From 99ad0c4ec46d4a1a37d6328639fc0e810bae550b Mon Sep 17 00:00:00 2001 From: Gionata Ghiggi Date: Wed, 5 Jun 2024 15:57:19 +0200 Subject: [PATCH] Update precommit rules and doc compilation (#217) * Update precommit rules and docs * Update docstrings * Fix dummy typo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yaml | 98 +- .github/workflows/lint.yml | 62 +- .github/workflows/release_to_pypi.yml | 112 +- .github/workflows/tests.yml | 154 +- .gitignore | 13 +- .pre-commit-config.yaml | 64 +- .readthedocs.yaml | 13 +- AUTHORS.md | 2 + CHANGELOG.md | 3 + CODE_OF_CONDUCT.md | 39 +- CONTRIBUTING.rst | 4 +- README.md | 55 +- SECURITY.md | 8 +- .../CAMPAIGN_NAME/metadata/station_name_1.yml | 81 +- .../CAMPAIGN_NAME/metadata/station_name_2.yml | 81 +- disdrodb/api/checks.py | 2 +- disdrodb/api/configs.py | 2 +- disdrodb/api/path.py | 8 +- disdrodb/l0/check_standards.py | 6 +- .../l0/configs/OTT_Parsivel/bins_diameter.yml | 132 +- .../l0/configs/OTT_Parsivel/bins_velocity.yml | 128 +- .../l0/configs/OTT_Parsivel/l0a_encodings.yml | 64 +- .../l0/configs/OTT_Parsivel/l0b_cf_attrs.yml | 42 +- .../l0/configs/OTT_Parsivel/l0b_encodings.yml | 14 +- .../configs/OTT_Parsivel/raw_data_format.yml | 154 +- .../configs/OTT_Parsivel2/bins_diameter.yml | 128 +- .../configs/OTT_Parsivel2/bins_velocity.yml | 128 +- .../configs/OTT_Parsivel2/l0a_encodings.yml | 78 +- .../l0/configs/OTT_Parsivel2/l0b_cf_attrs.yml | 46 +- .../configs/OTT_Parsivel2/l0b_encodings.yml | 14 +- .../configs/OTT_Parsivel2/raw_data_format.yml | 196 +- disdrodb/l0/configs/RD_80/bins_diameter.yml | 80 +- disdrodb/l0/configs/RD_80/l0a_encodings.yml | 32 +- disdrodb/l0/configs/RD_80/l0b_cf_attrs.yml | 4 +- disdrodb/l0/configs/RD_80/l0b_encodings.yml | 222 +- disdrodb/l0/configs/RD_80/raw_data_format.yml | 94 +- .../l0/configs/Thies_LPM/bins_diameter.yml | 88 +- .../l0/configs/Thies_LPM/bins_velocity.yml | 80 +- .../l0/configs/Thies_LPM/l0a_encodings.yml | 152 +- .../l0/configs/Thies_LPM/l0b_cf_attrs.yml | 121 +- .../l0/configs/Thies_LPM/l0b_encodings.yml | 6 +- .../l0/configs/Thies_LPM/raw_data_format.yml | 490 +- disdrodb/l0/l0_processing.py | 17 +- disdrodb/l0/l0a_processing.py | 20 +- disdrodb/l0/l0b_nc_processing.py | 14 +- disdrodb/l0/l0b_processing.py | 16 +- disdrodb/l0/template_tools.py | 22 +- .../Raw/EPFL/PARSIVEL_2007/metadata/10.yml | 89 +- .../Raw/UK/DIVEN/metadata/CAIRNGORM.yml | 65 +- .../CAMPAIGN_NAME/metadata/STATION_NAME.yml | 68 +- .../CAMPAIGN_NAME/metadata/STATION_NAME.yml | 102 +- disdrodb/tests/data/test_yaml/invalid.yaml | 2 - disdrodb/tests/data/test_yaml/valid.yaml | 18 +- .../CAMPAIGN_NAME/metadata/STATIONID.yml | 43 + .../CAMPAIGN_NAME/metadata/STATION_NAME.yml | 55 + .../fake_data_sample.parquet | Bin 0 -> 3009 bytes .../fake_data_sample_0.parquet | Bin 0 -> 3157 bytes .../fake_data_sample_1.parquet | Bin 0 -> 3157 bytes .../test_folders_files_creation/issue/123.yml | 1 + .../metadata/123.yml | 55 + .../mock_file_path.yml | 21 + disdrodb/tests/test_utils/test_utils_yaml.py | 6 - disdrodb/utils/compression.py | 1 + disdrodb/utils/logger.py | 14 +- disdrodb/utils/pandas.py | 2 +- disdrodb/utils/xarray.py | 6 +- docs/README.html | 16177 +++++++++++++++- docs/README.md | 26 +- docs/environment.yaml | 3 +- docs/source/conf.py | 129 +- docs/source/contribute_data.rst | 6 +- docs/source/index.rst | 4 - docs/source/installation.rst | 8 +- docs/source/reader_preparation.ipynb | 2679 --- docs/source/readers.rst | 2 +- docs/source/sensor_configs.rst | 2 +- docs/source/software_structure.rst | 4 +- pyproject.toml | 20 +- tutorials/reader_preparation.ipynb | 293 +- 79 files changed, 18129 insertions(+), 5161 deletions(-) delete mode 100644 disdrodb/tests/data/test_yaml/invalid.yaml create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATIONID.yml create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample.parquet create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_0.parquet create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_1.parquet create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/issue/123.yml create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/metadata/123.yml create mode 100644 disdrodb/tests/pytest_files/test_folders_files_creation/mock_file_path.yml delete mode 100644 docs/source/reader_preparation.ipynb diff --git a/.github/ISSUE_TEMPLATE/bug_report.yaml b/.github/ISSUE_TEMPLATE/bug_report.yaml index bb297209..91290caf 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yaml +++ b/.github/ISSUE_TEMPLATE/bug_report.yaml @@ -4,55 +4,55 @@ title: "[BUG] " labels: [bug, triage] assignees: ghiggi body: -- type: checkboxes - attributes: - label: Is there an existing issue for this? - description: Please search to see if an issue already exists for the bug you encountered. - options: - - label: I have searched the existing issues - required: true -- type: textarea - attributes: - label: Current Behavior - description: A concise description of what you're experiencing. - validations: - required: false -- type: textarea - attributes: - label: Expected Behavior - description: A concise description of what you expected to happen. - validations: - required: false -- type: textarea - attributes: - label: Steps To Reproduce - description: Steps to reproduce the behavior. - placeholder: | - 1. In this environment... - 2. With this config... - 3. Run '...' - 4. See error... - validations: - required: false -- type: textarea - attributes: - label: Environment - description: | - examples: - - **OS**: Ubuntu 20.04 - - **Python**: 3.8.2 - value: | + - type: checkboxes + attributes: + label: Is there an existing issue for this? + description: Please search to see if an issue already exists for the bug you encountered. + options: + - label: I have searched the existing issues + required: true + - type: textarea + attributes: + label: Current Behavior + description: A concise description of what you're experiencing. + validations: + required: false + - type: textarea + attributes: + label: Expected Behavior + description: A concise description of what you expected to happen. + validations: + required: false + - type: textarea + attributes: + label: Steps To Reproduce + description: Steps to reproduce the behavior. + placeholder: | + 1. In this environment... + 2. With this config... + 3. Run '...' + 4. See error... + validations: + required: false + - type: textarea + attributes: + label: Environment + description: | + examples: + - **OS**: Ubuntu 20.04 + - **Python**: 3.8.2 + value: | - OS: - python: - render: markdown - validations: - required: false -- type: textarea - attributes: - label: Anything else? - description: | - Links? References? Anything that will give us more context about the issue you are encountering! + render: markdown + validations: + required: false + - type: textarea + attributes: + label: Anything else? + description: | + Links? References? Anything that will give us more context about the issue you are encountering! - Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. - validations: - required: false + Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. + validations: + required: false diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 969e8cb9..59d13f58 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -1,31 +1,31 @@ -name: Lint -on: [push, pull_request, workflow_dispatch] - -jobs: - lint: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Setup Python - uses: actions/setup-python@v5 - with: - python-version: "3.8" - - - name: Ruff - if: success() || failure() - uses: pre-commit/action@v3.0.1 - with: - extra_args: ruff --all-files - - - name: Black - if: success() || failure() - uses: pre-commit/action@v3.0.1 - with: - extra_args: black --all-files --verbose - - - name: Blackdoc - if: success() || failure() - uses: pre-commit/action@v3.0.1 - with: - extra_args: blackdoc --all-files +name: Lint +on: [push, pull_request, workflow_dispatch] + +jobs: + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.8" + + - name: Ruff + if: success() || failure() + uses: pre-commit/action@v3.0.1 + with: + extra_args: ruff --all-files + + - name: Black + if: success() || failure() + uses: pre-commit/action@v3.0.1 + with: + extra_args: black --all-files --verbose + + - name: Blackdoc + if: success() || failure() + uses: pre-commit/action@v3.0.1 + with: + extra_args: blackdoc --all-files diff --git a/.github/workflows/release_to_pypi.yml b/.github/workflows/release_to_pypi.yml index 89bdaf9f..9dbce1ec 100644 --- a/.github/workflows/release_to_pypi.yml +++ b/.github/workflows/release_to_pypi.yml @@ -1,56 +1,56 @@ -name: Package Release to PyPI -on: - push: - tags: - - "v*" - -jobs: - build-n-publish: - name: Build and publish to PyPI - runs-on: ubuntu-latest - permissions: - contents: write - - steps: - - name: Checkout source - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.x" - - - name: Install pypa/build - run: >- - python -m - pip install build - - - name: Build a binary wheel - run: >- - python -m build - - - name: Publish distribution to PyPI - uses: pypa/gh-action-pypi-publish@release/v1 - with: - user: __token__ - password: ${{ secrets.PYPI_API_TOKEN }} - - - name: Get Asset name - if: success() || failure() - run: | - export PKG=$(ls dist/ | grep tar) - set -- $PKG - echo "asset_name=$1" >> $GITHUB_ENV - - - name: Create GitHub Release - if: success() || failure() - id: create_release - uses: softprops/action-gh-release@v2 - with: - name: ${{ github.ref }} - tag_name: ${{ github.ref }} - files: | - dist/${{ env.asset_name }} - draft: false - prerelease: false - token: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token +name: Package Release to PyPI +on: + push: + tags: + - "v*" + +jobs: + build-n-publish: + name: Build and publish to PyPI + runs-on: ubuntu-latest + permissions: + contents: write + + steps: + - name: Checkout source + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.x" + + - name: Install pypa/build + run: >- + python -m + pip install build + + - name: Build a binary wheel + run: >- + python -m build + + - name: Publish distribution to PyPI + uses: pypa/gh-action-pypi-publish@release/v1 + with: + user: __token__ + password: ${{ secrets.PYPI_API_TOKEN }} + + - name: Get Asset name + if: success() || failure() + run: | + export PKG=$(ls dist/ | grep tar) + set -- $PKG + echo "asset_name=$1" >> $GITHUB_ENV + + - name: Create GitHub Release + if: success() || failure() + id: create_release + uses: softprops/action-gh-release@v2 + with: + name: ${{ github.ref }} + tag_name: ${{ github.ref }} + files: | + dist/${{ env.asset_name }} + draft: false + prerelease: false + token: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 9eb0e3c1..e590bbad 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -1,77 +1,77 @@ -name: Tests -on: [push, pull_request, workflow_dispatch] - -jobs: - tests: - name: Test on ${{ matrix.os }}, Python ${{ matrix.python-version }}${{ matrix.experimental && ' with latest dependencies' || '' }} - runs-on: ${{ matrix.os }} - defaults: - run: - shell: bash -el {0} - - strategy: - fail-fast: false - matrix: - os: ["ubuntu-latest"] - python-version: ["3.9", "3.10", "3.11"] - experimental: [false] - include: - - os: "ubuntu-latest" - python-version: "3.11" - experimental: true - - - os: "ubuntu-latest" - python-version: "3.11" - experimental: false - coverage: true - - steps: - - uses: actions/checkout@v4 - - - name: Set up micromamba and install package dependencies - uses: mamba-org/setup-micromamba@v1 - with: - environment-file: ${{ matrix.experimental && 'ci/environment_latest.yml' || 'ci/environment.yml' }} - create-args: >- - python=${{ matrix.python-version }} - init-shell: >- - bash - - - name: Install the package - run: | - pip install -e . - - - name: Test with pytest - run: | - pytest # See pyproject.toml for cov options - - - name: Upload coverage reports to Codecov - uses: codecov/codecov-action@v4 - if: matrix.coverage - with: - token: ${{ secrets.CODECOV_TOKEN }} - files: coverage.xml - - - name: Upload coverage reports to Codacy - uses: codacy/codacy-coverage-reporter-action@v1 - if: matrix.coverage && github.ref == 'refs/heads/main' - with: - project-token: ${{ secrets.CODACY_PROJECT_TOKEN }} - coverage-reports: coverage.xml - - - name: Coveralls Parallel - uses: coverallsapp/github-action@v2 - if: matrix.coverage - with: - parallel: true - files: lcov.info - - end-coveralls: - needs: [tests] - if: ${{ always() }} - runs-on: ubuntu-latest - steps: - - name: Coveralls Finished - uses: coverallsapp/github-action@v2 - with: - parallel-finished: true +name: Tests +on: [push, pull_request, workflow_dispatch] + +jobs: + tests: + name: Test on ${{ matrix.os }}, Python ${{ matrix.python-version }}${{ matrix.experimental && ' with latest dependencies' || '' }} + runs-on: ${{ matrix.os }} + defaults: + run: + shell: bash -el {0} + + strategy: + fail-fast: false + matrix: + os: ["ubuntu-latest"] + python-version: ["3.9", "3.10", "3.11"] + experimental: [false] + include: + - os: "ubuntu-latest" + python-version: "3.11" + experimental: true + + - os: "ubuntu-latest" + python-version: "3.11" + experimental: false + coverage: true + + steps: + - uses: actions/checkout@v4 + + - name: Set up micromamba and install package dependencies + uses: mamba-org/setup-micromamba@v1 + with: + environment-file: ${{ matrix.experimental && 'ci/environment_latest.yml' || 'ci/environment.yml' }} + create-args: >- + python=${{ matrix.python-version }} + init-shell: >- + bash + + - name: Install the package + run: | + pip install -e . + + - name: Test with pytest + run: | + pytest # See pyproject.toml for cov options + + - name: Upload coverage reports to Codecov + uses: codecov/codecov-action@v4 + if: matrix.coverage + with: + token: ${{ secrets.CODECOV_TOKEN }} + files: coverage.xml + + - name: Upload coverage reports to Codacy + uses: codacy/codacy-coverage-reporter-action@v1 + if: matrix.coverage && github.ref == 'refs/heads/main' + with: + project-token: ${{ secrets.CODACY_PROJECT_TOKEN }} + coverage-reports: coverage.xml + + - name: Coveralls Parallel + uses: coverallsapp/github-action@v2 + if: matrix.coverage + with: + parallel: true + files: lcov.info + + end-coveralls: + needs: [tests] + if: ${{ always() }} + runs-on: ubuntu-latest + steps: + - name: Coveralls Finished + uses: coverallsapp/github-action@v2 + with: + parallel-finished: true diff --git a/.gitignore b/.gitignore index c80dff3f..3cd73988 100644 --- a/.gitignore +++ b/.gitignore @@ -44,6 +44,10 @@ share/python-wheels/ MANIFEST _version.py +# Ignore specific files and directories +docs/source/tutorials/ +docs/build/ + # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. @@ -160,12 +164,3 @@ cython_debug/ train_interactive.py tmp_fig.py log.txt - -# Ignore specific files -docs\source\reader_preparation.ipynb -\docs\source\reader_preparation.ipynb -/docs/source/csv/*.csv - -# Ignore specific directories -/disdrodb/tests/temp/ -disdrodb-dev diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 8976d7a6..23af6525 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,31 +1,71 @@ +--- repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.5.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - + - id: check-yaml + - id: check-toml + - id: check-symlinks + - id: check-docstring-first + - id: check-ast + - id: check-added-large-files - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.3.4 + rev: v0.3.5 hooks: - id: ruff args: [--fix] - - repo: https://github.com/psf/black rev: 24.3.0 hooks: - - id: black - language_version: python3 - + - id: black + language_version: python3 - repo: https://github.com/keewis/blackdoc rev: v0.3.9 hooks: - - id: blackdoc - additional_dependencies: ["black[jupyter]"] - + - id: blackdoc + additional_dependencies: ["black[jupyter]"] + - repo: https://github.com/pre-commit/mirrors-prettier + rev: "v3.1.0" + hooks: + - id: prettier + types_or: [yaml, html, css, scss, javascript, json] # markdown to avoid conflicts with mdformat - repo: https://github.com/codespell-project/codespell rev: v2.2.6 hooks: - - id: codespell - types_or: [python, markdown, rst] - additional_dependencies: [tomli] + - id: codespell + types_or: [python, markdown, rst] + additional_dependencies: [tomli] + - repo: https://github.com/asottile/pyupgrade + rev: v3.15.2 + hooks: + - id: pyupgrade + - repo: https://github.com/MarcoGorelli/madforhooks + rev: 0.4.1 + hooks: + # - id: conda-env-sorter # conflicts with prettier + - id: check-execution-order + - repo: https://github.com/executablebooks/mdformat + rev: 0.7.17 + hooks: + - id: mdformat + additional_dependencies: [mdformat-gfm, mdformat-black] + exclude: ^.github/ # this avoid the hook to wrongly reformats md issue template files + - repo: https://github.com/kynan/nbstripout + rev: 0.7.1 + hooks: + - id: nbstripout + args: [--keep-output] + - repo: https://github.com/nbQA-dev/nbQA + rev: 1.8.5 + hooks: + - id: nbqa-black + - id: nbqa-ruff + args: [--fix, "--ignore=E402,B018"] + - repo: https://github.com/pre-commit/pygrep-hooks + rev: "v1.10.0" + hooks: + - id: rst-backticks + - id: rst-directive-colons + - id: rst-inline-touching-normal diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 3119d645..ce1a661c 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -7,7 +7,6 @@ build: tools: python: "mambaforge-22.9" - sphinx: configuration: docs/source/conf.py @@ -15,14 +14,12 @@ sphinx: formats: - pdf -# Optional but recommended, declare the Python requirements required -# to build your documentation +# Declare the Python requirements required to build your documentation # See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html - -# python: -# install: -# - requirements: docs/requirements.txt - # See https://docs.readthedocs.io/en/stable/guides/conda.html#making-builds-faster-with-mamba conda: environment: docs/environment.yaml +python: + install: + - method: pip + path: . diff --git a/AUTHORS.md b/AUTHORS.md index 326af451..284f298b 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -3,7 +3,9 @@ The following people have made contributions to this project: <!--- Use your GitHub account or any other personal reference URL ---> + <!--- If you wish to not use your real name, please use your github username ---> + <!--- The list should be alphabetical by last name if possible, with github usernames at the bottom and the istitution ---> <!--- See https://gist.github.com/djhoese/52220272ec73b12eb8f4a29709be110d for auto-generating parts of this list ---> diff --git a/CHANGELOG.md b/CHANGELOG.md index 800520df..39f48990 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,12 +3,15 @@ ## Version 0.0.2 - Glorious Green - XXXX-XX-XX ### Enhancements + - TODO ### Changes + - TODO ### Bugfixes + - TODO ## Version 0.0.1 - DISDRODB Birth Date - 2023-01-30 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 25959d09..259566b2 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,4 +1,3 @@ - # Contributor Covenant Code of Conduct ## Our Pledge @@ -18,22 +17,22 @@ diverse, inclusive, and healthy community. Examples of behavior that contributes to a positive environment for our community include: -* Demonstrating empathy and kindness toward other people -* Being respectful of differing opinions, viewpoints, and experiences -* Giving and gracefully accepting constructive feedback -* Accepting responsibility and apologizing to those affected by our mistakes, - and learning from the experience -* Focusing on what is best not just for us as individuals, but for the overall community +- Demonstrating empathy and kindness toward other people +- Being respectful of differing opinions, viewpoints, and experiences +- Giving and gracefully accepting constructive feedback +- Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +- Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: -* The use of sexualized language or imagery, and sexual attention or advances of any kind -* Trolling, insulting or derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or email address, - without their explicit permission -* Other conduct which could reasonably be considered inappropriate in a - professional setting +- The use of sexualized language or imagery, and sexual attention or advances of any kind +- Trolling, insulting or derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or email address, + without their explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting ## Enforcement Responsibilities @@ -59,7 +58,7 @@ representative at an online or offline event. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at -[INSERT CONTACT METHOD]. +\[INSERT CONTACT METHOD\]. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the @@ -118,14 +117,14 @@ version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. Community Impact Guidelines were inspired by -[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. +[Mozilla's code of conduct enforcement ladder][mozilla coc]. For answers to common questions about this code of conduct, see the FAQ at -[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at +[https://www.contributor-covenant.org/faq][faq]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. +[faq]: https://www.contributor-covenant.org/faq [homepage]: https://www.contributor-covenant.org -[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html -[Mozilla CoC]: https://github.com/mozilla/diversity -[FAQ]: https://www.contributor-covenant.org/faq +[mozilla coc]: https://github.com/mozilla/diversity [translations]: https://www.contributor-covenant.org/translations +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 61deb3ef..470642f9 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -100,7 +100,7 @@ contribute to the project! Each contribution should be made in a separate new branch of your forked repository. For example, if you plan to contribute with new readers, please create a branch for every single reader. -Working on the main branch is reserved for `Core Contributors` only. +Working on the main branch is reserved for *Core Contributors* only. Core Contributors are developers that actively work and maintain the repository. They are the only ones who accept Pull Requests and push commits directly to the DISDRODB repository. @@ -135,7 +135,7 @@ We follow the `PEP 8 <https://pep8.org/>`__ style guide for python code. Another relevant style guide can be found in the `The Hitchhiker's Guide to Python <https://docs.python-guide.org/writing/style/>`__. To ensure a minimal style consistency, we use `black <https://black.readthedocs.io/en/stable/>`__ to auto-format the source code. -The `black` configuration used in the DISDRODB project is +The *black* configuration used in the DISDRODB project is defined in the `pyproject.toml <https://github.com/ltelab/disdrodb/blob/main/pyproject.toml>`__. diff --git a/README.md b/README.md index 8380e487..d4dc6a5c 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,22 @@ # πŸ“¦ DISDRODB - A package to standardize, process and analyze global disdrometer data. -| | | -| -------------------- | ---------------------------------------------- | -| Deployment | [![PyPI](https://badge.fury.io/py/disdrodb.svg?style=flat)](https://pypi.org/project/disdrodb/) [![Conda](https://img.shields.io/conda/vn/conda-forge/disdrodb.svg?logo=conda-forge&logoColor=white&style=flat)](https://anaconda.org/conda-forge/disdrodb) | -| Activity | [![PyPI Downloads](https://img.shields.io/pypi/dm/disdrodb.svg?label=PyPI%20downloads&style=flat)](https://pypi.org/project/disdrodb/) [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/disdrodb.svg?label=Conda%20downloads&style=flat)](https://anaconda.org/conda-forge/disdrodb) | -| Python Versions | [![Python Versions](https://img.shields.io/badge/Python-3.8%20%203.9%20%203.10%20%203.11%20%203.12-blue?style=flat)](https://www.python.org/downloads/) | -| Supported Systems | [![Linux](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests.yml?label=Linux&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![macOS](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests.yml?label=macOS&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![Windows](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests_windows.yml?label=Windows&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests_windows.yml) | -| Project Status | [![Project Status](https://www.repostatus.org/badges/latest/active.svg?style=flat)](https://www.repostatus.org/#active) | -| Build Status | [![Tests](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml/badge.svg?style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![Lint](https://github.com/ltelab/disdrodb/actions/workflows/lint.yml/badge.svg?style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/lint.yml) [![Docs](https://readthedocs.org/projects/disdrodb/badge/?version=latest&style=flat)](https://disdrodb.readthedocs.io/en/latest/) | -| Linting | [![Black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat)](https://github.com/psf/black) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json&style=flat)](https://github.com/astral-sh/ruff) [![Codespell](https://img.shields.io/badge/Codespell-enabled-brightgreen?style=flat)](https://github.com/codespell-project/codespell) | -| Code Coverage | [![Coveralls](https://coveralls.io/repos/github/ltelab/disdrodb/badge.svg?branch=main&style=flat)](https://coveralls.io/github/ltelab/disdrodb?branch=main) [![Codecov](https://codecov.io/gh/ltelab/disdrodb/branch/main/graph/badge.svg?style=flat)](https://codecov.io/gh/ltelab/disdrodb) | -| Code Quality | [![Codefactor](https://www.codefactor.io/repository/github/ltelab/disdrodb/badge?style=flat)](https://www.codefactor.io/repository/github/ltelab/disdrodb) [![Codebeat](https://codebeat.co/badges/14ff831b-f064-4bdd-a2e2-72ffdf28a35a?style=flat)](https://codebeat.co/projects/github-com-ltelab-disdrodb-main) [![Codacy](https://app.codacy.com/project/badge/Grade/d823c50a7ad14268bd347b5aba384623?style=flat)](https://app.codacy.com/gh/ltelab/disdrodb/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![Codescene](https://codescene.io/projects/36773/status-badges/code-health?style=flat)](https://codescene.io/projects/36773) | -| Code Review | [![pyOpenSci](https://tinyurl.com/XXXX)](#) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/XXXX/badge?style=flat)](#) | -| License | [![License](https://img.shields.io/github/license/ltelab/disdrodb?style=flat)](https://github.com/ltelab/disdrodb/blob/main/LICENSE) | -| Community | [![Slack](https://img.shields.io/badge/Slack-disdrodb-green.svg?logo=slack&style=flat)](https://join.slack.com/t/disdrodbworkspace/shared_invite/zt-25l4mvgo7-cfBdXalzlWGd4Pt7H~FqoA) [![GitHub Discussions](https://img.shields.io/badge/GitHub-Discussions-green?logo=github&style=flat)](https://github.com/ltelab/disdrodb/discussions) | -| Citation | [![JOSS](http://joss.theoj.org/papers/<DOI>/joss.<DOI>/status.svg?style=flat)](#) [![DOI](https://zenodo.org/badge/429018433.svg?style=flat)](https://zenodo.org/doi/10.5281/zenodo.7680581) | - - [**Slack**](https://join.slack.com/t/disdrodbworkspace/shared_invite/zt-25l4mvgo7-cfBdXalzlWGd4Pt7H~FqoA) | [**Docs**](https://disdrodb.readthedocs.io/en/latest/) +| | | +| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deployment | [![PyPI](https://badge.fury.io/py/disdrodb.svg?style=flat)](https://pypi.org/project/disdrodb/) [![Conda](https://img.shields.io/conda/vn/conda-forge/disdrodb.svg?logo=conda-forge&logoColor=white&style=flat)](https://anaconda.org/conda-forge/disdrodb) | +| Activity | [![PyPI Downloads](https://img.shields.io/pypi/dm/disdrodb.svg?label=PyPI%20downloads&style=flat)](https://pypi.org/project/disdrodb/) [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/disdrodb.svg?label=Conda%20downloads&style=flat)](https://anaconda.org/conda-forge/disdrodb) | +| Python Versions | [![Python Versions](https://img.shields.io/badge/Python-3.8%20%203.9%20%203.10%20%203.11%20%203.12-blue?style=flat)](https://www.python.org/downloads/) | +| Supported Systems | [![Linux](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests.yml?label=Linux&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![macOS](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests.yml?label=macOS&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![Windows](https://img.shields.io/github/actions/workflow/status/ltelab/disdrodb/.github/workflows/tests_windows.yml?label=Windows&style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests_windows.yml) | +| Project Status | [![Project Status](https://www.repostatus.org/badges/latest/active.svg?style=flat)](https://www.repostatus.org/#active) | +| Build Status | [![Tests](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml/badge.svg?style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/tests.yml) [![Lint](https://github.com/ltelab/disdrodb/actions/workflows/lint.yml/badge.svg?style=flat)](https://github.com/ltelab/disdrodb/actions/workflows/lint.yml) [![Docs](https://readthedocs.org/projects/disdrodb/badge/?version=latest&style=flat)](https://disdrodb.readthedocs.io/en/latest/) | +| Linting | [![Black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat)](https://github.com/psf/black) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json&style=flat)](https://github.com/astral-sh/ruff) [![Codespell](https://img.shields.io/badge/Codespell-enabled-brightgreen?style=flat)](https://github.com/codespell-project/codespell) | +| Code Coverage | [![Coveralls](https://coveralls.io/repos/github/ltelab/disdrodb/badge.svg?branch=main&style=flat)](https://coveralls.io/github/ltelab/disdrodb?branch=main) [![Codecov](https://codecov.io/gh/ltelab/disdrodb/branch/main/graph/badge.svg?style=flat)](https://codecov.io/gh/ltelab/disdrodb) | +| Code Quality | [![Codefactor](https://www.codefactor.io/repository/github/ltelab/disdrodb/badge?style=flat)](https://www.codefactor.io/repository/github/ltelab/disdrodb) [![Codebeat](https://codebeat.co/badges/14ff831b-f064-4bdd-a2e2-72ffdf28a35a?style=flat)](https://codebeat.co/projects/github-com-ltelab-disdrodb-main) [![Codacy](https://app.codacy.com/project/badge/Grade/d823c50a7ad14268bd347b5aba384623?style=flat)](https://app.codacy.com/gh/ltelab/disdrodb/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![Codescene](https://codescene.io/projects/36773/status-badges/code-health?style=flat)](https://codescene.io/projects/36773) | +| Code Review | [![pyOpenSci](https://tinyurl.com/XXXX)](#) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/XXXX/badge?style=flat)](#) | +| License | [![License](https://img.shields.io/github/license/ltelab/disdrodb?style=flat)](https://github.com/ltelab/disdrodb/blob/main/LICENSE) | +| Community | [![Slack](https://img.shields.io/badge/Slack-disdrodb-green.svg?logo=slack&style=flat)](https://join.slack.com/t/disdrodbworkspace/shared_invite/zt-25l4mvgo7-cfBdXalzlWGd4Pt7H~FqoA) [![GitHub Discussions](https://img.shields.io/badge/GitHub-Discussions-green?logo=github&style=flat)](https://github.com/ltelab/disdrodb/discussions) | +| Citation | [![JOSS](http://joss.theoj.org/papers/%3CDOI%3E/joss.%3CDOI%3E/status.svg?style=flat)](#) [![DOI](https://zenodo.org/badge/429018433.svg?style=flat)](https://zenodo.org/doi/10.5281/zenodo.7680581) | + +[**Slack**](https://join.slack.com/t/disdrodbworkspace/shared_invite/zt-25l4mvgo7-cfBdXalzlWGd4Pt7H~FqoA) | [**Docs**](https://disdrodb.readthedocs.io/en/latest/) DISDRODB is part of an initial effort to index, collect and homogenize drop size distribution (DSD) data sets across the globe, as well as to establish a global standard for disdrometers observations data sharing. @@ -53,7 +53,7 @@ Then clone the DISDRODB Metadata Archive repository with: git clone https://github.com/ltelab/disdrodb-data.git ``` -This will create a directory called ``disdrodb-data``, which is ready to be filled with data from the DISDRODB Decentralized Data Archive. +This will create a directory called `disdrodb-data`, which is ready to be filled with data from the DISDRODB Decentralized Data Archive. But before starting to download some data, we need to specify the location of the DISDRODB Local Archive. @@ -61,6 +61,7 @@ You can specify once forever the default DISDRODB Local Archive directory by run ```python import disdrodb + base_dir = "<path_to>/disdrodb-data/DISDRODB>" disdrodb.define_configs(base_dir=base_dir) ``` @@ -111,9 +112,9 @@ is hosted on GitHub at [https://github.com/ltelab/disdrodb-data](https://github. DISDRODB can be installed from PyPI with pip: - ```bash - pip install disdrodb - ``` +```bash +pip install disdrodb +``` ## πŸ’­ Feedback and Contributing Guidelines @@ -125,12 +126,12 @@ Feel free to also open a [GitHub Issue](https://github.com/ltelab/disdrodb/issue ## ✍️ Contributors -* [Gionata Ghiggi](https://people.epfl.ch/gionata.ghiggi) -* [Kim Candolfi](https://github.com/KimCandolfi) -* [RΓ©gis Longchamp](https://people.epfl.ch/regis.longchamp) -* [Charlotte GisΓ¨le Weil](https://people.epfl.ch/charlotte.weil) -* [Jacopo Grazioli](https://people.epfl.ch/jacopo.grazioli) -* [Alexis Berne](https://people.epfl.ch/alexis.berne?lang=en) +- [Gionata Ghiggi](https://people.epfl.ch/gionata.ghiggi) +- [Kim Candolfi](https://github.com/KimCandolfi) +- [RΓ©gis Longchamp](https://people.epfl.ch/regis.longchamp) +- [Charlotte GisΓ¨le Weil](https://people.epfl.ch/charlotte.weil) +- [Jacopo Grazioli](https://people.epfl.ch/jacopo.grazioli) +- [Alexis Berne](https://people.epfl.ch/alexis.berne?lang=en) ## Citation diff --git a/SECURITY.md b/SECURITY.md index b3b5c7e6..040e8ac2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -7,10 +7,10 @@ guarantee that releases before 1.0 will see security updates except for the most release. After 1.0, you can expect more stability in the interfaces and security fixes to be backported more regularly. -| Version | Supported | -| ------- | ------------------ | -| 0.x.x (latest) | :white_check_mark: | -| < 0.0.17 | :x: | +| Version | Supported | +| -------------- | ------------------ | +| 0.x.x (latest) | :white_check_mark: | +| \< 0.0.17 | :x: | ## Unsafe YAML Loading diff --git a/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_1.yml b/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_1.yml index 35f2519b..58d5240e 100644 --- a/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_1.yml +++ b/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_1.yml @@ -5,53 +5,54 @@ sensor_name: OTT_Parsivel reader: EPFL/LOCARNO_2018 raw_data_format: raw platform_type: fixed -disdrodb_data_url: '' -source: '' -source_convention: '' -source_processing_date: '' +disdrodb_data_url: "" +source: "" +source_convention: "" +source_processing_date: "" title: LOCARNO_2018 -description: '' -history: '' -keywords: '' -comment: '' -project_name: '' -station_id: '60' +description: "" +history: "" +keywords: "" +comment: "" +project_name: "" +station_id: "60" location: Cimetta country: Switzerland continent: Europe latitude: 46.2005117 longitude: 8.7916367 altitude: 1671 -deployment_status: '' -deployment mode: '' -platform_protection: '' -platform_orientation: '' +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" sensor_long_name: OTT Hydromet Parsivel -sensor_manufacturer: '' -sensor_wavelength: '' -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_length: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -calibration_sensitivity: '' -calibration_certification_date: '' -calibration_certification_url: '' -contributors: '' -authors: '' -authors_url: '' -contact: '' +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" contact_information: http://lte.epfl.ch -acknowledgement: '' -references: '' -documentation: '' -website: '' -institution: Laboratoire de Teledetection Environnementale - Ecole Polytechnique +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: + Laboratoire de Teledetection Environnementale - Ecole Polytechnique Federale de Lausanne -source_repository: '' -license: '' -doi: '' -summary: '' +source_repository: "" +license: "" +doi: "" +summary: "" diff --git a/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_2.yml b/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_2.yml index d01b54b1..12e2198d 100644 --- a/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_2.yml +++ b/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/station_name_2.yml @@ -5,53 +5,54 @@ sensor_name: OTT_Parsivel reader: EPFL/LOCARNO_2018 raw_data_format: raw platform_type: fixed -disdrodb_data_url: '' -source: '' -source_convention: '' -source_processing_date: '' +disdrodb_data_url: "" +source: "" +source_convention: "" +source_processing_date: "" title: LOCARNO_2018 -description: '' -history: '' -keywords: '' -comment: '' -project_name: '' -station_id: '61' +description: "" +history: "" +keywords: "" +comment: "" +project_name: "" +station_id: "61" location: Locarno-Monti country: Switzerland continent: Europe latitude: 46.17259 longitude: 8.7874267 altitude: -9999 -deployment_status: '' -deployment mode: '' -platform_protection: '' -platform_orientation: '' +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" sensor_long_name: OTT Hydromet Parsivel -sensor_manufacturer: '' -sensor_wavelength: '' -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_length: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -calibration_sensitivity: '' -calibration_certification_date: '' -calibration_certification_url: '' -contributors: '' -authors: '' -authors_url: '' -contact: '' +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" contact_information: http://lte.epfl.ch -acknowledgement: '' -references: '' -documentation: '' -website: '' -institution: Laboratoire de Teledetection Environnementale - Ecole Polytechnique +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: + Laboratoire de Teledetection Environnementale - Ecole Polytechnique Federale de Lausanne -source_repository: '' -license: '' -doi: '' -summary: '' +source_repository: "" +license: "" +doi: "" +summary: "" diff --git a/disdrodb/api/checks.py b/disdrodb/api/checks.py index 54a97fba..de5b5296 100644 --- a/disdrodb/api/checks.py +++ b/disdrodb/api/checks.py @@ -323,7 +323,7 @@ def check_valid_campaign_dir(campaign_dir): Used to check validity of ``raw_dir`` and ``processed_dir``. - The path must be ``*/DISDRODB/<Raw/Processed>/<DATA_SOURCE>/<CAMPAIGN_NAME>`` + The path must be ``/DISDRODB/<Raw/Processed>/<DATA_SOURCE>/<CAMPAIGN_NAME>``. """ last_component = os.path.basename(campaign_dir) tree_components = infer_disdrodb_tree_path_components(campaign_dir) diff --git a/disdrodb/api/configs.py b/disdrodb/api/configs.py index e335b173..c8f15cdc 100644 --- a/disdrodb/api/configs.py +++ b/disdrodb/api/configs.py @@ -93,7 +93,7 @@ def read_config_file(sensor_name: str, product: str, filename: str) -> dict: return dictionary -def available_sensor_names(product: str = "L0A") -> sorted: +def available_sensor_names(product: str = "L0A") -> list: """Get available names of sensors. Returns diff --git a/disdrodb/api/path.py b/disdrodb/api/path.py index c674ab25..ab4c6f0d 100644 --- a/disdrodb/api/path.py +++ b/disdrodb/api/path.py @@ -396,7 +396,7 @@ def define_l0a_filename(df, processed_dir, station_name: str) -> str: Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame L0A DataFrame processed_dir : str Path of the processed directory @@ -425,7 +425,7 @@ def define_l0b_filename(ds, processed_dir, station_name: str) -> str: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset L0B xarray Dataset processed_dir : str Path of the processed directory @@ -454,7 +454,7 @@ def define_l0a_filepath(df: pd.DataFrame, processed_dir: str, station_name: str) Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame L0A DataFrame. processed_dir : str Path of the processed directory. @@ -477,7 +477,7 @@ def define_l0b_filepath(ds: xr.Dataset, processed_dir: str, station_name: str, l Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset L0B xarray Dataset. processed_dir : str Path of the processed directory. diff --git a/disdrodb/l0/check_standards.py b/disdrodb/l0/check_standards.py index bd2c4307..88c30c5a 100644 --- a/disdrodb/l0/check_standards.py +++ b/disdrodb/l0/check_standards.py @@ -90,7 +90,7 @@ def _check_raw_fields_available(df: pd.DataFrame, sensor_name: str, verbose: boo Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Dataframe sensor_name : str Name of the sensor. @@ -124,7 +124,7 @@ def check_l0a_column_names(df: pd.DataFrame, sensor_name: str) -> None: Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -165,7 +165,7 @@ def check_l0a_standards(df: pd.DataFrame, sensor_name: str, verbose: bool = True Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame L0A dataframe. sensor_name : str Name of the sensor. diff --git a/disdrodb/l0/configs/OTT_Parsivel/bins_diameter.yml b/disdrodb/l0/configs/OTT_Parsivel/bins_diameter.yml index 39a97582..3da3a362 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/bins_diameter.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/bins_diameter.yml @@ -1,5 +1,5 @@ center: - 0: 0.062 # 0.0625 ? + 0: 0.062 # 0.0625 ? 1: 0.187 2: 0.312 3: 0.437 @@ -8,7 +8,7 @@ center: 6: 0.812 7: 0.937 8: 1.062 - 9: 1.187 # 1.1875 + 9: 1.187 # 1.1875 10: 1.375 11: 1.625 12: 1.875 @@ -33,101 +33,101 @@ center: 31: 24.5 bounds: 0: - - 0.0 - - 0.1245 + - 0.0 + - 0.1245 1: - - 0.1245 - - 0.2495 + - 0.1245 + - 0.2495 2: - - 0.2495 - - 0.3745 + - 0.2495 + - 0.3745 3: - - 0.3745 - - 0.4995 + - 0.3745 + - 0.4995 4: - - 0.4995 - - 0.6245 + - 0.4995 + - 0.6245 5: - - 0.6245 - - 0.7495 + - 0.6245 + - 0.7495 6: - - 0.7495 - - 0.8745 + - 0.7495 + - 0.8745 7: - - 0.8745 - - 0.9995 + - 0.8745 + - 0.9995 8: - - 0.9995 - - 1.1245 + - 0.9995 + - 1.1245 9: - - 1.1245 - - 1.25 + - 1.1245 + - 1.25 10: - - 1.25 - - 1.5 + - 1.25 + - 1.5 11: - - 1.5 - - 1.75 + - 1.5 + - 1.75 12: - - 1.75 - - 2.0 + - 1.75 + - 2.0 13: - - 2.0 - - 2.25 + - 2.0 + - 2.25 14: - - 2.25 - - 2.5 + - 2.25 + - 2.5 15: - - 2.5 - - 3.0 + - 2.5 + - 3.0 16: - - 3.0 - - 3.5 + - 3.0 + - 3.5 17: - - 3.5 - - 4.0 + - 3.5 + - 4.0 18: - - 4.0 - - 4.5 + - 4.0 + - 4.5 19: - - 4.5 - - 5.0 + - 4.5 + - 5.0 20: - - 5.0 - - 6.0 + - 5.0 + - 6.0 21: - - 6.0 - - 7.0 + - 6.0 + - 7.0 22: - - 7.0 - - 8.0 + - 7.0 + - 8.0 23: - - 8.0 - - 9.0 + - 8.0 + - 9.0 24: - - 9.0 - - 10.0 + - 9.0 + - 10.0 25: - - 10.0 - - 12.0 + - 10.0 + - 12.0 26: - - 12.0 - - 14.0 + - 12.0 + - 14.0 27: - - 14.0 - - 16.0 + - 14.0 + - 16.0 28: - - 16.0 - - 18.0 + - 16.0 + - 18.0 29: - - 18.0 - - 20.0 + - 18.0 + - 20.0 30: - - 20.0 - - 23.0 + - 20.0 + - 23.0 31: - - 23.0 - - 26.0 + - 23.0 + - 26.0 width: 0: 0.125 # 0.1245 ? 1: 0.125 diff --git a/disdrodb/l0/configs/OTT_Parsivel/bins_velocity.yml b/disdrodb/l0/configs/OTT_Parsivel/bins_velocity.yml index 5ea05595..07f591b1 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/bins_velocity.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/bins_velocity.yml @@ -33,101 +33,101 @@ center: 31: 20.8 bounds: 0: - - 0.0 - - 0.1 + - 0.0 + - 0.1 1: - - 0.1 - - 0.2 + - 0.1 + - 0.2 2: - - 0.2 - - 0.3 + - 0.2 + - 0.3 3: - - 0.3 - - 0.4 + - 0.3 + - 0.4 4: - - 0.4 - - 0.5 + - 0.4 + - 0.5 5: - - 0.5 - - 0.6 + - 0.5 + - 0.6 6: - - 0.6 - - 0.7 + - 0.6 + - 0.7 7: - - 0.7 - - 0.8 + - 0.7 + - 0.8 8: - - 0.8 - - 0.9 + - 0.8 + - 0.9 9: - - 0.9 - - 1.0 + - 0.9 + - 1.0 10: - - 1.0 - - 1.2 + - 1.0 + - 1.2 11: - - 1.2 - - 1.4 + - 1.2 + - 1.4 12: - - 1.4 - - 1.6 + - 1.4 + - 1.6 13: - - 1.6 - - 1.8 + - 1.6 + - 1.8 14: - - 1.8 - - 2.0 + - 1.8 + - 2.0 15: - - 2.0 - - 2.4 + - 2.0 + - 2.4 16: - - 2.4 - - 2.8 + - 2.4 + - 2.8 17: - - 2.8 - - 3.2 + - 2.8 + - 3.2 18: - - 3.2 - - 3.6 + - 3.2 + - 3.6 19: - - 3.6 - - 4.0 + - 3.6 + - 4.0 20: - - 4.0 - - 4.8 + - 4.0 + - 4.8 21: - - 4.8 - - 5.6 + - 4.8 + - 5.6 22: - - 5.6 - - 6.4 + - 5.6 + - 6.4 23: - - 6.4 - - 7.2 + - 6.4 + - 7.2 24: - - 7.2 - - 8.0 + - 7.2 + - 8.0 25: - - 8.0 - - 9.6 + - 8.0 + - 9.6 26: - - 9.6 - - 11.2 + - 9.6 + - 11.2 27: - - 11.2 - - 12.8 + - 11.2 + - 12.8 28: - - 12.8 - - 14.4 + - 12.8 + - 14.4 29: - - 14.4 - - 16.0 + - 14.4 + - 16.0 30: - - 16.0 - - 19.2 + - 16.0 + - 19.2 31: - - 19.2 - - 22.4 + - 19.2 + - 22.4 width: 0: 0.1 1: 0.1 diff --git a/disdrodb/l0/configs/OTT_Parsivel/l0a_encodings.yml b/disdrodb/l0/configs/OTT_Parsivel/l0a_encodings.yml index cafd4e33..48d2c1a0 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/l0a_encodings.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/l0a_encodings.yml @@ -1,32 +1,32 @@ -rainfall_rate_32bit: 'float32' -rainfall_accumulated_32bit: 'float32' -weather_code_synop_4680: 'float32' # 'uint32' -weather_code_synop_4677: 'float32' # 'uint32' -weather_code_metar_4678: 'str' -weather_code_nws: 'str' -reflectivity_32bit: 'float32' -mor_visibility: 'float32' # 'uint16' -sample_interval: 'float32' # 'uint16' -laser_amplitude: 'float32' # 'uint32' -number_particles: 'float32' # 'uint32' -sensor_temperature: 'float32' # 'int8' -sensor_serial_number: 'str' -firmware_iop: 'str' -firmware_dsp: 'str' -sensor_heating_current: 'float32' -sensor_battery_voltage: 'float32' -sensor_status: 'float32' # 'uint8' -start_time: 'str' -sensor_time: 'str' -sensor_date: 'str' -station_name: 'str' -station_number: 'str' -rainfall_amount_absolute_32bit: 'float32' -error_code: 'float32' # 'uint8' -rainfall_rate_16bit: 'float32' -rainfall_rate_12bit: 'float32' -rainfall_accumulated_16bit: 'float32' -reflectivity_16bit: 'float32' -raw_drop_concentration: 'str' -raw_drop_average_velocity: 'str' -raw_drop_number: 'str' +rainfall_rate_32bit: "float32" +rainfall_accumulated_32bit: "float32" +weather_code_synop_4680: "float32" # 'uint32' +weather_code_synop_4677: "float32" # 'uint32' +weather_code_metar_4678: "str" +weather_code_nws: "str" +reflectivity_32bit: "float32" +mor_visibility: "float32" # 'uint16' +sample_interval: "float32" # 'uint16' +laser_amplitude: "float32" # 'uint32' +number_particles: "float32" # 'uint32' +sensor_temperature: "float32" # 'int8' +sensor_serial_number: "str" +firmware_iop: "str" +firmware_dsp: "str" +sensor_heating_current: "float32" +sensor_battery_voltage: "float32" +sensor_status: "float32" # 'uint8' +start_time: "str" +sensor_time: "str" +sensor_date: "str" +station_name: "str" +station_number: "str" +rainfall_amount_absolute_32bit: "float32" +error_code: "float32" # 'uint8' +rainfall_rate_16bit: "float32" +rainfall_rate_12bit: "float32" +rainfall_accumulated_16bit: "float32" +reflectivity_16bit: "float32" +raw_drop_concentration: "str" +raw_drop_average_velocity: "str" +raw_drop_number: "str" diff --git a/disdrodb/l0/configs/OTT_Parsivel/l0b_cf_attrs.yml b/disdrodb/l0/configs/OTT_Parsivel/l0b_cf_attrs.yml index c3a761aa..b2d47401 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/l0b_cf_attrs.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/l0b_cf_attrs.yml @@ -9,19 +9,19 @@ rainfall_accumulated_32bit: weather_code_synop_4680: description: SYNOP weather code according to table 4680 of Parsivel documentation long_name: Weather code SYNOP 4680 - units: '' + units: "" weather_code_synop_4677: description: SYNOP weather code according to table 4677 of Parsivel documentation long_name: Weather code SYNOP 4677 - units: '' + units: "" weather_code_metar_4678: description: METAR/SPECI weather code according to table 4678 of Parsivel documentation long_name: Weather code METAR 4678 - units: '' + units: "" weather_code_nws: description: NWS weather code according to Parsivel documentation long_name: Weather code NWS - units: '' + units: "" reflectivity_32bit: description: Radar reflectivity long_name: Reflectivity 32 bit @@ -35,14 +35,15 @@ sample_interval: long_name: Sample interval units: s laser_amplitude: - description: Signal amplitude of the laser strip. A way to monitor if windows are + description: + Signal amplitude of the laser strip. A way to monitor if windows are dirty or not. long_name: Laser amplitude - units: '' + units: "" number_particles: description: Number of particles detected and validated long_name: Number of detected particles - units: '' + units: "" sensor_temperature: description: Temperature in sensor housing long_name: Temperature of the sensor @@ -50,18 +51,19 @@ sensor_temperature: sensor_serial_number: description: Sensor serial number long_name: Serial number of the sensor - units: '' + units: "" firmware_iop: description: Version IOP firmware long_name: Version IOP firmware - units: '' + units: "" firmware_dsp: description: Version DSP firmware long_name: Version DSP firmware - units: '' + units: "" sensor_heating_current: - description: 'Sensor head heating current. Optimum heating output of the sensor - head heating system can be guaranteed with a power supply voltage > 20 V ' + description: + "Sensor head heating current. Optimum heating output of the sensor + head heating system can be guaranteed with a power supply voltage > 20 V " long_name: Sensor heating current units: A sensor_battery_voltage: @@ -71,27 +73,27 @@ sensor_battery_voltage: sensor_status: description: Sensor status long_name: Sensor status - units: '' + units: "" start_time: description: Meausurement start time long_name: Start time - units: '' + units: "" sensor_time: description: Time of the sensor long_name: Sensor time - units: '' + units: "" sensor_date: description: Date of the sensor long_name: Sensor date - units: '' + units: "" station_name: description: Name of the station long_name: Station name - units: '' + units: "" station_number: description: Number of the station long_name: Station number - units: '' + units: "" rainfall_amount_absolute_32bit: description: Accumulated rain amount over the measurement interval long_name: Rainfall amount absolute 32 bit @@ -99,7 +101,7 @@ rainfall_amount_absolute_32bit: error_code: description: Error code long_name: Error code - units: '' + units: "" rainfall_rate_16bit: description: Rainfall rate long_name: Rainfall rate 16 bit @@ -127,4 +129,4 @@ raw_drop_average_velocity: raw_drop_number: description: Drop counts per diameter and velocity class long_name: Raw drop number - units: '' + units: "" diff --git a/disdrodb/l0/configs/OTT_Parsivel/l0b_encodings.yml b/disdrodb/l0/configs/OTT_Parsivel/l0b_encodings.yml index 4aab8cc1..d8b04830 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/l0b_encodings.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/l0b_encodings.yml @@ -247,8 +247,8 @@ raw_drop_concentration: fletcher32: false contiguous: false chunksizes: - - 5000 - - 32 + - 5000 + - 32 raw_drop_average_velocity: dtype: float32 zlib: true @@ -257,8 +257,8 @@ raw_drop_average_velocity: fletcher32: false contiguous: false chunksizes: - - 5000 - - 32 + - 5000 + - 32 raw_drop_number: dtype: uint16 zlib: true @@ -268,6 +268,6 @@ raw_drop_number: contiguous: false _FillValue: 65535 chunksizes: - - 5000 - - 32 - - 32 + - 5000 + - 32 + - 32 diff --git a/disdrodb/l0/configs/OTT_Parsivel/raw_data_format.yml b/disdrodb/l0/configs/OTT_Parsivel/raw_data_format.yml index 32925f18..192e7e2a 100644 --- a/disdrodb/l0/configs/OTT_Parsivel/raw_data_format.yml +++ b/disdrodb/l0/configs/OTT_Parsivel/raw_data_format.yml @@ -4,40 +4,40 @@ rainfall_rate_32bit: n_decimals: 3 n_naturals: 4 data_range: - - 0 - - 9999.999 + - 0 + - 9999.999 nan_flags: null - field_number: '01' + field_number: "01" rainfall_accumulated_32bit: n_digits: 6 n_characters: 7 n_decimals: 2 n_naturals: 4 data_range: - - 0 - - 300.0 + - 0 + - 300.0 nan_flags: null - field_number: '02' + field_number: "02" weather_code_synop_4680: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - 0 - - 89 + - 0 + - 89 nan_flags: null - field_number: '03' + field_number: "03" weather_code_synop_4677: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - 0 - - 90 + - 0 + - 90 nan_flags: null - field_number: '04' + field_number: "04" weather_code_metar_4678: n_digits: null n_characters: null @@ -45,7 +45,7 @@ weather_code_metar_4678: n_naturals: null data_range: null nan_flags: null - field_number: '05' + field_number: "05" weather_code_nws: n_digits: null n_characters: null @@ -53,25 +53,25 @@ weather_code_nws: n_naturals: null data_range: null nan_flags: null - field_number: '06' + field_number: "06" reflectivity_32bit: n_digits: 5 n_characters: 6 n_decimals: 3 n_naturals: 2 data_range: - - -9.999 - - 99.999 + - -9.999 + - 99.999 nan_flags: null - field_number: '07' + field_number: "07" mor_visibility: n_digits: 4 n_characters: 4 n_decimals: 0 n_naturals: 4 data_range: - - 0 - - 20000 + - 0 + - 20000 nan_flags: null field_number: 08 sample_interval: @@ -80,8 +80,8 @@ sample_interval: n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 3600 + - 0 + - 3600 nan_flags: null field_number: 09 laser_amplitude: @@ -90,30 +90,30 @@ laser_amplitude: n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '10' + field_number: "10" number_particles: n_digits: 5 n_characters: 5 n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '11' + field_number: "11" sensor_temperature: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - -99 - - 100 + - -99 + - 100 nan_flags: null - field_number: '12' + field_number: "12" sensor_serial_number: n_digits: null n_characters: null @@ -121,7 +121,7 @@ sensor_serial_number: n_naturals: null data_range: null nan_flags: null - field_number: '13' + field_number: "13" firmware_iop: n_digits: null n_characters: null @@ -129,7 +129,7 @@ firmware_iop: n_naturals: null data_range: null nan_flags: null - field_number: '14' + field_number: "14" firmware_dsp: n_digits: null n_characters: null @@ -137,42 +137,42 @@ firmware_dsp: n_naturals: null data_range: null nan_flags: null - field_number: '15' + field_number: "15" sensor_heating_current: n_digits: 3 n_characters: 4 n_decimals: 2 n_naturals: 1 data_range: - - 0 - - 4.0 + - 0 + - 4.0 nan_flags: null - field_number: '16' + field_number: "16" sensor_battery_voltage: n_digits: 3 n_characters: 4 n_decimals: 1 n_naturals: 2 data_range: - - 0 - - 30.0 + - 0 + - 30.0 nan_flags: null - field_number: '17' + field_number: "17" sensor_status: n_digits: 1 n_characters: 1 n_decimals: 0 n_naturals: 1 data_range: - - 0 - - 3 + - 0 + - 3 nan_flags: null valid_values: - - 0 - - 1 - - 2 - - 3 - field_number: '18' + - 0 + - 1 + - 2 + - 3 + field_number: "18" start_time: n_digits: null n_characters: null @@ -180,7 +180,7 @@ start_time: n_naturals: null data_range: null nan_flags: null - field_number: '19' + field_number: "19" sensor_time: n_digits: null n_characters: null @@ -188,7 +188,7 @@ sensor_time: n_naturals: null data_range: null nan_flags: null - field_number: '20' + field_number: "20" sensor_date: n_digits: null n_characters: null @@ -196,7 +196,7 @@ sensor_date: n_naturals: null data_range: null nan_flags: null - field_number: '21' + field_number: "21" station_name: n_digits: null n_characters: null @@ -204,7 +204,7 @@ station_name: n_naturals: null data_range: null nan_flags: null - field_number: '22' + field_number: "22" station_number: n_digits: null n_characters: null @@ -212,41 +212,41 @@ station_number: n_naturals: null data_range: null nan_flags: null - field_number: '23' + field_number: "23" rainfall_amount_absolute_32bit: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0 - - 999.999 + - 0 + - 999.999 nan_flags: null - field_number: '24' + field_number: "24" error_code: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - 0 - - 3 + - 0 + - 3 nan_flags: null valid_values: - - 0 - - 1 - - 2 - field_number: '25' + - 0 + - 1 + - 2 + field_number: "25" rainfall_rate_16bit: n_digits: 5 n_characters: 6 n_decimals: 3 n_naturals: 2 data_range: - - 0 - - 9999.999 + - 0 + - 9999.999 nan_flags: null - field_number: '30' + field_number: "30" rainfall_rate_12bit: n_digits: 5 n_characters: 6 @@ -254,27 +254,27 @@ rainfall_rate_12bit: n_naturals: 4 data_range: null nan_flags: null - field_number: '31' + field_number: "31" rainfall_accumulated_16bit: n_digits: 6 n_characters: 7 n_decimals: 2 n_naturals: 4 data_range: - - 0 - - 300.0 + - 0 + - 300.0 nan_flags: null - field_number: '32' + field_number: "32" reflectivity_16bit: n_digits: 4 n_characters: 5 n_decimals: 2 n_naturals: 2 data_range: - - -9.99 - - 99.99 + - -9.99 + - 99.99 nan_flags: null - field_number: '33' + field_number: "33" raw_drop_concentration: n_digits: 0 n_characters: 224 @@ -283,9 +283,9 @@ raw_drop_concentration: data_range: null nan_flags: null dimension_order: - - diameter_bin_center + - diameter_bin_center n_values: 32 - field_number: '90' + field_number: "90" raw_drop_average_velocity: n_digits: 0 n_characters: 224 @@ -294,9 +294,9 @@ raw_drop_average_velocity: data_range: null nan_flags: null dimension_order: - - velocity_bin_center + - velocity_bin_center n_values: 32 - field_number: '91' + field_number: "91" raw_drop_number: n_digits: 0 n_characters: 4096 @@ -305,7 +305,7 @@ raw_drop_number: data_range: null nan_flags: null dimension_order: - - velocity_bin_center - - diameter_bin_center + - velocity_bin_center + - diameter_bin_center n_values: 1024 - field_number: '93' + field_number: "93" diff --git a/disdrodb/l0/configs/OTT_Parsivel2/bins_diameter.yml b/disdrodb/l0/configs/OTT_Parsivel2/bins_diameter.yml index e8ce55d5..8052e18e 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/bins_diameter.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/bins_diameter.yml @@ -33,101 +33,101 @@ center: 31: 24.5 bounds: 0: - - 0.0 - - 0.1245 + - 0.0 + - 0.1245 1: - - 0.1245 - - 0.2495 + - 0.1245 + - 0.2495 2: - - 0.2495 - - 0.3745 + - 0.2495 + - 0.3745 3: - - 0.3745 - - 0.4995 + - 0.3745 + - 0.4995 4: - - 0.4995 - - 0.6245 + - 0.4995 + - 0.6245 5: - - 0.6245 - - 0.7495 + - 0.6245 + - 0.7495 6: - - 0.7495 - - 0.8745 + - 0.7495 + - 0.8745 7: - - 0.8745 - - 0.9995 + - 0.8745 + - 0.9995 8: - - 0.9995 - - 1.1245 + - 0.9995 + - 1.1245 9: - - 1.1245 - - 1.25 + - 1.1245 + - 1.25 10: - - 1.25 - - 1.5 + - 1.25 + - 1.5 11: - - 1.5 - - 1.75 + - 1.5 + - 1.75 12: - - 1.75 - - 2.0 + - 1.75 + - 2.0 13: - - 2.0 - - 2.25 + - 2.0 + - 2.25 14: - - 2.25 - - 2.5 + - 2.25 + - 2.5 15: - - 2.5 - - 3.0 + - 2.5 + - 3.0 16: - - 3.0 - - 3.5 + - 3.0 + - 3.5 17: - - 3.5 - - 4.0 + - 3.5 + - 4.0 18: - - 4.0 - - 4.5 + - 4.0 + - 4.5 19: - - 4.5 - - 5.0 + - 4.5 + - 5.0 20: - - 5.0 - - 6.0 + - 5.0 + - 6.0 21: - - 6.0 - - 7.0 + - 6.0 + - 7.0 22: - - 7.0 - - 8.0 + - 7.0 + - 8.0 23: - - 8.0 - - 9.0 + - 8.0 + - 9.0 24: - - 9.0 - - 10.0 + - 9.0 + - 10.0 25: - - 10.0 - - 12.0 + - 10.0 + - 12.0 26: - - 12.0 - - 14.0 + - 12.0 + - 14.0 27: - - 14.0 - - 16.0 + - 14.0 + - 16.0 28: - - 16.0 - - 18.0 + - 16.0 + - 18.0 29: - - 18.0 - - 20.0 + - 18.0 + - 20.0 30: - - 20.0 - - 23.0 + - 20.0 + - 23.0 31: - - 23.0 - - 26.0 + - 23.0 + - 26.0 width: 0: 0.125 1: 0.125 diff --git a/disdrodb/l0/configs/OTT_Parsivel2/bins_velocity.yml b/disdrodb/l0/configs/OTT_Parsivel2/bins_velocity.yml index 5ea05595..07f591b1 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/bins_velocity.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/bins_velocity.yml @@ -33,101 +33,101 @@ center: 31: 20.8 bounds: 0: - - 0.0 - - 0.1 + - 0.0 + - 0.1 1: - - 0.1 - - 0.2 + - 0.1 + - 0.2 2: - - 0.2 - - 0.3 + - 0.2 + - 0.3 3: - - 0.3 - - 0.4 + - 0.3 + - 0.4 4: - - 0.4 - - 0.5 + - 0.4 + - 0.5 5: - - 0.5 - - 0.6 + - 0.5 + - 0.6 6: - - 0.6 - - 0.7 + - 0.6 + - 0.7 7: - - 0.7 - - 0.8 + - 0.7 + - 0.8 8: - - 0.8 - - 0.9 + - 0.8 + - 0.9 9: - - 0.9 - - 1.0 + - 0.9 + - 1.0 10: - - 1.0 - - 1.2 + - 1.0 + - 1.2 11: - - 1.2 - - 1.4 + - 1.2 + - 1.4 12: - - 1.4 - - 1.6 + - 1.4 + - 1.6 13: - - 1.6 - - 1.8 + - 1.6 + - 1.8 14: - - 1.8 - - 2.0 + - 1.8 + - 2.0 15: - - 2.0 - - 2.4 + - 2.0 + - 2.4 16: - - 2.4 - - 2.8 + - 2.4 + - 2.8 17: - - 2.8 - - 3.2 + - 2.8 + - 3.2 18: - - 3.2 - - 3.6 + - 3.2 + - 3.6 19: - - 3.6 - - 4.0 + - 3.6 + - 4.0 20: - - 4.0 - - 4.8 + - 4.0 + - 4.8 21: - - 4.8 - - 5.6 + - 4.8 + - 5.6 22: - - 5.6 - - 6.4 + - 5.6 + - 6.4 23: - - 6.4 - - 7.2 + - 6.4 + - 7.2 24: - - 7.2 - - 8.0 + - 7.2 + - 8.0 25: - - 8.0 - - 9.6 + - 8.0 + - 9.6 26: - - 9.6 - - 11.2 + - 9.6 + - 11.2 27: - - 11.2 - - 12.8 + - 11.2 + - 12.8 28: - - 12.8 - - 14.4 + - 12.8 + - 14.4 29: - - 14.4 - - 16.0 + - 14.4 + - 16.0 30: - - 16.0 - - 19.2 + - 16.0 + - 19.2 31: - - 19.2 - - 22.4 + - 19.2 + - 22.4 width: 0: 0.1 1: 0.1 diff --git a/disdrodb/l0/configs/OTT_Parsivel2/l0a_encodings.yml b/disdrodb/l0/configs/OTT_Parsivel2/l0a_encodings.yml index 0f352974..9de9ebc7 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/l0a_encodings.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/l0a_encodings.yml @@ -1,39 +1,39 @@ -rainfall_rate_32bit: 'float32' -rainfall_accumulated_32bit: 'float32' -weather_code_synop_4680: 'float32' # 'uint32' -weather_code_synop_4677: 'float32' # 'uint32' -weather_code_metar_4678: 'str' -weather_code_nws: 'str' -reflectivity_32bit: 'float32' -mor_visibility: 'float32' # 'uint16' -sample_interval: 'float32' # 'uint16' -laser_amplitude: 'float32' #'uint32' -number_particles: 'float32' # 'uint32' -sensor_temperature: 'float32' # 'int8' -sensor_serial_number: 'str' -firmware_iop: 'str' -firmware_dsp: 'str' -sensor_heating_current: 'float32' -sensor_battery_voltage: 'float32' -sensor_status: 'float32' # 'uint8' -start_time: 'str' -sensor_time: 'str' -sensor_date: 'str' -station_name: 'str' -station_number: 'str' -rainfall_amount_absolute_32bit: 'float32' -error_code: 'float32' # 'uint8' -sensor_temperature_pcb: 'float32' # 'int8' -sensor_temperature_receiver: 'float32' # 'int8' -sensor_temperature_trasmitter: 'float32' # 'int8' -rainfall_rate_16_bit_30: 'float32' -rainfall_rate_16_bit_1200: 'float32' -rainfall_accumulated_16bit: 'float32' -reflectivity_16bit: 'float32' -rain_kinetic_energy: 'float32' -snowfall_rate: 'float32' -number_particles_all: 'float32' # 'float32' 'uint32' -list_particles: 'str' -raw_drop_concentration: 'str' -raw_drop_average_velocity: 'str' -raw_drop_number: 'str' +rainfall_rate_32bit: "float32" +rainfall_accumulated_32bit: "float32" +weather_code_synop_4680: "float32" # 'uint32' +weather_code_synop_4677: "float32" # 'uint32' +weather_code_metar_4678: "str" +weather_code_nws: "str" +reflectivity_32bit: "float32" +mor_visibility: "float32" # 'uint16' +sample_interval: "float32" # 'uint16' +laser_amplitude: "float32" #'uint32' +number_particles: "float32" # 'uint32' +sensor_temperature: "float32" # 'int8' +sensor_serial_number: "str" +firmware_iop: "str" +firmware_dsp: "str" +sensor_heating_current: "float32" +sensor_battery_voltage: "float32" +sensor_status: "float32" # 'uint8' +start_time: "str" +sensor_time: "str" +sensor_date: "str" +station_name: "str" +station_number: "str" +rainfall_amount_absolute_32bit: "float32" +error_code: "float32" # 'uint8' +sensor_temperature_pcb: "float32" # 'int8' +sensor_temperature_receiver: "float32" # 'int8' +sensor_temperature_trasmitter: "float32" # 'int8' +rainfall_rate_16_bit_30: "float32" +rainfall_rate_16_bit_1200: "float32" +rainfall_accumulated_16bit: "float32" +reflectivity_16bit: "float32" +rain_kinetic_energy: "float32" +snowfall_rate: "float32" +number_particles_all: "float32" # 'float32' 'uint32' +list_particles: "str" +raw_drop_concentration: "str" +raw_drop_average_velocity: "str" +raw_drop_number: "str" diff --git a/disdrodb/l0/configs/OTT_Parsivel2/l0b_cf_attrs.yml b/disdrodb/l0/configs/OTT_Parsivel2/l0b_cf_attrs.yml index 1e9639f1..7f326383 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/l0b_cf_attrs.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/l0b_cf_attrs.yml @@ -9,19 +9,19 @@ rainfall_accumulated_32bit: weather_code_synop_4680: description: SYNOP weather code according to table 4680 of Parsivel documentation long_name: Weather code SYNOP 4680 - units: '' + units: "" weather_code_synop_4677: description: SYNOP weather code according to table 4677 of Parsivel documentation long_name: Weather code SYNOP 4677 - units: '' + units: "" weather_code_metar_4678: description: METAR/SPECI weather code according to table 4678 of Parsivel documentation long_name: Weather code METAR 4678 - units: '' + units: "" weather_code_nws: description: NWS weather code according to Parsivel documentation long_name: Weather code NWS - units: '' + units: "" reflectivity_32bit: description: Radar reflectivity long_name: Reflectivity 32 bit @@ -35,14 +35,15 @@ sample_interval: long_name: Sample interval units: s laser_amplitude: - description: Signal amplitude of the laser strip. A way to monitor if windows are + description: + Signal amplitude of the laser strip. A way to monitor if windows are dirty or not. long_name: Laser amplitude - units: '' + units: "" number_particles: description: Number of particles detected and validated long_name: Number of detected particles - units: '' + units: "" sensor_temperature: description: Temperature in sensor housing long_name: Temperature of the sensor @@ -50,18 +51,19 @@ sensor_temperature: sensor_serial_number: description: Sensor serial number long_name: Serial number of the sensor - units: '' + units: "" firmware_iop: description: Version IOP firmware long_name: Version IOP firmware - units: '' + units: "" firmware_dsp: description: Version DSP firmware long_name: Version DSP firmware - units: '' + units: "" sensor_heating_current: - description: 'Sensor head heating current. Optimum heating output of the sensor - head heating system can be guaranteed with a power supply voltage > 20 V ' + description: + "Sensor head heating current. Optimum heating output of the sensor + head heating system can be guaranteed with a power supply voltage > 20 V " long_name: Sensor heating current units: A sensor_battery_voltage: @@ -71,27 +73,27 @@ sensor_battery_voltage: sensor_status: description: Sensor status long_name: Sensor status - units: '' + units: "" start_time: description: Meausurement start time long_name: Start time - units: '' + units: "" sensor_time: description: Time of the sensor long_name: Sensor time - units: '' + units: "" sensor_date: description: Date of the sensor long_name: Sensor date - units: '' + units: "" station_name: description: Name of the station long_name: Station name - units: '' + units: "" station_number: description: Number of the station long_name: Station number - units: '' + units: "" rainfall_amount_absolute_32bit: description: Accumulated rain amount over the measurement interval long_name: Rainfall amount absolute 32 bit @@ -99,7 +101,7 @@ rainfall_amount_absolute_32bit: error_code: description: Error code long_name: Error code - units: '' + units: "" sensor_temperature_pcb: description: Temperature in printed circuit board long_name: Sensor PCB temperature @@ -139,11 +141,11 @@ snowfall_rate: number_particles_all: description: Number of all particles detected (also not validated) long_name: Number of all measured particles - units: '' + units: "" list_particles: description: List of all particles detected. Including size and velocity. long_name: List particles detected - units: '' + units: "" raw_drop_concentration: description: Particle number concentrations per diameter class long_name: Raw drop concentration @@ -155,4 +157,4 @@ raw_drop_average_velocity: raw_drop_number: description: Drop counts per diameter and velocity class long_name: Raw drop number - units: '' + units: "" diff --git a/disdrodb/l0/configs/OTT_Parsivel2/l0b_encodings.yml b/disdrodb/l0/configs/OTT_Parsivel2/l0b_encodings.yml index 3a2df87f..88d4a6ed 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/l0b_encodings.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/l0b_encodings.yml @@ -307,8 +307,8 @@ raw_drop_concentration: fletcher32: false contiguous: false chunksizes: - - 5000 - - 32 + - 5000 + - 32 raw_drop_average_velocity: dtype: float32 zlib: true @@ -317,8 +317,8 @@ raw_drop_average_velocity: fletcher32: false contiguous: false chunksizes: - - 5000 - - 32 + - 5000 + - 32 raw_drop_number: dtype: uint16 zlib: true @@ -328,6 +328,6 @@ raw_drop_number: contiguous: false _FillValue: 65535 chunksizes: - - 5000 - - 32 - - 32 + - 5000 + - 32 + - 32 diff --git a/disdrodb/l0/configs/OTT_Parsivel2/raw_data_format.yml b/disdrodb/l0/configs/OTT_Parsivel2/raw_data_format.yml index aa64fbf6..5c271483 100644 --- a/disdrodb/l0/configs/OTT_Parsivel2/raw_data_format.yml +++ b/disdrodb/l0/configs/OTT_Parsivel2/raw_data_format.yml @@ -4,40 +4,40 @@ rainfall_rate_32bit: n_decimals: 3 n_naturals: 4 data_range: - - 0 - - 9999.999 + - 0 + - 9999.999 nan_flags: null - field_number: '01' + field_number: "01" rainfall_accumulated_32bit: n_digits: 6 n_characters: 7 n_decimals: 2 n_naturals: 4 data_range: - - 0 - - 300.0 + - 0 + - 300.0 nan_flags: null - field_number: '02' + field_number: "02" weather_code_synop_4680: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - 0 - - 89 + - 0 + - 89 nan_flags: null - field_number: '03' + field_number: "03" weather_code_synop_4677: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - 0 - - 90 + - 0 + - 90 nan_flags: null - field_number: '04' + field_number: "04" weather_code_metar_4678: n_digits: null n_characters: null @@ -45,7 +45,7 @@ weather_code_metar_4678: n_naturals: null data_range: null nan_flags: null - field_number: '05' + field_number: "05" weather_code_nws: n_digits: null n_characters: null @@ -53,25 +53,25 @@ weather_code_nws: n_naturals: null data_range: null nan_flags: null - field_number: '06' + field_number: "06" reflectivity_32bit: n_digits: 5 n_characters: 6 n_decimals: 3 n_naturals: 2 data_range: - - -9.999 - - 99.999 + - -9.999 + - 99.999 nan_flags: null - field_number: '07' + field_number: "07" mor_visibility: n_digits: 4 n_characters: 4 n_decimals: 0 n_naturals: 4 data_range: - - 0 - - 20000 + - 0 + - 20000 nan_flags: null field_number: 08 sample_interval: @@ -80,8 +80,8 @@ sample_interval: n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 3600 + - 0 + - 3600 nan_flags: null field_number: 09 laser_amplitude: @@ -90,30 +90,30 @@ laser_amplitude: n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '10' + field_number: "10" number_particles: n_digits: 5 n_characters: 5 n_decimals: 0 n_naturals: 5 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '11' + field_number: "11" sensor_temperature: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - -99 - - 100 + - -99 + - 100 nan_flags: null - field_number: '12' + field_number: "12" sensor_serial_number: n_digits: null n_characters: null @@ -121,7 +121,7 @@ sensor_serial_number: n_naturals: null data_range: null nan_flags: null - field_number: '13' + field_number: "13" firmware_iop: n_digits: null n_characters: null @@ -129,7 +129,7 @@ firmware_iop: n_naturals: null data_range: null nan_flags: null - field_number: '14' + field_number: "14" firmware_dsp: n_digits: null n_characters: null @@ -137,42 +137,42 @@ firmware_dsp: n_naturals: null data_range: null nan_flags: null - field_number: '15' + field_number: "15" sensor_heating_current: n_digits: 3 n_characters: 4 n_decimals: 2 n_naturals: 1 data_range: - - 0 - - 4.0 + - 0 + - 4.0 nan_flags: null - field_number: '16' + field_number: "16" sensor_battery_voltage: n_digits: 3 n_characters: 4 n_decimals: 1 n_naturals: 2 data_range: - - 0 - - 30.0 + - 0 + - 30.0 nan_flags: null - field_number: '17' + field_number: "17" sensor_status: n_digits: 1 n_characters: 1 n_decimals: 0 n_naturals: 1 data_range: - - 0 - - 3 + - 0 + - 3 nan_flags: null valid_values: - - 0 - - 1 - - 2 - - 3 - field_number: '18' + - 0 + - 1 + - 2 + - 3 + field_number: "18" start_time: n_digits: null n_characters: null @@ -180,7 +180,7 @@ start_time: n_naturals: null data_range: null nan_flags: null - field_number: '19' + field_number: "19" sensor_time: n_digits: null n_characters: null @@ -188,7 +188,7 @@ sensor_time: n_naturals: null data_range: null nan_flags: null - field_number: '20' + field_number: "20" sensor_date: n_digits: null n_characters: null @@ -196,7 +196,7 @@ sensor_date: n_naturals: null data_range: null nan_flags: null - field_number: '21' + field_number: "21" station_name: n_digits: null n_characters: null @@ -204,7 +204,7 @@ station_name: n_naturals: null data_range: null nan_flags: null - field_number: '22' + field_number: "22" station_number: n_digits: null n_characters: null @@ -212,131 +212,131 @@ station_number: n_naturals: null data_range: null nan_flags: null - field_number: '23' + field_number: "23" rainfall_amount_absolute_32bit: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0 - - 999.999 + - 0 + - 999.999 nan_flags: null - field_number: '24' + field_number: "24" error_code: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - 0 - - 3 + - 0 + - 3 nan_flags: null valid_values: - - 0 - - 1 - - 2 - field_number: '25' + - 0 + - 1 + - 2 + field_number: "25" sensor_temperature_pcb: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - -99 - - 100 + - -99 + - 100 nan_flags: null - field_number: '26' + field_number: "26" sensor_temperature_receiver: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - -99 - - 100 + - -99 + - 100 nan_flags: null - field_number: '27' + field_number: "27" sensor_temperature_trasmitter: n_digits: 3 n_characters: 3 n_decimals: 0 n_naturals: 3 data_range: - - -99 - - 100 + - -99 + - 100 nan_flags: null - field_number: '28' + field_number: "28" rainfall_rate_16_bit_30: n_digits: 5 n_characters: 6 n_decimals: 2 n_naturals: 3 data_range: - - 0 - - 30 + - 0 + - 30 nan_flags: null - field_number: '30' + field_number: "30" rainfall_rate_16_bit_1200: n_digits: 5 n_characters: 6 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 1200 + - 0 + - 1200 nan_flags: null - field_number: '31' + field_number: "31" rainfall_accumulated_16bit: n_digits: 6 n_characters: 7 n_decimals: 2 n_naturals: 4 data_range: - - 0 - - 300.0 + - 0 + - 300.0 nan_flags: null - field_number: '32' + field_number: "32" reflectivity_16bit: n_digits: 4 n_characters: 5 n_decimals: 2 n_naturals: 2 data_range: - - -9.99 - - 99.99 + - -9.99 + - 99.99 nan_flags: null - field_number: '33' + field_number: "33" rain_kinetic_energy: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0 - - 999.999 + - 0 + - 999.999 nan_flags: null - field_number: '34' + field_number: "34" snowfall_rate: n_digits: 6 n_characters: 7 n_decimals: 2 n_naturals: 4 data_range: - - 0 - - 999.999 + - 0 + - 999.999 nan_flags: null - field_number: '35' + field_number: "35" number_particles_all: n_digits: 8 n_characters: 8 n_decimals: 0 n_naturals: 8 data_range: - - 0 - - 8192 + - 0 + - 8192 nan_flags: null - field_number: '60' + field_number: "60" list_particles: n_digits: null n_characters: null @@ -344,7 +344,7 @@ list_particles: n_naturals: null data_range: null nan_flags: null - field_number: '61' + field_number: "61" raw_drop_concentration: n_digits: 0 n_characters: 224 @@ -353,9 +353,9 @@ raw_drop_concentration: data_range: null nan_flags: null dimension_order: - - diameter_bin_center + - diameter_bin_center n_values: 32 - field_number: '90' + field_number: "90" raw_drop_average_velocity: n_digits: 0 n_characters: 224 @@ -364,9 +364,9 @@ raw_drop_average_velocity: data_range: null nan_flags: null dimension_order: - - velocity_bin_center + - velocity_bin_center n_values: 32 - field_number: '91' + field_number: "91" raw_drop_number: n_digits: 0 n_characters: 4096 @@ -375,7 +375,7 @@ raw_drop_number: data_range: null nan_flags: null dimension_order: - - velocity_bin_center - - diameter_bin_center + - velocity_bin_center + - diameter_bin_center n_values: 1024 - field_number: '93' + field_number: "93" diff --git a/disdrodb/l0/configs/RD_80/bins_diameter.yml b/disdrodb/l0/configs/RD_80/bins_diameter.yml index 0495341b..b2439b01 100644 --- a/disdrodb/l0/configs/RD_80/bins_diameter.yml +++ b/disdrodb/l0/configs/RD_80/bins_diameter.yml @@ -22,65 +22,65 @@ center: bounds: 0: - - 0.313 - - 0.405 + - 0.313 + - 0.405 1: - - 0.405 - - 0.505 + - 0.405 + - 0.505 2: - - 0.505 - - 0.596 + - 0.505 + - 0.596 3: - - 0.596 - - 0.715 + - 0.596 + - 0.715 4: - - 0.715 - - 0.827 + - 0.715 + - 0.827 5: - - 0.827 - - 0.999 + - 0.827 + - 0.999 6: - - 0.999 - - 1.232 + - 0.999 + - 1.232 7: - - 1.232 - - 1.429 + - 1.232 + - 1.429 8: - - 1.429 - - 1.582 + - 1.429 + - 1.582 9: - - 1.582 - - 1.748 + - 1.582 + - 1.748 10: - - 1.748 - - 2.077 + - 1.748 + - 2.077 11: - - 2.077 - - 2.441 + - 2.077 + - 2.441 12: - - 2.441 - - 2.727 + - 2.441 + - 2.727 13: - - 2.727 - - 3.011 + - 2.727 + - 3.011 14: - - 3.011 - - 3.385 + - 3.011 + - 3.385 15: - - 3.385 - - 3.704 + - 3.385 + - 3.704 16: - - 3.704 - - 4.127 + - 3.704 + - 4.127 17: - - 4.127 - - 4.573 + - 4.127 + - 4.573 18: - - 4.573 - - 5.145 + - 4.573 + - 5.145 19: - - 5.145 - - 5.6 + - 5.145 + - 5.6 width: 0: 0.092 diff --git a/disdrodb/l0/configs/RD_80/l0a_encodings.yml b/disdrodb/l0/configs/RD_80/l0a_encodings.yml index 28245506..775d637b 100644 --- a/disdrodb/l0/configs/RD_80/l0a_encodings.yml +++ b/disdrodb/l0/configs/RD_80/l0a_encodings.yml @@ -1,16 +1,16 @@ -sensor_status: 'float32' # 'int8' -interval: 'float32' # 'uint8' -RI: 'float32' -RA: 'float32' -RAT: 'float32' -W: 'float32' -Wg: 'float32' -Z: 'float32' -ZdB: 'float32' -EK: 'float32' -EF: 'float32' -N0: 'float32' -slope: 'float32' -Dmax: 'float32' -ND: 'str' -raw_drop_number: 'str' +sensor_status: "float32" # 'int8' +interval: "float32" # 'uint8' +RI: "float32" +RA: "float32" +RAT: "float32" +W: "float32" +Wg: "float32" +Z: "float32" +ZdB: "float32" +EK: "float32" +EF: "float32" +N0: "float32" +slope: "float32" +Dmax: "float32" +ND: "str" +raw_drop_number: "str" diff --git a/disdrodb/l0/configs/RD_80/l0b_cf_attrs.yml b/disdrodb/l0/configs/RD_80/l0b_cf_attrs.yml index a94fe927..95baceae 100644 --- a/disdrodb/l0/configs/RD_80/l0b_cf_attrs.yml +++ b/disdrodb/l0/configs/RD_80/l0b_cf_attrs.yml @@ -1,7 +1,7 @@ sensor_status: description: Sensor status long_name: Sensor status - units: '' + units: "" interval: description: Time interval between measurement long_name: Time interval between measurement @@ -61,4 +61,4 @@ ND: raw_drop_number: description: Drop counts per diameter long_name: Drop counts per diameter - units: '' + units: "" diff --git a/disdrodb/l0/configs/RD_80/l0b_encodings.yml b/disdrodb/l0/configs/RD_80/l0b_encodings.yml index ad478974..bc6a63a0 100644 --- a/disdrodb/l0/configs/RD_80/l0b_encodings.yml +++ b/disdrodb/l0/configs/RD_80/l0b_encodings.yml @@ -1,127 +1,127 @@ sensor_status: - dtype: uint8 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 - _FillValue: 255 + dtype: uint8 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 + _FillValue: 255 interval: - dtype: uint8 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 - _FillValue: 255 + dtype: uint8 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 + _FillValue: 255 RI: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 RA: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 RAT: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 W: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 Wg: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 Z: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 ZdB: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 EK: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 EF: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 N0: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 slope: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 Dmax: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: 5000 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: 5000 ND: - dtype: float32 - zlib: true - complevel: 3 - shuffle: true - fletcher32: false - contiguous: false - chunksizes: - - 5000 - - 20 + dtype: float32 + zlib: true + complevel: 3 + shuffle: true + fletcher32: false + contiguous: false + chunksizes: + - 5000 + - 20 raw_drop_number: dtype: uint16 zlib: true @@ -131,5 +131,5 @@ raw_drop_number: contiguous: false _FillValue: 65535 chunksizes: - - 5000 - - 20 + - 5000 + - 20 diff --git a/disdrodb/l0/configs/RD_80/raw_data_format.yml b/disdrodb/l0/configs/RD_80/raw_data_format.yml index 2de8ed85..0b1cf856 100644 --- a/disdrodb/l0/configs/RD_80/raw_data_format.yml +++ b/disdrodb/l0/configs/RD_80/raw_data_format.yml @@ -5,7 +5,7 @@ time: n_naturals: null data_range: null nan_flags: null - field_number: '02' + field_number: "02" date: n_digits: null n_characters: null @@ -13,7 +13,7 @@ date: n_naturals: null data_range: null nan_flags: null - field_number: '01' + field_number: "01" sensor_status: n_digits: null n_characters: null @@ -22,139 +22,139 @@ sensor_status: data_range: null nan_flags: null valid_values: - - 0 - - 1 - field_number: '03' + - 0 + - 1 + field_number: "03" interval: n_digits: 4 n_characters: 4 n_decimals: 4 n_naturals: 0 data_range: - - 0 - - 9999 + - 0 + - 9999 nan_flags: null - field_number: '04' + field_number: "04" RI: n_digits: 7 n_characters: 8 n_decimals: 4 n_naturals: 3 data_range: - - 0 - - 999.9999 + - 0 + - 999.9999 nan_flags: null - field_number: '25' + field_number: "25" RA: n_digits: 5 n_characters: 6 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '26' + field_number: "26" RAT: n_digits: 5 n_characters: 6 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '27' + field_number: "27" W: n_digits: 5 n_characters: 6 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '28' + field_number: "28" Wg: n_digits: 5 n_characters: 6 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '29' + field_number: "29" Z: n_digits: 6 n_characters: 1 n_decimals: 4 n_naturals: 1 data_range: - - -99.9999 - - 99.9999 + - -99.9999 + - 99.9999 nan_flags: -Inf - field_number: '30' + field_number: "30" ZdB: n_digits: 6 n_characters: 1 n_decimals: 4 n_naturals: 1 data_range: - - -99.9999 - - 99.9999 + - -99.9999 + - 99.9999 nan_flags: -Inf - field_number: '31' + field_number: "31" EF: n_digits: 7 n_characters: 8 n_decimals: 4 n_naturals: 4 data_range: - - 0 - - 9999.999 + - 0 + - 9999.999 nan_flags: null - field_number: '33' + field_number: "33" EK: n_digits: 7 n_characters: 8 n_decimals: 4 n_naturals: 4 data_range: - - 0 - - 9999.999 + - 0 + - 9999.999 nan_flags: null - field_number: '32' + field_number: "32" N0: n_digits: 8 n_characters: 9 n_decimals: 4 n_naturals: 4 data_range: - - 0 - - 9999.9999 + - 0 + - 9999.9999 nan_flags: null - field_number: '34' + field_number: "34" slope: n_digits: 8 n_characters: 9 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '35' + field_number: "35" Dmax: n_digits: 8 n_characters: 9 n_decimals: 4 n_naturals: 1 data_range: - - 0 - - 9.9999 + - 0 + - 9.9999 nan_flags: null - field_number: '37' + field_number: "37" raw_drop_number: n_digits: 0 n_characters: null @@ -163,7 +163,7 @@ raw_drop_number: data_range: null nan_flags: null dimension_order: - - diameter_bin_center + - diameter_bin_center n_values: 20 field_number: null ND: @@ -174,6 +174,6 @@ ND: data_range: null nan_flags: null dimension_order: - - diameter_bin_center + - diameter_bin_center n_values: 20 - field_number: '36' + field_number: "36" diff --git a/disdrodb/l0/configs/Thies_LPM/bins_diameter.yml b/disdrodb/l0/configs/Thies_LPM/bins_diameter.yml index 470dbaf5..fb4f3456 100644 --- a/disdrodb/l0/configs/Thies_LPM/bins_diameter.yml +++ b/disdrodb/l0/configs/Thies_LPM/bins_diameter.yml @@ -23,71 +23,71 @@ center: 21: 54 bounds: 0: - - 0.125 - - 0.25 + - 0.125 + - 0.25 1: - - 0.25 - - 0.375 + - 0.25 + - 0.375 2: - - 0.375 - - 0.5 + - 0.375 + - 0.5 3: - - 0.5 - - 0.75 + - 0.5 + - 0.75 4: - - 0.75 - - 1.0 + - 0.75 + - 1.0 5: - - 1.0 - - 1.25 + - 1.0 + - 1.25 6: - - 1.25 - - 1.5 + - 1.25 + - 1.5 7: - - 1.5 - - 1.75 + - 1.5 + - 1.75 8: - - 1.75 - - 2.0 + - 1.75 + - 2.0 9: - - 2.0 - - 2.5 + - 2.0 + - 2.5 10: - - 2.5 - - 3.0 + - 2.5 + - 3.0 11: - - 3.0 - - 3.5 + - 3.0 + - 3.5 12: - - 3.5 - - 4.0 + - 3.5 + - 4.0 13: - - 4.0 - - 4.5 + - 4.0 + - 4.5 14: - - 4.5 - - 5.0 + - 4.5 + - 5.0 15: - - 5.0 - - 5.5 + - 5.0 + - 5.5 16: - - 5.5 - - 6.0 + - 5.5 + - 6.0 17: - - 6.0 - - 6.5 + - 6.0 + - 6.5 18: - - 6.5 - - 7.0 + - 6.5 + - 7.0 19: - - 7.0 - - 7.5 + - 7.0 + - 7.5 20: - - 7.5 - - 8.0 + - 7.5 + - 8.0 21: - - 8.0 - - 100 + - 8.0 + - 100 width: 0: 0.125 1: 0.125 diff --git a/disdrodb/l0/configs/Thies_LPM/bins_velocity.yml b/disdrodb/l0/configs/Thies_LPM/bins_velocity.yml index b0512fce..7868628c 100644 --- a/disdrodb/l0/configs/Thies_LPM/bins_velocity.yml +++ b/disdrodb/l0/configs/Thies_LPM/bins_velocity.yml @@ -21,65 +21,65 @@ center: 19: 15.0 bounds: 0: - - 0.0 - - 0.2 + - 0.0 + - 0.2 1: - - 0.2 - - 0.4 + - 0.2 + - 0.4 2: - - 0.4 - - 0.6 + - 0.4 + - 0.6 3: - - 0.6 - - 0.8 + - 0.6 + - 0.8 4: - - 0.8 - - 1.0 + - 0.8 + - 1.0 5: - - 1.0 - - 1.4 + - 1.0 + - 1.4 6: - - 1.4 - - 1.8 + - 1.4 + - 1.8 7: - - 1.8 - - 2.2 + - 1.8 + - 2.2 8: - - 2.2 - - 2.6 + - 2.2 + - 2.6 9: - - 2.6 - - 3.0 + - 2.6 + - 3.0 10: - - 3.0 - - 3.4 + - 3.0 + - 3.4 11: - - 3.4 - - 4.2 + - 3.4 + - 4.2 12: - - 4.2 - - 5.0 + - 4.2 + - 5.0 13: - - 5.0 - - 5.8 + - 5.0 + - 5.8 14: - - 5.8 - - 6.6 + - 5.8 + - 6.6 15: - - 6.6 - - 7.4 + - 6.6 + - 7.4 16: - - 7.4 - - 8.2 + - 7.4 + - 8.2 17: - - 8.2 - - 9.0 + - 8.2 + - 9.0 18: - - 9.0 - - 10.0 + - 9.0 + - 10.0 19: - - 10.0 - - 20.0 + - 10.0 + - 20.0 width: 0: 0.2 1: 0.2 diff --git a/disdrodb/l0/configs/Thies_LPM/l0a_encodings.yml b/disdrodb/l0/configs/Thies_LPM/l0a_encodings.yml index 265c64d0..e2e65f9a 100644 --- a/disdrodb/l0/configs/Thies_LPM/l0a_encodings.yml +++ b/disdrodb/l0/configs/Thies_LPM/l0a_encodings.yml @@ -1,79 +1,79 @@ -device_address: 'float32' # 'uint8' -weather_code_synop_4677_5min: 'float32' # 'uint8' -weather_code_synop_4680_5min : 'float32' # 'uint8' -weather_code_metar_4678_5min : 'str' -precipitation_rate_5min : 'float32' -weather_code_synop_4677: 'float32' # 'uint8' -weather_code_synop_4680: 'float32' # 'uint8' -weather_code_metar_4678: 'str' -precipitation_rate: 'float32' -rainfall_rate: 'float32' -snowfall_rate: 'float32' -precipitation_accumulated: 'float32' -mor_visibility: 'float32' # 'uint16' -reflectivity: 'float32' -quality_index: 'float32' # 'uint8' -max_hail_diameter: 'float32' -laser_status: 'float32' # 'uint8' -static_signal: 'float32' # 'uint8' -laser_temperature_analog_status: 'float32' # 'uint8' -laser_temperature_digital_status: 'float32' # 'uint8' -laser_current_analog_status: 'float32' # 'uint8' -laser_current_digital_status: 'float32' # 'uint8' -sensor_voltage_supply_status: 'float32' # 'uint8' -current_heating_pane_transmitter_head_status: 'float32' # 'uint8' -current_heating_pane_receiver_head_status: 'float32' # 'uint8' -temperature_sensor_status: 'float32' # 'uint8' -current_heating_voltage_supply_status: 'float32' # 'uint8' -current_heating_house_status: 'float32' # 'uint8' -current_heating_heads_status: 'float32' # 'uint8' -current_heating_carriers_status: 'float32' # 'uint8' -control_output_laser_power_status: 'float32' # 'uint8' -reserve_status: 'float32' # 'uint8' -temperature_interior : 'float32' # 'uint16' -laser_temperature: 'float32' # 'uint16' -laser_current_average: 'float32' # 'uint16' -control_voltage: 'float32' -optical_control_voltage_output: 'float32' -sensor_voltage_supply : 'str' -current_heating_pane_transmitter_head : 'str' -current_heating_pane_receiver_head: 'str' -temperature_ambient: 'float32' -current_heating_voltage_supply: 'str' -current_heating_house: 'str' -current_heating_heads: 'str' -current_heating_carriers: 'str' -number_particles: 'float32' # 'uint16' -number_particles_internal_data: 'float32' -number_particles_min_speed: 'float32' # 'uint16' -number_particles_min_speed_internal_data: 'float32' -number_particles_max_speed: 'float32' # 'uint16' -number_particles_max_speed_internal_data: 'float32' -number_particles_min_diameter: 'float32' # 'uint16' -number_particles_min_diameter_internal_data: 'float32' -number_particles_no_hydrometeor: 'float32' # 'uint16' -number_particles_no_hydrometeor_internal_data: 'float32' -number_particles_unknown_classification: 'float32' # 'uint16' -number_particles_unknown_classification_internal_data: 'float32' -number_particles_class_1: 'float32' # 'uint16' -number_particles_class_1_internal_data: 'float32' -number_particles_class_2: 'float32' # 'uint16' -number_particles_class_2_internal_data: 'float32' -number_particles_class_3: 'float32' # 'uint16' -number_particles_class_3_internal_data: 'float32' -number_particles_class_4: 'float32' # 'uint16' -number_particles_class_4_internal_data: 'float32' -number_particles_class_5: 'float32' # 'uint16' -number_particles_class_5_internal_data: 'float32' -number_particles_class_6: 'float32' # 'uint16' -number_particles_class_6_internal_data: 'float32' -number_particles_class_7: 'float32' # 'uint16' -number_particles_class_7_internal_data: 'float32' -number_particles_class_8: 'float32' # 'uint16' -number_particles_class_8_internal_data: 'float32' -number_particles_class_9: 'float32' # 'uint16' -number_particles_class_9_internal_data: 'float32' -raw_drop_number: 'str' +device_address: "float32" # 'uint8' +weather_code_synop_4677_5min: "float32" # 'uint8' +weather_code_synop_4680_5min: "float32" # 'uint8' +weather_code_metar_4678_5min: "str" +precipitation_rate_5min: "float32" +weather_code_synop_4677: "float32" # 'uint8' +weather_code_synop_4680: "float32" # 'uint8' +weather_code_metar_4678: "str" +precipitation_rate: "float32" +rainfall_rate: "float32" +snowfall_rate: "float32" +precipitation_accumulated: "float32" +mor_visibility: "float32" # 'uint16' +reflectivity: "float32" +quality_index: "float32" # 'uint8' +max_hail_diameter: "float32" +laser_status: "float32" # 'uint8' +static_signal: "float32" # 'uint8' +laser_temperature_analog_status: "float32" # 'uint8' +laser_temperature_digital_status: "float32" # 'uint8' +laser_current_analog_status: "float32" # 'uint8' +laser_current_digital_status: "float32" # 'uint8' +sensor_voltage_supply_status: "float32" # 'uint8' +current_heating_pane_transmitter_head_status: "float32" # 'uint8' +current_heating_pane_receiver_head_status: "float32" # 'uint8' +temperature_sensor_status: "float32" # 'uint8' +current_heating_voltage_supply_status: "float32" # 'uint8' +current_heating_house_status: "float32" # 'uint8' +current_heating_heads_status: "float32" # 'uint8' +current_heating_carriers_status: "float32" # 'uint8' +control_output_laser_power_status: "float32" # 'uint8' +reserve_status: "float32" # 'uint8' +temperature_interior: "float32" # 'uint16' +laser_temperature: "float32" # 'uint16' +laser_current_average: "float32" # 'uint16' +control_voltage: "float32" +optical_control_voltage_output: "float32" +sensor_voltage_supply: "str" +current_heating_pane_transmitter_head: "str" +current_heating_pane_receiver_head: "str" +temperature_ambient: "float32" +current_heating_voltage_supply: "str" +current_heating_house: "str" +current_heating_heads: "str" +current_heating_carriers: "str" +number_particles: "float32" # 'uint16' +number_particles_internal_data: "float32" +number_particles_min_speed: "float32" # 'uint16' +number_particles_min_speed_internal_data: "float32" +number_particles_max_speed: "float32" # 'uint16' +number_particles_max_speed_internal_data: "float32" +number_particles_min_diameter: "float32" # 'uint16' +number_particles_min_diameter_internal_data: "float32" +number_particles_no_hydrometeor: "float32" # 'uint16' +number_particles_no_hydrometeor_internal_data: "float32" +number_particles_unknown_classification: "float32" # 'uint16' +number_particles_unknown_classification_internal_data: "float32" +number_particles_class_1: "float32" # 'uint16' +number_particles_class_1_internal_data: "float32" +number_particles_class_2: "float32" # 'uint16' +number_particles_class_2_internal_data: "float32" +number_particles_class_3: "float32" # 'uint16' +number_particles_class_3_internal_data: "float32" +number_particles_class_4: "float32" # 'uint16' +number_particles_class_4_internal_data: "float32" +number_particles_class_5: "float32" # 'uint16' +number_particles_class_5_internal_data: "float32" +number_particles_class_6: "float32" # 'uint16' +number_particles_class_6_internal_data: "float32" +number_particles_class_7: "float32" # 'uint16' +number_particles_class_7_internal_data: "float32" +number_particles_class_8: "float32" # 'uint16' +number_particles_class_8_internal_data: "float32" +number_particles_class_9: "float32" # 'uint16' +number_particles_class_9_internal_data: "float32" +raw_drop_number: "str" # '521': air_temperature # '522': relative_humidity # '523': wind_speed diff --git a/disdrodb/l0/configs/Thies_LPM/l0b_cf_attrs.yml b/disdrodb/l0/configs/Thies_LPM/l0b_cf_attrs.yml index 7cc63b21..79abe16f 100644 --- a/disdrodb/l0/configs/Thies_LPM/l0b_cf_attrs.yml +++ b/disdrodb/l0/configs/Thies_LPM/l0b_cf_attrs.yml @@ -1,19 +1,19 @@ device_address: description: "Device address (factory setting \u201E00\") (NN)" long_name: Device address - units: '' + units: "" weather_code_synop_4677_5min: description: 5M SYNOP Tab.4677 (5 minutes mean value) (NN) long_name: 5 min SYNOP Tab.4677 - units: '' + units: "" weather_code_synop_4680_5min: description: 5M SYNOP Tab.4680 (5 minutes mean value) (NN) long_name: 5 min SYNOP Tab.4680 - units: '' + units: "" weather_code_metar_4678_5min: description: 5M METAR Tab.4678 (5 minutes mean value) (AAAAA) long_name: 5 min METAR Tab.4678 - units: '' + units: "" precipitation_rate_5min: description: "5M Intensit\xE4t [mm/h] (5 minutes mean value) (NNN.NNN)" long_name: 5 min precipitation rate @@ -21,15 +21,15 @@ precipitation_rate_5min: weather_code_synop_4677: description: 1M SYNOP Tab.4677 (1 minute value) (NN) long_name: SYNOP Tab.4677 - units: '' + units: "" weather_code_synop_4680: description: 1M SYNOP Tab.4680 (1 minute value) (NN) long_name: SYNOP Tab.4680 - units: '' + units: "" weather_code_metar_4678: description: 1M METAR Tab.4678 (1 minute value) (AAAAA) long_name: METAR Tab.4678 - units: '' + units: "" precipitation_rate: description: 1M Intensity [mm/h] total precipitation (1 minute value) (NNN.NNN) long_name: Total precipitation intensity @@ -49,7 +49,7 @@ precipitation_accumulated: mor_visibility: description: 1M Visibility in precipitation [0...99999m] (1 minute value) (NNNNN) long_name: Visibility in precipitation - units: '' + units: "" reflectivity: description: 1M Radar reflectivity [-9.9...99.9dBZ] (1 minute value) (NN.N) long_name: Radar reflectivity @@ -57,7 +57,7 @@ reflectivity: quality_index: description: 1M Measuring quality [0...100%] (1 minute value) (NNN) long_name: Quility index - units: '%' + units: "%" max_hail_diameter: description: 1M Maximum diameter hail [mm] (1 minute value) (N.N) long_name: Maximum diameter hail @@ -65,67 +65,67 @@ max_hail_diameter: laser_status: description: Status Laser (OK/on:0, off:1) long_name: Status laser - units: '' + units: "" static_signal: description: Static signal (OK:0, Error:1) long_name: Static signal - units: '' + units: "" laser_temperature_analog_status: description: Status Laser temperature (analogue) (OK:0, Error:1) long_name: Status analogue laser temperature - units: '' + units: "" laser_temperature_digital_status: description: Status Laser temperature (digital) (OK:0, Error:1) long_name: Status digital laser temperature - units: '' + units: "" laser_current_analog_status: description: Status Laser current (analogue) (OK:0, Error:1) long_name: Status analogue laser current - units: '' + units: "" laser_current_digital_status: description: Status Laser current (digital) (OK:0, Error:1) long_name: Status digital laser current - units: '' + units: "" sensor_voltage_supply_status: description: Status Sensor supply (OK:0, Error:1) long_name: Status sensor supply - units: '' + units: "" current_heating_pane_transmitter_head_status: description: Status Current pane heating laser head (OK:0, warning:1) long_name: Status current pane heating laser head - units: '' + units: "" current_heating_pane_receiver_head_status: description: Status Current pane heating receiver head (OK:0, warning:1) long_name: Status current pane heating reciver head - units: '' + units: "" temperature_sensor_status: description: Status Temperature sensor (OK:0, warning:1) long_name: Status temperature sensor - units: '' + units: "" current_heating_voltage_supply_status: description: Status Heating supply (OK:0, warning:1) long_name: Status heating supply - units: '' + units: "" current_heating_house_status: description: Status Current heating housing (OK:0, warning:1) long_name: Status current heating housing - units: '' + units: "" current_heating_heads_status: description: Status Current heating heads (OK:0, warning:1) long_name: Status current heating heads - units: '' + units: "" current_heating_carriers_status: description: Status Current heating carriers (OK:0, warning:1) long_name: Status current heating carriers - units: '' + units: "" control_output_laser_power_status: description: Status Control output laser power (OK:0, warning:1) long_name: Status control output laser - units: '' + units: "" reserve_status: description: Reserve Status (0) long_name: Reserve status - units: '' + units: "" temperature_interior: description: "Interior temperature [\xB0C] (NNN)" long_name: Interior temperature @@ -163,12 +163,14 @@ temperature_ambient: long_name: Ambient temperature units: "\xB0C" current_heating_voltage_supply: - description: Voltage Heating supply [1/10 V] (only 5.4110.x1.xxx, otherwise "999") + description: + Voltage Heating supply [1/10 V] (only 5.4110.x1.xxx, otherwise "999") (NNN) long_name: Voltage heating supply units: V current_heating_house: - description: Current heating housing [mA] (only 5.4110.x1.xxx, otherwise "9999") + description: + Current heating housing [mA] (only 5.4110.x1.xxx, otherwise "9999") (NNNN) long_name: Current heating housing units: mA @@ -177,131 +179,132 @@ current_heating_heads: long_name: Current heating head units: mA current_heating_carriers: - description: Current heating carriers [mA] (only 5.4110.x1.xxx, otherwise "9999") + description: + Current heating carriers [mA] (only 5.4110.x1.xxx, otherwise "9999") (NNNN) long_name: Current heating carriers units: mA number_particles: description: Number of all measured particles (NNNNN) long_name: Number of all measured particles - units: '' + units: "" number_particles_internal_data: description: "\u201E00000.000\" (internal data)" long_name: Number of all measured particles internal data - units: '' + units: "" number_particles_min_speed: description: Number of particles < minimal speed (0.15m/s) (NNNNN) long_name: Number of particles < minimal speed (0.15m/s) - units: '' + units: "" number_particles_min_speed_internal_data: description: "\u201E00000.000\" (internal data)" long_name: Number of particles < minimal speed (0.15m/s) internal data - units: '' + units: "" number_particles_max_speed: description: Number of particles > maximal speed (20m/s) (NNNNN) long_name: Number of particles > maximal speed (20m/s) - units: '' + units: "" number_particles_max_speed_internal_data: description: "\u201E00000.000\" (internal data)" long_name: Number of particles > maximal speed (20m/s) internal data - units: '' + units: "" number_particles_min_diameter: description: Number of particles < minimal diameter (0.15mm) (NNNNN) long_name: Number of particles < minimal diameter (0.15mm) - units: '' + units: "" number_particles_min_diameter_internal_data: description: "\u201E00000.000\" (internal data)" long_name: Number of particles < minimal diameter (0.15mm) internal data - units: '' + units: "" number_particles_no_hydrometeor: description: Number of particles no hydrometeor long_name: Number of particles no hydrometeor - units: '' + units: "" number_particles_no_hydrometeor_internal_data: description: Total volume (gross) of this class long_name: Total volume of this class - units: '' + units: "" number_particles_unknown_classification: description: Number of particles with unknown classification long_name: Number of particles with unknown classification - units: '' + units: "" number_particles_unknown_classification_internal_data: description: Total volume (gross) of this class long_name: Number of particles with unknown classification internal data - units: '' + units: "" number_particles_class_1: description: Number of particles class 1 long_name: Number of particles class 1 - units: '' + units: "" number_particles_class_1_internal_data: description: Total volume (gross) of class 1 long_name: Total volume of class 1 - units: '' + units: "" number_particles_class_2: description: Number of particles class 2 long_name: Number of particles class 2 - units: '' + units: "" number_particles_class_2_internal_data: description: Total volume (gross) of class 2 long_name: Total volume of class 2 - units: '' + units: "" number_particles_class_3: description: Number of particles class 3 long_name: Number of particles class 3 - units: '' + units: "" number_particles_class_3_internal_data: description: Total volume (gross) of class 3 long_name: Total volume of class 3 - units: '' + units: "" number_particles_class_4: description: Number of particles class 4 long_name: Number of particles class 4 - units: '' + units: "" number_particles_class_4_internal_data: description: Total volume (gross) of class 4 long_name: Total volume of class 4 - units: '' + units: "" number_particles_class_5: description: Number of particles class 5 long_name: Number of particles class 5 - units: '' + units: "" number_particles_class_5_internal_data: description: Total volume (gross) of class 5 long_name: Total volume of class 5 - units: '' + units: "" number_particles_class_6: description: Number of particles class 6 long_name: Number of particles class 6 - units: '' + units: "" number_particles_class_6_internal_data: description: Total volume (gross) of class 6 long_name: Total volume of class 6 - units: '' + units: "" number_particles_class_7: description: Number of particles class 7 long_name: Number of particles class 7 - units: '' + units: "" number_particles_class_7_internal_data: description: Total volume (gross) of class 7 long_name: Total volume of class 7 - units: '' + units: "" number_particles_class_8: description: Number of particles class 8 long_name: Number of particles class 8 - units: '' + units: "" number_particles_class_8_internal_data: description: Total volume (gross) of class 8 long_name: Total volume of class 8 - units: '' + units: "" number_particles_class_9: description: Number of particles class 9 long_name: Number of particles class 9 - units: '' + units: "" number_particles_class_9_internal_data: description: Total volume (gross) of class 9 long_name: Total volume of class 9 - units: '' + units: "" raw_drop_number: description: Precipitation spectrum long_name: Precipitation spectrum - units: '' + units: "" diff --git a/disdrodb/l0/configs/Thies_LPM/l0b_encodings.yml b/disdrodb/l0/configs/Thies_LPM/l0b_encodings.yml index b28d7467..ce1f50be 100644 --- a/disdrodb/l0/configs/Thies_LPM/l0b_encodings.yml +++ b/disdrodb/l0/configs/Thies_LPM/l0b_encodings.yml @@ -690,6 +690,6 @@ raw_drop_number: contiguous: false _FillValue: 65535 chunksizes: - - 5000 - - 22 # diameter - - 20 # velocity + - 5000 + - 22 # diameter + - 20 # velocity diff --git a/disdrodb/l0/configs/Thies_LPM/raw_data_format.yml b/disdrodb/l0/configs/Thies_LPM/raw_data_format.yml index 4cf714fa..a468bb8a 100644 --- a/disdrodb/l0/configs/Thies_LPM/raw_data_format.yml +++ b/disdrodb/l0/configs/Thies_LPM/raw_data_format.yml @@ -4,40 +4,40 @@ start_identifier: n_decimals: 0 n_naturals: 1 data_range: - - 0 - - 9 + - 0 + - 9 nan_flags: null - field_number: '01' + field_number: "01" device_address: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - 0 - - 99 + - 0 + - 99 nan_flags: null - field_number: '02' + field_number: "02" sensor_serial_number: n_digits: 4 n_characters: 4 n_decimals: 0 n_naturals: 4 data_range: - - 0 - - 9999 + - 0 + - 9999 nan_flags: null - field_number: '03' + field_number: "03" software_version: n_digits: 3 n_characters: 4 n_decimals: 1 n_naturals: 2 data_range: - - 0 - - 9.99 + - 0 + - 9.99 nan_flags: null - field_number: '04' + field_number: "04" sensor_date: n_digits: 6 n_characters: 8 @@ -45,7 +45,7 @@ sensor_date: n_naturals: 0 data_range: null nan_flags: null - field_number: '05' + field_number: "05" sensor_time: n_digits: 6 n_characters: 8 @@ -53,25 +53,25 @@ sensor_time: n_naturals: 0 data_range: null nan_flags: null - field_number: '06' + field_number: "06" weather_code_synop_4677_5min: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - -4 - - 90 + - -4 + - 90 nan_flags: -1 - field_number: '07' + field_number: "07" weather_code_synop_4680_5min: n_digits: 2 n_characters: 2 n_decimals: 0 n_naturals: 2 data_range: - - -1 - - 89 + - -1 + - 89 nan_flags: -1 field_number: 08 weather_code_metar_4678_5min: @@ -88,30 +88,30 @@ precipitation_rate_5min: n_decimals: 3 n_naturals: 3 data_range: - - 0.0 - - 999.999 + - 0.0 + - 999.999 nan_flags: null - field_number: '10' + field_number: "10" weather_code_synop_4677: n_digits: 0 n_characters: 5 n_decimals: 0 n_naturals: 0 data_range: - - -4 - - 90 + - -4 + - 90 nan_flags: -1 - field_number: '11' + field_number: "11" weather_code_synop_4680: n_digits: 0 n_characters: 5 n_decimals: 0 n_naturals: 0 data_range: - - -1 - - 89 + - -1 + - 89 nan_flags: -1 - field_number: '12' + field_number: "12" weather_code_metar_4678: n_digits: 0 n_characters: 5 @@ -119,319 +119,319 @@ weather_code_metar_4678: n_naturals: 0 data_range: null nan_flags: null - field_number: '13' + field_number: "13" precipitation_rate: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0.0 - - 999.999 + - 0.0 + - 999.999 nan_flags: null - field_number: '14' + field_number: "14" rainfall_rate: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0.0 - - 999.999 + - 0.0 + - 999.999 nan_flags: null - field_number: '15' + field_number: "15" snowfall_rate: n_digits: 6 n_characters: 7 n_decimals: 3 n_naturals: 3 data_range: - - 0.0 - - 999.999 + - 0.0 + - 999.999 nan_flags: null - field_number: '16' + field_number: "16" precipitation_accumulated: n_digits: 6 n_characters: 7 n_decimals: 4 n_naturals: 2 data_range: - - 0.0 - - 9999.99 + - 0.0 + - 9999.99 nan_flags: null - field_number: '17' + field_number: "17" mor_visibility: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '18' + field_number: "18" reflectivity: n_digits: 3 n_characters: 4 n_decimals: 2 n_naturals: 1 data_range: - - -9.9 - - 99.9 + - -9.9 + - 99.9 nan_flags: null - field_number: '19' + field_number: "19" quality_index: n_digits: 3 n_characters: 3 n_decimals: 3 n_naturals: 0 data_range: - - 0 - - 100 + - 0 + - 100 nan_flags: 999 - field_number: '20' + field_number: "20" max_hail_diameter: n_digits: 2 n_characters: 3 n_decimals: 1 n_naturals: 1 data_range: - - 0.0 - - 9.9 + - 0.0 + - 9.9 nan_flags: null - field_number: '21' + field_number: "21" laser_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '22' + - 0 + - 1 + field_number: "22" static_signal: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null - field_number: '23' + field_number: "23" laser_temperature_analog_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '24' + - 0 + - 1 + field_number: "24" laser_temperature_digital_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '25' + - 0 + - 1 + field_number: "25" laser_current_analog_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '26' + - 0 + - 1 + field_number: "26" laser_current_digital_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '27' + - 0 + - 1 + field_number: "27" sensor_voltage_supply_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '28' + - 0 + - 1 + field_number: "28" current_heating_pane_transmitter_head_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null - field_number: '29' + field_number: "29" current_heating_pane_receiver_head_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '30' + - 0 + - 1 + field_number: "30" temperature_sensor_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '31' + - 0 + - 1 + field_number: "31" current_heating_voltage_supply_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '32' + - 0 + - 1 + field_number: "32" current_heating_house_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '33' + - 0 + - 1 + field_number: "33" current_heating_heads_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '34' + - 0 + - 1 + field_number: "34" current_heating_carriers_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '35' + - 0 + - 1 + field_number: "35" control_output_laser_power_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 1 + - 0 + - 1 nan_flags: null valid_values: - - 0 - - 1 - field_number: '36' + - 0 + - 1 + field_number: "36" reserve_status: n_digits: 1 n_characters: 1 n_decimals: 1 n_naturals: 0 data_range: - - 0 - - 0 + - 0 + - 0 nan_flags: null valid_values: - - 0 - - 1 - field_number: '37' + - 0 + - 1 + field_number: "37" temperature_interior: n_digits: 3 n_characters: 3 n_decimals: 3 n_naturals: 0 data_range: - - -30 - - 999 + - -30 + - 999 nan_flags: null - field_number: '38' + field_number: "38" laser_temperature: n_digits: 2 n_characters: 2 n_decimals: 2 n_naturals: 0 data_range: - - 0 - - 80 + - 0 + - 80 nan_flags: null - field_number: '39' + field_number: "39" laser_current_average: n_digits: 4 n_characters: 4 n_decimals: 4 n_naturals: 0 data_range: - - 0 - - 9999 + - 0 + - 9999 nan_flags: null - field_number: '40' + field_number: "40" control_voltage: n_digits: 4 n_characters: 4 @@ -439,17 +439,17 @@ control_voltage: n_naturals: 0 data_range: null nan_flags: null - field_number: '41' + field_number: "41" optical_control_voltage_output: n_digits: 4 n_characters: 4 n_decimals: 4 n_naturals: 0 data_range: - - 0 - - 9999 + - 0 + - 9999 nan_flags: null - field_number: '42' + field_number: "42" sensor_voltage_supply: n_digits: 3 n_characters: 3 @@ -457,7 +457,7 @@ sensor_voltage_supply: n_naturals: 0 data_range: null nan_flags: null - field_number: '43' + field_number: "43" current_heating_pane_transmitter_head: n_digits: 3 n_characters: 3 @@ -465,7 +465,7 @@ current_heating_pane_transmitter_head: n_naturals: 0 data_range: null nan_flags: null - field_number: '44' + field_number: "44" current_heating_pane_receiver_head: n_digits: 3 n_characters: 3 @@ -473,17 +473,17 @@ current_heating_pane_receiver_head: n_naturals: 0 data_range: null nan_flags: null - field_number: '45' + field_number: "45" temperature_ambient: n_digits: 4 n_characters: 5 n_decimals: 3 n_naturals: 1 data_range: - - -999.9 - - 99999.0 + - -999.9 + - 99999.0 nan_flags: null - field_number: '46' + field_number: "46" current_heating_voltage_supply: n_digits: 3 n_characters: 3 @@ -491,7 +491,7 @@ current_heating_voltage_supply: n_naturals: 0 data_range: null nan_flags: null - field_number: '47' + field_number: "47" current_heating_house: n_digits: 4 n_characters: 4 @@ -499,7 +499,7 @@ current_heating_house: n_naturals: 0 data_range: null nan_flags: null - field_number: '48' + field_number: "48" current_heating_heads: n_digits: 4 n_characters: 4 @@ -507,7 +507,7 @@ current_heating_heads: n_naturals: 0 data_range: null nan_flags: null - field_number: '49' + field_number: "49" current_heating_carriers: n_digits: 4 n_characters: 4 @@ -515,307 +515,307 @@ current_heating_carriers: n_naturals: 0 data_range: null nan_flags: null - field_number: '50' + field_number: "50" number_particles: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '51' + field_number: "51" number_particles_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '52' + field_number: "52" number_particles_min_speed: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '53' + field_number: "53" number_particles_min_speed_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '54' + field_number: "54" number_particles_max_speed: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '55' + field_number: "55" number_particles_max_speed_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '56' + field_number: "56" number_particles_min_diameter: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '57' + field_number: "57" number_particles_min_diameter_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '58' + field_number: "58" number_particles_no_hydrometeor: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '59' + field_number: "59" number_particles_no_hydrometeor_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '60' + field_number: "60" number_particles_unknown_classification: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '61' + field_number: "61" number_particles_unknown_classification_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '62' + field_number: "62" number_particles_class_1: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '63' + field_number: "63" number_particles_class_1_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '64' + field_number: "64" number_particles_class_2: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '65' + field_number: "65" number_particles_class_2_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '66' + field_number: "66" number_particles_class_3: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '67' + field_number: "67" number_particles_class_3_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '68' + field_number: "68" number_particles_class_4: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '69' + field_number: "69" number_particles_class_4_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '70' + field_number: "70" number_particles_class_5: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '71' + field_number: "71" number_particles_class_5_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '72' + field_number: "72" number_particles_class_6: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '73' + field_number: "73" number_particles_class_6_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '74' + field_number: "74" number_particles_class_7: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '75' + field_number: "75" number_particles_class_7_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '76' + field_number: "76" number_particles_class_8: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '77' + field_number: "77" number_particles_class_8_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '78' + field_number: "78" number_particles_class_9: n_digits: 5 n_characters: 5 n_decimals: 5 n_naturals: 0 data_range: - - 0 - - 99999 + - 0 + - 99999 nan_flags: null - field_number: '79' + field_number: "79" number_particles_class_9_internal_data: n_digits: 8 n_characters: 9 n_decimals: 5 n_naturals: 3 data_range: - - 0.0 - - 99999.999 + - 0.0 + - 99999.999 nan_flags: null - field_number: '80' + field_number: "80" raw_drop_number: n_digits: 1317 n_characters: 1756 @@ -824,7 +824,7 @@ raw_drop_number: data_range: null nan_flags: null dimension_order: - - diameter_bin_center - - velocity_bin_center + - diameter_bin_center + - velocity_bin_center n_values: 440 - field_number: '81' + field_number: "81" diff --git a/disdrodb/l0/l0_processing.py b/disdrodb/l0/l0_processing.py index 44b5d866..d126c7af 100644 --- a/disdrodb/l0/l0_processing.py +++ b/disdrodb/l0/l0_processing.py @@ -385,38 +385,29 @@ def run_l0a( The path should have the following structure: ``<...>/DISDRODB/Processed/<DATA_SOURCE>/<CAMPAIGN_NAME>``. For testing purposes, this function exceptionally accepts also a directory path simply ending with ``<CAMPAIGN_NAME>`` (e.g., ``/tmp/<CAMPAIGN_NAME>``). - station_name : str - The name of the station. - + The name of the station. glob_patterns : str Glob pattern to search for data files in ``<raw_dir>/data/<station_name>``. - column_names : list Column names of the raw text file. - reader_kwargs : dict Arguments for Pandas ``read_csv`` function to open the text file. - df_sanitizer_fun : callable, optional Sanitizer function to format the DataFrame into DISDRODB L0A standard. Default is ``None``. - parallel : bool, optional If ``True``, process the files simultaneously in multiple processes. The number of simultaneous processes can be customized using the ``dask.distributed.LocalCluster``. If ``False``, process the files sequentially in a single process. Default is ``False``. - verbose : bool, optional If ``True``, print detailed processing information to the terminal. Default is ``False``. - force : bool, optional If ``True``, overwrite existing data in destination directories. If ``False``, raise an error if data already exists in destination directories. Default is ``False``. - debugging_mode : bool, optional If ``True``, reduce the amount of data to process. Processes only the first 100 rows of 3 raw data files. @@ -519,25 +510,20 @@ def run_l0b( - The ``<CAMPAIGN_NAME>`` must semantically match between: - the ``raw_dir`` and ``processed_dir`` directory paths; - with the key ``campaign_name`` within the metadata YAML files. - processed_dir : str The desired directory path for the processed DISDRODB L0A and L0B products. The path should have the following structure: ``<...>/DISDRODB/Processed/<DATA_SOURCE>/<CAMPAIGN_NAME>``. For testing purposes, this function exceptionally accepts also a directory path simply ending with ``<CAMPAIGN_NAME>`` (e.g., ``/tmp/<CAMPAIGN_NAME>``). - station_name : str The name of the station. - force : bool, optional If ``True``, overwrite existing data in destination directories. If ``False``, raise an error if data already exists in destination directories. Default is ``False``. - verbose : bool, optional If ``True``, print detailed processing information to the terminal. Default is ``True``. - parallel : bool, optional If ``True``, process the files simultaneously in multiple processes. The number of simultaneous processes can be customized using the ``dask.distributed.LocalCluster``. @@ -545,7 +531,6 @@ def run_l0b( Also, ensure to set the ``HDF5_USE_FILE_LOCKING`` environment variable to ``False``. If ``False``, process the files sequentially in a single process. Default is ``False``. - debugging_mode : bool, optional If ``True``, reduce the amount of data to process. Only the first 3 raw data files will be processed. diff --git a/disdrodb/l0/l0a_processing.py b/disdrodb/l0/l0a_processing.py index 03e25ae5..9c6540cb 100644 --- a/disdrodb/l0/l0a_processing.py +++ b/disdrodb/l0/l0a_processing.py @@ -159,7 +159,7 @@ def remove_rows_with_missing_time(df: pd.DataFrame, verbose: bool = False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. verbose : bool Whether to verbose the processing. The default is ``False``. @@ -193,7 +193,7 @@ def remove_duplicated_timesteps(df: pd.DataFrame, verbose: bool = False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. verbose : bool Whether to verbose the processing. The default is ``False``. @@ -251,7 +251,7 @@ def remove_issue_timesteps(df, issue_dict, verbose=False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. issue_dict : dict Issue dictionary. @@ -293,7 +293,7 @@ def cast_column_dtypes(df: pd.DataFrame, sensor_name: str) -> pd.DataFrame: Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -330,7 +330,7 @@ def coerce_corrupted_values_to_nan(df: pd.DataFrame, sensor_name: str) -> pd.Dat Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -361,7 +361,7 @@ def strip_string_spaces(df: pd.DataFrame, sensor_name: str) -> pd.DataFrame: Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -456,7 +456,7 @@ def replace_nan_flags(df, sensor_name, verbose=False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -491,7 +491,7 @@ def set_nan_outside_data_range(df, sensor_name, verbose=False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -530,7 +530,7 @@ def set_nan_invalid_values(df, sensor_name, verbose=False): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. sensor_name : str Name of the sensor. @@ -679,7 +679,7 @@ def write_l0a( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. filepath : str Output file path. diff --git a/disdrodb/l0/l0b_nc_processing.py b/disdrodb/l0/l0b_nc_processing.py index 9bac1db1..121b8185 100644 --- a/disdrodb/l0/l0b_nc_processing.py +++ b/disdrodb/l0/l0b_nc_processing.py @@ -141,7 +141,7 @@ def preprocess_raw_netcdf(ds, dict_names, sensor_name): Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Raw netCDF to be converted to DISDRODB standards. dict_names : dict Dictionary mapping raw netCDF variables/coordinates/dimension names @@ -151,7 +151,7 @@ def preprocess_raw_netcdf(ds, dict_names, sensor_name): Returns ------- - ds : xr.Dataset + ds : xarray.Dataset xarray Dataset with variables compliant to DISDRODB conventions. """ @@ -185,7 +185,7 @@ def replace_custom_nan_flags(ds, dict_nan_flags, verbose=False): Parameters ---------- - df : xr.Dataset + df : xarray.Dataset Input xarray dataset dict_nan_flags : dict Dictionary with nan flags value to set as ``np.nan``. @@ -218,7 +218,7 @@ def replace_nan_flags(ds, sensor_name, verbose): Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset dict_nan_flags : dict Dictionary with nan flags value to set as np.nan @@ -242,7 +242,7 @@ def set_nan_outside_data_range(ds, sensor_name, verbose): Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset sensor_name : str Name of the sensor. @@ -280,7 +280,7 @@ def set_nan_invalid_values(ds, sensor_name, verbose): Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset sensor_name : str Name of the sensor. @@ -323,7 +323,7 @@ def create_l0b_from_raw_nc( Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Raw xarray dataset dict_names : dict Dictionary mapping raw netCDF variables/coordinates/dimension names diff --git a/disdrodb/l0/l0b_processing.py b/disdrodb/l0/l0b_processing.py index 9f51f133..7741535d 100644 --- a/disdrodb/l0/l0b_processing.py +++ b/disdrodb/l0/l0b_processing.py @@ -203,7 +203,7 @@ def retrieve_l0b_arrays( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe sensor_name : str Name of the sensor @@ -287,7 +287,7 @@ def _convert_object_variables_to_string(ds: xr.Dataset) -> xr.Dataset: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input dataset. Returns @@ -306,7 +306,7 @@ def _set_variable_attributes(ds: xr.Dataset, sensor_name: str) -> xr.Dataset: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input dataset. sensor_name : str Name of the sensor. @@ -431,7 +431,7 @@ def create_l0b_from_l0a( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame DISDRODB L0A dataframe. attrs : dict Station metadata. @@ -503,7 +503,7 @@ def sanitize_encodings_dict(encoding_dict: dict, ds: xr.Dataset) -> dict: ---------- encoding_dict : dict Dictionary containing the encoding to write DISDRODB L0B netCDFs. - ds : xr.Dataset + ds : xarray.Dataset Input dataset. Returns @@ -525,7 +525,7 @@ def rechunk_dataset(ds: xr.Dataset, encoding_dict: dict) -> xr.Dataset: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset encoding_dict : dict Dictionary containing the encoding to write the xarray dataset as a netCDF. @@ -549,7 +549,7 @@ def set_encodings(ds: xr.Dataset, sensor_name: str) -> xr.Dataset: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset. sensor_name : str Name of the sensor. @@ -585,7 +585,7 @@ def write_l0b(ds: xr.Dataset, filepath: str, force=False) -> None: Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input xarray dataset. filepath : str Output file path. diff --git a/disdrodb/l0/template_tools.py b/disdrodb/l0/template_tools.py index 019deb6f..0a5ba21c 100644 --- a/disdrodb/l0/template_tools.py +++ b/disdrodb/l0/template_tools.py @@ -72,7 +72,7 @@ def print_df_column_names(df: pd.DataFrame) -> None: Parameters ---------- - df : dataframe + df : pandas.DataFrame The dataframe. """ for i, column in enumerate(df.columns): @@ -106,7 +106,7 @@ def print_df_with_any_nan_rows(df: pd.DataFrame) -> None: Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. """ df_bool_is_nan = df.isna() @@ -123,7 +123,7 @@ def print_df_first_n_rows(df: pd.DataFrame, n: int = 5, print_column_names: bool Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe. n : int, optional Number of row. The default is 5. @@ -178,7 +178,7 @@ def print_df_summary_stats( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe column_indices : Union[int,slice,list], optional Column indices. If ``None``, select all columns. @@ -219,7 +219,7 @@ def print_df_columns_unique_values( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe column_indices : Union[int,slice,list], optional Column indices. If ``None``, select all columns. @@ -247,7 +247,7 @@ def get_df_columns_unique_values_dict( Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe column_indices : Union[int,slice,list], optional Column indices. If ``None``, select all columns. @@ -274,7 +274,8 @@ def str_is_number(string: str) -> bool: Parameters ---------- - string : Input string. + string : str + Input string. Returns @@ -294,7 +295,8 @@ def str_is_integer(string: str) -> bool: Parameters ---------- - string : Input string. + string : str + Input string. Returns @@ -314,7 +316,7 @@ def str_has_decimal_digits(string: str) -> bool: Parameters ---------- - string : + string : str Input string. @@ -488,7 +490,7 @@ def infer_column_names(df: pd.DataFrame, sensor_name: str, row_idx: int = 1): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame The dataframe to analyse. sensor_name : str name of the sensor. diff --git a/disdrodb/tests/data/check_readers/DISDRODB/Raw/EPFL/PARSIVEL_2007/metadata/10.yml b/disdrodb/tests/data/check_readers/DISDRODB/Raw/EPFL/PARSIVEL_2007/metadata/10.yml index ccb3bea0..8e63baff 100644 --- a/disdrodb/tests/data/check_readers/DISDRODB/Raw/EPFL/PARSIVEL_2007/metadata/10.yml +++ b/disdrodb/tests/data/check_readers/DISDRODB/Raw/EPFL/PARSIVEL_2007/metadata/10.yml @@ -1,57 +1,58 @@ data_source: EPFL campaign_name: PARSIVEL_2007 -station_name: '10' +station_name: "10" sensor_name: OTT_Parsivel reader: EPFL/PARSIVEL_2007 raw_data_format: txt platform_type: fixed -disdrodb_data_url: '' -source: '' -source_convention: '' -source_processing_date: '' +disdrodb_data_url: "" +source: "" +source_convention: "" +source_processing_date: "" title: PARSIVEL_2007 -description: '' -project_name: '' -keywords: '' -summary: '' -history: '' -comment: '' -station_id: '10' -location: '' -country: '' -continent: '' +description: "" +project_name: "" +keywords: "" +summary: "" +history: "" +comment: "" +station_id: "10" +location: "" +country: "" +continent: "" latitude: -9999 longitude: -9999 altitude: -9999 -deployment_status: '' -deployment mode: '' -platform_protection: '' -platform_orientation: '' +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" sensor_long_name: OTT Hydromet Parsivel -sensor_manufacturer: '' -sensor_wavelength: '' -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_length: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -calibration_sensitivity: '' -calibration_certification_date: '' -calibration_certification_url: '' -contributors: '' -authors: '' -authors_url: '' -contact: '' +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" contact_information: http://lte.epfl.ch -acknowledgement: '' -references: '' -documentation: '' -website: '' -institution: Laboratoire de Teledetection Environnementale - Ecole Polytechnique +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: + Laboratoire de Teledetection Environnementale - Ecole Polytechnique Federale de Lausanne -source_repository: '' -license: '' -doi: '' +source_repository: "" +license: "" +doi: "" diff --git a/disdrodb/tests/data/check_readers/DISDRODB/Raw/UK/DIVEN/metadata/CAIRNGORM.yml b/disdrodb/tests/data/check_readers/DISDRODB/Raw/UK/DIVEN/metadata/CAIRNGORM.yml index 1e88fea4..54be96d8 100755 --- a/disdrodb/tests/data/check_readers/DISDRODB/Raw/UK/DIVEN/metadata/CAIRNGORM.yml +++ b/disdrodb/tests/data/check_readers/DISDRODB/Raw/UK/DIVEN/metadata/CAIRNGORM.yml @@ -6,52 +6,53 @@ reader: UK/DIVEN raw_data_format: netcdf platform_type: fixed source: NCAS Laser Precipitation Monitor -source_convention: '' -source_processing_date: '' +source_convention: "" +source_processing_date: "" title: cairngorm -description: 1 minute precipitation characteristics output for a single day from a +description: + 1 minute precipitation characteristics output for a single day from a DiVeN disdrometer at Cairngorm project_name: Disdrometer Verification Network (DiVeN) -keywords: '' -summary: '' -history: '' -comment: '' -station_id: '11' +keywords: "" +summary: "" +history: "" +comment: "" +station_id: "11" location: Cairngorm country: United Kingdom continent: Europe latitude: 57.0063 longitude: -3.6628 altitude: 781 -deployment_status: '' -deployment mode: '' -platform_protection: '' -platform_orientation: '' +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" sensor_long_name: Thies_LPM -sensor_manufacturer: '' +sensor_manufacturer: "" sensor_wavelength: 2143 mv mm-2 -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_length: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -calibration_sensitivity: '' -calibration_certification_date: '' -calibration_certification_url: '' +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" contributors: Ryan R. Neely III authors: Ben Pickering -authors_url: '' +authors_url: "" contact: Ben Pickering contact_information: ben.pickering@ncas.ac.uk -acknowledgement: '' -references: '' -documentation: '' -website: '' +acknowledgement: "" +references: "" +documentation: "" +website: "" institution: National Centre for Atmospheric Science (NCAS) source_repository: https://orcid.org/0000-0002-8474-9005 -license: '' -doi: '' -disdrodb_data_url: '' +license: "" +doi: "" +disdrodb_data_url: "" diff --git a/disdrodb/tests/data/test_dir_structure/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml b/disdrodb/tests/data/test_dir_structure/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml index db58546f..4d04c7be 100644 --- a/disdrodb/tests/data/test_dir_structure/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml +++ b/disdrodb/tests/data/test_dir_structure/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml @@ -1,15 +1,15 @@ -title: 'CABAUW' -description: '' -source: '' -history: '' -conventions: '' -campaign_name: 'DELFT' -project_name: '' -station_name: 'STATION_NAME' -station_id: '' -location: '' -country: 'Netherlands' -continent: 'Europe' +title: "CABAUW" +description: "" +source: "" +history: "" +conventions: "" +campaign_name: "DELFT" +project_name: "" +station_name: "STATION_NAME" +station_id: "" +location: "" +country: "Netherlands" +continent: "Europe" latitude: 51.96832449 longitude: 4.92921567 altitude: 1 @@ -19,25 +19,25 @@ EPSG: 4326 latitude_unit: DegreesNorth longitude_unit: DegreesEast altitude_unit: MetersAboveSeaLevel -sensor_name: 'sensor-name' -sensor_long_name: 'OTT Hydromet Parsivel 2' -sensor_wavelength: '' -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -contributors: '' -authors: '' -institution: '' -references: '' -documentation: '' -website: '' -source_repository: '' -doi: '' -contact: '' -contact_information: '' -source_data_format: '' -obs_type: '' +sensor_name: "sensor-name" +sensor_long_name: "OTT Hydromet Parsivel 2" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +contributors: "" +authors: "" +institution: "" +references: "" +documentation: "" +website: "" +source_repository: "" +doi: "" +contact: "" +contact_information: "" +source_data_format: "" +obs_type: "" diff --git a/disdrodb/tests/data/test_dir_structure/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml b/disdrodb/tests/data/test_dir_structure/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml index 5193b4e8..838e17f9 100644 --- a/disdrodb/tests/data/test_dir_structure/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml +++ b/disdrodb/tests/data/test_dir_structure/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml @@ -1,56 +1,56 @@ -data_source: 'DATA_SOURCE' -campaign_name: 'CAMPAIGN_NAME' -station_name: 'STATION_NAME' -sensor_name: 'OTT_Parsivel' -reader: 'EPFL/EPFL_2009' +data_source: "DATA_SOURCE" +campaign_name: "CAMPAIGN_NAME" +station_name: "STATION_NAME" +sensor_name: "OTT_Parsivel" +reader: "EPFL/EPFL_2009" raw_data_format: txt platform_type: fixed -disdrodb_data_url: '' -source: '' -source_convention: '' -source_processing_date: '' -title: '' -description: '' -summary: '' -history: '' -keywords: '' -project_name: '' -comment: '' -station_id: '' -location: '' -country: '' -continent: '' +disdrodb_data_url: "" +source: "" +source_convention: "" +source_processing_date: "" +title: "" +description: "" +summary: "" +history: "" +keywords: "" +project_name: "" +comment: "" +station_id: "" +location: "" +country: "" +continent: "" latitude: -9999 longitude: -9999 altitude: -9999 -deployment_status: '' -deployment mode: '' -platform_protection: '' -platform_orientation: '' -sensor_long_name: '' -sensor_manufacturer: '' -sensor_wavelength: '' -sensor_serial_number: '' -firmware_iop: '' -firmware_dsp: '' -firmware_version: '' -sensor_beam_length: '' -sensor_beam_width: '' -sensor_nominal_width: '' -measurement_interval: '' -calibration_sensitivity: '' -calibration_certification_date: '' -calibration_certification_url: '' -contributors: '' -authors: '' -authors_url: '' -contact: '' -contact_information: '' -acknowledgement: '' -references: '' -documentation: '' -website: '' -institution: '' -source_repository: '' -license: '' -doi: '' +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" +sensor_long_name: "" +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" +contact_information: "" +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: "" +source_repository: "" +license: "" +doi: "" diff --git a/disdrodb/tests/data/test_yaml/invalid.yaml b/disdrodb/tests/data/test_yaml/invalid.yaml deleted file mode 100644 index 31f7e619..00000000 --- a/disdrodb/tests/data/test_yaml/invalid.yaml +++ /dev/null @@ -1,2 +0,0 @@ -key1: value1 -key2: value2: diff --git a/disdrodb/tests/data/test_yaml/valid.yaml b/disdrodb/tests/data/test_yaml/valid.yaml index d694d072..e4d91c6b 100644 --- a/disdrodb/tests/data/test_yaml/valid.yaml +++ b/disdrodb/tests/data/test_yaml/valid.yaml @@ -1,9 +1,9 @@ -key1: value1 -key2: 2 -key3: 3.0 -key4: - - value4 -key5: - - 5 -key6: null -key7: "" +key1: value1 +key2: 2 +key3: 3.0 +key4: + - value4 +key5: + - 5 +key6: null +key7: "" diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATIONID.yml b/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATIONID.yml new file mode 100644 index 00000000..19bd7cf0 --- /dev/null +++ b/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATIONID.yml @@ -0,0 +1,43 @@ +title: "CABAUW" +description: "" +source: "" +history: "" +conventions: "" +campaign_name: "DELFT" +project_name: "" +station_id: "STATIONID" +station_name: "Cabauw" +location: "" +country: "Netherlands" +continent: "Europe" +latitude: 51.96832449 +longitude: 4.92921567 +altitude: 1 +crs: WGS84 +proj4_string: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs +EPSG: 4326 +latitude_unit: DegreesNorth +longitude_unit: DegreesEast +altitude_unit: MetersAboveSeaLevel +sensor_name: "OTT_Parsivel2" +sensor_long_name: "OTT Hydromet Parsivel 2" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +contributors: "" +authors: "" +institution: "" +references: "" +documentation: "" +website: "" +source_repository: "" +doi: "" +contact: "" +contact_information: "" +source_data_format: "" +obs_type: "" diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml b/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml new file mode 100644 index 00000000..5dfd8800 --- /dev/null +++ b/disdrodb/tests/pytest_files/test_folders_files_creation/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME/metadata/STATION_NAME.yml @@ -0,0 +1,55 @@ +data_source: "DATA_SOURCE" +campaign_name: "CAMPAIGN_NAME" +station_name: "STATION_NAME" +sensor_name: "OTT_Parsivel" +reader: "EPFL/EPFL_2009" +raw_data_format: txt +platform_type: fixed +source: "" +source_convention: "" +source_processing_date: "" +title: "" +description: "" +summary: "" +history: "" +keywords: "" +project_name: "" +comment: "" +station_id: "" +location: "" +country: "" +continent: "" +latitude: -9999 +longitude: -9999 +altitude: -9999 +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" +sensor_long_name: "" +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" +contact_information: "" +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: "" +source_repository: "" +license: "" +doi: "" diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample.parquet b/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample.parquet new file mode 100644 index 0000000000000000000000000000000000000000..4a667812a727ea045da6889be6df60cd0c7f6763 GIT binary patch literal 3009 zcmcgu-EQJW6dqWT%|=lbMKJ=23uHCcN=;IJ(`BVzj9G9@NwOgu8`!QQ+c;p>HWb?s z5~Zzr(W||xsy;-O2k2${3VDKF_rCW%GyEs8QnTG!FrGPo-#6#X9Qh?K#hC!J%6w8} z{0!PaXldPt5K1tJ*<ga(pj&5_ndQhHdg@Drd<oD!iTYs-_<RAxeH&rc!s&Ii91i(Z z#vcp>?O>Nna4QpsFbRSfKZ0Q$UI@(tOwI#LhJ498fPea@5VbMl&wlPNAH}3Tb1~PM z&Cw=TBCS7u{qE-PB|>UupP`kJW-<$+r*outi2bw_VFJTc1YE<?DYxHz!QXw{4*~A> zQDhQuGFuROF8@s?R3xc{BCUQ(iWGlerQ*~49{IW3$5V+@D7lN^_df2&$K36>ym2%S zIsA1Rg_a$yp<4>Gf|#`}KU8}Q6(fK#SZSkG0CaTwTbspq8<B0W7}+R8n_wUu2^SxB zbhwAspZm!5d3b(<?*hR9w-USwY$8a1Xt}M}HN^?tEM2jNUDGelw3g{vw!=P;zGfBn z&32R>NY1EQr;01qRxBM%Y|k{o;@r^9+S#mZ5ctgPw}~O^*@o7t=`2{=p4IM;6)m-? zYc7#nx~tR_R{`@YR=2_2u<OM4^i_W68GC&_Av4Q@Rf+oyiGk>YIFP+vZv&O#zF6%( z3KjjgDC9QZ2O3H5U_rX{k!m)rBh6mwwe7sQliqt<Q|=>D(H*#dS44_JYu3>fx7DF_ zH4U|+bo!*M?Y`3Kw0Z=(t9KluWfN&Uwj0}xvygdqh(4{{P46bzYZO|;rP7i7e9$IB zH>;nMCsTicF+PSrdA`Cph50(pV_e7CI+gMKjJ`*Nxv~D}I6GLwNTX1_h;a{tAG~Q0 zcvB}BEAZ!g6*>L3Qa%909{BTLh?eP91u0cd9Cp=x$yMzG<H+Du(aIU+lU`l4hH^90 zR;`k$3(}?7Y}MtYW2jc%t)7T`qL7ZOLeCMi1xt}7r<%`r6*+DaJJpsPH4AZUz5)IX zEuM+#WuEFpTk;h7uv-;QjbauXVC!o8=6P-3?9sT{LbqbI%}VlkpqHgsMJ_dJLXL0* z?vhtYWZo)r+9p_{DV>&#SW2|xnHlcGJ?Z_zn3rgYG-lN@O;sqJkN1)-7^3k}_<A+@ z!a0YW&hj`|#H0DY@~}&7RUXt~PEKM9spq1+KwWEsG_kSZJe2u#Iax|;P}?H|8&x46 ztClH7;xO-eT0AzdrVDbETa=^5LQWrcwgx<;7&+5T&5~e$nLasVP&-GF3;jyDkfvwO z=F>3mt4WP=tXWcDOPF2NfP0Y@FAlP)Ua^@oNZqIIK%!bMTq<(LqcNH}@9-pbDcUh| zUCI5$xIpeNbOd;cWT2NkE0Wd$3!X1g;VBuZJzWl_719s(n_&5wGqTS%sDVbtDSPNA z7}o^Tt%47rLBFUBQ9;prp^g0Ci$_||!6leJJI4GmhQ=abtYOmN2p9z+^W#(6sX$2< zbR18GQLBOw6U~<>6m2V=Z#>=I(c0}D+JvLSCL0~J-!S>w8voEQPY`+uAMgJFTRkSP literal 0 HcmV?d00001 diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_0.parquet b/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_0.parquet new file mode 100644 index 0000000000000000000000000000000000000000..411665c27920c93d762347106fa39604bec57e0e GIT binary patch literal 3157 zcmcgv&1)M+6d%d5u#KU{WQ7ERngxqMZC9VJW$0m6w&Yc1JC^HRNolb6OVTR4D^=23 zmW^o-ITRWw1X^f&?xA2vPodC~UP|b{&|8l^_K+T1`eyWH*-}ECri(N?^WK~H`^|fA z9?~704KXfeh6!gFCxhk@dh^pMM4t&oJvs_H(eh`XM<zkSAZDIfUjpkn#?83>tLU~P z>~(}eb!m@$baW)Cs)W$ftohqCC<D(NerDDen?r7&*C8`5kIQ3uo;m_TU~1sRM;s%@ zNnqyS1$Zx_<wigJ^k`!ot%%nVc`2>u6PIWy16mIz0#BTj7<Ds}*c`KVJzifwDQbUx z_4r~I{r$$Hk4Y4-hRi$l#l^QDUHmzT1~U)3Jwp0Y_J6qjjH??|S5r4NdVud8o==>C zA14F9yZi%)!E6lJ$TQv|XM~(wuL;#PpG39Czpl=VIAgagLL*O5bfSZASWL^}`Shju zOO*bJ=VwRY!F1reeqn^rC~m$FV#w=mC}qu*m?^}}E;^yI7E#7e2+@_&Te(0-1GIQ6 z^Kjn31QtCTscYkL`TV}jP3tsp<3b9&<|GDg9AeKyxTS$9&oAy<2r}Sx*Cb1o8s3Y^ zGgh}$?c`9Y8n$UQ*!6|Gti;}1T3|bpL#o!O;*7N<Qv(yrHVm*R>YAY*4$C@@51Z{8 zF=TB^SE{PUg0*FvwRT@ol`ERkByv-0N~+Y9!2FEWEHKxt64`t3esXz@JwG3i8D_!C zWcv__j_8#*kh7gH0hR8#Sminj75%p;<jP+M>PfF)LAbP@YB;Pt&1&?n)wp*hyw{E< zT}Py%D{%iVi4=wAtgcDTYMtiQ(B-;RZ<Dmv+ETq<Z4u~Wt=`b97LkU6E5Vfz3z28L z=$Dl%v6TpWj=Zxto81-HJDtLNG4nQg;-w9Y@g981^B%@g%$0Bw;}TAksEp%=^erOX z?CW>;5}h@S)C<+~7`HI&gEw^oZ@LM_JbbxUUX0z(=e7Z{1^c;oc+;>8DIuB*?;Oip zLQ}T3^<ABld2>_G?YBz2*_9jdnrvnbEhU`tm1;?hG<4ZaHVgawDxZpl<W#G{C(@=Q z3XMWCZs*02LF{BpXvhihV{v8JUsgi#pqAsPj<*C`5_gUZsRKQez&hAAl`W&FZW%4= zH<3Qhn>8aJ+3V;<A($7lWi_=)I0ARU&WGdoB{60ZEZz_fvU)JeTcP+6_wJdrcDm1t zH+kx_U>b&;$`<=)Nu+gNe=U4%RXk}FA*RDTw#V_P{I5K$Y)uwBc^DCsphWUHE-sK) zH6;veOsIz(7t2MmF$HqFt7E;8N(Ku#iV@yPHf<#o9F@~CG1?p#qw-izuU1<f9zu}R zbVV@*IA6R?Y7BDMki>L5pG(JR%~@Ow=50BmP>!l8w3V=NEbDMD68y<_BHGGSHg%Hs zOLrh#$fZvuF>X^Il}&qRKYGerL2_Nm{lz#<?k{u%crs+5mps9f)&UES%TnP18L2&8 zcBXmK54I{``BtN6pD0rUg^ZW%p`T!^ri^9*_5d36i+mRq6s--VxBrEBr1fl{g6Y~G z=Jqf&mK4ScCJl~&Q4liUJD{Bklw?82{xsEVRp4Tvc^7#XYEu0}TWc;Wwc0Xm!VBFd WyU=OBKJtUB|3P2E@47qihvFY+zEv9l literal 0 HcmV?d00001 diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_1.parquet b/disdrodb/tests/pytest_files/test_folders_files_creation/fake_data_sample_1.parquet new file mode 100644 index 0000000000000000000000000000000000000000..c3bd20ebbe19c3d8e4deaa2f43b8694e1290ecfb GIT binary patch literal 3157 zcmcgv-Af}^6u*fvsQ4h_gaiVW0mDMo=EGPc^x;lS;-pbGvCbsfE;?Vvi8GTl$rz(Z zUwkNZp-^a{AFnMGgzeK(+OmBqrT;?T`q;-lY#$3f_vXuJvP<1=hvd$^=bU?fzkANP zhvYuag%}qz#e_4AlR>ixz4qY*qR*J39vlRnXz8O*BI6)o5HriHFM{<9<7V9cRdml0 z_Bz6#y0%9?I5?10RYK@#&is86l!0FyerDPiok4D&*C8`5kIQ3uo;m_TV50BDM;s%@ zNnmE+1$fV+<wigJ^zp_hS`n`!@<LkA#;(y)`m`>`0#BTj7<Dt0*bK9FJ6>NrIoAIE z{PEQ^n)>?7OA^JKA@fdr`u#WWU;Q<X`ZEu@H$?hE_J6wljH?q=S3@^7dVud7o)4UX zAIAfKxcq&H{%i!;&@;}FGeS;o*M#btO`_`K-&UuFoUv0Dp`j-@bfUelSWL^}+2pnO zOO*bp=VwRYax(C3uP{Pr7&qU$QRH>kl_SlRm<hy8FF2vH7Es1d2+@_&UAaI<1GF%g zc{J-^1dFbX)Uolne12c%u5}u?b0Gy@aS{V}4zcGE+|s~==U4X}f(&@wRmoDNy7y}Q zoYgH=JKa|*hHYAPc76T<E3pq3=h?PopQ<&gIA<-%)WF2D4FfC=b<I%s2W4%?`;AtW z7_zpdD-~5^!P>ITYOANH$Yo7w5V@%}BvooiV1CYO7MSZ+iR|5bC%Lr7UR?Ca46<Nl zvVDL=Tl7*K$k{FyflB9Gta2NLivC*^a^-IWb)`43AY58kH5k^eW_9I(ReyLRytj@f z-A1IM8*u+Fi4=wAtfongN{!~#(B+y`Ymu~8TT-o7X%grYtyb477LkU6%faOk3z26# z=+~9Y(d7txfxOc=m)jB7+nvIDHT4F0Vx<j?aTdPhc^l&u%$0Bw;}VXSsEp$V^i3j6 z^z=K~czX>a^+NSL#!U?S;7y&tn{I-!0AH?I5TnP1{1zZKVL$gKZyI(nC9LGb+b8m- z(2%VyeMjeH-c0EE-DZh5J8~mdmCc-?rGzuSTq%i>x-Of^Msb&4<x|m+oNCtjc-oXi zp<YbJ?1C6Fh@ET+bvX`xEba*QA1R?&P|I^v$6JCeiQ6Z|)SjM+V;yW8%BFFsZW>MM zH=aHzm{p??$+q>P5G;tfBQ=#E9D%!F7s9b)NsL+qi#LS5oE}`^tx#-$d*@7AE8XM8 zn>_VdGz~*e<qmsiiKlg5e<gfvRXnX9LQDsFY>nbk{$F`mxvDI-^DrbPL5bvZR9qmh zYD(zam{1RSE}D<zq6*}8N5^_Gl?)d16eGNyY}iUDI4q|lVw4yaqoa|WUaq#<JcJ;r z>9S%9aK2cJ)EMNhE{W+@A)k)YnzOhl%v*9qp&V6HXenXiMAqS6#QD>$_)0TVPUs}> z*X}^Lm`|TcV$7yK$_aaWcjb(?g5<iA`-^d!++XMj@MOq9FL|6NtpgSum!rZxGE#fG zY)=cMA8eMv^7VSxK7K?E6f$13hkk;wnlc(i*aK+LFA5!0P&7A`?*8ZEk=C<y2BvFS z%w;h&mK4ScCJl~&Q4liE?$J&KO0uA1Z<^}1Dsa))yoJ2;RjKx#tu>aEYITV=;rULJ Woo}~aANj%6`=HO@cinyXL-7y6{#AJZ literal 0 HcmV?d00001 diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/issue/123.yml b/disdrodb/tests/pytest_files/test_folders_files_creation/issue/123.yml new file mode 100644 index 00000000..b4f46b72 --- /dev/null +++ b/disdrodb/tests/pytest_files/test_folders_files_creation/issue/123.yml @@ -0,0 +1 @@ +key: value diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/metadata/123.yml b/disdrodb/tests/pytest_files/test_folders_files_creation/metadata/123.yml new file mode 100644 index 00000000..a5e873db --- /dev/null +++ b/disdrodb/tests/pytest_files/test_folders_files_creation/metadata/123.yml @@ -0,0 +1,55 @@ +data_source: "" +campaign_name: "" +station_name: "" +sensor_name: "" +reader: "" +raw_data_format: txt +platform_type: fixed +source: "" +source_convention: "" +source_processing_date: "" +title: "" +description: "" +project_name: "" +keywords: "" +summary: "" +history: "" +comment: "" +station_id: "" +location: "" +country: "" +continent: "" +latitude: -9999 +longitude: -9999 +altitude: -9999 +deployment_status: "" +deployment mode: "" +platform_protection: "" +platform_orientation: "" +sensor_long_name: "" +sensor_manufacturer: "" +sensor_wavelength: "" +sensor_serial_number: "" +firmware_iop: "" +firmware_dsp: "" +firmware_version: "" +sensor_beam_length: "" +sensor_beam_width: "" +sensor_nominal_width: "" +measurement_interval: "" +calibration_sensitivity: "" +calibration_certification_date: "" +calibration_certification_url: "" +contributors: "" +authors: "" +authors_url: "" +contact: "" +contact_information: "" +acknowledgement: "" +references: "" +documentation: "" +website: "" +institution: "" +source_repository: "" +license: "" +doi: "" diff --git a/disdrodb/tests/pytest_files/test_folders_files_creation/mock_file_path.yml b/disdrodb/tests/pytest_files/test_folders_files_creation/mock_file_path.yml new file mode 100644 index 00000000..94b5a337 --- /dev/null +++ b/disdrodb/tests/pytest_files/test_folders_files_creation/mock_file_path.yml @@ -0,0 +1,21 @@ +# This file is used to store timesteps/time periods with wrong/corrupted observation. +# The specified timesteps are dropped during the L0 processing. +# The time format used is the isoformat : YYYY-mm-dd HH:MM:SS. +# The 'timesteps' key enable to specify the list of timesteps to be discarded. +# The 'time_period' key enable to specify the time periods to be dropped. +# Example: +# +# timesteps: +# - 2018-12-07 14:15:00 +# - 2018-12-07 14:17:00 +# - 2018-12-07 14:19:00 +# - 2018-12-07 14:25:00 +# time_period: +# - ['2018-08-01 12:00:00', '2018-08-01 14:00:00'] +# - ['2018-08-01 15:44:30', '2018-08-01 15:59:31'] +# - ['2018-08-02 12:44:30', '2018-08-02 12:59:31'] + +timesteps: + - "2018-12-07 14:15:00" +time_periods: + - ["2018-08-01 12:00:00", "2018-08-01 14:00:00"] diff --git a/disdrodb/tests/test_utils/test_utils_yaml.py b/disdrodb/tests/test_utils/test_utils_yaml.py index 413a210c..d4d079f5 100644 --- a/disdrodb/tests/test_utils/test_utils_yaml.py +++ b/disdrodb/tests/test_utils/test_utils_yaml.py @@ -21,7 +21,6 @@ import os import pytest -import yaml from disdrodb import __root_path__ from disdrodb.utils.yaml import read_yaml @@ -44,11 +43,6 @@ def test_read_yaml(): filepath = os.path.join(TEST_DATA_DIR, "test_yaml", "valid.yaml") assert read_yaml(filepath) == dictionary - # Test reading a YAML file with invalid syntax - invalid_yaml_filepath = os.path.join(TEST_DATA_DIR, "test_yaml", "invalid.yaml") - with pytest.raises(yaml.YAMLError): - read_yaml(invalid_yaml_filepath) - # Test reading a non-existent YAML file non_existent_filepath = os.path.join(TEST_DATA_DIR, "non_existent.yaml") with pytest.raises(FileNotFoundError): diff --git a/disdrodb/utils/compression.py b/disdrodb/utils/compression.py index 32d3069c..09cf1407 100644 --- a/disdrodb/utils/compression.py +++ b/disdrodb/utils/compression.py @@ -46,6 +46,7 @@ def unzip_file(filepath: str, dest_path: str) -> None: Path of the file to unzip. dest_path : str Path of the destination directory. + """ with zipfile.ZipFile(filepath, "r") as zip_ref: zip_ref.extractall(dest_path) diff --git a/disdrodb/utils/logger.py b/disdrodb/utils/logger.py index 0ed99fb8..2fae3d30 100644 --- a/disdrodb/utils/logger.py +++ b/disdrodb/utils/logger.py @@ -52,7 +52,7 @@ def close_logger(logger: logger) -> None: Parameters ---------- - logger : logger + logger : logging.Logger Logger object. """ handlers = logger.handlers[:] @@ -70,7 +70,7 @@ def log_debug(logger: logger, msg: str, verbose: bool = False) -> None: Parameters ---------- - logger : logger + logger : logging.Logger Log object. msg : str Message. @@ -88,7 +88,7 @@ def log_info(logger: logger, msg: str, verbose: bool = False) -> None: Parameters ---------- - logger : logger + logger : logging.Logger Log object. msg : str Message. @@ -106,7 +106,7 @@ def log_warning(logger: logger, msg: str, verbose: bool = False) -> None: Parameters ---------- - logger : logger + logger : logging.Logger Log object. msg : str Message. @@ -124,7 +124,7 @@ def log_error(logger: logger, msg: str, verbose: bool = False) -> None: Parameters ---------- - logger : logger + logger : logging.Logger Log object. msg : str Message. @@ -190,11 +190,11 @@ def define_summary_log(list_logs): The summary log select only logged lines with ``root``, ``WARNING`` and ``ERROR`` keywords. The problems log file select only logged lines with the ``ERROR`` keyword. - The two log files are saved in the parent directory of the input list_logs. + The two log files are saved in the parent directory of the input ``list_logs``. The function assume that the files logs are located at: - ``/DISDRODB/Processed/<DATA_SOURCE>/<CAMPAIGN_NAME>/logs/<product>/<station_name>/*.log`` + ``/DISDRODB/Processed/<DATA_SOURCE>/<CAMPAIGN_NAME>/logs/<product>/<station_name>/<filename>.log`` """ # LogCaptureHandler of pytest does not have baseFilename attribute, so it returns None diff --git a/disdrodb/utils/pandas.py b/disdrodb/utils/pandas.py index 2d6dbc1c..322d68c5 100644 --- a/disdrodb/utils/pandas.py +++ b/disdrodb/utils/pandas.py @@ -26,7 +26,7 @@ def get_dataframe_start_end_time(df: pd.DataFrame): Parameters ---------- - df : pd.DataFrame + df : pandas.DataFrame Input dataframe Returns diff --git a/disdrodb/utils/xarray.py b/disdrodb/utils/xarray.py index 9e5fd3ef..ebc3fed7 100644 --- a/disdrodb/utils/xarray.py +++ b/disdrodb/utils/xarray.py @@ -28,7 +28,7 @@ def get_dataset_start_end_time(ds: xr.Dataset): Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset Input dataset Returns @@ -48,7 +48,7 @@ def regularize_dataset(ds: xr.Dataset, freq: str, time_dim="time", method=None, Parameters ---------- - ds : xr.Dataset + ds : xarray.Dataset xarray Dataset. time_dim : str, optional The time dimension in the xr.Dataset. The default is ``"time"``. @@ -64,7 +64,7 @@ def regularize_dataset(ds: xr.Dataset, freq: str, time_dim="time", method=None, Returns ------- - ds_reindexed : xr.Dataset + ds_reindexed : xarray.Dataset Regularized dataset. """ diff --git a/docs/README.html b/docs/README.html index bc44164e..aa726b51 100644 --- a/docs/README.html +++ b/docs/README.html @@ -1,441 +1,15768 @@ -<!DOCTYPE html> +<!doctype html> <html> + <head> + <meta charset="utf-8" /> + <meta name="generator" content="pandoc" /> + <meta http-equiv="X-UA-Compatible" content="IE=EDGE" /> -<head> - -<meta charset="utf-8" /> -<meta name="generator" content="pandoc" /> -<meta http-equiv="X-UA-Compatible" content="IE=EDGE" /> - - - - -<title>README - - - - - - - - - - + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - -
-

DISDRODB documentation

-

DISDRODB’s documentation is built using the powerful Sphinx framework, -styled with Book -Theme.

-

All documentation source files are neatly organized in the -docs/ directory within the project’s repository.

-
-
-

Build the documentation

-

To build the documentation locally, follow the next three steps.

-

1. Set up the python environment for building the -documentation

-

The python packages required to build the documentation are listed in -the requirements.txt -file.

-

For an efficient setup, we recommend creating a dedicated virtual -environment. Navigate to the docs/ directory and execute -the following command. This will also install pandoc, a versatile document conversion -tool:

-
conda create -f environment.yaml
-

2. Activate the virtual environment

-

Once the environment is ready, activate it using:

-
conda activate build-doc-disdrodb
-

3. Generate the documentation

-

With the environment set and activated, you’re ready to generate the -documentation. Execute:

-
make clean html
-

This command will build the HTML version of the documentation. It -first cleans previous builds (make clean) and then -generates fresh documentation (html).

-

Note: It’s important to review the output of the -command. Look out for warnings or errors and address them to ensure the -documentation is accurate and complete.

-

By following these steps, you should have a local version of the -DISDRODB documentation in the docs/build/html/ directory, -ready for review or deployment!

-
-
- - - - -
- - + + + + + + + + + + + + + - + +
+ - +
+

DISDRODB documentation

+

+ DISDRODB’s documentation is built using the powerful + Sphinx framework, + styled with + Book Theme. +

+

+ All documentation source files are neatly organized in the + docs/ directory within the project’s repository. +

+
+
+

Build the documentation

+

+ To build the documentation locally, follow the next three steps. +

+

+ 1. Set up the python environment for building the + documentation +

+

+ The python packages required to build the documentation are listed + in the + requirements.txt + file. +

+

+ For an efficient setup, we recommend creating a dedicated virtual + environment. Navigate to the docs/ directory and + execute the following command. This will also install + pandoc, a versatile document + conversion tool: +

+ 
conda create -f environment.yaml
+

2. Activate the virtual environment

+

Once the environment is ready, activate it using:

+ 
conda activate build-doc-disdrodb
+

3. Generate the documentation

+

+ With the environment set and activated, you’re ready to generate the + documentation. Execute: +

+ 
make clean html
+

+ This command will build the HTML version of the documentation. It + first cleans previous builds (make clean) and then + generates fresh documentation (html). +

+

+ Note: It’s important to review the output of the + command. Look out for warnings or errors and address them to ensure + the documentation is accurate and complete. +

+

+ By following these steps, you should have a local version of the + DISDRODB documentation in the + docs/build/html/ directory, ready for review or + deployment! +

+
+
+
- -$(document).ready(function () { - $('.tabset-dropdown > .nav-tabs > li').click(function () { - $(this).parent().toggleClass('nav-tabs-open'); - }); -}); - + - + - - + - + + + diff --git a/docs/README.md b/docs/README.md index cb56777f..6b843a0c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,7 @@ styled with [Book Theme](https://sphinx-book-theme.readthedocs.io/en/stable/inde All documentation source files are neatly organized in the `docs/` directory within the project's repository. ------------------------------------------------------------------ +______________________________________________________________________ ### Build the documentation @@ -13,23 +13,23 @@ To build the documentation locally, follow the next three steps. **1. Set up the python environment for building the documentation** - The python packages required to build the documentation are listed in the [environment.yaml](https://github.com/ltelab/disdrodb/blob/main/docs/environment.yaml) file. +The python packages required to build the documentation are listed in the [environment.yaml](https://github.com/ltelab/disdrodb/blob/main/docs/environment.yaml) file. - For an efficient setup, we recommend creating a dedicated virtual environment. - Navigate to the `docs/` directory and execute the following command. - This will create a new environment and install the required packages. +For an efficient setup, we recommend creating a dedicated virtual environment. +Navigate to the `docs/` directory and execute the following command. +This will create a new environment and install the required packages. - ``` - conda create -f environment.yaml - ``` +``` +conda create -f environment.yaml +``` **2. Activate the virtual environment** - Once the environment is ready, activate it using: +Once the environment is ready, activate it using: - ``` - conda activate build-doc-disdrodb - ``` +``` +conda activate build-doc-disdrodb +``` **3. Generate the documentation** @@ -45,5 +45,5 @@ It first cleans previous builds (`make clean`) and then generates fresh document **Note**: It's important to review the output of the command. Look out for warnings or errors and address them to ensure the documentation is accurate and complete. -By following these steps, you should have a local version of the DISDRODB documentation in the ``docs/build/html/`` directory, +By following these steps, you should have a local version of the DISDRODB documentation in the `docs/build/html/` directory, ready for review or deployment! diff --git a/docs/environment.yaml b/docs/environment.yaml index 45c4754f..d7221fcc 100644 --- a/docs/environment.yaml +++ b/docs/environment.yaml @@ -9,7 +9,8 @@ dependencies: - pre-commit - pandoc - nbsphinx==0.9.3 - - sphinx==5.1.1 + - sphinx==7.2.6 - sphinx-book-theme==1.1.0 - sphinx-mdinclude==0.5.3 - sphinx-gallery + - docutils diff --git a/docs/source/conf.py b/docs/source/conf.py index 14255edb..0773bca3 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -13,6 +13,8 @@ import os import shutil import sys +import inspect +import disdrodb # sys.path.insert(0, os.path.abspath("..")) sys.path.insert(0, os.path.abspath("../..")) @@ -28,11 +30,11 @@ # Copy tutorial notebook root_path = os.path.dirname(os.path.dirname(os.getcwd())) - -in_path = os.path.join(root_path, "tutorials", "reader_preparation.ipynb") -out_path = os.path.join(os.getcwd(), "reader_preparation.ipynb") - -shutil.copyfile(in_path, out_path) +filenames = ["reader_preparation.ipynb"] +for filename in filenames: + in_path = os.path.join(root_path, "tutorials", filename) + out_path = os.path.join(os.getcwd(), "tutorials", filename) + shutil.copyfile(in_path, out_path) # -- General configuration --------------------------------------------------- @@ -41,14 +43,65 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - "sphinx.ext.napoleon", - "sphinx.ext.autodoc", + "sphinx.ext.coverage", "sphinx.ext.viewcode", + "sphinx.ext.intersphinx", + "sphinx.ext.coverage", + "sphinx.ext.linkcode", + # "sphinx_design", + # "sphinx_gallery.gen_gallery", # "sphinx.ext.autosectionlabel", - "nbsphinx", "sphinx_mdinclude", + "sphinx.ext.napoleon", + "sphinx.ext.autodoc", + "sphinx.ext.autosummary", + # "myst_parser", + "nbsphinx", +] + +# Set up mapping for other projects' docs +intersphinx_mapping = { + "matplotlib": ("https://matplotlib.org/stable/", None), + "numpy": ("https://numpy.org/doc/stable/", None), + "pandas": ("https://pandas.pydata.org/docs/", None), + "python": ("https://docs.python.org/3/", None), + "xarray": ("https://docs.xarray.dev/en/stable/", None), + "dask": ("https://docs.dask.org/en/stable/", None), + "pydantic": ("https://docs.pydantic.dev/latest/", None), +} +always_document_param_types = True + +# Warn when a reference is not found in docstrings +nitpicky = True +nitpick_ignore = [ + ("py:class", "optional"), + ("py:class", "array-like"), + ("py:class", "file-like object"), + # For traitlets docstrings + ("py:class", "All"), + ("py:class", "t.Any"), + ("py:class", "t.Iterable"), + ("py:class", "sorted"), + ("py:class", "dictionary"), +] +nitpick_ignore_regex = [ + ("py:class", r".*[cC]allable"), ] +# The suffix of source filenames. +source_suffix = [".rst", ".md"] + +# For a class, combine class and __init__ docstrings +autoclass_content = "both" + +# Napoleon settings +napoleon_google_docstring = False +napoleon_numpy_docstring = True +napoleon_include_init_with_doc = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = "sphinx" + # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] @@ -77,6 +130,7 @@ "use_download_button": True, # "use_sidenotes": True, "show_toc_level": 2, + "navigation_with_keys": False, } # Add any paths that contain custom static files (such as style sheets) here, @@ -95,8 +149,65 @@ def run_apidoc(_): module_dir = os.path.join(cur_dir, "..", "..", "disdrodb") output_dir = os.path.join(cur_dir, "api") - main(["-f", "-o", output_dir, module_dir]) + exclude = [os.path.join(module_dir, "tests")] + main(["-f", "-o", output_dir, module_dir, *exclude]) def setup(app): app.connect("builder-inited", run_apidoc) + + +# Function to resolve source code links for `linkcode` +# adapted from NumPy, Pandas implementations +def linkcode_resolve(domain, info): + """ + Determine the URL corresponding to Python object + """ + if domain != "py": + return None + + modname = info["module"] + fullname = info["fullname"] + + submod = sys.modules.get(modname) + if submod is None: + return None + + obj = submod + for part in fullname.split("."): + try: + obj = getattr(obj, part) + except AttributeError: + return None + + try: + fn = inspect.getsourcefile(inspect.unwrap(obj)) + except TypeError: + try: # property + fn = inspect.getsourcefile(inspect.unwrap(obj.fget)) + except (AttributeError, TypeError): + fn = None + if not fn: + return None + + try: + source, lineno = inspect.getsourcelines(obj) + except TypeError: + try: # property + source, lineno = inspect.getsourcelines(obj.fget) + except (AttributeError, TypeError): + lineno = None + except OSError: + lineno = None + + if lineno: + linespec = f"#L{lineno}-L{lineno + len(source) - 1}" + else: + linespec = "" + + fn = os.path.relpath(fn, start=os.path.dirname(disdrodb.__file__)) + + if "+" in disdrodb.__version__: + return f"https://github.com/ltelab/disdrodb/blob/main/disdrodb/{fn}{linespec}" + else: + return f"https://github.com/ltelab/disdrodb/blob/" f"v{disdrodb.__version__}/disdrodb/{fn}{linespec}" diff --git a/docs/source/contribute_data.rst b/docs/source/contribute_data.rst index 4cee6237..4172352a 100644 --- a/docs/source/contribute_data.rst +++ b/docs/source/contribute_data.rst @@ -32,7 +32,7 @@ The name you adopt for the ```` and ```` will be use * Avoid the usage of dash ( - ) and dots ( . ) to separate words. Use the underscore ( _ ) instead! - * For short-term campaigns, we suggest adding the year of the campaign at the end (i.e. `EPFL_2009`) + * For short-term campaigns, we suggest adding the year of the campaign at the end (i.e. *EPFL_2009*) Here below we provide a detailed description of the steps to follow to contribute your data to DISDRODB: @@ -430,7 +430,7 @@ For example, to process all stations of the EPFL_2008 campaign, you would run: For more details and options related to DISDRODB L0 processing, read the section :ref:`Run DISDRODB L0 Processing `. -The DISDRODB L0 processing generates the DISDRODB `Processed` directories tree illustrated here below. +The DISDRODB L0 processing generates the DISDRODB *Processed* directories tree illustrated here below. | πŸ“ DISDRODB | β”œβ”€β”€ πŸ“ Processed @@ -490,7 +490,7 @@ Below, we offer a utility designed to compress each raw file associated to a spe method="gzip", ) -After compressing the raw files, remember to update the reader `glob_patterns` to include the new file extension (i.e. .gz) +After compressing the raw files, remember to update the reader *glob_patterns* to include the new file extension (i.e. .gz) and rerun the DISDRODB L0 processing to check that everything works fine. If you arrived at this point and you didn't open yet a Pull Request in the `GitHub disdrodb repository `__, do it now so diff --git a/docs/source/index.rst b/docs/source/index.rst index 65e00243..9980395d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,7 +1,3 @@ -.. d documentation master file, created by - sphinx-quickstart on Wed Jul 13 14:44:07 2022. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. Welcome to DISDRODB ! ====================== diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 4c1437ff..5ce1d6de 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -34,13 +34,13 @@ or `conda `__ (recommended). or `anaconda `__ if you don't have it already installed. -* Create the `disdrodb-py311` (or any other custom name) conda environment: +* Create the *disdrodb-py311* (or any other custom name) conda environment: .. code-block:: bash conda create --name disdrodb-py311 python=3.11 --no-default-packages -* Activate the `disdrodb-py311` conda environment: +* Activate the *disdrodb-py311* conda environment: .. code-block:: bash @@ -174,10 +174,10 @@ Run DISDRODB on Jupyter Notebooks If you want to run disdrodb on a `Jupyter Notebook `__, you have to take care to set up the IPython kernel environment where disdrodb is installed. -For example, if your conda/virtual environment is named `disdrodb-dev`, run: +For example, if your conda/virtual environment is named ``disdrodb-dev``, run: .. code-block:: bash python -m ipykernel install --user --name=disdrodb-dev -When you will use the Jupyter Notebook, by clicking on `Kernel` and then `Change Kernel`, you will be able to select the `disdrodb-dev` kernel. +When you will use the Jupyter Notebook, by clicking on ``Kernel`` and then ``Change Kernel``, you will be able to select the ``disdrodb-dev`` kernel. diff --git a/docs/source/reader_preparation.ipynb b/docs/source/reader_preparation.ipynb deleted file mode 100644 index 880a97e3..00000000 --- a/docs/source/reader_preparation.ipynb +++ /dev/null @@ -1,2679 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# DISDRODB reader preparation tutorial" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This notebook aims to guide you through creating the reader for the raw files logged by a disdrometer device. \n", - "\n", - "In first place, this notebook will provide you with functions that will display and enable to investigate the content of your raw data files.\n", - "\n", - "Successively, you will define a series of parameters defining the reader behaviour. These pieces of code will be consolidated in the [reader_template.py](https://github.com/ltelab/disdrodb/blob/main/disdrodb/l0/readers/reader_template.py) file to generate a DISDRODB L0 reader.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this notebook, we uses a lightweight dataset for illustratory purposes. You may use it and readapt it for exploring your own dataset, when preparing a new reader. \n", - "\n", - "Following the documentation in [How to Contribute New Data to DISDRODB](https://disdrodb.readthedocs.io/en/latest/contribute_data.html), you should have already: \n", - "\n", - "* defined the metadata for the stations you aim to define the reader\n", - "* copied the raw data within the correct folder of the local DISDRODB archive\n", - "* copied the [reader_template.py](https://github.com/ltelab/disdrodb/blob/main/disdrodb/l0/readers/reader_template.py), place it in the correct `disdrodb.l0.` directory and renamed it as `.py`\n", - "\n", - " " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For this tutorial, we have prepared some sample data in the folder [data/DISDRODB](https://github.com/ltelab/disdrodb/tree/main/data/DISDRODB) of the [disdrodb](https://github.com/ltelab/disdrodb/) repository. In this tutorial, this `data/DISDRODB` directory will act as toy DISDRODB base directory.\n", - "\n", - "The data corresponds to some measurements taken at two stations (`station_name_1` and `station_name_2`) during two days of a field campaign led by the EPFL LTE laboratory.\n", - "\n", - "```\n", - "Β Β πŸ“ DISDRODB\n", - "Β Β β”œβ”€β”€ πŸ“ Raw\n", - "Β Β Β Β Β Β β”œβ”€β”€ πŸ“ DATA_SOURCE\n", - "Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ CAMPAIGN_NAME\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ data\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ station_name_1\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ file60_20180817.dat.gz\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ file60_20180818.dat.gz\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ station_name_2\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ file61_20180817.dat.gz\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ file61_20180818.dat.gz\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ info\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ issue\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ station_name_1.yml\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ station_name_2.yml\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“ metadata\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ station_name_1.yml\n", - "Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β β”œβ”€β”€ πŸ“œ station_name_2.yml\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 1: Read and analyse the data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The goal of Step 1 is to define the specifications to read the raw data into a dataframe and ensure that the dataframe columns match the DISDRODB standards.\n", - "At the end of this tutorial, you should be able to generate Apache Parquet files from your input raw data. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "--------------------------------------------------------------------\n", - "Here we load the modules and packages required. *Nothing must be changed here*. " - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/home/ghiggi/Python_Packages/disdrodb\n" - ] - } - ], - "source": [ - "# Define project base directory\n", - "import os\n", - "\n", - "root_path = os.path.dirname(os.getcwd()) # something like /home/ghiggi/Projects/disdrodb\n", - "print(root_path)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "import logging\n", - "import pandas as pd\n", - "\n", - "# Directory\n", - "from disdrodb.api.create_directories import create_l0_directory_structure\n", - "\n", - "# Tools to develop the reader\n", - "from disdrodb.l0.template_tools import (\n", - " check_column_names,\n", - " infer_column_names,\n", - " print_df_first_n_rows,\n", - " print_df_random_n_rows,\n", - " print_df_column_names,\n", - " print_valid_l0_column_names,\n", - " get_df_columns_unique_values_dict,\n", - " print_df_columns_unique_values,\n", - " print_df_summary_stats,\n", - ")\n", - "\n", - "# L0A processing\n", - "from disdrodb.l0.io import get_raw_filepaths\n", - "from disdrodb.l0.l0a_processing import (\n", - " read_raw_file,\n", - " read_raw_files,\n", - " cast_column_dtypes,\n", - " write_l0a,\n", - ")\n", - "\n", - "# L0B processing\n", - "from disdrodb.l0.l0b_processing import (\n", - " retrieve_l0b_arrays,\n", - " create_l0b_from_l0a,\n", - " set_encodings,\n", - ")\n", - "\n", - "# Metadata\n", - "from disdrodb.metadata import read_station_metadata\n", - "\n", - "# Standards\n", - "from disdrodb.api.path import define_campaign_dir\n", - "from disdrodb.api.info import infer_path_info_dict\n", - "from disdrodb.api.checks import check_sensor_name\n", - "from disdrodb.l0.check_standards import check_l0a_column_names\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**1. Define paths and running parameters**\n", - "\n", - "In the following section, define the raw and processed directory paths. *This may be changed if you are using another folder*.\n", - "\n", - "NB:\n", - "- In the real use case, the `DATA_SOURCE` and `CAMPAIGN_NAME` should be replaced by meaningul names ! \n", - "- The `raw_dir` and `processed_dir` must end with the same `CAMPAIGN_NAME` (in upper case format)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "raw_dir: /home/ghiggi/Python_Packages/disdrodb/data/DISDRODB/Raw/DATA_SOURCE/CAMPAIGN_NAME\n", - "processed_dir: /home/ghiggi/Python_Packages/disdrodb/data/DISDRODB/Processed/DATA_SOURCE/CAMPAIGN_NAME\n" - ] - } - ], - "source": [ - "base_dir = os.path.join(root_path, \"data\", \"DISDRODB\")\n", - "data_source = \"DATA_SOURCE\"\n", - "campaign_name = \"CAMPAIGN_NAME\"\n", - "raw_dir = define_campaign_dir(base_dir=base_dir, \n", - " product=\"RAW\",\n", - " data_source=data_source,\n", - " campaign_name=campaign_name, \n", - ")\n", - "processed_dir = define_campaign_dir(base_dir=base_dir, \n", - " product=\"L0A\",\n", - " data_source=data_source,\n", - " campaign_name=campaign_name, \n", - ")\n", - "assert os.path.exists(raw_dir), \"Raw directory does not exist\"\n", - "print(f\"raw_dir: {raw_dir}\")\n", - "print(f\"processed_dir: {processed_dir}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Then we define the reader execution parameters. When the new reader will be created, these parameters will be become the reader function arguments. Please have a look [at the documentation](https://disdrodb.readthedocs.io/en/latest/readers.html#runing-a-reader) to get a full description. " - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "force = True\n", - "parallel = False\n", - "verbose = True\n", - "debugging_mode = True\n", - "sensor_name = \"OTT_Parsivel\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**2. Selection of the station**\n", - "\n", - "In this example, we choose to implement and run the reader for station `station_name_1`. However, feel free to change the station name :)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "station_name = \"station_name_1\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**3. Initialization**\n", - "\n", - "We initiate some checks, and get some variable. *Nothing must be changed here.*" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "# Create directory structure\n", - "create_l0_directory_structure(\n", - " raw_dir=raw_dir,\n", - " processed_dir=processed_dir,\n", - " station_name=station_name,\n", - " force=force,\n", - " verbose=False,\n", - " product=\"L0A\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Please, be sure to run the cell above only one time. If it is run many times, the log file blocks the folder creation. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**4. Get the list of file to process**\n", - "\n", - "We now list all raw data files that are available for the selected station.\n", - "Here we need to specify the [glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)) that enables to select all the relevant data files. \n", - "Since the files in this case study are named like `file_