Use normalization to compute echo range scaling (#1463)

* use align vectors instead of from matrix for a channel-wise rotation

* use normalization logic, add warnings, and add tests

* simplify test

* small word change

* Update echopype/consolidate/ek_depth_utils.py

Co-authored-by: Wu-Jung Lee <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replace previous APA7 citation with recent ICES APA7 citation (#1455)

* pin scipy to temporarily ensure that rotation matrix calculation does not fail (#1460)

* [pre-commit.ci] pre-commit autoupdate (#1461)

updates:
- [github.com/PyCQA/flake8: 7.1.1 → 7.1.2](PyCQA/flake8@7.1.1...7.1.2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Check if there exist any swap files before cleaning them up [all tests ci] (#1451)

* check if zarr stores > 0

* add a test for echodata delete

* fix path

* add comments for readability

---------

Co-authored-by: Wu-Jung Lee <[email protected]>

* remove __setattr__ from EchoData (#1457)

* compute dask array before np array equal (#1452)

* Dataset.dims to Dataset.sizes [all tests ci] (#1453)

* dims to sizes

* dims to sizes when using ds.dims

* dims to sizes for test echodata combine

* Chunks as dictionaries in `_get_auto_chunk` [all tests ci] (#1454)

* chunks as dictionaries in _get_auto_chunk

* when zarr encoding get chunks as list-like and rename _get_auto_chunk to _get_dask_auto_chunk to show that there is a difference between Dask and Zarr chunking

* set decode timedelta to False since it will default to this in later xarray versions (#1462)

* Fix deprecation warning for truth value of an empty array [all tests ci] (#1450)

* fix deprecation warning for truth value of an empty array

* change test outcome from True to False when param is missing

* found logic error in checking parameter validity before loading XML

---------

Co-authored-by: Wu-Jung Lee <[email protected]>

* Fill in NaN for missing EK80 coefficients [all tests ci] (#1458)

* fill in nan for missing filter coeffs from all channels

* update test_convert_ek80_no_fil_coeff

* remove commented out section

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cast numpy float array to int

* add compute Sv test

* remove test mark

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: ctuguinay <[email protected]>

* Backward compatibility of raw-converted dataset with `xr.DataTree` [all tests ci] (#1447)

* EP-1420 resolved path issue with ek60_missing_channel_power

* EP-1420 fixed path to use string concatenation

* formatting

* adding conditionals for checking platform nmea and exchange from time1 to time_nmea

* working datatree for v0.9.0 w o calibration

* Drop Ping Time Duplicates (#1382)

* init commit

* revert change to fix merge conflict

* test only one file

* use other file

* move test duplicate to test convert ek

* add extra line

* move test back to ek80 convert

* pin zarr and add check unique ping time duplicates and tests

* fix test message

* test remove zarr pin

* add back zarr pin

* Fix tests that fail from new Xarray variable and attribute assignment updates (#1433)

* add .values to change water level scalar value

* add .values to fillna

* fix assign attributes

* remove test1

* update echodata properly

* remove print statement

* Assemble AD2CP timestamp with nanosecond precision (#1436)

* assemble ad2cp timestamp at ns instead of us

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add noqa

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use `import_resources.files` instead of the legacy `open_text` (#1434)

* use import_resources.files to open yaml files

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* import from importlib.resources

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* bump codespell version, add exceptions (#1438)

* Pin `zarr` and `netcdf4` temporarily [all tests ci] (#1429)

* pin zarr and netcdf4 in requirements.txt

* change np.alltrue to np.all

* use np.argmin(da.data) instead of da.argmin()

* use S instead s for MVBS ping_time_bin

* extract single element in computing nsamples and L in tapered_chirp

* change from S to s in testing.py

* Add `type-extensions` to requirements.txt (#1440)

* add type-extensions

* sort requirements.txt

* add manual trigger to pypi workflow (#1442)

* update cff with ices paper data (#1443)

* add assign actual range utility function (#1435)

* chore(deps): bump actions/setup-python from 5.3.0 to 5.4.0 (#1445)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.3.0 to 5.4.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v5.3.0...v5.4.0)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1446)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/PyCQA/isort: 5.13.2 → 6.0.0](PyCQA/isort@5.13.2...6.0.0)
- [github.com/psf/black: 24.10.0 → 25.1.0](psf/black@24.10.0...25.1.0)
- [github.com/codespell-project/codespell: v2.4.0 → v2.4.1](codespell-project/codespell@v2.4.0...v2.4.1)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* added pooch and bound the test registry to the echodata test

* EP-1420 added legacy test files >=v0.8.4, created pooch registry, parameterized test

* EP-1420 checked tests

* EP-1420 debugging TestEchoData test failures, test_nbytes, test_setattr, & test_setitem

* EP-1447 troubleshooting bug still

* EP-1420 removed pooch and added normal test path resources

* EP-1420 fixed test_nbytes, commented out test_setattr

* EP-1420 changed "time_nmea" to "nmea_time", added ek80 legacy_datatree tests, fixed readme

* EP-1420 removed import

* trying to add sonar channel_all

* EP-1420 checked sonar for channel and renamed to channel_all

* check legacy data differently

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove v0.9.2 zarr test files

* remove test_setattr that is no longer needed, see #1457 that removes EchoData.__setattr__

* Update echopype/tests/echodata/test_echodata.py

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Caesar Tuguinay <[email protected]>
Co-authored-by: Wu-Jung Lee <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add v0.9.1 and v0.10.0 release notes to docs (#1465)

* add v0.9.1 and v0.10.0 whats new

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update workflows to use python 3.12 and ubuntu 22.04 [all tests ci] (#1466)

* update to use python 3.12 and ubuntu 22.04

* update v0.10.0 release notes

* Replace `pkg_resources.resource_string` with `importlib.resources.files` [all tests ci] (#1468)

* replace pkg_resources with importlib.resources

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused pkg_resources import

* update __name__ to __package__ and use read_text

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused pkg_resources again

* update whats new

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore(deps): bump pypa/gh-action-pypi-publish from 1.12.3 to 1.12.4 (#1430)

Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.12.3 to 1.12.4.
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.12.3...v1.12.4)

---
updated-dependencies:
- dependency-name: pypa/gh-action-pypi-publish
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/cache from 4.2.0 to 4.2.1 (#1469)

Bumps [actions/cache](https://github.com/actions/cache) from 4.2.0 to 4.2.1.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4.2.0...v4.2.1)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* not -> no (#1471)

* [pre-commit.ci] pre-commit autoupdate (#1472)

updates:
- [github.com/PyCQA/isort: 6.0.0 → 6.0.1](PyCQA/isort@6.0.0...6.0.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore(deps): bump actions/cache from 4.2.1 to 4.2.2 (#1473)

Bumps [actions/cache](https://github.com/actions/cache) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4.2.1...v4.2.2)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/setup-python from 5.4.0 to 5.5.0 (#1481)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v5.4.0...v5.5.0)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: 5.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1480)

updates:
- [github.com/PyCQA/flake8: 7.1.2 → 7.2.0](PyCQA/flake8@7.1.2...7.2.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore(deps): bump actions/cache from 4.2.2 to 4.2.3 (#1479)

Bumps [actions/cache](https://github.com/actions/cache) from 4.2.2 to 4.2.3.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4.2.2...v4.2.3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* expand ping time preemptively and add corresponding chunking tests (#1483)

* Enhance `compute_MVBS` feasability  (#1470)

* add range var max and reindex to compute_MVBS parameters and drop flox maximum version

* add tests for compute MVBS changes

* add todo for compute NASC

* add np.nan fill value

* Update echopype/commongrid/api.py

Co-authored-by: Wu-Jung Lee <[email protected]>

---------

Co-authored-by: Wu-Jung Lee <[email protected]>

* indent line of code

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Wu-Jung Lee <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: oftfrfbf <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jan Meischner <[email protected]>

Author: 6 people

echopype/consolidate/ek_depth_utils.py

@@ -74,36 +74,39 @@ def ek_use_platform_angles(platform_ds: xr.Dataset, ping_time_da: xr.DataArray)
     return align_to_ping_time(echo_range_scaling, "time2", ping_time_da)
 
 
-def ek_use_beam_angles(
-    beam_ds: xr.Dataset,
-) -> xr.DataArray:
+def ek_use_beam_angles(beam_ds: xr.Dataset) -> xr.DataArray:
     """
-    Use `beam_direction_x`, `beam_direction_y`, and `beam_direction_z` from the EK Beam group to
-    compute echo range rotational values.
+    Compute echo range scaling from beam_direction components. For each channel, we expect that
+    the beam direction vector is normalized. If not, then we normalize the z direction and set
+    that as the echo range scaling.
+    Additionally, if a nonzero vector is not normalized, a warning is issued and it is normalized.
+    If a channel-wise beam direction vector is zero, a warning is issued and the returned z value
+    is set to NaN.
     """
-    # Check and log NaNs if they exist in the Beam group variables
+    # Check and log NaNs if they exist in the Beam direction variables
     _check_and_log_nans(
         beam_ds, "Sonar/Beam_group1", ["beam_direction_x", "beam_direction_y", "beam_direction_z"]
     )
 
-    # Grab beam angles from beam group
     beam_direction_x = beam_ds["beam_direction_x"]
     beam_direction_y = beam_ds["beam_direction_y"]
     beam_direction_z = beam_ds["beam_direction_z"]
 
-    # Compute echo range scaling from pitch roll rotations
-    beam_dir_rotmatrix_stack = [
-        [
-            np.array([0, 0, beam_direction_x[c]]),
-            np.array([0, 0, beam_direction_y[c]]),
-            np.array([0, 0, beam_direction_z[c]]),
-        ]
-        for c in range(len(beam_direction_x))
-    ]
-    rot_beam_direction = R.from_matrix(beam_dir_rotmatrix_stack)
-    echo_range_scaling = rot_beam_direction.as_matrix()[:, -1, -1]
-    echo_range_scaling = xr.DataArray(
-        echo_range_scaling, dims="channel", coords={"channel": beam_ds["channel"]}
-    )
+    # Calculate the norm for each channel
+    norm = np.sqrt(beam_direction_x**2 + beam_direction_y**2 + beam_direction_z**2)
+
+    # Warn if any nonzero vector is not normalized
+    tolerance = 1e-8
+    if ((norm > tolerance) & (np.abs(norm - 1) > tolerance)).any():
+        logger.warning(
+            "Beam direction vector was not normalized; applying normalization. "
+            "By definition, it should have been normalized."
+        )
+
+    # Warn if any channel has a (nearly) zero vector
+    if (norm < tolerance).any():
+        logger.warning("Some beam direction vectors are zero. Outputting NaN for those channels.")
 
-    return echo_range_scaling
+    # For channels with near-zero norm, we return NaN. Otherwise, we return the normalized
+    # z component.
+    return xr.where(norm < tolerance, np.nan, beam_direction_z / norm)

echopype/tests/consolidate/test_add_depth.py

@@ -104,28 +104,74 @@ def test_ek_use_platform_angles_output():
 
 
 @pytest.mark.unit
-def test_ek_use_beam_angles_output():
+def test_ek_use_beam_angles_output(caplog):
     """
-    Test `use_beam_angle` outputs for 2 sideways looking beams and 1 vertical looking beam.
+    Test `use_beam_angle` outputs for 2 sideways looking beams, 1 vertical looking beam, and 1
+    beam that does not have normalized directions.
     """
     # Create a Dataset with channel-specific beam direction vectors
     # In channels 1 and 2, the transducers are not pointing vertically at all so the echo
     # range scaling should be 0 (i.e zeros out the entire depth).
     # In channel 3, the transducer is completely vertical so the echo range scaling should
     # be 1 (i.e no change).
-    # In channel 4, the transducer is tilted to the x direction by 30 deg, so the
-    # echo range scaling should be sqrt(3)/2.
+    # In channel 4, we do not have a normalized vector so the echo range scaling should be
+    # sqrt(3)/2 divided by the norm of channel 4.
     channel_da = xr.DataArray(["chan1", "chan2", "chan3", "chan4"], dims=("channel"))
     beam_ds = xr.Dataset(
         {
-            "beam_direction_x": xr.DataArray([1, 0, 0, 1/2], dims=("channel")),
+            "beam_direction_x": xr.DataArray([1, 0, 0, 1], dims=("channel")),
             "beam_direction_y": xr.DataArray([0, 1, 0, 0], dims=("channel")),
             "beam_direction_z": xr.DataArray([0, 0, 1, np.sqrt(3)/2], dims=("channel")),
         },
         coords={"channel": channel_da}
     )
+
+    # Turn on logger verbosity
+    ep.utils.log.verbose(override=False)
+
+    # Compute beam angle echo range scaling
     echo_range_scaling = ep.consolidate.ek_depth_utils.ek_use_beam_angles(beam_ds)
-    assert np.allclose(echo_range_scaling.values, np.array([0.0, 0.0, 1.0, np.sqrt(3)/2]))
+
+    # Turn off logger verbosity
+    ep.utils.log.verbose(override=True)
+
+    # Verify the correct warning
+    assert "Beam direction vector was not normalized" in caplog.text
+
+    # Compute and compare manual test values with function values
+    fourth_value = (np.sqrt(3)/2) / np.sqrt(((np.sqrt(3)/2) ** 2) + 1)
+    assert np.allclose(echo_range_scaling.values, np.array([0.0, 0.0, 1.0, fourth_value]))
+
+
+@pytest.mark.unit
+def test_warning_zero_vector(caplog):
+    """
+    Test that a warning is logged and NaN is returned for channels with zero beam direction vector.
+    """
+    # Create a dataset with two channels: zero vector [0, 0, 0] and nonzero normalized vector [0, 1, 0].
+    beam_ds = xr.Dataset(
+        {
+            "beam_direction_x": xr.DataArray([0, 0], dims=("channel")),
+            "beam_direction_y": xr.DataArray([0, 1], dims=("channel")),
+            "beam_direction_z": xr.DataArray([0, 0], dims=("channel")),
+        }
+    )
+    
+    # Turn on logger verbosity
+    ep.utils.log.verbose(override=False)
+    
+    # Compute beam angle echo range scaling
+    echo_range_scaling = ep.consolidate.ek_depth_utils.ek_use_beam_angles(beam_ds)
+    
+    # Verify the correct warning
+    assert "Some beam direction vectors are zero" in caplog.text
+
+    # Check that channel 0 output is NaN and channel 1 output is 0
+    assert np.isnan(echo_range_scaling.values[0])
+    assert np.isclose(echo_range_scaling.values[1], 0.0)
+    
+    # Turn off logger verbosity
+    ep.utils.log.verbose(override=True)
 
 
 @pytest.mark.integration