bug: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility

[jmp75]

Description of the bug

Running unit tests (pytest) from the command line tasker, unit tests fail (some expected to at this stage) but not for the right reason.

To Reproduce

at commit 0c8ccba

make test causes errors.

Curiously, running the unit tests from the VSCode debugger does not fail or show any runtime warning.

Full traceback

  ....F                                                                    [100%]
  ==================================== ERRORS ====================================
  ______________________ ERROR collecting test_read_file.py ______________________
  tests/test_read_file.py:6: in <module>
      from efts_io.wrapper import EftsDataSet
  src/efts_io/wrapper.py:11: in <module>
      from efts_io.conventions import (
  src/efts_io/conventions.py:6: in <module>
      import netCDF4 as nc
  .venvs/3.12/lib/python3.12/site-packages/netCDF4/__init__.py:3: in <module>
      from ._netCDF4 import *
  src/netCDF4/_netCDF4.pyx:1: in init netCDF4._netCDF4
      ???
  E   RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject```

Possible fix

xarray/issues/7259

[jmp75]

Triggers the error:

python -c "import numpy;import warnings;warnings.filterwarnings('error');import netCDF4"

passes:

python -c "import netCDF4;import numpy;import warnings;warnings.filterwarnings('error');import netCDF4"

[jmp75]

but, even if I load netCDF4 as early as possible in the codebase in the imports, no luck still fails

[jmp75]

I renounced having an explicit dependency on netCDF4 for now. However I still have when using open_dataset an issue:

  tests/test_read_file.py:47: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  src/efts_io/wrapper.py:118: in __init__
      x = xr.open_dataset(data, decode_times=False)
  .venvs/3.12/lib/python3.12/site-packages/xarray/backends/api.py:588: in open_dataset
      backend_ds = backend.open_dataset(
  .venvs/3.12/lib/python3.12/site-packages/xarray/backends/netCDF4_.py:645: in open_dataset
      store = NetCDF4DataStore.open(
  .venvs/3.12/lib/python3.12/site-packages/xarray/backends/netCDF4_.py:375: in open
      import netCDF4
  .venvs/3.12/lib/python3.12/site-packages/netCDF4/__init__.py:3: in <module>
      from ._netCDF4 import *
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  >   ???
  E   RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

I am pretty sure I previously had this test going further. What is going on ??

[jmp75]

I recreated the virtual environments from scratch with make setup; still the same story. But I understand a bit better what is going on.

with duty test two runtime warnings cause test failures:

FAILED config/test_read_file.py::test_round_trip - NameError: name 'UNITS_ATTR_KEY' is not defined
  FAILED config/test_read_file.py::test_read_thing - RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibi...
  FAILED config/test_create.py::test_create_new_efts - FutureWarning: The return type of `Dataset.dims` will be changed to return ...

but with coverage run -m pytest, no such failure (other later causes).

================================================================== short test summary info ===================================================================
FAILED tests/test_read_file.py::test_read_thing - ValueError: first_where: Invalid condition, no element is true
FAILED tests/test_read_file.py::test_round_trip - NameError: name 'UNITS_ATTR_KEY' is not defined
========================================================== 2 failed, 5 passed, 4 warnings in 1.76s ====================

duty test must be setting settings such that warnings are treated as errors (a good thing). This warning appears to be a red flag, but I am a bit stuck for now, at least cannot use make test as part of the workflow.