Loaders: spectrum cube importers

[kecnry]

Description

This pull request adds a "3D Spectrum" importer, partially updates cubeviz plugins to work within deconfigged (more follow-up work will be needed, but they don't prevent data from loading), and temporarily redirects load_data through load to test what failures exist with current CI.

To test locally - set app.state.dev_loaders = True. Note that tests were run with that enabled and as of the last effort here, there were 20 remaining failures that will need to be addressed before removing the dev-flag.

FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_cubeviz_aperphot.py::test_cubeviz_aperphot_cube_orig_flux - AssertionError: assert '2.700e+01 ' == '4.894e+00 um'
  
  - 4.894e+00 um
  + 2.700e+01
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_cubeviz_aperphot.py::test_cubeviz_aperphot_generated_3d_gaussian_smooth - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 1 / 1 (100%)
Max absolute difference among violations: 3.64674246e-16
Max relative difference among violations: 0.75113548
 ACTUAL: Quantity(1.208231e-16, unit='erg / (Angstrom s cm2)')
 DESIRED: Quantity(4.854973e-16, unit='erg / (Angstrom s cm2)')
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/model_fitting/tests/test_fitting.py::test_cube_fitting_backend[1-zeros] - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

(shapes (200, 14, 15), (15, 14, 200) mismatch)
 ACTUAL: array([[[3.999145, 4.057785, 3.994165, ..., 4.000475, 4.000631,
         3.99054 ],
        [4.012648, 4.005936, 4.050553, ..., 3.99292 , 3.987368,...
 DESIRED: array([[[3.999145, 3.999145, 3.999145, ..., 3.999145, 3.999145,
         3.999145],
        [4.012648, 4.012648, 4.012648, ..., 4.012648, 4.012648,...
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/model_fitting/tests/test_fitting.py::test_cube_fitting_backend[1-None] - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

(shapes (200, 14, 15), (15, 14, 200) mismatch)
 ACTUAL: array([[[3.999145, 4.057785, 3.994165, ..., 4.000475, 4.000631,
         3.99054 ],
        [4.012648, 4.005936, 4.050553, ..., 3.99292 , 3.987368,...
 DESIRED: array([[[3.999145, 3.999145, 3.999145, ..., 3.999145, 3.999145,
         3.999145],
        [4.012648, 4.012648, 4.012648, ..., 4.012648, 4.012648,...
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/model_fitting/tests/test_fitting.py::test_cube_fitting_backend[None-zeros] - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

(shapes (200, 14, 15), (15, 14, 200) mismatch)
 ACTUAL: array([[[3.999145, 4.057785, 3.994165, ..., 4.000475, 4.000631,
         3.99054 ],
        [4.012648, 4.005936, 4.050553, ..., 3.99292 , 3.987368,...
 DESIRED: array([[[3.999145, 3.999145, 3.999145, ..., 3.999145, 3.999145,
         3.999145],
        [4.012648, 4.012648, 4.012648, ..., 4.012648, 4.012648,...
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/model_fitting/tests/test_fitting.py::test_cube_fitting_backend[None-None] - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

(shapes (200, 14, 15), (15, 14, 200) mismatch)
 ACTUAL: array([[[3.999145, 4.057785, 3.994165, ..., 4.000475, 4.000631,
         3.99054 ],
        [4.012648, 4.005936, 4.050553, ..., 3.99292 , 3.987368,...
 DESIRED: array([[[3.999145, 3.999145, 3.999145, ..., 3.999145, 3.999145,
         3.999145],
        [4.012648, 4.012648, 4.012648, ..., 4.012648, 4.012648,...
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/spectral_extraction/tests/test_spectral_extraction.py::test_cube_extraction_with_nan - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 10 / 10 (100%)
Max absolute difference among violations: 1.6e-16
Max relative difference among violations: 0.16666667
 ACTUAL: array([8.e-16, 8.e-16, 1.e-15, 1.e-15, 1.e-15, 1.e-15, 1.e-15, 1.e-15,
       1.e-15, 1.e-15])
 DESIRED: array(9.6e-16)
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/subset_tools/tests/test_subset_tools.py::test_import_sky_region_in_cubeviz - AssertionError: assert [(<CircleSkyR...trum [FLUX]')] == []
  
  Left contains one more item: (<CircleSkyRegion(center=<SkyCoord (ICRS): (ra, dec) in deg\n    (339.0149557, 33.97591)>, radius=0.5 arcsec)>, 'Failed to load: _orig_spatial_wcs meta tag not in 3D Spectrum [FLUX]')
  
  Full diff:
  - []
  + [
  +     (
  +         <CircleSkyRegion(center=<SkyCoord (ICRS): (ra, dec) in deg
  +     (339.0149557, 33.97591)>, radius=0.5 arcsec)>,
  +         'Failed to load: _orig_spatial_wcs meta tag not in 3D Spectrum [FLUX]',
  +     ),
  + ]
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/subset_tools/tests/test_subset_tools.py::test_get_regions - UserWarning: Regions skipped: Subset 2
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/model_fitting/tests/test_plugin.py::test_spectral_first_cube - astropy.units.errors.UnitConversionError: 'm' (length) and '' (dimensionless) are not convertible
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/tests/test_utils.py::test_wildcard_match_basic - TypeError: BaseImporterToDataCollection.__init__() missing 1 required positional argument: 'parser'
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_parsers.py::test_fits_image_hdu_with_microns - assert Unit("m") == Unit("um")
 +  where Unit("m") = <SpectralAxis \n   (observer to target:\n      radial_velocity=0.0 km / s\n      redshift=0.0)\n  [4.89049987e-06, 4.89149....89249987e-06, 4.89349987e-06,\n   4.89449987e-06, 4.89549987e-06, 4.89649987e-06, 4.89749987e-06,\n   4.89849987e-06] m>.unit
 +    where <SpectralAxis \n   (observer to target:\n      radial_velocity=0.0 km / s\n      redshift=0.0)\n  [4.89049987e-06, 4.89149....89249987e-06, 4.89349987e-06,\n   4.89449987e-06, 4.89549987e-06, 4.89649987e-06, 4.89749987e-06,\n   4.89849987e-06] m> = <Spectrum(flux=[[[9.99999983775159e-18 ... 0.0]]] erg / (Angstrom s cm2 pix2) (shape=(8, 9, 10), mean=0.00000 erg / (A...87e-06 4.89249987e-06 ... 4.89649987e-06\n 4.89749987e-06 4.89849987e-06] m> (length=9); uncertainty=StdDevUncertainty)>.spectral_axis
 +  and   Unit("um") = u.um
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_parsers.py::test_fits_image_hdu_parse_from_file - AssertionError: assert ('Pixel x=-1....148973 (deg)') == ('Pixel x=-1....148908 (deg)')
  
  At index 1 diff: 'World 13h41m46.5994s +26d59m58.6136s (ICRS)' != 'World 13h41m46.6368s +26d59m58.6136s (ICRS)'
  
  Full diff:
    (
        'Pixel x=-1.0 y=00.0',
  -     'World 13h41m46.6368s +26d59m58.6136s (ICRS)',
  ?                     ^^^^
  +     'World 13h41m46.5994s +26d59m58.6136s (ICRS)',
  ?                     ^^^^
  -     '205.4443201084 26.9996148908 (deg)',
  ?               -----            ^^
  +     '205.4441642302 26.9996148973 (deg)',
  ?             ++++ +             ^^
    )
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_parsers.py::test_numpy_cube - Failed: DID NOT RAISE <class 'TypeError'>
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/tests/test_data_menu.py::test_remove_all_data_cubeviz - astropy.units.errors.UnitConversionError: 'm' (length) and '' (dimensionless) are not convertible
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_regions.py::TestLoadRegions::test_regions_pixel - UserWarning: Regions skipped: Subset 1
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_regions.py::TestLoadRegions::test_regions_sky_has_wcs - AssertionError: assert 1 == 0
 +  where 1 = len([(<CircleSkyRegion(center=<SkyCoord (ICRS): (ra, dec) in deg\n    (205.4397, 27.0035)>, radius=0.0004 deg)>, 'Failed to load: _orig_spatial_wcs meta tag not in has_microns[FLUX]')])
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_regions.py::TestLoadRegions::test_spatial_spectral_mix - astropy.units.errors.UnitConversionError: 'm' (length) and '' (dimensionless) are not convertible
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/tests/test_subsets.py::TestRegionsFromSubsets::test_get_regions_from_subsets_cubeviz - assert False
 +  where False = isinstance(None, CircleSkyRegion)
FAILED ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/default/plugins/subset_tools/subset_tools.py::jdaviz.configs.default.plugins.subset_tools.subset_tools.SubsetTools.get_regions
ERROR ../../.tox/py312-test/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_regions.py::TestLoadRegions::test_spatial_spectral_mix - astropy.units.errors.UnitConversionError: 'm' (length) and '' (dimensionless) are not convertible

This PR will need to be rebased back to main once #3739 is merged.

Note: this work brings up the question - do we want to keep flux, uncertainty, and mask as separate data-collection entries? This PR maintains the current behavior/support.... but should we deprecate this in favor of a single data collection entry? Or do we want to keep this behavior to allow multiple masks? Right now, how do you re-assemble the input spectrum from multiple DC cubes?

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• If new remote data is added that uses MAST, is the URI added to the cache-download.yml workflow?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[kecnry]

removed the no-changelog-entry-needed label now that this is actually deprecating load_data

[kecnry]

this is now sending s2d files that used to go through fits to now go through specutils, losing the ability to select an extension. Any idea why? Seems this isn't causing a test failure though, so I can try to add a test that fails and enforces the previous/preferred behavior.

[MatthewPortman]

Memory report for loading in the Manga cube in test_autoconfig.py

In FormatSelect._update_items:

• parser specutils.Spectrum: 524 MB (BaseResolver.output)
• specutils.Spectrum > 3D Spectrum: 477 MB
• object > 3D Spectrum: 382 MB

in Spectrum3DImporter.__call__:

• BaseImporterToDataCollection.__call__ via super: 191 MB
• Creating the cube mask to add to dc: 191 MB
• Spectral Extraction: 263 MB

Total: ~2000 MB
Total for the entire process ~2076 MB

Profiling information for the curious