CalibrationDate not available in factory mode

[oceancolorcoder]

@ARamsay17 pySAS sample data test in Factory regime:

Using LUT interpolations.
Calculating Zhang glint correction (LUT).
Interpolating Z17 LUT using cubic method
Zhang17 LUT Elapsed Time: 1.5 s
Traceback (most recent call last):
  File "/Users/daurin/GitRepos/HyperCP/Main.py", line 584, in singleL2Clicked
    self.processSingle("L2")
  File "/Users/daurin/GitRepos/HyperCP/Main.py", line 558, in processSingle
    Controller.processFilesSingleLevel(
  File "/Users/daurin/GitRepos/HyperCP/Source/Controller.py", line 904, in processFilesSingleLevel
    Controller.processSingleLevel(pathOut, fp, calibrationMap, level)
  File "/Users/daurin/GitRepos/HyperCP/Source/Controller.py", line 772, in processSingleLevel
    root = Controller.processL2(root,outFilePath)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/Controller.py", line 470, in processL2
    node = ProcessL2.processL2(root,station)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1877, in processL2
    if not ProcessL2.stationsEnsemblesReflectance(node, root,station):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1769, in stationsEnsemblesReflectance
    if not ProcessL2.ensemblesReflectance(node, sasGroup, referenceGroup, ancGroup,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1390, in ensemblesReflectance
    l1b_unc, x_breakdown_unc = sensor.ClassBased(node, uncGroup, stats)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/PIU/BaseInstrument.py", line 126, in ClassBased
    PDS = pds(node, uncGrp)
          ^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/PIU/PIUDataStore.py", line 73, in __init__
    [self.read_uncertainties(root, input, sensor) for sensor in self.sensors]
  File "/Users/daurin/GitRepos/HyperCP/Source/PIU/PIUDataStore.py", line 73, in <listcomp>
    [self.read_uncertainties(root, input, sensor) for sensor in self.sensors]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daurin/GitRepos/HyperCP/Source/PIU/PIUDataStore.py", line 317, in read_uncertainties
    cal_date  = dt.strptime(root.getGroup(calDate_string).attributes['CalibrationDate'], "%Y%m%d%H%M%S")
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'CalibrationDate'

[ARamsay17]

solved with 0e60abc included in PR #404

[oceancolorcoder]

@ARamsay17 did you only test this on the sample pySAS data? I think it may be working with that example because it finds FidRadDB RADCAL files for that instrument (available_files_calTime0 in the fidRadDB path PL1b L65). How will CalibrationDate be defined for instruments that have no FidRadDB files, or are using factory rad cal that are more recent than the FidRadDB RADCAL on file?

Alternatively, when in Factory regime, the CalibrationDate attribute is no longer defined and no longer queried in PL2? This seems to be what I am seeing, in which case, we should be fine. Please confirm. Thanks.

[ARamsay17]

@oceancolorcoder that is a very interesting point. I think you're correct and we need to have a closer look at this. Perhaps a more straightforward solution would be to default to 1% stability uncertainty for the factory regime? Also does this imply that our calibration stability calculation only works in the case where we have cal/char from FidradDB? Even in Sensor-specific and Class-based.

[oceancolorcoder]

As per today's discussion, we need to incorporate the CalibrationDate attribute for factory regime at or before PL1b, and perform the pro-rated 1% stability in L2 for uncertainty as in class-based regime.

[oceancolorcoder]

@ARamsay17 @juanchossn The plot thickens. I find that although my mac and linux machines are on the same dev commit, my mac version is getting over this CalibrationDate error (albeit erroneously; more below), while my linux version is crashing after multiple ensembles. Again, this is the OXR20250306 dataset on the box.

Investigation shows that the problem is here in PIUDataStore.read_uncertainties L317:

cal_date = dt.strptime(root.getGroup(calDate_string).attributes['CalibrationDate'], "%Y%m%d%H%M%S")

This is only looking for the CalibrationDate attribute in the *_LIGHT_L1AQC groups. On mac, it gets through all three ensembles and returns the date found in the FidRadDB file. As this is a factory regime run with a newer factory cal from 2024, and the date in the mis-indentified FidRadDB file is from 2022, this is incorrect.

Even stranger: on linux, the cal_date line runs successfully for the first two ensembles, finding CalibrationDate in the attributes of ES_LIGHT_L1AQC, it does NOT find it in the third ensemble and crashes. Very weird!

[UPDATE: My mistake. The linux version had worked on all ensembles of that file, but then moved onto another file that had not been updated with CalibrationDate in the L1B-onward files. One mystery solved. To Do below still stands and issue will remain open.]

[oceancolorcoder]

To do: The CalibrationDate attribute in the propagated L1AQC groups going to L2 (apparently the only groups that use this later) should be populated by the correct calibrations; either factory cal file for regime 1, or FidRadDB RADCAL for 2 and 3.

[oceancolorcoder]

Another anomaly I discovered is that even when running in regime 1 (factory), the configuration files lists the FidRadDB files as "needed":

"fL1bCal": 1,
   "fL1bThermal": 1,
   "neededCalCharsFRM": {
       "LI": [
           "SAT0385_RADCAL",
           "SAT0385_STRAY",
           "SAT0385_THERMAL",
           "SAT0385_POLAR"
       ],
       "ES": [
           "SAT0488_RADCAL",
           "SAT0488_STRAY",
           "SAT0488_THERMAL",
           "SAT0488_ANGULAR"
       ],
       "LT": [
           "SAT0386_RADCAL",
           "SAT0386_STRAY",
           "SAT0386_THERMAL",
           "SAT0386_POLAR"
       ]
   },

This may be harmless if the "needed" files are not actually needed by the code, but it's confusing.

[Edit: we may want to shift this to its own issue associated with PL1b CalChar]

[oceancolorcoder]

Partial resolution in commits:
866389d
081cc32
b7a7a45
fa320a9

[oceancolorcoder]

L1AQC carry-over group still problematic after PL1b L247