[Bug]: DIST-S1 HIST - Time Based Error - SAS Error

[hhlee445]

Checked for duplicates

Yes - I've already checked

Describe the bug

This error was reported by INT. It seems the issue occurred because PCM passed the input lists incorrectly. PCM needs to ensure that it passes the correct number of current RTC bursts and the corresponding matching baseline bursts.

See full Run information in Attached file below:

Test Run Results taken from (9/26/2025 10:42 AM A. K. Roth)
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/9574427&group_by=tests:assignedto_id&group_order=desc&group_id=-1

142 SCIFLO_L3_DIST_S1 failures with this error:

Exception Type: <class 'RuntimeError'>

Got exception running:
docker run --init --rm -u 10005:10005 -w /home/ops -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_runconfig_dir:/home/ops/runconfig:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_input_dir:/home/ops/input_dir:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_output_dir:/home/ops/output_dir -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_scratch_dir:/home/ops/scratch_dir opera_pge/dist_s1:6.0.0-rc.2.0 --file /home/ops/runconfig/RunConfig.yaml
STDOUT/STDERR:
Running preprocessor for DistS1PreProcessorMixin
Starting SAS execution for DistS1Executor
Traceback (most recent call last):
File "/home/ops/opera/scripts/pge_main.py", line 194, in
pge_main()
~~~~~~~~^^
File "/home/ops/opera/scripts/pge_main.py", line 190, in pge_main
pge_start(run_config_filename)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/home/ops/opera/scripts/pge_main.py", line 164, in pge_start
pge.run()
~~~~~~~^^
File "/home/ops/opera/pge/base/base_pge.py", line 800, in run
self.run_sas_executable(**kwargs)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/ops/opera/pge/base/base_pge.py", line 779, in run_sas_executable
elapsed_time = time_and_execute(
command_line, self.logger, self.runconfig.execute_via_shell
)
File "/home/ops/opera/util/run_utils.py", line 257, in time_and_execute
logger.critical(module_name, ErrorCode.SAS_PROGRAM_FAILED, error_msg)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ops/opera/util/logger.py", line 426, in critical
raise RuntimeError(description)
RuntimeError: Command "/opt/conda/envs/dist-s1-env/bin/dist-s1 run_sas --run_config_path /home/ops/scratch_dir/RunConfig_sas.yaml" failed with exit code 1
ERROR conda.cli.main_run:execute(125): conda run sh -c exec ${CONDA_ROOT}/bin/pge_docker_entrypoint.sh "${@}" – --file /home/ops/runconfig/RunConfig.yaml failed. (See above for error)

Traceback (most recent call last):
File "/home/ops/verdi/ops/opera-pcm/util/exec_util.py", line 54, in call_noerr
output = check_output(cmd, stderr=STDOUT, shell=True)
File "/opt/conda/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/opt/conda/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker run --init --rm -u 10005:10005 -w /home/ops -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_runconfig_dir:/home/ops/runconfig:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_input_dir:/home/ops/input_dir:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_output_dir:/home/ops/output_dir -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_scratch_dir:/home/ops/scratch_dir opera_pge/dist_s1:6.0.0-rc.2.0 --file /home/ops/runconfig/RunConfig.yaml' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ops/verdi/ops/opera-pcm/opera_chimera/opera_pge_job_submitter.py", line 137, in perform_adaptation_tasks
output_datasets = run_pipeline(job_json, self._base_work_dir)
File "/home/ops/verdi/ops/opera-pcm/wrapper/opera_pge_wrapper.py", line 159, in run_pipeline
exec_pge_command(
File "/home/ops/verdi/ops/opera-pcm/wrapper/opera_pge_wrapper.py", line 262, in exec_pge_command
call_noerr(cmd_line, work_dir)
File "/home/ops/verdi/ops/opera-pcm/util/exec_util.py", line 67, in call_noerr
raise RuntimeError("Got exception running:\n{}\nSTDOUT/STDERR:\n{}".format(cmd, e.output.decode()))
RuntimeError: Got exception running:
docker run --init --rm -u 10005:10005 -w /home/ops -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_runconfig_dir:/home/ops/runconfig:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_input_dir:/home/ops/input_dir:ro -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_output_dir:/home/ops/output_dir -v /data/work/jobs/2025/09/26/16/49/job-WF-SCIFLO_L3_DIST_S1-batch-p33PTR_3_S1A_a356-20250926T104240.914754Z/pge_scratch_dir:/home/ops/scratch_dir opera_pge/dist_s1:6.0.0-rc.2.0 --file /home/ops/runconfig/RunConfig.yaml
STDOUT/STDERR:
Running preprocessor for DistS1PreProcessorMixin
Starting SAS execution for DistS1Executor
Traceback (most recent call last):
File "/home/ops/opera/scripts/pge_main.py", line 194, in
pge_main()
~~~~~~~~^^
File "/home/ops/opera/scripts/pge_main.py", line 190, in pge_main
pge_start(run_config_filename)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/home/ops/opera/scripts/pge_main.py", line 164, in pge_start
pge.run()
~~~~~~~^^
File "/home/ops/opera/pge/base/base_pge.py", line 800, in run
self.run_sas_executable(**kwargs)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/ops/opera/pge/base/base_pge.py", line 779, in run_sas_executable
elapsed_time = time_and_execute(
command_line, self.logger, self.runconfig.execute_via_shell
)
File "/home/ops/opera/util/run_utils.py", line 257, in time_and_execute
logger.critical(module_name, ErrorCode.SAS_PROGRAM_FAILED, error_msg)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ops/opera/util/logger.py", line 426, in critical
raise RuntimeError(description)
RuntimeError: Command "/opt/conda/envs/dist-s1-env/bin/dist-s1 run_sas --run_config_path /home/ops/scratch_dir/RunConfig_sas.yaml" failed with exit code 1
ERROR conda.cli.main_run:execute(125): conda run sh -c exec ${CONDA_ROOT}/bin/pge_docker_entrypoint.sh "${@}" – --file /home/ops/runconfig/RunConfig.yaml failed. (See above for error)

What did you expect?

I expected [...]

Reproducible steps

1.
2.
3.
...

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...

[hhlee445]

Reported by Steven.

While trying to run the DIST-S1 E2E Historical Time Specific Tile and Date Range by ADT, I encountered this error for both Tiles (11SLT and 30UXC). I was able to get the first two granules for 11SLT, and the first six for 30UXC. See the TestRail results (through the link) for full details.