Skip to content

Conversation

@chrisjrd
Copy link
Contributor

Purpose

implement DSWx-NI grace period boilerplate.

Proposed Changes

  • [CHANGE] DSWx-NI query and download to mark records as submitted_for_download to support future grace period implementation.

Issues

#872

Testing

tested locally

@chrisjrd chrisjrd self-assigned this Dec 17, 2025
@chrisjrd chrisjrd requested a review from hhlee445 December 17, 2025 18:34
@hhlee445
Copy link
Contributor

hhlee445 commented Jan 7, 2026

Getting an error with a command

python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c NISAR_L2_GCOV_BETA_V1 --native-id=NISAR_L2_PR_GCOV_088_039_D_085_2005_SHSH_A_20251114T220420_20251114T220427_T00407_N_P_J_001 --endpoint=UAT --job-queue=opera-job_worker-gcov_download

[2026-01-07 19:23:07,843: INFO/supply_job_id] Running outside of job context. Generating random job ID. job_id='94694eb7-1d57-456b-bde3-046ebe6c3cad'
Traceback (most recent call last):
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 165, in <module>
    main()
  File "/export/home/hysdsops/mozart/ops/opera-pcm/util/exec_util.py", line 35, in wrapper
    status = func(*args, **kwargs)
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 45, in main
    run(sys.argv)
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 73, in run
    results["query"] = run_query(args, token, es_conn, cmr, job_id, settings)
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 104, in run_query
    cmr_query = NisarGcovCmrQuery(args, token, es_conn, cmr, job_id, settings)
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/gcov/gcov_query.py", line 17, in __init__
    super().__init__(args, token, es_conn, cmr, job_id, settings)
  File "/export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/query.py", line 41, in __init__
    self.query_replacement_file = args.query_replacement_file
AttributeError: 'Namespace' object has no attribute 'query_replacement_file'

@hhlee445
Copy link
Contributor

hhlee445 commented Jan 8, 2026

DSWX-NI SCIFLO jobs failed with

Exception Type: <class 'ValueError'>
Exception Value: Number of processes must be at least 1
Traceback (most recent call last):
  File "/root/verdi/ops/sciflo-1.3.7/sciflo/grid/workUnit.py", line 130, in run
    result = self._run()
  File "/root/verdi/ops/sciflo-1.3.7/sciflo/grid/workUnit.py", line 172, in _run
    return func(*funcArgs)
  File "/tmp/scifloWork-ops/sciflowuid-15818421-01538571-0788082026-ae0179ad4569b9b623d8273d60e8d13c/run_pge_docker.py", line 87, in submit_pge_job
    job_json = cls_object.submit_job()
  File "/root/verdi/ops/chimera-2.2.4/chimera/pge_job_submitter.py", line 216, in submit_job
    job_json = self.perform_adaptation_tasks(job_json)
  File "/home/ops/verdi/ops/opera-pcm/opera_chimera/opera_pge_job_submitter.py", line 99, in perform_adaptation_tasks
    with get_context("spawn").Pool(
  File "/opt/conda/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/opt/conda/lib/python3.9/multiprocessing/pool.py", line 205, in __init__
    raise ValueError("Number of processes must be at least 1")
ValueError: Number of processes must be at least 1

It submitted jobs over and over again. Need to check the trigger logic

@RKuttruff
Copy link
Contributor

@hhlee445 This isn't a triggering issue. This is a bug that was introduced with the parallelized localization work I did (essentially trying to create a process pool with 0 processes). I've pushed a fix

@RKuttruff RKuttruff changed the title Issue #872: DSWx-NI grace period boilerplate Issue #872: DSWx-NI triggering logic Jan 16, 2026
rileykk added 4 commits January 20, 2026 08:47
should hopefully clarify situations where granules are catalogued but no evaluation happens due to all granules being over water
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants