Monitoring of duplicates and missed products

[LucaCinquini]

We need to have better accountability for all of our products, especially for CSLC-S1 (which is input to DISP-S1) but also DSWx-S1, DSWx-HLS and RTC-S1.

Every day, we need to execute a cron job that:
a) For each product, queries CMR for # of duplicates and # of missing granules, for the past 10 days (configurable)
b) Creates a text report of it, stores it on S3
c) Creates plots on S3 for visual embedding in public GitHub pages

We might be able to use lots of the infrastructure already setup by Alvin.
@riverma please coordinate with @RKuttruff .

[riverma]

@RKuttruff - as discussed, we should expand the scope of this ticket to handle the deployment and visualization for both the duplicate detection and the missing granules scripts.

The missing granules wrapper script that @achongjpl worked on needs to be tweaked for DSWX-S1. Right now, it's not giving fully accurate results because it skips a post-processing step. We need to update the wrapper to run the following command to get that sorted:

python ~/mozart/ops/opera-pcm/tools/ops/data_subscriber/data_subscriber_client.py --rtc-native-ids-file <missing_txt_from_cmr_audit> --output <compressed_missing_rtc_list_txt>

Once that’s fixed, the plan is to have that wrapper running on the OPERA INT-FWD via crontab, pushing reports to Amazon S3 and metrics to OpenSearch (something that's already turned on via a crontab but we should verify for DSWX-S1). We should then circle back to the duplicate detection script and ensure it’s aligned with that workflow - running on INT-FWD and publishing to the same S3 folder structure and OpenSearch index.

Once we have both data streams flowing into S3 and OpenSearch, we can build out a dashboard to track the volume of duplicates and missing granules over time. We can also create a Python script that queries Elasticsearch to generate PNGs of those charts, saves them to S3, and renders them on the OPERA SDS README. I think doing all that together will allow OPS to have much greater time series insights into duplicates and missing products and also allow project team members to see the status of operations on our GitHub page. Thanks for all your help and let me know what you think about this. It'll be a super important audit report for the team!