Assess how the broker responds to high alert rates and determine areas in need of improvement

[troyraen]

Since LSST's first light is getting close, we should do another assessment of how our broker pipeline responds to high alert rates. I have done this before (see especially #96) but there are one or two significant problems still outstanding, plus, our pipeline has changed since then. The scope of this issue is to assess, determine problem areas and solutions, and make an implementation plan.

[troyraen]

Here are my current thoughts based on what I've seen in the past:

Our number one problem is the BigQuery rate limit discussed in #244 (discovered in #96). We need to:

• Implement Replace 'store_in_BigQuery' Cloud Function with a Pub/Sub to BigQuery Subscription #244 and then assess that solution. Does it fix the problem for that module? Can we use it for the other modules as well?
• If "no" to any of the above, what code changes do we need to make? I think the answer is (at least) create the client at the module level and get rid of the API call that gets the table before putting the data. pittgoogle-client will help for both.

Cloud Functions have some rate limits that are fine for ZTF but will be a problem for LSST. We need to:

• Use Cloud Run for LSST modules.

The module that dumps Avro files to Cloud Storage responds very well. Need to:

• Double check it at LSST scale. It is particularly important that this module runs smoothly because we use it to deduplicate, which means that it's the choke point (it feeds the rest of the pipeline). The code checks whether the object is already in the bucket before putting it there. I've been very pleasantly surprised so far at how fast it does this. I haven't seen any hints of problems. But it's possible that it hasn't been subjected to rates that are high enough for long enough for problems to show up.

Consumer module also responds well. Need to:

• Look at how many tasks we're using (config that determines parallelism; ideal number depends on the number of partitions used by the upstream Kafka Broker).
• Consider whether we need to run the Connector in distributed mode.

My assessments have been done using anomalous, rare events where we've received alerts from ZTF at approximately LSST rates for a short period of time (order 10 minutes). Need to:

• Intentionally test everything at LSST scale for long enough that we're convinced we understand what will happen.

@hernandezc1 may have more insights from the ELAsTiCC Challenge.

[troyraen]

ZTF had trouble publishing alerts Jan 1 & 2 due to the year change. This resulted in a backlog of alerts being published yesterday (Jan 2-3). I tried to manage the flow into BigQuery by reducing the number of instances (per module) allowed to run concurrently. I'm not sure whether it helped. We started getting logs saying "The request was aborted because there was no available instance." and the average function execution times didn't seem to go down with the number of concurrent instances. It might be worth doing a postmortem review of the metrics and logs from that period. Peak incoming rate happened around Jan 2, 2025, 10:35 pm PST.