Test broker performance under LSST-sized loads using the Rubin test stream

[troyraen]

#123 is complete. Now we need to test our system under the load.

"Tasks" in the Kafka Connector (i.e., our consumer) do the actual data copying and determine the parallelism of the consumer. When I used 10 max tasks (as we do with ZTF), the consumer VM (size e2-standard-2) ran out of memory. For now I used 1 task. We should test our system under the realistic LSST load and experiment with the number of tasks, machine size etc. The number of tasks is set in the config file: broker/consumer/rubin/ps-connector.properties.

#123 and the README referenced in that PR have more information about ingesting the Rubin stream.

Update: I have since realized that I left a setting (from the ZTF consumer) that reset the Kafka offset to “earliest”. This means that I was not ingesting the live stream, but a backlog. This explains the very high incoming rate (high even for LSST... it was ~304,000 in 2 minutes). So we should try using 10 max tasks on the live stream.