Something marked all task instances as failed

[mat-pielat]

Something caused task states of all DAGs to become failed - including ones that have run successfully in the past. DAG states were unaffected, so one could observe successful DAG runs with all their tasks marked as failed.

• DAGs that have started since the occurrence of this event (that flipped everything in the past) execute normally, i.e. tasks are marked correctly and their state doesn't change later.
• We don't know whether the affected tasks instances were simply marked as failed or their state was reset and they subsequently entered the failed state. Log shows no trace of retried executions.
• We managed to narrow down the likely time of the occurrence of this event (let's call it "point T"). I figured that point T is somewhere between execution times of the last affected task and the first unaffected one, which ended up being a ~25 second wide window.
• Database metrics show a big spike in row updates at point T but we don't know the query that caused it.

This is the third time we observe such phenomenon. It happened in three different deployments but considering the size of our fleet, this is a very rare occurrence. For each of these deployments it's a one off and we don't know what are the conditions that can trigger the situation. It happened once with Airflow version 2.4.3 and twice with 2.5.3.

One thing is common for all three cases was that clear command was used around the time of the occurrence. The audit log of the recently affected Airflow deployment contains these two entries:

dttm: 2023-XX-XX, XX:XX:45
event: clear
extra: [('dag_id', 'foobar'), ('dag_run_id', 'manual__...'), ('confirmed', 'false'), ('execution_date', '...'), ('past', 'false'), ('future', 'false'), ('upstream', 'false'), ('downstream', 'false'), ('recursive', 'false'), ('only_failed', 'true'), ('task_id', 'fizzbuzz')]

dttm: 2023-XX-XX, XX:XX:53
event: clear
extra: [('dag_id', 'foobar'), ('dag_run_id', 'manual__...'), ('confirmed', 'true'), ('execution_date', '...'), ('past', 'false'), ('future', 'false'), ('upstream', 'false'), ('downstream', 'false'), ('recursive', 'false'), ('only_failed', 'true'), ('task_id', 'fizzbuzz')]

However, these were logged 1h 40m before point T, so this might not be the smoking gun we're looking for.

Can this situation be explained in any way?

• An obvious answer is to say that some DAG executed UPDATE query on all rows of task_instance table. To the best of our knowledge, this wasn't the case.
• Another theory is that this is indeed related to clear command. Technically, this command only affect the specific {dagrun, task} pair but maybe a bug can cause a big spill somehow?

Has anything like this been observed before?

[potiuk]

Hmm.. Maybe it's this one #32684

[potiuk]

But that one would delete task instances - so I am not sure if that's the case?