Why the primary key of dag_run from metadata database use int instead of bigint?

[houyingkun]

Our use case is to trigger 70000 dag runs per minute, each run is for a specific customer's data extract from the authorized third party platform.
We've calculated that the type of int would use up in about one month. ( 4294967295 / 60 / 24 / 365 / 70000 ≈ 0.117 )

Is there any advice for our use case? Should we modify the column data type from int to bigint directly?

[turbaszek]

@houyingkun I think having a custom migration would be a good idea. But if I were you I would open a PR with this change so others don't face this issue in future.

[XD-DENG]

I'm not sure if this is a generic use case. In Postgres, integer supports up to 2147483647 (which may be more than enough for most users). Of course bigint supports much bigger limit, 9223372036854775807, but the cost is doubling the storage size.

Reference: https://www.postgresql.org/docs/current/datatype-numeric.html

[houyingkun]

Our use case is:

Step1:
TriggerDag:get all need data extract seller id.

Step2:
Triggered Dag:
    T1: get single seller info from the business database by seller id.
    T2: Use the seller info to request orders from the third-party platform.
    T3: Request all order items by order id from T2.
    T4: Combine data from T2 and T3 and load data to the business database.

Because our customers are sensitive to data in real-time, we decided to set the interval time as one minute.
I am not sure the design is suitable for airflow, could you give me some advice?

[XD-DENG]

Hi @houyingkun , one question out of curiosity, which executor (CeleryExecutor or KubernetesExecutor) does your team use to serve 70000 DagRuns per minute? And how long by average does each DagRun take?

• If each of your DagRun/Job is not that small, I personally haven't seen such a scale for Airflow
• If each of your DagRun/Job is very samll/granular, the overheads brought by the scheduling may make the whole thing too expensive, and you may want to sort of merge your jobs to make them less granular.

@turbaszek please feel free to add or correct me.

[turbaszek]

@XD-DENG while I agree with your doubts it's not the first time we see users using Airflow at such scale. So I assume we should be prepare to handle such cases. A PR with migration should be easy to cut - @houyingkun will need it anyway - and it would be a better place for discussion.

[XD-DENG]

I am definitely open to see a PR raised to change the column type from integer to bigint. I may not be a big fan of it, but I would like to see how other folks think.