Unable to Create Work Pool

[JTunis]

Not sure if this is specifically related to prefect-helm, but because of the custom setup I'm going to assume we probably have something misconfigured here. We're deploying prefect-server via helm and unable to create work pools (either created by helm, UI, or CLI inside or outside of the server pod).

I see the appropriate queries make it to the prefect-server DB, but they end up hanging.

In pg_stat_activity I see the following query remains stuck in idle in transaction and never finishes:

UPDATE work_pool SET default_queue_id=$1::UUID, updated=now() WHERE [work_pool.id](http://work_pool.id/) = $2::UUID RETURNING work_pool.updated

via this query

SELECT pid, usename, application_name, state,
       wait_event_type, wait_event,
       now() - xact_start AS xact_age,
       query
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY xact_start
LIMIT 50;

Ultimately, this results in timeouts from the API.

The db is an Aurora Postgres DB running PG 17.4.

Here are my server chart values (including various db tuning configs that I've tried):

global:
  prefect:
    image:
      repository: prefecthq/prefect
      prefectTag: 3.4.14-python3.13
      pullPolicy: IfNotPresent

server:
  replicaCount: 1

  resources:
    requests:
      cpu: "1"
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi

  env:
    - name: PREFECT_LOGGING_LEVEL
      value: "DEBUG"
    - name: PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT
      value: "60"
    - name: PREFECT_SERVER_DATABASE_TIMEOUT
      value: "120"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE
      value: "0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE
      value: "0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE
      value: "1800"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
      value: "60"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
      value: "15"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
      value: "15"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME
      value: "prefect-server"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_PRE_PING
      value: "true"

  uiConfig:
    prefectUiApiUrl: "http://${base_prefect_server_url}:4200/api"

backgroundServices:
  runAsSeparateDeployment: true

  env:
    - name: PREFECT_LOGGING_LEVEL
      value: "DEBUG"
    - name: PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT
      value: "60"
    - name: PREFECT_SERVER_DATABASE_TIMEOUT
      value: "120"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE
      value: "0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE
      value: "0"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE
      value: "1800"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
      value: "60"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
      value: "15"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
      value: "15"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME
      value: "prefect-server"
    - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_PRE_PING
      value: "true"

  resources:
    requests:
      cpu: "1"
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi

  serviceAccount:
    create: true
    name: "prefect-server-background-services"

  messaging:
    redis:
      host: "${redis_host}"
      port: 6379
      db: 0
      username: "${redis_username}"
      password: "${redis_password}"
      ssl: false

serviceAccount:
  create: true
  name: "prefect-server"

service:
  type: LoadBalancer
  annotations: {
    "service.beta.kubernetes.io/aws-load-balancer-type": "nlb",
    "service.beta.kubernetes.io/aws-load-balancer-internal": "true"
  }

ingress:
  enabled: false

secret:
  create: false
  name: "${postgres_secret_name}"

postgresql:
  enabled: false

redis:
  enabled: false

Chart version: 2025.8.21160848
Prefect version: 3.4.14

Let me know if there is anything else I can do to help debug.

[mitchnielsen]

Thanks for the detailed report @JTunis, we can try to dig into this a bit and see what's going on.

Will also cc backend folks who might have more context here and/or know what else to ask for to help debug: @zzstoatzz @desertaxle

[bdalpe]

May be related to #530

[bdalpe]

@JTunis I think what's happening is the background server and API server are attempting to do database migrations at the same time and getting deadlocked.

The query in question can be found in the schema migration scripts: https://github.com/PrefectHQ/prefect/blob/28d8999852ced8f119016467505890d10f42602c/src/prefect/server/database/_migrations/versions/postgresql/2023_01_31_110543_f98ae6d8e2cc_work_queue_data_migration.py#L77-L78

Can you try one of two things?

Option 1

Upgrade to chart version 2025.9.5190948 which includes the relevant environment variable set. diff: 2025.8.21160848...2025.9.5190948

Option 2

Set this in your values file:

backgroundServices:
  env:
    - name: PREFECT_API_DATABASE_MIGRATE_ON_START
      value: "false"

This results in only the API server running migrations. Ensure you restart all Pods.

[JTunis]

@bdalpe Hm, I tried upgrading the chart version to 2025.9.12160156 and saw the same issue. I then added PREFECT_API_DATABASE_MIGRATE_ON_START=false to the background-services which also didn't solve the issue.

I also tried adding PREFECT_API_DATABASE_MIGRATE_ON_START=false to both the server and background-services, ran the db migrations manually successfully, but still failing to creating a work pool with the same transaction hanging

[bdalpe]

@JTunis thanks for the update. We'll keep looking into this.