`ServerDisconnectedError` when querying tables created after PostgREST startup since v13+

[bolinocroustibat]

Description

Since upgrading from PostgREST v12.2.3 to v13.0.8 and v14.0.0, we observe a reproducible issue where some recently created tables cannot be queried through PostgREST.
Requests to these tables fail with a connection abort (ServerDisconnectedError / RemoteDisconnected).

This issue does not occur on v12.2.3.

Context

We run PostgREST as part of a public API gateway (tabular-api.data.gouv.fr) that dynamically exposes a large PostgreSQL database (tens of thousands of tables).
New tables are created regularly, while PostgREST continues to run.

Deployment: Nomad + Consul (Blue/Green, typically 2 instances).

Observed behavior

Starting with v13, queries to some recently created tables fail, while older tables continue to work normally.

From our observations, the problematic tables appear to be tables created after PostgREST has started.

The error is typically:
requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

Notes:

• The issue affects only a subset of newly created tables.
• Tables that already existed at startup behave normally.
• Restarting PostgREST makes the problematic tables work again — until new tables are created.

This behavior is not present in v12.2.3.

What we’ve tried

• Restarting PostgREST: temporarily fixes the issue.
• Purging the number of tables (120k → 55k): no effect.
• Reverting to v12.2.3: issue disappears entirely.
• Upgrading to v14.0.0: same behavior as v13.

Environment

• PostgreSQL: 15.8 (Debian package)
• PostgREST versions tested:
  • ✅ 12.2.3 — works correctly
  • ❌ 13.0.8 — new tables trigger ServerDisconnectedError
  • ❌ 14.0.0 — same issue
• Deployment: Nomad + Consul
• Schema size: ~55K tables (dynamic workload)
• Requests: hundreds/minute (read-only)

Notes

This suggests a regression introduced in v13 regarding table discovery or schema reloading?

A separate issue we opened describes a continuous memory growth / OOM regression that also appears starting in v13.
We are not sure whether the two problems share a root cause, but both regressions disappear when reverting to v12.

Happy to provide logs if needed.

[steve-chavez]

ServerDisconnectedError / RemoteDisconnected)
The error is typically:
requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

Those errors look like they're coming from a wrapper library so unfortunately they don't help us debug. Could you share the plain HTTP errors? (HTTP status, body)

[steve-chavez]

One thing that did change in v13 that could be related is that now we look for a table in our schema cache before making the query to the db: #3869.

[wolfgangwalther]

It would also be interesting whether you can find any errors in your PostgreSQL server log related to this.

PostgreSQL: 15.8 (Debian package)

While at it, you should seriously update this to the latest minor release for the 15-series, which is 15.15 - 15.8 is more than a year old.

[bolinocroustibat]

While at it, you should seriously update this to the latest minor release for the 15-series, which is 15.15 - 15.8 is more than a year old.

We’re running Debian Bookworm and currently use the PostgreSQL version from the default Debian repositories, which is 15.8 (Debian package).
Debian stable tends to lag behind upstream minor releases and mainly backports security/stability fixes instead of tracking every upstream 15.x point release.

[wolfgangwalther]

We’re running Debian Bookworm and currently use the PostgreSQL version from the default Debian repositories, which is 15.8 (Debian package).

That looks very different to me: https://packages.debian.org/bookworm/postgresql-15

Package: postgresql-15 (15.14-0+deb12u1)

[bolinocroustibat]

That looks very different to me: https://packages.debian.org/bookworm/postgresql-15

This page is about the software package, not the PostgreSQL cluster created on initial install/upgrade of the OS. The PostgreSQL cluster was created when PostgreSQL was in version 15.8 and Debian doesn't modify the cluster version number internally, even though it applies security patches (the OS is regularly updated).
Anyway we doubt the issue here could be related to PostgreSQL version but we could always be wrong :)

EDIT: After double-checking on the server, that’s not accurate in our case: we’re actually using PostgreSQL from the official PostgreSQL APT repository (PGDG), not the stock Debian postgresql-15 package. So ater checking you’re absolutely right that this is behind the latest 15.x minor (15.15). We’ll plan an upgrade of the PGDG packages to the latest 15.x but in our case it's not trivial

[bolinocroustibat]

Those errors look like they're coming from a wrapper library so unfortunately they don't help us debug. Could you share the plain HTTP errors? (HTTP status, body)

On our Load Balancers (HAProxy) logs, after extracting we see repeated 500s on a recently created table/resource (fff0cc27-977b-40d5-9c11-f7e4e79a0b72), with very long backend times (Tr ≈ 90–110s) but a normal termination state (----):

2025-11-10T00:13:12.248+01:00 ... WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/www-tabular-api5 0/0/3/98721/+98724 500 +2561 - - ---- 724/394/3/1/0 0/0 "GET /api/resources/fff0cc27-977b-40d5-9c11-f7e4e79a0b72/data/?page_size=200&page=1&SIRET__contains=41254894300056&Date_inspection__sort=desc HTTP/1.1"
2025-11-10T00:14:01.720+01:00 ... WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/www-tabular-api2 0/0/10/106361/+106371 500 +2561 - - ---- 925/529/2/0/0 0/0 "GET /api/resources/fff0cc27-977b-40d5-9c11-f7e4e79a0b72/data/?page_size=200&page=1&SIRET__contains=84158649800013&Date_inspection__sort=desc HTTP/1.1"
...

These are “regular” HTTP 500 responses (HAProxy sees ---- as termination state), they are not RemoteDisconnected but show the same new table/resource is problematic.

When the Python client sees:

requests.exceptions.ConnectionError:
  ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

HAProxy logs show the backend closing the connection prematurely, with status -1 or 503 and a non-normal termination state (CC--, CR--, SC--):

haproxy.log.4.gz:2025-11-10T08:24:05.088517+01:00 haproxy[782407]: xxx.xxx.xxx.xxx:33462 [10/Nov/2025:08:24:05.085] WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/www-tabular-api1 0/0/-1/-1/+2 -1 +0 - - CC-- 1560/870/0/0/0 0/0 "GET https://tabular-api.data.gouv.fr/api/resources/c5d02b42-1008-4406-83f5-3a81c8b936a3/data/?SIREN%20organisme%20d%C3%A9signant__exact=835195405 HTTP/2.0"
...
haproxy.log.4.gz:2025-11-10T08:40:04.730804+01:00 haproxy[782407]: xxx.xxx.xxx.xxx:16856 [10/Nov/2025:08:40:04.730] WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/<NOSRV> 0/-1/-1/-1/+0 -1 +0 - - CR-- 1846/1027/0/0/0 0/0 "GET https://tabular-api.data.gouv.fr/api/resources/c45b6dee-e948-41f2-8351-43c17092abf4/data/?page_size=20&page=1 HTTP/2.0"
...
WWW-TABULAR-API-DATAGOUVFR/<NOSRV> 0/-1/-1/-1/+0 503 +106 - - SC-- 2552/1421/0/0/0 0/0 "GET /api/resources/c13d05e5-9e55-4d03-bf7e-042a2ade7e49/data/?page_size=1&DATE_DEBUT__sort=desc&P_SOUSCRITE__exact=6 HTTP/1.1"
...
haproxy.log.4.gz:2025-11-10T09:39:08.167461+01:00 haproxy[782407]: xxx.xxx.xxx.xxx:41282 [10/Nov/2025:09:39:08.165] WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/<NOSRV> 0/-1/-1/-1/+0 503 +106 - - SC-- 2552/1421/0/0/0 0/0 "GET /api/resources/c13d05e5-9e55-4d03-bf7e-042a2ade7e49/data/?page_size=1&DATE_DEBUT__sort=desc&P_SOUSCRITE__exact=6 HTTP/1.1"
...
haproxy.log.4.gz:2025-11-10T09:39:08.596367+01:00 haproxy[782407]: xxx.xxx.xxx.xxx:50638 [10/Nov/2025:09:39:08.595] WILDCARD~ WWW-TABULAR-API-DATAGOUVFR/<NOSRV> 0/-1/-1/-1/+0 503 +106 - - SC-- 2543/1419/0/0/0 0/0 "GET https://tabular-api.data.gouv.fr/api/resources/c5d02b42-1008-4406-83f5-3a81c8b936a3/data/?SIREN%20organisme%20d%C3%A9signant__exact=870801958 HTTP/2.0"

• -1 status + CC-- / CR-- → server (PostgREST) closed the backend connection before sending a valid HTTP response.
• 503 + SC-- / → HAProxy aborted the backend connection or had no available backend to route to
• In all these cases, HAProxy does not see a full HTTP response from the backend, which matches the client-side RemoteDisconnected('Remote end closed connection without response').

[maudetes]

Here are some more details on the issues when querying newly created tables.

Context

At the moment, we don't have a postgREST ^v13 deployed in production, we don't have detailed HTTP logs.
In order to debug this issue, we deployed the v13.0.8 on our preprod with similar specs and context, except for
Schema size: ~11K tables
Requests: 1/minute (read-only)

Observations

We don't reproduce the ServerDisconnectedError in the context of our preprod env.
However, we do make interesting observations with the following steps:

1. postgREST v13.0.8 running
2. new (small) table created in the targeted schema
3. querying this table from PostgREST returns a 404 :

>>> requests.get(f"http://{postgrest}/64affa3d080f3180c60b7be3fb97c8b1?limit=20&order=__id.asc")
send: b'GET /64affa3d080f3180c60b7be3fb97c8b1?limit=20&order=__id.asc HTTP/1.1\r\nHost: X.X.X.X:3000\r\nUser-Agent: python-requests/2.32.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Fri, 14 Nov 2025 10:13:47 GMT
header: Server: postgrest/13.0.8
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 149
header: Proxy-Status: PostgREST; error=PGRST205
<Response [404]>

4. the table does exist and running select count(*) from "64affa3d080f3180c60b7be3fb97c8b1"; in psql returns correctly.

Notes

• RAM increase when running the query for the first time and is not released after
• the query takes up to ~4/5secs (vs less than 100ms for a found table)
• if/when we restart postgREST, the table is now available and returned correctly when querying postgREST

[mkleczek]

@maudetes do you reload the schema cache after adding the table? 404 would suggest it is missing from the schema cache.

[bolinocroustibat]

See here for update