Continuous memory growth and OOM kills since PostgREST v13+

[bolinocroustibat]

Description

After upgrading from PostgREST v12.2.3 → v13.0.8 and v14.0.0, we observe a reproducible memory issue in production:
RAM usage continuously increases over time until the process is killed by the OOM killer.
This regression does not occur with v12, where memory usage remains stable even under heavy load.

Context

We run PostgREST as part of a public API gateway (tabular-api.data.gouv.fr), which dynamically exposes a large PostgreSQL database (tens of thousands of tables).
Each dataset in the DB corresponds to one physical table, so the total number of tables can vary — currently around 55K, but previously up to 120K.

PostgREST is deployed via Nomad/Consul (Blue/Green deployments, 2 instances, 6–8 GB RAM each).

Observed behavior

• With v13+, memory usage starts around 25–50% at startup and grows steadily until the process reaches 100% and is OOM-killed (every 1–2 hours under load).
• With v12.2.3, memory usage may start higher, but remains stable over time under identical load and schema size.
• Reducing the number of tables (from 120K → 55K) does not change the behavior.
• Restarting PostgREST temporarily resets memory, but the same growth pattern repeats.

Positive change since v14

Startup appears very significantly smoother and faster in v14+ — lower initial RAM consumption and faster readiness.
However, this comes at the cost of continuous memory growth over time at least for v13+, which was not observed in previous releases.

What we’ve tried

• Increasing PostgREST memory allocation (6 GB → 8 GB): no effect, still OOM after a few hours.
• Purging half of the tables: no effect.
• Reverting to v12.2.3: memory stable again, no OOM.

Environment

• PostgreSQL: 15.8 (Debian package)
• PostgREST versions tested:
  • ✅ 12.2.3 — stable memory behavior
  • ❌ 13.0.8 — continuous memory growth → OOM
  • ❌ 14.0.0 — same issue
• Deployment: Nomad + Consul (Blue/Green, 2 instances, 6–8 GB RAM each)
• Approx. tables: ~55,000
• Approx. requests: hundreds/minute (read-only)

Expected behavior

Memory usage should stay stable over time after startup, as in previous versions.

Actual behavior

Memory usage keeps increasing until the process is terminated by the system OOM killer.

Notes

This may be related to caching or schema metadata management changes introduced in v13+?

EDIT: Separately, we reported an issue where queries to some recently created tables fail with ServerDisconnectedError starting in v13.
We are not sure whether both problems share a common cause, but both regressions vanish when reverting to v12.

Happy to provide detailed metrics or logs if needed.

[steve-chavez]

@bolinocroustibat Could you share your configuration? (redacted jwt-secret and db-uri)

I'd like to know if you're using the JWT cache introduced in https://github.com/PostgREST/postgrest/releases/tag/v12.0.0.

[bolinocroustibat]

@steve-chavez Thank you for your very quick answer!
We don't use JWT tokens.
Here is an example of the DB URI we use: PGRST_DB_URI="postgres://<user>:<password>@<ip>:5432/<db-name>"

[steve-chavez]

@bolinocroustibat Could you share the log traces when starting up the postgREST v14 version on your database?

It should show something like:

06/May/2024:08:16:11 -0500: Starting PostgREST 12.1...
06/May/2024:08:16:11 -0500: Successfully connected to PostgreSQL 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
06/May/2024:08:16:11 -0500: Connection Pool initialized with a maximum size of 10 connections
06/May/2024:08:16:11 -0500: API server listening on port 3000
06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel
06/May/2024:08:16:11 -0500: Config reloaded
06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds
06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 Functions, 0 Domain Representations, 4 Media Type Handlers
06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel
06/May/2024:14:11:27 -0500: Config reloaded

[bolinocroustibat]

@steve-chavez Sure. Here is a log of a start on freshly deployed v14 before we started to run into issues:

28/Oct/2025:10:32:04 +0000: Starting PostgREST 14.0...
28/Oct/2025:10:32:04 +0000: API server listening on 0.0.0.0:3000
28/Oct/2025:10:32:04 +0000: Listening for database notifications on the "pgrst" channel
28/Oct/2025:10:32:04 +0000: Successfully connected to PostgreSQL 15.8 (Debian 15.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
28/Oct/2025:10:32:04 +0000: Connection Pool initialized with a maximum size of 10 connections
28/Oct/2025:10:32:04 +0000: Config reloaded
28/Oct/2025:10:32:11 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:21 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:31 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:42 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:47 +0000: Schema cache queried in 9074.3 milliseconds
28/Oct/2025:10:32:47 +0000: Schema cache loaded 125236 Relations, 0 Relationships, 0 Functions, 0 Domain Representations, 4 Media Type Handlers, 1196 Timezones
28/Oct/2025:10:32:47 +0000: Schema cache loaded in 243.1 milliseconds

[wolfgangwalther]

Something is surely not right with logging the schema cache load time, right? The log says this is roughly 40 seconds, but the numbers are not that. I think we saw that elsewhere already.

[steve-chavez]

The logs here:

28/Oct/2025:10:32:11 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:21 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:31 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:42 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}

Suggest that the scache queries were failing at around 10 seconds and being retried. Then:

28/Oct/2025:10:32:47 +0000: Schema cache queried in 9074.3 milliseconds

This succeeds in around ~9 seconds. @bolinocroustibat Do you have statement_timeout=10 (or another timeout) defined for the role which you connect postgREST to?

28/Oct/2025:10:32:47 +0000: Schema cache loaded 125236 Relations,

125K tables is definitely big, I'm lead to believe the problem is related to #3869

---

The log says this is roughly 40 seconds, but the numbers are not that. I think we saw that elsewhere already.

Perhaps you're remembering this one? #3046 (comment)

Not sure if it's the same case, but it's weird that these log lines:

28/Oct/2025:10:32:42 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:47 +0000: Schema cache queried in 9074.3 milliseconds

Have 5 seconds in difference according to the timestamp when the query finished in ~9 seconds. There's definitely something off in the time calculation.

[laurenceisla]

The logs here:

28/Oct/2025:10:32:11 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:21 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:31 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}
28/Oct/2025:10:32:42 +0000: {"code":"PGRST002","details":null,"hint":null,"message":"Could not query the database for the schema cache. Retrying."}

Suggest that the scache queries were failing at around 10 seconds and being retried.

This was a fix done in #4367, this should be triggered by a request, however the 503 HTTP error is not logged... @bolinocroustibat perhaps the log-level = "crit"? If it is, I'd suggest changing it to "error" to debug better. if it's not then there's another issue here.

Although the message says "retrying", I'm not sure if it does, it should respond the error and nothing else (the double schema load on startup was also fixed in the same PR).

• With v13+, memory usage starts around 25–50% at startup and grows steadily until the process reaches 100% and is OOM-killed (every 1–2 hours under load).

Do you see a pattern in every step of the memory increase? From the graphic I see it jumps at certain moments, for example could be after a schema restart or something similar?

[steve-chavez]

125K tables is definitely big, I'm lead to believe the problem is related to #3869

Made a revert on #4468, @bolinocroustibat if possible use the binary at the bottom of the page of https://github.com/PostgREST/postgrest/actions/runs/19379695783?pr=4468 and check if the OOM persists.