chore(infra): derive DAO API database memory settings from cgroup limit - #2099
Open
pikonha wants to merge 2 commits into
Open
chore(infra): derive DAO API database memory settings from cgroup limit#2099pikonha wants to merge 2 commits into
pikonha wants to merge 2 commits into
Conversation
All 14 DAO databases run the stock postgres-ssl image with no tuning, and every one of them carries a hand-applied `max_connections = 500` in postgresql.auto.conf — undocumented, invisible outside the volume, and applied across four different memory tiers (2/4/6/8 GB). 500 backends is roughly 2.5 GB of unreclaimable memory. On the 2 GB tier (nouns-db, shutter-db) that is an OOM the container cannot survive, and PostgreSQLHighConnections cannot catch it: it fires at 80% sustained for 10m, which the OOM beats. Wrap the stock image so the entrypoint reads /sys/fs/cgroup/memory.max and appends the derived values as -c flags. Command-line options outrank postgresql.auto.conf, so this neutralises the existing 500 without writing to any volume, and reverts by pointing a service back at the stock image. 2 GB -> max_connections=75 effective_cache_size=1.3 GB 4 GB -> max_connections=80 effective_cache_size=2.6 GB 6 GB -> max_connections=120 effective_cache_size=3.9 GB 8 GB -> max_connections=160 effective_cache_size=5.2 GB shared_buffers is deliberately left at its 128 MB default: raising it converts reclaimable page cache into unreclaimable shmem, which would spend safety margin on the two tiers that have the least of it. Separate change. Verify with: bash infra/dao-api-db/entrypoint.dao-api-db.sh --self-test Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pikonha
requested review from
LeonardoVieira1630,
PedroBinotto,
alextnetto and
brunod-e
as code owners
August 1, 2026 16:05
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
|
🚅 Deployed to the anticapture-pr-2099 environment in anticapture-infra
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 074b1e2c74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # The base image's ENTRYPOINT (wrapper.sh) sets up SSL certs and pgbackrest, so | ||
| # we don't replace it — entrypoint.dao-api-db.sh derives a couple of settings from | ||
| # the container's memory limit and then execs wrapper.sh with them appended. | ||
| COPY entrypoint.dao-api-db.sh /entrypoint.sh |
There was a problem hiding this comment.
With the repository-root build context selected by dao-api-db.railway.toml, this source resolves to /entrypoint.dao-api-db.sh, but a repo-wide search shows the file exists only under infra/dao-api-db/; the image build therefore fails before deployment. Docker's COPY documentation specifies that local sources are resolved relative to the root of the build context, so this must copy infra/dao-api-db/entrypoint.dao-api-db.sh (or explicitly change the build context).
Useful? React with 👍 / 👎.
The fleet is not on one major: production runs 17 Postgres services across 16 (x8), 17 (x4) and 18 (x5). A hardcoded :16 base would start against a newer PGDATA and exit with "database files are incompatible with server", crashlooping 9 of 17 databases. Take the major from a PG_MAJOR build arg, which Railway populates from the service's variables. No default on purpose: an unset PG_MAJOR fails the build and leaves the running container serving, whereas a wrong default would build cleanly and then crashloop. Booted and verified on all three majors at a 2 GB limit — each reports max_connections=75, effective_cache_size=1331MB, ssl=on. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Contributor
|
🔍 Vercel preview: https://anticapture-c9dodln4b-ful.vercel.app |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
All DAO databases run the stock
postgres-sslimage with zero tuning, and every one sampled carries a hand-appliedmax_connections = 500inpostgresql.auto.conf— undocumented, invisible outside the volume, applied across four different memory tiers.memory.maxauto.confmax_connections = 500max_connections = 500max_connections = 500max_connections = 500max_connections = 500max_connections = 500500 backends is ~2.5 GB of unreclaimable memory. On the 2 GB tier that is an OOM the container cannot survive.
PostgreSQLHighConnectionscannot catch it either — it fires at 80% sustainedfor: 10m, which the OOM beats.What
Wrap the stock image. The entrypoint reads
/sys/fs/cgroup/memory.maxand appends derived values as-cflags.max_connectionseffective_cache_sizeDerived rather than hardcoded because the fleet spans four tiers off one image, and a Railway plan change moves a service between tiers without touching the repo.
-cflags outrankpostgresql.auto.conf, so this neutralises the existing500without writing to any volume, and reverts by pointing a service back at the stock image.shared_buffersdeliberately stays at its 128 MB default — raising it converts reclaimable page cache into unreclaimable shmem, spending safety margin on the two tiers with the least of it. Separate change.PG_MAJORProduction runs 17 Postgres services across three majors: 16 (×8), 17 (×4), 18 (×5). The base image version comes from a
PG_MAJORbuild arg, which Railway populates from the service's variables.There is no default on purpose. An unset
PG_MAJORfails the build, leaving the running container serving. A wrong default would build cleanly and then crashloop ondatabase files are incompatible with server.Verified locally
--self-testpasses 8 cases: 4 fleet tiers, both clamp boundaries, 2 fallback pathsPGDATA; each reportsmax_connections=75,effective_cache_size=1331MB,ssl=onat a 2 GB limitPG_MAJORfails the build, as designed75) and 6 GiB (120)ssl = on— the entrypoint wrap does not break the base image's cert setupmax_connections = 500written into a realauto.confand restarted:SHOW max_connections→75,source→command lineNot in this PR
shared_buffersat 128 MB fleet-wideREINDEX DATABASEbeforeREFRESH COLLATION VERSIONpostgres-exporter'sDATA_SOURCE_NAMEholds plaintext superuser credentials for 14 production databases in one variable🤖 Generated with Claude Code