fix(db): self-heal DATABASE_URL secret after Aurora password rotation

[elamaran11]

Problem

Aurora's RDS-managed master user secret rotates automatically (7-day default). The in-cluster accelbench-db secret, however, is a static snapshot taken at terraform apply time — nothing re-syncs it after a rotation.

Result: after each rotation the API can no longer authenticate and crashloops with:

failed SASL auth: FATAL: password authentication failed for user "accelbench" (SQLSTATE 28P01)

In the UI this shows up as "Login service unavailable" — the web frontend stays up, but the accelbench-api backend is down, so the login endpoint is unreachable. It recurs roughly weekly and today required a manual secret patch + API restart to recover.

The README's documented fallback ("run terraform apply to refresh the secret") does not currently work either — see Known limitation below.

Fix

A self-contained hourly CronJob that keeps the secret in sync — no new cluster operators (no External Secrets Operator / CSI driver):

1. Reads the Aurora RDS-managed master secret (via the existing accelbench-api Pod Identity).
2. Rebuilds DATABASE_URL, preserving the host/port/db path from the current URL and swapping only the credentials (URL-encoded with jq @uri). This avoids depending on database.host, which is unset in the existingSecret deployment path.
3. Only when the value changed, patches the accelbench-db secret and kubectl rollout restarts the API. Otherwise it's a clean no-op — the API is never restarted unnecessarily.

Changes

File	What
helm/accelbench/templates/cronjob-db-rotation-sync.yaml	Hourly CronJob (0 * * * *), concurrencyPolicy: Forbid, reuses the accelbench-api ServiceAccount.
helm/accelbench/templates/db-rotation-sync-rbac.yaml	Least-privilege Role/RoleBinding — get/patch scoped by resourceNames to only the accelbench-db secret and the accelbench-api deployment, namespaced.
helm/accelbench/values.yaml	New dbRotationSync block: enabled, schedule, masterSecretArn (required when enabled), image, resources.
docker/Dockerfile.db-sync	aws-cli v2 (AL2 base) + kubectl v1.31.4 + jq. Published as accelbench-tools:db-sync — a distinct tag so it never collides with the API's TOOLS_IMAGE (accelbench-tools:latest).
terraform/main.tf	aws_iam_role_policy.api_db_master_secret_read — grants the API role secretsmanager:GetSecretValue/DescribeSecret on only the master secret ARN.

Deployment notes

• Set dbRotationSync.masterSecretArn to the Aurora RDS-managed master secret ARN (Terraform output aurora_master_user_secret[0].secret_arn).
• Build/push the image: podman build --platform linux/amd64 -f docker/Dockerfile.db-sync -t <registry>/accelbench-tools:db-sync . && podman push ....
  • Build gotcha: do not execute the amd64 kubectl/aws binaries during build on Apple Silicon — they crash under QEMU emulation. The Dockerfile only downloads them; verification happens at runtime on real amd64 nodes.

Validation

Deployed to the live accelbench-eks cluster and exercised end-to-end:

• Run 1 (secret stale): job read the master secret, patched accelbench-db, and restarted the API → API recovered, 2/2 Running, no 28P01.
• Run 2 (secret current): DATABASE_URL already current — no-op — confirms idempotency; the API is not restarted when nothing changed.
• Public endpoints: /login → 200, /api/v1/status → {"error":"missing_token"} (expected — auth on).

Known limitation

terraform plan/apply currently fails repo-wide with an unrelated, pre-existing error in terraform/modules/karpenter/main.tf (Unsupported \moved` across resource types—gavinbunney/kubectl1.19.0 doesn't supportmoved). Because of this, the Terraform IAM policy in this PR was applied out-of-band via aws iam put-role-policyon the live cluster; the committedterraform/main.tf` change is the IaC record and will reconcile once that karpenter blocker is fixed (worth a separate PR).