v2 postgres: update idcreate runbooks for PostgreSQL + provisioning adapter

- idcreate-vrsctest-deploy.md: add postgres container, provisioning HTTP adapter,
  4-container stack (postgres + provisioning + api + worker), PostgreSQL backup cmd,
  remove 'database is locked' troubleshooting entry (v1 only)
- idcreate-vrsctest-update.md: update for postgres health wait, container list updated

Author: dream-hermes-agent

runbooks/idcreate-vrsctest-deploy.md

@@ -16,6 +16,8 @@ parent: Deployment
 **Service IP:** `10.200.0.14`
 **URL:** `https://idcreate.vrsctest.buildwithdreams.com`
 
+> **v2 change (2026-05-15):** PostgreSQL replaces SQLite as the data store. A `provisioning` container runs the HTTP provisioning adapter. Total containers: 4 (`postgres`, `provisioning`, `api`, `worker`).
+
 ---
 
 ## Prerequisites
@@ -43,7 +45,7 @@ VRSC mainnet and VRSCTEST use identical octets on their respective networks —
 
 ---
 
-## Architecture
+## Architecture (v2)
 
 ```
 Internet
@@ -60,19 +62,34 @@ Caddy (mains_blue_caddy-caddy-1)
   idcreate-api container (dev200_idcreate-api-1)
     at 10.200.0.14:5003 (net-vrsctest only)
         │
-        ▼ (background)
-  idcreate-worker container (dev200_idcreate-worker-1)
-    polls pending registrations, advances state
+        ├──────────────────────────────┐
+        ▼                              ▼
+  idcreate-worker              idcreate-provisioning
+  (background state machine)   (Node.js HTTP adapter)
+        │                              │
+        ▼                              ▼
+  PostgreSQL 16-alpine               svc-provisioning scripts
+  dev200_idcreate-postgres-1         (provisioning/engine.py)
+  5432 (host: 127.0.0.1:5432)
         │
         ▼
-  VRSCTEST verusd daemon at 10.200.0.11:18842
+  VRSCTEST verusd daemon at 10.200.0.11:18843
 ```
 
-> **Caddy is dual-homed.** It is already connected to `net-vrsctest` from the qrcodes deployment (playbook `37`). No network changes needed for this deployment.
+---
+
+## Containers (v2)
+
+| Container | Service | IP | Port |
+|---|---|---|---|
+| `dev200_idcreate-postgres-1` | PostgreSQL 16-alpine | auto | `5432` (host: `127.0.0.1:5432`) |
+| `dev200_idcreate-provisioning-1` | Node.js provisioning scripts | auto | `5055` (host: `127.0.0.1:5055`) |
+| `dev200_idcreate-api-1` | FastAPI | `10.200.0.14` | `5003` (host: `127.0.0.1:5003`) |
+| `dev200_idcreate-worker-1` | Polling worker | auto | — (internal) |
 
 ---
 
-## Environment Variables Written by Playbook 41
+## Environment Variables Set by Playbook 41
 
 Playbook `41-idcreate-deploy.yml` reads VRSCTEST RPC credentials from `~/docker-verusd/vrsctest/data_dir/vrsctest.conf` and writes these to `~/svc-idcreate/.env`:
 
@@ -86,18 +103,18 @@ verusd_vrsc_rpc_port=18843
 verusd_vrsc_rpc_host=10.200.0.11
 ```
 
-> **SFConstants mapping:** The VRSCTEST chain hijacks the `DAEMON_VERUSD_VRSC` slot in `SFConstants.py` via `NATIVE_COIN="VRSCTEST"`. There is no separate VRSCTEST daemon slot.
-
----
-
-## Containers
+**New in v2 (added by playbook 41):**
 
-| Container | Service | IP | Port |
-|---|---|---|---|
-| `dev200_idcreate-api-1` | FastAPI | `10.200.0.14` | `5003` (host: `127.0.0.1:5003`) |
-| `dev200_idcreate-worker-1` | Polling worker | `10.200.0.14` | — (internal) |
+```
+DATABASE_URL=postgresql://idcreate:<auto-generated-password>@postgres:5432/idcreate
+PROVISIONING_ADAPTER_MODE=http
+PROVISIONING_SERVICE_URL=http://127.0.0.1:5055
+PROVISIONING_HTTP_TIMEOUT_SECONDS=10
+PROVISIONING_RETRY_COUNT=1
+PROVISIONING_LOG_LEVEL=INFO
+```
 
-No provisioning service — not ready yet. Focus is on the id creation base Python service.
+> **SFConstants mapping:** The VRSCTEST chain hijacks the `DAEMON_VERUSD_VRSC` slot in `SFConstants.py` via `NATIVE_COIN="VRSCTEST"`. There is no separate VRSCTEST daemon slot.
 
 ---
 
@@ -135,15 +152,17 @@ Delegate: ansible-playbook -i inventory.ini playbooks/41-idcreate-deploy.yml
 
 This playbook:
 1. Reads VRSCTEST RPC credentials from `~/docker-verusd/vrsctest/data_dir/vrsctest.conf`
-2. Writes `~/svc-idcreate/.env` with all required vars (RPC, NATIVE_COIN, etc.)
-3. Removes any existing containers (`dev200_idcreate-api-1`, `dev200_idcreate-worker-1`)
-4. Writes `docker-compose.yml` targeting `net-vrsctest` at IP `10.200.0.14`
-5. Runs `docker compose up -d --force-recreate`
-6. Waits for api container to appear in `docker ps`
+2. Writes `~/svc-idcreate/.env` with all required vars (RPC, NATIVE_COIN, DATABASE_URL, provisioning adapter, etc.)
+3. Auto-generates a strong PostgreSQL password and writes it to `DATABASE_URL`
+4. Removes any existing containers
+5. Writes `docker-compose.yml` with 4 services: postgres + provisioning + api + worker
+6. Runs `docker compose up -d --force-recreate`
+7. Waits for postgres container to become healthy (`pg_isready`)
+8. Waits for api container to appear in `docker ps`
 
-> If `.env` already exists, step 1 (copy from env.sample) is skipped and RPC vars are updated in place — safe to re-run.
+> If `.env` already exists, step 1 (copy from env.sample) is skipped and all vars are updated in place — safe to re-run.
 
-### Step 3.5 — Issue API key
+### Step 3.5 — Issue API key (if not already set)
 
 ```
 Delegate: ansible-playbook -i inventory.ini playbooks/43-idcreate-add-api-key.yml
@@ -155,8 +174,8 @@ This playbook:
 3. Prints the key to the ansible output — **copy it now**, it will not be shown again
 
 > After running this, you **must restart the containers** for the new env var to take effect:
-> ```bash
-> ssh bwd "cd ~/svc-idcreate && docker compose -p dev200_idcreate restart"
+> ```
+> Delegate: ansible-playbook -i inventory.ini playbooks/47-idcreate-restart.yml
 > ```
 
 ### Step 4 — Add Caddy route
@@ -165,7 +184,7 @@ This playbook:
 Delegate: ansible-playbook -i inventory.ini playbooks/42-idcreate-caddy-route.yml
 ```
 
-This playbook:
+This playbook (unchanged in v2):
 1. **Pre-flight check** — verifies Caddy is connected to `net-vrsctest`; aborts if missing
 2. Appends `idcreate.vrsctest.buildwithdreams.com` route block to the host Caddyfile using `blockinfile`
 3. Copies the updated Caddyfile into the container (`docker cp` — needed because the container mount is read-only)
@@ -187,14 +206,34 @@ Expected: `HTTP/2 200` from the FastAPI service.
 
 To verify container health from the server:
 ```bash
-ssh bwd "docker ps | grep idcreate"
+ssh bwd "docker ps | grep dev200_idcreate"
 ssh bwd "curl -s http://10.200.0.14:5003/health"
 ```
 
+To verify PostgreSQL is running:
+```bash
+ssh bwd "docker ps | grep postgres"
+ssh bwd "docker exec dev200_idcreate-postgres-1 pg_isready -U idcreate -d idcreate"
+```
+
 To check logs:
 ```bash
 ssh bwd "docker logs dev200_idcreate-api-1 --tail 50"
 ssh bwd "docker logs dev200_idcreate-worker-1 --tail 50"
+ssh bwd "docker logs dev200_idcreate-postgres-1 --tail 20"
+```
+
+---
+
+## PostgreSQL Backup
+
+Compose stores DB data on volume `idcreate_postgres_data`.
+
+Recommended: periodic host-level volume snapshots, keep at least daily backups, keep longer retention for operational audits.
+
+Example backup command:
+```bash
+ssh bwd "cd ~/svc-idcreate && docker compose -p dev200_idcreate exec -T postgres pg_dump -U idcreate -d idcreate > idcreate_backup.sql"
 ```
 
 ---
@@ -206,14 +245,16 @@ ssh bwd "docker logs dev200_idcreate-worker-1 --tail 50"
 | `502 Bad Gateway` | Caddy not on `net-vrsctest` | Run `37-qrcodes-caddy-network.yml` (should already be done) |
 | `Connection refused` | idcreate container not running | Run `41-idcreate-deploy.yml`; check `docker logs dev200_idcreate-api-1` |
 | `500 Internal Server Error` | RPC connection failed | Verify VRSCTEST daemon running; check `docker logs dev200_idcreate-api-1` for RPC errors |
+| `database is locked` (v1 only) | SQLite concurrent write conflict | Upgrade to v2 — PostgreSQL handles concurrent access |
 | `curl: (6) Could not resolve host` | DNS not propagated | Wait 5-10 minutes for Let's Encrypt DNS propagation |
 | Container keeps restarting | Health check failing or bad env vars | `docker logs dev200_idcreate-api-1`; verify `.env` has correct RPC port and credentials |
-| Route not responding | Caddyfile stale | Run `42-idcreate-caddy-route.yml` again to reload |
+| PostgreSQL container not healthy | postgres not ready yet | Wait up to 150s; check `docker logs dev200_idcreate-postgres-1` |
 
 **Container restart loop:**
 ```bash
-ssh bwd "docker rm -f dev200_idcreate-api-1 dev200_idcreate-worker-1"
+ssh bwd "cd ~/svc-idcreate && docker compose -p dev200_idcreate down"
 # Then re-run Step 3
+Delegate: ansible-playbook -i inventory.ini playbooks/41-idcreate-deploy.yml
 ```
 
 **RPC connection refused — common cause:**
@@ -252,7 +293,7 @@ Delegate: ansible-playbook -i inventory.ini playbooks/42-idcreate-caddy-route.ym
 |---|---|---|
 | `39` | `39-idcreate-clone.yml` | Clone repo |
 | `40` | `40-idcreate-build.yml` | Build Docker image |
-| `41` | `41-idcreate-deploy.yml` | Deploy api+worker on `net-vrsctest` at `10.200.0.14`; writes `.env` |
+| `41` | `41-idcreate-deploy.yml` | Deploy full stack (postgres + provisioning + api + worker) |
 | `42` | `42-idcreate-caddy-route.yml` | Add HTTPS route in Caddy |
 | `43` | `43-idcreate-add-api-key.yml` | Generate and persist `REGISTRAR_API_KEYS` test key |
 | `08b` | `08b-start-vrsctest.yml` | Start VRSCTEST daemon (prereq) |
@@ -263,4 +304,5 @@ Delegate: ansible-playbook -i inventory.ini playbooks/42-idcreate-caddy-route.ym
 
 ## History
 
-- **2026-04-29** — Created. Mirrors the qrcodes VRSCTEST deployment pattern. Deploys api + worker only; provisioning service excluded (not ready). VRSCTEST uses the `DAEMON_VERUSD_VRSC` slot via `NATIVE_COIN="VRSCTEST"` — no separate VRSCTEST slot in `SFConstants.py`.
+- **2026-04-29** — Created. Mirrors the qrcodes VRSCTEST deployment pattern. Deploys api + worker only; provisioning service excluded (not ready). VRSCTEST uses the `DAEMON_VERUSD_VRSC` slot via `NATIVE_COIN="VRSCTEST"`.
+- **2026-05-15** — v2 postgres migration. Replaces SQLite with PostgreSQL 16-alpine. Adds provisioning adapter in HTTP mode (`PROVISIONING_ADAPTER_MODE=http`). Total containers increases from 2 to 4. `database is locked` errors are resolved.

runbooks/idcreate-vrsctest-update.md

@@ -16,6 +16,8 @@ parent: Deployment
 **Service IP:** `10.200.0.14`
 **URL:** `https://idcreate.vrsctest.buildwithdreams.com`
 
+> **v2 change (2026-05-15):** PostgreSQL container is now part of the stack. The update playbook waits for postgres to be healthy before declaring success. Total containers: 4.
+
 ---
 
 ## Prerequisites
@@ -41,7 +43,8 @@ This playbook (in order):
 4. **Stops** running containers
 5. **Rebuilds** `buildwithdreams/svc-idcreate:local` from latest code
 6. **Starts** containers with `docker compose up -d --force-recreate`
-7. **Verifies** api container appears in `docker ps`
+7. **Waits** for postgres container to become healthy (`pg_isready`)
+8. **Verifies** api container appears in `docker ps`
 
 > `.env` is restored unconditionally after every pull. This is required because git operations can remove or replace untracked files during merge/rebase.
 
@@ -64,6 +67,8 @@ ssh bwd "curl -s http://10.200.0.14:5003/health"
 ```bash
 ssh bwd "docker logs dev200_idcreate-api-1 --tail 30"
 ssh bwd "docker logs dev200_idcreate-worker-1 --tail 30"
+ssh bwd "docker logs dev200_idcreate-postgres-1 --tail 20"
+ssh bwd "docker logs dev200_idcreate-provisioning-1 --tail 30"
 ```
 
 ---
@@ -91,6 +96,7 @@ Common causes:
 | Symptom | Check | Fix |
 |---|---|---|
 | Container not starting | `docker logs dev200_idcreate-api-1` | Check for import errors or missing env vars |
+| PostgreSQL not healthy | `docker logs dev200_idcreate-postgres-1` | Check `pg_isready` output; disk space |
 | RPC unreachable | `docker logs dev200_idcreate-api-1` | Verify VRSCTEST daemon at `10.200.0.11:18843` is running |
 | .env missing | `ls ~/svc-idcreate/.env*` on server | `cp ~/.env.backup ~/.env` then restart |
 | Image missing | `docker images | grep svc-idcreate` | Run `40-idcreate-build.yml` first |
@@ -132,9 +138,11 @@ Delegate: ansible-playbook -i inventory.ini playbooks/45-idcreate-update.yml
 | `45` | `45-idcreate-update.yml` | Pull + rebuild + restart idcreate stack |
 | `40` | `40-idcreate-build.yml` | Build image (used by 45, rarely needed alone) |
 | `41` | `41-idcreate-deploy.yml` | Full deploy — writes `.env` fresh |
+| `47` | `47-idcreate-restart.yml` | Restart to pick up .env changes (includes postgres health wait) |
 
 ---
 
 ## History
 
 - **2026-04-30** — Created. Pull/rebuild/restart pattern for svc-idcreate on VRSCTEST. `.env` is backed up before pull and restored unconditionally after to survive git merge side-effects.
+- **2026-05-15** — v2 postgres migration. PostgreSQL 16-alpine added to stack. Update playbook now waits for `postgres` container to be healthy before completing. `database is locked` errors from v1 are resolved.