feat: add cached catalog read model

[knzeng-e]

Outcome

Dotify Home now loads its production catalog through one cacheable API request
instead of making browser RPC work grow with every artist, release, and royalty
recipient.

Artist SmartRuntimes remain authoritative. The new backend read model is a
performance projection of on-chain state, not a replacement source of truth.
Security-sensitive access decisions still read current runtime policy when a
listener opens a protected track.

Closes #86

Local scope and acceptance record:
docs/backlog/26-cached-catalog-read-model.md.

Issue and context

Before this PR, initial browsing performed a fan-out from the browser:

ArtistDirectory
  -> every artist/runtime
    -> every track hash
      -> every track record
        -> every royalty split
          -> startup access probes

The number of browser RPC calls therefore grew approximately with artists,
tracks, and split recipients. A larger sovereign catalog made the first useful
screen slower, even though browsing does not require a security decision for
every release.

That mixed two different responsibilities:

• Browsing projection: fast, cacheable, tolerant of bounded staleness.
• Access authority: current, direct, and fail-closed before protected
  playback or transactions.

This PR separates those responsibilities. That separation is the central
architectural idea to review.

Architecture and key concepts

flowchart LR
  D[ArtistDirectory events] --> G[Viem chain gateway]
  R[SmartRuntime music events] --> G
  G --> M[Catalog read model]
  M -->|atomic write| S[(JSON snapshot)]
  S --> M
  M --> A[Fastify catalog API]
  A -->|ETag + SWR + lag| C[Browser catalog service]
  C --> U[React catalog state]
  U -->|protected track intent| X[Direct access check / key service]
  X --> R

Loading

Read model

A read model is a query-optimized projection of authoritative data. Here,
SmartRuntimes own catalog truth; the backend stores the shape Home needs so
clients do not reconstruct it repeatedly.

This is a small CQRS-style separation: contract writes and security checks use
the authoritative model, while catalog queries use a projection.

Confirmed checkpoint

The indexer observes the chain head but indexes only:

safe head = chain head - CATALOG_CONFIRMATIONS

Each snapshot records the safe block number and hash. If that hash no longer
matches the canonical chain, the checkpoint crossed a reorg and the model
performs a deterministic full reindex.

Event indexing plus reconciliation

Events identify what changed cheaply, but the events do not contain every
catalog field. The indexer therefore:

1. consumes confirmed directory and track events;
2. re-reads changed release records and royalty splits;
3. periodically enumerates complete on-chain state to repair missed events or
   provider gaps.

This combines low steady-state cost with a deterministic recovery path.

Stale-while-revalidate

Cache-Control: public, max-age=30, stale-while-revalidate=120 lets a client or
edge cache serve known catalog content while refreshing it in the background.
The ETag is derived from catalog content, not every observed head movement, so
unchanged releases remain cacheable. Fresh state and lag are also exposed in
headers.

How it works

1. Startup and persistence

CatalogReadModel.start() loads the validated snapshot before starting RPC
polling. Snapshot writes use a temporary file and atomic rename, so readers
never observe a partially written JSON document.

The snapshot includes:

• artists and runtime addresses;
• active and inactive releases;
• access mode, price, personhood requirement, and media/metadata refs;
• cover variants and royalty summaries;
• registration/source blocks;
• indexed block/hash, observed head, and reconciliation timestamps.

2. Incremental synchronization

On each poll:

1. read chain head and calculate the confirmed safe head;
2. compare the stored checkpoint hash with the canonical block hash;
3. reindex on reorg, chain rewind, identity mismatch, reconciliation deadline,
   or directory change;
4. otherwise replay changed release events and re-read only affected records;
5. persist the next checkpoint only after the complete projection is built.

An event from an unknown runtime triggers reindex rather than advancing past
data the current projection cannot explain.

3. API behavior

The API exposes:

• GET /api/catalog
• GET /api/catalog/artists/:artistAddress
• GET /api/catalog/releases/:contentHash

Pagination cursors include the catalog revision. If data changes between pages,
the server returns STALE_CATALOG_CURSOR instead of silently mixing revisions.

Every catalog response describes one of these states:

State	Meaning	Data behavior
fresh	Current snapshot available	Serve it
stale-cache	Snapshot is older than the configured threshold	Serve it and disclose staleness
rpc-outage	Chain polling failed	Serve last snapshot when available
indexer-outage	Snapshot load/write failed	Serve in-memory snapshot when available
empty	Successful index found no releases	Return a truthful empty catalog

4. Browser behavior

When VITE_DOTIFY_API_URL is configured:

1. web/src/services/catalog.ts paints a validated browser-cached response;
2. one conditional API request refreshes it with If-None-Match;
3. useCatalog maps the API model into the existing UI domain model;
4. CatalogProvider does not preflight every protected release on startup;
5. selecting a protected release still performs the direct access check.

When the API is not configured, local/demo mode preserves direct chain
enumeration. A configured production API never silently falls back to the
browser fan-out.

Design decisions and tradeoffs

Atomic JSON before a database

The repository has no database dependency today. An atomic JSON snapshot gives
the first production read boundary with low operational and migration cost.

Tradeoff: it is deliberately single-writer and unsuitable for horizontal API
replicas. CATALOG_SNAPSHOT_PATH must be mounted on durable storage. Shared
transactional storage is a follow-up before horizontal scaling.

Events are hints; contract reads remain canonical

A pure event-sourced projection would be faster to replay but current events do
not carry descriptions, media refs, full metadata, or splits. Re-reading
changed records avoids inventing event-only truth and lets periodic
reconciliation repair drift.

Bounded-stale browsing, current access

Serving stale catalog metadata during an RPC outage is acceptable for
discovery. Releasing protected content from stale policy is not. The catalog
projection is never accepted as an authorization decision.

Revision-bound offset pagination

Offset pagination is sufficient for the production seed size and simple for
clients. Binding the offset to the content revision prevents cross-revision
page mixtures. Keyset pagination can replace it if catalog scale demands it.

Alternatives not chosen

• Keep browser enumeration: rejected because latency and RPC load grow with
  catalog size.
• Backend proxy without persistence: rejected because it remains cold and
  RPC-dependent on every request.
• Pure log indexer: rejected because current events do not contain complete
  release data and provider gaps need repair.
• Add PostgreSQL now: deferred because the repository has no database
  operational contract yet; the storage interface isolates that future change.

Security, failure, and operations

• SmartRuntimes remain the catalog and access source of truth.
• Catalog responses never grant playback access or release a content key.
• Protected-track intent still uses current direct access checks.
• Ambiguous release ownership remains an explicit error.
• Production does not silently fall back to demo signers, browser secrets, or
  startup RPC enumeration.
• Snapshot validation rejects malformed persisted data.
• Persistence failure is observable as indexer-outage; it does not erase the
  in-memory last-known projection.
• RPC failure is observable as rpc-outage; key delivery continues to fail
  closed through its existing path.
• Operators can force deterministic recovery with
  cd services/api && npm run catalog:reindex.
• Multiple processes must not write the same JSON snapshot.

New server settings are documented in services/api/.env.example and
docs/reference/environment-variables.md:

• CATALOG_SNAPSHOT_PATH
• CATALOG_POLL_INTERVAL_MS
• CATALOG_RECONCILE_INTERVAL_MS
• CATALOG_STALE_AFTER_MS
• CATALOG_CONFIRMATIONS

Review guide

Suggested order

1. Scope and invariants

   • Read docs/backlog/26-cached-catalog-read-model.md.
   • Confirm the delivered boundary matches [Performance][P0] Build a cached catalog read model #86 and that public p75 evidence is
     clearly separated from implementation evidence.

2. Domain contract

   • Read services/api/src/services/catalog/types.ts.
   • Check that persisted/API fields are sufficient for browsing without
     smuggling authorization decisions into the projection.

3. Chain adapter

   • Read services/api/src/services/catalog/chain.ts.
   • Compare minimal event/function ABIs with the generated/contract ABIs.
   • Verify access mode, personhood, native amount, block, and royalty mappings.

4. Consistency engine

   • Read services/api/src/services/catalog/readModel.ts, then
     snapshotStore.ts.
   • Trace full reindex, event replay, checkpoint movement, reorg detection, and
     persistence failure.

5. HTTP contract

   • Read services/api/src/routes/catalog.ts.
   • Check ETag/304 behavior, cache headers, cursor revision binding, detail
     ambiguity, and typed 4xx/5xx responses.

6. Browser cutover

   • Read web/src/services/catalog.ts, then the API branch in
     web/src/hooks/useCatalog.ts, then
     web/src/app/providers/CatalogProvider.tsx.
   • Confirm production performs one browsing request and only defers, never
     bypasses, security-sensitive access reads.

7. Recovery evidence

   • Read readModel.test.ts, catalog.test.ts, and the frontend catalog/status
     tests.
   • Match each failure state and replay invariant to a deterministic test.

8. Operational truth

   • Review env, README, spec, technical memory, backlog, and public page
     updates.
   • Confirm the single-writer boundary and remaining performance evidence are
     not overstated.

Verify carefully

• A checkpoint never advances past a change the model cannot understand.
• Reorg detection compares the persisted block hash before replay.
• Full reconciliation produces deterministic ordering and counts.
• Snapshot writes are atomic and validated.
• Persistence/RPC failures preserve usable last-known data without hiding
  the outage state.
• Event ABIs and tuple mappings exactly match deployed contracts.
• ETags and browser cache keys distinguish response variants.
• A stale pagination cursor cannot combine two catalog revisions.
• API mode never triggers startup runtime/royalty/access fan-out.
• Direct access checks still run before protected playback.
• Local/demo behavior remains available only when the API is unconfigured.
• No secret or access grant is introduced into catalog persistence.

Learning checkpoint

After reviewing, a maintainer should be able to explain:

1. why browsing and authorization need different consistency models;
2. how confirmed checkpoints and reconciliation recover from reorgs or missed
   events;
3. why content-derived ETags improve cacheability;
4. where JSON persistence stops being sufficient;
5. how to replace the snapshot store without changing the chain, route, or
   frontend contracts.

Validation

Evidence	What it proves
cd services/api && npm run build	Backend types and emitted build are valid
cd services/api && npm test (74 tests)	Routes, replay, reorg, persistence failure, auth, uploads, and existing API behavior
cd web && npm run test:unit (173 tests)	Catalog client/status mapping and existing frontend domain behavior
cd web && npm run build	Production TypeScript/Vite integration
cd web && npm run lint	0 errors; three pre-existing hook dependency warnings remain
node scripts/backlog-sync.mjs --check --offline	Local issue/doc/manifest mapping
node scripts/backlog-sync.mjs --check --live	Active issues remain visible in Project 5
PR CI and Netlify preview	EVM, web, formatting, backlog, review, and deploy-preview checks pass

Known limitations and follow-ups

• Public warm p75 < 250 ms, cold p75 < 800 ms, and useful-content
  < 800 ms still require deployment traces at the production seed size.
• The JSON store is single-writer. Shared storage is required before
  horizontally scaling the API.
• The first API page is capped at 100 releases. Keyset pagination or client
  pagination is needed beyond the initial production seed.
• Publish-to-index latency is bounded by confirmations and polling; this PR
  does not add a privileged public reindex endpoint.

This PR remains draft until the public performance evidence is attached. Once
that evidence satisfies #86, marking the PR ready and merging it will close the
ticket.

Metadata

• Project: Dotify sprints (Project 5)
• Status: In Progress
• Priority: P0
• Track: Production spine
• Phase: Now
• Type: Work
• Backlog doc: docs/backlog/26-cached-catalog-read-model.md
• Assignee: knzeng-e
• Labels: dotify-backlog, P0, backend, frontend, tests,
  observability
• Milestone: none exists in the repository
• Reviewers: none requested because no separate code owner is recorded
• Draft state: intentional, pending public p75 evidence

[netlify]

✅ Deploy Preview for muzinga canceled.

Name	Link
🔨 Latest commit	79dcd0a
🔍 Latest deploy log	https://app.netlify.com/projects/muzinga/deploys/6a61bd5dabb7f200080ff6bf