Anonymous catalog pages are never cached at the Cloudflare edge

[tiankaima]

Summary

Anonymous GET requests for public catalog pages (/catalog/sections/*, /catalog/courses/*, /catalog/teachers/*, etc.) are never cached at the Cloudflare edge. Every request — including the heavy AI-crawler traffic tracked in the linked issue — runs the full Worker pipeline: hooks, DB queries, SSR. Making these pages CDN-cacheable for anonymous traffic turns most crawler hits into cheap cache reads.

Current behavior

• src/hooks.server.ts:315-317 sets Cache-Control: no-store on every HTML response that doesn't already carry a Cache-Control header:

  if (!mutableResponse.headers.has("Cache-Control")) {
    mutableResponse.headers.set("Cache-Control", "no-store");
  }

• None of the catalog page loads set their own cache headers, so every catalog HTML page is no-store.

• The header constants for exactly this purpose already exist but are only used by a few API routes: PUBLIC_CATALOG_HEADERS / PUBLIC_LOCALE_CATALOG_HEADERS in src/lib/public-cache-control.ts (public, max-age=0, stale-while-revalidate=300 for browsers; Cloudflare-CDN-Cache-Control: public, max-age=60, stale-while-revalidate=300 for the edge).

• src/routes/robots.txt/+server.ts and src/routes/sitemap.xml/+server.ts already use Cloudflare-CDN-Cache-Control via createCrawlerDiscoveryResponse, proving the pattern works on this zone.

Correctness constraints (must hold)

1. Never cache personalized content. Authenticated requests (session cookie / bearer / better-auth cookie — see hasRequestAuthSignal in src/lib/auth/request-auth-signal.ts) must keep no-store, and the zone cache configuration must not serve cached anonymous HTML to logged-in users for URLs whose render depends on locals.authUser (subscribe state, viewer flags).
2. Catalog content is public and identical for all anonymous viewers, except:
   • Locale: rendered language depends on the locale cookie / Accept-Language (negotiateLocale in hooks). A cached page in one locale may be served to a client preferring another; acceptable short-term (60 s TTL) but should be noted, or the locale should be part of the cache key.
   • CSP nonce: hooks inject a per-request script nonce (addScriptNonce). A cached HTML document carries a fixed nonce that still matches its own scripts, so it remains valid; it only slightly weakens the nonce's freshness guarantee.
3. SSR'd data (__data requests, form actions, POSTs) are out of scope — only anonymous HTML GETs on public routes.

Proposed fix

• In src/hooks.server.ts (or the catalog page loads), for GET requests on public catalog routes without an auth signal, set Cache-Control: public, max-age=0, stale-while-revalidate=300 and Cloudflare-CDN-Cache-Control: public, max-age=60, stale-while-revalidate=300 instead of the blanket no-store. Authenticated requests keep no-store via the existing fallback.
• Add a zone-level Cache Rule making catalog HTML eligible for cache only when the request carries no session cookie (so personalized pages can never be served from cache), respecting origin CDN-Cache-Control for TTL.
• Optionally include locale in the cache key (or skip caching when a non-default locale is negotiated) to avoid cross-locale cache hits.

Expected impact

Crawler/bot traffic (historically hundreds of thousands of requests/day) becomes mostly edge cache hits with ~0 ms Worker CPU, instead of full SSR renders.