Skip to content

Add renderStreet endpoint: managed-street JSON β†’ labeled beauty-shot image - #1860

Open
kfarr wants to merge 11 commits into
mainfrom
claude/street-json-visual-endpoint-59qeln
Open

Add renderStreet endpoint: managed-street JSON β†’ labeled beauty-shot image#1860
kfarr wants to merge 11 commits into
mainfrom
claude/street-json-visual-endpoint-59qeln

Conversation

@kfarr

@kfarr kfarr commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Public HTTP endpoint (renderStreet Cloud Function + /render-street hosting rewrite) that turns a managed-street JSON blob into a labeled, angled pseudo-orthographic "beauty shot" PNG, aimed at LLM/skill/MCP callers as a zero-friction on-ramp: every response carries an #managed-street-json: editor deep link. Full API contract and deploy notes in docs/street-render-endpoint.md.

Commit 1 adds the endpoint: render.html + a lean editor-free webpack bundle (street-render.js) with a street-render-harness scene component (payload input, model-load quiescence readiness, bbox corner-fit camera framing, capture API), and the puppeteer-core + @sparticuz/chromium Cloud Function.

Commit 2 is a visual pass tuned side-by-side against NACTO street illustrations:

  • default azimuth 45Β° β†’ 20Β°; screen-space refit after the analytic corner-fit so the street is centered and fills the frame at any aspect (was off-center with excess sky at 1:1)
  • default street length 27m when the payload omits it (fit camera β‡’ shorter street = tighter frame)
  • street-label restyle (shared with the editor): white cells + segment-colored accent band on the street-surface edge, bold width over lighter name, flat unlit material with shadows off; bar tucked flush against the street face instead of floating 4m out
  • street-environment (shared): shadow camera box was a lopsided ~70Γ—50m around the origin leaving everything beyond it unshadowed; now symmetric 90Γ—90m at the same 2048 map (no added GPU memory, ~4.4cm texels). Harness aligns streets length: middle so a 60m street fits the origin-centered box (default-alignment follow-up: managed-street: default length alignment should be middleΒ #1863)

App-wide impact

street-label and street-environment are shared components: the editor gets the restyled cross-section bar and the larger shadow coverage. Both are visual-only; geometry, alignment handling, units, and save/load paths untouched. Shadow box is strictly larger than before on all four sides at unchanged resolution.

Test plan

  • npm test (774 passing, incl. managed-street JSON round-trip contract)
  • npm run lint
  • Headless renders verified via playwright against render.html at 1:1 and 16:10, azimuths 15/20/25/45, lengths 27/40/50/60: framing centered, label flush + legible, full shadow coverage to the far end of a 60m street
  • Editor smoke test: eyeball the restyled label bar on a normal scene
  • Staging deploy (hosting before function; RENDER_PAGE_URL/EDITOR_BASE_URL overrides in .env.dev-3dstreet) + curl smoke test

πŸ€– Generated with Claude Code

claude and others added 2 commits July 19, 2026 04:32
…hot image

Public HTTP Cloud Function (POST/GET, /render-street hosting rewrite) that
renders a managed-street JSON blob into a pseudo-orthographic beauty shot β€”
perspective camera at 45Β° azimuth with a narrow FOV, cross-section label
bar, environment lighting β€” using the same A-Frame component stack the app
uses. Aimed at LLM/skill/MCP callers as a zero-friction on-ramp: every
response carries an #managed-street-json: editor deep link so a generated
image funnels users into 3dstreet.app for 3D maps, AI rendering, and
editing.

- render.html + src/render/ (new `render` webpack entry, street-render.js):
  lean editor-free bundle; street-render-harness scene component handles
  payload input, model-load quiescence readiness (THREE
  DefaultLoadingManager busy-tracking + event quiet window), bounding-box
  corner-fit camera framing with auto side selection (shoot from the side
  with the lower boundary so buildings backdrop rather than occlude), and
  a capture API with title/branding annotations.
- public/functions/render-street.js: v2 onRequest, puppeteer-core +
  @sparticuz/chromium (SwiftShader WebGL), input validation and option
  clamping, PNG/JPEG bytes or format=json with dataURL + openInEditorUrl.
- docs/street-render-endpoint.md: API contract, JSON format summary,
  architecture, deploy notes.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WKpg1pMhRpJB3AxmantgV3
Tuning from side-by-side comparison against NACTO street illustrations:

- Camera: default azimuth 45 -> 20 (shallower, more NACTO-like); new
  screen-space refit after the analytic corner-fit β€” project bbox corners
  through a probe camera, pan to center the on-screen bounds and rescale
  distance to fill the frame, fixing the off-center street + excess sky
  at square aspects.
- Default street length 27m (was managed-street's 60m) when the payload
  omits length; the fit camera makes shorter streets fill the frame.
- street-label restyle (shared with editor): white cells with a
  segment-colored accent band on the street-surface edge (segment color,
  falling back to per-surface swatches), bold width over lighter name,
  hairline separators; flat unlit material with shadows off so scene
  lighting can't tint it and vehicles can't shadow it. Label bar now
  tucked flush against the street's start face instead of floating 4m out.
- street-environment (shared): shadow camera box was a lopsided ~70x50m
  around the origin, leaving everything beyond it unshadowed; now a
  symmetric 90x90m at the same 2048 map (no extra GPU memory, ~4.4cm
  texels). Render harness aligns streets length: middle so they sit
  inside the origin-centered box β€” a start-aligned 60m street exited the
  shadow volume halfway.

Co-Authored-By: Claude Fable 5 <[email protected]>
@kfarr kfarr changed the title (experimental) Add renderStreet endpoint: managed-street JSON β†’ labeled 45Β° beauty-s… Add renderStreet endpoint: managed-street JSON β†’ labeled beauty-shot image Jul 19, 2026
concurrency 2 x maxInstances 10 allowed 20 concurrent chromium renders:
~$1.4k/mo compute if saturated 24/7 (10 busy instances x 2vCPU+2GiB at
Cloud Run rates) plus ~$120 egress. Two instances bound that at ~$280/mo
while still serving 4 concurrent renders; excess traffic gets 429s.

Co-Authored-By: Claude Fable 5 <[email protected]>
Successful renders are cached in the default bucket under
renders/v1/<hash>.<png|jpg> (hash = truncated sha256 of the sorted-key
canonicalized street JSON + sanitized options) with a <hash>.json input
sidecar, and the response now carries a durable URL β€” imageUrl in
format=json, X-3DStreet-Image-Url on byte responses β€” that LLM callers
can embed in artifacts. The URL is a proxy contract: hosting rewrite
/render/img/** β†’ new lightweight serveRenderImage function (256MiB,
concurrency 80) β†’ Storage, served with immutable year-long cache headers
so the hosting CDN absorbs repeat traffic; the storage backend can move
later without breaking published links. Cache writes are best-effort
(imageUrl: null on failure, render still succeeds). The v1 path segment
allows clean invalidation when the renderer's look changes; share page +
eviction/re-render-on-miss tracked in #1864.

Co-Authored-By: Claude Fable 5 <[email protected]>
@kfarr

kfarr commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator Author

Added in 8e2e94f: stable image URLs. Every successful render is cached at renders/v1/<hash>.<ext> in the default bucket (hash = truncated sha256 of canonicalized street + options, input stored as a .json sidecar) and the response now returns a durable proxy URL (imageUrl in format=json, X-3DStreet-Image-Url on byte responses): /render/img/v1/<hash>.png, served by a new lightweight serveRenderImage function behind a hosting rewrite with immutable cache headers. Cache writes are best-effort; a storage failure degrades to imageUrl: null, not a failed render.

Not covered by local testing (no storage emulator in this repo's emulator config), so the staging smoke test should verify: curl -s 'https://.../render-street?format=json' -d @street.json | jq .imageUrl, then fetch that URL and confirm image bytes + Cache-Control: immutable. Share page and cache eviction/re-render-on-miss are tracked in #1864.

@kfarr
kfarr marked this pull request as ready for review July 19, 2026 08:09
@github-project-automation github-project-automation Bot moved this to Backlog (Not Ready) in 3DStreet Dev Tracking Jul 19, 2026
@kfarr kfarr moved this from Backlog (Not Ready) to For Review in 3DStreet Dev Tracking Jul 19, 2026
@kfarr

kfarr commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator Author

Manual QA checklist

  • Basic render: POST the docs' curl example β†’ PNG looks right (lanes, label bar, vehicles, shadows, title + branding)
  • Latency: time 1 cold render and 2-3 warm renders of unique streets; note the numbers here (cold vs warm is the open question)
  • Cache: repeat an identical request β†’ near-instant response; open X-3DStreet-Image-Url in a browser β†’ same image at the stable URL
  • JSON mode: ?format=json returns image, imageUrl, openInEditorUrl, meta; openInEditorUrl loads the street in the editor and matches the image (include one street with no length field: known gap, image renders 27m but editor opens 60m)
  • Headers: normal street β†’ X-3DStreet-Editor-Url present; huge street (max segments, long names/colors) β†’ response still succeeds, header omitted
  • GET mode: ?data=<base64url> renders; adding &width=1600&type=jpg overrides options inside the payload (jpg at 1600px)
  • Options spot check: environment=sunset, units=imperial, labels=false, boundaries=false each do what they say
  • Errors: empty segments β†’ 400 with a clear message; unknown model id in modelsArray β†’ still renders, meta.timedOut: true
  • Editor regression: open a normal scene in the editor β†’ label bar and shadows look right (street-label and street-environment were touched by this PR)
  • Load sanity: fire ~4 concurrent renders β†’ no 500s or instance OOM (429s on overflow are expected)

kfarr and others added 7 commits July 19, 2026 01:18
…rop autoSide

- Serve identical repeat requests from the render cache before spinning up
  a puppeteer render; sidecar now stores meta so cache hits answer
  ?format=json completely
- Single-flight browser relaunch: close the dead Chromium and reuse a
  concurrent request's fresh launch instead of orphaning a second one
- Omit X-3DStreet-Editor-Url when the deep link exceeds 8KB (hosting caps
  response headers); shared sendRenderResponse for hit/miss paths
- GET query params now win over options inside the ?data payload, and the
  flat merge only applies to GET
- Remove autoSide camera flip (v1 overkill; also fixes explicit azimuth=20
  being treated as unset)
- Docs + CLAUDE.md updated (serveRenderImage, header/GET behavior)
…kill

v1 "caller surface" for the renderStreet endpoint (#1864 Β§2) β€” a thin
instructions + schema layer over the existing public HTTP endpoint. No MCP
server, no new Cloud Function, no auth.

- public/openapi/render-street.yaml: OpenAPI 3.1 spec for /render-street.
  Single servers entry (https://3dstreet.app), POST (renderStreet) and GET
  (renderStreetLink) operations, reuses the managed-street Segment schema from
  nonCommandTools.js. Response modeled as ?format=json returning only imageUrl +
  openInEditorUrl + meta (no base64 image field, which blows the GPT Action
  response-size limit). Imports cleanly as a GPT Action.
- public/firebase.json: serve /openapi/** with application/yaml content-type +
  CORS so the GPT builder can import the spec by URL (static-file precedence
  already serves it at /openapi/render-street.yaml).
- docs/integrations/chatgpt-gpt.md: full custom-GPT setup package β€” name,
  description, system instructions, conversation starters, Action config, a
  worked suburban-arterial example, and a pre-publish note (privacy-policy URL,
  cold-start latency mitigation).
- .claude/skills/street-cross-section/SKILL.md: Claude Code Agent Skill teaching
  the same flow via curl, with the embedded segment schema and worked example.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LaYFYQ18Z4URsNmgbyM2fZ
CI builds the PR merge commit (branch + main). main's commit 2d19b95
("Geospatial map opacity + 3d-tiles-renderer 0.5.0 upgrade") deleted
src/aframe-components/blending-opacity.js and removed its require from
src/index.js. This branch's render entry (src/render/index.js) still required
that component, so the merge build failed with "Can't resolve
'../aframe-components/blending-opacity.js'". blending-opacity is a Google 3D
Tiles opacity component used only by street-geo.js, which the render bundle
never loads β€” so the render entry never needed it. Drop the require.

Also set minInstances: 1 on renderStreet to keep one instance warm and skip the
30-90s cold start (@sparticuz/chromium unpack + first-render asset downloads)
that GPT Actions / agent callers time out on. Idle min-instances bill CPU at
~10% of the active rate, so the warm floor is ~$25-30/mo (2vCPU+2GiB) versus the
~$139/mo busy-instance rate; maxInstances 2 still caps worst-case cost. Docs
updated to match.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LaYFYQ18Z4URsNmgbyM2fZ
Mirror the AI generators' Discord showcase for the render endpoint: a
successful fresh render posts an embed (rendered image, street name/stats, and
the "Open in editor" deep link) to the shared DISCORD_WEBHOOK_URL webhook.

- Anonymous by design (the endpoint has no auth/user); footer "3DStreet Render
  API", teal accent to distinguish from the generators' purple AI posts.
- Only fresh renders with a stable imageUrl post β€” cache hits already posted
  once, and a base64 data URL can't embed β€” so one post per new render.
- Awaited before the response (v2/Cloud Run throttles CPU after the response is
  sent, so fire-and-forget would stall) but internally bounded by a 4s
  AbortController, so a slow/down Discord never fails or noticeably delays a
  render (which already took 30-90s).
- The editor deep link embeds the whole street JSON and can be many KB; Discord
  rejects over-long embed URLs, so it's used as the clickable link only when
  short, else we fall back to the always-short imageUrl.

Adds the DISCORD_WEBHOOK_URL secret to the function config. Docs updated.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LaYFYQ18Z4URsNmgbyM2fZ
…y flags)

minInstances: 1 was hardcoded, so every environment reserved a warm 2GiB
instance (~$21/mo per Firebase's estimate) β€” wasteful on dev/staging that nobody
hits. Gate it on the target project instead: warm only in prod (dstreet-305604),
0 everywhere else. Resolution uses GCLOUD_PROJECT, which is set at both deploy
discovery and runtime, so a plain `firebase deploy --only functions` bakes the
right value per project with no per-deploy flags β€” the same staging-vs-prod
pattern rad-dispatch.js already uses.

Same mechanism also fixes RENDER_PAGE_URL / EDITOR_BASE_URL: they now default
per project (prod -> 3dstreet.app, else -> dev-3dstreet.web.app) instead of
always pointing at prod. This corrects a latent dev bug where the dev function
rendered via the prod page and returned prod image/editor URLs. Explicit env
vars still override (e.g. local dev). Docs updated.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LaYFYQ18Z4URsNmgbyM2fZ
The /render-street hosting rewrite is capped at Firebase Hosting's 60s limit, so
a cold render (30-90s) returns a 502 even though the function completes and
caches server-side. Document the mitigations (prod warm instance + content
cache; use the direct function URL for cold/one-off renders, notably on
dev/staging where minInstances is 0) and note GPT Actions' own ~45s ceiling.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LaYFYQ18Z4URsNmgbyM2fZ
@kfarr

kfarr commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

Decision: render natively in Blender; the browser-free path is managed-street JSON β†’ GLB β†’ Blender β†’ PNG

TL;DR: keeping a browser in the render path costs 23–59s/render, so we're
switching to a native renderer (Blender + Cycles, CPU). The new "bare-metal"
Cloud Function takes managed-street JSON, assembles the geometry in headless
THREE (no DOM, no A-Frame), and exports a GLB. Blender loads that GLB and
renders the beauty shot. The endpoint returns both the image and the GLB
({ imageUrl, glbUrl }). The current browser-based endpoint stays working as-is
until the native path is proven. Full next-steps prompt at the bottom.

We are deliberately keeping this managed-street-only for now. A more
generalized "serialized JSON scene manifest (IR)" is explicitly NOT part of this
work β€” later we can either morph this function or add a second one for that.
Don't build the IR; use GLB as the interchange.

Why the browser is out

The current renderStreet (public/functions/render-street.js) drives headless
Chrome -> A-Frame -> SwiftShader (software WebGL, no GPU) to turn managed-street
JSON into a PNG. Instrumented warm+new renders on dev:

scene segments wall time
residential (light) 7 ~23s
complete street w/ bus + median 7 ~48s
avenue w/ buildings 9 ~59s

The killer is rasterization on SwiftShader: the harness does ~3 settle passes at
2–8s each plus a final capture, ~30–50s of the total. Warm instances only remove
cold start; a browser GPU is ~$650–800/mo always-on; screenshot services shoot
half-loaded scenes. None fix the per-render floor.

Why native, and why Blender

The dependency isn't "a browser," it's A-Frame + WebGL. THREE.js runs
headless in Node, and Blender imports glTF natively. So we split the two jobs the
browser conflated β€” assembling the street (cheap) and rasterizing it (the
expensive part) β€” and hand geometry to Blender as a standard GLB instead of
re-rasterizing in software WebGL.

  • Blender + Cycles (CPU) is the only first-class CPU path (Godot has no CPU
    renderer; Eevee is GPU-only). Path-traced, ~3–15s, cheap, nicer beauty shot,
    and it loads GLB out of the box.
  • Godot is the pick only if we later accept a GPU. Settle empirically with a
    bake-off on a golden GLB before committing.

The architecture (managed-street only; no IR)

managed-street JSON
  -> [DOM-free Node assembler: pure THREE, no A-Frame]   ==> GLB (geometry + placed models [+ camera + sun])
  -> [Blender Cycles: load GLB, add sky/ground rig, render]   ==> PNG
  -> [Pillow: composite 2D cross-section label + "made with 3DStreet" branding]
returns { imageUrl, glbUrl }

What lives where:

  • GLB (the interchange): every segment surface (the below-box slabs) with
    its material/color, plus every generated model placed at its final transform.
    Optionally also the framed camera and the sun (glTF supports both). GLB is the
    ONLY structured output β€” no custom manifest.
  • Blender adds what isn't geometry: the environment sky/lighting preset,
    the ground plane, and the camera+sun if we don't bake them into the GLB.
  • Pillow overlays the 2D label bar + branding after render (same as the
    current capture() β€” do NOT bake text into 3D).

Next-steps prompt (paste into a fresh session)

Build the browser-free street renderer for 3DStreet. Repo: /Users/kieranfarr/dev/
3dstreet, branch claude/street-json-visual-endpoint-59qeln (PR #1860). Read the
"Decision" comment above this prompt for the rationale; read
docs/street-render-endpoint.md for the current endpoint contract.

GOAL: a DOM-free Node function that takes managed-street JSON and returns a GLB,
then a Blender step that renders that GLB to a PNG. Managed-street ONLY. Do NOT
build a generalized JSON scene-manifest / IR β€” GLB is the interchange.

WORKING AGREEMENT (read first):

  • Start writing code within your first few actions. Hard cap ~6 file reads before
    you create the first new file. Don't survey the whole component family β€” the
    facts you need are below. Build, run, iterate; don't narrate exploration.

THE PIPELINE:
managed-street JSON -> [pure-THREE assembler, no A-Frame/DOM] -> GLB
-> [Blender Cycles] -> PNG (+ return the GLB)

TASK 1 β€” the assembler (this is the actual raw function; do this first):
Write a Node module street-to-glb that takes managed-street JSON and returns a
GLB Buffer, using THREE directly (no A-Frame, no jsdom). It must:

  1. For each segment, build the surface slab: a THREE.BoxGeometry matching the
    below-box geometry (width = segment width, depth = street length,
    height ~0.2, top face at y=0), textured/colored per segment surface+color.
    Position segments across X exactly like src/aframe-components/street-align.js
    (center => start at -totalWidth/2, then walk cumulative widths; y = elevation
    height). Length alignment: center on origin (matches the render harness's
    length: middle).
  2. Place generated models. Port the placement arithmetic from
    src/aframe-components/street-generated-clones.js (fixed/random/single/fit),
    -stencil, -striping, -pedestrians, -rail. Load each referenced catalog model
    GLB with THREE GLTFLoader in Node and put it at the computed transform.
    • Model URLs/dims are in src/catalog.json (STREET.catalog); building tiling
      widths are the hardcoded buildingWidths table in
      street-generated-clones.js generateFit() (fallback ||10).
    • random/fit modes use a SEEDED mulberry32 RNG (src/lib/rng.js). Replicate it
      byte-for-byte so JSON->GLB is deterministic (needed for the endpoint's
      content-hash cache). Pin an explicit seed on random/fit entries in tests.
  3. Export the assembled THREE.Scene to GLB with GLTFExporter (Node needs a few
    globals shimmed). Optionally include the framed camera and a sun light so the
    GLB is self-renderable; otherwise pass those to Blender separately.
    Reference/oracle for correct GLB output (do NOT reuse its DOM path β€” it's the
    browser exporter): src/editor/lib/exportUtils.js, src/batch-models.js.
    SCOPE: the managed-street family only β€” managed-street, street-segment,
    street-align, street-generated-{clones,stencil,striping,pedestrians,rail},
    street-label, street-environment. OUT: legacy street/streetmix-loader,
    intersection, geo, arbitrary user entities.

TASK 2 β€” Blender render:
A Blender (Cycles, CPU) script that imports the GLB, adds the environment rig
(day/night/sunset: sky + sun + ambient β€” port the presets from
src/aframe-components/street-environment.js setEnvOption()), a ground plane, and
frames the camera via the corner-fit math in
src/render/street-render-harness.js frameCamera() (analytic corner-fit + the
screen-space refit loop β€” both are pure THREE, port them). Render to PNG.
Then composite the 2D cross-section label bar + "made with 3DStreet" branding
with Pillow (label cell widths/text/accent come from the travelled-way segments,
same as street-label.js) β€” do NOT bake labels into the GLB.

VALIDATE with two golden streets end to end (JSON -> GLB -> PNG):

  1. suburban-arterial β€” payload in .claude/skills/street-cross-section/SKILL.md
    (sidewalks + pedestrians + 3 random drive lanes); add explicit seeds.
  2. avenue-with-boundaries β€” drive lanes + sidewalks + boundary variant
    'brownstone' side left AND right (exercises buildingWidths/fit tiling).
    DONE = both GLBs open in Blender and render to a labeled beauty shot comparable
    to the current browser output, and the assembler is deterministic for a fixed
    seed. Do a Cycles-CPU bake-off on one golden GLB to confirm the ~3–15s target
    before wiring it into the Cloud Function.

WIRE-UP (after the pipeline works):
New/updated Cloud Function behind the same POST contract as renderStreet,
returning { imageUrl, glbUrl } (cache both at content-hash paths, mirroring the
existing /render/img/v1/ cache in public/functions/render-street.js). Keep
the old browser endpoint working until this is proven; don't rip it out.

KEY FILES:

  • Endpoint (current, keep working): public/functions/render-street.js
  • Camera framing math to port: src/render/street-render-harness.js frameCamera()
  • Assembler logic to port: the managed-street family listed in TASK 1
  • GLB exporter oracle (browser, reference only): src/editor/lib/exportUtils.js,
    src/batch-models.js
  • Catalog (model URLs + dims): src/catalog.json
  • Environment presets: src/aframe-components/street-environment.js
  • Docs/contract: docs/street-render-endpoint.md,
    .claude/skills/street-cross-section/SKILL.md, public/openapi/render-street.yaml

NOTES:

  • Deferred, separate from this work: the generalized scene-manifest/IR function;
    prod deploy; the ChatGPT custom GPT; privacy-policy URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: For Review

Development

Successfully merging this pull request may close these issues.

2 participants