Skip to content

feat(dsl): seeded domain randomization for background, light and distractors - #97

Closed
Nitjsefnie wants to merge 8 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/41-domain-randomization
Closed

feat(dsl): seeded domain randomization for background, light and distractors#97
Nitjsefnie wants to merge 8 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/41-domain-randomization

Conversation

@Nitjsefnie

@Nitjsefnie Nitjsefnie commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Typed, seeded domain-randomization knobs for the Scene builder — background, light and N distractors — sampled deterministically from a pydantic spec, applied through a single SceneBuilder.randomize(spec, seed=...) entry point. Everything is additive and off by default: a scene that never calls randomize builds a byte-identical manifest.

Related Issues and Pull Requests

Fixes #41

Changes

  • src/multicam_sim/randomization.py (new) — RandomizationSpec composed of BackgroundSpec, LightSpec and DistractorSpec, all frozen pydantic models in the style of noise.py. Every knob is a closed (min, max) interval; a model_validator rejects inverted and non-finite bounds. sample(seed) returns a concrete typed RandomizationSample via numpy.random.default_rng(seed), with a fixed draw order.
  • src/multicam_sim/dsl/builder.pyrandomize(spec, *, seed=0) samples, stashes background and light for build(), and adds each sampled position through the existing distractor(...) method so randomized distractors are ordinary entities. A per-builder counter keeps generated ids unique.
  • src/multicam_sim/scene.pyScene.background, Scene.light and Scene.randomization (a RandomizationRecord of spec + seed, so a randomized run is reproducible from its own output). All default to None; the manifest builder never reads them.
  • src/multicam_sim/dsl/render.py — the pyrender backend consumes scene.background / scene.light when set, with a _light_pose helper. Absent them the render is unchanged.
  • DESIGN.md — the knobs, their units, their default intervals, and the reproducibility contract.
  • tests/test_randomization.py (new) — 35 tests.

Only three of the issue's four knobs are implemented. Textures are not, deliberately: the codebase has no texture concept to hang one off — entities render as untextured spheres coloured by a stable id-derived RGB — so a texture knob would first need a material field on the render specs. That is renderer-side schema work and your call, not something to guess at here.

Testing

uv run ruff check ., uv run ruff format --check ., uv run mypy src all clean. uv run pytest: 284 passed, 4 skipped on upstream/main319 passed, 4 skipped on this branch, same plain uv sync both sides.

Manifests for all example scenes are byte-identical to upstream/main when randomize is not called — checked by sha256 and byte length, not by inspection.

This PR went through an adversarial review before it was opened, and four things came back that are worth stating plainly, because three of them were wrong in the first version:

  1. The determinism tests originally pinned nothing. sample() being byte-identical for the same seed holds for any deterministic implementation, so a mutation that reordered the draws — or used seed + 1 — changed every sampled value and still left all tests green. There is now a golden test asserting the shipped sample for one fixed spec and seed against pasted-in literals that share no constant with production code; both mutations turn it red.
  2. Calling randomize twice produced duplicate entity ids. The ids restarted at rand_distractor_0, so the manifest carried duplicates and frames_by_id silently kept only the second. Fixed with a per-builder counter, plus a test asserting all ids are unique after two calls.
  3. Background colour was already configurable and the first draft of the docs said otherwise. PyrenderBackend(bg=...) exists on main (render.py:110); what is new here is scene-level and randomizable control. The docs now say that, and state the precedence — scene.background wins over the constructor bg when set. Say if you would rather the explicit constructor argument win.
  4. NaN bounds passed validation (every NaN comparison is false, so the min > max check never fired) and then failed inside sample() with numpy's OverflowError. Non-finite bounds are now rejected at construction, with 12 parametrized cases.

Bugs Discovered

That one is pre-existing and independent of this PR — two plain .distractor("dup", ...) calls reproduce it on main with no randomization involved. I deliberately did not add a guard to entity() or distractor() here, since that changes existing public behaviour and belongs in its own change rather than buried in a feature PR.

Follow-ups / Known Limitations

  • The Kubric backend does not consume the sampled light or background — only the pyrender backend does. Wiring it would mean extending KubricSceneSpec with a light field. The values are on the Scene and in the provenance sidecar, so any backend can pick them up later.
  • Whole-Scene JSON now carries "background": null, "light": null, "randomization": null on un-randomized scenes, the same shape the possession sidecar introduced. The byte-golden contract is the manifest (Manifest.to_json(exclude_none=True)), which is untouched, and no production code serializes whole scenes to disk.
  • Old-shape scene JSON loads into the new models fine. New JSON loaded by older code also succeeds — pydantic ignores the extra keys — but re-serializing from older code silently drops the provenance. Inherent to an additive field; noted rather than solved.

Nitjsefnie and others added 8 commits July 31, 2026 11:57
Typed pydantic spec module: background, light, and distractor sub-specs,
each knob a closed (min, max) interval validated against inversion. Sampling
is pure and seeded via numpy.random.default_rng with a fixed draw order, so a
spec + seed deterministically yields a concrete RandomizationSample.

Co-Authored-By: Kimi K3 <[email protected]>
…add#41)

SceneBuilder.randomize(spec, seed=...) samples the spec and applies it:
sampled background/light land on new optional Scene fields, distractors go
through the existing .distractor entry point, and a RandomizationRecord
sidecar (spec + seed) rides on the scene. All fields default to None, so an
un-randomized scene is byte-identical; the pyrender backend consumes the
sampled background/light when present.

Co-Authored-By: Kimi K3 <[email protected]>
Covers byte-identical sampling per seed, divergence across seeds, inverted-
interval rejection, un-randomized scenes staying byte-identical (manifest and
scene JSON), the distractor count knob flowing through the existing distractor
path, and the provenance sidecar round-tripping and reproducing the sample.

Co-Authored-By: Kimi K3 <[email protected]>
New DESIGN.md section under DSL assembly: what each knob means, its units,
its default interval, and the same-spec-plus-same-seed reproducibility
contract.

Co-Authored-By: Kimi K3 <[email protected]>
A second .randomize(...) call restarted distractor numbering at 0, producing
duplicate entity ids in the scene and manifest and silently overwriting the
first batch in build()'s frames_by_id. Ids now come from a per-builder
counter, so a single call keeps the stable rand_distractor_0..N-1 names and
repeat calls cannot collide.

Co-Authored-By: Kimi K3 <[email protected]>
NaN bounds slip past every min > max comparison and then crashed inside
sample() with numpy's OverflowError. The shared interval check now rejects
non-finite bounds, so LightSpec/DistractorSpec fail fast with a
ValidationError naming the field (BackgroundSpec's channel range already
rejected them).

Co-Authored-By: Kimi K3 <[email protected]>
…d#41)

PyrenderBackend(bg=...) already made the background configurable per renderer;
the new knob is scene-level, randomizable control that takes precedence over
that constructor default. Say so in DESIGN.md and the BackgroundSpec
docstring, and document the precedence at the override point in render.py.

Co-Authored-By: Kimi K3 <[email protected]>
bamdadd#41)

test_golden_sample_pins_the_shipped_draw_order asserts the sampled values for
one fixed spec and seed against pasted-in numeric literals, so a change to
draw order or seed handling goes red even though sampling stays
deterministic (verified: both the reordered-draws and default_rng(seed+1)
mutations fail it). Also covers unique ids across repeated randomize calls
and NaN/inf rejection for every spec.

Co-Authored-By: Kimi K3 <[email protected]>
@bamdadd

bamdadd commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Merged as f1e97ad. Thank you @Nitjsefnie — this is a lovely, disciplined feature: frozen typed specs mirroring noise.py, a pure seeded sample() with a fixed draw order, and the additive off-by-default contract verified by byte-identical golden manifests. The pinned golden-sample test and the NaN/inverted-bound rejection are exactly the right rigor. Appreciated the honest write-up of what was fixed in review and the pre-existing #96 you surfaced rather than smuggled a guard for. 329 passed locally, ruff/format/mypy clean.

@bamdadd

bamdadd commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Landed via local squash-merge (author preserved as @Nitjsefnie) in f1e97ad on main; closing this fork PR.

@bamdadd bamdadd closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Domain randomization: backgrounds, lighting, distractor objects

2 participants