Skip to content

feat(dsl): skeletal motion DSL — walk, reach and wave gaits for COCO-17 - #98

Closed
Nitjsefnie wants to merge 6 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/29-skeletal-motion-dsl
Closed

feat(dsl): skeletal motion DSL — walk, reach and wave gaits for COCO-17#98
Nitjsefnie wants to merge 6 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/29-skeletal-motion-dsl

Conversation

@Nitjsefnie

@Nitjsefnie Nitjsefnie commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

A skeletal-motion layer: parametric walk / reach / wave gaits that generate a PoseTrajectory
over frames for the COCO-17 skeleton, so a scene no longer needs 17 hand-authored joint positions per
frame. Additive — a new dsl/gait.py and its exports, nothing existing modified.

Related Issues and Pull Requests

Fixes #29

Touches DESIGN.md, as does #97 — whichever lands second needs a one-hunk rebase there.

Changes

  • src/multicam_sim/dsl/gait.py (new) — WalkGait, ReachGait, WaveGait (frozen pydantic,
    validators in the motion.py idiom) behind a Gait.walk(...) / .reach(...) / .wave(...)
    namespace mirroring Path. to_pose_trajectory(entity_id, *, fps, num_frames) is the entry point.
  • src/multicam_sim/dsl/__init__.py — exports.
  • DESIGN.md — the gait layer, its parameters and its contracts.
  • tests/test_gait.py (new) — 12 tests.

Timing is reused, not reimplemented. to_pose_trajectory drives the existing
root.compile_frames(fps, num_frames, ...) and samples the gait at the same t = frame / fps, so
over(seconds), at_speed(v) and untimed stretch all behave exactly as they do for a plain path.
Verified as bit-exact: across six root variants (over, two at_speed, untimed stretch, a circle,
and a sequence composite), the gait's joint position matches plain path point + standing offset to
0.000e+00 over 91 frames, including past-duration clamping.

Root translation composes with joint offsets: a gait emits body-local offsets, any PathUnion
carries the skeleton, and the world joint is root + offset.

Testing

ruff check, ruff format --check, mypy src clean. pytest: 294 passed, 4 skipped on
upstream/main306 passed, 4 skipped here, the delta being exactly the 12 new tests. Manifests
for the example scenes are byte-identical to upstream/main — checked by sha256 and byte length
across all nine shipped scenes, not by inspection.

This went through an adversarial review before opening, and three things came out of it that are
worth stating plainly rather than burying:

1. The first version's limbs were not rigid, and that is now fixed. Knees were placed at the
hip–ankle midpoint, so hip→knee ranged 0.4165–0.4657 and knee→ankle stretched about 24% at full
swing; the reach forearm shrank to 48% of its rest length at the target. Bones that change length
are wrong output for pose ground truth, whatever it looks like on screen. Limbs now use closed-form
two-link IK with fixed segment lengths derived from height, and all 19 COCO-17 edges hold constant
to ~1e-13. There is a test asserting exactly that, and it fails against the old midpoint
construction.

Two things fell out of that rework: walk feet ride a sphere about the hip so the leg never clamps
(which keeps the continuity bound fully analytic), and an unreachable reach target is clamped to just
inside the reachable sphere rather than onto it — an exact-sphere clamp has a velocity singularity,
measured as a 9.4 m/s knee spike before the redesign.

2. The continuity test's bound was unsound for non-linear roots. It derived the root term from
length()/total_duration(), an average speed, so a Bezier or unequal-legged Waypoint root could
exceed it and fail the test spuriously — reproduced at 0.0821 against a 0.0492 bound. The bound now
uses the measured maximum per-frame root displacement from the compiled frames, and the test covers
both root types including that previously-failing config. The gait's own max_local_speed() was
sound and is unchanged; a 1671-combination sweep of the validator-legal space found no violation.

3. The determinism test was self-consistency only. Compiling twice in one process would also pass
against something seeded once at class level. The gaits contain no randomness at all — confirmed by
audit, not by that test — so there is now a golden test pinning actual joint coordinates at fixed
frames against pasted-in literals with no shared constants.

Follow-ups / Known Limitations

  • examples/assembly_station.py already animates a PoseTrajectory procedurally (a sinusoidal wrist
    reach), so this is not the first moving skeleton in the repo — the difference is that it hand-authors
    all 17 joint offsets in a literal table with no timing model, root path or reusable API. The
    docstring credits it.
  • over() / at_speed() retime the root translation only — gait cadence is wall-clock and does
    not stretch with them. Documented, and called out here because it is genuinely surprising.
  • ReachGait.target is not constrained, so a target inside the torso would send the arm through the
    body. Left unconstrained deliberately rather than guessing at a body model — say if you want a
    validator and what shape it should take.
  • Gait parameters are a first pass tuned for structural correctness, not realism. Happy to take
    numbers if you have opinions about what a walk should look like.

Nitjsefnie and others added 6 commits July 31, 2026 13:00
New dsl/gait.py: frozen pydantic gait models emit body-local joint offsets;
an existing root PathUnion translates the skeleton, and frame compilation is
driven through the motion DSL's own _PathNode.compile_frames so over/at_speed
behave identically. Fully kinematic and deterministic; no new dependencies.

Co-Authored-By: Kimi K3 <[email protected]>
Covers all three gaits: 17 COCO joints per frame, per-frame displacement
bounded by each gait's analytic max_local_speed plus root speed, root
translation over the requested duration, determinism, and a two-camera
manifest round-trip labelling every joint per camera.

Co-Authored-By: Kimi K3 <[email protected]>
Short note under the pose manifest extension: gaits generate PoseTrajectory
via body-local offsets plus a root path, reusing the motion DSL timing model.

Co-Authored-By: Kimi K3 <[email protected]>
Knees and the reaching elbow were interpolated positions, so hip->knee,
knee->ankle and shoulder->elbow/elbow->wrist changed length frame to frame
(up to ~24% leg stretch, reach forearm down to 48% at the target). Segment
lengths are now fixed from height and the mid joint is solved by two-link IK
with an explicit bend direction (knees forward, elbows outward-down). Walk
feet ride a sphere about the hip so the leg solve stays well-conditioned and
fully analytic; unreachable reach targets are clamped once to the reachable
sphere so clamping adds no velocity spike. Also credits assembly_station.py
in the module docstring and flags that over()/at_speed() retime only the
root translation, never the gait cadence.

Co-Authored-By: Kimi K3 <[email protected]>
New rigidity test asserts all 19 COCO-17 edges keep constant length across
frames for all gaits, including a reach clamped past full extension (red
against the old midpoint construction). The continuity bound now measures
the root's actual max per-frame displacement instead of the unsound
average-speed term, and covers Bezier and unequal-legged Waypoint roots
that false-failed under the average. A golden test pins literal joint
coordinates for fixed configs.

Co-Authored-By: Kimi K3 <[email protected]>
@bamdadd

bamdadd commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Merged as 4d85fc7. Thank you @Nitjsefnie — this is genuinely impressive work. The closed-form two-link IK keeping every COCO-17 edge rigid frame-to-frame is exactly the right call for ground-truth output, and I appreciated the honesty in max_local_speed(): analytic bounds where they exist, a densely-sampled safety-factored fallback for the IK mid-joints where they don't, and the reasoning written down. Clean composition with the existing PathUnion/timing DSL, all-deterministic, and the rigidity/continuity/golden-joint tests nail down the contract. 341 passed locally (both gait and randomization suites green together), ruff/format/mypy clean.

@bamdadd

bamdadd commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Landed via local squash-merge (author preserved as @Nitjsefnie) in 4d85fc7 on main; closing this fork PR.

@bamdadd bamdadd closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skeletal motion DSL: walk / reach / wave gaits for the COCO-17 skeleton

2 participants