feat(dsl): skeletal motion DSL — walk, reach and wave gaits for COCO-17 - #98
feat(dsl): skeletal motion DSL — walk, reach and wave gaits for COCO-17#98Nitjsefnie wants to merge 6 commits into
Conversation
New dsl/gait.py: frozen pydantic gait models emit body-local joint offsets; an existing root PathUnion translates the skeleton, and frame compilation is driven through the motion DSL's own _PathNode.compile_frames so over/at_speed behave identically. Fully kinematic and deterministic; no new dependencies. Co-Authored-By: Kimi K3 <[email protected]>
Covers all three gaits: 17 COCO joints per frame, per-frame displacement bounded by each gait's analytic max_local_speed plus root speed, root translation over the requested duration, determinism, and a two-camera manifest round-trip labelling every joint per camera. Co-Authored-By: Kimi K3 <[email protected]>
Short note under the pose manifest extension: gaits generate PoseTrajectory via body-local offsets plus a root path, reusing the motion DSL timing model. Co-Authored-By: Kimi K3 <[email protected]>
Knees and the reaching elbow were interpolated positions, so hip->knee, knee->ankle and shoulder->elbow/elbow->wrist changed length frame to frame (up to ~24% leg stretch, reach forearm down to 48% at the target). Segment lengths are now fixed from height and the mid joint is solved by two-link IK with an explicit bend direction (knees forward, elbows outward-down). Walk feet ride a sphere about the hip so the leg solve stays well-conditioned and fully analytic; unreachable reach targets are clamped once to the reachable sphere so clamping adds no velocity spike. Also credits assembly_station.py in the module docstring and flags that over()/at_speed() retime only the root translation, never the gait cadence. Co-Authored-By: Kimi K3 <[email protected]>
New rigidity test asserts all 19 COCO-17 edges keep constant length across frames for all gaits, including a reach clamped past full extension (red against the old midpoint construction). The continuity bound now measures the root's actual max per-frame displacement instead of the unsound average-speed term, and covers Bezier and unequal-legged Waypoint roots that false-failed under the average. A golden test pins literal joint coordinates for fixed configs. Co-Authored-By: Kimi K3 <[email protected]>
Co-Authored-By: Kimi K3 <[email protected]>
|
Merged as 4d85fc7. Thank you @Nitjsefnie — this is genuinely impressive work. The closed-form two-link IK keeping every COCO-17 edge rigid frame-to-frame is exactly the right call for ground-truth output, and I appreciated the honesty in max_local_speed(): analytic bounds where they exist, a densely-sampled safety-factored fallback for the IK mid-joints where they don't, and the reasoning written down. Clean composition with the existing PathUnion/timing DSL, all-deterministic, and the rigidity/continuity/golden-joint tests nail down the contract. 341 passed locally (both gait and randomization suites green together), ruff/format/mypy clean. |
|
Landed via local squash-merge (author preserved as @Nitjsefnie) in 4d85fc7 on main; closing this fork PR. |
Summary
A skeletal-motion layer: parametric
walk/reach/wavegaits that generate aPoseTrajectoryover frames for the COCO-17 skeleton, so a scene no longer needs 17 hand-authored joint positions per
frame. Additive — a new
dsl/gait.pyand its exports, nothing existing modified.Related Issues and Pull Requests
Fixes #29
Touches
DESIGN.md, as does #97 — whichever lands second needs a one-hunk rebase there.Changes
src/multicam_sim/dsl/gait.py(new) —WalkGait,ReachGait,WaveGait(frozen pydantic,validators in the
motion.pyidiom) behind aGait.walk(...)/.reach(...)/.wave(...)namespace mirroring
Path.to_pose_trajectory(entity_id, *, fps, num_frames)is the entry point.src/multicam_sim/dsl/__init__.py— exports.DESIGN.md— the gait layer, its parameters and its contracts.tests/test_gait.py(new) — 12 tests.Timing is reused, not reimplemented.
to_pose_trajectorydrives the existingroot.compile_frames(fps, num_frames, ...)and samples the gait at the samet = frame / fps, soover(seconds),at_speed(v)and untimed stretch all behave exactly as they do for a plain path.Verified as bit-exact: across six root variants (
over, twoat_speed, untimed stretch, a circle,and a sequence composite), the gait's joint position matches
plain path point + standing offsetto0.000e+00over 91 frames, including past-duration clamping.Root translation composes with joint offsets: a gait emits body-local offsets, any
PathUnioncarries the skeleton, and the world joint is root + offset.
Testing
ruff check,ruff format --check,mypy srcclean.pytest: 294 passed, 4 skipped onupstream/main→ 306 passed, 4 skipped here, the delta being exactly the 12 new tests. Manifestsfor the example scenes are byte-identical to
upstream/main— checked by sha256 and byte lengthacross all nine shipped scenes, not by inspection.
This went through an adversarial review before opening, and three things came out of it that are
worth stating plainly rather than burying:
1. The first version's limbs were not rigid, and that is now fixed. Knees were placed at the
hip–ankle midpoint, so hip→knee ranged 0.4165–0.4657 and knee→ankle stretched about 24% at full
swing; the reach forearm shrank to 48% of its rest length at the target. Bones that change length
are wrong output for pose ground truth, whatever it looks like on screen. Limbs now use closed-form
two-link IK with fixed segment lengths derived from
height, and all 19 COCO-17 edges hold constantto ~1e-13. There is a test asserting exactly that, and it fails against the old midpoint
construction.
Two things fell out of that rework: walk feet ride a sphere about the hip so the leg never clamps
(which keeps the continuity bound fully analytic), and an unreachable reach target is clamped to just
inside the reachable sphere rather than onto it — an exact-sphere clamp has a velocity singularity,
measured as a 9.4 m/s knee spike before the redesign.
2. The continuity test's bound was unsound for non-linear roots. It derived the root term from
length()/total_duration(), an average speed, so a Bezier or unequal-legged Waypoint root couldexceed it and fail the test spuriously — reproduced at 0.0821 against a 0.0492 bound. The bound now
uses the measured maximum per-frame root displacement from the compiled frames, and the test covers
both root types including that previously-failing config. The gait's own
max_local_speed()wassound and is unchanged; a 1671-combination sweep of the validator-legal space found no violation.
3. The determinism test was self-consistency only. Compiling twice in one process would also pass
against something seeded once at class level. The gaits contain no randomness at all — confirmed by
audit, not by that test — so there is now a golden test pinning actual joint coordinates at fixed
frames against pasted-in literals with no shared constants.
Follow-ups / Known Limitations
examples/assembly_station.pyalready animates aPoseTrajectoryprocedurally (a sinusoidal wristreach), so this is not the first moving skeleton in the repo — the difference is that it hand-authors
all 17 joint offsets in a literal table with no timing model, root path or reusable API. The
docstring credits it.
over()/at_speed()retime the root translation only — gait cadence is wall-clock and doesnot stretch with them. Documented, and called out here because it is genuinely surprising.
ReachGait.targetis not constrained, so a target inside the torso would send the arm through thebody. Left unconstrained deliberately rather than guessing at a body model — say if you want a
validator and what shape it should take.
numbers if you have opinions about what a walk should look like.