Bag-replay scoring: validate parameter choices against real recordings#52
Open
MJohnson459 wants to merge 1 commit into
Open
Bag-replay scoring: validate parameter choices against real recordings#52MJohnson459 wants to merge 1 commit into
MJohnson459 wants to merge 1 commit into
Conversation
Offline harness that replays a real recorded mapping bag through SLAM (or
kinematic_icp) under N parameter sets and scores the results with truth-free
proxies, then writes a side-by-side comparison report (metrics table + rendered
maps). The reality check that complements the sim benchmark (`pixi run bench`),
which needs Gazebo ground truth.
pixi run bag-replay -- --bag ~/.mote/bags/mapping/<ts> \
--params baseline.yaml sparse_no_loop.yaml
- Reuses the sim benchmark's ROS-free metrics.py, not a fork: adds truth-free
map_quality (wall-thickness/speckle/coverage crispness proxies, numpy-only
erosion) and loop_drift (odometry self-consistency), reuses path_length.
- Each param set replays in its own freshly-launched stack under a random
ROS_DOMAIN_ID, so the harness can never reach a live robot and sequential runs
can't leak into each other. replayer.py streams the bag's scan/tf/tf_static
back in sim-time order (driving /clock from bag stamps), stripping the tf edge
the node under test republishes (map->odom for slam) so the recorded copy
can't fight the fresh one; in slam mode the recorded odom->base is fed to slam
as its odometry prior, exactly as on the robot.
- replay_stack_launch.py references env packages only (slam_toolbox /
kinematic_icp / nav2_lifecycle_manager), so the harness runs against a bag
without the workspace being built.
- Maps render with cv2 (map_saver convention: walls black, unknown grey, north
up) with the trajectory overlaid.
- Limitations documented honestly (README + report): proxies not error measures
— loop drift needs an actual loop, crispness catches blur/noise not a
confidently-wrong map, no absolute ATE without a survey.
Verified: `pixi run lint` passes; metrics + bag-replay unit tests green (no ROS,
isolated); demonstrated end-to-end on real bag 20260706_135320 — baseline vs
sparse_no_loop produced a discriminating report (baseline explored 25.7 vs
21.8 m2 with 0.040 vs 0.072 speckle) with rendered maps.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_019WNpDuxGXctLchGh6X4sDN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bag-replay scoring harness: replay a real recorded mapping bag through SLAM (or kinematic_icp) under N parameter sets and score with truth-free proxies, producing a side-by-side comparison report (metrics table + rendered maps). The reality check complementing the sim benchmark (which needs Gazebo ground truth).
New: pixi run bag-replay in mote_simulation/tools/bag_replay/ (replay.py orchestrator, replayer.py ROS client, replay_stack_launch.py, render.py, report.py, examples/sparse_no_loop.yaml, README, test_bag_replay.py). Extends the sim benchmark's ROS-free metrics.py (reused, not forked) with truth-free map_quality (numpy-only crispness: wall-thickness via erosion, speckle, coverage) and loop_drift, plus tests.
Design: each param set replays in its own freshly-launched stack under a random ROS_DOMAIN_ID (can never reach a live robot; sequential runs can't leak). replayer.py streams the bag's scan/tf/tf_static back in sim-time order, driving /clock from bag stamps, and strips the tf edge the node-under-test republishes (map->odom for slam) so the recorded copy can't fight the fresh node; in slam mode the recorded odom->base is fed to slam as its prior, exactly as on the robot. replay_stack_launch.py uses env packages only, so it runs without a workspace build. Limitations documented honestly (proxies not error measures; loop drift needs an actual loop; crispness catches blur/noise not a confidently-wrong map; no absolute ATE without a survey).
Acceptance: one command (bag + N param files) -> report.md + run.json + per-set map.png/npz/series.json, all saved. Demonstrated end-to-end on real bag 20260706_135320: baseline vs sparse_no_loop gave a discriminating report (baseline explored 25.7 vs 21.8 m2 with 0.040 vs 0.072 speckle). No live-robot path from tests (ROS-free + isolated domain id). pixi run lint passes; metrics (10) + bag-replay (4) unit tests green.
Note: a settle-loop bug that raced sim-time to ~1200s (12k phantom samples) was found and fixed by wall-pacing the post-bag settle and making trajectory sampling gap-robust.