The Comprehensive Ecosystem for ROS 2 Navigation & SLAM Benchmarking
BenchBot is a complete lifecycle ecosystem for professional ROS 2 development. From initial integration to final validation, it empowers teams to master their navigation stack.
- 🧩 Integrate: Plug in any SLAM algorithm with a modular plugin system.
- ⚙️ Optimize: Use the AI Auto-Tuner to automatically discover the perfect parameters for your robot.
- 📈 Monitor: Track evolution with industrial-grade metrics (ATE, SSIM, Coverage) over time.
- ✅ Validate: Ensure production readiness with automated CI/CD pipelines and reproducible Docker environments.
benchmark_demo.mp4
- Multi-Simulator: Switch between Gazebo (Classic) and O3DE (PhysX 5.0).
- High-Fidelity Physics: "Sim-to-Real" tuning with realistic wheel slip, friction, and IMU/Lidar noise (Details).
- Advanced Metrics:
- Trajectory: ATE (Absolute Trajectory Error) with automatic alignment
- Map Quality: Coverage %, IoU, SSIM (Structural Similarity), Wall Thickness Analysis
- System: Real-time CPU Usage %, Max RAM (MB)
- Anomaly Detection: Stuck robot, TF jumps, massive drift detection
- Docker Support: Full containerization for 100% reproducible benchmarks across environments
- Automated Reporting: One-click PDF generation with trajectory plots, metrics tables, and health indicators
- Centralized Logging: Rotating file logs, colored console output, automatic crash reports (JSON)
- Comprehensive Testing: 70+ unit tests with pytest, 80%+ code coverage, CI/CD ready
- Headless CI: Run full benchmarks on servers without a display (
runner/run_matrix.py). - Modern GUI: Dashboard, Analysis Comparison, 3D Visualizer, Robot Manager, and Settings.
- Intelligent Comparison: Overlay up to 3 trajectories with ground truth and detailed anomaly tooltips.
- 3D Real-Time Monitoring: Live LIDAR point cloud, robot pose, and trajectory with "Follow Robot" camera mode.
- Stress Testing: Dynamic sensor degradation (LIDAR noise/range) and actuator limiting.
See the full Demo Report for interactive analysis.
Here is a sample comparison between SLAM Toolbox, Cartographer, and GMapping in a simulated office environment.
| Metric | SLAM Toolbox | Cartographer | GMapping |
|---|---|---|---|
| ATE RMSE (m) | 0.0130 🥇 | 0.0151 🥈 | 0.0197 🥉 |
| Map SSIM | 0.9175 | 0.8517 | 0.9212 🥇 |
| CPU Usage | 196% 🥇 | 1476% | 1284% |
| RAM Usage | 1212 MB | 1068 MB | 1027 MB 🥇 |
| Map IoU | 0.1457 | 0.0079 | 0.1695 🥇 |
Analysis: SLAM Toolbox offers the best trajectory accuracy (lowest ATE), while GMapping produces the highest quality maps (best SSIM & IoU) but consumes significantly more resources. Cartographer struggled with loop closure in this specific scenario, resulting in lower map quality scores.
Full documentation is available at https://benchbot.guillaumeschneider.fr.
- 🚀 Quick Start
- 📦 Installation Guide
- 📊 Metrics Explained
- 🎮 Simulators (Gazebo vs O3DE)
- 🧠 AI Auto-Tuner
- 🖥️ Headless Mode (CI/CD)
- OS: Ubuntu 22.04 LTS
- ROS 2: Humble Hawksbill
- Python: 3.10+
# Clone the repository
git clone https://github.com/guillaume-schneider/BenchBot.git
cd BenchBot
# Run automated installer
./install.shFor detailed instructions, see the Installation Guide.
We welcome contributions! Please see our Contributing Guide for details.
This project is licensed under the MIT License - see the LICENSE file for details.