I've completed a large-scale perception audit of Alpamayo 1 10B across 312 NuRec intersection scenes (61,150 timesteps). The full 23-page report has been shared privately with the team.
Objective: Determine whether the model’s perception claims are geometrically consistent with what the cameras can actually observe.
Methodology:
- Colocated AlpaSim deployment on Lambda Labs 4×H100 SXM5 (~$442 total compute)
- 4-camera configuration at 1080p/10Hz: front_tele, front_wide, cross_left, cross_right
- Camera-aware visibility model with angular occlusion culling and per-type motion thresholds
- Audit script extracts two data streams per scene (ASL Telemetry Ground Truth and CoC Text) and aligns them at 10Hz
- 7 verdicts: Agreement, Dynamic Hallucination, Phantom Object, Geometric Hallucination, Potential Under report, Planning Only, Unclassified.
- Bumper proximity scoring (Critical <2m, Warning <5m, Info ≥5m from front bumper)
- Range sensitivity validated across ±50% camera range variation (±0.4pp spread)
Key findings have been shared privately with the team. Happy to share more details once the team has had a chance to review.
This work reflects independent research conducted in a personal capacity.
Dennis Darrow
I've completed a large-scale perception audit of Alpamayo 1 10B across 312 NuRec intersection scenes (61,150 timesteps). The full 23-page report has been shared privately with the team.
Objective: Determine whether the model’s perception claims are geometrically consistent with what the cameras can actually observe.
Methodology:
Key findings have been shared privately with the team. Happy to share more details once the team has had a chance to review.
This work reflects independent research conducted in a personal capacity.
Dennis Darrow