You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The semantic map is only ever populated by passive detection while driving, using a fixed launch-time prompt list (DEFAULT_PROMPTS in semantic_detector.py). Two compounding limitations:
Static detection vocabulary - the GroundingDINO prompt list is a launch-time ROS parameter, not derived from the task command, so the detector can never be asked to look for something outside its preset list - even though GroundingDINO itself supports arbitrary open-vocabulary prompts.
No active-search behavior - if a target isn't already in the semantic map (never observed, or outside the prompt vocabulary), there's no fallback exploration. The planner's own empty-plan fallback message literally says "no perceived labels yet, explore first" (llm_planner.py), but no explore behavior is implemented anywhere - it's just a log/summary string.
Proposed design
Feed the task command's parsed target words into the detector's prompt list at runtime (dynamic vocabulary), rather than relying solely on the static launch-time list.
Blocked on #8: the planner doesn't reliably report "unresolvable" today (it can substitute a wrong known label instead), so search can't be triggered correctly until that's fixed first.
A command referencing an object outside the current semantic map and outside the static prompt list triggers an active search rather than an immediate failure or wrong-label substitution.
Detection vocabulary can be extended at runtime from the task command's target labels.
Behavior validated with at least one scripted repro (target genuinely never seen before).
Problem
The semantic map is only ever populated by passive detection while driving, using a fixed launch-time prompt list (
DEFAULT_PROMPTSinsemantic_detector.py). Two compounding limitations:"no perceived labels yet, explore first"(llm_planner.py), but no explore behavior is implemented anywhere - it's just a log/summary string.Proposed design
Dependency
Blocked on #8: the planner doesn't reliably report "unresolvable" today (it can substitute a wrong known label instead), so search can't be triggered correctly until that's fixed first.
Related
Acceptance criteria