Skip to content

Active-search behavior + dynamic detection vocabulary when a target is unresolved #12

Description

@adharshvenkat

Problem

The semantic map is only ever populated by passive detection while driving, using a fixed launch-time prompt list (DEFAULT_PROMPTS in semantic_detector.py). Two compounding limitations:

  1. Static detection vocabulary - the GroundingDINO prompt list is a launch-time ROS parameter, not derived from the task command, so the detector can never be asked to look for something outside its preset list - even though GroundingDINO itself supports arbitrary open-vocabulary prompts.
  2. No active-search behavior - if a target isn't already in the semantic map (never observed, or outside the prompt vocabulary), there's no fallback exploration. The planner's own empty-plan fallback message literally says "no perceived labels yet, explore first" (llm_planner.py), but no explore behavior is implemented anywhere - it's just a log/summary string.

Proposed design

  • Feed the task command's parsed target words into the detector's prompt list at runtime (dynamic vocabulary), rather than relying solely on the static launch-time list.
  • Add an active-search / explore task type the planner can emit when a target is genuinely unresolved (also suggested in Planner substitutes a wrong known label instead of reporting the target unresolvable #8) - drive the robot through unexplored/likely areas looking for the target before giving up.

Dependency

Blocked on #8: the planner doesn't reliably report "unresolvable" today (it can substitute a wrong known label instead), so search can't be triggered correctly until that's fixed first.

Related

Acceptance criteria

  • A command referencing an object outside the current semantic map and outside the static prompt list triggers an active search rather than an immediate failure or wrong-label substitution.
  • Detection vocabulary can be extended at runtime from the task command's target labels.
  • Behavior validated with at least one scripted repro (target genuinely never seen before).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions