Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ LLM_MODEL=qwen2.5:3b
# Ollama server URL (used only when LLM_PROVIDER=ollama)
OLLAMA_HOST=http://localhost:11434

# API key required for anthropic/openai/xai, leave blank for ollama
# API key: required for anthropic/openai/xai, leave blank for ollama
LLM_API_KEY=

# Override the OpenAI-compatible base URL (used by openai/xai; xai auto-defaults
Expand Down
14 changes: 5 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# ─────────────────────────────────────────────────────────────────────────────
# Stage 1: builder
# Installs build tools, compiles GroundingDINO C++ extensions, builds the
# natural_nav ROS2 package. Nothing from this stage leaks into runtime.
# ─────────────────────────────────────────────────────────────────────────────
FROM osrf/ros:jazzy-desktop AS builder

ENV DEBIAN_FRONTEND=noninteractive
Expand All @@ -25,7 +23,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \

# CUDA 12.8 toolkit (nvcc) from NVIDIA's repo. Ubuntu's packaged nvidia-cuda-toolkit
# is only 12.0 and cannot target consumer Blackwell (sm_120); GroundingDINO's custom
# ms_deform_attn op must be compiled with nvcc >= 12.8. Builder-only the runtime
# ms_deform_attn op must be compiled with nvcc >= 12.8. Builder-only: the runtime
# stage copies just the built artifacts, so none of this toolkit lands in the image.
RUN wget -qO /tmp/cuda-keyring.deb \
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb \
Expand All @@ -42,14 +40,14 @@ ENV FORCE_CUDA=1
# +PTX embeds forward-compatible PTX so the op still loads on newer archs.
ENV TORCH_CUDA_ARCH_LIST="12.0+PTX"

# Step 1: torch must be installed before GroundingDINO its setup.py imports torch.
# Step 1: torch must be installed before GroundingDINO, since its setup.py imports torch.
# cu128 wheels (torch >= 2.7) are the first with Blackwell/sm_120 kernels; the old
# cu124/2.6 wheels top out at sm_90 and crash on the RTX 50-series at runtime.
RUN pip install --no-cache-dir --break-system-packages \
--index-url https://download.pytorch.org/whl/cu128 \
"torch==2.7.*" "torchvision==0.22.*"

# Step 2a: deps that may try to upgrade torch installed with explicit cu128 index
# Step 2a: deps that may try to upgrade torch, installed with explicit cu128 index
# so any transitive torch reinstall keeps the matching CUDA wheel.
# --ignore-installed numpy: apt-managed numpy can't be removed by pip; install ours alongside.
RUN pip install --no-cache-dir --break-system-packages --ignore-installed numpy \
Expand Down Expand Up @@ -92,17 +90,15 @@ RUN . /opt/ros/jazzy/setup.sh && \
--packages-select natural_nav


# ─────────────────────────────────────────────────────────────────────────────
# Stage 2: runtime
# Clean base copies only built artifacts from builder.
# Clean base: copies only built artifacts from builder.
# No compilers, no pip cache, no build-essential.
# ─────────────────────────────────────────────────────────────────────────────
FROM osrf/ros:jazzy-desktop AS runtime

ENV DEBIAN_FRONTEND=noninteractive

# Runtime deps only (no build tools)
# ros:jazzy-desktop base already includes Gazebo we add robot-specific pkgs
# ros:jazzy-desktop base already includes Gazebo; we add robot-specific pkgs
RUN apt-get update && apt-get install -y --no-install-recommends \
python3-pip \
ros-jazzy-navigation2 \
Expand Down
14 changes: 6 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@

> *"Go inspect the shelf next to the pallet."*

NaturalNav takes a natural-language command, decomposes it with an LLM into a structured task graph, and dispatches navigation goals to a TurtleBot4 in a Gazebo warehouse — but task targets aren't hardcoded coordinates. They're resolved through an **open-vocabulary semantic map** the robot builds online from its RGBD camera using GroundingDINO. The whole stack runs in `docker compose up`.

Mobile-robot project in a multi-modality portfolio focused on perception, planning, and reasoning.
NaturalNav takes a natural-language command, decomposes it with an LLM into a structured task graph, and dispatches navigation goals to a TurtleBot4 in a Gazebo warehouse. Task targets aren't hardcoded coordinates: they're resolved through an **open-vocabulary semantic map** the robot builds online from its RGBD camera using GroundingDINO. The full stack (Gazebo, Nav2, perception, and LLM planning) is fully Dockerized with GPU passthrough.

---

Expand Down Expand Up @@ -103,24 +101,24 @@ git clone https://github.com/adharshvenkat/natural_nav.git
cd natural_nav

cp .env.example .env
# edit .env pick LLM_PROVIDER, paste your API key, etc. (defaults to local Ollama)
# edit .env: pick LLM_PROVIDER, paste your API key, etc. (defaults to local Ollama)

xhost +local:docker # let the container draw on your X display
docker compose up -d ollama # start local LLM (skip if using a cloud provider)
./scripts/setup_ollama.sh # pull qwen2.5:3b into the ollama volume (one-time)
./scripts/setup_groundingdino.sh # pull GroundingDINO weights into the volume (one-time)

# Terminal 1 Gazebo + Nav2 + RViz
# Terminal 1: Gazebo + Nav2 + RViz
docker compose run --rm --name nn naturalnav \
ros2 launch natural_nav simulation.launch.py
```

Gazebo + RViz will open. In RViz, click **2D Pose Estimate** and place the robot pose roughly where it sits in Gazebo so AMCL localizes.

Then start the NaturalNav nodes (planner, executor, semantic detector) — the `simulation.launch.py` above only brings up the sim and Nav2:
Then start the NaturalNav nodes (planner, executor, semantic detector). The `simulation.launch.py` above only brings up the sim and Nav2:

```bash
# Terminal 2 LLM planner + task executor + semantic detector
# Terminal 2: LLM planner + task executor + semantic detector
docker exec -it nn /entrypoint.sh \
ros2 launch natural_nav naturalnav.launch.py
```
Expand Down Expand Up @@ -189,7 +187,7 @@ natural_nav/

## Configuration

`.env` (gitignored copy from `.env.example`):
`.env` (gitignored, copy from `.env.example`):

```bash
LLM_PROVIDER=ollama # ollama | anthropic | openai | xai
Expand Down
20 changes: 10 additions & 10 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,34 +7,34 @@ services:
image: naturalnav:latest
container_name: naturalnav

# ── LLM configuration ────────────────────────────────────────────────────
# LLM configuration
environment:
- LLM_PROVIDER=${LLM_PROVIDER:-ollama}
- LLM_MODEL=${LLM_MODEL:-qwen2.5:3b}
- OLLAMA_HOST=${OLLAMA_HOST:-http://localhost:11434}
- LLM_API_KEY=${LLM_API_KEY:-}
- LLM_BASE_URL=${LLM_BASE_URL:-}
# X11 forwarding lets Gazebo render on the host display
# X11 forwarding: lets Gazebo render on the host display
- DISPLAY=${DISPLAY}
# NVIDIA GPU access for Gazebo sensor rendering
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=all
- __NV_PRIME_RENDER_OFFLOAD=1
- __GLX_VENDOR_LIBRARY_NAME=nvidia

# ── Volume mounts ─────────────────────────────────────────────────────────
# Volume mounts
volumes:
# X11 socket required for Gazebo GUI
# X11 socket, required for Gazebo GUI
- /tmp/.X11-unix:/tmp/.X11-unix
# Model weights cache avoids re-downloading GroundingDINO weights
# Model weights cache, avoids re-downloading GroundingDINO weights
- naturalnav_weights:/root/.cache/naturalnav
# Gazebo fuel cache avoids re-downloading warehouse models every run
# Gazebo fuel cache, avoids re-downloading warehouse models every run
- gz_fuel_cache:/root/.gz/fuel

# ── Networking ────────────────────────────────────────────────────────────
# Networking
network_mode: host

# ── GPU (NVIDIA Container Toolkit) ────────────────────────────────────────
# GPU (NVIDIA Container Toolkit)
runtime: nvidia
deploy:
resources:
Expand All @@ -44,14 +44,14 @@ services:
count: all
capabilities: [gpu]

# ── Runtime ───────────────────────────────────────────────────────────────
# Runtime
privileged: false
restart: unless-stopped

stdin_open: true
tty: true

# ── Local LLM server (Ollama) ─────────────────────────────────────────────
# Local LLM server (Ollama)
ollama:
image: ollama/ollama:latest
container_name: ollama
Expand Down
6 changes: 3 additions & 3 deletions src/natural_nav/natural_nav/fleet_orchestrator.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ def __init__(self):
self.get_logger().info(
f'Task Executor ready, waiting on Nav2 action {nav_action!r}')

# ── subscriptions ────────────────────────────────────────────────────────
# subscriptions

def _on_task_graph(self, msg: String):
try:
Expand Down Expand Up @@ -123,7 +123,7 @@ def _on_semantic_map(self, msg: String):
except (json.JSONDecodeError, ValueError) as e:
self.get_logger().warn(f'Bad semantic map snapshot: {e}')

# ── execution loop ───────────────────────────────────────────────────────
# execution loop

def _tick(self):
if not self._tasks or self._active_task_id is not None:
Expand Down Expand Up @@ -256,7 +256,7 @@ def _request_replan(self, failed_task: Task):
self.get_logger().info(
f'Requested replan for task {failed_task.id}: {failed_task.failure_reason}')

# ── helpers ──────────────────────────────────────────────────────────────
# helpers

def _publish_status(self):
snapshot = {
Expand Down
12 changes: 6 additions & 6 deletions src/natural_nav/natural_nav/llm_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
user message, and returns the model's text response.

Configured via environment / ROS params:
LLM_PROVIDER 'anthropic', 'openai', 'xai', or 'ollama'
LLM_API_KEY API key (not needed for ollama)
LLM_MODEL model name (e.g. claude-opus-4-8, gpt-4o, grok-4.3, qwen2.5:3b)
OLLAMA_HOST ollama server URL (default http://localhost:11434)
LLM_BASE_URL override the OpenAI-compatible endpoint (openai/xai)
LLM_PROVIDER : 'anthropic', 'openai', 'xai', or 'ollama'
LLM_API_KEY : API key (not needed for ollama)
LLM_MODEL : model name (e.g. claude-opus-4-8, gpt-4o, grok-4.3, qwen2.5:3b)
OLLAMA_HOST : ollama server URL (default http://localhost:11434)
LLM_BASE_URL : override the OpenAI-compatible endpoint (openai/xai)
"""

from __future__ import annotations
Expand Down Expand Up @@ -100,7 +100,7 @@ def get_client(
return OllamaClient(model or 'qwen2.5:3b', host)

if not api_key:
raise ValueError(f'LLM_API_KEY not set required for provider {provider!r}')
raise ValueError(f'LLM_API_KEY not set, required for provider {provider!r}')

base_url = os.environ.get('LLM_BASE_URL', '') or None

Expand Down
12 changes: 6 additions & 6 deletions src/natural_nav/natural_nav/llm_planner.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@

CRITICAL: the "target" field MUST be one of the labels the robot has actually
perceived. Use ONLY labels from this list (case-insensitive, exact match
preferred the executor will fuzzy-match if needed):
preferred; the executor will fuzzy-match if needed):

{known_labels}

Expand Down Expand Up @@ -102,7 +102,7 @@ def __init__(self):

self.get_logger().info('LLM Planner ready')

# ── subscriptions ────────────────────────────────────────────────────────
# subscriptions

def _on_semantic_map(self, msg: String):
try:
Expand All @@ -128,7 +128,7 @@ def _on_replan_request(self, msg: String):
return
if not self._last_command:
self.get_logger().warn(
'Replan requested but no prior command ignoring')
'Replan requested but no prior command, ignoring')
return
# Pull the freshest known_labels from the request (executor sends them).
self._known_labels = sorted(ctx.get('known_labels', self._known_labels))
Expand All @@ -138,17 +138,17 @@ def _on_replan_request(self, msg: String):
f'{ctx.get("failed_task", {}).get("reason", "(no reason)")}')
self._plan(self._last_command)

# ── planning ─────────────────────────────────────────────────────────────
# planning

def _plan(self, command: str):
self._publish_status(f'planning: {command}')

if not self._known_labels:
self.get_logger().warn(
'No known labels in semantic map yet emitting empty plan. '
'No known labels in semantic map yet, emitting empty plan. '
'Drive the robot around to populate the map first.')
empty = {
'summary': 'no perceived labels yet explore first',
'summary': 'no perceived labels yet, explore first',
'tasks': [],
'replan_on_failure': False,
}
Expand Down
9 changes: 4 additions & 5 deletions src/natural_nav/natural_nav/projection.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""
Pixel-to-3D projection helpers.

Deliberately free of ROS and torch imports so the geometry the part most
likely to carry a silent bug can be unit-tested on a plain Python host
Deliberately free of ROS and torch imports so the geometry, the part most
likely to carry a silent bug, can be unit-tested on a plain Python host
without the GPU sim. semantic_detector imports these; the test suite exercises
them directly.

Expand Down Expand Up @@ -61,9 +61,8 @@ def unproject(
"""Back-project pixel (u, v) at metric depth z to a point in the camera
optical frame via the pinhole model.

(u, v) and (cx, cy) must live in the *same* pixel space — i.e. unproject
with the intrinsics that match the coordinates you pass, not a rescaled
copy. semantic_detector samples depth in the depth image's resolution but
(u, v) and (cx, cy) must live in the *same* pixel space: unproject with
the intrinsics that match the coordinates you pass, not a rescaled copy. semantic_detector samples depth in the depth image's resolution but
unprojects in the camera_info (RGB) resolution, keeping this consistent.
"""
x = (u - cx) * z / fx
Expand Down
10 changes: 5 additions & 5 deletions src/natural_nav/natural_nav/semantic_detector.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ def __init__(self):
f'inference={1.0/detect_period:.1f} Hz, '
f'publish={1.0/publish_period:.1f} Hz')

# ── model load ──────────────────────────────────────────────────────────
# model load

def _load_model(self):
from groundingdino.util.inference import load_model
Expand All @@ -160,7 +160,7 @@ def _load_model(self):
self._gdino_model = load_model(str(config), str(weights), device=self._device)
self.get_logger().info('GroundingDINO loaded')

# ── subscriptions ────────────────────────────────────────────────────────
# subscriptions

def _on_rgb(self, msg: Image):
try:
Expand Down Expand Up @@ -194,7 +194,7 @@ def _on_camera_info(self, msg: CameraInfo):
width=msg.width, height=msg.height,
)

# ── inference + projection ──────────────────────────────────────────────
# inference + projection

def _run_inference(self):
if not self._inference_lock.acquire(blocking=False):
Expand Down Expand Up @@ -300,7 +300,7 @@ def _project_and_update(
if z is None:
continue
# Unproject in the RGB/camera_info pixel space using the matching
# intrinsics never the depth-scaled coords with RGB intrinsics.
# intrinsics, never the depth-scaled coords with RGB intrinsics.
x_cam, y_cam, z_cam = unproject(
det['cx'], det['cy'], z,
intr.fx, intr.fy, intr.cx, intr.cy)
Expand All @@ -317,7 +317,7 @@ def _project_and_update(
self._semantic_map.update(
det['label'], point_map.point.x, point_map.point.y, det['score'])

# ── publishing ──────────────────────────────────────────────────────────
# publishing

def _publish_map(self):
if len(self._semantic_map) == 0:
Expand Down
2 changes: 1 addition & 1 deletion src/natural_nav/test/test_projection.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""
Geometry tests for natural_nav.projection.

Pure numpyno ROS, no torch, no GPU sim so the depth-to-3D math that
Pure numpy: no ROS, no torch, no GPU sim, so the depth-to-3D math that
semantic_detector relies on is actually exercised in CI and on a dev host.
Runs under colcon test and standalone: `python3 -m pytest test/test_projection.py`.
"""
Expand Down
2 changes: 1 addition & 1 deletion src/natural_nav/worlds/naturalnav.world
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
<direction>-0.5 0.1 -0.9</direction>
</light>

<!-- ── Warehouse artifacts ───────────────────────────────────────────────
<!-- Warehouse artifacts.
Positioned to match location->pose map in fleet_orchestrator.py so
navigation targets align with visible, detectable objects. -->

Expand Down
Loading