diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md deleted file mode 100644 index e6a4e7982a..0000000000 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -linkTitle: "Debug a motion plan" -title: "Debug a motion plan" -weight: 45 -layout: "docs" -type: "docs" -description: "Use the 3D scene to inspect the frame system and obstacle geometry behind a motion plan that failed or produced unexpected results." ---- - -When a motion plan fails with a collision error or produces a path that -does not look right, the **3D SCENE** tab lets you inspect the static view -of the world the planner sees at the moment you look: configured frame -positions, collision geometries, and the arm's current pose. Most -motion planning failures come down to one of a few problems that are -immediately visible in 3D. - -The **3D SCENE** tab does not replay motion plans and has no timeline, -scrubber, or plan-playback control. It is an inspector of the -configured frame system and live component poses. If you need to watch -a plan execute over time, step the arm to intermediate joint positions -manually and reinspect. - -## Prerequisites - -- A machine with an arm or gantry configured and at least one frame defined. -- A motion plan that is failing or producing unexpected results. - -## Diagnose the problem - -### 1. Open the 3D SCENE tab - -Navigate to your machine in the [Viam app](https://app.viam.com) and -click the **3D SCENE** tab. The scene loads your current frame system -configuration. - -If your machine is online, the scene connects for live pose data. -If your machine is offline, the scene still renders the configured -frame positions. - -### 2. Check frame positions - -In the **World** panel in the upper-left, expand the tree and click -each component in turn. The Details panel on the right shows the -selected entity's **world position** and **world orientation**, plus -editable **local position** and **local orientation** relative to the -parent frame. Compare these values to your physical measurements. - -Three common mismatches to look for: - -- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. -- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. -- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. - -### 3. Check obstacle geometry - -Obstacles appear as translucent shapes in the scene and as child rows -under their parent frame in the **World** panel. Select each obstacle -from the tree to see its **geometry** type and **dimensions** in the -Details panel. - -Verify that: - -- Every physical obstacle in your workspace has a corresponding - geometry in the scene. -- Each geometry covers the actual physical object. If a box geometry - is too small, the planner will find paths that clip the real - obstacle. -- Geometries are positioned correctly. A table surface defined at - `z: 0` when the arm base is at `z: 500` will not protect against - table collisions. - -If obstacles are missing or misplaced, see -[Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). - -### 4. Look for impossible targets - -If the motion plan target is outside the arm's reach or inside an -obstacle, the planner cannot find a path. - -In the **World** panel, expand the arm and select its tip link or the -gripper frame (whichever is configured as the motion target). Read the -**world position** and compare it to the target pose your code -commanded. Then mentally place the target: is it inside an obstacle -geometry? Is it further than the arm can reach from its base? - -If the target is inside an obstacle geometry, either move the target -or adjust the obstacle definition. - -### 5. Check for self-collision geometry - -Some arm models include collision geometry for each link. If a motion -plan fails with a self-collision error, inspect the arm's link -geometries for overlap in the current configuration. - -If the arm renders without collision geometry, the feature may be -disabled: open **Settings → Scene → Arm Models** and verify that the -rendering mode includes colliders. - -Self-collisions can happen when wrist joints are commanded to -positions that bring adjacent links too close together. If the -overlap is a modeling artifact rather than a real collision, allow -the frame pair with -[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). - -## Common causes of motion plan failures - -| Symptom | Likely cause | What to check in 3D SCENE | -| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | -| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | -| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | -| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | -| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | -| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | diff --git a/docs/motion-planning/_index.md b/docs/motion-planning/_index.md index fd457100fe..915fd0b515 100644 --- a/docs/motion-planning/_index.md +++ b/docs/motion-planning/_index.md @@ -80,7 +80,7 @@ returns a collision-free path from the current pose to your target. {{< cards >}} {{% card link="/motion-planning/frame-system/" noimage="true" %}} {{% card link="/motion-planning/obstacles/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/" noimage="true" %}} +{{% card link="/visualization/3d-scene/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/" noimage="true" %}} {{< /cards >}} @@ -89,7 +89,7 @@ returns a collision-free path from the current pose to your target. {{< cards >}} {{% card link="/motion-planning/move-gantry/" noimage="true" %}} {{% card link="/motion-planning/verify-a-plan/" noimage="true" %}} -{{% card link="/motion-planning/debug-motion-with-cli/" noimage="true" %}} +{{% card link="/motion-planning/debug-motion-plan/" noimage="true" %}} {{< /cards >}} ## Concept pages diff --git a/docs/motion-planning/debug-motion-plan.md b/docs/motion-planning/debug-motion-plan.md new file mode 100644 index 0000000000..553622e7df --- /dev/null +++ b/docs/motion-planning/debug-motion-plan.md @@ -0,0 +1,205 @@ +--- +linkTitle: "Debug a motion plan" +title: "Debug a motion plan" +weight: 84 +layout: "docs" +type: "docs" +description: "Find why a motion plan failed or moved unexpectedly by checking frames, obstacles, and reach in the 3D scene or from the Viam CLI." +aliases: + - /motion-planning/3d-scene/debug-motion-plan/ + - /motion-planning/debug-motion-with-cli/ + - /motion-planning/motion-how-to/debug-motion-with-cli/ +--- + +When a motion plan fails with a collision error, or the arm ends up somewhere unexpected, +the first question is the same: is the frame system wrong, or is the plan wrong? You can +answer it two ways. If your machine has the **3D SCENE** tab, inspect the world the planner +sees by eye. From any shell, the Viam CLI reports the same frame poses and tests +reachability. This page covers both. + +To render the plan's trajectory itself, the path the arm takes from start to goal, see +[Visualize a motion plan](/motion-planning/visualize-a-motion-plan/). + +## Prerequisites + +- A machine with an arm or gantry configured and at least one frame defined. +- A motion plan that is failing or producing unexpected results. +- For the CLI checks: the Viam CLI installed and authenticated, and the `--part` identifier + for the machine part you want to inspect. + +## In the 3D scene + +Open the **3D SCENE** tab on your machine's page in the [Viam app](https://app.viam.com). It +loads your frame system configuration and, when the machine is online, connects for live +pose data. Work through the checks below; most planning failures show up in one of them. + +### Check frame positions + +In the **World** panel in the upper-left, expand the tree and click each component in turn. +The Details panel on the right shows the selected entity's **world position** and **world +orientation**, plus editable **local position** and **local orientation** relative to the +parent frame. Compare these values to your physical measurements. + +Three common mismatches to look for: + +- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. +- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. +- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. + +### Check obstacle geometry + +Obstacles appear as translucent shapes in the scene and as child rows under their parent +frame in the **World** panel. Select each obstacle from the tree to see its **geometry** type +and **dimensions** in the Details panel. + +Verify that: + +- Every physical obstacle in your workspace has a corresponding geometry in the scene. +- Each geometry covers the actual physical object. If a box geometry is too small, the + planner will find paths that clip the real obstacle. +- Geometries are positioned correctly. A table surface defined at `z: 0` when the arm base + is at `z: 500` will not protect against table collisions. + +If obstacles are missing or misplaced, see +[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). + +### Look for impossible targets + +If the motion plan target is outside the arm's reach or inside an obstacle, the planner +cannot find a path. + +In the **World** panel, expand the arm and select its tip link or the gripper frame +(whichever is configured as the motion target). Read the **world position** and compare it to +the target pose your code commanded. Then place the target: is it inside an obstacle +geometry? Is it further than the arm can reach from its base? + +If the target is inside an obstacle geometry, either move the target or adjust the obstacle +definition. + +### Check for self-collision geometry + +Some arm models include collision geometry for each link. If a motion plan fails with a +self-collision error, inspect the arm's link geometries for overlap in the current +configuration. + +If the arm renders without collision geometry, the feature may be disabled: open **Settings → +Scene → Arm Models** and verify that the rendering mode includes colliders. + +Self-collisions can happen when wrist joints are commanded to positions that bring adjacent +links too close together. If the overlap is a modeling artifact rather than a real collision, +allow the frame pair with +[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). + +## With the CLI + +When the 3D scene tab is not available, or you want to script the checks, the Viam CLI +reports the same information from the shell. Three commands cover most cases: `print-config` +dumps the configured frame tree, `print-status` prints every frame's current world-frame +pose, and `set-pose` drives a component to a pose to test reachability. For the full flag +reference, see [Motion CLI commands](/motion-planning/reference/cli-commands/). + +### Frames are in the wrong place + +The arm reports reaching `x=300, y=200` but physically sits elsewhere, or the +scene shows the gripper off to the side of the arm. Both symptoms point to a frame +configuration that disagrees with the physical setup. Dump the configured frame tree: + +```sh +viam machines part motion print-config --part "my-machine-main" +``` + +Each frame part shows its name, parent, translation, and orientation. Compare against your +JSON configuration and physical measurements. Common issues: a **wrong parent** (a component +parented to the world frame instead of the arm, or vice versa), **wrong units** (centimeters +instead of millimeters), or a **missing frame** (a component with no entry, usually because +its frame configuration did not save). + +Then check the live world-frame poses: + +```sh +viam machines part motion print-status --part "my-machine-main" +``` + +`print-status` prints one line per frame part with its computed world-frame pose: + +```text + my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 + my-gripper : X: 0.00 Y: 0.00 Z: 110.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 90.00 +``` + +A pose that does not match where the component physically sits means the frame configuration +is wrong. Edit the configuration in the Viam app, then run both commands again to confirm the +change took effect. + +### Find where one component is + +To read a single component's pose without the rest of the frame tree: + +```sh +viam machines part motion get-pose \ + --part "my-machine-main" \ + --component "my-arm" +``` + +The output is the same single-line format as `print-status`, limited to the component. This +is useful for scripting: log it, compare between runs, or read the arm's position before and +after a physical move to check it matches what the arm reports. + +### Check whether a target pose is reachable + +Drive toward the target in small steps to find where planning fails. First read the current +pose with `get-pose`, then move a small distance: + +```sh +viam machines part motion set-pose \ + --part "my-machine-main" \ + --component "my-arm" \ + --x 100 +``` + +`set-pose` overrides only the fields you pass, so this moves the arm to `X=100` while keeping +the current Y, Z, and orientation. If this small step fails, the planner cannot reach any pose +near the current position, which usually means a configuration error rather than a +target-reachability issue. If it succeeds, increase the delta and work toward the pose you +want. The last successful pose and the first failing pose bracket the problem. If `set-pose` +fails at a pose that should be reachable, return to `print-config` and `print-status`: frame +configuration errors often show up as unexpected unreachability. + +### The arm moved to the wrong place after a motion call + +Your code called `motion.Move` or `arm.MoveToPosition` and the arm ended up somewhere other +than the commanded pose. Immediately after the motion, capture the actual final pose with +`get-pose` and compare it to the pose you commanded. Differences beyond the arm's positioning +tolerance suggest a frame configuration mismatch or a kinematics calibration problem. + +If `print-status` shows the arm where it physically is but that does not match the pose you +commanded, the pose may have been interpreted in an unexpected reference frame. Check the +`reference_frame` on the target `PoseInFrame`: the same `(x, y, z)` in the arm's frame and in +the world frame describes two different places. + +The CLI commands `print-status`, `get-pose`, and `set-pose` call the motion service's +`GetPose`, which is deprecated in favor of the frame system service's `GetPose`; the +commands and their output format are stable. +`set-pose` calls the motion service's `Move` and blocks until the motion finishes or fails, +returning a non-zero exit status with the error message on failure. + +## Common causes of motion plan failures + +| Symptom | Likely cause | What to check | +| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | +| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | +| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | +| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | +| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | +| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | + +## What's next + +- [Visualize a motion plan](/motion-planning/visualize-a-motion-plan/): + render the plan's trajectory and goals as custom visuals when a visual check is not enough. +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): + check obstacle geometry against the real workspace. +- [Motion CLI commands](/motion-planning/reference/cli-commands/): + the full flag reference for the motion commands. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. diff --git a/docs/motion-planning/debug-motion-with-cli.md b/docs/motion-planning/debug-motion-with-cli.md deleted file mode 100644 index a346c6e989..0000000000 --- a/docs/motion-planning/debug-motion-with-cli.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -linkTitle: "Debug with the CLI" -title: "Debug motion with the Viam CLI" -weight: 40 -layout: "docs" -type: "docs" -description: "Diagnose frame configuration and motion issues from the command line using the Viam CLI motion commands." -aliases: - - /motion-planning/motion-how-to/debug-motion-with-cli/ ---- - -A motion plan fails, or the arm ends up somewhere unexpected, and the first -question is always the same: is the frame system wrong, or is the plan wrong? -Answering that from application code requires writing instrumentation. The -Viam CLI answers it at the shell. Three commands cover most cases: -`print-config` dumps the configured frame tree, `print-status` prints every -frame's current world-frame pose, and `set-pose` drives a component to a pose -so you can isolate reachability. - -This guide walks through the most common debugging flows. For the -visual equivalent on machines where the **3D SCENE** tab is available, -see [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/). -For the dry command reference, see -[Motion CLI commands](/motion-planning/reference/cli-commands/). - -## Before you start - -- The Viam CLI installed and authenticated. -- A machine with configured components and at least one frame defined. -- The `--part` identifier for the machine part you want to inspect. - -## Symptom: frames are in the wrong place - -The arm reaches where it thinks is x=300, y=200, and it physically ends up -somewhere else. Or the **3D SCENE** tab shows the gripper dangling off to the -side of the arm. In both cases, the frame configuration does not match what -the machine actually does. Two CLI commands will tell you where the -disagreement is. - -### 1. Dump the configured frame system - -```sh -viam machines part motion print-config --part "my-machine-main" -``` - -Read through the output. Each frame part shows its name, parent, -translation, and orientation. Compare against your JSON configuration -and against physical measurements. - -Common issues you will catch at this step: - -- **Wrong parent**: a component you intended to attach to the arm is - parented to the world frame (or vice versa). -- **Wrong units**: translations expressed in centimeters rather than - millimeters. -- **Missing frames**: a component that should have a frame has no entry - in the output at all, which usually means its frame configuration did - not save. - -### 2. Check the live world-frame poses - -```sh -viam machines part motion print-status --part "my-machine-main" -``` - -`print-status` prints one line per frame part with its computed world-frame pose: - -```text - my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 - my-gripper : X: 0.00 Y: 0.00 Z: 110.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 90.00 -``` - -Compare these poses against where each component physically is. A -mismatch means the frame configuration is wrong. - -### 3. Adjust the config and re-check - -Edit the frame configuration in the Viam app. Run `print-config` and -`print-status` again to confirm the change took effect. - -## Symptom: I need to know where one component is right now - -You want a single component's pose without the rest of the frame tree. - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Output is the same single-line format as `print-status`, limited to -the specified component. - -This is useful for scripting: pipe the output to a log, compare between -runs, or read the arm's position before and after a physical move to -verify it matches what the arm reports. - -## Symptom: I am not sure if a target pose is reachable - -Write the target pose in world coordinates and try to drive there. - -### 1. Read the current pose - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Note the X, Y, Z values. - -### 2. Move a small distance first - -This small step isolates the problem. If it fails, the planner cannot reach -any pose near the current position, which usually means a configuration -error rather than a target-reachability issue. If it succeeds, the current -position is not the problem and you can work outward from there. - -```sh -viam machines part motion set-pose \ - --part "my-machine-main" \ - --component "my-arm" \ - --x 100 -``` - -`set-pose` overrides only the fields you pass. In this example, it -moves the arm to X=100 while keeping the current Y, Z, and orientation. - -### 3. Move toward the real target incrementally - -If the small step succeeds, increase the delta. Work toward the pose -you ultimately want. If an intermediate step fails, the last successful -pose and the first failing pose bracket the problem: the planner can -reach the first but not the second. - -### 4. Compare against the frame system - -If `set-pose` fails at a pose that should be reachable, return to -`print-config` and `print-status` to check whether the planner's view -of the arm's location matches reality. Frame configuration errors -often manifest as unexpected unreachability. - -## Symptom: the arm moved to the wrong place after a motion call - -Your code called `motion.Move` or `arm.MoveToPosition` and the arm -ended up somewhere other than the commanded pose. - -### 1. Capture the actual final pose - -Immediately after the motion completes: - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Compare this to the pose you commanded. Differences that exceed the -arm's positioning tolerance suggest either a frame configuration -mismatch or a kinematics calibration problem. - -### 2. Cross-check with arm joint values - -The arm's own `GetJointPositions` reports joint angles. For a built-in -arm model, you can verify those match the expected kinematic solution -for the commanded pose. A large offset between forward-kinematic -prediction and reality points to incorrect kinematic parameters. - -### 3. If frames look correct, look at the target - -If `print-status` shows the arm where it physically is, but that does -not match the pose you commanded, the pose you commanded may have been -interpreted in an unexpected reference frame. Double-check the -`reference_frame` on the target `PoseInFrame`: the same `(x, y, z)` in -the arm's frame and in the world frame describes two different places. - -## Limitations - -- **Internal RPC is deprecated but the CLI output is not.** - `print-status`, `get-pose`, and `set-pose` call `Motion.GetPose` under the - hood, which is deprecated in favor of `Robot.GetPose`. The CLI output - format and the commands themselves are stable; only the backing call is - scheduled to change. -- `set-pose` calls the motion service's `Move`, which blocks until the - motion completes or fails. Plan failures surface as the CLI returning - a non-zero exit status with an error message. Collect that message as - the starting point for the rest of your debugging. - -## What's next - -- [Motion CLI commands](/motion-planning/reference/cli-commands/): the - full flag reference. -- [Frame system](/motion-planning/frame-system/): the concept these - commands inspect. -- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): - the 3D-scene equivalent when your machine has the visualization tab. -- [How motion planning works](/motion-planning/how-planning-works/): - understand when the planner fails and what the failure means. diff --git a/docs/motion-planning/frame-system/camera-calibration.md b/docs/motion-planning/frame-system/camera-calibration.md index 23e741ab1a..187ef75774 100644 --- a/docs/motion-planning/frame-system/camera-calibration.md +++ b/docs/motion-planning/frame-system/camera-calibration.md @@ -233,10 +233,10 @@ fmt.Printf(" z=%.1f mm\n", pt.Z) If the computed position is within 10-20 mm of the measured position at a working distance of 500-1000 mm, your calibration is good. -For a visual sanity check, open the [3D SCENE tab](/motion-planning/3d-scene/). +For a visual sanity check, open the [3D SCENE tab](/visualization/3d-scene/). The camera frame should sit in the correct position and orientation relative to the arm, and any visible obstacles should appear in plausible locations. -See [Calibrate frame offsets](/motion-planning/3d-scene/calibrate-frame-offsets/) +See [Measuring between frames](/visualization/3d-scene/measuring-between-frames/) for the full workflow. ## Troubleshooting diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index ad2c73ca1f..4ce305ec07 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -258,7 +258,7 @@ different things, and only one affects a `Move` call. plan this motion, and they are gone when the call returns. The transform in the previous section is a `WorldState` transform, so it shapes the plan. - The world state store service is a separate service that holds transforms for - client-side visualization in the [3D scene](/motion-planning/3d-scene/). + client-side visualization in the [3D scene](/visualization/3d-scene/). To change where the arm goes, put the transform in the `WorldState` you pass to `Move`, as shown above. To render a custom visual that leaves planning unchanged, @@ -274,7 +274,7 @@ same axes drawn in the diagrams on this page. A frame you add in code with a `WorldState` transform lasts only for that `Move` call, so it stays out of the 3D scene by default. To draw a code-defined frame in -the [3D scene](/motion-planning/3d-scene/), publish it through a world state store +the [3D scene](/visualization/3d-scene/), publish it through a world state store service. A module that implements this service holds your transforms and streams them to the scene, which renders a frame with no geometry as a set of coordinate axes alongside the static frames. diff --git a/docs/motion-planning/frame-system/overview.md b/docs/motion-planning/frame-system/overview.md index 20fc3c9720..fb6a2bd9b4 100644 --- a/docs/motion-planning/frame-system/overview.md +++ b/docs/motion-planning/frame-system/overview.md @@ -195,7 +195,7 @@ A typical frame configuration looks like this: } ``` -After saving, the [3D SCENE tab](/motion-planning/3d-scene/) renders the +After saving, the [3D SCENE tab](/visualization/3d-scene/) renders the frame at its computed world pose so you can verify visually. For hardware-specific walkthroughs (table-mounted arm with gripper and camera, mobile manipulator, mobile base with sensors), see the cards at @@ -271,7 +271,7 @@ Common uses: ## Verify visually with the 3D SCENE tab -The [3D SCENE tab](/motion-planning/3d-scene/) renders your frame hierarchy in +The [3D SCENE tab](/visualization/3d-scene/) renders your frame hierarchy in 3D: every configured component appears at its computed world pose, with geometry and axes drawn. Open it after a configuration change to confirm the arm is mounted on the table, the camera is above the arm, and any obstacles diff --git a/docs/motion-planning/how-planning-works.md b/docs/motion-planning/how-planning-works.md index 377d9db7b3..81f8c782ec 100644 --- a/docs/motion-planning/how-planning-works.md +++ b/docs/motion-planning/how-planning-works.md @@ -1,7 +1,7 @@ --- linkTitle: "How planning works" title: "How motion planning works" -weight: 90 +weight: 80 layout: "docs" type: "docs" description: "The algorithm Viam uses to plan robot arm motion, why it works for arms, and what to try when planning fails." diff --git a/docs/motion-planning/obstacles/allow-frame-collisions.md b/docs/motion-planning/obstacles/allow-frame-collisions.md index e272f5f6d7..3dfca37669 100644 --- a/docs/motion-planning/obstacles/allow-frame-collisions.md +++ b/docs/motion-planning/obstacles/allow-frame-collisions.md @@ -244,7 +244,7 @@ silencing them can mask real hardware problems. reject those. 3. Visually confirm the motion in the **3D SCENE** tab before running on hardware. See - [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/). + [Debug a motion plan](/motion-planning/debug-motion-plan/). ## Troubleshooting @@ -288,5 +288,5 @@ geometry is usually the right answer. other three constraint types (Linear, Orientation, Pseudolinear). - [Define obstacles](/motion-planning/obstacles/): the obstacle geometry that the planner checks against. -- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): +- [Debug a motion plan](/motion-planning/debug-motion-plan/): visualize frame overlaps in the **3D SCENE** tab. diff --git a/docs/motion-planning/obstacles/configure-workspace-obstacles.md b/docs/motion-planning/obstacles/configure-workspace-obstacles.md index 975260d13e..cb65d7f1be 100644 --- a/docs/motion-planning/obstacles/configure-workspace-obstacles.md +++ b/docs/motion-planning/obstacles/configure-workspace-obstacles.md @@ -327,7 +327,7 @@ For objects the robot grasps and releases dynamically, see [Attach and detach ge 3. Orbit the scene. Each configured obstacle appears as a translucent shape at the position you configured. Each component shows up under the **World** panel; click a component to see its pose and geometry in the **Details** panel. 4. Check coverage against the physical workspace. If the arm can reach past an obstacle into the physical object, the geometry is too small. -For the full visualization and verification workflow, see [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). +For the full visualization and verification workflow, see [Verify obstacles](/motion-planning/obstacles/verify-obstacles/). ## Try it @@ -377,6 +377,6 @@ Fix the geometry in the component's attributes and save again. ## What's next - [Plan collision-free paths](/motion-planning/obstacles/avoid-obstacles/): pass runtime obstacles through `WorldState` for objects that change between calls. -- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): full verification workflow in the **3D SCENE** tab. +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): full verification workflow in the **3D SCENE** tab. - [Attach and detach geometries](/motion-planning/obstacles/attach-detach-geometries/): attach a grasped object to the gripper frame for the duration of a pickup. - [Allow specific frames to collide](/motion-planning/obstacles/allow-frame-collisions/): permit expected contact between frames. diff --git a/docs/motion-planning/obstacles/overview.md b/docs/motion-planning/obstacles/overview.md index 9e7caaec04..fae2322071 100644 --- a/docs/motion-planning/obstacles/overview.md +++ b/docs/motion-planning/obstacles/overview.md @@ -151,6 +151,7 @@ Practical consequences: {{< cards >}} {{% card link="/motion-planning/obstacles/configure-workspace-obstacles/" noimage="true" %}} +{{% card link="/motion-planning/obstacles/verify-obstacles/" noimage="true" %}} {{% card link="/motion-planning/obstacles/avoid-obstacles/" noimage="true" %}} {{% card link="/motion-planning/obstacles/attach-detach-geometries/" noimage="true" %}} {{% card link="/motion-planning/obstacles/allow-frame-collisions/" noimage="true" %}} diff --git a/docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md b/docs/motion-planning/obstacles/verify-obstacles.md similarity index 88% rename from docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md rename to docs/motion-planning/obstacles/verify-obstacles.md index f74a90004e..92e6e9d6fe 100644 --- a/docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md +++ b/docs/motion-planning/obstacles/verify-obstacles.md @@ -1,13 +1,15 @@ --- linkTitle: "Verify obstacles" title: "Verify obstacles" -weight: 40 +weight: 25 layout: "docs" type: "docs" description: "Visualize and adjust obstacle geometry so the motion planner routes around physical objects." +aliases: + - /motion-planning/3d-scene/set-up-obstacle-avoidance/ --- -The motion planner avoids obstacles only if it knows about them, and it knows about them only as geometries you define: boxes, spheres, capsules, or cylinders positioned in the frame system. That definition is invisible in JSON. A box specified as `{x: 800, y: 1200, z: 20}` at some parent-relative translation either covers the table or it doesn't, and you can't tell which from the numbers. The **3D SCENE** tab lets you see what the planner sees, so you can check coverage before running a plan. +The motion planner routes around an obstacle only when a geometry represents it: a box, sphere, capsule, or cylinder positioned in the frame system. The **3D SCENE** tab renders each configured geometry in place, so you can confirm that a box defined as `{x: 800, y: 1200, z: 20}` at a parent-relative translation actually covers the table, and check coverage of the whole workspace, before you run a plan. ## Prerequisites diff --git a/docs/motion-planning/reference/cli-commands.md b/docs/motion-planning/reference/cli-commands.md index 7660a63b83..7169355a8f 100644 --- a/docs/motion-planning/reference/cli-commands.md +++ b/docs/motion-planning/reference/cli-commands.md @@ -150,11 +150,11 @@ position. - **Quick reachability test**: run `get-pose` to read the current pose, then run `set-pose` with a small offset to confirm the motion service can drive the arm. Start with small offsets; large ones risk collisions. For a step-by-step debugging workflow using these commands, see -[Debug motion with the CLI](/motion-planning/debug-motion-with-cli/). +[Debug motion with the CLI](/motion-planning/debug-motion-plan/). ## What's next -- [Debug motion with the CLI](/motion-planning/debug-motion-with-cli/): +- [Debug motion with the CLI](/motion-planning/debug-motion-plan/): step-by-step debugging using these commands. - [Frame system](/motion-planning/frame-system/): the concept the CLI inspects. diff --git a/docs/motion-planning/verify-a-plan.md b/docs/motion-planning/verify-a-plan.md index ded000a45b..967999893b 100644 --- a/docs/motion-planning/verify-a-plan.md +++ b/docs/motion-planning/verify-a-plan.md @@ -1,7 +1,7 @@ --- -linkTitle: "Verify a plan" +linkTitle: "Verify a plan before running it" title: "Verify a motion plan without executing it" -weight: 60 +weight: 82 layout: "docs" type: "docs" description: "Use armplanning.PlanMotion to compute and inspect a trajectory before the arm moves, and learn how to get the same plan over the motion service API." @@ -184,5 +184,5 @@ and the motion service is all you have. plan one trajectory through an ordered list of goals. - [How motion planning works](/motion-planning/how-planning-works/): why a plan can be infeasible and what to adjust. -- [Debug motion with the CLI](/motion-planning/debug-motion-with-cli/): +- [Debug motion with the CLI](/motion-planning/debug-motion-plan/): inspect frames and poses from the command line. diff --git a/docs/motion-planning/visualize-a-motion-plan.md b/docs/motion-planning/visualize-a-motion-plan.md new file mode 100644 index 0000000000..fa4fb90ed5 --- /dev/null +++ b/docs/motion-planning/visualize-a-motion-plan.md @@ -0,0 +1,163 @@ +--- +linkTitle: "Visualize a motion plan" +title: "Visualize a motion plan" +weight: 86 +layout: "docs" +type: "docs" +description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path against obstacles and reach." +aliases: + - /motion-planning/3d-scene/visualize-a-motion-plan/ +--- + +The **3D SCENE** tab renders your configured frame system and live component poses, and the +[visual checks for a failing plan](/motion-planning/debug-motion-plan/) find most +problems by eye. When those checks do not reveal the problem, or you want to see the +trajectory itself, publish the plan as **custom visuals** through a world state store +service. The scene then renders the path you can otherwise only read as numbers. + +This page shows how to turn a plan into transforms the scene can draw, and how to +use the rendered path to debug a plan that failed or moved unexpectedly. + +## Why publish the plan as custom visuals + +A plan is a sequence of joint configurations. Rendered in the scene, the path +shows immediately whether it clips an obstacle, swings wide, or aims at a +target outside the arm's reach. Publishing +the plan as world state store transforms puts the trajectory and goals in the +same 3D view as the frames and obstacle geometry the planner used, so you can see +the path and the world together. + +## Convert the trajectory into poses + +A plan's `Trajectory` is a sequence of joint configurations +(`FrameSystemInputs`), one per step. The scene places geometry by pose, so +convert each step into end-effector poses with the frame system. `ComputePoses` +takes a configuration and returns the pose of each frame: + +```go +var transforms []*commonpb.Transform +for i, step := range plan.Trajectory() { + poses, err := step.ComputePoses(fs) + if err != nil { + return err + } + gripperPose := poses["my-gripper"].Pose() + // stepMarker (next section) turns this pose into a visual. + transform, err := stepMarker(i, gripperPose) + if err != nil { + return err + } + transforms = append(transforms, transform) +} +// Serve transforms through a world state store service (last section). +``` + +Each `step` is the arm's configuration at that point in the trajectory, and +`poses["my-gripper"]` is where the gripper sits in that configuration. + +## Build transforms for the plan + +With a pose per step, build the visuals with the `draw` library: a marker per +trajectory step and a marker for each goal. Give each a stable UUID so the scene +can update them when you re-plan. + +```go +import ( + "github.com/viam-labs/motion-tools/draw" + commonpb "go.viam.com/api/common/v1" + "go.viam.com/rdk/spatialmath" +) + +// Distinct colors keep the trajectory and its goals apart in the scene. +var ( + stepColor = draw.NewColor(draw.WithName("blue")) + goalColor = draw.NewColor(draw.WithName("green")) +) + +func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { + id := fmt.Sprintf("step-%d", i) + // Build the sphere at the origin; WithPose below places it at the step pose. + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 5, id) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(stepColor)) + if err != nil { + return nil, err + } + // WithID derives a stable UUID from the string, so re-planning updates each + // marker in place instead of drawing a duplicate. + return drawn.Draw(id, draw.WithID(id), draw.WithPose(pose)) +} + +// goalMarker draws a destination pose. Unlike the trajectory poses, a goal pose is +// not a ComputePoses output: it is the destination you passed to the planner. +func goalMarker(i int, goalPose spatialmath.Pose) (*commonpb.Transform, error) { + id := fmt.Sprintf("goal-%d", i) + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 8, id) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(goalColor)) + if err != nil { + return nil, err + } + return drawn.Draw(id, draw.WithID(id), draw.WithPose(goalPose)) +} +``` + +The trajectory poses come from `ComputePoses`, but the goal pose is the destination +you passed to the planner. `goalMarker` draws it with a distinct color, so the target +stands out from the trajectory leading to it. + +## Serve the transforms to the scene + +Serve the transforms through a world state store service so the **3D SCENE** tab +renders them. Return your step and goal markers from the service's `ListUUIDs`, +`GetTransform`, and `StreamTransformChanges` methods, and the plan streams in alongside +the frames and obstacle geometry the scene already shows. For those methods, the +poll-and-update loop, and how a module pulls data from its dependencies, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## Diagnose a failed or surprising plan + +With the plan rendered, debugging becomes visual. Compare the trajectory against +the rest of the scene: + +- **Where does the path collide?** If a step marker passes through an obstacle + geometry, that is where the planner reports a collision. Check whether the + obstacle is real or an oversized geometry. +- **Does the goal fall outside the arm's reach?** If a goal marker sits far from + any reachable arm configuration, the goal is out of reach. Move the goal or + check the frame system. +- **Why the detour?** An unexpected route usually means an obstacle is forcing + the planner around it. Look for stray geometry between the start and goal. + +For checking the obstacle geometry itself, separate from the plan, see +[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). + +## When to visualize versus inspect or verify + +The 3D scene serves three distinct purposes, and it helps to keep them straight: + +- **Visualize a plan** (this page): publish the trajectory and goals as custom + visuals to see the path in context. +- **[Inspect static frames and geometry](/motion-planning/debug-motion-plan/)**: + use the stock scene to check frame positions and obstacle coverage with no plan involved. +- **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is + reachable and a path exists before you visualize or run anything. + +Visualization shows you _what the path looks like_; static inspection shows you +_what the world looks like_; feasibility checking tells you _whether a plan +exists at all_. Reach for the one that matches the question you are asking. + +## What's next + +- [Debug a motion plan](/motion-planning/debug-motion-plan/): + the no-code visual checks to try first. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + the world state store service that serves these transforms. +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): + check obstacle geometry against the real workspace. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. diff --git a/docs/reference/apis/services/world-state-store.md b/docs/reference/apis/services/world-state-store.md index 8edcd13159..f5d4868c3c 100644 --- a/docs/reference/apis/services/world-state-store.md +++ b/docs/reference/apis/services/world-state-store.md @@ -4,7 +4,7 @@ linkTitle: "World state store" weight: 90 type: "docs" tags: ["world_state_store", "services"] -description: "Retrieve a list of world objects." +description: "List, get, and stream the transforms a world state store service publishes for the 3D scene to draw." icon: true images: ["/icons/components/generic.svg"] date: "2025-09-12" @@ -13,8 +13,11 @@ aliases: - /dev/reference/apis/services/world-state-store/ --- -The world state store service API allows you to retrieve a list of world objects. -You can use this list to create custom visualizers to render spatial data related to a machine on the machine's **VISUALIZE** tab. +The world state store service API lets a client list, get, and stream the transforms a +world state store service publishes. The **3D SCENE** tab uses this API to render a +machine's [custom visuals](/visualization/visuals-and-collisions/), and a custom +visualizer you build can consume it the same way. To implement the service in a module, +see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/). The world state store service supports the following methods: diff --git a/docs/visualization/3d-scene/3d-scene-widgets.md b/docs/visualization/3d-scene/3d-scene-widgets.md new file mode 100644 index 0000000000..c5abb41317 --- /dev/null +++ b/docs/visualization/3d-scene/3d-scene-widgets.md @@ -0,0 +1,56 @@ +--- +linkTitle: "3D scene widgets" +title: "3D scene widgets" +weight: 40 +layout: "docs" +type: "docs" +description: "Overlay live data on the 3D scene with widgets: an arm's joint positions and a camera's live feed." +aliases: + - /visualization/3d-scene-tools/3d-scene-widgets/ +--- + +Widgets are floating panels you overlay on the 3D scene to read live data next to the 3D +view. Each widget is draggable and resizable. + +## Enable a widget + +Open the **Settings** panel (the gear icon in the scene toolbar) and select **Widgets**. +Toggle a widget on and it appears floating over the viewport; toggle it off to remove it. The +panel lists two kinds of widget: **Arm positions**, and a **Camera widgets** section with one +toggle per configured camera. A third widget, the [frame POV](#frame-pov), opens from the +Details panel instead. + +## Arm positions + +The **Arm positions** widget shows the live joint angles of an arm. It lists each joint by +index with its current position in degrees, and updates as the arm moves: + +| Joint | Position (degrees) | +| ----- | ------------------ | +| 0 | -0.00 | +| 1 | -11.29 | +| 2 | -20.05 | +| ... | ... | + +If your machine has more than one arm, a **Select arm** dropdown at the top of the widget +chooses which arm's joints to show. Use this widget to read an arm's current configuration +while you inspect its pose in the 3D view. + +## Camera widgets + +A camera widget shows a camera's live feed as a panel over the 3D scene, with its current +frame rate, for example `20.0fps`. Each configured camera has its own toggle under +**Camera widgets**. + +A resolution dropdown on the widget sets the feed size: **Default**, `1280x720`, `640x360`, +`320x180`, `160x90`, or `80x44`. A lower resolution costs less bandwidth, which helps when +you watch several cameras at once or work over a slow connection. For how a camera's data +appears in the scene itself, see [Cameras](/visualization/perception/cameras/). + +## Frame POV + +The frame POV widget renders the scene from a selected frame's perspective, so you can +check what a camera's view covers, or what an end effector approaches, from its configured +pose. Select an entity in the **World** panel, then click the camera icon (**View from this +frame**) in the Details panel. A panel titled **POV** opens for that frame; open one per +frame you want to watch. diff --git a/docs/visualization/3d-scene/_index.md b/docs/visualization/3d-scene/_index.md new file mode 100644 index 0000000000..0606c0d61a --- /dev/null +++ b/docs/visualization/3d-scene/_index.md @@ -0,0 +1,89 @@ +--- +linkTitle: "3D scene" +title: "Visualizing with the 3D scene" +weight: 5 +layout: "docs" +type: "docs" +description: "What the 3D scene renders, where each element comes from, and how built-in configuration content differs from custom visuals a module publishes at runtime." +aliases: + - /visualization/visualizing-with-the-3d-scene/ + - /visualization/3d-scene-tools/ + - /motion-planning/3d-scene/ +--- + +The **3D SCENE** tab renders your machine in an interactive 3D view: the frames of every +component, the geometry attached to them, point clouds from depth cameras, and any custom +visuals a module publishes while the machine runs. It turns configuration you would +otherwise read as JSON numbers into a picture you can inspect, so you can confirm a gripper +sits where its frame configuration places it or watch a motion plan against the obstacles +around it. + +This page describes what the scene shows, where each element comes from, and how the scene +stays current. For the tab's panels, navigation, and settings, see +[The 3D scene interface](/visualization/3d-scene/the-3d-scene-interface/). + +## What the scene renders + +The scene draws four kinds of element, each with its own appearance: + +- **Component frames** render as sets of red, green, and blue coordinate axes (X, Y, and + Z), one per component, positioned by each frame's translation and orientation. +- **Geometries** render as translucent shapes (a box, sphere, or capsule) at the frame they + are attached to. +- **Point clouds** from depth cameras render as colored point sets. +- **Custom visuals** render as whatever a module draws them as: markers, arrows, lines, + meshes, or extra geometries, published while the machine runs. + +## Where each element comes from + +Every element in the scene traces back to one source. The **World panel** lists every entity +in a tree rooted at the world frame, so you can select anything on screen and follow it back +to the component or module that produced it. + +| Element in the scene | Comes from | +| -------------------- | ----------------------------------------------------------- | +| Component frame | The frame system, from each component's frame configuration | +| Attached geometry | A component's `frame.geometry` in configuration | +| Point cloud | A depth camera, streamed live when the machine is online | +| Custom visual | A module, published through a world state store service | + +The first three appear because they are part of the machine's configuration, and the scene +reads that configuration whenever you open the tab. Custom visuals appear only when a module +publishes them. + +## Built-in content versus custom visuals + +The distinction between the sources matters because it tells you _where to make a change_: + +- **Built-in content** is the frame system and configured geometry. The scene shows it with + no code. To change a frame's position or a geometry's shape, edit the component's + configuration. +- **Custom visuals** are everything a module computes or senses at runtime: a detection, a + planned path, a sensor's obstacle readings. The scene shows them only while the module + publishes them. To change a custom visual, change the module's code. + +If an element looks wrong, this split tells you where to look: a misplaced frame or geometry +is a configuration problem, while a missing or stale custom visual is a module problem. + +## How the scene stays current + +Built-in content reflects your configuration and, when the machine is online, each +component's live pose. Custom visuals stay current a different way: a module publishes them +to a [world state store service](/reference/apis/services/world-state-store/), and the scene +subscribes to that service's stream of transform changes. As the module adds, updates, and +removes transforms, the scene applies each change to the one affected visual instead of +redrawing everything, so a busy scene keeps up as the underlying data changes. + +To publish your own custom visuals this way, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## What's next + +{{< cards >}} +{{% card link="/visualization/3d-scene/the-3d-scene-interface/" noimage="true" %}} +{{% card link="/visualization/3d-scene/measuring-between-frames/" noimage="true" %}} +{{% card link="/visualization/3d-scene/editing-frames-visually/" noimage="true" %}} +{{% card link="/visualization/3d-scene/3d-scene-widgets/" noimage="true" %}} +{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} +{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/motion-planning/3d-scene/edit-frames.md b/docs/visualization/3d-scene/editing-frames-visually.md similarity index 88% rename from docs/motion-planning/3d-scene/edit-frames.md rename to docs/visualization/3d-scene/editing-frames-visually.md index abf3fc8143..a3a9003329 100644 --- a/docs/motion-planning/3d-scene/edit-frames.md +++ b/docs/visualization/3d-scene/editing-frames-visually.md @@ -1,15 +1,18 @@ --- -linkTitle: "Edit frames visually" -title: "Edit frames visually" -weight: 50 +linkTitle: "Editing frames visually" +title: "Editing frames visually" +weight: 30 layout: "docs" type: "docs" description: "Add, edit, and attach geometry to frames directly in the 3D scene instead of editing JSON configuration." +aliases: + - /motion-planning/3d-scene/edit-frames/ + - /visualization/3d-scene-tools/editing-frames-visually/ --- The **3D SCENE** tab can serve as a configuration editor: you can add, move, re-parent, and reshape frames without writing JSON. -Visual editing is most useful while you are still figuring out where things go. Typing coordinates into JSON and reloading the 3D view to check them is slow; editing in the viewport and seeing the result immediately is faster. The trade-off is that the visual editor writes the same JSON fields through a smaller surface area, so it is less suited to bulk changes or cross-machine-part frames. Changes flow back to the machine configuration, and the app surfaces an unsaved-changes banner on the CONFIGURE tab where you save them with **Save** or `⌘/Ctrl+S`. +Visual editing is most useful while you are still figuring out where things go. Typing coordinates into JSON and reloading the 3D view to check them is slow; editing in the viewport and seeing the result immediately is faster. The trade-off is that the visual editor edits fewer fields than the JSON editor, so it is less suited to bulk changes or cross-machine-part frames. Changes flow back to the machine configuration, and the app shows an unsaved-changes banner on the CONFIGURE tab where you save them with **Save** or `⌘/Ctrl+S`. ## Prerequisites @@ -86,10 +89,10 @@ After the AI applies changes, save or discard buttons appear in the **3D SCENE** Visual editing covers most cases, but a few are faster in JSON: -- **Bulk changes** (renaming many frames, regenerating a layout) — JSON +- **Bulk changes** (renaming many frames, regenerating a layout): JSON edits are easier in a text editor. -- **Frames that reference components on a different machine part** — +- **Frames that reference components on a different machine part**: the visual editor's parent dropdown only shows local frames. - **Complex orientations** (rotations expressed in `axis_angles` or - `quaternion` rather than `ov_degrees`) — the visual editor surfaces + `quaternion` rather than `ov_degrees`): the visual editor offers only the orientation vector form. diff --git a/docs/motion-planning/3d-scene/calibrate-frame-offsets.md b/docs/visualization/3d-scene/measuring-between-frames.md similarity index 93% rename from docs/motion-planning/3d-scene/calibrate-frame-offsets.md rename to docs/visualization/3d-scene/measuring-between-frames.md index 383100f136..6117b42171 100644 --- a/docs/motion-planning/3d-scene/calibrate-frame-offsets.md +++ b/docs/visualization/3d-scene/measuring-between-frames.md @@ -1,10 +1,13 @@ --- -linkTitle: "Calibrate frame offsets" -title: "Calibrate frame offsets" +linkTitle: "Measuring between frames" +title: "Measuring between frames" weight: 20 layout: "docs" type: "docs" description: "Verify and adjust the spatial relationship between components using the 3D scene and measurement tool." +aliases: + - /motion-planning/3d-scene/calibrate-frame-offsets/ + - /visualization/3d-scene-tools/measuring-between-frames/ --- When you configure a camera on an arm, or a sensor on a base, the frame system needs the exact translation and orientation between the two components. A 15 mm error in a camera offset places a detected object 15 mm off; the arm then reaches for the wrong spot, or the point cloud sits behind the table instead of on it. The **3D SCENE** tab lets you verify offsets visually and measure distances directly, so you can catch these errors before they produce bad motion. @@ -58,7 +61,7 @@ If the component is a depth camera, enable its point cloud at **Settings → Poi The point cloud should align with the physical objects in your workspace. If the point cloud appears shifted or rotated relative to where objects actually are, the camera's frame offset or orientation is wrong. -See [Verify point cloud alignment](/motion-planning/3d-scene/verify-point-cloud-alignment/) for more on working with point clouds. +See [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/) for more on working with point clouds. ## Iterative adjustment diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/visualization/3d-scene/the-3d-scene-interface.md similarity index 68% rename from docs/motion-planning/3d-scene/_index.md rename to docs/visualization/3d-scene/the-3d-scene-interface.md index 18241314db..e5d89f9327 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/visualization/3d-scene/the-3d-scene-interface.md @@ -1,17 +1,16 @@ --- -linkTitle: "3D scene tab" -title: "3D scene tab" -weight: 30 +linkTitle: "The 3D scene interface" +title: "The 3D scene interface" +weight: 10 layout: "docs" type: "docs" -no_list: true -description: "Visualize your machine's frame system, geometries, and point clouds in an interactive 3D view." +description: "The panels, toolbar, navigation, and settings of the 3D SCENE tab in the Viam app." --- -The **3D SCENE** tab renders your machine's frame system as an interactive 3D visualization on your machine's page in the [Viam app](https://app.viam.com). -Frame configuration is otherwise invisible: a JSON translation of `{x: 50, y: 0, z: 110}` tells you nothing about whether the gripper actually sits where the arm needs it. The **3D SCENE** tab makes that spatial relationship visible so you can catch misconfigurations before a motion plan fails. - -The tab reads your machine's configuration and, when the machine is online, connects for live data. Each component's frame appears as a set of coordinate axes positioned by its translation and orientation relative to its parent frame. Attached geometries render as translucent shapes, and point clouds from depth cameras render as colored point sets. +The **3D SCENE** tab on your machine's page in the [Viam app](https://app.viam.com) is where +you inspect the 3D scene. This page is a reference to its panels, toolbar, navigation, and +settings. For what the scene renders and where each element comes from, see +[Visualizing with the 3D scene](/visualization/3d-scene/). ## The interface @@ -34,18 +33,18 @@ It includes: - **local position** (mm) and **local orientation** (deg): pose relative to the parent frame. Editable for configurable frames; these correspond to the `translation` and `orientation` in your frame configuration. - **geometry**: four buttons (`None` / `Box` / `Sphere` / `Capsule`) plus **dimensions** (`x / y / z` for Box, `r / l` for Capsule, `r` for Sphere, all in mm). -The panel header includes a **Zoom to object** button (centers the camera on the selected entity) and a copy-to-clipboard button next to the `Details` heading that exports the entity's pose and geometry as JSON. +The panel header includes a **Zoom to object** button (centers the camera on the selected entity), a **View from this frame** button (camera icon) that opens a [frame POV widget](/visualization/3d-scene/3d-scene-widgets/#frame-pov), and a copy-to-clipboard button next to the `Details` heading that exports the entity's pose and geometry as JSON. Entities that can be removed (for example, dropped PCD files) also show a **Remove from scene** button in the header. **Dashboard toolbar** (top-center): Visible buttons, left to right: -- **Orthographic / Perspective** toggle — switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. -- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Edit frames visually](/motion-planning/3d-scene/edit-frames/). -- **Measurement** (ruler icon) — activate to measure distance between two points you pick in the viewport. Click the icon again to clear. -- **Measurement settings** (sliders icon next to the ruler) — toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. -- **AI Scene Builder** — opens a prompt panel for editing frames with natural language. See [Edit frames with AI](/motion-planning/3d-scene/edit-frames/#edit-frames-with-ai). -- **Logs** — shows a count badge for errors/warnings from the scene renderer. -- **Settings** (gear icon) — opens the Settings panel. +- **Orthographic / Perspective** toggle: switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. +- **Add frames**: opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene/editing-frames-visually/). +- **Measurement** (ruler icon): activate to measure distance between two points you pick in the viewport. Click the icon again to clear. +- **Measurement settings** (sliders icon next to the ruler): toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. +- **AI Scene Builder**: opens a prompt panel for editing frames with natural language. See [Edit frames with AI](/visualization/3d-scene/editing-frames-visually/#edit-frames-with-ai). +- **Logs**: shows a count badge for errors/warnings from the scene renderer. +- **Settings** (gear icon): opens the Settings panel. ## Navigation controls @@ -97,13 +96,3 @@ To add a HoverLink: After the link is added, hovering a point in the source entity highlights the matching point in the target entity (and updates the hover tooltip with both points' positions). Existing links appear under **Relationships** in the Details panel and have per-link remove buttons. This is useful for comparing point clouds that should align (a registered scan against a transformed version, ground-truth points against predicted points) without flipping back and forth between separate views. - -## How-to guides - -{{< cards >}} -{{% card link="/motion-planning/3d-scene/calibrate-frame-offsets/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/verify-point-cloud-alignment/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/edit-frames/" noimage="true" %}} -{{< /cards >}} diff --git a/docs/visualization/_index.md b/docs/visualization/_index.md new file mode 100644 index 0000000000..a759cbfca4 --- /dev/null +++ b/docs/visualization/_index.md @@ -0,0 +1,10 @@ +--- +linkTitle: "Visualization" +title: "Visualization" +weight: 165 +layout: "docs" +type: "docs" +no_list: true +manualLink: "/visualization/overview/" +description: "See your machine in 3D: frames, geometries, point clouds, and custom visuals your modules publish." +--- diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md new file mode 100644 index 0000000000..acd618faf7 --- /dev/null +++ b/docs/visualization/overview.md @@ -0,0 +1,84 @@ +--- +linkTitle: "Overview" +title: "Visualization" +weight: 1 +layout: "docs" +type: "docs" +description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam visualizer, custom apps you build, and time-series dashboards." +--- + +A machine produces two kinds of data you can watch: spatial state (frames, geometries, +collisions, point clouds, motion plans) and time series readings (temperatures, speeds, +counts). For spatial state, use the 3D scene view in +the Viam app, or Viam Visualization, a standalone visualizer you run yourself. You can also +read this data from components with a Viam SDK and present it in a custom user interface you +build, which runs in a browser, on a phone, or on a server. For time series data, services +and modules push readings to the cloud, where the Viam app's Teleop workspaces and dashboards +show live information across a machine or fleet. + +## The 3D scene + +The **3D SCENE** tab on your machine's page renders an interactive 3D view of your +machine: component frames from the frame system, configured geometries, depth-camera +point clouds, and custom visuals a module publishes at runtime. + +- Use it while configuring or debugging a machine, to see frames, geometry, and live poses + with no code. +- Best for catching frame and obstacle misconfigurations and inspecting a motion plan in + context. It runs right in the Viam app, ready to use. + +{{< cards >}} +{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} +{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} +{{% card link="/motion-planning/visualize-a-motion-plan/" noimage="true" %}} +{{% card link="/visualization/troubleshoot-the-3d-scene/" noimage="true" %}} +{{< /cards >}} + +## Viam Visualization + +Viam Visualization is a standalone 3D visualizer you run yourself to preview and debug +spatial data from a Go client. It shares the same `draw` library as the in-app 3D scene, so +the visuals you build work either way. + +- Use it while developing, to preview spatial data such as a point cloud, detections, or a + planned path straight from a script or test. +- Best when you want to iterate from Go quickly, without deploying a module or opening the + Viam app. + +{{< cards >}} +{{% card link="/visualization/viam-visualization/" noimage="true" %}} +{{< /cards >}} + +## Custom apps + +A custom app uses a Viam SDK to read your machine's data and present it however you design. +It runs outside the machine, in a browser, on a phone, or on a server, so you can build a +custom dashboard, an operator console, or a fleet view. A Svelte app can also +[embed the 3D scene's own renderer](https://viamrobotics.github.io/visualization/guides/embedding/) +as a component, so a custom UI gets the full 3D view alongside your own controls. + +- Use it when operators or stakeholders need a tailored UI beyond the built-in scene. +- Best when you want to choose exactly which data to show and how, for one machine or a + whole fleet. + +{{< cards >}} +{{% card link="/build-apps/overview/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-dashboard/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-fleet/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-flutter-app/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-monitoring-service/" noimage="true" %}} +{{< /cards >}} + +## Time series data + +Services and modules push readings to the cloud, where you watch them live with Viam's +Teleop workspaces and dashboards. + +- Use it for values that change over time: temperatures, speeds, counts, and sensor + readings. +- Best for live monitoring and historical trends across a single machine or a whole fleet. + +{{< cards >}} +{{% card link="/monitor/teleop-workspaces/" noimage="true" %}} +{{% card link="/monitor/dashboards/overview/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md new file mode 100644 index 0000000000..e6a8ec2664 --- /dev/null +++ b/docs/visualization/perception/_index.md @@ -0,0 +1,19 @@ +--- +linkTitle: "Perception" +title: "Perception in the 3D scene" +weight: 60 +layout: "docs" +type: "docs" +no_list: true +description: "View and compare live perception data, such as depth-camera point clouds, in the 3D scene." +--- + +The 3D scene renders live perception data alongside your frame system. You see what your +machine senses in the same space as the components that sense it. + +{{< cards >}} +{{% card link="/visualization/perception/point-clouds/" noimage="true" %}} +{{% card link="/visualization/perception/cameras/" noimage="true" %}} +{{% card link="/visualization/perception/vision-services/" noimage="true" %}} +{{% card link="/visualization/perception/verify-point-cloud-alignment/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/perception/cameras.md b/docs/visualization/perception/cameras.md new file mode 100644 index 0000000000..f0512cccd5 --- /dev/null +++ b/docs/visualization/perception/cameras.md @@ -0,0 +1,31 @@ +--- +linkTitle: "Cameras" +title: "Cameras in the 3D scene" +weight: 20 +layout: "docs" +type: "docs" +description: "View a camera's live feed alongside the 3D scene, and relate it to the camera's frame and point cloud." +--- + +A camera contributes to the 3D scene in more than one way: its frame positions it in space, +its live feed shows what it currently sees, and a depth camera also produces a point cloud. + +## See a camera's live feed + +To watch a camera's feed next to the 3D view, open the **Settings** panel (gear icon), select +**Widgets**, and toggle the camera on under **Camera widgets**. For the widget's resolution +and frame-rate controls, see [3D scene widgets](/visualization/3d-scene/3d-scene-widgets/). +To view the scene from the camera's own perspective, open its +[frame POV](/visualization/3d-scene/3d-scene-widgets/#frame-pov). + +## Point clouds from depth cameras + +A depth camera also reports the distance to points in front of it, which the scene renders as +a point cloud. For how point clouds render, display, and compare, see +[Point clouds](/visualization/perception/point-clouds/). + +## Camera frames + +A camera appears in the scene as a coordinate frame, and its point cloud renders at that +frame. If a camera's point cloud appears in the wrong place, check the camera's frame +configuration rather than the camera itself. diff --git a/docs/visualization/perception/point-clouds.md b/docs/visualization/perception/point-clouds.md new file mode 100644 index 0000000000..4af7602a7a --- /dev/null +++ b/docs/visualization/perception/point-clouds.md @@ -0,0 +1,43 @@ +--- +linkTitle: "Point clouds" +title: "Point clouds in the 3D scene" +weight: 10 +layout: "docs" +type: "docs" +description: "How depth-camera point clouds render in the 3D scene, how to adjust their display, and how to load and compare external point clouds." +--- + +A depth camera reports the distance to points in front of it. The 3D scene renders that data +as a colored point set, placed in the scene by the camera's frame, so you can see what the +camera perceives in the same space as the rest of your machine. + +## Live point clouds + +When your machine is online, the scene streams point clouds from your depth cameras and +draws each as a set of points at the camera's frame. Because each point cloud sits at its +camera's frame, a point cloud that appears in the wrong place usually points to a wrong camera +frame, not wrong perception. + +Vision services can also contribute point-cloud entities, such as the points behind a +detection. These render the same way. + +## Adjust how point clouds display + +Open the **Settings** panel (gear icon) in the 3D scene tab to control point-cloud rendering: + +- **Point size and color**: set the default size and color the scene draws points at. +- **Enabled cameras**: turn each camera's point cloud on or off, so you can focus on one + camera at a time. +- **Vision**: enable or disable vision-service point-cloud entities. + +## Load and compare external point clouds + +To inspect a saved capture, drag a `.pcd` or `.ply` file onto the viewport. The scene loads +it as a point cloud you can view alongside your live data, which is useful for comparing a +saved SLAM map or scan against the current frame system. + +To compare two point clouds that should align, such as a scan and a transformed copy, link +them with **HoverLink**: select one point cloud, add a HoverLink relationship to the other, +and hovering a point in one highlights the matching point in the other. For a full alignment +walkthrough, see +[Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). diff --git a/docs/motion-planning/3d-scene/verify-point-cloud-alignment.md b/docs/visualization/perception/verify-point-cloud-alignment.md similarity index 93% rename from docs/motion-planning/3d-scene/verify-point-cloud-alignment.md rename to docs/visualization/perception/verify-point-cloud-alignment.md index 6cc06052a6..c7870115ee 100644 --- a/docs/motion-planning/3d-scene/verify-point-cloud-alignment.md +++ b/docs/visualization/perception/verify-point-cloud-alignment.md @@ -7,11 +7,12 @@ type: "docs" description: "Display a depth camera's point cloud in the 3D scene to verify that the data aligns with your frame system and workspace geometry." aliases: - /motion-planning/3d-scene/inspect-point-clouds/ + - /motion-planning/3d-scene/verify-point-cloud-alignment/ --- Depth cameras produce point clouds: sets of 3D points that represent the surfaces the camera sees. The **3D SCENE** tab renders those points in your frame system, so you can check two things at once: the camera is producing usable data, and the data lines up with the rest of the workspace. -Misalignment usually means one of two things: the camera's frame offset is wrong, or the camera itself has a problem. Finding that out now, before a motion plan runs or an ML detector ships, costs minutes; finding it out later costs a day of debugging a downstream pipeline. +Misalignment usually means one of two things: the camera's frame offset is wrong, or the camera itself has a problem. Finding that out now, before a motion plan runs or you deploy an ML detector, costs minutes; finding it out later costs a day of debugging a downstream pipeline. ## Prerequisites @@ -46,7 +47,7 @@ The point cloud sits wherever the camera's frame puts it. If the frame configura - The floor appears at the expected height relative to the world frame. If the point cloud appears shifted, rotated, or in an unexpected location, the camera's frame offset or orientation is likely wrong. -See [Calibrate frame offsets](/motion-planning/3d-scene/calibrate-frame-offsets/). +See [Measuring between frames](/visualization/3d-scene/measuring-between-frames/). ### 4. Check data quality diff --git a/docs/visualization/perception/vision-services.md b/docs/visualization/perception/vision-services.md new file mode 100644 index 0000000000..5cd2a75801 --- /dev/null +++ b/docs/visualization/perception/vision-services.md @@ -0,0 +1,78 @@ +--- +linkTitle: "Vision services" +title: "Vision services in the 3D scene" +weight: 30 +layout: "docs" +type: "docs" +description: "Render a vision service's segmented objects in the 3D scene and tune detection parameters against the live view." +--- + +A vision service with a 3D segmenter returns the objects it finds as point clouds, one +per object, through `GetObjectPointClouds`. The 3D scene polls each vision service on +your machine and renders those objects in place, so you see what the detector found in +the same space as the camera, the arm, and the workspace geometry. + +## Prerequisites + +- A vision service configured with a 3D segmenter, such as the + [`obstacles_pointcloud` module](https://app.viam.com/module/viam/obstacles-pointcloud). +- A depth camera that supports point clouds, with its frame configured. + +## Display a vision service's objects + +Open the **3D SCENE** tab on your machine's page. When the machine is online, the scene +calls each vision service's `GetObjectPointClouds` and draws the returned objects as +point-cloud entities. Each object renders at the pose the service reports, placed in the +scene by the camera's frame. + +To control which services render, open the **Settings** panel (gear icon) and select +**Vision**: each vision service has its own toggle. The polling rate lives under +**Connection**, with the scene's other data-stream rates; slow it down when a busy scene +lags, or turn it off while you work on something else. + +## Read an object's placement + +An object's position in the scene combines two sources: the segmenter's output (the +object's points and pose relative to the camera) and the camera's frame configuration +(where the camera sits in the frame system). When an object renders in the wrong place, +this split tells you where to look: + +- **Every object is offset the same way**: the camera's frame configuration is wrong. + Verify it with [Measuring between frames](/visualization/3d-scene/measuring-between-frames/) + and [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). +- **One object is wrong or split in two**: the segmenter's parameters group the points + incorrectly. Tune the service configuration. + +## Tune a detector against the live view + +The scene gives you a visual feedback loop for segmenter parameters: + +1. Place representative objects in the camera's view. +2. Watch the rendered objects in the 3D scene next to the raw point cloud (enable the + camera under **Settings** > **Pointclouds**). +3. Adjust the vision service's configuration, for example the minimum points per object + or the plane-removal threshold, and save. +4. Compare the new segmentation against the physical objects, then repeat. + +Segmentation quality shows up directly: an over-aggressive threshold drops small +objects, and a loose one merges neighbors into one blob. + +## When no objects appear + +Work through these checks in order: + +1. The vision service's toggle is on under **Settings** > **Vision**. +2. The vision polling rate under **Settings** > **Connection** is not off. +3. The service returns objects at all: call `GetObjectPointClouds` from the + [vision service API](/reference/apis/services/vision/) and check the count. +4. The camera behind the service supports point clouds: cameras that report + `supports_pcd=false` produce nothing for a 3D segmenter to segment. + +## What's next + +- [Point clouds](/visualization/perception/point-clouds/): how raw depth-camera data + renders and displays. +- [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/): + confirm the camera's frame before you tune a detector. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): draw + custom annotations the built-in vision layer does not cover. diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md new file mode 100644 index 0000000000..7cf731ff7b --- /dev/null +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -0,0 +1,248 @@ +--- +linkTitle: "Publish visuals from a module" +title: "Publish custom visuals from a module" +weight: 20 +layout: "docs" +type: "docs" +description: "Implement a world state store service that pulls data from other resources, builds transforms with the draw library, and streams them to the 3D scene." +--- + +The 3D scene shows your frame system and configured geometry by default. To draw +anything else, a detected object, a planned path, a sensor's live readings as +shapes, you implement a **world state store service**. A module that implements +this service is a producer: it reads data from other resources, turns that data +into transforms, and streams them to the scene as they change. + +This page covers when to implement the service, the methods to implement, how +to build transforms with the `draw` library, and the poll-and-update loop that +keeps the scene in sync. The pattern throughout is **pull**: the module depends +on the resources it visualizes and reads their data, rather than having those +resources push into it. + +## When to implement a world state store service + +Implement one when you want custom visuals in the 3D scene beyond the default +content. The scene already draws component frames and configured geometry on its +own. A module adds anything computed or sensed at runtime: a vision service's +detections, a sensor's obstacle readings, a motion plan's trajectory, or any +annotation specific to your application. + +## Implement the service methods + +The world state store service exposes three read methods, which the 3D scene +calls to discover and follow your visuals: + +- `ListUUIDs`: return the UUID of every transform you currently publish. +- `GetTransform`: return the transform for a given UUID. +- `StreamTransformChanges`: return a stream of change events so the scene + follows additions, updates, and removals without re-fetching everything. + +A typical implementation keeps the current transforms in a map keyed by UUID, +serves `ListUUIDs` and `GetTransform` from that map, and fans out change events +to subscribers from `StreamTransformChanges`. In the snippets below, `commonpb` +is `go.viam.com/api/common/v1` and `pb` is +`go.viam.com/api/service/worldstatestore/v1`. + +{{% alert title="Implementing in Python" color="tip" %}} +The Python SDK also includes the service, in `viam.services.worldstatestore`: implement +the same three methods there. The `draw` helper library below is Go-only; a Python +module builds its `Transform` protos directly, with the geometry constructors shown in +[Visuals and collisions](/visualization/visuals-and-collisions/). +{{% /alert %}} + +```go +func (s *visualizer) ListUUIDs( + ctx context.Context, extra map[string]any, +) ([][]byte, error) { + s.mu.RLock() + defer s.mu.RUnlock() + uuids := make([][]byte, 0, len(s.transforms)) + for id := range s.transforms { + uuids = append(uuids, []byte(id)) + } + return uuids, nil +} +``` + +`GetTransform` is a map lookup by UUID. `StreamTransformChanges` hands the caller a +stream backed by a fresh subscriber channel, and the poll loop pushes each change +onto every subscriber through a small `emit` helper: + +```go +func (s *visualizer) GetTransform( + ctx context.Context, uuid []byte, extra map[string]any, +) (*commonpb.Transform, error) { + s.mu.RLock() + defer s.mu.RUnlock() + tf, ok := s.transforms[string(uuid)] + if !ok { + return nil, fmt.Errorf("no transform with uuid %s", uuid) + } + return tf, nil +} + +func (s *visualizer) StreamTransformChanges( + ctx context.Context, extra map[string]any, +) (*worldstatestore.TransformChangeStream, error) { + ch := make(chan worldstatestore.TransformChange, 32) + s.mu.Lock() + s.subscribers = append(s.subscribers, ch) + s.mu.Unlock() + return worldstatestore.NewTransformChangeStreamFromChannel(ctx, ch), nil +} + +// emit fans one change out to every stream subscriber. +func (s *visualizer) emit( + tf *commonpb.Transform, kind pb.TransformChangeType, updated []string, +) { + change := worldstatestore.TransformChange{ + ChangeType: kind, Transform: tf, UpdatedFields: updated, + } + s.mu.RLock() + defer s.mu.RUnlock() + for _, ch := range s.subscribers { + select { + case ch <- change: + default: // skip a subscriber whose buffer is full + } + } +} +``` + +## Build transforms with the draw library + +Rather than assembling `commonpb.Transform` protos by hand, use the `draw` +library from `github.com/viam-labs/motion-tools/draw`. You wrap a +`spatialmath.Geometry` with styling, then `Draw` it with an ID and pose to get a +`*commonpb.Transform`: + +```go +import ( + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +func buildTransform(o obstacle) (*commonpb.Transform, error) { + // Build the box at the origin; WithPose below places it at the obstacle pose. + box, err := spatialmath.NewBox(spatialmath.NewZeroPose(), o.Dims, o.ID) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(box, draw.WithGeometryColor(colorFor(o))) + if err != nil { + return nil, err + } + return drawn.Draw(o.ID, draw.WithID(o.ID), draw.WithPose(o.Pose)) +} +``` + +The library produces standard `commonpb.Transform` values, the same type the +service methods return, so the transforms you build this way flow straight +through `ListUUIDs`, `GetTransform`, and `StreamTransformChanges` to the scene. + +## Drive a poll-and-update loop + +The module owns its update cadence. A ticker drives a loop that reads the +module's dependencies, builds the current set of transforms, diffs it against the +cached set, and emits one change event per difference: + +```go +func (s *visualizer) pollLoop() { + for range s.ticker.C { + readings, err := s.sensor.Readings(s.ctx, nil) + if err != nil { + continue + } + next := buildTransforms(readings) + + // Diff next against s.last and emit one event per change. + for id, tf := range next { + if prev, ok := s.last[id]; !ok { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_ADDED, nil) + } else if changed(prev, tf) { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_UPDATED, changedFields(prev, tf)) + } + } + for id, tf := range s.last { + if _, ok := next[id]; !ok { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_REMOVED, nil) + } + } + s.last = next + } +} +``` + +Emitting added, updated, and removed events (with `UpdatedFields` on updates) +lets the scene apply incremental changes instead of re-rendering. The diff +against cached state is what turns a full snapshot each tick into a stream of +minimal updates. + +## Pull from the resources you visualize + +The module is the producer, and the resources it visualizes are its +dependencies. The 3D scene reads only the world state store service; the service +reads everything else. Data flows one way: + +> dependency resources → world state store module (reads, builds transforms) → 3D scene + +This is why the loop above calls `s.sensor.Readings(...)`: the sensor is a +dependency, and the module pulls from it. The same pattern visualizes any other +resource. A module whose primary job is something else (an arm, a sensor, a +planner) stays focused on that job. To visualize it, you write a separate world +state store module that takes that resource as a dependency and pulls from its +existing API: + +- a sensor's `Readings` +- a component's geometry getters +- any resource's `DoCommand` + +The visualized resource needs no changes and no awareness that it is being drawn. +The store depends on it, not the other way around. + +```go +func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstatestore.Service, error) { + obstacleSensor, err := sensor.FromProvider(deps, "obstacle-sensor") + if err != nil { + return nil, fmt.Errorf("getting obstacle-sensor: %w", err) + } + // The module depends on obstacle-sensor and pulls its readings on a loop. + return startVisualizer(obstacleSensor), nil +} +``` + +## Add the service to your machine + +The module registers a model against the world state store API, so you configure it like +any other modular service: add the module to your machine, then add a service with +`"api": "rdk:service:world_state_store"` and your model. Name the resources the module +pulls from in `depends_on`, so the module starts after them: + +```json +"services": [ + { + "name": "visualizer", + "api": "rdk:service:world_state_store", + "model": "::", + "depends_on": ["obstacle-sensor"] + } +] +``` + +Once the machine is online with the service running, the **3D SCENE** tab discovers the +service and streams its transforms. For packaging and deploying the module itself, see +[Build modules](/build-modules/). + +## What's next + +- [Implementing WorldStateStoreService](https://viamrobotics.github.io/visualization/guides/worldstatestore/): + the upstream guide this pattern follows, with a + [complete runnable module](https://github.com/viam-labs/motion-tools/tree/main/docs/examples/worldstatestore). +- [Visuals and collisions](/visualization/visuals-and-collisions/): + what a transform contains, and which geometry the planner collision-checks. +- [Viam Visualization](/visualization/viam-visualization/): + preview the same visuals from a script with the standalone visualizer. +- [Transform metadata](/visualization/reference/transform-metadata/): + the styling keys the scene reads and their wire formats. +- [Frame system](/motion-planning/frame-system/): position the transforms you + publish. diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md new file mode 100644 index 0000000000..8c1998801f --- /dev/null +++ b/docs/visualization/reference/_index.md @@ -0,0 +1,18 @@ +--- +linkTitle: "Reference" +title: "Visualization reference" +weight: 90 +layout: "docs" +type: "docs" +no_list: true +description: "Reference material for the types and services behind the 3D scene and custom visuals." +--- + +Reference pages for the types and services that back the 3D scene and custom visuals. + +{{< cards >}} +{{% card link="/reference/apis/services/world-state-store/" noimage="true" %}} +{{% card link="/visualization/reference/world-state/" noimage="true" %}} +{{% card link="/visualization/reference/transform-metadata/" noimage="true" %}} +{{% card link="/visualization/reference/draw-library/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/reference/draw-library.md b/docs/visualization/reference/draw-library.md new file mode 100644 index 0000000000..1a0f654125 --- /dev/null +++ b/docs/visualization/reference/draw-library.md @@ -0,0 +1,90 @@ +--- +linkTitle: "draw library" +title: "draw library" +weight: 30 +layout: "docs" +type: "docs" +description: "Lookup tables for the draw library: placement and identity options, shape constructors, colors, metadata, and snapshot options." +--- + +The [`draw` library](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) +(`github.com/viam-labs/motion-tools/draw`) builds the transforms and entities that the 3D +scene and [Viam Visualization](/visualization/viam-visualization/) render. This page lists +the options you reach for while writing a producer. The library lives in `viam-labs` and +moves faster than the RDK, so treat the project's +[generated API docs](https://viamrobotics.github.io/visualization/api/draw/) and +[pkg.go.dev](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) as the full and +current inventory. + +## Placement and identity + +Options for `Draw` calls and `NewDrawConfig`, all of type `DrawableOption`: + +| Option | What it sets | +| ---------------- | ------------------------------------------------------------------------------------------ | +| `WithParent` | The reference frame the pose is expressed in. Defaults to `world`. | +| `WithPose` | The entity's pose in the parent frame. | +| `WithCenter` | An offset applied at the entity's own center. | +| `WithID` | A string identity; the library derives a stable UUID from it, so re-sends update in place. | +| `WithUUID` | An explicit UUID, when you manage identity yourself. | +| `WithAxesHelper` | Draws a coordinate triad at the entity's origin. | +| `WithInvisible` | Hides the entity by default; the viewer can re-enable it. | + +## Shapes + +| Constructor | Builds | +| ------------------------------------- | ----------------------------------------------------------------------------------- | +| `NewDrawnGeometry` | A styled `spatialmath.Geometry`; its `Draw` method returns a `*commonpb.Transform`. | +| `NewShape` + `WithArrows` | Arrows, for directions, normals, or vectors. | +| `NewShape` + `WithLine` / `NewLine` | A line, for paths, segments, or connections. | +| `NewShape` + `WithPoints` | Points, for sampled data or markers. | +| `NewShape` + `WithModel` | A detailed 3D mesh model. | +| `NewShape` + `WithNurbs` / `NewNurbs` | A smooth NURBS curve or surface. | + +Style a geometry with `WithGeometryColor` (one color) or `WithGeometryColors` (per-point +colors) when you call `NewDrawnGeometry`. + +## Colors + +Build a `Color` with `NewColor` plus one option, or use the one-call helpers: + +| With `NewColor` | Helper | Input | +| --------------- | --------------- | -------------------------------------- | +| `WithRGB` | `ColorFromRGB` | `r, g, b` as 0 to 255 | +| `WithRGBA` | `ColorFromRGBA` | `r, g, b` plus alpha for opacity | +| `WithName` | `ColorFromName` | A CSS color name, such as `dodgerblue` | +| `WithHex` | `ColorFromHex` | A hex string | +| `WithHSV` | `ColorFromHSV` | Hue, saturation, value | + +## Metadata + +Options for `NewDrawing` and `NewTransform`, all of type `DrawMetadataOption`. Each writes +one of the [metadata keys](/visualization/reference/transform-metadata/) the scene reads: + +| Option | Metadata it writes | +| --------------------------- | -------------------------------------------- | +| `WithMetadataColors` | `colors` (and `opacities` from alpha) | +| `WithMetadataAxesHelper` | `show_axes_helper` | +| `WithMetadataInvisible` | `invisible` | +| `WithMetadataRelationships` | `relationships`, for links such as HoverLink | + +## Snapshots + +Options for `NewSnapshot`, which builds a loadable +[scene snapshot](/visualization/viam-visualization/#save-and-load-scene-snapshots): + +| Option | What it sets | +| ------------------------------------------- | -------------------------------------------------- | +| `WithSceneCamera` | Where the scene camera starts. | +| `WithGrid`, `WithGridCellSize` | The reference grid and its cell size. | +| `WithScenePointSize`, `WithScenePointColor` | Default point rendering. | +| `WithRenderArmModels` | Whether arms render as colliders, models, or both. | + +## What's next + +- [Transform metadata](/visualization/reference/transform-metadata/): the wire formats + behind the metadata options. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the + library in a world state store module. +- [Viam Visualization](/visualization/viam-visualization/): the library from a script, + pushed to the standalone visualizer. diff --git a/docs/visualization/reference/transform-metadata.md b/docs/visualization/reference/transform-metadata.md new file mode 100644 index 0000000000..c45bec7a68 --- /dev/null +++ b/docs/visualization/reference/transform-metadata.md @@ -0,0 +1,65 @@ +--- +linkTitle: "Transform metadata" +title: "Transform metadata" +weight: 20 +layout: "docs" +type: "docs" +description: "The metadata keys the 3D scene reads from a transform, with the wire format for each: colors, opacities, axes helper, and visibility." +--- + +A transform's `metadata` field is a protobuf `Struct` of rendering attributes. The 3D +scene reads seven keys from it and ignores every other key. If a visual renders with +default styling, check the key names and formats on this page first: an unrecognized +key fails silently. + +Metadata affects rendering only. The motion planner reads its geometry from the frame +system and the [`WorldState`](/visualization/reference/world-state/) you pass to `Move`, +so no metadata key changes what the planner plans around. + +## The keys the scene reads + +| Key | Type | What it controls | +| ------------------ | --------------- | --------------------------------------------------------------------------------- | +| `colors` | string (base64) | Fill color, or one color per point for point cloud geometry. | +| `color_format` | number | How the color bytes are laid out. The `draw` library writes the RGB format. | +| `opacities` | string (base64) | Transparency, `0` (invisible) to `255` (opaque). One byte, or one byte per point. | +| `show_axes_helper` | bool | Draws a coordinate triad at the visual's origin. | +| `invisible` | bool | Hides the visual by default; the viewer can re-enable it in the World panel. | +| `chunks` | object | Streams a large entity in pieces: `chunk_size`, `total`, and `stride`. | +| `relationships` | list | Links this entity to others, which powers features such as HoverLink. | + +## Color and opacity wire format + +The binary values travel as base64-encoded strings, because a protobuf `Struct` has no +bytes type: + +- `colors` packs 3 bytes per color, in R, G, B order. A single color styles the whole + shape; for point cloud geometry, supply one color per point. +- `opacities` packs 1 byte per value. A single byte applies one opacity to the whole + visual; a byte per point sets per-point transparency. + +## Produce metadata with the draw library + +The [`draw` library](https://github.com/viam-labs/motion-tools) encodes these formats +for you, so producer code sets options instead of packing bytes: + +```go +import "github.com/viam-labs/motion-tools/draw" + +// Red at half opacity. WithName, WithHex, WithRGB, and WithHSV also build colors. +drawn, err := draw.NewDrawnGeometry( + box, + draw.WithGeometryColor(draw.NewColor(draw.WithRGBA(255, 0, 0, 128))), +) +``` + +`WithMetadataColors`, `WithMetadataAxesHelper`, and `WithMetadataInvisible` set the +corresponding keys on a drawing. When you assemble the `Struct` in another language, +follow the wire formats in the table above. + +## What's next + +- [Visuals and collisions](/visualization/visuals-and-collisions/): the transform the + metadata belongs to, and which geometry the planner collision-checks. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): serve + styled transforms to the 3D scene. diff --git a/docs/visualization/reference/world-state.md b/docs/visualization/reference/world-state.md new file mode 100644 index 0000000000..4adbc20b48 --- /dev/null +++ b/docs/visualization/reference/world-state.md @@ -0,0 +1,44 @@ +--- +linkTitle: "WorldState" +title: "WorldState" +weight: 10 +layout: "docs" +type: "docs" +description: "What a WorldState is: the per-request set of obstacles and frame transforms you pass to a single Move call for the planner to plan around." +--- + +A `WorldState` is the argument you pass to a single `Move` call. It carries the obstacles +and frame transforms that the motion planner should account for on that one request. When +the call returns, the `WorldState` is gone. + +## What a WorldState carries + +A `WorldState` holds two kinds of item: + +- **Obstacles**: geometries the planner treats as things to avoid, each expressed in a + reference frame. Use them for objects that are not part of the machine's configured frame + system, such as a pallet detected at runtime. +- **Transforms**: frames added to the frame system for this request only. A transform can + reposition where the arm moves (for example, a frame at a grasped object's tip) or carry a + geometry that travels with a component. + +## A WorldState applies to one request + +A `WorldState` is per-request. The motion service uses it to plan the single motion you +attach it to, then discards it. To carry the same obstacles into a later `Move` call, pass +a `WorldState` again. To build obstacles and transforms and attach them to a +`Move` call, see [Define obstacles](/motion-planning/obstacles/) and +[Arm and end effector frames](/motion-planning/frame-system/end-effector-frames/). + +## WorldState versus the world state store service + +The names are similar, but the two are different things, and only one affects a plan: + +- **`WorldState`** is planner input. The obstacles and transforms in it shape the motion the + planner computes for one `Move` call. +- The [world state store service](/reference/apis/services/world-state-store/) is + visualization output. It holds transforms that the 3D scene draws, and the planner never + reads it. + +The same geometry can take both paths, but they are separate: put it in a `WorldState` for +the planner to avoid, and publish it to the world state store service for the scene to draw. diff --git a/docs/visualization/troubleshoot-the-3d-scene.md b/docs/visualization/troubleshoot-the-3d-scene.md new file mode 100644 index 0000000000..ec4cc149b7 --- /dev/null +++ b/docs/visualization/troubleshoot-the-3d-scene.md @@ -0,0 +1,90 @@ +--- +linkTitle: "Troubleshoot the 3D scene" +title: "Troubleshoot the 3D scene" +weight: 80 +layout: "docs" +type: "docs" +description: "Trace a wrong or missing element in the 3D scene to its source: the machine configuration, a module, or the connection." +--- + +Every element in the 3D scene comes from one of three layers: the machine's saved +configuration (frames and geometry), a module publishing at runtime (custom visuals and +point clouds), or the live connection that streams poses and data. When something looks +wrong, identify the layer first; the fix lives there, and edits to the wrong layer waste +time. + +## Read the scene's own signals first + +Three signals tell you what the scene is receiving before you change anything: + +- **Online state.** An offline machine renders the saved frame configuration only, with + every component at its configured pose. Live poses, point clouds, and custom visuals + need the machine online and connected. +- **The Logs button** in the top-center toolbar shows a count of renderer errors and + warnings. A nonzero badge means the scene received data it could not draw; open it and + read the message before assuming data is missing. +- **Polling rates** under **Settings** > **Connection** control how often the scene + fetches each data stream. A stream set to off explains a layer that never updates. + +## A frame or geometry is in the wrong place + +Frames and configured geometry come from the machine configuration, so a misplaced one +is a configuration problem. Select the entity in the **World** panel and compare its +**local position** and **parent frame** to your physical measurements. Then fix the +values in the configuration or +[edit the frame in the scene](/visualization/3d-scene/editing-frames-visually/). +For the full checklist of frame and obstacle checks, see +[Debug a motion plan](/motion-planning/debug-motion-plan/). + +## A point cloud is missing + +Work down the camera pipeline: + +1. The camera's toggle is on under **Settings** > **Pointclouds** > **Enabled cameras**. +2. The camera supports point clouds. A camera that reports `supports_pcd=false` is + auto-disabled in that list; check its module documentation. +3. The camera's frame is configured. A point cloud renders at its camera's frame, so a + camera with no frame has nowhere to draw. + +A point cloud that renders in the wrong place, rather than missing, is a frame problem: +see [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). + +## A custom visual is missing or stale + +Custom visuals come from a module through a +[world state store service](/reference/apis/services/world-state-store/), so a missing or +stale one is a module problem. Check the module's output at the service boundary: + +1. `ListUUIDs` returns the visual's UUID. An empty list means the module publishes + nothing; check the module's logs and its dependencies. +2. The module emits a change event when the data changes. A visual that appears once and + never moves usually means the poll loop stopped or emits no `TransformChange` events. +3. The UUID is stable across updates. A module that generates a fresh UUID each tick + piles up duplicates instead of updating one visual. + +The implementation side of these checks is covered in +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## Know what the scene can tell you + +Match your conclusion to what the scene renders: + +- **Machine offline**: trust the frame tree and configured geometry; treat every pose as + the configured default, and expect point clouds, live poses, and custom visuals to be + absent. +- **Machine online**: poses, point clouds, and custom visuals are live, subject to the + polling rates you set. + +A plan's obstacles are a separate case: `WorldState` obstacles passed to a single `Move` +call are planner input for that request and are never drawn. To see them, publish the +same geometry as a custom visual; see +[Visuals and collisions](/visualization/visuals-and-collisions/). + +## What's next + +- [Debug a motion plan](/motion-planning/debug-motion-plan/): the frame, obstacle, and + reach checks for planning failures. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the + service methods and poll loop behind custom visuals. +- [The 3D scene interface](/visualization/3d-scene/the-3d-scene-interface/): every + panel, setting, and toolbar button. diff --git a/docs/visualization/viam-visualization.md b/docs/visualization/viam-visualization.md new file mode 100644 index 0000000000..4429d6fd26 --- /dev/null +++ b/docs/visualization/viam-visualization.md @@ -0,0 +1,128 @@ +--- +linkTitle: "Viam Visualization" +title: "Viam Visualization" +weight: 30 +layout: "docs" +type: "docs" +description: "Run the standalone Viam Visualization app locally and push geometries, point clouds, and frame systems to it from a Go client." +aliases: + - /visualization/drawing-library/ +--- + +Viam Visualization is a standalone 3D visualizer for monitoring, testing, and debugging +spatial data: you start it locally and push visuals to it from your own Go code while +you develop. It renders the same entities the in-app **3D SCENE** tab renders, built +with the same `draw` library, so the visuals you construct work in either place. + +This page covers running the app, pushing visuals from a Go client, and saving scenes +as snapshots. + +## Viam Visualization versus the 3D scene tab + +The two render 3D visuals, but they are different tools for different moments: + +- The **3D SCENE tab** lives in the Viam app and renders a configured machine: + its frames, geometry, point clouds, and the custom visuals published to its + world state store service. It is the in-app view of a machine. +- **Viam Visualization** is a standalone visualizer you run on your own machine and push + to from a client. It is for previewing and debugging spatial data while you + develop, without deploying a module or opening the Viam app. + +Reach for the 3D scene tab to inspect a running machine; reach for Viam +Visualization to iterate on spatial data from a script or test. + +## Run the app + +From the [motion-tools repository](https://github.com/viam-labs/motion-tools), run +`make setup` once, then `make up` to start the app at `http://localhost:5173`. It +renders in your browser. For prerequisites and incremental-rebuild details, see the +[Running locally guide](https://viamrobotics.github.io/visualization/guides/local-usage/). + +To try the visualizer before installing anything, open the hosted +[playground](https://viamrobotics.github.io/visualization/playground/snapshot), which +renders a sample scene snapshot in your browser. + +## Push visuals from a Go client + +With the app running, push visuals to it with the client API from +`github.com/viam-labs/motion-tools/client/api`. Reusing an entity ID updates that +visual in place, so an iterating script animates state instead of piling up duplicates; +a new or empty ID adds another entity: + +```go +import ( + "github.com/viam-labs/motion-tools/client/api" + "github.com/viam-labs/motion-tools/draw" +) + +// Reuse an ID to update that visual in place; change or omit it to add another. +_, err := api.DrawGeometry(api.DrawGeometryOptions{ + ID: "obstacle-1", + Geometry: box, + Color: draw.NewColor(draw.WithName("red")), +}) +``` + +The client API has a call per data kind: geometries, point clouds, lines, NURBS, GLTF +models, single frames, and whole frame systems. Beyond plain shapes, the `draw` package +supplies the primitives you push, arrows for directions and normals, lines for paths, +points for sampled data, and styling options for each: + +```go +import "github.com/viam-labs/motion-tools/draw" + +shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) +``` + +This lets you preview spatial data, a point cloud, a set of detections, a planned +path, straight from a script or test. For every call and option, see the generated +[client API reference](https://viamrobotics.github.io/visualization/api/client-api/). + +## Connect to a live machine + +The visualizer can also connect to a Viam machine and render its frame system, arms, and +cameras, the way the in-app **3D SCENE** tab does. Put the machine's credentials in a +.env.local file at the repository root, run `make up`, then pick the machine +in the machine config panel (lower right). For the credential format, see the +[Running locally guide](https://viamrobotics.github.io/visualization/guides/local-usage/#connecting-to-a-viam-machine). + +## Save and load scene snapshots + +A snapshot captures a scene as a JSON file you can share, commit as a test fixture, or +reload later. Build one in Go with the `draw` package, then drag the file onto either +viewer's viewport to load it: + +```go +import "github.com/viam-labs/motion-tools/draw" + +snapshot := draw.NewSnapshot( + draw.WithSceneCamera(camera), // where the scene camera starts + draw.WithGrid(true), +) +if err := snapshot.DrawGeometry(box, boxPose, "world", draw.ColorFromName("dodgerblue")); err != nil { + return err +} +data, err := snapshot.MarshalJSON() +``` + +Name the output file with a `visualization_snapshot` prefix, for example +visualization_snapshot_grasp_test.json: the drag-and-drop loader accepts +only files that match that prefix. + +## How updates reach the browser + +The app runs a **draw service** that the client API calls. Each push becomes an +`AddEntity`, `UpdateEntity`, or `RemoveEntity` operation, and the service fans that +single change out over a `StreamEntity` stream the browser subscribes to. The browser +applies the one change instead of re-rendering the scene. This is the same add, +update, and remove model the world state store service uses to feed the in-app 3D +scene, so a busy scene stays in sync as your data changes. + +## What's next + +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + use the same `draw` library to serve transforms to the in-app 3D scene. +- [Transform metadata](/visualization/reference/transform-metadata/): + the styling attributes behind colors, opacity, and visibility. +- [Visuals and collisions](/visualization/visuals-and-collisions/): + what a transform contains and how the scene renders it. diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md new file mode 100644 index 0000000000..ecdb29ba08 --- /dev/null +++ b/docs/visualization/visuals-and-collisions.md @@ -0,0 +1,368 @@ +--- +linkTitle: "Visuals and collisions" +title: "Visuals and collisions" +weight: 10 +layout: "docs" +type: "docs" +description: "How a Transform defines a custom visual, and which geometry the motion planner actually collision-checks." +--- + +A geometry is a simple shape, such as a box, sphere, or capsule, that represents an +object's physical extent. Viam uses one set of geometry types for two jobs: drawing a +custom visual in the 3D scene, and telling the motion planner about an obstacle. + +To create a custom visual, you attach a geometry to a transform's `physical_object` +field and provide +that transform to a [world state store service](/reference/apis/services/world-state-store/). +This service holds the transforms you publish and streams them to the 3D scene. This +page covers what a transform contains, the difference between planner geometry and +visualization geometry, and how to build each geometry type. + +## Anatomy of a transform + +A `Transform` carries four things that together place and style a visual: + +- **Reference frame and pose**: the visual's origin. +- **Geometry** (the `physical_object` field): the shape to draw (a box, sphere, + capsule, mesh, or point cloud). +- **Metadata**: styling such as color and opacity. +- **UUID**: a stable identifier for this specific visual. + +The reference frame and pose decide _where_, the geometry decides _what shape_, +and the metadata decides _how it looks_. + +## The UUID gives a visual a stable identity + +Each transform has a UUID. That identifier is what lets a module change one +visual without disturbing the rest of the scene: to update a visual, the module +re-sends a transform with the same UUID; to remove it, it references that UUID; +to add a new one, it uses a fresh UUID. Without stable identifiers the client +would have to re-render everything on every change. With them, the scene applies +incremental add, update, and remove operations to individual visuals. + +## Metadata styles the visual + +The metadata is a set of rendering attributes the scene reads when it draws the +geometry: + +- `colors`: the fill color, or one color per point for point cloud geometry +- `opacities`: how transparent the shape is +- `show_axes_helper`: draws a coordinate triad at the visual's origin +- `invisible`: hides the visual by default + +The scene reads only the keys it recognizes and ignores the rest. These are all +**visualization attributes**: they control how the visual looks. +The planner reads its solid geometry from the frame +system and the [`WorldState`](/motion-planning/obstacles/) you pass to `Move`, so +metadata changes what you see without changing what the planner plans around. + +## The scene draws, the planner collision-checks + +The geometry on a world state store transform renders in the 3D scene: publishing a +box draws a box. The motion planner collision-checks a separate geometry, which it +reads from two places: + +- The **frame system**: each component's `frame.geometry`. +- The **`WorldState`** you pass to a `Move` call: obstacles and transforms + supplied for that single planning request. + +Despite the similar names, these are different things: the +[world state store service](/reference/apis/services/world-state-store/) holds transforms +for the scene to draw, and the [`WorldState`](/motion-planning/obstacles/) you pass to +`Move` carries obstacles for the planner to avoid. A world state store transform and a +`WorldState` obstacle travel two paths, each with its own job: one is drawn in the scene, +the other is planned around. The same shape can take both paths. + +## Making a geometry both visible and collision-checked + +If you want a geometry to appear in the scene _and_ be avoided by the planner, +you do both, separately: + +- **For the scene**: publish it as a transform through the world state store + service (see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/)). +- **For planning**: add it to the frame system, or include it in the + `WorldState` you pass to `Move`. + +Today these are two separate outputs you produce from the same source data: one +transform for the scene, one geometry for the planner. + +## Geometry types + +The supported types are: + +- **box**: dimensions in millimeters +- **sphere**: a radius +- **capsule**: a radius and length +- **mesh**: an arbitrary triangle mesh +- **point cloud**: a set of points + +Choose the type that matches what you are representing: a box or capsule to +approximate a physical object, a mesh for a precise model, a point cloud for +sensor data. For guidance on matching a geometry type to a physical object, see the +[geometry types table](/motion-planning/obstacles/overview/#geometry-types). + +You build a geometry as a `Geometry` proto, the same type a world state store +transform and a `WorldState` obstacle both carry. The Python SDK and the Go SDK +construct that proto directly, with no helper library. The box, sphere, and +capsule primitives also have a machine config (JSON) form. The transform you +attach the geometry to supplies its reference frame and pose. + +### Box + +A box takes its `x`, `y`, and `z` dimensions in millimeters. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "box", "x": 100, "y": 100, "z": 100 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Geometry, RectangularPrism, Vector3 + +box = Geometry( + label="box", + box=RectangularPrism(dims_mm=Vector3(x=100, y=100, z=100)), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +box := &commonpb.Geometry{ + Label: "box", + GeometryType: &commonpb.Geometry_Box{ + Box: &commonpb.RectangularPrism{ + DimsMm: &commonpb.Vector3{X: 100, Y: 100, Z: 100}, + }, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Sphere + +A sphere takes a radius in millimeters: the `r` key in JSON, the `radius_mm` +field in the proto. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "sphere", "r": 50 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Geometry, Sphere + +sphere = Geometry(label="sphere", sphere=Sphere(radius_mm=50)) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +sphere := &commonpb.Geometry{ + Label: "sphere", + GeometryType: &commonpb.Geometry_Sphere{ + Sphere: &commonpb.Sphere{RadiusMm: 50}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Capsule + +A capsule takes a radius and a length in millimeters. The length must be at +least twice the radius. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "capsule", "r": 50, "l": 200 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Capsule, Geometry + +capsule = Geometry( + label="capsule", + capsule=Capsule(radius_mm=50, length_mm=200), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +capsule := &commonpb.Geometry{ + Label: "capsule", + GeometryType: &commonpb.Geometry_Capsule{ + Capsule: &commonpb.Capsule{RadiusMm: 50, LengthMm: 200}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Mesh + +A mesh comes from an STL or PLY file. Read the file and embed its bytes in the +geometry with a `content_type`. The renderer draws PLY, so convert an STL file +to PLY first. + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from pathlib import Path + +from viam.proto.common import Geometry, Mesh + +mesh = Geometry( + label="mesh", + mesh=Mesh(content_type="ply", mesh=Path("model.ply").read_bytes()), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import ( + "os" + + commonpb "go.viam.com/api/common/v1" +) + +plyBytes, err := os.ReadFile("model.ply") +if err != nil { + return err +} +mesh := &commonpb.Geometry{ + Label: "mesh", + GeometryType: &commonpb.Geometry_Mesh{ + Mesh: &commonpb.Mesh{ContentType: "ply", Mesh: plyBytes}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Point cloud + +A point cloud is sensor output, so you read it as PCD bytes in binary PCD format +and embed them in the geometry. Add a color per point in the PCD data itself. +A point cloud has no machine config form, so you build it in code. + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from pathlib import Path + +from viam.proto.common import Geometry, PointCloud + +point_cloud = Geometry( + label="point-cloud", + pointcloud=PointCloud(point_cloud=Path("cloud.pcd").read_bytes()), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import ( + "os" + + commonpb "go.viam.com/api/common/v1" +) + +pcdBytes, err := os.ReadFile("cloud.pcd") +if err != nil { + return err +} +pointCloud := &commonpb.Geometry{ + Label: "point-cloud", + GeometryType: &commonpb.Geometry_Pointcloud{ + Pointcloud: &commonpb.PointCloud{PointCloud: pcdBytes}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +## Assemble a transform + +The geometry above is only the shape. To render it as a visual, wrap it in a transform +that carries the other anatomy fields: a reference frame and pose, metadata, and a UUID. +The `draw` library assembles them for you, so you do not build the `Transform` proto by +hand: + +```go +import ( + "github.com/golang/geo/r3" + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +// A red box at (400, 0, 200) in the world frame. +box, err := spatialmath.NewBox( + spatialmath.NewZeroPose(), r3.Vector{X: 100, Y: 100, Z: 100}, "target-box", +) +if err != nil { + return nil, err +} +drawn, err := draw.NewDrawnGeometry( + box, + draw.WithGeometryColor(draw.NewColor(draw.WithName("red"))), // metadata: color +) +if err != nil { + return nil, err +} +transform, err := drawn.Draw( + "target-box", // the name of this visual's frame + draw.WithID("target-box"), // UUID: a stable identity for later updates + draw.WithParent("world"), // reference frame the pose is expressed in + draw.WithPose(spatialmath.NewPoseFromPoint(r3.Vector{X: 400, Y: 0, Z: 200})), // pose +) +``` + +Publish `transform` through a world state store service to draw it. For the full module +that serves transforms this way, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## What's next + +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + implement a world state store service that publishes transforms. +- [Define obstacles](/motion-planning/obstacles/): the geometry the planner + collision-checks. +- [Frame system](/motion-planning/frame-system/): how the planner gets the + geometry and frames it plans around.