From 4bf7a0af3827818ae565aae5b2823890115d0a0d Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 18 Jun 2026 16:32:35 -0600 Subject: [PATCH 01/46] Add motion planning pages: end effector frames, waypoints, pose clouds, plan verification Add four new pages under Motion Planning and wire them into navigation: - Frame System > End effector frames: how Move targets a named frame, the frame attribute, WorldState transforms, and WorldState vs the world state store service. - Move an arm > Moving through multiple waypoints: multi-goal PlanRequest with armplanning.PlanMotion, continuous trajectory, return_partial_plan. - Move an arm > Pose clouds: relax a goal into a region with NewPoseInFrameWithGoalCloud; briefly noted and linked from the constraints page. - Verifying a Motion Plan: armplanning.PlanMotion vs Move, assembling a PlanRequest, reading the Plan, and the motion service plan DoCommand verb. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/motion-planning/_index.md | 1 + .../frame-system/end-effector-frames.md | 185 ++++++++++++++++++ docs/motion-planning/frame-system/overview.md | 9 + .../move-an-arm/constraints.md | 9 + .../move-an-arm/multiple-waypoints.md | 164 ++++++++++++++++ docs/motion-planning/move-an-arm/overview.md | 2 + .../move-an-arm/pose-clouds.md | 126 ++++++++++++ docs/motion-planning/verify-a-plan.md | 183 +++++++++++++++++ 8 files changed, 679 insertions(+) create mode 100644 docs/motion-planning/frame-system/end-effector-frames.md create mode 100644 docs/motion-planning/move-an-arm/multiple-waypoints.md create mode 100644 docs/motion-planning/move-an-arm/pose-clouds.md create mode 100644 docs/motion-planning/verify-a-plan.md diff --git a/docs/motion-planning/_index.md b/docs/motion-planning/_index.md index 2727c85842..fd457100fe 100644 --- a/docs/motion-planning/_index.md +++ b/docs/motion-planning/_index.md @@ -88,6 +88,7 @@ returns a collision-free path from the current pose to your target. {{< cards >}} {{% card link="/motion-planning/move-gantry/" noimage="true" %}} +{{% card link="/motion-planning/verify-a-plan/" noimage="true" %}} {{% card link="/motion-planning/debug-motion-with-cli/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md new file mode 100644 index 0000000000..6ab5dfbc04 --- /dev/null +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -0,0 +1,185 @@ +--- +linkTitle: "End effector frames" +title: "End effector frames" +weight: 5 +layout: "docs" +type: "docs" +description: "How the motion service moves a named frame to a target pose, and how the frame attribute and WorldState transforms control which point on the robot that frame represents." +--- + +When you ask the motion service to move an arm, you do not move "the arm." You +move a _frame_. The destination you pass is a pose for a named frame in the +frame system, and the planner solves for joint angles that place that frame's +origin at the destination. Understanding which frame you are moving, and where +that frame sits on the physical robot, is the difference between the tool tip +landing on the target and the arm's mounting flange landing there instead. + +This page explains how the motion service targets frames, how the `frame` +attribute defines an end effector frame, and how a `WorldState` transform lets +you add or reposition that frame for a single request. + +## Move targets a frame, not a component + +The `Move` request takes a component name and a destination. The component name +is not treated as a piece of hardware to actuate directly; it is resolved to a +frame in the frame system. The planner then searches for an arm configuration +that brings that frame's origin to the destination pose. + +```python +await motion_service.move( + component_name="my-gripper", + destination=destination, +) +``` + +Passing `"my-gripper"` here moves the gripper's frame, so the destination +describes where the gripper should end up. Passing `"my-arm"` instead moves the +arm's frame, which sits at the arm's mounting flange. The two are different +physical points, so the same destination pose produces different motions. Name +the frame whose position you actually care about. + +This is why motion is expressed in terms of frames: a single arm carries many +frames worth targeting (the flange, a gripper's tool center point, a wrist +camera), and the planner needs to know which one you mean. + +## Define an end effector frame with the frame attribute + +A component's frame is defined through its `frame` attribute, the same +parent, translation, and orientation you configure for any component. For an +end effector, the translation is what moves the frame origin from the arm's +flange out to the working point. + +A gripper whose tool center point sits 120 mm past the flange uses a frame like +this: + +```json +{ + "parent": "my-arm", + "translation": { "x": 0, "y": 0, "z": 120 }, + "orientation": { + "type": "ov_degrees", + "value": { "x": 0, "y": 0, "z": 1, "th": 0 } + } +} +``` + +With this frame configured, calling `Move` with `component_name="my-gripper"` +positions the point 120 mm past the flange at the destination. Without it, the +planner has no model of the tool's length and positions the flange instead, so +the physical tool overshoots the target by 120 mm. For a worked example with an +arm, gripper, and camera, see +[Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/). + +## How the frame system maintains frames + +The frame system stores every configured frame in a single tree rooted at the +world frame, and recomputes each frame's pose as the arm's joints move. Because +the gripper frame is a child of the arm, it tracks the flange automatically: +when the arm moves, the gripper's tool center point moves with it, and the +planner reasons about the tool's position at every step of a candidate path. + +This tree is built from your saved configuration. To target a frame that is not +in the saved configuration, or to shift an existing one for a single motion, you +extend the tree at request time with a transform. + +## Extend the frame system with a WorldState transform + +`WorldState` carries supplementary transforms alongside obstacles. A transform +adds a frame to the frame system for the duration of one request, without +editing the saved configuration. You give the new frame a name, a parent frame +to attach to, and an offset pose relative to that parent. + +Attaching a transform as a child of the end effector creates a new frame offset +from it. This is how you reposition the point the planner aims for: define a +frame at the tip of a held object or at an offset from the gripper, then target +that frame in the destination. + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import ( + Transform, PoseInFrame, Pose, WorldState +) + +# A frame named "object-tip" 80mm beyond the gripper, attached to the gripper. +object_tip = Transform( + reference_frame="object-tip", + pose_in_observer_frame=PoseInFrame( + reference_frame="my-gripper", + pose=Pose(x=0, y=0, z=80, o_x=0, o_y=0, o_z=1, theta=0), + ), +) + +world_state = WorldState(transforms=[object_tip]) + +# Now Move can target "object-tip" as the frame to place at the destination. +await motion_service.move( + component_name="object-tip", + destination=destination, + world_state=world_state, +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +// A frame named "object-tip" 80mm beyond the gripper, attached to the gripper. +objectTip := referenceframe.NewLinkInFrame( + "my-gripper", + spatialmath.NewPoseFromPoint(r3.Vector{X: 0, Y: 0, Z: 80}), + "object-tip", + nil, +) + +worldState, err := referenceframe.NewWorldState( + nil, []*referenceframe.LinkInFrame{objectTip}) +if err != nil { + logger.Fatal(err) +} + +// Now Move can target "object-tip" as the frame to place at the destination. +_, err = motionService.Move(ctx, motion.MoveReq{ + ComponentName: "object-tip", + Destination: destination, + WorldState: worldState, +}) +``` + +{{% /tab %}} +{{< /tabs >}} + +Because the transform is a child of `my-gripper`, the new frame moves with the +gripper, and the planner places `object-tip` rather than the gripper's own +origin at the destination. When the request finishes, the frame disappears: the +saved frame system is unchanged. + +## WorldState versus the world state store service + +The names are similar, but `WorldState` and the world state store service are +different things, and only one affects a `Move`. + +- `WorldState` is the argument you pass to a single `Move` call. The obstacles + and transforms in it exist only for that request: the planner uses them to + plan this motion, and they are gone when the call returns. The transform in + the previous section is a `WorldState` transform, so it shapes the plan. +- The world state store service is a separate service that holds transforms for + client-side visualization in the [3D scene](/motion-planning/3d-scene/). The + motion planner does not read it, so publishing a transform there draws it in + the scene but has no effect on a `Move`. + +To change where the arm goes, put the transform in the `WorldState` you pass to +`Move`, as shown above. To render a custom visual without affecting planning, +publish it to the world state store service instead. + +## What's next + +- [Frame system overview](/motion-planning/frame-system/overview/): + configure parent, translation, orientation, and geometry for every component. +- [Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/): + a worked end effector configuration. +- [Move an arm to a pose](/motion-planning/move-an-arm/move-to-pose/): + target a frame with the motion service. +- [Attach and detach geometries](/motion-planning/obstacles/attach-detach-geometries/): + carry a grasped object's geometry with the end effector. diff --git a/docs/motion-planning/frame-system/overview.md b/docs/motion-planning/frame-system/overview.md index c4750aeca5..723ba6f9be 100644 --- a/docs/motion-planning/frame-system/overview.md +++ b/docs/motion-planning/frame-system/overview.md @@ -284,9 +284,18 @@ If a component's pose looks wrong, check the translation and orientation values in its frame configuration. For details on all CLI motion commands, see [Motion Service Configuration](/motion-planning/reference/motion-service/#cli-commands). +## Targeting frames in motion + +The motion service moves a named frame to a target pose, so which frame you +configure for an end effector determines where the arm actually goes. For how +`Move` resolves a component name to a frame, and how a `WorldState` transform +adds or repositions that frame for a single request, see +[End effector frames](/motion-planning/frame-system/end-effector-frames/). + ## Configure frames for your hardware {{< cards >}} +{{% card link="/motion-planning/frame-system/end-effector-frames/" noimage="true" %}} {{% card link="/motion-planning/frame-system/arm-gripper-camera/" noimage="true" %}} {{% card link="/motion-planning/frame-system/arm-fixed-camera/" noimage="true" %}} {{% card link="/motion-planning/frame-system/mobile-base-sensors/" noimage="true" %}} diff --git a/docs/motion-planning/move-an-arm/constraints.md b/docs/motion-planning/move-an-arm/constraints.md index 91e14eec51..bc0a72af33 100644 --- a/docs/motion-planning/move-an-arm/constraints.md +++ b/docs/motion-planning/move-an-arm/constraints.md @@ -113,6 +113,15 @@ constraints), see For `CollisionSpecification` (allow specific frame pairs to collide), see [Allow frame collisions](/motion-planning/obstacles/allow-frame-collisions/). +## Relax the goal with a pose cloud + +Constraints restrict the _path_ between poses. The complementary tool is to +relax the _goal_ itself. If your task does not require an exact destination pose, +you can give the planner a region of acceptable poses instead of a single one, +which enlarges the solution set and makes planning faster and more reliable. This +is a pose cloud, and it is often the right move when a goal is failing or +planning slowly. See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/). + ## Performance considerations Every constraint adds a check to every candidate path segment, and diff --git a/docs/motion-planning/move-an-arm/multiple-waypoints.md b/docs/motion-planning/move-an-arm/multiple-waypoints.md new file mode 100644 index 0000000000..bd02b59f80 --- /dev/null +++ b/docs/motion-planning/move-an-arm/multiple-waypoints.md @@ -0,0 +1,164 @@ +--- +linkTitle: "Move through waypoints" +title: "Move an arm through multiple waypoints" +weight: 40 +layout: "docs" +type: "docs" +description: "Plan a single continuous trajectory through an ordered list of intermediate goals using armplanning.PlanMotion." +--- + +Sometimes the endpoint is not enough. You need the arm to pass through a +specific approach point before reaching a target, to thread between two pieces +of equipment in a set order, or to follow a sequence of stations in one motion. +A single `Move` to the final pose lets the planner choose any collision-free +path, which may skip the intermediate points you care about. + +`armplanning.PlanMotion` accepts an ordered list of goals and plans one +continuous trajectory that hits each in turn. This page shows how to build a +multi-goal request, mix goal types across waypoints, and recover a partial +result when a later waypoint is infeasible. + +{{% alert title="Go SDK only" color="note" %}} +Routing one plan through multiple goals uses `armplanning.PlanMotion`, an +in-process Go function. The motion service `Move` API takes a single +destination, so multi-waypoint planning runs in Go alongside the planner rather +than over the service API. For the difference between planning in process and +calling the service, see [Verify a motion plan](/motion-planning/verify-a-plan/). +{{% /alert %}} + +## Why route through ordered goals + +Chaining separate `Move` calls is not the same as one multi-goal plan: + +- Each `Move` plans and executes independently, so the arm stops at every + waypoint instead of flowing through it. +- Each call plans from scratch with no knowledge of the goals that follow, so + it can reach a waypoint in a configuration that makes the next one + unreachable. +- There is no single trajectory you can inspect, validate, or replay. + +A multi-goal `PlanRequest` produces one trajectory spanning every segment. The +planner hits each goal in the order you list it, and uses each waypoint's solved +configuration as the start of the next segment, so the whole motion is +continuous and consistent. + +## Build a multi-goal PlanRequest + +`PlanRequest.Goals` is an ordered list of `PlanState` values. Each `PlanState` +is either a set of Cartesian poses (`FrameSystemPoses`) or a joint configuration +(`FrameSystemInputs`), and you can mix the two across waypoints. The +`StartState` gives the configuration the plan begins from. + +```go +import ( + "go.viam.com/rdk/motionplan/armplanning" + "go.viam.com/rdk/referenceframe" + "go.viam.com/rdk/spatialmath" + "github.com/golang/geo/r3" +) + +// Build the frame system from the running machine. +fsCfg, err := machine.FrameSystemConfig(ctx) +if err != nil { + logger.Fatal(err) +} +fs, err := referenceframe.NewFrameSystem("robot", fsCfg.Parts, nil) +if err != nil { + logger.Fatal(err) +} + +// Start from a known arm configuration. +startState := armplanning.NewPlanState(nil, referenceframe.FrameSystemInputs{ + "my-arm": []referenceframe.Input{0, 0, 0, 0, 0, 0}, +}) + +goals := []*armplanning.PlanState{ + // Waypoint 1: a Cartesian approach pose for the gripper. + armplanning.NewPlanState(referenceframe.FrameSystemPoses{ + "my-gripper": referenceframe.NewPoseInFrame("world", + spatialmath.NewPose( + r3.Vector{X: 400, Y: 0, Z: 300}, + &spatialmath.OrientationVectorDegrees{OZ: -1}, + )), + }, nil), + // Waypoint 2: a specific joint configuration. + armplanning.NewPlanState(nil, referenceframe.FrameSystemInputs{ + "my-arm": []referenceframe.Input{0, -1, 1, 0, 1, 0}, + }), + // Waypoint 3: a relaxed goal expressed as a pose cloud. + armplanning.NewPlanState(referenceframe.FrameSystemPoses{ + "my-gripper": referenceframe.NewPoseInFrameWithGoalCloud("world", + spatialmath.NewPose( + r3.Vector{X: 350, Y: 200, Z: 250}, + &spatialmath.OrientationVectorDegrees{OZ: -1}, + ), + &referenceframe.PoseCloud{X: 5, Y: 5, Z: 5, Theta: 10}), + }, nil), +} + +plan, _, err := armplanning.PlanMotion(ctx, logger, &armplanning.PlanRequest{ + FrameSystem: fs, + StartState: startState, + Goals: goals, +}) +if err != nil { + logger.Fatal(err) +} +``` + +Each goal names the frame it constrains, so a Cartesian waypoint can target the +gripper frame while a configuration waypoint pins the arm's joints. Pose clouds +work as goals too, which lets you relax the waypoints that do not need an exact +pose. See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/) for when to +use them. + +## Read the trajectory + +The returned `Plan` is one continuous trajectory across all segments. +`plan.Trajectory()` is a slice of joint configurations, one per step, from the +start state through the last goal: + +```go +for i, step := range plan.Trajectory() { + fmt.Printf("step %d: %v\n", i, step["my-arm"]) +} +``` + +There is no break in the trajectory at a waypoint: the steps that reach +waypoint 1 flow directly into the steps that leave it for waypoint 2. This is +the property you lose when you chain separate `Move` calls, where each call is +its own plan with a full stop in between. + +## Recover a partial plan when a waypoint fails + +If a later waypoint is unreachable, the whole request fails by default and you +get no trajectory, even for the waypoints that did solve. Set +`ReturnPartialPlan` to get the trajectory up to the last waypoint the planner +solved: + +```go +opts := armplanning.NewBasicPlannerOptions() +opts.ReturnPartialPlan = true + +plan, _, err := armplanning.PlanMotion(ctx, logger, &armplanning.PlanRequest{ + FrameSystem: fs, + StartState: startState, + Goals: goals, + PlannerOptions: opts, +}) +``` + +The partial trajectory ends at the last solved waypoint, which tells you which +waypoint is infeasible: if a three-goal request returns a plan that stops at +waypoint 2, waypoint 3 is the one to investigate. From there, loosen that +waypoint (for example with a pose cloud), move it, or check that it is reachable +and collision-free. + +## What's next + +- [Pose clouds](/motion-planning/move-an-arm/pose-clouds/): + relax a waypoint into a region of acceptable poses. +- [Verify a motion plan](/motion-planning/verify-a-plan/): + plan without executing to check feasibility first. +- [How motion planning works](/motion-planning/how-planning-works/): + why a goal can be infeasible and what to try. diff --git a/docs/motion-planning/move-an-arm/overview.md b/docs/motion-planning/move-an-arm/overview.md index b88e2c4c49..e51bd94129 100644 --- a/docs/motion-planning/move-an-arm/overview.md +++ b/docs/motion-planning/move-an-arm/overview.md @@ -57,6 +57,8 @@ from the table above. {{% card link="/motion-planning/move-an-arm/move-to-pose/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/move-with-constraints/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/move-by-joint-positions/" noimage="true" %}} +{{% card link="/motion-planning/move-an-arm/multiple-waypoints/" noimage="true" %}} +{{% card link="/motion-planning/move-an-arm/pose-clouds/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/pick-an-object/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/place-an-object/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/move-an-arm/pose-clouds.md b/docs/motion-planning/move-an-arm/pose-clouds.md new file mode 100644 index 0000000000..b54ba8f51c --- /dev/null +++ b/docs/motion-planning/move-an-arm/pose-clouds.md @@ -0,0 +1,126 @@ +--- +linkTitle: "Pose clouds" +title: "Relax a goal with a pose cloud" +weight: 45 +layout: "docs" +type: "docs" +description: "Give the planner a region of acceptable destinations instead of a single exact pose, so it can find a solution faster and more reliably." +--- + +A single target pose tells the planner exactly where the end effector must end +up, down to the millimeter and the degree. For many tasks that is more precise +than you need. Dropping a part into a bin, touching a flat surface, or pointing a +tool roughly at a target all have slack in them, and demanding an exact pose +throws that slack away. An over-constrained goal makes the planner work harder, +fail more often, and sometimes reject a goal that a slightly looser target would +have reached. + +A _pose cloud_ replaces the exact goal with a region of acceptable poses. You +give the planner a target pose plus a set of tolerances, and any pose within +those tolerances counts as having arrived. This page explains what a pose cloud +relaxes, why its tolerances are measured in the target's own frame, and how to +pass one to the motion service. + +## How a pose cloud relaxes a goal + +A pose cloud defines a tolerance, or leeway, on each component of the goal pose. +Each leeway permits deviation in the range from its negative to its positive +value, so the goal becomes a box around the target pose rather than a single +point. + +| Field | Units | Relaxes | +| ---------------- | -------- | ----------------------------------------------------------------- | +| `X`, `Y`, `Z` | mm | Position along each axis | +| `OX`, `OY`, `OZ` | unitless | The orientation vector components (the direction the tool points) | +| `Theta` | degrees | Rotation about the orientation axis | + +A leeway of zero on a field holds that component to the exact target value; a +larger leeway gives the planner more room on that component. You relax only what +the task allows: a part dropped into a wide bin can take large `X` and `Y` +leeways while keeping `Z` tight, and a tool that may spin freely about its axis +can take a large `Theta` while keeping its pointing direction fixed. + +The more freedom you give the planner, the larger the set of arm configurations +that satisfy the goal, so inverse kinematics is more likely to find a solution +and the search reaches it faster. A goal that fails as an exact pose often +succeeds as a pose cloud. + +## Tolerances are measured in the target's frame + +The leeways are evaluated in the reference frame of the target, not in world +coordinates. This matters whenever the target is tilted relative to the world. + +Consider a gripper approaching an inclined surface. You want to allow slack +_along_ the surface but stay tight _into_ it, so the tool does not push through. +Expressed in world coordinates that is an awkward mix of all three axes, because +the surface is tilted. Expressed in the surface's own frame it is simple: large +leeway on the two in-plane axes, tight leeway on the axis normal to the surface. +Because pose clouds use the target's frame, you describe the tolerance the way +the task is shaped rather than translating it into world axes by hand. + +## Use a pose cloud with the motion service + +In Go, attach a pose cloud to the destination with +`NewPoseInFrameWithGoalCloud`, then pass that destination to `Move`. The pose is +the center of the region and the `PoseCloud` is the leeway around it. + +```go +import ( + "go.viam.com/rdk/services/motion" + "go.viam.com/rdk/referenceframe" + "go.viam.com/rdk/spatialmath" +) + +// Point the tool at the cup, but allow slack in the approach orientation. +destination := referenceframe.NewPoseInFrameWithGoalCloud( + "cup", + spatialmath.NewPoseFromOrientation( + &spatialmath.OrientationVectorDegrees{OX: 1, OZ: 0, Theta: 180}, + ), + &referenceframe.PoseCloud{OX: 1, OY: 1, OZ: 0.1, Theta: 10}, +) + +_, err = motionService.Move(ctx, motion.MoveReq{ + ComponentName: "my-gripper", + Destination: destination, +}) +if err != nil { + logger.Fatal(err) +} +``` + +Here the destination is expressed in the `cup` frame, so the leeways apply +relative to the cup. The planner is free to satisfy the goal with any gripper +pose inside the cloud. + +{{% alert title="Available through the Go SDK" color="note" %}} +The goal cloud travels on the destination's `PoseInFrame`, which is part of the +`Move` request sent to the motion service, so the planner honors it server side. +The convenience constructor `NewPoseInFrameWithGoalCloud` is part of the Go SDK; +other SDKs do not yet provide an equivalent helper. +{{% /alert %}} + +## When to reach for a pose cloud + +- **Slack placement.** Dropping or setting down an object where any spot in a + region is acceptable. Relax the in-plane position and the spin about the + vertical. +- **Surface contact.** Touching, wiping, or sanding a surface where the contact + point can move along the surface but must stay on it. Relax the in-plane axes, + keep the normal tight. +- **Faster, more reliable planning.** Any goal that is failing or planning + slowly as an exact pose. Relaxing the components the task does not constrain + enlarges the solution set and often turns a failing plan into a quick success. + +If the task genuinely requires an exact pose, use a plain destination. A pose +cloud is for the common case where "close enough" is correct and exactness only +makes the planner's job harder. + +## What's next + +- [Configure motion constraints](/motion-planning/move-an-arm/constraints/): + restrict the path between poses, the complement to relaxing the goal. +- [Move through waypoints](/motion-planning/move-an-arm/multiple-waypoints/): + use pose clouds for the waypoints that do not need an exact pose. +- [How motion planning works](/motion-planning/how-planning-works/): + why a larger solution set makes the search succeed more often. diff --git a/docs/motion-planning/verify-a-plan.md b/docs/motion-planning/verify-a-plan.md new file mode 100644 index 0000000000..b2b3507fe7 --- /dev/null +++ b/docs/motion-planning/verify-a-plan.md @@ -0,0 +1,183 @@ +--- +linkTitle: "Verify a plan" +title: "Verify a motion plan without executing it" +weight: 60 +layout: "docs" +type: "docs" +description: "Use armplanning.PlanMotion to compute and inspect a trajectory before the arm moves, and learn how to get the same plan over the motion service API." +--- + +Before an arm moves, you often want to know whether the motion is even possible: +whether a goal is reachable, whether a path exists around the obstacles you have +declared, or what trajectory the planner would produce. Executing a `Move` to +find out is slow and, on real hardware, risky. Planning without executing +answers those questions first, then leaves it to you whether to run the result. + +`armplanning.PlanMotion` runs the same planner the motion service uses, but +returns the `Plan` instead of moving the arm. This page shows how to compute and +inspect a plan in process, and closes with how to get the same plan over the +motion service API from a remote client. + +## When to plan without executing + +- **Preview a motion.** Inspect the trajectory the planner produces before + committing the arm to it. +- **Check feasibility.** Confirm a goal is reachable and a collision-free path + exists, without driving the arm into a failed attempt. +- **Validate from a hypothetical state.** Plan from a start configuration the + arm is not currently in, to test reachability before you move there. + +## PlanMotion compared to Move + +`Move` and `PlanMotion` share the planner, but differ in what they do with the +result and where they run: + +- `Move` is a motion service API call. It plans, then executes, moving the arm. +- `PlanMotion` is an in-process Go function. It plans and returns the trajectory. + Nothing moves until you execute the result yourself. + +Because `PlanMotion` runs in process, you assemble the planning inputs directly +rather than letting the service collect them from the robot. That is more setup, +but it gives you the plan as data with no side effects. + +## Assemble a PlanRequest + +A `PlanRequest` carries everything the planner needs: the frame system, the +world state, the start state, and one or more goals. Build the frame system from +the running machine, set the start state to the configuration you want to plan +from, and give the goal as a pose for a named frame. + +```go +import ( + "go.viam.com/rdk/motionplan/armplanning" + "go.viam.com/rdk/referenceframe" + "go.viam.com/rdk/spatialmath" + "github.com/golang/geo/r3" +) + +// Frame system from the running machine. +fsCfg, err := machine.FrameSystemConfig(ctx) +if err != nil { + logger.Fatal(err) +} +fs, err := referenceframe.NewFrameSystem("robot", fsCfg.Parts, nil) +if err != nil { + logger.Fatal(err) +} + +// Plan from a known start configuration. Use the arm's current joints to +// verify a real move, or any configuration to test a hypothetical one. +startState := armplanning.NewPlanState(nil, referenceframe.FrameSystemInputs{ + "my-arm": []referenceframe.Input{0, 0, 0, 0, 0, 0}, +}) + +// Goal: place the gripper frame at a pose in the world frame. +goal := armplanning.NewPlanState(referenceframe.FrameSystemPoses{ + "my-gripper": referenceframe.NewPoseInFrame("world", + spatialmath.NewPose( + r3.Vector{X: 400, Y: 0, Z: 300}, + &spatialmath.OrientationVectorDegrees{OZ: -1}, + )), +}, nil) + +// Fail fast on an infeasible goal instead of grinding for the full default. +opts := armplanning.NewBasicPlannerOptions() +opts.Timeout = 15 + +plan, _, err := armplanning.PlanMotion(ctx, logger, &armplanning.PlanRequest{ + FrameSystem: fs, + StartState: startState, + Goals: []*armplanning.PlanState{goal}, + PlannerOptions: opts, +}) +``` + +The default planner timeout is 300 seconds, so an infeasible plan grinds for +five minutes before failing. When you are verifying feasibility, set a shorter +`Timeout` so an impossible goal fails quickly. + +## Read the plan + +If `PlanMotion` returns without an error, the goal is feasible from the start +state: the planner found a complete, collision-free trajectory. `plan.Trajectory()` +is that trajectory, a sequence of joint configurations the arm would pass +through. + +```go +if err != nil { + logger.Infof("goal is not feasible: %v", err) + return +} + +traj := plan.Trajectory() +logger.Infof("feasible: %d steps", len(traj)) +for i, step := range traj { + fmt.Printf("step %d: %v\n", i, step["my-arm"]) +} +``` + +An error means no plan was found within the timeout, which is your feasibility +answer. A returned trajectory is yours to inspect, log, compare against an +expected path, or hand to the arm for execution once you are satisfied. + +## Plan over the wire with the motion service + +`PlanMotion` requires running Go in process with the machine. A remote client +that only has the motion service API can still get a plan without executing, by +sending the plan request through the service's `DoCommand` interface. The +built-in motion service handles a `"plan"` command that runs the same planner as +`Move` and returns the trajectory without moving the arm. + +The mechanism is: build a `MoveRequest`, serialize it, and send it under the key +`"plan"`. + +```go +import "google.golang.org/protobuf/encoding/protojson" + +req := motion.MoveReq{ + ComponentName: "my-gripper", + Destination: destination, +} +pbReq, err := req.ToProto("builtin") +if err != nil { + logger.Fatal(err) +} +payload, err := protojson.Marshal(pbReq) +if err != nil { + logger.Fatal(err) +} + +// The key must be the lowercase string "plan". +resp, err := motionService.DoCommand(ctx, map[string]interface{}{ + "plan": string(payload), +}) +if err != nil { + logger.Fatal(err) +} +// resp carries the planned trajectory under the "plan" key. +``` + +The trade-offs versus the in-process call: + +- The response is untyped. The trajectory comes back as generic map data, and + its exact shape depends on the transport, so you parse it by hand rather than + receiving a typed `Plan`. +- You drive it with string keys instead of typed parameters, which is easier to + get wrong. +- It works for any remote client of the motion service, with no need to + assemble the frame system or pull the planner into your process. + +Use the in-process `PlanMotion` when you have Go access to the machine and want +the plan as typed data. Use the `"plan"` `DoCommand` when you are a remote client +and the motion service is all you have. The +[viamkit](https://github.com/viam-labs/viamkit) library wraps the `DoCommand` +form and smooths over the response parsing. + +## What's next + +- [Move through waypoints](/motion-planning/move-an-arm/multiple-waypoints/): + plan one trajectory through an ordered list of goals. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. +- [Debug motion with the CLI](/motion-planning/debug-motion-with-cli/): + inspect frames and poses from the command line. From e56850f3faa494de944ebba7c60c04b057cb3fbe Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 18 Jun 2026 17:04:16 -0600 Subject: [PATCH 02/46] Add Visualization section and reframe Debug a motion plan Add a new top-level Visualization section covering the 3D scene and the world state store service, and reframe the motion planning debug page around publishing a plan as custom visuals. - visualization/_index.md: what the 3D scene renders, an improved frame system description, and built-in vs custom visuals. - visualization/visuals-and-collisions.md: transform anatomy, UUIDs, geometry types, and why a visual is not an obstacle the planner avoids. - visualization/publish-visuals-from-a-module.md: implement a world state store service, build transforms with the draw library, poll-and-update loop, and visualizing a non-WSS resource by depending on it. - visualization/drawing-library.md: the draw library and the standalone Viam Visualization app. - motion-planning/3d-scene/debug-motion-plan.md: publish a plan's trajectory and goals as transforms (ComputePoses + draw), then diagnose against obstacles and reach. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../3d-scene/debug-motion-plan.md | 213 ++++++++++-------- docs/visualization/_index.md | 94 ++++++++ docs/visualization/drawing-library.md | 96 ++++++++ .../publish-visuals-from-a-module.md | 170 ++++++++++++++ docs/visualization/visuals-and-collisions.md | 108 +++++++++ 5 files changed, 582 insertions(+), 99 deletions(-) create mode 100644 docs/visualization/_index.md create mode 100644 docs/visualization/drawing-library.md create mode 100644 docs/visualization/publish-visuals-from-a-module.md create mode 100644 docs/visualization/visuals-and-collisions.md diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index e6a4e7982a..0017c38efc 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -1,113 +1,128 @@ --- linkTitle: "Debug a motion plan" -title: "Debug a motion plan" +title: "Visualize and debug a motion plan" weight: 45 layout: "docs" type: "docs" -description: "Use the 3D scene to inspect the frame system and obstacle geometry behind a motion plan that failed or produced unexpected results." +description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." --- -When a motion plan fails with a collision error or produces a path that -does not look right, the **3D SCENE** tab lets you inspect the static view -of the world the planner sees at the moment you look: configured frame -positions, collision geometries, and the arm's current pose. Most -motion planning failures come down to one of a few problems that are -immediately visible in 3D. - -The **3D SCENE** tab does not replay motion plans and has no timeline, -scrubber, or plan-playback control. It is an inspector of the -configured frame system and live component poses. If you need to watch -a plan execute over time, step the arm to intermediate joint positions -manually and reinspect. - -## Prerequisites - -- A machine with an arm or gantry configured and at least one frame defined. -- A motion plan that is failing or producing unexpected results. - -## Diagnose the problem - -### 1. Open the 3D SCENE tab - -Navigate to your machine in the [Viam app](https://app.viam.com) and -click the **3D SCENE** tab. The scene loads your current frame system -configuration. - -If your machine is online, the scene connects for live pose data. -If your machine is offline, the scene still renders the configured -frame positions. - -### 2. Check frame positions - -In the **World** panel in the upper-left, expand the tree and click -each component in turn. The Details panel on the right shows the -selected entity's **world position** and **world orientation**, plus -editable **local position** and **local orientation** relative to the -parent frame. Compare these values to your physical measurements. - -Three common mismatches to look for: - -- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. -- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. -- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. - -### 3. Check obstacle geometry - -Obstacles appear as translucent shapes in the scene and as child rows -under their parent frame in the **World** panel. Select each obstacle -from the tree to see its **geometry** type and **dimensions** in the -Details panel. - -Verify that: - -- Every physical obstacle in your workspace has a corresponding - geometry in the scene. -- Each geometry covers the actual physical object. If a box geometry - is too small, the planner will find paths that clip the real - obstacle. -- Geometries are positioned correctly. A table surface defined at - `z: 0` when the arm base is at `z: 500` will not protect against - table collisions. - -If obstacles are missing or misplaced, see +The **3D SCENE** tab does not show motion plans on its own. It is a static +inspector of the configured frame system and live component poses: it has no +timeline, no scrubber, and no plan playback. To see a plan, the trajectory the +arm will follow, the goals it aims for, and how that path relates to your +obstacles, you publish the plan as **custom visuals** through a world state store +service. The scene then renders the path you can otherwise only read as numbers. + +This page shows how to turn a plan into transforms the scene can draw, and how to +use the rendered path to debug a plan that failed or moved unexpectedly. + +## Why publish the plan as custom visuals + +A plan is a sequence of joint configurations. Read as numbers it tells you +little; rendered in the scene it tells you immediately whether the path clips an +obstacle, swings wide, or aims at a target outside the arm's reach. Publishing +the plan as world state store transforms puts the trajectory and goals in the +same 3D view as the frames and obstacle geometry the planner used, so you can see +the path and the world together. + +## Convert the trajectory into poses + +A plan's `Trajectory` is a sequence of joint configurations +(`FrameSystemInputs`), one per step. The scene places geometry by pose, so +convert each step into end-effector poses with the frame system. `ComputePoses` +takes a configuration and returns the pose of each frame: + +```go +for i, step := range plan.Trajectory() { + poses, err := step.ComputePoses(fs) + if err != nil { + return err + } + gripperPose := poses["my-gripper"].Pose() + // Place a marker for this step at gripperPose (next section). + _ = i + _ = gripperPose +} +``` + +Each `step` is the arm's configuration at that point in the trajectory, and +`poses["my-gripper"]` is where the gripper sits in that configuration. + +## Build transforms for the plan + +With a pose per step, build the visuals with the `draw` library: a marker per +trajectory step and a marker for each goal. Give each a stable UUID so the scene +can update them when you re-plan. + +```go +import ( + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { + sphere, err := spatialmath.NewSphere(pose, 5, fmt.Sprintf("step-%d", i)) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(stepColor)) + if err != nil { + return nil, err + } + return drawn.Draw(fmt.Sprintf("step-%d", i), draw.WithPose(pose)) +} +``` + +Draw the goal poses the same way with a distinct color, so the target stands out +from the trajectory leading to it. + +## Serve the transforms to the scene + +Serve the transforms through a world state store service so the **3D SCENE** tab +renders them. The plan markers stream in alongside the frames and obstacle +geometry the scene already shows. For the service interface, the poll-and-update +loop, and how a module pulls data from its dependencies, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## Diagnose a failed or surprising plan + +With the plan rendered, debugging becomes visual. Compare the trajectory against +the rest of the scene: + +- **Where does the path collide?** If a step marker passes through an obstacle + geometry, that is where the planner reports a collision. Check whether the + obstacle is real or an oversized geometry. +- **Does the goal fall outside the arm's reach?** If a goal marker sits far from + any reachable arm configuration, the planner cannot get there. Move the goal or + check the frame system. +- **Why the detour?** An unexpected route usually means an obstacle is forcing + the planner around it. Look for geometry between the start and goal you did not + intend to add. + +For checking the obstacle geometry itself, separate from the plan, see [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). -### 4. Look for impossible targets - -If the motion plan target is outside the arm's reach or inside an -obstacle, the planner cannot find a path. - -In the **World** panel, expand the arm and select its tip link or the -gripper frame (whichever is configured as the motion target). Read the -**world position** and compare it to the target pose your code -commanded. Then mentally place the target: is it inside an obstacle -geometry? Is it further than the arm can reach from its base? - -If the target is inside an obstacle geometry, either move the target -or adjust the obstacle definition. - -### 5. Check for self-collision geometry +## When to visualize versus inspect or verify -Some arm models include collision geometry for each link. If a motion -plan fails with a self-collision error, inspect the arm's link -geometries for overlap in the current configuration. +The 3D scene serves three distinct purposes, and it helps to keep them straight: -If the arm renders without collision geometry, the feature may be -disabled: open **Settings → Scene → Arm Models** and verify that the -rendering mode includes colliders. +- **Visualize a plan** (this page): publish the trajectory and goals as custom + visuals to see the path in context. +- **Inspect static frames and geometry**: use the stock scene to check frame + positions and obstacle coverage with no plan involved. +- **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is + reachable and a path exists before you visualize or execute anything. -Self-collisions can happen when wrist joints are commanded to -positions that bring adjacent links too close together. If the -overlap is a modeling artifact rather than a real collision, allow -the frame pair with -[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). +Visualization shows you _what the path looks like_; static inspection shows you +_what the world looks like_; feasibility checking tells you _whether a plan +exists at all_. Reach for the one that matches the question you are asking. -## Common causes of motion plan failures +## What's next -| Symptom | Likely cause | What to check in 3D SCENE | -| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | -| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | -| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | -| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | -| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | -| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + the world state store service that serves these transforms. +- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): + check obstacle geometry against the real workspace. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. diff --git a/docs/visualization/_index.md b/docs/visualization/_index.md new file mode 100644 index 0000000000..247a330c02 --- /dev/null +++ b/docs/visualization/_index.md @@ -0,0 +1,94 @@ +--- +linkTitle: "Visualization" +title: "Visualization" +weight: 165 +layout: "docs" +type: "docs" +no_list: true +description: "See your machine in 3D: frames, geometries, point clouds, and custom visuals your modules publish." +--- + +Spatial configuration is invisible in JSON. A frame translation of +`{x: 50, y: 0, z: 110}` tells you nothing about whether the gripper actually +sits where the arm needs it, and a list of obstacle geometries tells you nothing +about whether they cover the real workspace. Visualization renders these in 3D +so you can see the spatial model your machine is actually working with, catch +misconfigurations before they cause a failure, and watch live data in context. + +This section covers the **3D SCENE** tab in the Viam app, the transforms and +metadata that make up a custom visual, how a module publishes visuals, and the +standalone Viam Visualization app for previewing spatial data outside the app. + +## What the 3D scene renders + +The **3D SCENE** tab on your machine's page renders an interactive 3D view of +your machine. It draws four kinds of content, each from a different source: + +- **Component frames**, from the [frame system](#the-frame-system). Each + configured component appears as a set of coordinate axes at its computed + position. +- **Geometries**, from the components' `frame.geometry` configuration and from + obstacles. These render as translucent shapes the motion planner uses for + collision checking. +- **Point clouds**, from depth cameras, rendered as colored point sets. +- **Custom visuals**, from a [world state store service](/visualization/publish-visuals-from-a-module/). + A module publishes these at runtime, and the scene streams them in as they + change. + +The scene reads your saved configuration, and when the machine is online it +connects for live data, so frames move to their current poses and point clouds +update as the cameras report them. + +## The frame system + +Everything in the scene is positioned by the **frame system**: the single +coordinate tree that records where every component sits relative to every other. + +Each component has a frame with a parent, a translation, and an orientation. +A component's frame is defined relative to its parent's frame, not to the world +directly, so the frames form a tree rooted at the fixed **world frame**. A +gripper's frame is a child of the arm's, the arm's is a child of the world, and +so on. Because the tree is connected, the frame system can compute the transform +between any two frames: it walks the path between them, composing each frame's +local transform along the way. When the arm's joints move, every frame below it +in the tree moves with it automatically. + +This is what makes the scene meaningful. A point cloud from a wrist camera and an +obstacle defined in world coordinates can be drawn in the same view because the +frame system relates both back to the world frame. The same machinery lets the +motion planner reason about where the gripper is while the arm moves, and lets a +custom visual attach itself to a moving component. + +For configuring frames in detail (parent, translation, orientation, geometry), +see the [frame system documentation](/motion-planning/frame-system/). + +## Built-in content versus custom visuals + +Two kinds of content reach the scene by two different routes, and the difference +determines where you change them: + +- **Built-in content** (component frames and configured geometry) comes from + your machine **configuration**. You change it by editing the frame system and + obstacle config. The scene reads it directly. +- **Custom visuals** come from a **module** that implements a world state store + service and publishes transforms at runtime. You change them in **code**, not + config, and the scene streams them in as the module adds, updates, and removes + them. + +If a frame or geometry looks wrong, fix the configuration. If a custom visual +looks wrong, the module that publishes it is where to look. + +## Topics + +{{< cards >}} +{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} +{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} +{{% card link="/visualization/drawing-library/" noimage="true" %}} +{{< /cards >}} + +## Use the 3D scene with motion planning + +{{< cards >}} +{{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} +{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md new file mode 100644 index 0000000000..0cbbb14f5e --- /dev/null +++ b/docs/visualization/drawing-library.md @@ -0,0 +1,96 @@ +--- +linkTitle: "Drawing library" +title: "The drawing library and Viam visualization" +weight: 30 +layout: "docs" +type: "docs" +description: "Use the draw library to build visuals, and run the standalone Viam Visualization app to preview spatial data from a Go client." +--- + +Viam Visualization is a standalone 3D app you run yourself. Unlike the **3D +SCENE** tab in the Viam app, which renders a machine's world state store service, +Viam Visualization is a separate tool for monitoring, testing, and debugging +spatial data: you start it locally and push visuals to it from your own Go code. +It shares the same `draw` library used to build world state store transforms, so +the visuals you construct are the same either way. + +This page covers what the drawing library is, the primitives it exposes, and how +to run the app and drive it from a Go client. + +## Viam Visualization versus the 3D scene tab + +The two render 3D visuals, but they are different tools for different moments: + +- The **3D SCENE tab** lives in the Viam app and renders a machine's world state + store service. It is the in-app view of a configured machine. +- **Viam Visualization** is a standalone app you run on your own machine and push + to from a client. It is for previewing and debugging spatial data while you + develop, without deploying a module or opening the Viam app. + +Reach for the 3D scene tab to inspect a running machine; reach for Viam +Visualization to iterate on spatial data from a script or test. + +## What the drawing library is + +The `draw` library turns geometry, pose, and styling metadata into the entities +the renderer displays. Instead of assembling proto structs by hand, you describe +a shape and its appearance and the library produces a correct, fully-formed +visual with the right identifiers and metadata. The same library backs both +render targets: it builds the `commonpb.Transform` values a world state store +module serves, and the entities you push to Viam Visualization. + +## Drawing primitives and styling + +The library exposes a set of drawable shapes and styling options. Pick the +primitive that matches what you are drawing: + +- **arrows**: directions, normals, or vectors +- **lines**: paths, segments, or connections +- **points**: sampled data or markers +- **models**: detailed 3D meshes +- **NURBS**: smooth curves and surfaces + +Each visual takes styling options: color, opacity, per-point colors for point +data, and a label. Choosing the right primitive and styling keeps a busy scene +readable, for example drawing a trajectory as a line, its waypoints as points, +and approach directions as arrows. + +## Construct visuals with the library + +Building visuals with the library rather than by hand keeps producer code +readable and guarantees the identifiers and metadata are correct. You construct +a shape with a styling option and let the library assemble the entity: + +```go +import "github.com/viam-labs/motion-tools/draw" + +shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) +``` + +Because the library owns the proto details, you work in terms of shapes and +styles, not field-by-field struct assembly. + +## Run the app and push from a Go client + +Viam Visualization runs locally and renders in your browser. You start the app, +then push visuals to it from Go with the client API. Reusing an entity ID updates +that visual in place; a new ID adds another: + +```go +import "github.com/viam-labs/motion-tools/client/api" + +// Update in place by reusing the ID; add a new entity with a different ID. +err := api.DrawGeometry(box, api.WithID("obstacle-1"), api.WithColor(red)) +``` + +This lets you preview spatial data, a point cloud, a set of detections, a planned +path, straight from a script or test, without deploying a module or connecting +through the Viam app. For setup, the local server, and the full client API, see +the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). + +## What's next + +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + use the same `draw` library to serve transforms to the in-app 3D scene. +- [Visuals and collisions](/visualization/visuals-and-collisions/): + what a transform contains and how the scene renders it. diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md new file mode 100644 index 0000000000..5630200fe5 --- /dev/null +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -0,0 +1,170 @@ +--- +linkTitle: "Publish visuals from a module" +title: "Publish custom visuals from a module" +weight: 20 +layout: "docs" +type: "docs" +description: "Implement a world state store service that pulls data from other resources, builds transforms with the draw library, and streams them to the 3D scene." +--- + +The 3D scene shows your frame system and configured geometry by default. To draw +anything else, a detected object, a planned path, a sensor's live readings as +shapes, you implement a **world state store service**. A module that implements +this service is a producer: it reads data from other resources, turns that data +into transforms, and streams them to the scene as they change. + +This page covers when to implement the service, the interface to implement, how +to build transforms with the `draw` library, and the poll-and-update loop that +keeps the scene in sync. The pattern throughout is **pull**: the module depends +on the resources it visualizes and reads their data, rather than having those +resources push into it. + +## When to implement a world state store service + +Implement one when you want custom visuals in the 3D scene that the default +content cannot show. The scene already draws component frames and configured +geometry, so you do not need a module to see those. You do need one to visualize +anything computed or sensed at runtime: a vision service's detections, a sensor's +obstacle readings, a motion plan's trajectory, or any annotation specific to your +application. + +## Implement the service interface + +The world state store service exposes three read methods, which the 3D scene +calls to discover and follow your visuals: + +- `ListUUIDs`: return the UUID of every transform you currently publish. +- `GetTransform`: return the transform for a given UUID. +- `StreamTransformChanges`: return a stream of change events so the scene + follows additions, updates, and removals without re-fetching everything. + +A typical implementation keeps the current transforms in a map keyed by UUID, +serves `ListUUIDs` and `GetTransform` from that map, and fans out change events +to subscribers from `StreamTransformChanges`. + +```go +func (s *visualizer) ListUUIDs( + ctx context.Context, extra map[string]any, +) ([][]byte, error) { + s.mu.RLock() + defer s.mu.RUnlock() + uuids := make([][]byte, 0, len(s.transforms)) + for id := range s.transforms { + uuids = append(uuids, []byte(id)) + } + return uuids, nil +} +``` + +## Build transforms with the draw library + +Rather than assembling `commonpb.Transform` protos by hand, use the `draw` +library from `github.com/viam-labs/motion-tools/draw`. You wrap a +`spatialmath.Geometry` with styling, then `Draw` it with an ID and pose to get a +`*commonpb.Transform`: + +```go +import ( + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +func buildTransform(o obstacle) (*commonpb.Transform, error) { + box, err := spatialmath.NewBox(o.Pose, o.Dims, o.ID) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(box, draw.WithGeometryColor(colorFor(o))) + if err != nil { + return nil, err + } + return drawn.Draw(o.ID, draw.WithID(o.ID), draw.WithPose(o.Pose)) +} +``` + +The library produces standard `commonpb.Transform` values, the same type the +service interface returns, so the transforms you build this way flow straight +through `ListUUIDs`, `GetTransform`, and `StreamTransformChanges` to the scene. + +## Drive a poll-and-update loop + +The module owns its update cadence. A ticker drives a loop that reads the +module's dependencies, builds the current set of transforms, diffs it against the +cached set, and emits one change event per difference: + +```go +func (s *visualizer) pollLoop() { + for range s.ticker.C { + readings, err := s.sensor.Readings(s.ctx, nil) + if err != nil { + continue + } + next := buildTransforms(readings) + + // Diff next against s.last and emit one event per change. + for id, tf := range next { + if prev, ok := s.last[id]; !ok { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_ADDED, nil) + } else if changed(prev, tf) { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_UPDATED, changedFields(prev, tf)) + } + } + for id, tf := range s.last { + if _, ok := next[id]; !ok { + s.emit(tf, pb.TransformChangeType_TRANSFORM_CHANGE_TYPE_REMOVED, nil) + } + } + s.last = next + } +} +``` + +Emitting added, updated, and removed events (with `UpdatedFields` on updates) +lets the scene apply incremental changes instead of re-rendering. The diff +against cached state is what turns a full snapshot each tick into a stream of +minimal updates. + +## The producer reads its dependencies + +The module is the producer, and the resources it visualizes are its +dependencies. The 3D scene reads only the world state store service; the service +reads everything else. Data flows one way: + +> dependency resources → world state store module (reads, builds transforms) → 3D scene + +This is why the loop above calls `s.sensor.Readings(...)`: the sensor is a +dependency, and the module pulls from it. + +## Visualize a resource that is not a world state store + +A module whose primary job is something else, an arm, a sensor, a planner, does +not push visuals into the scene. Instead, you write a world state store module +that takes that resource as a dependency and pulls from its existing API: + +- a sensor's `Readings` +- a component's geometry getters +- any resource's `DoCommand` + +The world state store module converts what it reads into transforms and streams +them. The visualized resource needs no changes and no awareness that it is being +drawn. The store depends on it, not the other way around. + +```go +func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstatestore.Service, error) { + obstacleSensor, err := sensor.FromProvider(deps, "obstacle-sensor") + if err != nil { + return nil, fmt.Errorf("getting obstacle-sensor: %w", err) + } + // The module depends on obstacle-sensor and pulls its readings on a loop. + return startVisualizer(obstacleSensor), nil +} +``` + +## What's next + +- [Visuals and collisions](/visualization/visuals-and-collisions/): + what a transform contains and why a visual is not an obstacle. +- [The drawing library and Viam visualization](/visualization/drawing-library/): + the `draw` primitives and the standalone visualizer app. +- [Frame system](/motion-planning/frame-system/): position the transforms you + publish. diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md new file mode 100644 index 0000000000..a1e74cb333 --- /dev/null +++ b/docs/visualization/visuals-and-collisions.md @@ -0,0 +1,108 @@ +--- +linkTitle: "Visuals and collisions" +title: "Visuals and collisions" +weight: 10 +layout: "docs" +type: "docs" +description: "How a Transform defines a custom visual, and why a visual in the scene is not the same as geometry the motion planner avoids." +--- + +A custom visual in the 3D scene is a `Transform`: a piece of geometry placed +somewhere in the frame system, with styling that controls how it draws. The +world state store service streams these transforms to the scene. This page +covers what a transform contains, how the scene tracks it over time, and the +distinction that trips people up most: a visual you can see is not automatically +an obstacle the motion planner avoids. + +## Anatomy of a transform + +A `Transform` carries four things that together place and style one visual: + +- **Reference frame and pose**: where the visual sits. The pose is given in a + parent reference frame, so the visual is positioned in the frame system and + moves with its parent. +- **Geometry**: the shape to draw (a box, sphere, capsule, mesh, or point + cloud). +- **Metadata**: styling such as color and opacity. +- **UUID**: a stable identifier for this specific visual. + +The reference frame and pose decide _where_, the geometry decides _what shape_, +and the metadata decides _how it looks_. + +## The UUID gives a visual a stable identity + +Each transform has a UUID. That identifier is what lets a module change one +visual without disturbing the rest of the scene: to update a visual, the module +re-sends a transform with the same UUID; to remove it, it references that UUID; +to add a new one, it uses a fresh UUID. Without stable identifiers the client +would have to re-render everything on every change. With them, the scene applies +incremental add, update, and remove operations to individual visuals. + +## Geometry types + +A transform's geometry is the shape the scene draws. The supported types are: + +- **box**: dimensions in millimeters +- **sphere**: a radius +- **capsule**: a radius and length +- **mesh**: an arbitrary triangle mesh +- **point cloud**: a set of points + +Choose the type that matches what you are representing: a box or capsule to +approximate a physical object, a mesh for a precise model, a point cloud for +sensor data. + +## Metadata styles the visual + +The metadata is a set of rendering attributes the scene reads when it draws the +geometry: + +- `color`: the fill color +- `opacity`: how transparent the shape is +- per-point colors: for point cloud geometry +- `collision_allowed`: a hint about whether the geometry represents an allowed + collision + +These are **visualization attributes**, not planning inputs. They change how a +visual looks in the scene. They do not change what the motion planner does, +including `collision_allowed`: setting it affects how the visual is presented, +not whether the planner treats anything as solid. + +## A visual is not an obstacle + +This is the distinction to internalize: the geometry on a world state store +transform renders in the 3D scene, but the motion planner does not read the +world state store. Publishing a box to the scene draws a box. It does not add an +obstacle the arm will avoid. + +The geometry the planner actually collision-checks comes from two places: + +- The **frame system**: each component's `frame.geometry`. +- The **`WorldState`** you pass to a `Move` call: obstacles and transforms + supplied for that single planning request. + +A transform in the world state store and an obstacle in a `WorldState` are +therefore different things on different paths, even when they describe the same +shape. + +## Making a geometry both visible and collision-checked + +If you want a geometry to appear in the scene _and_ be avoided by the planner, +you do both, separately: + +- **For the scene**: publish it as a transform through the world state store + service (see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/)). +- **For planning**: add it to the frame system, or include it in the + `WorldState` you pass to `Move`. + +There is no single field today that does both. Treat the visual and the +collision geometry as two outputs you produce from the same source data. + +## What's next + +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + implement a world state store service that publishes transforms. +- [Define obstacles](/motion-planning/obstacles/): the geometry the planner + collision-checks. +- [Frame system](/motion-planning/frame-system/): how the planner gets the + geometry and frames it plans around. From 9bbd7d2d3a5f431710dbe64d5292a47b7972754b Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Mon, 22 Jun 2026 11:03:39 -0600 Subject: [PATCH 03/46] Revise end-effector frames page per review Apply house-style fixes from the doc review: remove the negation-based opening and the "Understanding..." learning-objective leak, replace the vague "targets frames" wording with what Move actually does, convert "is not treated as hardware" to the positive "resolves to a frame", reframe the world state store contrast positively (noun-plus-job), and trim the description under the Hugo length limit. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- .../frame-system/end-effector-frames.md | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index 6ab5dfbc04..ccea002e46 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -4,26 +4,24 @@ title: "End effector frames" weight: 5 layout: "docs" type: "docs" -description: "How the motion service moves a named frame to a target pose, and how the frame attribute and WorldState transforms control which point on the robot that frame represents." +description: "How Move places a named frame at a target pose, and how the frame attribute and WorldState transforms set which point on the robot a frame represents." --- -When you ask the motion service to move an arm, you do not move "the arm." You -move a _frame_. The destination you pass is a pose for a named frame in the -frame system, and the planner solves for joint angles that place that frame's -origin at the destination. Understanding which frame you are moving, and where -that frame sits on the physical robot, is the difference between the tool tip -landing on the target and the arm's mounting flange landing there instead. +When you move an arm with the motion service, you move a _frame_. The +destination you pass is a pose for a named frame in the frame system, and the +planner solves for joint angles that place that frame's origin at the +destination. The frame you name decides whether the tool tip lands on the target +or the arm's mounting flange lands there instead. -This page explains how the motion service targets frames, how the `frame` -attribute defines an end effector frame, and how a `WorldState` transform lets -you add or reposition that frame for a single request. +This page explains how `Move` resolves a component name to a frame, how the +`frame` attribute defines an end effector frame, and how a `WorldState` +transform lets you add or reposition that frame for a single request. -## Move targets a frame, not a component +## Which frame Move acts on The `Move` request takes a component name and a destination. The component name -is not treated as a piece of hardware to actuate directly; it is resolved to a -frame in the frame system. The planner then searches for an arm configuration -that brings that frame's origin to the destination pose. +resolves to a frame in the frame system. The planner then searches for an arm +configuration that brings that frame's origin to the destination pose. ```python await motion_service.move( @@ -78,8 +76,8 @@ the gripper frame is a child of the arm, it tracks the flange automatically: when the arm moves, the gripper's tool center point moves with it, and the planner reasons about the tool's position at every step of a candidate path. -This tree is built from your saved configuration. To target a frame that is not -in the saved configuration, or to shift an existing one for a single motion, you +This tree is built from your saved configuration. To target a frame absent from +the saved configuration, or to shift an existing one for a single motion, you extend the tree at request time with a transform. ## Extend the frame system with a WorldState transform @@ -166,8 +164,9 @@ different things, and only one affects a `Move`. the previous section is a `WorldState` transform, so it shapes the plan. - The world state store service is a separate service that holds transforms for client-side visualization in the [3D scene](/motion-planning/3d-scene/). The - motion planner does not read it, so publishing a transform there draws it in - the scene but has no effect on a `Move`. + motion planner reads only the `WorldState` you pass to `Move`, so a transform + published to the store renders in the scene while planning stays driven by + `WorldState` alone. To change where the arm goes, put the transform in the `WorldState` you pass to `Move`, as shown above. To render a custom visual without affecting planning, From 47fc4e062e3b9433fb146f64eb6263d9f8f06bac Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 23 Jun 2026 09:00:46 -0600 Subject: [PATCH 04/46] updated the end effector frame doc and added images --- .../frame-system/arm-vs-gripper-frame.svg | 62 ++ .../frame-system/drill-bit-frame.svg | 43 ++ .../frame-system/end-effector-types.svg | 635 ++++++++++++++++++ .../frame-system/held-object-frame.svg | 476 +++++++++++++ .../frame-system/held-object-geometry.svg | 45 ++ 5 files changed, 1261 insertions(+) create mode 100644 assets/motion-planning/frame-system/arm-vs-gripper-frame.svg create mode 100644 assets/motion-planning/frame-system/drill-bit-frame.svg create mode 100644 assets/motion-planning/frame-system/end-effector-types.svg create mode 100644 assets/motion-planning/frame-system/held-object-frame.svg create mode 100644 assets/motion-planning/frame-system/held-object-geometry.svg diff --git a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg new file mode 100644 index 0000000000..e8d42024ed --- /dev/null +++ b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg @@ -0,0 +1,62 @@ + + + + + + + + + x + + z + + y + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + target_pose + + + + arm frame + + drill-tip frame + + + + + + The same target_pose moves a different point depending on which frame you name. + diff --git a/assets/motion-planning/frame-system/drill-bit-frame.svg b/assets/motion-planning/frame-system/drill-bit-frame.svg new file mode 100644 index 0000000000..c5e33dc086 --- /dev/null +++ b/assets/motion-planning/frame-system/drill-bit-frame.svg @@ -0,0 +1,43 @@ + + + + + + + + x + + z + + y + + + + + Adding a drill-tip frame to a cobot + + + + + + + + + + + + + + + + 80 mm + + + + my-arm frame + + drill-tip frame + + The drill-tip frame sits 80 mm along z from the arm's frame. A WorldState + transform adds it for one Move call, so the planner places the bit on target. + diff --git a/assets/motion-planning/frame-system/end-effector-types.svg b/assets/motion-planning/frame-system/end-effector-types.svg new file mode 100644 index 0000000000..befeb06ed0 --- /dev/null +++ b/assets/motion-planning/frame-system/end-effector-types.svg @@ -0,0 +1,635 @@ + + + + + + + + + + + + + + + + x + + z + + y + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Two-finger gripper + + + + arm frame + + gripper frame + + world frame + + Drill + + + + arm frame + + + + + + drill-tip frame + + world frame + Every end effector frame is defined relative to the arm frame at the flange and the world frame at the base. + diff --git a/assets/motion-planning/frame-system/held-object-frame.svg b/assets/motion-planning/frame-system/held-object-frame.svg new file mode 100644 index 0000000000..11cf5e9466 --- /dev/null +++ b/assets/motion-planning/frame-system/held-object-frame.svg @@ -0,0 +1,476 @@ + + + + + + + + + + + + + + + + x + + z + + y + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + grasped object + + + + arm frame + + + object-tip frame + gripper frame + Grasp an object, then add a frame at its tip + diff --git a/assets/motion-planning/frame-system/held-object-geometry.svg b/assets/motion-planning/frame-system/held-object-geometry.svg new file mode 100644 index 0000000000..ceb5536ba9 --- /dev/null +++ b/assets/motion-planning/frame-system/held-object-geometry.svg @@ -0,0 +1,45 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + collision geometry + + + + + + + obstacle + + Attach the object's geometry to the frame, and the planner routes the held object around obstacles. + From d69ce98d5dde280309108aa4cd8bc7c1ceb0edbe Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 23 Jun 2026 09:08:26 -0600 Subject: [PATCH 05/46] missed the frame file --- .../frame-system/end-effector-frames.md | 256 ++++++++++++------ 1 file changed, 177 insertions(+), 79 deletions(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index ccea002e46..340e5d3a94 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -7,48 +7,66 @@ type: "docs" description: "How Move places a named frame at a target pose, and how the frame attribute and WorldState transforms set which point on the robot a frame represents." --- -When you move an arm with the motion service, you move a _frame_. The -destination you pass is a pose for a named frame in the frame system, and the -planner solves for joint angles that place that frame's origin at the -destination. The frame you name decides whether the tool tip lands on the target -or the arm's mounting flange lands there instead. +When you move an arm with the motion service, you move a _frame_ from its current +pose to a destination pose. That frame is called the **end effector frame**, also called +the tool control point (TCP) or the manipulation control frame. Where the end effector +frame sits depends on the arm, the tool, and the task, and that frame can change partway +through a job. This page explains why the end effector frame exists, how `Move` resolves it, +and how you define and visualize it with Viam. + +First, look at where the end effector frame could be defined for two example use cases: + +{{}} + +The end effector frame is hardware and task dependent. For a two-finger gripper, the +end effector frame sits between the fingertips, a short offset below the arm's frame. For +a drill, it sits at the bit tip, offset from the arm's frame both forward and down, +and the orientation may be flipped, so a positive z would mean to drill deeper. +When you call `Move`, you have the option to specify which frame you are moving, and +using the arm frame is always an option. By creating an end effector frame at the +correct location, a `Move` call becomes easier and better represents the task being +performed. + +Now look at how the motion service describes the move. The snippets on this page +assume a connected motion service client, `motion_service` in Python and +`motionService` in Go. For how to set this up in more detail, see +[Access the motion service in your code](/reference/services/motion/#access-the-motion-service-in-your-code). -This page explains how `Move` resolves a component name to a frame, how the -`frame` attribute defines an end effector frame, and how a `WorldState` -transform lets you add or reposition that frame for a single request. - -## Which frame Move acts on +```python +from viam.proto.common import Pose, PoseInFrame -The `Move` request takes a component name and a destination. The component name -resolves to a frame in the frame system. The planner then searches for an arm -configuration that brings that frame's origin to the destination pose. +# A pose in world frame facing Z Down +target_pose = PoseInFrame( + reference_frame="world", + pose=Pose(x=400, y=0, z=300, o_x=0, o_y=0, o_z=-1, theta=0), +) -```python +# Move the Gripper Frame to the target_pose await motion_service.move( component_name="my-gripper", - destination=destination, + destination=target_pose, ) ``` -Passing `"my-gripper"` here moves the gripper's frame, so the destination -describes where the gripper should end up. Passing `"my-arm"` instead moves the -arm's frame, which sits at the arm's mounting flange. The two are different -physical points, so the same destination pose produces different motions. Name -the frame whose position you actually care about. +You pass two parameters to `Move`: a string `component_name` and a `destination` +pose. -This is why motion is expressed in terms of frames: a single arm carries many -frames worth targeting (the flange, a gripper's tool center point, a wrist -camera), and the planner needs to know which one you mean. +## Which frame Move resolves -## Define an end effector frame with the frame attribute +The `destination` you pass is a target pose for the resolved frame, and the planner +tries to find a collision-free motion that moves the frame to the destination. -A component's frame is defined through its `frame` attribute, the same -parent, translation, and orientation you configure for any component. For an -end effector, the translation is what moves the frame origin from the arm's -flange out to the working point. +The `component_name` can be a component name or a frame name. The motion service +looks for a frame with that name in the frame system. When you pass a component name +such as `"my-arm"` or `"my-gripper"`, Viam supplies a frame for it behind the scenes: -A gripper whose tool center point sits 120 mm past the flange uses a frame like -this: +- **An arm**: its kinematics file (URDF or Viam JSON) defines a single output frame at the + terminal link of the kinematic chain, commonly called the tool flange. That output frame + is the arm's built-in end effector frame. +- **Another component**, such as a gripper, defines a static frame through its `frame` + attribute, which you set on the component's configuration card. + +Here is a gripper `frame` attribute from the [Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/) example, whose origin sits 120 mm past the flange: ```json { @@ -61,36 +79,46 @@ this: } ``` -With this frame configured, calling `Move` with `component_name="my-gripper"` -positions the point 120 mm past the flange at the destination. Without it, the -planner has no model of the tool's length and positions the flange instead, so -the physical tool overshoots the target by 120 mm. For a worked example with an -arm, gripper, and camera, see -[Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/). +## Why the frame you name matters + +A `Move` call that uses the arm frame produces different motion than one that +uses the drill-tip frame. The arm's frame sits at the flange, and the drill-tip +frame sits at the bit tip, so the same `target_pose` produces different motions. +In the example below, the target pose is defined on the face of a box. If you use +the drill-tip frame, the bit moves to the box face. If you use the arm frame, the +drill drives into the box, because the planner tries to move the arm frame to the +target pose. + +{{}} -## How the frame system maintains frames +## Define an end effector frame in code -The frame system stores every configured frame in a single tree rooted at the -world frame, and recomputes each frame's pose as the arm's joints move. Because -the gripper frame is a child of the arm, it tracks the flange automatically: -when the arm moves, the gripper's tool center point moves with it, and the -planner reasons about the tool's position at every step of a candidate path. +The `frame` attribute described above works well when the end effector frame is static. +When the end effector frame moves during a task, or when defining it in code suits +the job better, add a new frame to the frame system with a `WorldState` transform. -This tree is built from your saved configuration. To target a frame absent from -the saved configuration, or to shift an existing one for a single motion, you -extend the tree at request time with a transform. +A great example is a grasped part. Prior to pickup, the end effector frame is located +with the gripper. After pickup, the end effector frame may need to move to the part for +an assembly step. The grasped part may not have been in the machine configuration, +so you add its frame to the frame system at request time with a `WorldState` transform, +attached to the gripper so it moves with the grasp: -## Extend the frame system with a WorldState transform +{{}} -`WorldState` carries supplementary transforms alongside obstacles. A transform -adds a frame to the frame system for the duration of one request, without -editing the saved configuration. You give the new frame a name, a parent frame -to attach to, and an offset pose relative to that parent. +To extend that frame system for +one motion, pass a +[`WorldState`](https://python.viam.dev/autoapi/viam/proto/common/index.html#viam.proto.common.WorldState) +to `Move` with extra frames. The steps differ slightly by language, but the shape is +the same: -Attaching a transform as a child of the end effector creates a new frame offset -from it. This is how you reposition the point the planner aims for: define a -frame at the tip of a held object or at an offset from the gripper, then target -that frame in the destination. +1. Define the translation and orientation from the parent frame to the origin of the + new frame. In Python, this is a `PoseInFrame`; in Go, it is the pose you pass to + `NewLinkInFrame`. The parent is usually an arm or gripper component name, such as + `"my-gripper"` for a grasped object. +2. Build a transform from that pose and add it to the `WorldState`. + +The `WorldState` applies to a single `Move` call. A later call needs its own +`WorldState` to keep using the transform. {{< tabs >}} {{% tab name="Python" %}} @@ -100,21 +128,22 @@ from viam.proto.common import ( Transform, PoseInFrame, Pose, WorldState ) -# A frame named "object-tip" 80mm beyond the gripper, attached to the gripper. +# A frame named "object-tip", 100 mm beyond the gripper, attached to it. object_tip = Transform( - reference_frame="object-tip", + reference_frame="object-tip", # the name of the new frame pose_in_observer_frame=PoseInFrame( - reference_frame="my-gripper", - pose=Pose(x=0, y=0, z=80, o_x=0, o_y=0, o_z=1, theta=0), + reference_frame="my-gripper", # the parent: the object moves with the gripper + # 70 mm across and 130 mm down from the gripper, turned so z points along the bent object. + pose=Pose(x=70, y=0, z=130, o_x=1, o_y=0, o_z=0, theta=90), ), ) world_state = WorldState(transforms=[object_tip]) -# Now Move can target "object-tip" as the frame to place at the destination. +# Move can now place "object-tip" at the destination. await motion_service.move( component_name="object-tip", - destination=destination, + destination=target_pose, world_state=world_state, ) ``` @@ -123,24 +152,28 @@ await motion_service.move( {{% tab name="Go" %}} ```go -// A frame named "object-tip" 80mm beyond the gripper, attached to the gripper. -objectTip := referenceframe.NewLinkInFrame( - "my-gripper", - spatialmath.NewPoseFromPoint(r3.Vector{X: 0, Y: 0, Z: 80}), - "object-tip", - nil, +// A frame named "object-tip", 100 mm beyond the gripper, attached to it. +objectTipLink := referenceframe.NewLinkInFrame( + "my-gripper", // the parent: the object moves with the gripper + // 70 mm across and 130 mm down from the gripper, turned so z points along the bent object. + spatialmath.NewPose( + r3.Vector{X: 70, Y: 0, Z: 130}, + &spatialmath.OrientationVectorDegrees{OX: 1, OY: 0, OZ: 0, Theta: 90}, + ), + "object-tip", // the name of the new frame + nil, // optional collision geometry; nil means a pure coordinate frame ) worldState, err := referenceframe.NewWorldState( - nil, []*referenceframe.LinkInFrame{objectTip}) + nil, []*referenceframe.LinkInFrame{objectTipLink}) if err != nil { logger.Fatal(err) } -// Now Move can target "object-tip" as the frame to place at the destination. +// Move can now place "object-tip" at the destination. _, err = motionService.Move(ctx, motion.MoveReq{ ComponentName: "object-tip", - Destination: destination, + Destination: targetPose, // your goal *referenceframe.PoseInFrame WorldState: worldState, }) ``` @@ -148,30 +181,95 @@ _, err = motionService.Move(ctx, motion.MoveReq{ {{% /tab %}} {{< /tabs >}} -Because the transform is a child of `my-gripper`, the new frame moves with the -gripper, and the planner places `object-tip` rather than the gripper's own -origin at the destination. When the request finishes, the frame disappears: the -saved frame system is unchanged. +### Attach a geometry to a transform + +A transform can also carry a collision geometry, a shape the planner avoids while it +tries to find collision-free motion to the destination. For a grasped object, it can be +helpful to attach geometry so the planner routes the grasped object around obstacles. +Give the transform a box, sphere, capsule, or mesh defined in the frame's own coordinates: + +{{}} + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import ( + Transform, PoseInFrame, Pose, Geometry, RectangularPrism, Vector3, +) + +object_tip = Transform( + reference_frame="object-tip", + pose_in_observer_frame=PoseInFrame( + reference_frame="my-gripper", + pose=Pose(x=70, y=0, z=130, o_x=1, o_y=0, o_z=0, theta=90), + ), + physical_object=Geometry( # the collision geometry for this frame + center=Pose(x=0, y=0, z=-50, o_x=0, o_y=0, o_z=1, theta=0), # centered on the grasped object + box=RectangularPrism(dims_mm=Vector3(x=30, y=30, z=100)), + label="object-tip", + ), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +// Make a new 30 x 30 x 100 mm box covering the grasped object. +geometry, err := spatialmath.NewBox( + spatialmath.NewPoseFromPoint(r3.Vector{X: 0, Y: 0, Z: -50}), // centered on the grasped object + r3.Vector{X: 30, Y: 30, Z: 100}, // full dimensions in mm + "object-tip", +) +if err != nil { + logger.Fatal(err) +} + +objectTipLink := referenceframe.NewLinkInFrame( + "my-gripper", + spatialmath.NewPose( + r3.Vector{X: 70, Y: 0, Z: 130}, + &spatialmath.OrientationVectorDegrees{OX: 1, OY: 0, OZ: 0, Theta: 90}, + ), + "object-tip", + geometry, // the planner avoids collisions with this geometry +) +``` + +{{% /tab %}} +{{< /tabs >}} ## WorldState versus the world state store service The names are similar, but `WorldState` and the world state store service are -different things, and only one affects a `Move`. +different things, and only one affects a `Move` call. - `WorldState` is the argument you pass to a single `Move` call. The obstacles and transforms in it exist only for that request: the planner uses them to plan this motion, and they are gone when the call returns. The transform in the previous section is a `WorldState` transform, so it shapes the plan. - The world state store service is a separate service that holds transforms for - client-side visualization in the [3D scene](/motion-planning/3d-scene/). The - motion planner reads only the `WorldState` you pass to `Move`, so a transform - published to the store renders in the scene while planning stays driven by - `WorldState` alone. + client-side visualization in the [3D scene](/motion-planning/3d-scene/). To change where the arm goes, put the transform in the `WorldState` you pass to -`Move`, as shown above. To render a custom visual without affecting planning, +`Move`, as shown above. To render a custom visual that leaves planning unchanged, publish it to the world state store service instead. +## Visualize the end effector frame + +Static frames render in the 3D scene automatically. Open the **3D SCENE** tab on +your machine's page, and the arm's built-in end effector frame and any `frame` +attributes you configured appear as coordinate axes at their computed poses, the +same axes drawn in the diagrams on this page. + +A frame you add in code with a `WorldState` transform lasts only for that `Move` +call, so it stays out of the 3D scene by default. To draw a code-defined frame in +the [3D scene](/motion-planning/3d-scene/), publish it through a world state store +service. A module that implements this service holds your transforms and streams +them to the scene, which renders a frame with no geometry as a set of coordinate axes +alongside the static frames. + ## What's next - [Frame system overview](/motion-planning/frame-system/overview/): From 84418de7121a7fb6479c8d26a1a2961804ff8a25 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 23 Jun 2026 11:46:25 -0600 Subject: [PATCH 06/46] Revise move-an-arm pages: pose clouds, waypoints, verify, constraints Apply house-style and accuracy fixes across the move-an-arm motion pages, and add diagrams for the concepts that read better visually. - Pose clouds: rework the example around a grasped cup with a target frame and an orientation constraint, add a units/default table and a PoseCloud snippet, and add two diagrams (the cup pose cloud and the tilted-surface target frame with the orientation-vector tolerance). - Multiple waypoints: answer the start-state and frame-system questions against source, and add a single-Move-vs-multi-goal contrast diagram. - Verify a plan and constraints: convert negations to positive statements, replace jargon, and fix a banned idiom in a heading. All code is verified against the RDK source (NewPoseInFrameWithGoalCloud, PoseCloud, OrientationConstraint, armplanning.PlanMotion, PlanRequest). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- .../move-an-arm/pose-cloud-cup.svg | 516 ++++++++++++++++++ .../move-an-arm/pose-cloud-target-frame.svg | 257 +++++++++ .../move-an-arm/waypoint-trajectory.svg | 373 +++++++++++++ .../move-an-arm/constraints.md | 8 +- .../move-an-arm/multiple-waypoints.md | 24 +- .../move-an-arm/pose-clouds.md | 157 +++--- docs/motion-planning/verify-a-plan.md | 10 +- 7 files changed, 1262 insertions(+), 83 deletions(-) create mode 100644 assets/motion-planning/move-an-arm/pose-cloud-cup.svg create mode 100644 assets/motion-planning/move-an-arm/pose-cloud-target-frame.svg create mode 100644 assets/motion-planning/move-an-arm/waypoint-trajectory.svg diff --git a/assets/motion-planning/move-an-arm/pose-cloud-cup.svg b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg new file mode 100644 index 0000000000..d9da65a855 --- /dev/null +++ b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg @@ -0,0 +1,516 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + pose cloud + validdestination + + + + Planner can find a solution anywhere in the pose cloud + + + + + + + + x + + z + + y + + + diff --git a/assets/motion-planning/move-an-arm/pose-cloud-target-frame.svg b/assets/motion-planning/move-an-arm/pose-cloud-target-frame.svg new file mode 100644 index 0000000000..687b9d6806 --- /dev/null +++ b/assets/motion-planning/move-an-arm/pose-cloud-target-frame.svg @@ -0,0 +1,257 @@ + + + + + + + + + + + + + + + + + + + + + Tolerances are read in the target's frame + + + + + + + + pose cloud + target frame + + + + + x + + y + + + + Pose CloudOrientation vector + + + + The pose cloud follows the tilted surface, and OX, OY, and Theta bound the orientation. + diff --git a/assets/motion-planning/move-an-arm/waypoint-trajectory.svg b/assets/motion-planning/move-an-arm/waypoint-trajectory.svg new file mode 100644 index 0000000000..58187b4b03 --- /dev/null +++ b/assets/motion-planning/move-an-arm/waypoint-trajectory.svg @@ -0,0 +1,373 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + target slot + + + + 2 + 1 + approach + + + single Move: any path, skips the approach + + multi-goal plan: through waypoint 1 & 2, then down + A single Move reaches the goal by any path. A multi-goal plan routes through the waypoints you require, in order. + diff --git a/docs/motion-planning/move-an-arm/constraints.md b/docs/motion-planning/move-an-arm/constraints.md index bc0a72af33..5e1f6dd85c 100644 --- a/docs/motion-planning/move-an-arm/constraints.md +++ b/docs/motion-planning/move-an-arm/constraints.md @@ -90,11 +90,11 @@ lets you whitelist specific pairs. | --------- | ------------------- | ---------------------------------------------------- | | `allows` | list of frame pairs | Each entry has `frame1` and `frame2` (string names). | -This is useful when: +Allow a collision when: - A gripper is expected to contact the object it is picking up. - Two components are physically close and their simplified collision geometries - overlap, but the real components do not collide. + overlap, while the real components stay clear of each other. Frame names support hierarchical matching: specifying `"my-arm"` matches all sub-geometries of the arm (such as `my-arm:upper_arm_link`, @@ -116,8 +116,8 @@ For `CollisionSpecification` (allow specific frame pairs to collide), see ## Relax the goal with a pose cloud Constraints restrict the _path_ between poses. The complementary tool is to -relax the _goal_ itself. If your task does not require an exact destination pose, -you can give the planner a region of acceptable poses instead of a single one, +relax the _goal_ itself. When close enough is acceptable, give the planner a +region of acceptable poses instead of a single one, which enlarges the solution set and makes planning faster and more reliable. This is a pose cloud, and it is often the right move when a goal is failing or planning slowly. See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/). diff --git a/docs/motion-planning/move-an-arm/multiple-waypoints.md b/docs/motion-planning/move-an-arm/multiple-waypoints.md index bd02b59f80..9731e50cb8 100644 --- a/docs/motion-planning/move-an-arm/multiple-waypoints.md +++ b/docs/motion-planning/move-an-arm/multiple-waypoints.md @@ -7,11 +7,11 @@ type: "docs" description: "Plan a single continuous trajectory through an ordered list of intermediate goals using armplanning.PlanMotion." --- -Sometimes the endpoint is not enough. You need the arm to pass through a -specific approach point before reaching a target, to thread between two pieces -of equipment in a set order, or to follow a sequence of stations in one motion. -A single `Move` to the final pose lets the planner choose any collision-free -path, which may skip the intermediate points you care about. +Sometimes you need the arm to pass through a specific approach point before +reaching a target, to thread between two pieces of equipment in a set order, +or to follow a sequence of stations in one motion. A single `Move` to the final +pose lets the planner choose any collision-free path, which may skip the intermediate +poses you care about. `armplanning.PlanMotion` accepts an ordered list of goals and plans one continuous trajectory that hits each in turn. This page shows how to build a @@ -28,7 +28,7 @@ calling the service, see [Verify a motion plan](/motion-planning/verify-a-plan/) ## Why route through ordered goals -Chaining separate `Move` calls is not the same as one multi-goal plan: +One multi-goal plan differs from chaining separate `Move` calls: - Each `Move` plans and executes independently, so the arm stops at every waypoint instead of flowing through it. @@ -42,6 +42,8 @@ planner hits each goal in the order you list it, and uses each waypoint's solved configuration as the start of the next segment, so the whole motion is continuous and consistent. +{{}} + ## Build a multi-goal PlanRequest `PlanRequest.Goals` is an ordered list of `PlanState` values. Each `PlanState` @@ -62,12 +64,15 @@ fsCfg, err := machine.FrameSystemConfig(ctx) if err != nil { logger.Fatal(err) } + +// Assemble the configured parts into a frame system the planner can use. fs, err := referenceframe.NewFrameSystem("robot", fsCfg.Parts, nil) if err != nil { logger.Fatal(err) } -// Start from a known arm configuration. +// The start state is required and cannot be nil. To plan from where the arm is +// now, read its current joints with arm.JointPositions and use those here. startState := armplanning.NewPlanState(nil, referenceframe.FrameSystemInputs{ "my-arm": []referenceframe.Input{0, 0, 0, 0, 0, 0}, }) @@ -108,9 +113,8 @@ if err != nil { Each goal names the frame it constrains, so a Cartesian waypoint can target the gripper frame while a configuration waypoint pins the arm's joints. Pose clouds -work as goals too, which lets you relax the waypoints that do not need an exact -pose. See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/) for when to -use them. +work as goals too, which lets you relax the waypoints where close enough is fine. +See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/) for when to use them. ## Read the trajectory diff --git a/docs/motion-planning/move-an-arm/pose-clouds.md b/docs/motion-planning/move-an-arm/pose-clouds.md index b54ba8f51c..b9e4b96b68 100644 --- a/docs/motion-planning/move-an-arm/pose-clouds.md +++ b/docs/motion-planning/move-an-arm/pose-clouds.md @@ -7,120 +7,149 @@ type: "docs" description: "Give the planner a region of acceptable destinations instead of a single exact pose, so it can find a solution faster and more reliably." --- -A single target pose tells the planner exactly where the end effector must end -up, down to the millimeter and the degree. For many tasks that is more precise -than you need. Dropping a part into a bin, touching a flat surface, or pointing a -tool roughly at a target all have slack in them, and demanding an exact pose -throws that slack away. An over-constrained goal makes the planner work harder, -fail more often, and sometimes reject a goal that a slightly looser target would -have reached. - -A _pose cloud_ replaces the exact goal with a region of acceptable poses. You +By default `Move` requires a destination pose, which defines where the end effector must +move to, and the planner holds the end effector to it tightly. For many tasks that is more +precise than needed. Dropping a part into a bin, sanding a flat surface, or pointing a +camera at an injection molding station all have different tolerance requirements for the destination pose, +and demanding tight tolerances on the target pose can lead to issues. An over-constrained goal makes the planner work harder, fail more often, and sometimes reject a goal that would have been valid for the task. + +A `pose cloud` replaces the exact goal with a region of acceptable poses. You give the planner a target pose plus a set of tolerances, and any pose within -those tolerances counts as having arrived. This page explains what a pose cloud -relaxes, why its tolerances are measured in the target's own frame, and how to -pass one to the motion service. +those tolerances is considered valid. This page explains what a pose cloud +relaxes and how to use a `pose cloud` with the motion service. + +## When to reach for a pose cloud -## How a pose cloud relaxes a goal +- **Target regions.** When the destination is a region rather than one exact pose, use a + pose cloud. For example, an object that may rest anywhere on a table, or a soup can that + may spin about its long axis. +- **Dynamic collisions.** When the target area may contain obstacles, a pose cloud lets + the planner place the object at a clear spot while avoiding collisions at the destination. +- **Kinematic constraints.** When a goal fails or plans slowly because the arm struggles + to reach that exact pose, relaxing the destination tolerance enlarges the solution space + and often turns a failing plan into a success. -A pose cloud defines a tolerance, or leeway, on each component of the goal pose. -Each leeway permits deviation in the range from its negative to its positive -value, so the goal becomes a box around the target pose rather than a single -point. +Pose clouds are powerful when coupled with motion constraints: together they describe the motion you want without pinning down the destination exactly. Moving a cup of water is a good example. You define an +orientation constraint to keep the cup upright, but you may not know which exact target +pose lets the arm hold that orientation across its workspace. A pose cloud lets the planner +choose a destination within a region instead: -| Field | Units | Relaxes | -| ---------------- | -------- | ----------------------------------------------------------------- | -| `X`, `Y`, `Z` | mm | Position along each axis | -| `OX`, `OY`, `OZ` | unitless | The orientation vector components (the direction the tool points) | -| `Theta` | degrees | Rotation about the orientation axis | +{{}} -A leeway of zero on a field holds that component to the exact target value; a -larger leeway gives the planner more room on that component. You relax only what +## Relaxing the destination pose + +A pose cloud defines a tolerance on each component of the goal pose. +Each permits +/- deviation, so the goal becomes a box around the target pose rather than a single +pose. The destination pose is the center of the box. + +```go +&referenceframe.PoseCloud{ + X: 50, // 50 mm of slack in x + Y: 50, // 50 mm of slack in y + Theta: 90, // up to 90 degrees of rotation about the orientation axis + // Z, OX, OY, and OZ are omitted, so they default to 0 and stay exact. +} +``` + +| Field | Units | Relaxes | Default | +| ---------------- | -------- | ----------------------------------------------------------------- | ------------------------- | +| `X`, `Y`, `Z` | mm | Position along each axis | `0`: position held exact | +| `OX`, `OY`, `OZ` | unitless | The orientation vector components (the direction the tool points) | `0`: direction held exact | +| `Theta` | degrees | Rotation about the orientation axis | `0`: rotation held exact | + +A tolerance of zero on a field holds that component to the exact target value; a +larger tolerance gives the planner more room on that component. You relax only what the task allows: a part dropped into a wide bin can take large `X` and `Y` -leeways while keeping `Z` tight, and a tool that may spin freely about its axis +tolerance while keeping `Z` tight, and a tool that may spin freely about its axis can take a large `Theta` while keeping its pointing direction fixed. +The position tolerances (`X`, `Y`, `Z`) are millimeters and `Theta` is degrees. The +orientation-vector tolerances (`OX`, `OY`, `OZ`) are unitless deviations of the pointing +direction on a unit sphere, not angles: a value of `1` accepts any value for that +component, and `0` holds it exact. + The more freedom you give the planner, the larger the set of arm configurations that satisfy the goal, so inverse kinematics is more likely to find a solution -and the search reaches it faster. A goal that fails as an exact pose often -succeeds as a pose cloud. +and the search reaches it faster. ## Tolerances are measured in the target's frame -The leeways are evaluated in the reference frame of the target, not in world +The tolerances are evaluated in the reference frame of the target, not in world coordinates. This matters whenever the target is tilted relative to the world. -Consider a gripper approaching an inclined surface. You want to allow slack -_along_ the surface but stay tight _into_ it, so the tool does not push through. -Expressed in world coordinates that is an awkward mix of all three axes, because -the surface is tilted. Expressed in the surface's own frame it is simple: large -leeway on the two in-plane axes, tight leeway on the axis normal to the surface. +Consider a gripper approaching an inclined surface. You want to allow tolerance +_along_ the surface but stay tight _into_ it, so the tool rides the surface +instead of driving through it. Expressed in world coordinates that is an awkward +mix of all three axes, because the surface is tilted. Expressed in the surface's +own frame, you give large tolerance on the two in-plane axes and tight tolerance on the +axis normal to the surface. Because pose clouds use the target's frame, you describe the tolerance the way the task is shaped rather than translating it into world axes by hand. +{{}} + ## Use a pose cloud with the motion service -In Go, attach a pose cloud to the destination with +A pose cloud attaches to the destination object. In Go, attach a pose cloud to the destination with `NewPoseInFrameWithGoalCloud`, then pass that destination to `Move`. The pose is -the center of the region and the `PoseCloud` is the leeway around it. +the center of the region and the `PoseCloud` is the tolerance around it. This example includes an orientation constraint for the water example, so the cup stays upright for the whole motion, not only at +the goal. ```go import ( - "go.viam.com/rdk/services/motion" + "github.com/golang/geo/r3" + "go.viam.com/rdk/motionplan" "go.viam.com/rdk/referenceframe" + "go.viam.com/rdk/services/motion" "go.viam.com/rdk/spatialmath" ) -// Point the tool at the cup, but allow slack in the approach orientation. +// Place the cup somewhere on the table. The exact spot does not matter, and the +// cup may end up rotated about vertical, but it must stay upright. destination := referenceframe.NewPoseInFrameWithGoalCloud( - "cup", - spatialmath.NewPoseFromOrientation( - &spatialmath.OrientationVectorDegrees{OX: 1, OZ: 0, Theta: 180}, + "table", // the target frame: tolerances follow the table surface + spatialmath.NewPose( + r3.Vector{X: 0, Y: 0, Z: 0}, // center of the region, on the table surface + &spatialmath.OrientationVectorDegrees{OZ: -1}, // upright: the tool's z-axis points into the table ), - &referenceframe.PoseCloud{OX: 1, OY: 1, OZ: 0.1, Theta: 10}, + // Tolerances are applied in the table frame, each as [-value, +value]. + &referenceframe.PoseCloud{ + X: 75, // mm of slack across the table surface in x + Y: 75, // mm of slack across the table surface in y + Z: 0, // hold the cup on the surface + Theta: 180, // degrees: any rotation about vertical; spinning the cup does not spill it + // OX, OY, OZ stay 0, so the upright pointing direction is held exact. + }, ) +// Keep the cup upright along the whole path, within 5 degrees, not just at the goal. +constraints := &motionplan.Constraints{ + OrientationConstraint: []motionplan.OrientationConstraint{ + {OrientationToleranceDegs: 5}, + }, +} + _, err = motionService.Move(ctx, motion.MoveReq{ ComponentName: "my-gripper", Destination: destination, + Constraints: constraints, }) if err != nil { logger.Fatal(err) } ``` -Here the destination is expressed in the `cup` frame, so the leeways apply -relative to the cup. The planner is free to satisfy the goal with any gripper -pose inside the cloud. - {{% alert title="Available through the Go SDK" color="note" %}} +Today the Go SDK provides the `NewPoseInFrameWithGoalCloud` constructor; The goal cloud travels on the destination's `PoseInFrame`, which is part of the `Move` request sent to the motion service, so the planner honors it server side. -The convenience constructor `NewPoseInFrameWithGoalCloud` is part of the Go SDK; -other SDKs do not yet provide an equivalent helper. {{% /alert %}} -## When to reach for a pose cloud - -- **Slack placement.** Dropping or setting down an object where any spot in a - region is acceptable. Relax the in-plane position and the spin about the - vertical. -- **Surface contact.** Touching, wiping, or sanding a surface where the contact - point can move along the surface but must stay on it. Relax the in-plane axes, - keep the normal tight. -- **Faster, more reliable planning.** Any goal that is failing or planning - slowly as an exact pose. Relaxing the components the task does not constrain - enlarges the solution set and often turns a failing plan into a quick success. - -If the task genuinely requires an exact pose, use a plain destination. A pose -cloud is for the common case where "close enough" is correct and exactness only -makes the planner's job harder. - ## What's next - [Configure motion constraints](/motion-planning/move-an-arm/constraints/): restrict the path between poses, the complement to relaxing the goal. - [Move through waypoints](/motion-planning/move-an-arm/multiple-waypoints/): - use pose clouds for the waypoints that do not need an exact pose. + use pose clouds for the waypoints where close enough is fine. - [How motion planning works](/motion-planning/how-planning-works/): why a larger solution set makes the search succeed more often. diff --git a/docs/motion-planning/verify-a-plan.md b/docs/motion-planning/verify-a-plan.md index b2b3507fe7..1da3581d2e 100644 --- a/docs/motion-planning/verify-a-plan.md +++ b/docs/motion-planning/verify-a-plan.md @@ -24,8 +24,8 @@ motion service API from a remote client. committing the arm to it. - **Check feasibility.** Confirm a goal is reachable and a collision-free path exists, without driving the arm into a failed attempt. -- **Validate from a hypothetical state.** Plan from a start configuration the - arm is not currently in, to test reachability before you move there. +- **Validate from a hypothetical state.** Plan from a start configuration other + than the arm's current one, to test reachability before you move there. ## PlanMotion compared to Move @@ -34,7 +34,7 @@ result and where they run: - `Move` is a motion service API call. It plans, then executes, moving the arm. - `PlanMotion` is an in-process Go function. It plans and returns the trajectory. - Nothing moves until you execute the result yourself. + Nothing moves until you run the result yourself. Because `PlanMotion` runs in process, you assemble the planning inputs directly rather than letting the service collect them from the robot. That is more setup, @@ -120,11 +120,11 @@ An error means no plan was found within the timeout, which is your feasibility answer. A returned trajectory is yours to inspect, log, compare against an expected path, or hand to the arm for execution once you are satisfied. -## Plan over the wire with the motion service +## Plan from a remote client `PlanMotion` requires running Go in process with the machine. A remote client that only has the motion service API can still get a plan without executing, by -sending the plan request through the service's `DoCommand` interface. The +sending the plan request through the service's `DoCommand` method. The built-in motion service handles a `"plan"` command that runs the same planner as `Move` and returns the trajectory without moving the arm. From a018a9772def0c4f661a00764018362b975be366 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 23 Jun 2026 12:05:22 -0600 Subject: [PATCH 07/46] Verify a plan: read current joints for the start state, fix prose Use arm.JointPositions to seed the start state from the arm's current position (with arm.FromRobot), and explain the extra/nil parameter. Correct the PlanRequest description (the world state is optional, not required), and fix grammar and code-comment typos. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- docs/motion-planning/verify-a-plan.md | 65 ++++++++++++++------------- 1 file changed, 35 insertions(+), 30 deletions(-) diff --git a/docs/motion-planning/verify-a-plan.md b/docs/motion-planning/verify-a-plan.md index 1da3581d2e..ded000a45b 100644 --- a/docs/motion-planning/verify-a-plan.md +++ b/docs/motion-planning/verify-a-plan.md @@ -9,13 +9,13 @@ description: "Use armplanning.PlanMotion to compute and inspect a trajectory bef Before an arm moves, you often want to know whether the motion is even possible: whether a goal is reachable, whether a path exists around the obstacles you have -declared, or what trajectory the planner would produce. Executing a `Move` to -find out is slow and, on real hardware, risky. Planning without executing -answers those questions first, then leaves it to you whether to run the result. +declared, or what trajectory the planner would produce. Running a `Move` to find out +makes the arm move, which is slow and, on real hardware, risky. Planning without +executing answers those questions first, then leaves it to you whether to run the result. `armplanning.PlanMotion` runs the same planner the motion service uses, but returns the `Plan` instead of moving the arm. This page shows how to compute and -inspect a plan in process, and closes with how to get the same plan over the +inspect a plan in process, and shows how to get the same plan over the motion service API from a remote client. ## When to plan without executing @@ -36,39 +36,51 @@ result and where they run: - `PlanMotion` is an in-process Go function. It plans and returns the trajectory. Nothing moves until you run the result yourself. -Because `PlanMotion` runs in process, you assemble the planning inputs directly -rather than letting the service collect them from the robot. That is more setup, -but it gives you the plan as data with no side effects. - ## Assemble a PlanRequest -A `PlanRequest` carries everything the planner needs: the frame system, the -world state, the start state, and one or more goals. Build the frame system from -the running machine, set the start state to the configuration you want to plan -from, and give the goal as a pose for a named frame. +To use `PlanMotion` you build a `PlanRequest`. It carries the frame system, the start +state, and one or more goals, plus optional obstacles as a world state. Build the +frame system from the running machine, set the start state to the configuration +you want to plan from, and give the goal as a pose for a named frame. ```go import ( + "github.com/golang/geo/r3" + "go.viam.com/rdk/components/arm" "go.viam.com/rdk/motionplan/armplanning" "go.viam.com/rdk/referenceframe" "go.viam.com/rdk/spatialmath" - "github.com/golang/geo/r3" ) -// Frame system from the running machine. +// Get the frame system config from the running machine. fsCfg, err := machine.FrameSystemConfig(ctx) if err != nil { logger.Fatal(err) } + +// Create a frame system from the config. fs, err := referenceframe.NewFrameSystem("robot", fsCfg.Parts, nil) if err != nil { logger.Fatal(err) } -// Plan from a known start configuration. Use the arm's current joints to -// verify a real move, or any configuration to test a hypothetical one. +// Read the arm's current joints so the plan starts from where the arm is now. +// JointPositions returns []referenceframe.Input, the same type a start +// configuration holds. The second argument, extra, passes optional +// driver-specific parameters to the component; nil means none. +myArm, err := arm.FromRobot(machine, "my-arm") +if err != nil { + logger.Fatal(err) +} +current, err := myArm.JointPositions(ctx, nil) +if err != nil { + logger.Fatal(err) +} + +// Plan from that start configuration. Supply any joint values instead to test +// a hypothetical start without moving the arm. startState := armplanning.NewPlanState(nil, referenceframe.FrameSystemInputs{ - "my-arm": []referenceframe.Input{0, 0, 0, 0, 0, 0}, + "my-arm": current, }) // Goal: place the gripper frame at a pose in the world frame. @@ -80,9 +92,9 @@ goal := armplanning.NewPlanState(referenceframe.FrameSystemPoses{ )), }, nil) -// Fail fast on an infeasible goal instead of grinding for the full default. +// Set a max timeout so the plan fails quickly if it can't find a solution. opts := armplanning.NewBasicPlannerOptions() -opts.Timeout = 15 +opts.Timeout = 15 // The default planner timeout is 300 seconds. plan, _, err := armplanning.PlanMotion(ctx, logger, &armplanning.PlanRequest{ FrameSystem: fs, @@ -92,10 +104,6 @@ plan, _, err := armplanning.PlanMotion(ctx, logger, &armplanning.PlanRequest{ }) ``` -The default planner timeout is 300 seconds, so an infeasible plan grinds for -five minutes before failing. When you are verifying feasibility, set a shorter -`Timeout` so an impossible goal fails quickly. - ## Read the plan If `PlanMotion` returns without an error, the goal is feasible from the start @@ -122,13 +130,13 @@ expected path, or hand to the arm for execution once you are satisfied. ## Plan from a remote client -`PlanMotion` requires running Go in process with the machine. A remote client +The examples above use `PlanMotion` through the armplanning Go library. A remote client that only has the motion service API can still get a plan without executing, by sending the plan request through the service's `DoCommand` method. The built-in motion service handles a `"plan"` command that runs the same planner as `Move` and returns the trajectory without moving the arm. -The mechanism is: build a `MoveRequest`, serialize it, and send it under the key +This is done by building a `MoveRequest`, serializing it, and sending it under the key `"plan"`. ```go @@ -162,16 +170,13 @@ The trade-offs versus the in-process call: - The response is untyped. The trajectory comes back as generic map data, and its exact shape depends on the transport, so you parse it by hand rather than receiving a typed `Plan`. -- You drive it with string keys instead of typed parameters, which is easier to - get wrong. +- You drive it with string keys instead of typed parameters, which can be tricky to get right. - It works for any remote client of the motion service, with no need to assemble the frame system or pull the planner into your process. Use the in-process `PlanMotion` when you have Go access to the machine and want the plan as typed data. Use the `"plan"` `DoCommand` when you are a remote client -and the motion service is all you have. The -[viamkit](https://github.com/viam-labs/viamkit) library wraps the `DoCommand` -form and smooths over the response parsing. +and the motion service is all you have. ## What's next From 5f2830cd8319335d8df4a2e13bacd650b26bd5d1 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 23 Jun 2026 15:59:46 -0600 Subject: [PATCH 08/46] Visualization: positive prose, overview hub, code accuracy Convert negation-heavy framing to noun-plus-job across the visualization pages (a visual and its collision geometry each get a positive role), and fix soft negatives vale does not flag (invisible, nothing, cannot). Move the section index into an overview page (thin _index + manualLink, matching the frame-system convention) and restructure it as a hub with a short section per visualization approach: the 3D scene, Viam Visualization, component data in apps, and time-series dashboards. Fix code against source: api.DrawGeometry takes a DrawGeometryOptions struct, and add the draw service's add/update/remove fan-out to the browser. Improve the frame system description to lead concrete. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- .../3d-scene/debug-motion-plan.md | 20 ++--- docs/visualization/_index.md | 86 +------------------ docs/visualization/drawing-library.md | 25 +++++- docs/visualization/overview.md | 57 ++++++++++++ .../publish-visuals-from-a-module.md | 25 +++--- docs/visualization/visuals-and-collisions.md | 38 ++++---- 6 files changed, 118 insertions(+), 133 deletions(-) create mode 100644 docs/visualization/overview.md diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index 0017c38efc..e0b49d79ff 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -7,12 +7,11 @@ type: "docs" description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." --- -The **3D SCENE** tab does not show motion plans on its own. It is a static -inspector of the configured frame system and live component poses: it has no -timeline, no scrubber, and no plan playback. To see a plan, the trajectory the -arm will follow, the goals it aims for, and how that path relates to your -obstacles, you publish the plan as **custom visuals** through a world state store -service. The scene then renders the path you can otherwise only read as numbers. +The **3D SCENE** tab is a static inspector: it shows the configured frame system +and live component poses. To see a motion plan, the trajectory the arm will follow, +the goals it aims for, and how that path relates to your obstacles, you publish the +plan as **custom visuals** through a world state store service. The scene then renders +the path you can otherwise only read as numbers. This page shows how to turn a plan into transforms the scene can draw, and how to use the rendered path to debug a plan that failed or moved unexpectedly. @@ -81,7 +80,7 @@ from the trajectory leading to it. Serve the transforms through a world state store service so the **3D SCENE** tab renders them. The plan markers stream in alongside the frames and obstacle -geometry the scene already shows. For the service interface, the poll-and-update +geometry the scene already shows. For the service methods, the poll-and-update loop, and how a module pulls data from its dependencies, see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/). @@ -94,11 +93,10 @@ the rest of the scene: geometry, that is where the planner reports a collision. Check whether the obstacle is real or an oversized geometry. - **Does the goal fall outside the arm's reach?** If a goal marker sits far from - any reachable arm configuration, the planner cannot get there. Move the goal or + any reachable arm configuration, the goal is out of reach. Move the goal or check the frame system. - **Why the detour?** An unexpected route usually means an obstacle is forcing - the planner around it. Look for geometry between the start and goal you did not - intend to add. + the planner around it. Look for stray geometry between the start and goal. For checking the obstacle geometry itself, separate from the plan, see [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). @@ -112,7 +110,7 @@ The 3D scene serves three distinct purposes, and it helps to keep them straight: - **Inspect static frames and geometry**: use the stock scene to check frame positions and obstacle coverage with no plan involved. - **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is - reachable and a path exists before you visualize or execute anything. + reachable and a path exists before you visualize or run anything. Visualization shows you _what the path looks like_; static inspection shows you _what the world looks like_; feasibility checking tells you _whether a plan diff --git a/docs/visualization/_index.md b/docs/visualization/_index.md index 247a330c02..a759cbfca4 100644 --- a/docs/visualization/_index.md +++ b/docs/visualization/_index.md @@ -5,90 +5,6 @@ weight: 165 layout: "docs" type: "docs" no_list: true +manualLink: "/visualization/overview/" description: "See your machine in 3D: frames, geometries, point clouds, and custom visuals your modules publish." --- - -Spatial configuration is invisible in JSON. A frame translation of -`{x: 50, y: 0, z: 110}` tells you nothing about whether the gripper actually -sits where the arm needs it, and a list of obstacle geometries tells you nothing -about whether they cover the real workspace. Visualization renders these in 3D -so you can see the spatial model your machine is actually working with, catch -misconfigurations before they cause a failure, and watch live data in context. - -This section covers the **3D SCENE** tab in the Viam app, the transforms and -metadata that make up a custom visual, how a module publishes visuals, and the -standalone Viam Visualization app for previewing spatial data outside the app. - -## What the 3D scene renders - -The **3D SCENE** tab on your machine's page renders an interactive 3D view of -your machine. It draws four kinds of content, each from a different source: - -- **Component frames**, from the [frame system](#the-frame-system). Each - configured component appears as a set of coordinate axes at its computed - position. -- **Geometries**, from the components' `frame.geometry` configuration and from - obstacles. These render as translucent shapes the motion planner uses for - collision checking. -- **Point clouds**, from depth cameras, rendered as colored point sets. -- **Custom visuals**, from a [world state store service](/visualization/publish-visuals-from-a-module/). - A module publishes these at runtime, and the scene streams them in as they - change. - -The scene reads your saved configuration, and when the machine is online it -connects for live data, so frames move to their current poses and point clouds -update as the cameras report them. - -## The frame system - -Everything in the scene is positioned by the **frame system**: the single -coordinate tree that records where every component sits relative to every other. - -Each component has a frame with a parent, a translation, and an orientation. -A component's frame is defined relative to its parent's frame, not to the world -directly, so the frames form a tree rooted at the fixed **world frame**. A -gripper's frame is a child of the arm's, the arm's is a child of the world, and -so on. Because the tree is connected, the frame system can compute the transform -between any two frames: it walks the path between them, composing each frame's -local transform along the way. When the arm's joints move, every frame below it -in the tree moves with it automatically. - -This is what makes the scene meaningful. A point cloud from a wrist camera and an -obstacle defined in world coordinates can be drawn in the same view because the -frame system relates both back to the world frame. The same machinery lets the -motion planner reason about where the gripper is while the arm moves, and lets a -custom visual attach itself to a moving component. - -For configuring frames in detail (parent, translation, orientation, geometry), -see the [frame system documentation](/motion-planning/frame-system/). - -## Built-in content versus custom visuals - -Two kinds of content reach the scene by two different routes, and the difference -determines where you change them: - -- **Built-in content** (component frames and configured geometry) comes from - your machine **configuration**. You change it by editing the frame system and - obstacle config. The scene reads it directly. -- **Custom visuals** come from a **module** that implements a world state store - service and publishes transforms at runtime. You change them in **code**, not - config, and the scene streams them in as the module adds, updates, and removes - them. - -If a frame or geometry looks wrong, fix the configuration. If a custom visual -looks wrong, the module that publishes it is where to look. - -## Topics - -{{< cards >}} -{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} -{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} -{{% card link="/visualization/drawing-library/" noimage="true" %}} -{{< /cards >}} - -## Use the 3D scene with motion planning - -{{< cards >}} -{{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} -{{< /cards >}} diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index 0cbbb14f5e..57a58f92db 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -77,10 +77,17 @@ then push visuals to it from Go with the client API. Reusing an entity ID update that visual in place; a new ID adds another: ```go -import "github.com/viam-labs/motion-tools/client/api" - -// Update in place by reusing the ID; add a new entity with a different ID. -err := api.DrawGeometry(box, api.WithID("obstacle-1"), api.WithColor(red)) +import ( + "github.com/viam-labs/motion-tools/client/api" + "github.com/viam-labs/motion-tools/draw" +) + +// Reuse an ID to update that visual in place; change or omit it to add another. +_, err := api.DrawGeometry(api.DrawGeometryOptions{ + ID: "obstacle-1", + Geometry: box, + Color: draw.NewColor(draw.WithName("red")), +}) ``` This lets you preview spatial data, a point cloud, a set of detections, a planned @@ -88,6 +95,16 @@ path, straight from a script or test, without deploying a module or connecting through the Viam app. For setup, the local server, and the full client API, see the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). +## How updates reach the browser + +The Viam Visualization app runs a **draw service** that the client API calls. The service +exposes `AddEntity`, `UpdateEntity`, and `RemoveEntity` for the changes you push, and a +`StreamEntity` stream the browser subscribes to. When you draw, update, or remove an +entity, the service fans that single change out over `StreamEntity` to the browser, which +applies it incrementally instead of re-rendering the whole scene. This is the same add, +update, and remove model the world state store service uses to feed the in-app 3D scene, +so a busy scene stays in sync as your data changes. + ## What's next - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md new file mode 100644 index 0000000000..2a90c3c78e --- /dev/null +++ b/docs/visualization/overview.md @@ -0,0 +1,57 @@ +--- +linkTitle: "Overview" +title: "Visualization" +weight: 1 +layout: "docs" +type: "docs" +description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam Visualization app, component data in apps, and time-series dashboards." +--- + +There are several approaches to visualizing information in Viam. For things like frames, +robot state, motion, collisions, and perception data, you can use the 3D scene view or an +offline visualizer called Viam Visualization. This data can be retrieved from components +and used in Viam apps or custom user applications. For time series data, services and +modules can push data to the cloud and visualize it with Viam's Teleop workspaces and +dashboards to get live information across a machine or fleet. + +## The 3D scene + +The **3D SCENE** tab on your machine's page renders an interactive 3D view of your +machine: component frames from the frame system, configured geometries, depth-camera +point clouds, and custom visuals a module publishes at runtime. Use it to check spatial +configuration and watch live data in context. + +{{< cards >}} +{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} +{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} +{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} +{{< /cards >}} + +## Viam Visualization + +Viam Visualization is a standalone 3D app you run yourself to preview and debug spatial +data from a Go client, without deploying a module or opening the Viam app. It shares the +same `draw` library as the in-app 3D scene, so the visuals you build work either way. + +{{< cards >}} +{{% card link="/visualization/drawing-library/" noimage="true" %}} +{{< /cards >}} + +## Component data in your applications + +Components report spatial data, geometries and poses, through their APIs. Read it with an +SDK and render or process it yourself in a Viam app or a custom application. + +{{< cards >}} +{{% card link="/build-apps/" noimage="true" %}} +{{< /cards >}} + +## Time series data + +Services and modules push readings to the cloud, where you watch them live with Viam's +Teleop workspaces and dashboards across a single machine or a whole fleet. + +{{< cards >}} +{{% card link="/monitor/teleop-workspaces/" noimage="true" %}} +{{% card link="/monitor/dashboards/overview/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 5630200fe5..8945069fe0 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -13,7 +13,7 @@ shapes, you implement a **world state store service**. A module that implements this service is a producer: it reads data from other resources, turns that data into transforms, and streams them to the scene as they change. -This page covers when to implement the service, the interface to implement, how +This page covers when to implement the service, the methods to implement, how to build transforms with the `draw` library, and the poll-and-update loop that keeps the scene in sync. The pattern throughout is **pull**: the module depends on the resources it visualizes and reads their data, rather than having those @@ -21,14 +21,13 @@ resources push into it. ## When to implement a world state store service -Implement one when you want custom visuals in the 3D scene that the default -content cannot show. The scene already draws component frames and configured -geometry, so you do not need a module to see those. You do need one to visualize -anything computed or sensed at runtime: a vision service's detections, a sensor's -obstacle readings, a motion plan's trajectory, or any annotation specific to your -application. +Implement one when you want custom visuals in the 3D scene beyond the default +content. The scene already draws component frames and configured geometry on its +own. A module adds anything computed or sensed at runtime: a vision service's +detections, a sensor's obstacle readings, a motion plan's trajectory, or any +annotation specific to your application. -## Implement the service interface +## Implement the service methods The world state store service exposes three read methods, which the 3D scene calls to discover and follow your visuals: @@ -83,7 +82,7 @@ func buildTransform(o obstacle) (*commonpb.Transform, error) { ``` The library produces standard `commonpb.Transform` values, the same type the -service interface returns, so the transforms you build this way flow straight +service methods return, so the transforms you build this way flow straight through `ListUUIDs`, `GetTransform`, and `StreamTransformChanges` to the scene. ## Drive a poll-and-update loop @@ -135,10 +134,10 @@ reads everything else. Data flows one way: This is why the loop above calls `s.sensor.Readings(...)`: the sensor is a dependency, and the module pulls from it. -## Visualize a resource that is not a world state store +## Visualize any other resource -A module whose primary job is something else, an arm, a sensor, a planner, does -not push visuals into the scene. Instead, you write a world state store module +A module whose primary job is something else (an arm, a sensor, a planner) stays +focused on that job. To visualize it, you write a separate world state store module that takes that resource as a dependency and pulls from its existing API: - a sensor's `Readings` @@ -163,7 +162,7 @@ func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstate ## What's next - [Visuals and collisions](/visualization/visuals-and-collisions/): - what a transform contains and why a visual is not an obstacle. + what a transform contains, and which geometry the planner collision-checks. - [The drawing library and Viam visualization](/visualization/drawing-library/): the `draw` primitives and the standalone visualizer app. - [Frame system](/motion-planning/frame-system/): position the transforms you diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index a1e74cb333..7cfb44ace1 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -4,15 +4,15 @@ title: "Visuals and collisions" weight: 10 layout: "docs" type: "docs" -description: "How a Transform defines a custom visual, and why a visual in the scene is not the same as geometry the motion planner avoids." +description: "How a Transform defines a custom visual, and which geometry the motion planner actually collision-checks." --- A custom visual in the 3D scene is a `Transform`: a piece of geometry placed somewhere in the frame system, with styling that controls how it draws. The world state store service streams these transforms to the scene. This page -covers what a transform contains, how the scene tracks it over time, and the -distinction that trips people up most: a visual you can see is not automatically -an obstacle the motion planner avoids. +covers what a transform contains, how the scene tracks it over time, and the point +that trips people up most: the scene draws your visual, while the planner avoids a +separate collision geometry. ## Anatomy of a transform @@ -63,27 +63,25 @@ geometry: - `collision_allowed`: a hint about whether the geometry represents an allowed collision -These are **visualization attributes**, not planning inputs. They change how a -visual looks in the scene. They do not change what the motion planner does, -including `collision_allowed`: setting it affects how the visual is presented, -not whether the planner treats anything as solid. +These are **visualization attributes**: the scene reads them when it draws the +geometry. They control how the visual looks, including `collision_allowed`, which is +a rendering hint about the visual. The planner reads its solid geometry from the frame +system and `WorldState` instead, so these attributes shape the picture while the +planner's obstacles come from elsewhere. -## A visual is not an obstacle +## The scene draws, the planner collision-checks -This is the distinction to internalize: the geometry on a world state store -transform renders in the 3D scene, but the motion planner does not read the -world state store. Publishing a box to the scene draws a box. It does not add an -obstacle the arm will avoid. - -The geometry the planner actually collision-checks comes from two places: +The geometry on a world state store transform renders in the 3D scene: publishing a +box draws a box. The motion planner collision-checks a separate geometry, which it +reads from two places: - The **frame system**: each component's `frame.geometry`. - The **`WorldState`** you pass to a `Move` call: obstacles and transforms supplied for that single planning request. -A transform in the world state store and an obstacle in a `WorldState` are -therefore different things on different paths, even when they describe the same -shape. +A world state store transform and a `WorldState` obstacle travel two paths, each with +its own job: one is drawn in the scene, the other is planned around. The same shape can +take both paths. ## Making a geometry both visible and collision-checked @@ -95,8 +93,8 @@ you do both, separately: - **For planning**: add it to the frame system, or include it in the `WorldState` you pass to `Move`. -There is no single field today that does both. Treat the visual and the -collision geometry as two outputs you produce from the same source data. +Today these are two separate outputs you produce from the same source data: one +transform for the scene, one geometry for the planner. ## What's next From 6b20f3a407bccdb3fe8dd01fdb400b39a124f3b1 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 24 Jun 2026 09:04:19 -0600 Subject: [PATCH 09/46] Visualization: redirect the section index to the overview Make /visualization/ forward to /visualization/overview/ with the empty-node layout and a canonical link, matching how other section indexes (reference/apis/services, tutorials/control) redirect. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- docs/visualization/_index.md | 6 +++--- docs/visualization/overview.md | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/visualization/_index.md b/docs/visualization/_index.md index a759cbfca4..6e4b0a5b88 100644 --- a/docs/visualization/_index.md +++ b/docs/visualization/_index.md @@ -2,9 +2,9 @@ linkTitle: "Visualization" title: "Visualization" weight: 165 -layout: "docs" +empty_node: true +layout: "empty" +canonical: "/visualization/overview/" type: "docs" -no_list: true -manualLink: "/visualization/overview/" description: "See your machine in 3D: frames, geometries, point clouds, and custom visuals your modules publish." --- diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index 2a90c3c78e..cc844ad514 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -7,12 +7,12 @@ type: "docs" description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam Visualization app, component data in apps, and time-series dashboards." --- -There are several approaches to visualizing information in Viam. For things like frames, -robot state, motion, collisions, and perception data, you can use the 3D scene view or an -offline visualizer called Viam Visualization. This data can be retrieved from components -and used in Viam apps or custom user applications. For time series data, services and -modules can push data to the cloud and visualize it with Viam's Teleop workspaces and -dashboards to get live information across a machine or fleet. +There are several approaches to visualizing information in Viam. For things like motion, +visualizing frames, checking the robot state, reviewing collisions, and viewing live perception +data, you can use the 3D scene view or an offline visualizer called Viam Visualization. This +data can be retrieved from components and used in Viam apps or custom user applications. +For time series data, services and modules can push data to the cloud to be visualized with +Viam's Teleop workspaces and dashboards to get live information across a machine or fleet. ## The 3D scene From 0d5f35602fe99a75dcb298f6bf1623aca1fa4bac Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 24 Jun 2026 10:40:05 -0600 Subject: [PATCH 10/46] Restructure visualization overview and add geometry code examples - Overview: add per-approach when/why bullets, name the Viam web app and the apps you build with an SDK, and link the build-apps tutorials. - Redirect /build-apps/ to its overview page (empty_node + canonical). - Visuals and collisions: add JSON/Python/Go examples for each geometry type, building the Geometry proto directly (no helper library), per the example-visualizations-go and example-visualizations-python modules. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- docs/build-apps/_index.md | 6 +- docs/visualization/overview.md | 53 ++-- docs/visualization/visuals-and-collisions.md | 239 ++++++++++++++++++- 3 files changed, 269 insertions(+), 29 deletions(-) diff --git a/docs/build-apps/_index.md b/docs/build-apps/_index.md index 6599d00c95..f25dc1165b 100644 --- a/docs/build-apps/_index.md +++ b/docs/build-apps/_index.md @@ -2,10 +2,10 @@ linkTitle: "Build apps" title: "Build apps" weight: 65 -layout: "docs" +empty_node: true +layout: "empty" +canonical: "/build-apps/overview/" type: "docs" -no_list: true -manualLink: "/build-apps/overview/" description: "Build client apps that talk to your Viam machines and the Viam cloud." date: "2026-04-10" aliases: diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index cc844ad514..53bc1dbd65 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -4,22 +4,27 @@ title: "Visualization" weight: 1 layout: "docs" type: "docs" -description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam Visualization app, component data in apps, and time-series dashboards." +description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam Visualization app, your own apps, and time-series dashboards." --- There are several approaches to visualizing information in Viam. For things like motion, -visualizing frames, checking the robot state, reviewing collisions, and viewing live perception -data, you can use the 3D scene view or an offline visualizer called Viam Visualization. This -data can be retrieved from components and used in Viam apps or custom user applications. -For time series data, services and modules can push data to the cloud to be visualized with -Viam's Teleop workspaces and dashboards to get live information across a machine or fleet. +frames, robot state, collisions, and live perception data, you can use the 3D scene view in +the Viam web app or the standalone Viam Visualization app. You can also read this data from +components with a Viam SDK and present it in an app you build, which runs in a browser, on a +phone, or on a server. For time series data, services and modules push readings to the +cloud, where the Viam web app's Teleop workspaces and dashboards show live information across +a machine or fleet. ## The 3D scene The **3D SCENE** tab on your machine's page renders an interactive 3D view of your machine: component frames from the frame system, configured geometries, depth-camera -point clouds, and custom visuals a module publishes at runtime. Use it to check spatial -configuration and watch live data in context. +point clouds, and custom visuals a module publishes at runtime. + +- Use it while configuring or debugging a machine, to see frames, geometry, and live poses + with no code. +- Best for catching frame and obstacle misconfigurations and inspecting a motion plan in + context. It runs right in the Viam app, ready to use. {{< cards >}} {{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} @@ -30,26 +35,44 @@ configuration and watch live data in context. ## Viam Visualization Viam Visualization is a standalone 3D app you run yourself to preview and debug spatial -data from a Go client, without deploying a module or opening the Viam app. It shares the -same `draw` library as the in-app 3D scene, so the visuals you build work either way. +data from a Go client. It shares the same `draw` library as the in-app 3D scene, so the +visuals you build work either way. + +- Use it while developing, to preview spatial data such as a point cloud, detections, or a + planned path straight from a script or test. +- Best when you want to iterate from Go quickly, without deploying a module or opening the + Viam app. {{< cards >}} {{% card link="/visualization/drawing-library/" noimage="true" %}} {{< /cards >}} -## Component data in your applications +## Viam apps -Components report spatial data, geometries and poses, through their APIs. Read it with an -SDK and render or process it yourself in a Viam app or a custom application. +A Viam app uses a Viam SDK to read your machine's data and present it however you design. +It runs outside the machine, in a browser, on a phone, or on a server, so you can build a +custom dashboard, an operator console, or a fleet view. + +- Use it when operators or stakeholders need a tailored UI beyond the built-in scene. +- Best when you want to choose exactly which data to show and how, for one machine or a + whole fleet. {{< cards >}} -{{% card link="/build-apps/" noimage="true" %}} +{{% card link="/build-apps/overview/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-dashboard/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-fleet/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-flutter-app/" noimage="true" %}} +{{% card link="/build-apps/app-tutorials/tutorial-monitoring-service/" noimage="true" %}} {{< /cards >}} ## Time series data Services and modules push readings to the cloud, where you watch them live with Viam's -Teleop workspaces and dashboards across a single machine or a whole fleet. +Teleop workspaces and dashboards. + +- Use it for values that change over time: temperatures, speeds, counts, and sensor + readings. +- Best for live monitoring and historical trends across a single machine or a whole fleet. {{< cards >}} {{% card link="/monitor/teleop-workspaces/" noimage="true" %}} diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index 7cfb44ace1..24b7176af0 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -7,20 +7,21 @@ type: "docs" description: "How a Transform defines a custom visual, and which geometry the motion planner actually collision-checks." --- -A custom visual in the 3D scene is a `Transform`: a piece of geometry placed -somewhere in the frame system, with styling that controls how it draws. The -world state store service streams these transforms to the scene. This page -covers what a transform contains, how the scene tracks it over time, and the point -that trips people up most: the scene draws your visual, while the planner avoids a -separate collision geometry. +Viam uses several standard types to define static and dynamic geometries, and +the same types are used to create visuals or to tell the motion planner about +obstacles. + +To create a custom visual in the 3D scene, you must create a geometry and +attach it to `Transform.Geometry` then provide that transform to a World State Store Service. +The world state store service streams these transforms to the 3d scene. This page +covers what a transform contains, how the scene tracks it over time, and discusses +the difference between planner geometry and visualization geometry. ## Anatomy of a transform -A `Transform` carries four things that together place and style one visual: +A `Transform` carries four things that together place and style a visual: -- **Reference frame and pose**: where the visual sits. The pose is given in a - parent reference frame, so the visual is positioned in the frame system and - moves with its parent. +- **Reference frame and pose**: This defines the visual's origin. - **Geometry**: the shape to draw (a box, sphere, capsule, mesh, or point cloud). - **Metadata**: styling such as color and opacity. @@ -40,7 +41,7 @@ incremental add, update, and remove operations to individual visuals. ## Geometry types -A transform's geometry is the shape the scene draws. The supported types are: +The supported types are: - **box**: dimensions in millimeters - **sphere**: a radius @@ -52,6 +53,222 @@ Choose the type that matches what you are representing: a box or capsule to approximate a physical object, a mesh for a precise model, a point cloud for sensor data. +You build a geometry as a `Geometry` proto, the same type a world state store +transform and a `WorldState` obstacle both carry. The Python SDK and the Go SDK +construct that proto directly, with no helper library. The box, sphere, and +capsule primitives also have a machine config (JSON) form. The transform you +attach the geometry to supplies its reference frame and pose. + +### Box + +A box takes its `x`, `y`, and `z` dimensions in millimeters. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "box", "x": 100, "y": 100, "z": 100 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Geometry, RectangularPrism, Vector3 + +box = Geometry( + label="box", + box=RectangularPrism(dims_mm=Vector3(x=100, y=100, z=100)), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +box := &commonpb.Geometry{ + Label: "box", + GeometryType: &commonpb.Geometry_Box{ + Box: &commonpb.RectangularPrism{ + DimsMm: &commonpb.Vector3{X: 100, Y: 100, Z: 100}, + }, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Sphere + +A sphere takes a radius `radius_mm` in millimeters. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "sphere", "r": 50 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Geometry, Sphere + +sphere = Geometry(label="sphere", sphere=Sphere(radius_mm=50)) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +sphere := &commonpb.Geometry{ + Label: "sphere", + GeometryType: &commonpb.Geometry_Sphere{ + Sphere: &commonpb.Sphere{RadiusMm: 50}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Capsule + +A capsule takes a radius and a length in millimeters. The length must be at +least twice the radius. + +{{< tabs >}} +{{% tab name="JSON" %}} + +```json +{ "type": "capsule", "r": 50, "l": 200 } +``` + +{{% /tab %}} +{{% tab name="Python" %}} + +```python +from viam.proto.common import Capsule, Geometry + +capsule = Geometry( + label="capsule", + capsule=Capsule(radius_mm=50, length_mm=200), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import commonpb "go.viam.com/api/common/v1" + +capsule := &commonpb.Geometry{ + Label: "capsule", + GeometryType: &commonpb.Geometry_Capsule{ + Capsule: &commonpb.Capsule{RadiusMm: 50, LengthMm: 200}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Mesh + +A mesh comes from an STL or PLY file. Read the file and embed its bytes in the +geometry with a `content_type`. The renderer draws PLY, so convert an STL file +to PLY first. + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from pathlib import Path + +from viam.proto.common import Geometry, Mesh + +mesh = Geometry( + label="mesh", + mesh=Mesh(content_type="ply", mesh=Path("model.ply").read_bytes()), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import ( + "os" + + commonpb "go.viam.com/api/common/v1" +) + +plyBytes, err := os.ReadFile("model.ply") +if err != nil { + return err +} +mesh := &commonpb.Geometry{ + Label: "mesh", + GeometryType: &commonpb.Geometry_Mesh{ + Mesh: &commonpb.Mesh{ContentType: "ply", Mesh: plyBytes}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + +### Point cloud + +A point cloud is sensor output, so you read it as PCD bytes in binary PCD format +and embed them in the geometry. Add a color per point in the PCD data itself. +A point cloud has no machine config form, so you build it in code. + +{{< tabs >}} +{{% tab name="Python" %}} + +```python +from pathlib import Path + +from viam.proto.common import Geometry, PointCloud + +point_cloud = Geometry( + label="point-cloud", + pointcloud=PointCloud(point_cloud=Path("cloud.pcd").read_bytes()), +) +``` + +{{% /tab %}} +{{% tab name="Go" %}} + +```go +import ( + "os" + + commonpb "go.viam.com/api/common/v1" +) + +pcdBytes, err := os.ReadFile("cloud.pcd") +if err != nil { + return err +} +pointCloud := &commonpb.Geometry{ + Label: "point-cloud", + GeometryType: &commonpb.Geometry_Pointcloud{ + Pointcloud: &commonpb.PointCloud{PointCloud: pcdBytes}, + }, +} +``` + +{{% /tab %}} +{{< /tabs >}} + ## Metadata styles the visual The metadata is a set of rendering attributes the scene reads when it draws the From 45d586afe7945949d6061e6c24e7c657ad392cdf Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 24 Jun 2026 13:41:45 -0600 Subject: [PATCH 11/46] Fix frame-system SVG triads per Dan's review - Flip arm/tool/object frames to Z-down (DH convention) while keeping the world frame Z-up, per Dan's orientation-axis comment. - Use a right-handed Z-down triad with green Y on the world frame's lower-left side (X-left, Y-down-left), so all green axes sit on one side; give Y a longer line so it reads clearly. - Label one triad per image (the rest are color-coded), draw red X over green Y, and remove the per-axis labels that collided. - Trim the end-effector-types caption to drop the ambiguous "world frame at the base" clause; widen the canvas so "drill-tip frame" is not clipped. - Delete the orphaned drill-bit-frame.svg (superseded by arm-vs-gripper-frame). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- .../frame-system/arm-vs-gripper-frame.svg | 16 ++-- .../frame-system/drill-bit-frame.svg | 43 ----------- .../frame-system/end-effector-types.svg | 74 +++++++------------ .../frame-system/held-object-frame.svg | 70 +++++++----------- 4 files changed, 62 insertions(+), 141 deletions(-) delete mode 100644 assets/motion-planning/frame-system/drill-bit-frame.svg diff --git a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg index e8d42024ed..7aa40ef97e 100644 --- a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg +++ b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg @@ -5,12 +5,9 @@ - - x - - z - - y + + + @@ -48,9 +45,12 @@ target_pose - + - arm frame + x + y + z + arm frame drill-tip frame diff --git a/assets/motion-planning/frame-system/drill-bit-frame.svg b/assets/motion-planning/frame-system/drill-bit-frame.svg deleted file mode 100644 index c5e33dc086..0000000000 --- a/assets/motion-planning/frame-system/drill-bit-frame.svg +++ /dev/null @@ -1,43 +0,0 @@ - - - - - - - - x - - z - - y - - - - - Adding a drill-tip frame to a cobot - - - - - - - - - - - - - - - - 80 mm - - - - my-arm frame - - drill-tip frame - - The drill-tip frame sits 80 mm along z from the arm's frame. A WorldState - transform adds it for one Move call, so the planner places the bit on target. - diff --git a/assets/motion-planning/frame-system/end-effector-types.svg b/assets/motion-planning/frame-system/end-effector-types.svg index befeb06ed0..bdc626af01 100644 --- a/assets/motion-planning/frame-system/end-effector-types.svg +++ b/assets/motion-planning/frame-system/end-effector-types.svg @@ -1,8 +1,8 @@ - x + marker-end="url(#aG)" + id="line5" /> - z - y + marker-end="url(#aR)" + id="line3" /> + marker-end="url(#aG)" + id="line8" /> + marker-end="url(#aR)" + id="line6" /> arm frame + x + y + z + transform="matrix(0.5234801,0,0,0.5234801,346.254,136.608)" /> Every end effector frame is defined relative to the arm frame at the flange and the world frame at the base. + id="text34">Every end effector frame is defined relative to the arm frame at the flange. diff --git a/assets/motion-planning/frame-system/held-object-frame.svg b/assets/motion-planning/frame-system/held-object-frame.svg index 11cf5e9466..bf39bfb762 100644 --- a/assets/motion-planning/frame-system/held-object-frame.svg +++ b/assets/motion-planning/frame-system/held-object-frame.svg @@ -74,51 +74,30 @@ - x + marker-end="url(#aG)" + id="line5" /> - z - y + marker-end="url(#aR)" + id="line3" /> + marker-end="url(#aG)" + id="line8" /> + marker-end="url(#aR)" + id="line6" /> object-tip frame gripper frame + x + y + z Date: Wed, 24 Jun 2026 14:46:52 -0600 Subject: [PATCH 12/46] Update docs/motion-planning/frame-system/end-effector-frames.md Co-authored-by: Shannon Bradshaw --- docs/motion-planning/frame-system/end-effector-frames.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index 340e5d3a94..7f50ef4366 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -19,7 +19,7 @@ First, look at where the end effector frame could be defined for two example use {{}} The end effector frame is hardware and task dependent. For a two-finger gripper, the -end effector frame sits between the fingertips, a short offset below the arm's frame. For +end effector frame typically sits between the fingertips, a short offset below the arm's frame. For a drill, it sits at the bit tip, offset from the arm's frame both forward and down, and the orientation may be flipped, so a positive z would mean to drill deeper. When you call `Move`, you have the option to specify which frame you are moving, and From 1eb509aa0e3aecf8b916d64bd24514ea52f97a5f Mon Sep 17 00:00:00 2001 From: btshrewsbury-viam Date: Wed, 24 Jun 2026 15:30:01 -0600 Subject: [PATCH 13/46] Update docs/motion-planning/frame-system/end-effector-frames.md Co-authored-by: Shannon Bradshaw --- docs/motion-planning/frame-system/end-effector-frames.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index 7f50ef4366..9428d1e0aa 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -82,7 +82,7 @@ Here is a gripper `frame` attribute from the [Arm with gripper and wrist camera] ## Why the frame you name matters A `Move` call that uses the arm frame produces different motion than one that -uses the drill-tip frame. The arm's frame sits at the flange, and the drill-tip +uses the end-effector frame. The arm's frame sits at the flange, and the end effector frame sits at the bit tip, so the same `target_pose` produces different motions. In the example below, the target pose is defined on the face of a box. If you use the drill-tip frame, the bit moves to the box face. If you use the arm frame, the From bc2480bf209b254abf65baee42dbebc189b38258 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 24 Jun 2026 16:33:41 -0600 Subject: [PATCH 14/46] Document orientation-vector tolerance coupling in pose clouds Per Dan's review: a pose cloud's OX/OY/OZ leeways are checked against a normalized orientation vector, so the three share one unit-sphere budget and do not move independently. Explain that you widen all three together to let the pointing direction vary, and that a wide OX/OY with a tight OZ holds the tool aligned because OZ pins the direction and the unit-sphere constraint then holds OX and OY at 0. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GdVfSBN5zBzSHStG1HoPDK --- docs/motion-planning/move-an-arm/pose-clouds.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/motion-planning/move-an-arm/pose-clouds.md b/docs/motion-planning/move-an-arm/pose-clouds.md index b9e4b96b68..aec7b7cf7a 100644 --- a/docs/motion-planning/move-an-arm/pose-clouds.md +++ b/docs/motion-planning/move-an-arm/pose-clouds.md @@ -65,8 +65,16 @@ can take a large `Theta` while keeping its pointing direction fixed. The position tolerances (`X`, `Y`, `Z`) are millimeters and `Theta` is degrees. The orientation-vector tolerances (`OX`, `OY`, `OZ`) are unitless deviations of the pointing -direction on a unit sphere, not angles: a value of `1` accepts any value for that -component, and `0` holds it exact. +direction on a unit sphere, not angles: on its own, a value of `1` accepts any value for +that component, and `0` holds it exact. + +Because the pointing direction is a unit vector, `OX`, `OY`, and `OZ` share one budget: the +squares of the three always sum to 1. They move together rather than one at a time. Tilting +the tool grows `OX` and `OY` and lowers `OZ` at the same time. To let the direction vary, +widen all three as a set; to hold it fixed, leave all three at `0`. Consider the combination +`OX: 1, OY: 1, OZ: 0`. It looks like "point in any direction", but `OZ: 0` holds `OZ` fixed, +and the unit-sphere constraint then keeps `OX` and `OY` at `0`. The tool stays exactly +aligned despite the wide `OX` and `OY` leeways. The more freedom you give the planner, the larger the set of arm configurations that satisfy the goal, so inverse kinematics is more likely to find a solution From d24137ab3499868e37db1a497f6da5eb51d07cce Mon Sep 17 00:00:00 2001 From: btshrewsbury-viam Date: Mon, 29 Jun 2026 13:30:10 -0600 Subject: [PATCH 15/46] Update docs/motion-planning/frame-system/overview.md Co-authored-by: Shannon Bradshaw --- docs/motion-planning/frame-system/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/frame-system/overview.md b/docs/motion-planning/frame-system/overview.md index 723ba6f9be..ea8fd1001c 100644 --- a/docs/motion-planning/frame-system/overview.md +++ b/docs/motion-planning/frame-system/overview.md @@ -284,7 +284,7 @@ If a component's pose looks wrong, check the translation and orientation values in its frame configuration. For details on all CLI motion commands, see [Motion Service Configuration](/motion-planning/reference/motion-service/#cli-commands). -## Targeting frames in motion +## Target an end effector for motion planning The motion service moves a named frame to a target pose, so which frame you configure for an end effector determines where the arm actually goes. For how From a1db84c0c0761f04757726a5f29ec2faaf4d1f0a Mon Sep 17 00:00:00 2001 From: btshrewsbury-viam Date: Mon, 29 Jun 2026 13:30:28 -0600 Subject: [PATCH 16/46] Update docs/motion-planning/move-an-arm/constraints.md Co-authored-by: Shannon Bradshaw --- docs/motion-planning/move-an-arm/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/move-an-arm/constraints.md b/docs/motion-planning/move-an-arm/constraints.md index 5e1f6dd85c..5474a1a0aa 100644 --- a/docs/motion-planning/move-an-arm/constraints.md +++ b/docs/motion-planning/move-an-arm/constraints.md @@ -118,7 +118,7 @@ For `CollisionSpecification` (allow specific frame pairs to collide), see Constraints restrict the _path_ between poses. The complementary tool is to relax the _goal_ itself. When close enough is acceptable, give the planner a region of acceptable poses instead of a single one, -which enlarges the solution set and makes planning faster and more reliable. This +which enlarges the solution set and makes planning faster. This is a pose cloud, and it is often the right move when a goal is failing or planning slowly. See [Pose clouds](/motion-planning/move-an-arm/pose-clouds/). From 8add3b1cd13cb04a12ee001b12f4e21e235dfc7f Mon Sep 17 00:00:00 2001 From: btshrewsbury-viam Date: Mon, 29 Jun 2026 13:31:17 -0600 Subject: [PATCH 17/46] Update docs/motion-planning/move-an-arm/pose-clouds.md Co-authored-by: Shannon Bradshaw --- docs/motion-planning/move-an-arm/pose-clouds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/move-an-arm/pose-clouds.md b/docs/motion-planning/move-an-arm/pose-clouds.md index aec7b7cf7a..594b35958c 100644 --- a/docs/motion-planning/move-an-arm/pose-clouds.md +++ b/docs/motion-planning/move-an-arm/pose-clouds.md @@ -83,7 +83,7 @@ and the search reaches it faster. ## Tolerances are measured in the target's frame The tolerances are evaluated in the reference frame of the target, not in world -coordinates. This matters whenever the target is tilted relative to the world. +coordinates. This matters when the target is tilted relative to the world. Consider a gripper approaching an inclined surface. You want to allow tolerance _along_ the surface but stay tight _into_ it, so the tool rides the surface From 5156732e6076840a64e241ab1200088ddc51faa9 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 17:12:17 -0600 Subject: [PATCH 18/46] Revise end effector frames page Restructure end-effector-frames.md into the arm frame, end effector frames, component-frame definition, moving with the motion service, and code-defined frame sections. Add the arm/joint-frames figure and fix the frame-system diagram triads (right-handed axes, DH joint orientation). Link the world state store service and WorldState references. --- .../frame-system/arm-joint-frames.svg | 71 +++++++++ .../frame-system/arm-vs-gripper-frame.svg | 12 +- .../frame-system/end-effector-types.svg | 142 ++++++++--------- .../frame-system/held-object-frame.svg | 82 +++++----- .../frame-system/end-effector-frames.md | 147 ++++++++++-------- 5 files changed, 269 insertions(+), 185 deletions(-) create mode 100644 assets/motion-planning/frame-system/arm-joint-frames.svg diff --git a/assets/motion-planning/frame-system/arm-joint-frames.svg b/assets/motion-planning/frame-system/arm-joint-frames.svg new file mode 100644 index 0000000000..c5864c3533 --- /dev/null +++ b/assets/motion-planning/frame-system/arm-joint-frames.svg @@ -0,0 +1,71 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + world frame + + + + + + + + + + my-arm:shoulder_lift_joint + + + my-arm:elbow_joint + + + my-arm:wrist_1_joint + + + my-arm frame + + Viam creates a frame at every joint and at the tool flange. + diff --git a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg index 7aa40ef97e..fadcf70015 100644 --- a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg +++ b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg @@ -1,13 +1,13 @@ - - - + + + - + @@ -47,8 +47,8 @@ - x - y + x + y z arm frame diff --git a/assets/motion-planning/frame-system/end-effector-types.svg b/assets/motion-planning/frame-system/end-effector-types.svg index bdc626af01..b0681d7aaf 100644 --- a/assets/motion-planning/frame-system/end-effector-types.svg +++ b/assets/motion-planning/frame-system/end-effector-types.svg @@ -34,37 +34,37 @@ id="defs23"> @@ -92,7 +92,7 @@ @@ -493,8 +493,8 @@ font-size="12" fill="#41506a" id="text26">arm frame - x - y + x + y z + transform="matrix(0.78,0,0,0.78,160.020,63.500)" /> + transform="matrix(0.78,0,0,0.78,160.372,-1.277)" /> + transform="matrix(0.78,0,0,0.78,-270.708,35.328)" /> + transform="matrix(0.78,0,0,0.78,-271.989,-1.059)" /> @@ -92,7 +92,7 @@ @@ -411,11 +411,11 @@ arm frame @@ -424,13 +424,13 @@ x="356" y="272" id="use25" - transform="matrix(0.67427158,-0.26554146,0.26554146,0.67427158,89.205933,188.71285)" /> + transform="matrix(0.7443,-0.2932,0.2932,0.7443,56.65,179.45)" /> + transform="matrix(0.80,0,0,0.80,71.22,-26.16)" /> gripper frame - x - y + x + y z ` after the kinematics file: link frames such as `my-arm:forearm_link` and +joint frames such as `my-arm:elbow_joint`. Parent a component to one of these frames to +mount it partway along the arm. For more information, see [Parent to an intermediate arm link](/motion-planning/frame-system/overview/#parent-to-an-intermediate-arm-link). + +{{}} + +## End effector frames + +The built-in arm frame is enough when you want to move the arm without an attached gripper or end effector. For most tasks, you attach a gripper or a tool to the end of the arm, and then the location you want to control is +where the tool acts, not where the arm flange is. An **end effector frame** marks that location. +The end effector frame is a user-defined frame that is both hardware and task dependent. For a two-finger gripper, the +end effector frame is typically between the fingertips, a short offset below the arm's +frame. For a drill, it is at the bit tip. {{}} -The end effector frame is hardware and task dependent. For a two-finger gripper, the -end effector frame typically sits between the fingertips, a short offset below the arm's frame. For -a drill, it sits at the bit tip, offset from the arm's frame both forward and down, -and the orientation may be flipped, so a positive z would mean to drill deeper. -When you call `Move`, you have the option to specify which frame you are moving, and -using the arm frame is always an option. By creating an end effector frame at the -correct location, a `Move` call becomes easier and better represents the task being -performed. +## Define an end effector frame through a component frame + +For end effector locations that are statically defined, such as a two-finger gripper or another tool that mounts rigidly to the arm's flange, you define a static frame through the component's `frame` attribute. +On the component's configuration card, set a `parent` to attach to (usually the arm), a `translation` that offsets the component's origin from the parent in millimeters, and an `orientation` that rotates its axes. Viam creates a frame named after the component, and uses it whenever you name that component. + +Here is a gripper `frame` attribute from the [Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/) example, which is 120 mm past the flange: + +```json +{ + "parent": "my-arm", + "translation": { "x": 0, "y": 0, "z": 120 }, + "orientation": { + "type": "ov_degrees", + "value": { "x": 0, "y": 0, "z": 1, "th": 0 } + } +} +``` -Now look at how the motion service describes the move. The snippets on this page -assume a connected motion service client, `motion_service` in Python and -`motionService` in Go. For how to set this up in more detail, see -[Access the motion service in your code](/reference/services/motion/#access-the-motion-service-in-your-code). +## Moving an arm with the motion service + +With the arm and component frames defined, you can move any of them to a pose. The +**motion service** moves a named frame to a destination pose. Its `Move` method takes a +frame name and a target pose, and the planner finds a collision-free path that brings that +frame to the pose: ```python from viam.proto.common import Pose, PoseInFrame -# A pose in world frame facing Z Down +# A target pose in the world frame, tool pointing down target_pose = PoseInFrame( reference_frame="world", pose=Pose(x=400, y=0, z=300, o_x=0, o_y=0, o_z=-1, theta=0), ) -# Move the Gripper Frame to the target_pose +# Move the my-arm frame to the target pose await motion_service.move( - component_name="my-gripper", + component_name="my-arm", destination=target_pose, ) ``` -You pass two parameters to `Move`: a string `component_name` and a `destination` -pose. - -## Which frame Move resolves - -The `destination` you pass is a target pose for the resolved frame, and the planner -tries to find a collision-free motion that moves the frame to the destination. +You pass two parameters to `Move`: a string `component_name` and a `destination`, a +`PoseInFrame` that pairs the target pose with the reference frame it is expressed in +(here, `world`). The `component_name` can be a component name or a frame name. The motion service looks for a frame with that name in the frame system. When you pass a component name -such as `"my-arm"` or `"my-gripper"`, Viam supplies a frame for it behind the scenes: +such as `"my-arm"` or `"my-gripper"`, Viam supplies a frame for it automatically: - **An arm**: its kinematics file (URDF or Viam JSON) defines a single output frame at the terminal link of the kinematic chain, commonly called the tool flange. That output frame @@ -66,24 +94,11 @@ such as `"my-arm"` or `"my-gripper"`, Viam supplies a frame for it behind the sc - **Another component**, such as a gripper, defines a static frame through its `frame` attribute, which you set on the component's configuration card. -Here is a gripper `frame` attribute from the [Arm with gripper and wrist camera](/motion-planning/frame-system/arm-gripper-camera/) example, whose origin sits 120 mm past the flange: - -```json -{ - "parent": "my-arm", - "translation": { "x": 0, "y": 0, "z": 120 }, - "orientation": { - "type": "ov_degrees", - "value": { "x": 0, "y": 0, "z": 1, "th": 0 } - } -} -``` - ## Why the frame you name matters A `Move` call that uses the arm frame produces different motion than one that -uses the end-effector frame. The arm's frame sits at the flange, and the end effector -frame sits at the bit tip, so the same `target_pose` produces different motions. +uses the end-effector frame. The arm's frame is at the flange and the end effector +frame at the bit tip, so the same `target_pose` produces different motions. In the example below, the target pose is defined on the face of a box. If you use the drill-tip frame, the bit moves to the box face. If you use the arm frame, the drill drives into the box, because the planner tries to move the arm frame to the @@ -94,31 +109,25 @@ target pose. ## Define an end effector frame in code The `frame` attribute described above works well when the end effector frame is static. -When the end effector frame moves during a task, or when defining it in code suits -the job better, add a new frame to the frame system with a `WorldState` transform. +When a task needs the end effector frame to change, or when defining it in code suits +the job better, add a new frame to the frame system with a [`WorldState`](https://python.viam.dev/autoapi/viam/proto/common/index.html#viam.proto.common.WorldState) transform. -A great example is a grasped part. Prior to pickup, the end effector frame is located +One example is when your end effector grasps an object. Before pickup, the end effector frame is located with the gripper. After pickup, the end effector frame may need to move to the part for -an assembly step. The grasped part may not have been in the machine configuration, -so you add its frame to the frame system at request time with a `WorldState` transform, -attached to the gripper so it moves with the grasp: +an assembly step: {{}} -To extend that frame system for -one motion, pass a -[`WorldState`](https://python.viam.dev/autoapi/viam/proto/common/index.html#viam.proto.common.WorldState) -to `Move` with extra frames. The steps differ slightly by language, but the shape is +The `WorldState` holds obstacles and frames that the motion service uses to plan collision-free motion. `Move` can consume a `world_state` to expand the obstacles and frames it plans around. To create a new frame, the steps differ slightly by language, but the approach is the same: -1. Define the translation and orientation from the parent frame to the origin of the +1. Create a WorldState Object +2. Define the translation and orientation from the parent frame to the new frame. In Python, this is a `PoseInFrame`; in Go, it is the pose you pass to - `NewLinkInFrame`. The parent is usually an arm or gripper component name, such as - `"my-gripper"` for a grasped object. -2. Build a transform from that pose and add it to the `WorldState`. - -The `WorldState` applies to a single `Move` call. A later call needs its own -`WorldState` to keep using the transform. + `NewLinkInFrame`. In the example above, the parent frame is the gripper + (`"my-gripper"`). +3. Build a transform from that pose and add it to a `WorldState`. +4. Call move using the `WorldState` object {{< tabs >}} {{% tab name="Python" %}} @@ -181,10 +190,13 @@ _, err = motionService.Move(ctx, motion.MoveReq{ {{% /tab %}} {{< /tabs >}} +The motion service keeps its own copy of the `WorldState` with any frames and collision geometries created by the machine's components and services. The `WorldState` you pass to `Move` is not added to it, so later calls must pass the `WorldState` again for the service to see those new frames or collisions. + ### Attach a geometry to a transform -A transform can also carry a collision geometry, a shape the planner avoids while it -tries to find collision-free motion to the destination. For a grasped object, it can be +A transform can also carry a collision geometry, a 3D shape the planner avoids while it +tries to find collision-free motion to the destination. A frame marks a location; a +geometry gives it size. For a grasped object, it can be helpful to attach geometry so the planner routes the grasped object around obstacles. Give the transform a box, sphere, capsule, or mesh defined in the frame's own coordinates: @@ -242,7 +254,7 @@ objectTipLink := referenceframe.NewLinkInFrame( ## WorldState versus the world state store service -The names are similar, but `WorldState` and the world state store service are +The names are similar, but [`WorldState`](https://python.viam.dev/autoapi/viam/proto/common/index.html#viam.proto.common.WorldState) and the [world state store service](/reference/apis/services/world-state-store/) are different things, and only one affects a `Move` call. - `WorldState` is the argument you pass to a single `Move` call. The obstacles @@ -258,7 +270,8 @@ publish it to the world state store service instead. ## Visualize the end effector frame -Static frames render in the 3D scene automatically. Open the **3D SCENE** tab on +A static frame stays fixed relative to its parent. Static frames render in the 3D scene +automatically. Open the **3D SCENE** tab on your machine's page, and the arm's built-in end effector frame and any `frame` attributes you configured appear as coordinate axes at their computed poses, the same axes drawn in the diagrams on this page. From cff1ac5b9e3e8b93a9824e615befdb386d67034e Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 17:38:11 -0600 Subject: [PATCH 19/46] Revise pose clouds page Rework pose-clouds.md: when-to-use guidance, the cup running example, the pose-cloud-as-box model, tolerances measured in the target frame, and the orientation-vector tolerance coupling plus the normalization sharp edge. Update the pose-cloud-cup and pose-cloud-target-frame diagrams (right-handed triads, label fixes). --- .../move-an-arm/pose-cloud-cup.svg | 8 +-- .../move-an-arm/pose-cloud-target-frame.svg | 12 ++-- .../move-an-arm/pose-clouds.md | 60 +++++++++---------- 3 files changed, 38 insertions(+), 42 deletions(-) diff --git a/assets/motion-planning/move-an-arm/pose-cloud-cup.svg b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg index d9da65a855..f83797f7ad 100644 --- a/assets/motion-planning/move-an-arm/pose-cloud-cup.svg +++ b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg @@ -493,15 +493,15 @@ pose cloud target frame @@ -178,15 +178,15 @@ }} ## Use a pose cloud with the motion service -A pose cloud attaches to the destination object. In Go, attach a pose cloud to the destination with +A pose cloud travels with the destination pose. In Go, attach the pose cloud to the destination with `NewPoseInFrameWithGoalCloud`, then pass that destination to `Move`. The pose is the center of the region and the `PoseCloud` is the tolerance around it. This example includes an orientation constraint for the water example, so the cup stays upright for the whole motion, not only at the goal. @@ -148,9 +145,8 @@ if err != nil { ``` {{% alert title="Available through the Go SDK" color="note" %}} -Today the Go SDK provides the `NewPoseInFrameWithGoalCloud` constructor; -The goal cloud travels on the destination's `PoseInFrame`, which is part of the -`Move` request sent to the motion service, so the planner honors it server side. +The Go SDK provides `NewPoseInFrameWithGoalCloud` to attach a pose cloud to the destination. +The goal cloud travels in the `Move` request, so the motion service honors it on the server. {{% /alert %}} ## What's next From 2338abb4d0e018dfec7eb9a3480e0f565f55aad6 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 17:48:59 -0600 Subject: [PATCH 20/46] Define frames and poses in the overview; fix the frame link Add brief frame and pose definitions to the frame-system overview so they are referenceable (per review), and repoint the end-effector-frames link to the new anchor. --- .../frame-system/end-effector-frames.md | 2 +- docs/motion-planning/frame-system/overview.md | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index 183a332eec..8be1755d76 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -8,7 +8,7 @@ description: "How Move places a named frame at a target pose, and how the frame --- When you move an arm with the motion service, you move a -[_frame_](/motion-planning/frame-system/overview/#frame) from its current +[_frame_](/motion-planning/frame-system/overview/#frames-and-poses) from its current pose to a destination pose. That frame is called the **end effector frame**, also called the tool control point (TCP) or the manipulation control frame. The end effector frame's location depends on the arm, the tool, and the task, and it can change partway diff --git a/docs/motion-planning/frame-system/overview.md b/docs/motion-planning/frame-system/overview.md index ea8fd1001c..20fc3c9720 100644 --- a/docs/motion-planning/frame-system/overview.md +++ b/docs/motion-planning/frame-system/overview.md @@ -37,6 +37,16 @@ things are in physical space. ## Concepts +### Frames and poses + +A **frame** is a coordinate system: an origin and three axes (x, y, z) attached to a +component or to a fixed point in the world. The frame system tracks where every frame sits +relative to the others. + +A **pose** is a position and orientation expressed in a particular frame. It pairs a +translation (the x, y, z offset from the frame's origin) with an orientation (the direction +something faces), so a pose only has meaning together with its reference frame. + ### The world frame The world frame is the fixed root of your frame system. You do not configure it From fce84ee30cb9e0befe1cc7820c630c724a98346b Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 18:16:20 -0600 Subject: [PATCH 21/46] Tighten frame-system figure captions and fix arm-frame label collision --- assets/motion-planning/frame-system/arm-vs-gripper-frame.svg | 4 ++-- assets/motion-planning/frame-system/end-effector-types.svg | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg index fadcf70015..271ea55c3b 100644 --- a/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg +++ b/assets/motion-planning/frame-system/arm-vs-gripper-frame.svg @@ -50,7 +50,7 @@ x y z - arm frame + arm frame drill-tip frame @@ -58,5 +58,5 @@ - The same target_pose moves a different point depending on which frame you name. + The frame you use changes how the arm moves. diff --git a/assets/motion-planning/frame-system/end-effector-types.svg b/assets/motion-planning/frame-system/end-effector-types.svg index b0681d7aaf..0d9ea3fd21 100644 --- a/assets/motion-planning/frame-system/end-effector-types.svg +++ b/assets/motion-planning/frame-system/end-effector-types.svg @@ -613,5 +613,5 @@ text-anchor="middle" font-size="13" fill="#41506a" - id="text34">Every end effector frame is defined relative to the arm frame at the flange. + id="text34">The tool frame sits where the tool acts, not at the arm flange. From bd105ebbca23d9e59b3a88073faa98d0462c78a9 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 18:17:31 -0600 Subject: [PATCH 22/46] Replace end-effector target-frame bullets with directive prose --- .../frame-system/end-effector-frames.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index 8be1755d76..ad2c73ca1f 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -84,15 +84,11 @@ You pass two parameters to `Move`: a string `component_name` and a `destination` `PoseInFrame` that pairs the target pose with the reference frame it is expressed in (here, `world`). -The `component_name` can be a component name or a frame name. The motion service -looks for a frame with that name in the frame system. When you pass a component name -such as `"my-arm"` or `"my-gripper"`, Viam supplies a frame for it automatically: - -- **An arm**: its kinematics file (URDF or Viam JSON) defines a single output frame at the - terminal link of the kinematic chain, commonly called the tool flange. That output frame - is the arm's built-in end effector frame. -- **Another component**, such as a gripper, defines a static frame through its `frame` - attribute, which you set on the component's configuration card. +Pass a component name, and the motion service uses the frame defined for that component. +For an arm, that frame is the `my-arm` frame at the tool flange, shown above. For a gripper +or another tool, it is the static frame you defined through its `frame` attribute, at the +point where the tool acts rather than at the flange. You can also pass a frame name directly, +such as one you add in code with a `WorldState` transform. ## Why the frame you name matters From a435c52c98fdf59dc0e30f5520fa37b976499a63 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 18:18:01 -0600 Subject: [PATCH 23/46] Fix pose-cloud-target-frame alt text to match the single-panel figure --- docs/motion-planning/move-an-arm/pose-clouds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/motion-planning/move-an-arm/pose-clouds.md b/docs/motion-planning/move-an-arm/pose-clouds.md index 95c01c7cb2..b1366f0d1d 100644 --- a/docs/motion-planning/move-an-arm/pose-clouds.md +++ b/docs/motion-planning/move-an-arm/pose-clouds.md @@ -91,7 +91,7 @@ own frame: large on the two in-plane axes so the tool can ride along the surface on the axis normal to the surface so it stays on the surface. Because a pose cloud uses the target's frame, these tolerances follow the tilt of the surface directly. -{{}} +{{}} ## Use a pose cloud with the motion service From c8088632fb931eaa5caad8a43221c37f6b22f4ed Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 30 Jun 2026 18:38:02 -0600 Subject: [PATCH 24/46] Lengthen pose-cloud-cup triad vectors so stems read clearly --- .../move-an-arm/pose-cloud-cup.svg | 46 +++++++++---------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/assets/motion-planning/move-an-arm/pose-cloud-cup.svg b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg index f83797f7ad..211e7bde8c 100644 --- a/assets/motion-planning/move-an-arm/pose-cloud-cup.svg +++ b/assets/motion-planning/move-an-arm/pose-cloud-cup.svg @@ -301,37 +301,37 @@ @@ -461,14 +461,14 @@ Date: Wed, 1 Jul 2026 11:26:25 -0600 Subject: [PATCH 25/46] Refine visualization pages: accuracy fixes, LO-driven ordering, clearer WorldState vs world state store --- .../3d-scene/debug-motion-plan.md | 7 +- docs/visualization/drawing-library.md | 29 ++--- .../publish-visuals-from-a-module.md | 19 ++-- docs/visualization/visuals-and-collisions.md | 106 +++++++++--------- 4 files changed, 79 insertions(+), 82 deletions(-) diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index e0b49d79ff..ac22d1d4ee 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -61,7 +61,8 @@ import ( ) func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { - sphere, err := spatialmath.NewSphere(pose, 5, fmt.Sprintf("step-%d", i)) + id := fmt.Sprintf("step-%d", i) + sphere, err := spatialmath.NewSphere(pose, 5, id) if err != nil { return nil, err } @@ -69,7 +70,9 @@ func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { if err != nil { return nil, err } - return drawn.Draw(fmt.Sprintf("step-%d", i), draw.WithPose(pose)) + // WithID derives a stable UUID from the string, so re-planning updates each + // marker in place instead of drawing a duplicate. + return drawn.Draw(id, draw.WithID(id), draw.WithPose(pose)) } ``` diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index 57a58f92db..2ae084d912 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -8,9 +8,10 @@ description: "Use the draw library to build visuals, and run the standalone Viam --- Viam Visualization is a standalone 3D app you run yourself. Unlike the **3D -SCENE** tab in the Viam app, which renders a machine's world state store service, -Viam Visualization is a separate tool for monitoring, testing, and debugging -spatial data: you start it locally and push visuals to it from your own Go code. +SCENE** tab in the Viam app, which renders a configured machine's frames, +geometry, point clouds, and published visuals, Viam Visualization is a separate tool for +monitoring, testing, and debugging spatial data: you start it locally and push +visuals to it from your own Go code. It shares the same `draw` library used to build world state store transforms, so the visuals you construct are the same either way. @@ -21,8 +22,9 @@ to run the app and drive it from a Go client. The two render 3D visuals, but they are different tools for different moments: -- The **3D SCENE tab** lives in the Viam app and renders a machine's world state - store service. It is the in-app view of a configured machine. +- The **3D SCENE tab** lives in the Viam app and renders a configured machine: + its frames, geometry, point clouds, and the custom visuals published to its + world state store service. It is the in-app view of a machine. - **Viam Visualization** is a standalone app you run on your own machine and push to from a client. It is for previewing and debugging spatial data while you develop, without deploying a module or opening the Viam app. @@ -92,18 +94,11 @@ _, err := api.DrawGeometry(api.DrawGeometryOptions{ This lets you preview spatial data, a point cloud, a set of detections, a planned path, straight from a script or test, without deploying a module or connecting -through the Viam app. For setup, the local server, and the full client API, see -the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). - -## How updates reach the browser - -The Viam Visualization app runs a **draw service** that the client API calls. The service -exposes `AddEntity`, `UpdateEntity`, and `RemoveEntity` for the changes you push, and a -`StreamEntity` stream the browser subscribes to. When you draw, update, or remove an -entity, the service fans that single change out over `StreamEntity` to the browser, which -applies it incrementally instead of re-rendering the whole scene. This is the same add, -update, and remove model the world state store service uses to feed the in-app 3D scene, -so a busy scene stays in sync as your data changes. +through the Viam app. Each push updates the app incrementally, the same add, +update, and remove model the world state store service uses to feed the in-app 3D +scene, so a busy scene stays in sync as your data changes. For setup, the local +server, and the full client API, see the +[Viam Visualization documentation](https://viamrobotics.github.io/visualization/). ## What's next diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 8945069fe0..e1cd97714d 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -123,7 +123,7 @@ lets the scene apply incremental changes instead of re-rendering. The diff against cached state is what turns a full snapshot each tick into a stream of minimal updates. -## The producer reads its dependencies +## Pull from the resources you visualize The module is the producer, and the resources it visualizes are its dependencies. The 3D scene reads only the world state store service; the service @@ -132,21 +132,18 @@ reads everything else. Data flows one way: > dependency resources → world state store module (reads, builds transforms) → 3D scene This is why the loop above calls `s.sensor.Readings(...)`: the sensor is a -dependency, and the module pulls from it. - -## Visualize any other resource - -A module whose primary job is something else (an arm, a sensor, a planner) stays -focused on that job. To visualize it, you write a separate world state store module -that takes that resource as a dependency and pulls from its existing API: +dependency, and the module pulls from it. The same pattern visualizes any other +resource. A module whose primary job is something else (an arm, a sensor, a +planner) stays focused on that job. To visualize it, you write a separate world +state store module that takes that resource as a dependency and pulls from its +existing API: - a sensor's `Readings` - a component's geometry getters - any resource's `DoCommand` -The world state store module converts what it reads into transforms and streams -them. The visualized resource needs no changes and no awareness that it is being -drawn. The store depends on it, not the other way around. +The visualized resource needs no changes and no awareness that it is being drawn. +The store depends on it, not the other way around. ```go func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstatestore.Service, error) { diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index 24b7176af0..22834d1148 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -7,15 +7,15 @@ type: "docs" description: "How a Transform defines a custom visual, and which geometry the motion planner actually collision-checks." --- -Viam uses several standard types to define static and dynamic geometries, and -the same types are used to create visuals or to tell the motion planner about -obstacles. +A geometry is a simple shape, such as a box, sphere, or capsule, that represents an +object's physical extent. Viam uses one set of geometry types for two jobs: drawing a +custom visual in the 3D scene, and telling the motion planner about an obstacle. -To create a custom visual in the 3D scene, you must create a geometry and -attach it to `Transform.Geometry` then provide that transform to a World State Store Service. -The world state store service streams these transforms to the 3d scene. This page -covers what a transform contains, how the scene tracks it over time, and discusses -the difference between planner geometry and visualization geometry. +To create a custom visual, you attach a geometry to `Transform.Geometry` and provide +that transform to a [world state store service](/reference/apis/services/world-state-store/). +This service holds the transforms you publish and streams them to the 3D scene. This +page covers what a transform contains, the difference between planner geometry and +visualization geometry, and how to build each geometry type. ## Anatomy of a transform @@ -39,6 +39,52 @@ to add a new one, it uses a fresh UUID. Without stable identifiers the client would have to re-render everything on every change. With them, the scene applies incremental add, update, and remove operations to individual visuals. +## Metadata styles the visual + +The metadata is a set of rendering attributes the scene reads when it draws the +geometry: + +- `color`: the fill color +- `opacity`: how transparent the shape is +- per-point colors: for point cloud geometry +- `collision_allowed`: a rendering hint that marks the visual as a permitted + contact, for display only + +These are all **visualization attributes**: they control how the visual looks, +`collision_allowed` included. The planner reads its solid geometry from the frame +system and the [`WorldState`](/motion-planning/obstacles/) you pass to `Move`, so +metadata changes what you see without changing what the planner plans around. + +## The scene draws, the planner collision-checks + +The geometry on a world state store transform renders in the 3D scene: publishing a +box draws a box. The motion planner collision-checks a separate geometry, which it +reads from two places: + +- The **frame system**: each component's `frame.geometry`. +- The **`WorldState`** you pass to a `Move` call: obstacles and transforms + supplied for that single planning request. + +Despite the similar names, these are different things: the +[world state store service](/reference/apis/services/world-state-store/) holds transforms +for the scene to draw, and the [`WorldState`](/motion-planning/obstacles/) you pass to +`Move` carries obstacles for the planner to avoid. A world state store transform and a +`WorldState` obstacle travel two paths, each with its own job: one is drawn in the scene, +the other is planned around. The same shape can take both paths. + +## Making a geometry both visible and collision-checked + +If you want a geometry to appear in the scene _and_ be avoided by the planner, +you do both, separately: + +- **For the scene**: publish it as a transform through the world state store + service (see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/)). +- **For planning**: add it to the frame system, or include it in the + `WorldState` you pass to `Move`. + +Today these are two separate outputs you produce from the same source data: one +transform for the scene, one geometry for the planner. + ## Geometry types The supported types are: @@ -269,50 +315,6 @@ pointCloud := &commonpb.Geometry{ {{% /tab %}} {{< /tabs >}} -## Metadata styles the visual - -The metadata is a set of rendering attributes the scene reads when it draws the -geometry: - -- `color`: the fill color -- `opacity`: how transparent the shape is -- per-point colors: for point cloud geometry -- `collision_allowed`: a hint about whether the geometry represents an allowed - collision - -These are **visualization attributes**: the scene reads them when it draws the -geometry. They control how the visual looks, including `collision_allowed`, which is -a rendering hint about the visual. The planner reads its solid geometry from the frame -system and `WorldState` instead, so these attributes shape the picture while the -planner's obstacles come from elsewhere. - -## The scene draws, the planner collision-checks - -The geometry on a world state store transform renders in the 3D scene: publishing a -box draws a box. The motion planner collision-checks a separate geometry, which it -reads from two places: - -- The **frame system**: each component's `frame.geometry`. -- The **`WorldState`** you pass to a `Move` call: obstacles and transforms - supplied for that single planning request. - -A world state store transform and a `WorldState` obstacle travel two paths, each with -its own job: one is drawn in the scene, the other is planned around. The same shape can -take both paths. - -## Making a geometry both visible and collision-checked - -If you want a geometry to appear in the scene _and_ be avoided by the planner, -you do both, separately: - -- **For the scene**: publish it as a transform through the world state store - service (see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/)). -- **For planning**: add it to the frame system, or include it in the - `WorldState` you pass to `Move`. - -Today these are two separate outputs you produce from the same source data: one -transform for the scene, one geometry for the planner. - ## What's next - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): From dbdc58cb2b0fbc319afef3a9e32d1932e67421dc Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 1 Jul 2026 13:17:27 -0600 Subject: [PATCH 26/46] Fix marker pose double-application; close LO gaps in visualization how-tos - Build geometries at NewZeroPose() and place via WithPose (was applying the pose twice: geometry center + transform pose) - publish-visuals: add GetTransform, StreamTransformChanges, and emit fan-out - debug-motion-plan: add goalMarker and clarify goal-pose source - drawing-library: restore the draw-service add/update/remove fan-out passage --- .../3d-scene/debug-motion-plan.md | 23 +++++++-- docs/visualization/drawing-library.md | 14 ++++-- .../publish-visuals-from-a-module.md | 48 ++++++++++++++++++- 3 files changed, 77 insertions(+), 8 deletions(-) diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index ac22d1d4ee..1b754f2c67 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -62,7 +62,8 @@ import ( func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { id := fmt.Sprintf("step-%d", i) - sphere, err := spatialmath.NewSphere(pose, 5, id) + // Build the sphere at the origin; WithPose below places it at the step pose. + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 5, id) if err != nil { return nil, err } @@ -74,10 +75,26 @@ func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { // marker in place instead of drawing a duplicate. return drawn.Draw(id, draw.WithID(id), draw.WithPose(pose)) } + +// goalMarker draws a destination pose. Unlike the trajectory poses, a goal pose is +// not a ComputePoses output: it is the destination you passed to the planner. +func goalMarker(i int, goalPose spatialmath.Pose) (*commonpb.Transform, error) { + id := fmt.Sprintf("goal-%d", i) + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 8, id) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(goalColor)) + if err != nil { + return nil, err + } + return drawn.Draw(id, draw.WithID(id), draw.WithPose(goalPose)) +} ``` -Draw the goal poses the same way with a distinct color, so the target stands out -from the trajectory leading to it. +The trajectory poses come from `ComputePoses`, but the goal pose is the destination +you passed to the planner. `goalMarker` draws it with a distinct color, so the target +stands out from the trajectory leading to it. ## Serve the transforms to the scene diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index 2ae084d912..09de3a0112 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -94,12 +94,18 @@ _, err := api.DrawGeometry(api.DrawGeometryOptions{ This lets you preview spatial data, a point cloud, a set of detections, a planned path, straight from a script or test, without deploying a module or connecting -through the Viam app. Each push updates the app incrementally, the same add, -update, and remove model the world state store service uses to feed the in-app 3D -scene, so a busy scene stays in sync as your data changes. For setup, the local -server, and the full client API, see the +through the Viam app. For setup, the local server, and the full client API, see the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). +## How updates reach the browser + +The app runs a **draw service** that the client API calls. Each push becomes an +`AddEntity`, `UpdateEntity`, or `RemoveEntity` operation, and the service fans that +single change out over a `StreamEntity` stream the browser subscribes to. The browser +applies the one change instead of re-rendering the scene. This is the same add, +update, and remove model the world state store service uses to feed the in-app 3D +scene, so a busy scene stays in sync as your data changes. + ## What's next - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index e1cd97714d..c57e58baf5 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -55,6 +55,51 @@ func (s *visualizer) ListUUIDs( } ``` +`GetTransform` is a map lookup by UUID. `StreamTransformChanges` hands the caller a +stream backed by a fresh subscriber channel, and the poll loop pushes each change +onto every subscriber through a small `emit` helper: + +```go +func (s *visualizer) GetTransform( + ctx context.Context, uuid []byte, extra map[string]any, +) (*commonpb.Transform, error) { + s.mu.RLock() + defer s.mu.RUnlock() + tf, ok := s.transforms[string(uuid)] + if !ok { + return nil, fmt.Errorf("no transform with uuid %s", uuid) + } + return tf, nil +} + +func (s *visualizer) StreamTransformChanges( + ctx context.Context, extra map[string]any, +) (*worldstatestore.TransformChangeStream, error) { + ch := make(chan worldstatestore.TransformChange, 32) + s.mu.Lock() + s.subscribers = append(s.subscribers, ch) + s.mu.Unlock() + return worldstatestore.NewTransformChangeStreamFromChannel(ctx, ch), nil +} + +// emit fans one change out to every stream subscriber. +func (s *visualizer) emit( + tf *commonpb.Transform, kind pb.TransformChangeType, updated []string, +) { + change := worldstatestore.TransformChange{ + ChangeType: kind, Transform: tf, UpdatedFields: updated, + } + s.mu.RLock() + defer s.mu.RUnlock() + for _, ch := range s.subscribers { + select { + case ch <- change: + default: // skip a subscriber whose buffer is full + } + } +} +``` + ## Build transforms with the draw library Rather than assembling `commonpb.Transform` protos by hand, use the `draw` @@ -69,7 +114,8 @@ import ( ) func buildTransform(o obstacle) (*commonpb.Transform, error) { - box, err := spatialmath.NewBox(o.Pose, o.Dims, o.ID) + // Build the box at the origin; WithPose below places it at the obstacle pose. + box, err := spatialmath.NewBox(spatialmath.NewZeroPose(), o.Dims, o.ID) if err != nil { return nil, err } From fa4d646fea92867c18dfbdb2ce8459370f144dac Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 1 Jul 2026 13:21:31 -0600 Subject: [PATCH 27/46] Close remaining visualization LO gaps - visuals-and-collisions: add a worked full-Transform assembly example (reference frame, pose, geometry, metadata, UUID) covering LO1 and metadata-in-code - drawing-library: add the make setup / make up local-run command (LO6) - debug-motion-plan: name the service methods that serve the plan markers (LO4) --- .../3d-scene/debug-motion-plan.md | 7 ++-- docs/visualization/drawing-library.md | 8 ++-- docs/visualization/visuals-and-collisions.md | 40 +++++++++++++++++++ 3 files changed, 49 insertions(+), 6 deletions(-) diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index 1b754f2c67..3751d85f4b 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -99,9 +99,10 @@ stands out from the trajectory leading to it. ## Serve the transforms to the scene Serve the transforms through a world state store service so the **3D SCENE** tab -renders them. The plan markers stream in alongside the frames and obstacle -geometry the scene already shows. For the service methods, the poll-and-update -loop, and how a module pulls data from its dependencies, see +renders them. Return your step and goal markers from the service's `ListUUIDs`, +`GetTransform`, and `StreamTransformChanges` methods, and the plan streams in alongside +the frames and obstacle geometry the scene already shows. For those methods, the +poll-and-update loop, and how a module pulls data from its dependencies, see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/). ## Diagnose a failed or surprising plan diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index 09de3a0112..83736a993f 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -74,9 +74,11 @@ styles, not field-by-field struct assembly. ## Run the app and push from a Go client -Viam Visualization runs locally and renders in your browser. You start the app, -then push visuals to it from Go with the client API. Reusing an entity ID updates -that visual in place; a new ID adds another: +Viam Visualization runs locally and renders in your browser. From the +[motion-tools repository](https://github.com/viam-labs/motion-tools), run `make setup` +once, then `make up` to start the app at `http://localhost:5173`. With the app running, +push visuals to it from Go with the client API. Reusing an entity ID updates that visual +in place; a new ID adds another: ```go import ( diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index 22834d1148..70142f3c88 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -315,6 +315,46 @@ pointCloud := &commonpb.Geometry{ {{% /tab %}} {{< /tabs >}} +## Assemble a transform + +The geometry above is only the shape. To render it as a visual, wrap it in a transform +that carries the other anatomy fields: a reference frame and pose, metadata, and a UUID. +The `draw` library assembles them for you, so you do not build the `Transform` proto by +hand: + +```go +import ( + "github.com/golang/geo/r3" + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +// A red box at (400, 0, 200) in the world frame. +box, err := spatialmath.NewBox( + spatialmath.NewZeroPose(), r3.Vector{X: 100, Y: 100, Z: 100}, "target-box", +) +if err != nil { + return nil, err +} +drawn, err := draw.NewDrawnGeometry( + box, + draw.WithGeometryColor(draw.NewColor(draw.WithName("red"))), // metadata: color +) +if err != nil { + return nil, err +} +transform, err := drawn.Draw( + "target-box", // the name of this visual's frame + draw.WithID("target-box"), // UUID: a stable identity for later updates + draw.WithParent("world"), // reference frame the pose is expressed in + draw.WithPose(spatialmath.NewPoseFromPoint(r3.Vector{X: 400, Y: 0, Z: 200})), // pose +) +``` + +Publish `transform` through a world state store service to draw it. For the full module +that serves transforms this way, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + ## What's next - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): From 9c32cbb2d493d2517ac4fdb56fb70723d503886b Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 1 Jul 2026 13:47:49 -0600 Subject: [PATCH 28/46] Add new visualization pages: 3D-scene explainer, point clouds, World State reference - Visualizing with the 3D scene: the section's 3D-scene explainer, owning the four 3D-scene learning objectives (identify elements, sources/trace-to-origin, built-in-vs-custom, streamed updates) - Perception > Point Clouds: how depth-camera point clouds render, display, and compare - Reference > World State: what a per-request WorldState is, vs the world state store service - Moves/renames of existing pages are deferred to the later reorg --- docs/visualization/perception/_index.md | 16 ++++ docs/visualization/perception/point-clouds.md | 43 ++++++++++ docs/visualization/reference/_index.md | 15 ++++ docs/visualization/reference/world-state.md | 44 ++++++++++ .../visualizing-with-the-3d-scene.md | 82 +++++++++++++++++++ 5 files changed, 200 insertions(+) create mode 100644 docs/visualization/perception/_index.md create mode 100644 docs/visualization/perception/point-clouds.md create mode 100644 docs/visualization/reference/_index.md create mode 100644 docs/visualization/reference/world-state.md create mode 100644 docs/visualization/visualizing-with-the-3d-scene.md diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md new file mode 100644 index 0000000000..5b0ff6845e --- /dev/null +++ b/docs/visualization/perception/_index.md @@ -0,0 +1,16 @@ +--- +linkTitle: "Perception" +title: "Perception in the 3D scene" +weight: 60 +layout: "docs" +type: "docs" +no_list: true +description: "View and compare live perception data, such as depth-camera point clouds, in the 3D scene." +--- + +The 3D scene renders live perception data alongside your frame system, so you can see what +your machine senses in the same space as the components that sense it. + +{{< cards >}} +{{% card link="/visualization/perception/point-clouds/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/perception/point-clouds.md b/docs/visualization/perception/point-clouds.md new file mode 100644 index 0000000000..1a2e879cc6 --- /dev/null +++ b/docs/visualization/perception/point-clouds.md @@ -0,0 +1,43 @@ +--- +linkTitle: "Point Clouds" +title: "Point clouds in the 3D scene" +weight: 10 +layout: "docs" +type: "docs" +description: "How depth-camera point clouds render in the 3D scene, how to adjust their display, and how to load and compare external point clouds." +--- + +A depth camera reports the distance to points in front of it. The 3D scene renders that data +as a colored point set, placed in the scene by the camera's frame, so you can see what the +camera perceives in the same space as the rest of your machine. + +## Live point clouds + +When your machine is online, the scene streams point clouds from your depth cameras and +draws each as a set of points at the camera's frame. Because each point cloud sits at its +camera's frame, a point cloud that lands in the wrong place usually points to a wrong camera +frame, not wrong perception. + +Vision services can also contribute point-cloud entities, such as the points behind a +detection. These render the same way. + +## Adjust how point clouds display + +Open the **Settings** panel (gear icon) in the 3D scene tab to control point-cloud rendering: + +- **Point size and color**: set the default size and color the scene draws points at. +- **Enabled cameras**: turn each camera's point cloud on or off, so you can focus on one + camera at a time. +- **Vision**: enable or disable vision-service point-cloud entities. + +## Load and compare external point clouds + +To inspect a saved capture, drag a `.pcd` or `.ply` file onto the viewport. The scene loads +it as a point cloud you can view alongside your live data, which is useful for comparing a +saved SLAM map or scan against the current frame system. + +To compare two point clouds that should align, such as a scan and a transformed copy, link +them with **HoverLink**: select one point cloud, add a HoverLink relationship to the other, +and hovering a point in one highlights the matching point in the other. For a full alignment +walkthrough, see +[Verify point cloud alignment](/motion-planning/3d-scene/verify-point-cloud-alignment/). diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md new file mode 100644 index 0000000000..6402e93f34 --- /dev/null +++ b/docs/visualization/reference/_index.md @@ -0,0 +1,15 @@ +--- +linkTitle: "Reference" +title: "Visualization reference" +weight: 90 +layout: "docs" +type: "docs" +no_list: true +description: "Reference material for the types and services behind Viam visualization." +--- + +Reference pages for the types and services that back the 3D scene and custom visuals. + +{{< cards >}} +{{% card link="/visualization/reference/world-state/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/reference/world-state.md b/docs/visualization/reference/world-state.md new file mode 100644 index 0000000000..1ffb2b7b05 --- /dev/null +++ b/docs/visualization/reference/world-state.md @@ -0,0 +1,44 @@ +--- +linkTitle: "World State" +title: "WorldState" +weight: 10 +layout: "docs" +type: "docs" +description: "What a WorldState is: the per-request set of obstacles and frame transforms you pass to a single Move call for the planner to plan around." +--- + +A `WorldState` is the argument you pass to a single `Move` call. It carries the obstacles +and frame transforms that the motion planner should account for on that one request, and +nothing else uses it. When the call returns, the `WorldState` is gone. + +## What a WorldState carries + +A `WorldState` holds two kinds of item: + +- **Obstacles**: geometries the planner treats as things to avoid, each expressed in a + reference frame. Use them for objects that are not part of the machine's configured frame + system, such as a pallet detected at runtime. +- **Transforms**: frames added to the frame system for this request only. A transform can + reposition where the arm moves (for example, a frame at a grasped object's tip) or carry a + geometry that travels with a component. + +## It applies to one request + +A `WorldState` is per-request. The motion service uses it to plan the single motion you +attach it to, then discards it. It is not stored, so a later `Move` call sees none of it +unless you pass a `WorldState` again. To build obstacles and transforms and attach them to a +`Move` call, see [Define obstacles](/motion-planning/obstacles/) and +[Arm and end effector frames](/motion-planning/frame-system/end-effector-frames/). + +## WorldState versus the world state store service + +The names are similar, but the two are different things, and only one affects a plan: + +- **`WorldState`** is planner input. The obstacles and transforms in it shape the motion the + planner computes for one `Move` call. +- The [world state store service](/reference/apis/services/world-state-store/) is + visualization output. It holds transforms that the 3D scene draws, and the planner never + reads it. + +The same geometry can take both paths, but they are separate: put it in a `WorldState` for +the planner to avoid, and publish it to the world state store service for the scene to draw. diff --git a/docs/visualization/visualizing-with-the-3d-scene.md b/docs/visualization/visualizing-with-the-3d-scene.md new file mode 100644 index 0000000000..1a3414930e --- /dev/null +++ b/docs/visualization/visualizing-with-the-3d-scene.md @@ -0,0 +1,82 @@ +--- +linkTitle: "Visualizing with the 3D scene" +title: "Visualizing with the 3D scene" +weight: 5 +layout: "docs" +type: "docs" +description: "What the 3D scene renders, where each element comes from, and how built-in configuration content differs from custom visuals a module publishes at runtime." +--- + +The **3D SCENE** tab renders your machine in an interactive 3D view: the frames of every +component, the geometry attached to them, point clouds from depth cameras, and any custom +visuals a module publishes while the machine runs. It turns configuration you would +otherwise read as JSON numbers into a picture you can inspect, so you can confirm a gripper +sits where the arm expects it or watch a motion plan against the obstacles around it. + +This page describes what the scene shows, where each element comes from, and how the scene +stays current. For the tab's panels, navigation, and settings, see the +[3D scene tab reference](/motion-planning/3d-scene/). + +## What the scene renders + +The scene draws four kinds of element, each with its own appearance: + +- **Component frames** render as sets of red, green, and blue coordinate axes, one per + component, positioned by each frame's translation and orientation. +- **Geometries** render as translucent shapes (a box, sphere, or capsule) at the frame they + are attached to. +- **Point clouds** from depth cameras render as colored point sets. +- **Custom visuals** render as whatever a module draws them as: markers, arrows, lines, + meshes, or extra geometries, published while the machine runs. + +## Where each element comes from + +Every element in the scene traces back to one source. The **World panel** lists every entity +in a tree rooted at the world frame, so you can select anything on screen and follow it back +to the component or module that produced it. + +| Element in the scene | Comes from | +| -------------------- | ----------------------------------------------------------- | +| Component frame | The frame system, from each component's frame configuration | +| Attached geometry | A component's `frame.geometry` in configuration | +| Point cloud | A depth camera, streamed live when the machine is online | +| Custom visual | A module, published through a world state store service | + +The first three appear because they are part of the machine's configuration, and the scene +reads that configuration whenever you open the tab. Custom visuals appear only when a module +publishes them. + +## Built-in content versus custom visuals + +The distinction between the sources matters because it tells you _where to make a change_: + +- **Built-in content** is the frame system and configured geometry. The scene shows it with + no code. To change a frame's position or a geometry's shape, edit the component's + configuration. +- **Custom visuals** are everything a module computes or senses at runtime: a detection, a + planned path, a sensor's obstacle readings. The scene shows them only while the module + publishes them. To change a custom visual, change the module's code. + +If an element looks wrong, this split tells you where to look: a misplaced frame or geometry +is a configuration problem, while a missing or stale custom visual is a module problem. + +## How the scene stays current + +Built-in content reflects your configuration and, when the machine is online, each +component's live pose. Custom visuals stay current a different way: a module publishes them +to a [world state store service](/reference/apis/services/world-state-store/), and the scene +subscribes to that service's stream of transform changes. As the module adds, updates, and +removes transforms, the scene applies each change to the one affected visual instead of +redrawing everything, so a busy scene keeps up as the underlying data changes. + +To publish your own custom visuals this way, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## What's next + +{{< cards >}} +{{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} +{{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} +{{% card link="/visualization/drawing-library/" noimage="true" %}} +{{% card link="/motion-planning/3d-scene/" noimage="true" %}} +{{< /cards >}} From 5af83a772ef6951d2173867c769d650676a88b3f Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 1 Jul 2026 13:58:25 -0600 Subject: [PATCH 29/46] Add 3D Scene Widgets and Cameras pages, sourced from the live 3D scene - 3D Scene Tools > 3D Scene Widgets: the widget overlays (Arm positions joint table, Camera widgets), enabled from Settings > Widgets - Perception > Cameras: a camera's live feed widget (resolution, fps), its point cloud, and its frame Content verified against the running app's 3D SCENE tab (viam-server 0.132.0). --- .../3d-scene-tools/3d-scene-widgets.md | 41 +++++++++++++++++++ docs/visualization/3d-scene-tools/_index.md | 16 ++++++++ docs/visualization/perception/_index.md | 1 + docs/visualization/perception/cameras.md | 33 +++++++++++++++ 4 files changed, 91 insertions(+) create mode 100644 docs/visualization/3d-scene-tools/3d-scene-widgets.md create mode 100644 docs/visualization/3d-scene-tools/_index.md create mode 100644 docs/visualization/perception/cameras.md diff --git a/docs/visualization/3d-scene-tools/3d-scene-widgets.md b/docs/visualization/3d-scene-tools/3d-scene-widgets.md new file mode 100644 index 0000000000..8553a3251e --- /dev/null +++ b/docs/visualization/3d-scene-tools/3d-scene-widgets.md @@ -0,0 +1,41 @@ +--- +linkTitle: "3D Scene Widgets" +title: "3D scene widgets" +weight: 30 +layout: "docs" +type: "docs" +description: "Overlay live data on the 3D scene with widgets: an arm's joint positions and a camera's live feed." +--- + +Widgets are floating panels you overlay on the 3D scene to read live data next to the 3D +view. Each widget is draggable and resizable, and you turn them on and off in the scene's +settings. + +## Enable a widget + +Open the **Settings** panel (the gear icon in the scene toolbar) and select **Widgets**. +Toggle a widget on and it appears floating over the viewport; toggle it off to remove it. The +panel lists two kinds of widget: **Arm positions**, and a **Camera widgets** section with one +toggle per configured camera. + +## Arm positions + +The **Arm positions** widget shows the live joint angles of an arm. It lists each joint by +index with its current position in degrees, and updates as the arm moves: + +| Joint | Position (degrees) | +| ----- | ------------------ | +| 0 | -0.00 | +| 1 | -11.29 | +| 2 | -20.05 | +| ... | ... | + +If your machine has more than one arm, a **Select arm** dropdown at the top of the widget +chooses which arm's joints to show. Use this widget to read an arm's current configuration +while you inspect its pose in the 3D view. + +## Camera widgets + +A camera widget shows a camera's live feed as a panel over the 3D scene. Each configured +camera has its own toggle under **Camera widgets**. For the feed's resolution and frame-rate +controls, see [Cameras](/visualization/perception/cameras/). diff --git a/docs/visualization/3d-scene-tools/_index.md b/docs/visualization/3d-scene-tools/_index.md new file mode 100644 index 0000000000..cc62965a85 --- /dev/null +++ b/docs/visualization/3d-scene-tools/_index.md @@ -0,0 +1,16 @@ +--- +linkTitle: "3D Scene Tools" +title: "3D scene tools" +weight: 40 +layout: "docs" +type: "docs" +no_list: true +description: "Tools built into the 3D scene: overlay widgets for live data, measurement, and visual frame editing." +--- + +The 3D scene includes tools for reading live data and adjusting your frame system without +leaving the view. + +{{< cards >}} +{{% card link="/visualization/3d-scene-tools/3d-scene-widgets/" noimage="true" %}} +{{< /cards >}} diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md index 5b0ff6845e..47f88bfaf8 100644 --- a/docs/visualization/perception/_index.md +++ b/docs/visualization/perception/_index.md @@ -13,4 +13,5 @@ your machine senses in the same space as the components that sense it. {{< cards >}} {{% card link="/visualization/perception/point-clouds/" noimage="true" %}} +{{% card link="/visualization/perception/cameras/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/perception/cameras.md b/docs/visualization/perception/cameras.md new file mode 100644 index 0000000000..8c9635933c --- /dev/null +++ b/docs/visualization/perception/cameras.md @@ -0,0 +1,33 @@ +--- +linkTitle: "Cameras" +title: "Cameras in the 3D scene" +weight: 20 +layout: "docs" +type: "docs" +description: "View a camera's live feed alongside the 3D scene, and relate it to the camera's frame and point cloud." +--- + +A camera contributes to the 3D scene in more than one way: its frame positions it in space, +its live feed shows what it currently sees, and a depth camera also produces a point cloud. + +## See a camera's live feed + +To watch a camera's feed next to the 3D view, open the **Settings** panel (gear icon), select +**Widgets**, and toggle the camera on under **Camera widgets**. A floating panel then shows +the live feed with its current frame rate, for example `20.0fps`. + +A resolution dropdown on the widget sets the feed size: **Default**, `1280x720`, `640x360`, +`320x180`, `160x90`, or `80x44`. A lower resolution costs less bandwidth, which helps when you +watch several cameras at once or work over a slow connection. + +## Point clouds from depth cameras + +A depth camera also reports the distance to points in front of it, which the scene renders as +a point cloud. For how point clouds render, display, and compare, see +[Point clouds](/visualization/perception/point-clouds/). + +## Camera frames + +A camera appears in the scene as a coordinate frame, and its point cloud renders at that +frame. If a camera's point cloud lands in the wrong place, check the camera's frame +configuration rather than the camera itself. From a0a1d988508bcf063f0da918949b5ff37e1fb10d Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 08:27:15 -0600 Subject: [PATCH 30/46] Reword Debug a motion plan intro; drop 'static inspector' --- docs/motion-planning/3d-scene/debug-motion-plan.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index 3751d85f4b..4b55e5f71f 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -7,11 +7,11 @@ type: "docs" description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." --- -The **3D SCENE** tab is a static inspector: it shows the configured frame system -and live component poses. To see a motion plan, the trajectory the arm will follow, -the goals it aims for, and how that path relates to your obstacles, you publish the -plan as **custom visuals** through a world state store service. The scene then renders -the path you can otherwise only read as numbers. +The **3D SCENE** tab shows your configured frame system and live component poses. To +also see a motion plan, the trajectory the arm will follow, the goals it aims for, and +how that path relates to your obstacles, you publish the plan as **custom visuals** +through a world state store service. The scene then renders the path you can otherwise +only read as numbers. This page shows how to turn a plan into transforms the scene can draw, and how to use the rendered path to debug a plan that failed or moved unexpectedly. From 94f616627194416934d80a3aa715580d32c7fc2e Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 08:34:33 -0600 Subject: [PATCH 31/46] Revert build-apps _index (out of scope); make visualization nav consistent - Restore build-apps/_index.md to main: the empty_node/layout:empty change belongs in the later reorg PR, not here - visualization/_index.md: drop empty_node/layout:empty, use layout:docs + manualLink to /visualization/overview/ so the section renders as a normal nav link like every other section instead of a non-clickable header --- docs/build-apps/_index.md | 6 +++--- docs/visualization/_index.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/build-apps/_index.md b/docs/build-apps/_index.md index f25dc1165b..6599d00c95 100644 --- a/docs/build-apps/_index.md +++ b/docs/build-apps/_index.md @@ -2,10 +2,10 @@ linkTitle: "Build apps" title: "Build apps" weight: 65 -empty_node: true -layout: "empty" -canonical: "/build-apps/overview/" +layout: "docs" type: "docs" +no_list: true +manualLink: "/build-apps/overview/" description: "Build client apps that talk to your Viam machines and the Viam cloud." date: "2026-04-10" aliases: diff --git a/docs/visualization/_index.md b/docs/visualization/_index.md index 6e4b0a5b88..a759cbfca4 100644 --- a/docs/visualization/_index.md +++ b/docs/visualization/_index.md @@ -2,9 +2,9 @@ linkTitle: "Visualization" title: "Visualization" weight: 165 -empty_node: true -layout: "empty" -canonical: "/visualization/overview/" +layout: "docs" type: "docs" +no_list: true +manualLink: "/visualization/overview/" description: "See your machine in 3D: frames, geometries, point clouds, and custom visuals your modules publish." --- From 477106d86ee4061d51eee2847efe688226124d15 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 08:47:58 -0600 Subject: [PATCH 32/46] Split Debug a motion plan: restore no-code checks, move viz to its own page The PR reframe replaced the original visual-inspection guide with a code-heavy publish-as-custom-visuals technique. Restore the practical no-code checks (frame positions, obstacle geometry, reach, self-collision, common-causes table) as 'Debug a motion plan', and move the visualization technique to a new 'Visualize a motion plan' page. Each links to the other; Debug is the start, Visualize is the deeper step. Repoint cards accordingly. --- docs/motion-planning/3d-scene/_index.md | 1 + .../3d-scene/debug-motion-plan.md | 227 ++++++++---------- .../3d-scene/visualize-a-motion-plan.md | 149 ++++++++++++ docs/visualization/overview.md | 2 +- 4 files changed, 250 insertions(+), 129 deletions(-) create mode 100644 docs/motion-planning/3d-scene/visualize-a-motion-plan.md diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/motion-planning/3d-scene/_index.md index 41b862de16..e428b5c4f0 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/motion-planning/3d-scene/_index.md @@ -104,5 +104,6 @@ This is useful for comparing point clouds that should align (a registered scan a {{% card link="/motion-planning/3d-scene/verify-point-cloud-alignment/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} +{{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/edit-frames/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index 4b55e5f71f..82851b8e49 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -1,146 +1,117 @@ --- linkTitle: "Debug a motion plan" -title: "Visualize and debug a motion plan" +title: "Debug a motion plan" weight: 45 layout: "docs" type: "docs" -description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." +description: "Use the 3D scene to inspect the frame system and obstacle geometry behind a motion plan that failed or produced unexpected results." --- -The **3D SCENE** tab shows your configured frame system and live component poses. To -also see a motion plan, the trajectory the arm will follow, the goals it aims for, and -how that path relates to your obstacles, you publish the plan as **custom visuals** -through a world state store service. The scene then renders the path you can otherwise -only read as numbers. - -This page shows how to turn a plan into transforms the scene can draw, and how to -use the rendered path to debug a plan that failed or moved unexpectedly. - -## Why publish the plan as custom visuals - -A plan is a sequence of joint configurations. Read as numbers it tells you -little; rendered in the scene it tells you immediately whether the path clips an -obstacle, swings wide, or aims at a target outside the arm's reach. Publishing -the plan as world state store transforms puts the trajectory and goals in the -same 3D view as the frames and obstacle geometry the planner used, so you can see -the path and the world together. - -## Convert the trajectory into poses - -A plan's `Trajectory` is a sequence of joint configurations -(`FrameSystemInputs`), one per step. The scene places geometry by pose, so -convert each step into end-effector poses with the frame system. `ComputePoses` -takes a configuration and returns the pose of each frame: - -```go -for i, step := range plan.Trajectory() { - poses, err := step.ComputePoses(fs) - if err != nil { - return err - } - gripperPose := poses["my-gripper"].Pose() - // Place a marker for this step at gripperPose (next section). - _ = i - _ = gripperPose -} -``` - -Each `step` is the arm's configuration at that point in the trajectory, and -`poses["my-gripper"]` is where the gripper sits in that configuration. - -## Build transforms for the plan - -With a pose per step, build the visuals with the `draw` library: a marker per -trajectory step and a marker for each goal. Give each a stable UUID so the scene -can update them when you re-plan. - -```go -import ( - "github.com/viam-labs/motion-tools/draw" - "go.viam.com/rdk/spatialmath" -) - -func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { - id := fmt.Sprintf("step-%d", i) - // Build the sphere at the origin; WithPose below places it at the step pose. - sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 5, id) - if err != nil { - return nil, err - } - drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(stepColor)) - if err != nil { - return nil, err - } - // WithID derives a stable UUID from the string, so re-planning updates each - // marker in place instead of drawing a duplicate. - return drawn.Draw(id, draw.WithID(id), draw.WithPose(pose)) -} - -// goalMarker draws a destination pose. Unlike the trajectory poses, a goal pose is -// not a ComputePoses output: it is the destination you passed to the planner. -func goalMarker(i int, goalPose spatialmath.Pose) (*commonpb.Transform, error) { - id := fmt.Sprintf("goal-%d", i) - sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 8, id) - if err != nil { - return nil, err - } - drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(goalColor)) - if err != nil { - return nil, err - } - return drawn.Draw(id, draw.WithID(id), draw.WithPose(goalPose)) -} -``` - -The trajectory poses come from `ComputePoses`, but the goal pose is the destination -you passed to the planner. `goalMarker` draws it with a distinct color, so the target -stands out from the trajectory leading to it. - -## Serve the transforms to the scene - -Serve the transforms through a world state store service so the **3D SCENE** tab -renders them. Return your step and goal markers from the service's `ListUUIDs`, -`GetTransform`, and `StreamTransformChanges` methods, and the plan streams in alongside -the frames and obstacle geometry the scene already shows. For those methods, the -poll-and-update loop, and how a module pulls data from its dependencies, see -[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). - -## Diagnose a failed or surprising plan - -With the plan rendered, debugging becomes visual. Compare the trajectory against -the rest of the scene: - -- **Where does the path collide?** If a step marker passes through an obstacle - geometry, that is where the planner reports a collision. Check whether the - obstacle is real or an oversized geometry. -- **Does the goal fall outside the arm's reach?** If a goal marker sits far from - any reachable arm configuration, the goal is out of reach. Move the goal or - check the frame system. -- **Why the detour?** An unexpected route usually means an obstacle is forcing - the planner around it. Look for stray geometry between the start and goal. - -For checking the obstacle geometry itself, separate from the plan, see +When a motion plan fails with a collision error or takes a path that does not look +right, start in the **3D SCENE** tab. It shows the world the planner sees: configured +frame positions, collision geometries, and the arm's current pose. Most motion planning +failures come down to a few problems you can spot in 3D before writing any code. + +The checks below find those problems by eye. To render the plan's trajectory itself, the +path the arm takes from start to goal, see +[Visualize a motion plan](/motion-planning/3d-scene/visualize-a-motion-plan/). + +## Prerequisites + +- A machine with an arm or gantry configured and at least one frame defined. +- A motion plan that is failing or producing unexpected results. + +## Diagnose the problem + +### 1. Open the 3D SCENE tab + +Navigate to your machine in the [Viam app](https://app.viam.com) and +click the **3D SCENE** tab. The scene loads your current frame system +configuration. + +If your machine is online, the scene connects for live pose data. +If your machine is offline, the scene still renders the configured +frame positions. + +### 2. Check frame positions + +In the **World** panel in the upper-left, expand the tree and click +each component in turn. The Details panel on the right shows the +selected entity's **world position** and **world orientation**, plus +editable **local position** and **local orientation** relative to the +parent frame. Compare these values to your physical measurements. + +Three common mismatches to look for: + +- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. +- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. +- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. + +### 3. Check obstacle geometry + +Obstacles appear as translucent shapes in the scene and as child rows +under their parent frame in the **World** panel. Select each obstacle +from the tree to see its **geometry** type and **dimensions** in the +Details panel. + +Verify that: + +- Every physical obstacle in your workspace has a corresponding + geometry in the scene. +- Each geometry covers the actual physical object. If a box geometry + is too small, the planner will find paths that clip the real + obstacle. +- Geometries are positioned correctly. A table surface defined at + `z: 0` when the arm base is at `z: 500` will not protect against + table collisions. + +If obstacles are missing or misplaced, see [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). -## When to visualize versus inspect or verify +### 4. Look for impossible targets + +If the motion plan target is outside the arm's reach or inside an +obstacle, the planner cannot find a path. + +In the **World** panel, expand the arm and select its tip link or the +gripper frame (whichever is configured as the motion target). Read the +**world position** and compare it to the target pose your code +commanded. Then mentally place the target: is it inside an obstacle +geometry? Is it further than the arm can reach from its base? + +If the target is inside an obstacle geometry, either move the target +or adjust the obstacle definition. + +### 5. Check for self-collision geometry + +Some arm models include collision geometry for each link. If a motion +plan fails with a self-collision error, inspect the arm's link +geometries for overlap in the current configuration. + +If the arm renders without collision geometry, the feature may be +disabled: open **Settings → Scene → Arm Models** and verify that the +rendering mode includes colliders. -The 3D scene serves three distinct purposes, and it helps to keep them straight: +Self-collisions can happen when wrist joints are commanded to +positions that bring adjacent links too close together. If the +overlap is a modeling artifact rather than a real collision, allow +the frame pair with +[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). -- **Visualize a plan** (this page): publish the trajectory and goals as custom - visuals to see the path in context. -- **Inspect static frames and geometry**: use the stock scene to check frame - positions and obstacle coverage with no plan involved. -- **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is - reachable and a path exists before you visualize or run anything. +## Common causes of motion plan failures -Visualization shows you _what the path looks like_; static inspection shows you -_what the world looks like_; feasibility checking tells you _whether a plan -exists at all_. Reach for the one that matches the question you are asking. +| Symptom | Likely cause | What to check in 3D SCENE | +| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | +| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | +| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | +| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | +| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | +| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | ## What's next -- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): - the world state store service that serves these transforms. +- [Visualize a motion plan](/motion-planning/3d-scene/visualize-a-motion-plan/): + render the plan's trajectory and goals as custom visuals when a visual check is not enough. - [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): check obstacle geometry against the real workspace. - [How motion planning works](/motion-planning/how-planning-works/): diff --git a/docs/motion-planning/3d-scene/visualize-a-motion-plan.md b/docs/motion-planning/3d-scene/visualize-a-motion-plan.md new file mode 100644 index 0000000000..f4924ac7a3 --- /dev/null +++ b/docs/motion-planning/3d-scene/visualize-a-motion-plan.md @@ -0,0 +1,149 @@ +--- +linkTitle: "Visualize a motion plan" +title: "Visualize a motion plan" +weight: 50 +layout: "docs" +type: "docs" +description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." +--- + +The **3D SCENE** tab renders your configured frame system and live component poses, and the +[visual checks for a failing plan](/motion-planning/3d-scene/debug-motion-plan/) find most +problems by eye. When those checks do not reveal the problem, or you want to see the +trajectory itself, publish the plan as **custom visuals** through a world state store +service. The scene then renders the path you can otherwise only read as numbers. + +This page shows how to turn a plan into transforms the scene can draw, and how to +use the rendered path to debug a plan that failed or moved unexpectedly. + +## Why publish the plan as custom visuals + +A plan is a sequence of joint configurations. Read as numbers it tells you +little; rendered in the scene it tells you immediately whether the path clips an +obstacle, swings wide, or aims at a target outside the arm's reach. Publishing +the plan as world state store transforms puts the trajectory and goals in the +same 3D view as the frames and obstacle geometry the planner used, so you can see +the path and the world together. + +## Convert the trajectory into poses + +A plan's `Trajectory` is a sequence of joint configurations +(`FrameSystemInputs`), one per step. The scene places geometry by pose, so +convert each step into end-effector poses with the frame system. `ComputePoses` +takes a configuration and returns the pose of each frame: + +```go +for i, step := range plan.Trajectory() { + poses, err := step.ComputePoses(fs) + if err != nil { + return err + } + gripperPose := poses["my-gripper"].Pose() + // Place a marker for this step at gripperPose (next section). + _ = i + _ = gripperPose +} +``` + +Each `step` is the arm's configuration at that point in the trajectory, and +`poses["my-gripper"]` is where the gripper sits in that configuration. + +## Build transforms for the plan + +With a pose per step, build the visuals with the `draw` library: a marker per +trajectory step and a marker for each goal. Give each a stable UUID so the scene +can update them when you re-plan. + +```go +import ( + "github.com/viam-labs/motion-tools/draw" + "go.viam.com/rdk/spatialmath" +) + +func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { + id := fmt.Sprintf("step-%d", i) + // Build the sphere at the origin; WithPose below places it at the step pose. + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 5, id) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(stepColor)) + if err != nil { + return nil, err + } + // WithID derives a stable UUID from the string, so re-planning updates each + // marker in place instead of drawing a duplicate. + return drawn.Draw(id, draw.WithID(id), draw.WithPose(pose)) +} + +// goalMarker draws a destination pose. Unlike the trajectory poses, a goal pose is +// not a ComputePoses output: it is the destination you passed to the planner. +func goalMarker(i int, goalPose spatialmath.Pose) (*commonpb.Transform, error) { + id := fmt.Sprintf("goal-%d", i) + sphere, err := spatialmath.NewSphere(spatialmath.NewZeroPose(), 8, id) + if err != nil { + return nil, err + } + drawn, err := draw.NewDrawnGeometry(sphere, draw.WithGeometryColor(goalColor)) + if err != nil { + return nil, err + } + return drawn.Draw(id, draw.WithID(id), draw.WithPose(goalPose)) +} +``` + +The trajectory poses come from `ComputePoses`, but the goal pose is the destination +you passed to the planner. `goalMarker` draws it with a distinct color, so the target +stands out from the trajectory leading to it. + +## Serve the transforms to the scene + +Serve the transforms through a world state store service so the **3D SCENE** tab +renders them. Return your step and goal markers from the service's `ListUUIDs`, +`GetTransform`, and `StreamTransformChanges` methods, and the plan streams in alongside +the frames and obstacle geometry the scene already shows. For those methods, the +poll-and-update loop, and how a module pulls data from its dependencies, see +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## Diagnose a failed or surprising plan + +With the plan rendered, debugging becomes visual. Compare the trajectory against +the rest of the scene: + +- **Where does the path collide?** If a step marker passes through an obstacle + geometry, that is where the planner reports a collision. Check whether the + obstacle is real or an oversized geometry. +- **Does the goal fall outside the arm's reach?** If a goal marker sits far from + any reachable arm configuration, the goal is out of reach. Move the goal or + check the frame system. +- **Why the detour?** An unexpected route usually means an obstacle is forcing + the planner around it. Look for stray geometry between the start and goal. + +For checking the obstacle geometry itself, separate from the plan, see +[Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). + +## When to visualize versus inspect or verify + +The 3D scene serves three distinct purposes, and it helps to keep them straight: + +- **Visualize a plan** (this page): publish the trajectory and goals as custom + visuals to see the path in context. +- **[Inspect static frames and geometry](/motion-planning/3d-scene/debug-motion-plan/)**: + use the stock scene to check frame positions and obstacle coverage with no plan involved. +- **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is + reachable and a path exists before you visualize or run anything. + +Visualization shows you _what the path looks like_; static inspection shows you +_what the world looks like_; feasibility checking tells you _whether a plan +exists at all_. Reach for the one that matches the question you are asking. + +## What's next + +- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): + the no-code visual checks to try first. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + the world state store service that serves these transforms. +- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): + check obstacle geometry against the real workspace. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index 53bc1dbd65..a85e2da758 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -29,7 +29,7 @@ point clouds, and custom visuals a module publishes at runtime. {{< cards >}} {{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} {{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} +{{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} {{< /cards >}} ## Viam Visualization From 6b46fe2258c033572b55880a8c4fa0b90b7c134d Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 08:56:36 -0600 Subject: [PATCH 33/46] Standardize visualization surface naming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Differentiate the three confusable 'apps': - the Viam app (app.viam.com, hosts the 3D SCENE tab) — no more 'Viam web app' - Viam Visualization / the standalone Viam visualizer — no more 'Viam Visualization app' - a custom app / custom user interface you build (was 'an app you build') Also capitalize the product name Viam Visualization consistently. --- docs/visualization/drawing-library.md | 12 +++++----- docs/visualization/overview.md | 22 +++++++++---------- .../publish-visuals-from-a-module.md | 4 ++-- docs/visualization/reference/_index.md | 2 +- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index 83736a993f..c803c01bd1 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -1,13 +1,13 @@ --- linkTitle: "Drawing library" -title: "The drawing library and Viam visualization" +title: "The drawing library and Viam Visualization" weight: 30 layout: "docs" type: "docs" -description: "Use the draw library to build visuals, and run the standalone Viam Visualization app to preview spatial data from a Go client." +description: "Use the draw library to build visuals, and run the standalone Viam visualizer to preview spatial data from a Go client." --- -Viam Visualization is a standalone 3D app you run yourself. Unlike the **3D +Viam Visualization is a standalone 3D visualizer you run yourself. Unlike the **3D SCENE** tab in the Viam app, which renders a configured machine's frames, geometry, point clouds, and published visuals, Viam Visualization is a separate tool for monitoring, testing, and debugging spatial data: you start it locally and push @@ -16,7 +16,7 @@ It shares the same `draw` library used to build world state store transforms, so the visuals you construct are the same either way. This page covers what the drawing library is, the primitives it exposes, and how -to run the app and drive it from a Go client. +to run the visualizer and drive it from a Go client. ## Viam Visualization versus the 3D scene tab @@ -25,7 +25,7 @@ The two render 3D visuals, but they are different tools for different moments: - The **3D SCENE tab** lives in the Viam app and renders a configured machine: its frames, geometry, point clouds, and the custom visuals published to its world state store service. It is the in-app view of a machine. -- **Viam Visualization** is a standalone app you run on your own machine and push +- **Viam Visualization** is a standalone visualizer you run on your own machine and push to from a client. It is for previewing and debugging spatial data while you develop, without deploying a module or opening the Viam app. @@ -72,7 +72,7 @@ shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) Because the library owns the proto details, you work in terms of shapes and styles, not field-by-field struct assembly. -## Run the app and push from a Go client +## Run the visualizer and push from a Go client Viam Visualization runs locally and renders in your browser. From the [motion-tools repository](https://github.com/viam-labs/motion-tools), run `make setup` diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index a85e2da758..1f6387785e 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -4,16 +4,16 @@ title: "Visualization" weight: 1 layout: "docs" type: "docs" -description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam Visualization app, your own apps, and time-series dashboards." +description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam visualizer, custom apps you build, and time-series dashboards." --- There are several approaches to visualizing information in Viam. For things like motion, frames, robot state, collisions, and live perception data, you can use the 3D scene view in -the Viam web app or the standalone Viam Visualization app. You can also read this data from -components with a Viam SDK and present it in an app you build, which runs in a browser, on a -phone, or on a server. For time series data, services and modules push readings to the -cloud, where the Viam web app's Teleop workspaces and dashboards show live information across -a machine or fleet. +the Viam app, or Viam Visualization, a standalone visualizer you run yourself. You can also +read this data from components with a Viam SDK and present it in a custom user interface you +build, which runs in a browser, on a phone, or on a server. For time series data, services +and modules push readings to the cloud, where the Viam app's Teleop workspaces and dashboards +show live information across a machine or fleet. ## The 3D scene @@ -34,9 +34,9 @@ point clouds, and custom visuals a module publishes at runtime. ## Viam Visualization -Viam Visualization is a standalone 3D app you run yourself to preview and debug spatial -data from a Go client. It shares the same `draw` library as the in-app 3D scene, so the -visuals you build work either way. +Viam Visualization is a standalone 3D visualizer you run yourself to preview and debug +spatial data from a Go client. It shares the same `draw` library as the in-app 3D scene, so +the visuals you build work either way. - Use it while developing, to preview spatial data such as a point cloud, detections, or a planned path straight from a script or test. @@ -47,9 +47,9 @@ visuals you build work either way. {{% card link="/visualization/drawing-library/" noimage="true" %}} {{< /cards >}} -## Viam apps +## Custom apps -A Viam app uses a Viam SDK to read your machine's data and present it however you design. +A custom app uses a Viam SDK to read your machine's data and present it however you design. It runs outside the machine, in a browser, on a phone, or on a server, so you can build a custom dashboard, an operator console, or a fleet view. diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index c57e58baf5..84df27cbd7 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -206,7 +206,7 @@ func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstate - [Visuals and collisions](/visualization/visuals-and-collisions/): what a transform contains, and which geometry the planner collision-checks. -- [The drawing library and Viam visualization](/visualization/drawing-library/): - the `draw` primitives and the standalone visualizer app. +- [The drawing library and Viam Visualization](/visualization/drawing-library/): + the `draw` primitives and the standalone visualizer. - [Frame system](/motion-planning/frame-system/): position the transforms you publish. diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md index 6402e93f34..edefb72b2b 100644 --- a/docs/visualization/reference/_index.md +++ b/docs/visualization/reference/_index.md @@ -5,7 +5,7 @@ weight: 90 layout: "docs" type: "docs" no_list: true -description: "Reference material for the types and services behind Viam visualization." +description: "Reference material for the types and services behind the 3D scene and custom visuals." --- Reference pages for the types and services that back the 3D scene and custom visuals. From 65c02c48f92d85d845e8ff515bcfe0440e5df9f3 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 09:17:05 -0600 Subject: [PATCH 34/46] Rearrange 3D-scene pages into the visualization section per target IA Moves (with aliases from old URLs, cross-links + section cards updated): - calibrate-frame-offsets -> visualization/3d-scene-tools/measuring-between-frames (renamed) - edit-frames -> visualization/3d-scene-tools/editing-frames-visually (renamed) - verify-point-cloud-alignment -> visualization/perception/ Surface the World State Store Service reference under Visualization > Reference. Content split of motion-planning/3d-scene/_index.md is deferred. --- docs/motion-planning/3d-scene/_index.md | 5 +---- .../motion-planning/frame-system/camera-calibration.md | 2 +- docs/visualization/3d-scene-tools/_index.md | 2 ++ .../3d-scene-tools/editing-frames-visually.md} | 8 +++++--- .../3d-scene-tools/measuring-between-frames.md} | 10 ++++++---- docs/visualization/perception/_index.md | 1 + docs/visualization/perception/point-clouds.md | 2 +- .../perception}/verify-point-cloud-alignment.md | 3 ++- docs/visualization/reference/_index.md | 1 + 9 files changed, 20 insertions(+), 14 deletions(-) rename docs/{motion-planning/3d-scene/edit-frames.md => visualization/3d-scene-tools/editing-frames-visually.md} (96%) rename docs/{motion-planning/3d-scene/calibrate-frame-offsets.md => visualization/3d-scene-tools/measuring-between-frames.md} (94%) rename docs/{motion-planning/3d-scene => visualization/perception}/verify-point-cloud-alignment.md (96%) diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/motion-planning/3d-scene/_index.md index e428b5c4f0..3cca6e4a55 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/motion-planning/3d-scene/_index.md @@ -40,7 +40,7 @@ Entities that can be removed (for example, dropped PCD files) also show a **Remo **Dashboard toolbar** (top-center): Visible buttons, left to right: - **Orthographic / Perspective** toggle — switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. -- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Edit frames visually](/motion-planning/3d-scene/edit-frames/). +- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene-tools/editing-frames-visually/). - **Measurement** (ruler icon) — activate to measure distance between two points you pick in the viewport. Click the icon again to clear. - **Measurement settings** (sliders icon next to the ruler) — toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. - **Logs** — shows a count badge for errors/warnings from the scene renderer. @@ -100,10 +100,7 @@ This is useful for comparing point clouds that should align (a registered scan a ## How-to guides {{< cards >}} -{{% card link="/motion-planning/3d-scene/calibrate-frame-offsets/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/verify-point-cloud-alignment/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/edit-frames/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/frame-system/camera-calibration.md b/docs/motion-planning/frame-system/camera-calibration.md index 23e741ab1a..145a10e290 100644 --- a/docs/motion-planning/frame-system/camera-calibration.md +++ b/docs/motion-planning/frame-system/camera-calibration.md @@ -236,7 +236,7 @@ working distance of 500-1000 mm, your calibration is good. For a visual sanity check, open the [3D SCENE tab](/motion-planning/3d-scene/). The camera frame should sit in the correct position and orientation relative to the arm, and any visible obstacles should appear in plausible locations. -See [Calibrate frame offsets](/motion-planning/3d-scene/calibrate-frame-offsets/) +See [Measuring between frames](/visualization/3d-scene-tools/measuring-between-frames/) for the full workflow. ## Troubleshooting diff --git a/docs/visualization/3d-scene-tools/_index.md b/docs/visualization/3d-scene-tools/_index.md index cc62965a85..914431250c 100644 --- a/docs/visualization/3d-scene-tools/_index.md +++ b/docs/visualization/3d-scene-tools/_index.md @@ -12,5 +12,7 @@ The 3D scene includes tools for reading live data and adjusting your frame syste leaving the view. {{< cards >}} +{{% card link="/visualization/3d-scene-tools/measuring-between-frames/" noimage="true" %}} +{{% card link="/visualization/3d-scene-tools/editing-frames-visually/" noimage="true" %}} {{% card link="/visualization/3d-scene-tools/3d-scene-widgets/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/3d-scene/edit-frames.md b/docs/visualization/3d-scene-tools/editing-frames-visually.md similarity index 96% rename from docs/motion-planning/3d-scene/edit-frames.md rename to docs/visualization/3d-scene-tools/editing-frames-visually.md index 3f3c4bbb5e..e060939a6b 100644 --- a/docs/motion-planning/3d-scene/edit-frames.md +++ b/docs/visualization/3d-scene-tools/editing-frames-visually.md @@ -1,10 +1,12 @@ --- -linkTitle: "Edit frames visually" -title: "Edit frames visually" -weight: 50 +linkTitle: "Editing frames visually" +title: "Editing frames visually" +weight: 20 layout: "docs" type: "docs" description: "Add, edit, and attach geometry to frames directly in the 3D scene instead of editing JSON configuration." +aliases: + - /motion-planning/3d-scene/edit-frames/ --- The **3D SCENE** tab can serve as a configuration editor: you can add, move, re-parent, and reshape frames without writing JSON. diff --git a/docs/motion-planning/3d-scene/calibrate-frame-offsets.md b/docs/visualization/3d-scene-tools/measuring-between-frames.md similarity index 94% rename from docs/motion-planning/3d-scene/calibrate-frame-offsets.md rename to docs/visualization/3d-scene-tools/measuring-between-frames.md index 383100f136..91cde182cc 100644 --- a/docs/motion-planning/3d-scene/calibrate-frame-offsets.md +++ b/docs/visualization/3d-scene-tools/measuring-between-frames.md @@ -1,10 +1,12 @@ --- -linkTitle: "Calibrate frame offsets" -title: "Calibrate frame offsets" -weight: 20 +linkTitle: "Measuring between frames" +title: "Measuring between frames" +weight: 10 layout: "docs" type: "docs" description: "Verify and adjust the spatial relationship between components using the 3D scene and measurement tool." +aliases: + - /motion-planning/3d-scene/calibrate-frame-offsets/ --- When you configure a camera on an arm, or a sensor on a base, the frame system needs the exact translation and orientation between the two components. A 15 mm error in a camera offset places a detected object 15 mm off; the arm then reaches for the wrong spot, or the point cloud sits behind the table instead of on it. The **3D SCENE** tab lets you verify offsets visually and measure distances directly, so you can catch these errors before they produce bad motion. @@ -58,7 +60,7 @@ If the component is a depth camera, enable its point cloud at **Settings → Poi The point cloud should align with the physical objects in your workspace. If the point cloud appears shifted or rotated relative to where objects actually are, the camera's frame offset or orientation is wrong. -See [Verify point cloud alignment](/motion-planning/3d-scene/verify-point-cloud-alignment/) for more on working with point clouds. +See [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/) for more on working with point clouds. ## Iterative adjustment diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md index 47f88bfaf8..352794c80a 100644 --- a/docs/visualization/perception/_index.md +++ b/docs/visualization/perception/_index.md @@ -14,4 +14,5 @@ your machine senses in the same space as the components that sense it. {{< cards >}} {{% card link="/visualization/perception/point-clouds/" noimage="true" %}} {{% card link="/visualization/perception/cameras/" noimage="true" %}} +{{% card link="/visualization/perception/verify-point-cloud-alignment/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/perception/point-clouds.md b/docs/visualization/perception/point-clouds.md index 1a2e879cc6..51a03f44fd 100644 --- a/docs/visualization/perception/point-clouds.md +++ b/docs/visualization/perception/point-clouds.md @@ -40,4 +40,4 @@ To compare two point clouds that should align, such as a scan and a transformed them with **HoverLink**: select one point cloud, add a HoverLink relationship to the other, and hovering a point in one highlights the matching point in the other. For a full alignment walkthrough, see -[Verify point cloud alignment](/motion-planning/3d-scene/verify-point-cloud-alignment/). +[Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). diff --git a/docs/motion-planning/3d-scene/verify-point-cloud-alignment.md b/docs/visualization/perception/verify-point-cloud-alignment.md similarity index 96% rename from docs/motion-planning/3d-scene/verify-point-cloud-alignment.md rename to docs/visualization/perception/verify-point-cloud-alignment.md index 6cc06052a6..63dc6b9098 100644 --- a/docs/motion-planning/3d-scene/verify-point-cloud-alignment.md +++ b/docs/visualization/perception/verify-point-cloud-alignment.md @@ -7,6 +7,7 @@ type: "docs" description: "Display a depth camera's point cloud in the 3D scene to verify that the data aligns with your frame system and workspace geometry." aliases: - /motion-planning/3d-scene/inspect-point-clouds/ + - /motion-planning/3d-scene/verify-point-cloud-alignment/ --- Depth cameras produce point clouds: sets of 3D points that represent the surfaces the camera sees. The **3D SCENE** tab renders those points in your frame system, so you can check two things at once: the camera is producing usable data, and the data lines up with the rest of the workspace. @@ -46,7 +47,7 @@ The point cloud sits wherever the camera's frame puts it. If the frame configura - The floor appears at the expected height relative to the world frame. If the point cloud appears shifted, rotated, or in an unexpected location, the camera's frame offset or orientation is likely wrong. -See [Calibrate frame offsets](/motion-planning/3d-scene/calibrate-frame-offsets/). +See [Measuring between frames](/visualization/3d-scene-tools/measuring-between-frames/). ### 4. Check data quality diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md index edefb72b2b..19cf55cab9 100644 --- a/docs/visualization/reference/_index.md +++ b/docs/visualization/reference/_index.md @@ -11,5 +11,6 @@ description: "Reference material for the types and services behind the 3D scene Reference pages for the types and services that back the 3D scene and custom visuals. {{< cards >}} +{{% card link="/reference/apis/services/world-state-store/" noimage="true" %}} {{% card link="/visualization/reference/world-state/" noimage="true" %}} {{< /cards >}} From 51e398da6b3bb8994438b756e2540e23ccfb4a70 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 09:37:40 -0600 Subject: [PATCH 35/46] Move Verify obstacles into the Obstacles section Move motion-planning/3d-scene/set-up-obstacle-avoidance.md -> motion-planning/obstacles/verify-obstacles.md (alias from old URL), add it to the obstacles how-to cards, and repoint inbound links. --- docs/motion-planning/3d-scene/_index.md | 2 +- docs/motion-planning/3d-scene/debug-motion-plan.md | 4 ++-- docs/motion-planning/3d-scene/visualize-a-motion-plan.md | 4 ++-- .../obstacles/configure-workspace-obstacles.md | 4 ++-- docs/motion-planning/obstacles/overview.md | 1 + .../verify-obstacles.md} | 4 +++- 6 files changed, 11 insertions(+), 8 deletions(-) rename docs/motion-planning/{3d-scene/set-up-obstacle-avoidance.md => obstacles/verify-obstacles.md} (98%) diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/motion-planning/3d-scene/_index.md index 3cca6e4a55..de0813893e 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/motion-planning/3d-scene/_index.md @@ -100,7 +100,7 @@ This is useful for comparing point clouds that should align (a registered scan a ## How-to guides {{< cards >}} -{{% card link="/motion-planning/3d-scene/set-up-obstacle-avoidance/" noimage="true" %}} +{{% card link="/motion-planning/obstacles/verify-obstacles/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} {{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md index 82851b8e49..4b31d12d20 100644 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ b/docs/motion-planning/3d-scene/debug-motion-plan.md @@ -66,7 +66,7 @@ Verify that: table collisions. If obstacles are missing or misplaced, see -[Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). +[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). ### 4. Look for impossible targets @@ -112,7 +112,7 @@ the frame pair with - [Visualize a motion plan](/motion-planning/3d-scene/visualize-a-motion-plan/): render the plan's trajectory and goals as custom visuals when a visual check is not enough. -- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): check obstacle geometry against the real workspace. - [How motion planning works](/motion-planning/how-planning-works/): why a plan can be infeasible and what to adjust. diff --git a/docs/motion-planning/3d-scene/visualize-a-motion-plan.md b/docs/motion-planning/3d-scene/visualize-a-motion-plan.md index f4924ac7a3..1884a20489 100644 --- a/docs/motion-planning/3d-scene/visualize-a-motion-plan.md +++ b/docs/motion-planning/3d-scene/visualize-a-motion-plan.md @@ -120,7 +120,7 @@ the rest of the scene: the planner around it. Look for stray geometry between the start and goal. For checking the obstacle geometry itself, separate from the plan, see -[Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). +[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). ## When to visualize versus inspect or verify @@ -143,7 +143,7 @@ exists at all_. Reach for the one that matches the question you are asking. the no-code visual checks to try first. - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the world state store service that serves these transforms. -- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): check obstacle geometry against the real workspace. - [How motion planning works](/motion-planning/how-planning-works/): why a plan can be infeasible and what to adjust. diff --git a/docs/motion-planning/obstacles/configure-workspace-obstacles.md b/docs/motion-planning/obstacles/configure-workspace-obstacles.md index a1c26e48f7..d7779fa754 100644 --- a/docs/motion-planning/obstacles/configure-workspace-obstacles.md +++ b/docs/motion-planning/obstacles/configure-workspace-obstacles.md @@ -327,7 +327,7 @@ For objects the robot grasps and releases dynamically, see [Attach and detach ge 3. Orbit the scene. Each configured obstacle appears as a translucent shape at the position you configured. Each component shows up under the **World** panel; click a component to see its pose and geometry in the **Details** panel. 4. Check coverage against the physical workspace. If the arm can reach past an obstacle into the physical object, the geometry is too small. -For the full visualization and verification workflow, see [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/). +For the full visualization and verification workflow, see [Verify obstacles](/motion-planning/obstacles/verify-obstacles/). ## Try it @@ -377,6 +377,6 @@ Fix the geometry in the component's attributes and save again. ## What's next - [Plan collision-free paths](/motion-planning/obstacles/avoid-obstacles/): pass runtime obstacles through `WorldState` for objects that change between calls. -- [Verify obstacles](/motion-planning/3d-scene/set-up-obstacle-avoidance/): full verification workflow in the **3D SCENE** tab. +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): full verification workflow in the **3D SCENE** tab. - [Attach and detach geometries](/motion-planning/obstacles/attach-detach-geometries/): attach a grasped object to the gripper frame for the duration of a pickup. - [Allow specific frames to collide](/motion-planning/obstacles/allow-frame-collisions/): permit expected contact between frames. diff --git a/docs/motion-planning/obstacles/overview.md b/docs/motion-planning/obstacles/overview.md index 9e7caaec04..fae2322071 100644 --- a/docs/motion-planning/obstacles/overview.md +++ b/docs/motion-planning/obstacles/overview.md @@ -151,6 +151,7 @@ Practical consequences: {{< cards >}} {{% card link="/motion-planning/obstacles/configure-workspace-obstacles/" noimage="true" %}} +{{% card link="/motion-planning/obstacles/verify-obstacles/" noimage="true" %}} {{% card link="/motion-planning/obstacles/avoid-obstacles/" noimage="true" %}} {{% card link="/motion-planning/obstacles/attach-detach-geometries/" noimage="true" %}} {{% card link="/motion-planning/obstacles/allow-frame-collisions/" noimage="true" %}} diff --git a/docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md b/docs/motion-planning/obstacles/verify-obstacles.md similarity index 98% rename from docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md rename to docs/motion-planning/obstacles/verify-obstacles.md index f74a90004e..2230395865 100644 --- a/docs/motion-planning/3d-scene/set-up-obstacle-avoidance.md +++ b/docs/motion-planning/obstacles/verify-obstacles.md @@ -1,10 +1,12 @@ --- linkTitle: "Verify obstacles" title: "Verify obstacles" -weight: 40 +weight: 25 layout: "docs" type: "docs" description: "Visualize and adjust obstacle geometry so the motion planner routes around physical objects." +aliases: + - /motion-planning/3d-scene/set-up-obstacle-avoidance/ --- The motion planner avoids obstacles only if it knows about them, and it knows about them only as geometries you define: boxes, spheres, capsules, or cylinders positioned in the frame system. That definition is invisible in JSON. A box specified as `{x: 800, y: 1200, z: 20}` at some parent-relative translation either covers the table or it doesn't, and you can't tell which from the numbers. The **3D SCENE** tab lets you see what the planner sees, so you can check coverage before running a plan. From c528e9d91157a884663e4324ae459fc8e4d02ceb Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 09:43:19 -0600 Subject: [PATCH 36/46] Reorganize the plan-troubleshooting pages - Move Debug a motion plan and Visualize a motion plan out of 3d-scene/ to the motion-planning root - Merge 'Debug with the CLI' into 'Debug a motion plan' as a 'With the CLI' section alongside the '3D scene' checks (aliases from both old URLs) - Retitle verify-a-plan linkTitle to 'Verify a plan before running it' - Reweight the cluster in order: how-works(80), verify(82), debug(84), visualize(86) - Repoint inbound links; aliases cover old URLs --- docs/motion-planning/3d-scene/_index.md | 4 +- .../3d-scene/debug-motion-plan.md | 118 ---------- docs/motion-planning/_index.md | 2 +- docs/motion-planning/debug-motion-plan.md | 204 ++++++++++++++++++ docs/motion-planning/debug-motion-with-cli.md | 198 ----------------- docs/motion-planning/how-planning-works.md | 2 +- .../obstacles/allow-frame-collisions.md | 4 +- .../motion-planning/reference/cli-commands.md | 4 +- docs/motion-planning/verify-a-plan.md | 6 +- .../{3d-scene => }/visualize-a-motion-plan.md | 10 +- docs/visualization/overview.md | 2 +- 11 files changed, 222 insertions(+), 332 deletions(-) delete mode 100644 docs/motion-planning/3d-scene/debug-motion-plan.md create mode 100644 docs/motion-planning/debug-motion-plan.md delete mode 100644 docs/motion-planning/debug-motion-with-cli.md rename docs/motion-planning/{3d-scene => }/visualize-a-motion-plan.md (95%) diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/motion-planning/3d-scene/_index.md index de0813893e..4a9669278e 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/motion-planning/3d-scene/_index.md @@ -101,6 +101,6 @@ This is useful for comparing point clouds that should align (a registered scan a {{< cards >}} {{% card link="/motion-planning/obstacles/verify-obstacles/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/debug-motion-plan/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} +{{% card link="/motion-planning/debug-motion-plan/" noimage="true" %}} +{{% card link="/motion-planning/visualize-a-motion-plan/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/3d-scene/debug-motion-plan.md b/docs/motion-planning/3d-scene/debug-motion-plan.md deleted file mode 100644 index 4b31d12d20..0000000000 --- a/docs/motion-planning/3d-scene/debug-motion-plan.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -linkTitle: "Debug a motion plan" -title: "Debug a motion plan" -weight: 45 -layout: "docs" -type: "docs" -description: "Use the 3D scene to inspect the frame system and obstacle geometry behind a motion plan that failed or produced unexpected results." ---- - -When a motion plan fails with a collision error or takes a path that does not look -right, start in the **3D SCENE** tab. It shows the world the planner sees: configured -frame positions, collision geometries, and the arm's current pose. Most motion planning -failures come down to a few problems you can spot in 3D before writing any code. - -The checks below find those problems by eye. To render the plan's trajectory itself, the -path the arm takes from start to goal, see -[Visualize a motion plan](/motion-planning/3d-scene/visualize-a-motion-plan/). - -## Prerequisites - -- A machine with an arm or gantry configured and at least one frame defined. -- A motion plan that is failing or producing unexpected results. - -## Diagnose the problem - -### 1. Open the 3D SCENE tab - -Navigate to your machine in the [Viam app](https://app.viam.com) and -click the **3D SCENE** tab. The scene loads your current frame system -configuration. - -If your machine is online, the scene connects for live pose data. -If your machine is offline, the scene still renders the configured -frame positions. - -### 2. Check frame positions - -In the **World** panel in the upper-left, expand the tree and click -each component in turn. The Details panel on the right shows the -selected entity's **world position** and **world orientation**, plus -editable **local position** and **local orientation** relative to the -parent frame. Compare these values to your physical measurements. - -Three common mismatches to look for: - -- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. -- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. -- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. - -### 3. Check obstacle geometry - -Obstacles appear as translucent shapes in the scene and as child rows -under their parent frame in the **World** panel. Select each obstacle -from the tree to see its **geometry** type and **dimensions** in the -Details panel. - -Verify that: - -- Every physical obstacle in your workspace has a corresponding - geometry in the scene. -- Each geometry covers the actual physical object. If a box geometry - is too small, the planner will find paths that clip the real - obstacle. -- Geometries are positioned correctly. A table surface defined at - `z: 0` when the arm base is at `z: 500` will not protect against - table collisions. - -If obstacles are missing or misplaced, see -[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). - -### 4. Look for impossible targets - -If the motion plan target is outside the arm's reach or inside an -obstacle, the planner cannot find a path. - -In the **World** panel, expand the arm and select its tip link or the -gripper frame (whichever is configured as the motion target). Read the -**world position** and compare it to the target pose your code -commanded. Then mentally place the target: is it inside an obstacle -geometry? Is it further than the arm can reach from its base? - -If the target is inside an obstacle geometry, either move the target -or adjust the obstacle definition. - -### 5. Check for self-collision geometry - -Some arm models include collision geometry for each link. If a motion -plan fails with a self-collision error, inspect the arm's link -geometries for overlap in the current configuration. - -If the arm renders without collision geometry, the feature may be -disabled: open **Settings → Scene → Arm Models** and verify that the -rendering mode includes colliders. - -Self-collisions can happen when wrist joints are commanded to -positions that bring adjacent links too close together. If the -overlap is a modeling artifact rather than a real collision, allow -the frame pair with -[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). - -## Common causes of motion plan failures - -| Symptom | Likely cause | What to check in 3D SCENE | -| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | -| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | -| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | -| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | -| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | -| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | - -## What's next - -- [Visualize a motion plan](/motion-planning/3d-scene/visualize-a-motion-plan/): - render the plan's trajectory and goals as custom visuals when a visual check is not enough. -- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): - check obstacle geometry against the real workspace. -- [How motion planning works](/motion-planning/how-planning-works/): - why a plan can be infeasible and what to adjust. diff --git a/docs/motion-planning/_index.md b/docs/motion-planning/_index.md index fd457100fe..1f35e6c94f 100644 --- a/docs/motion-planning/_index.md +++ b/docs/motion-planning/_index.md @@ -89,7 +89,7 @@ returns a collision-free path from the current pose to your target. {{< cards >}} {{% card link="/motion-planning/move-gantry/" noimage="true" %}} {{% card link="/motion-planning/verify-a-plan/" noimage="true" %}} -{{% card link="/motion-planning/debug-motion-with-cli/" noimage="true" %}} +{{% card link="/motion-planning/debug-motion-plan/" noimage="true" %}} {{< /cards >}} ## Concept pages diff --git a/docs/motion-planning/debug-motion-plan.md b/docs/motion-planning/debug-motion-plan.md new file mode 100644 index 0000000000..edb0794525 --- /dev/null +++ b/docs/motion-planning/debug-motion-plan.md @@ -0,0 +1,204 @@ +--- +linkTitle: "Debug a motion plan" +title: "Debug a motion plan" +weight: 84 +layout: "docs" +type: "docs" +description: "Find why a motion plan failed or moved unexpectedly by inspecting frames, obstacles, and reach, either visually in the 3D scene or from the command line with the Viam CLI." +aliases: + - /motion-planning/3d-scene/debug-motion-plan/ + - /motion-planning/debug-motion-with-cli/ + - /motion-planning/motion-how-to/debug-motion-with-cli/ +--- + +When a motion plan fails with a collision error, or the arm ends up somewhere unexpected, +the first question is the same: is the frame system wrong, or is the plan wrong? You can +answer it two ways. If your machine has the **3D SCENE** tab, inspect the world the planner +sees by eye. From any shell, the Viam CLI reports the same frame poses and tests +reachability. This page covers both. + +To render the plan's trajectory itself, the path the arm takes from start to goal, see +[Visualize a motion plan](/motion-planning/visualize-a-motion-plan/). + +## Prerequisites + +- A machine with an arm or gantry configured and at least one frame defined. +- A motion plan that is failing or producing unexpected results. +- For the CLI checks: the Viam CLI installed and authenticated, and the `--part` identifier + for the machine part you want to inspect. + +## In the 3D scene + +Open the **3D SCENE** tab on your machine's page in the [Viam app](https://app.viam.com). It +loads your frame system configuration and, when the machine is online, connects for live +pose data. Work through the checks below; most planning failures show up in one of them. + +### Check frame positions + +In the **World** panel in the upper-left, expand the tree and click each component in turn. +The Details panel on the right shows the selected entity's **world position** and **world +orientation**, plus editable **local position** and **local orientation** relative to the +parent frame. Compare these values to your physical measurements. + +Three common mismatches to look for: + +- **Wrong location**: translation values in the frame configuration do not match the physical setup. Compare **local position** (mm) to your physical measurements. +- **Wrong orientation**: the arm base is rotated 90 degrees, or a camera points the wrong direction. Check **local orientation** in the Details panel. +- **Wrong parent**: the component is attached to the wrong parent, which places it in an unexpected part of the scene. Check the **parent frame** field in the Details panel. + +### Check obstacle geometry + +Obstacles appear as translucent shapes in the scene and as child rows under their parent +frame in the **World** panel. Select each obstacle from the tree to see its **geometry** type +and **dimensions** in the Details panel. + +Verify that: + +- Every physical obstacle in your workspace has a corresponding geometry in the scene. +- Each geometry covers the actual physical object. If a box geometry is too small, the + planner will find paths that clip the real obstacle. +- Geometries are positioned correctly. A table surface defined at `z: 0` when the arm base + is at `z: 500` will not protect against table collisions. + +If obstacles are missing or misplaced, see +[Verify obstacles](/motion-planning/obstacles/verify-obstacles/). + +### Look for impossible targets + +If the motion plan target is outside the arm's reach or inside an obstacle, the planner +cannot find a path. + +In the **World** panel, expand the arm and select its tip link or the gripper frame +(whichever is configured as the motion target). Read the **world position** and compare it to +the target pose your code commanded. Then place the target: is it inside an obstacle +geometry? Is it further than the arm can reach from its base? + +If the target is inside an obstacle geometry, either move the target or adjust the obstacle +definition. + +### Check for self-collision geometry + +Some arm models include collision geometry for each link. If a motion plan fails with a +self-collision error, inspect the arm's link geometries for overlap in the current +configuration. + +If the arm renders without collision geometry, the feature may be disabled: open **Settings → +Scene → Arm Models** and verify that the rendering mode includes colliders. + +Self-collisions can happen when wrist joints are commanded to positions that bring adjacent +links too close together. If the overlap is a modeling artifact rather than a real collision, +allow the frame pair with +[`CollisionSpecification`](/motion-planning/obstacles/allow-frame-collisions/). + +## With the CLI + +When the 3D scene tab is not available, or you want to script the checks, the Viam CLI +reports the same information from the shell. Three commands cover most cases: `print-config` +dumps the configured frame tree, `print-status` prints every frame's current world-frame +pose, and `set-pose` drives a component to a pose to test reachability. For the full flag +reference, see [Motion CLI commands](/motion-planning/reference/cli-commands/). + +### Frames are in the wrong place + +The arm reaches where it thinks is `x=300, y=200` and physically ends up elsewhere, or the +scene shows the gripper off to the side of the arm. The frame configuration does not match +what the machine does. Dump the configured frame tree: + +```sh +viam machines part motion print-config --part "my-machine-main" +``` + +Each frame part shows its name, parent, translation, and orientation. Compare against your +JSON configuration and physical measurements. Common issues: a **wrong parent** (a component +parented to the world frame instead of the arm, or vice versa), **wrong units** (centimeters +instead of millimeters), or a **missing frame** (a component with no entry, usually because +its frame configuration did not save). + +Then check the live world-frame poses: + +```sh +viam machines part motion print-status --part "my-machine-main" +``` + +`print-status` prints one line per frame part with its computed world-frame pose: + +```text + my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 + my-gripper : X: 0.00 Y: 0.00 Z: 110.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 90.00 +``` + +A pose that does not match where the component physically sits means the frame configuration +is wrong. Edit the configuration in the Viam app, then run both commands again to confirm the +change took effect. + +### Find where one component is + +To read a single component's pose without the rest of the frame tree: + +```sh +viam machines part motion get-pose \ + --part "my-machine-main" \ + --component "my-arm" +``` + +The output is the same single-line format as `print-status`, limited to the component. This +is useful for scripting: log it, compare between runs, or read the arm's position before and +after a physical move to check it matches what the arm reports. + +### Check whether a target pose is reachable + +Drive toward the target in small steps to find where planning fails. First read the current +pose with `get-pose`, then move a small distance: + +```sh +viam machines part motion set-pose \ + --part "my-machine-main" \ + --component "my-arm" \ + --x 100 +``` + +`set-pose` overrides only the fields you pass, so this moves the arm to `X=100` while keeping +the current Y, Z, and orientation. If this small step fails, the planner cannot reach any pose +near the current position, which usually means a configuration error rather than a +target-reachability issue. If it succeeds, increase the delta and work toward the pose you +want. The last successful pose and the first failing pose bracket the problem. If `set-pose` +fails at a pose that should be reachable, return to `print-config` and `print-status`: frame +configuration errors often show up as unexpected unreachability. + +### The arm moved to the wrong place after a motion call + +Your code called `motion.Move` or `arm.MoveToPosition` and the arm ended up somewhere other +than the commanded pose. Immediately after the motion, capture the actual final pose with +`get-pose` and compare it to the pose you commanded. Differences beyond the arm's positioning +tolerance suggest a frame configuration mismatch or a kinematics calibration problem. + +If `print-status` shows the arm where it physically is but that does not match the pose you +commanded, the pose may have been interpreted in an unexpected reference frame. Check the +`reference_frame` on the target `PoseInFrame`: the same `(x, y, z)` in the arm's frame and in +the world frame describes two different places. + +The CLI commands `print-status`, `get-pose`, and `set-pose` call `Motion.GetPose`, which is +deprecated in favor of `Robot.GetPose`; the commands and their output format are stable. +`set-pose` calls the motion service's `Move` and blocks until the motion finishes or fails, +returning a non-zero exit status with the error message on failure. + +## Common causes of motion plan failures + +| Symptom | Likely cause | What to check | +| ----------------------------- | --------------------------------------------- | -------------------------------------------------------------------------- | +| "no valid path found" | Target unreachable or blocked by obstacles | Is the target inside an obstacle? Is it within the arm's reach? | +| Collision error | Obstacle geometry intersects the planned path | Are obstacles positioned correctly? Are they the right size? | +| Path goes through the table | Table obstacle missing or too small | Is there a geometry covering the table surface? Does it extend far enough? | +| Arm takes an unexpected route | Obstacles force the planner to go around | Are there obstacles you did not intend to add? Is geometry oversized? | +| Self-collision error | Arm links collide with each other | Do link geometries overlap in the failing configuration? | + +## What's next + +- [Visualize a motion plan](/motion-planning/visualize-a-motion-plan/): + render the plan's trajectory and goals as custom visuals when a visual check is not enough. +- [Verify obstacles](/motion-planning/obstacles/verify-obstacles/): + check obstacle geometry against the real workspace. +- [Motion CLI commands](/motion-planning/reference/cli-commands/): + the full flag reference for the motion commands. +- [How motion planning works](/motion-planning/how-planning-works/): + why a plan can be infeasible and what to adjust. diff --git a/docs/motion-planning/debug-motion-with-cli.md b/docs/motion-planning/debug-motion-with-cli.md deleted file mode 100644 index a346c6e989..0000000000 --- a/docs/motion-planning/debug-motion-with-cli.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -linkTitle: "Debug with the CLI" -title: "Debug motion with the Viam CLI" -weight: 40 -layout: "docs" -type: "docs" -description: "Diagnose frame configuration and motion issues from the command line using the Viam CLI motion commands." -aliases: - - /motion-planning/motion-how-to/debug-motion-with-cli/ ---- - -A motion plan fails, or the arm ends up somewhere unexpected, and the first -question is always the same: is the frame system wrong, or is the plan wrong? -Answering that from application code requires writing instrumentation. The -Viam CLI answers it at the shell. Three commands cover most cases: -`print-config` dumps the configured frame tree, `print-status` prints every -frame's current world-frame pose, and `set-pose` drives a component to a pose -so you can isolate reachability. - -This guide walks through the most common debugging flows. For the -visual equivalent on machines where the **3D SCENE** tab is available, -see [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/). -For the dry command reference, see -[Motion CLI commands](/motion-planning/reference/cli-commands/). - -## Before you start - -- The Viam CLI installed and authenticated. -- A machine with configured components and at least one frame defined. -- The `--part` identifier for the machine part you want to inspect. - -## Symptom: frames are in the wrong place - -The arm reaches where it thinks is x=300, y=200, and it physically ends up -somewhere else. Or the **3D SCENE** tab shows the gripper dangling off to the -side of the arm. In both cases, the frame configuration does not match what -the machine actually does. Two CLI commands will tell you where the -disagreement is. - -### 1. Dump the configured frame system - -```sh -viam machines part motion print-config --part "my-machine-main" -``` - -Read through the output. Each frame part shows its name, parent, -translation, and orientation. Compare against your JSON configuration -and against physical measurements. - -Common issues you will catch at this step: - -- **Wrong parent**: a component you intended to attach to the arm is - parented to the world frame (or vice versa). -- **Wrong units**: translations expressed in centimeters rather than - millimeters. -- **Missing frames**: a component that should have a frame has no entry - in the output at all, which usually means its frame configuration did - not save. - -### 2. Check the live world-frame poses - -```sh -viam machines part motion print-status --part "my-machine-main" -``` - -`print-status` prints one line per frame part with its computed world-frame pose: - -```text - my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 - my-gripper : X: 0.00 Y: 0.00 Z: 110.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 90.00 -``` - -Compare these poses against where each component physically is. A -mismatch means the frame configuration is wrong. - -### 3. Adjust the config and re-check - -Edit the frame configuration in the Viam app. Run `print-config` and -`print-status` again to confirm the change took effect. - -## Symptom: I need to know where one component is right now - -You want a single component's pose without the rest of the frame tree. - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Output is the same single-line format as `print-status`, limited to -the specified component. - -This is useful for scripting: pipe the output to a log, compare between -runs, or read the arm's position before and after a physical move to -verify it matches what the arm reports. - -## Symptom: I am not sure if a target pose is reachable - -Write the target pose in world coordinates and try to drive there. - -### 1. Read the current pose - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Note the X, Y, Z values. - -### 2. Move a small distance first - -This small step isolates the problem. If it fails, the planner cannot reach -any pose near the current position, which usually means a configuration -error rather than a target-reachability issue. If it succeeds, the current -position is not the problem and you can work outward from there. - -```sh -viam machines part motion set-pose \ - --part "my-machine-main" \ - --component "my-arm" \ - --x 100 -``` - -`set-pose` overrides only the fields you pass. In this example, it -moves the arm to X=100 while keeping the current Y, Z, and orientation. - -### 3. Move toward the real target incrementally - -If the small step succeeds, increase the delta. Work toward the pose -you ultimately want. If an intermediate step fails, the last successful -pose and the first failing pose bracket the problem: the planner can -reach the first but not the second. - -### 4. Compare against the frame system - -If `set-pose` fails at a pose that should be reachable, return to -`print-config` and `print-status` to check whether the planner's view -of the arm's location matches reality. Frame configuration errors -often manifest as unexpected unreachability. - -## Symptom: the arm moved to the wrong place after a motion call - -Your code called `motion.Move` or `arm.MoveToPosition` and the arm -ended up somewhere other than the commanded pose. - -### 1. Capture the actual final pose - -Immediately after the motion completes: - -```sh -viam machines part motion get-pose \ - --part "my-machine-main" \ - --component "my-arm" -``` - -Compare this to the pose you commanded. Differences that exceed the -arm's positioning tolerance suggest either a frame configuration -mismatch or a kinematics calibration problem. - -### 2. Cross-check with arm joint values - -The arm's own `GetJointPositions` reports joint angles. For a built-in -arm model, you can verify those match the expected kinematic solution -for the commanded pose. A large offset between forward-kinematic -prediction and reality points to incorrect kinematic parameters. - -### 3. If frames look correct, look at the target - -If `print-status` shows the arm where it physically is, but that does -not match the pose you commanded, the pose you commanded may have been -interpreted in an unexpected reference frame. Double-check the -`reference_frame` on the target `PoseInFrame`: the same `(x, y, z)` in -the arm's frame and in the world frame describes two different places. - -## Limitations - -- **Internal RPC is deprecated but the CLI output is not.** - `print-status`, `get-pose`, and `set-pose` call `Motion.GetPose` under the - hood, which is deprecated in favor of `Robot.GetPose`. The CLI output - format and the commands themselves are stable; only the backing call is - scheduled to change. -- `set-pose` calls the motion service's `Move`, which blocks until the - motion completes or fails. Plan failures surface as the CLI returning - a non-zero exit status with an error message. Collect that message as - the starting point for the rest of your debugging. - -## What's next - -- [Motion CLI commands](/motion-planning/reference/cli-commands/): the - full flag reference. -- [Frame system](/motion-planning/frame-system/): the concept these - commands inspect. -- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): - the 3D-scene equivalent when your machine has the visualization tab. -- [How motion planning works](/motion-planning/how-planning-works/): - understand when the planner fails and what the failure means. diff --git a/docs/motion-planning/how-planning-works.md b/docs/motion-planning/how-planning-works.md index 377d9db7b3..81f8c782ec 100644 --- a/docs/motion-planning/how-planning-works.md +++ b/docs/motion-planning/how-planning-works.md @@ -1,7 +1,7 @@ --- linkTitle: "How planning works" title: "How motion planning works" -weight: 90 +weight: 80 layout: "docs" type: "docs" description: "The algorithm Viam uses to plan robot arm motion, why it works for arms, and what to try when planning fails." diff --git a/docs/motion-planning/obstacles/allow-frame-collisions.md b/docs/motion-planning/obstacles/allow-frame-collisions.md index e272f5f6d7..3dfca37669 100644 --- a/docs/motion-planning/obstacles/allow-frame-collisions.md +++ b/docs/motion-planning/obstacles/allow-frame-collisions.md @@ -244,7 +244,7 @@ silencing them can mask real hardware problems. reject those. 3. Visually confirm the motion in the **3D SCENE** tab before running on hardware. See - [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/). + [Debug a motion plan](/motion-planning/debug-motion-plan/). ## Troubleshooting @@ -288,5 +288,5 @@ geometry is usually the right answer. other three constraint types (Linear, Orientation, Pseudolinear). - [Define obstacles](/motion-planning/obstacles/): the obstacle geometry that the planner checks against. -- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): +- [Debug a motion plan](/motion-planning/debug-motion-plan/): visualize frame overlaps in the **3D SCENE** tab. diff --git a/docs/motion-planning/reference/cli-commands.md b/docs/motion-planning/reference/cli-commands.md index 7660a63b83..7169355a8f 100644 --- a/docs/motion-planning/reference/cli-commands.md +++ b/docs/motion-planning/reference/cli-commands.md @@ -150,11 +150,11 @@ position. - **Quick reachability test**: run `get-pose` to read the current pose, then run `set-pose` with a small offset to confirm the motion service can drive the arm. Start with small offsets; large ones risk collisions. For a step-by-step debugging workflow using these commands, see -[Debug motion with the CLI](/motion-planning/debug-motion-with-cli/). +[Debug motion with the CLI](/motion-planning/debug-motion-plan/). ## What's next -- [Debug motion with the CLI](/motion-planning/debug-motion-with-cli/): +- [Debug motion with the CLI](/motion-planning/debug-motion-plan/): step-by-step debugging using these commands. - [Frame system](/motion-planning/frame-system/): the concept the CLI inspects. diff --git a/docs/motion-planning/verify-a-plan.md b/docs/motion-planning/verify-a-plan.md index ded000a45b..967999893b 100644 --- a/docs/motion-planning/verify-a-plan.md +++ b/docs/motion-planning/verify-a-plan.md @@ -1,7 +1,7 @@ --- -linkTitle: "Verify a plan" +linkTitle: "Verify a plan before running it" title: "Verify a motion plan without executing it" -weight: 60 +weight: 82 layout: "docs" type: "docs" description: "Use armplanning.PlanMotion to compute and inspect a trajectory before the arm moves, and learn how to get the same plan over the motion service API." @@ -184,5 +184,5 @@ and the motion service is all you have. plan one trajectory through an ordered list of goals. - [How motion planning works](/motion-planning/how-planning-works/): why a plan can be infeasible and what to adjust. -- [Debug motion with the CLI](/motion-planning/debug-motion-with-cli/): +- [Debug motion with the CLI](/motion-planning/debug-motion-plan/): inspect frames and poses from the command line. diff --git a/docs/motion-planning/3d-scene/visualize-a-motion-plan.md b/docs/motion-planning/visualize-a-motion-plan.md similarity index 95% rename from docs/motion-planning/3d-scene/visualize-a-motion-plan.md rename to docs/motion-planning/visualize-a-motion-plan.md index 1884a20489..230f367f23 100644 --- a/docs/motion-planning/3d-scene/visualize-a-motion-plan.md +++ b/docs/motion-planning/visualize-a-motion-plan.md @@ -1,14 +1,16 @@ --- linkTitle: "Visualize a motion plan" title: "Visualize a motion plan" -weight: 50 +weight: 86 layout: "docs" type: "docs" description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." +aliases: + - /motion-planning/3d-scene/visualize-a-motion-plan/ --- The **3D SCENE** tab renders your configured frame system and live component poses, and the -[visual checks for a failing plan](/motion-planning/3d-scene/debug-motion-plan/) find most +[visual checks for a failing plan](/motion-planning/debug-motion-plan/) find most problems by eye. When those checks do not reveal the problem, or you want to see the trajectory itself, publish the plan as **custom visuals** through a world state store service. The scene then renders the path you can otherwise only read as numbers. @@ -128,7 +130,7 @@ The 3D scene serves three distinct purposes, and it helps to keep them straight: - **Visualize a plan** (this page): publish the trajectory and goals as custom visuals to see the path in context. -- **[Inspect static frames and geometry](/motion-planning/3d-scene/debug-motion-plan/)**: +- **[Inspect static frames and geometry](/motion-planning/debug-motion-plan/)**: use the stock scene to check frame positions and obstacle coverage with no plan involved. - **Check feasibility**: use `armplanning.PlanMotion` to confirm a goal is reachable and a path exists before you visualize or run anything. @@ -139,7 +141,7 @@ exists at all_. Reach for the one that matches the question you are asking. ## What's next -- [Debug a motion plan](/motion-planning/3d-scene/debug-motion-plan/): +- [Debug a motion plan](/motion-planning/debug-motion-plan/): the no-code visual checks to try first. - [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the world state store service that serves these transforms. diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index 1f6387785e..5bf5214c2c 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -29,7 +29,7 @@ point clouds, and custom visuals a module publishes at runtime. {{< cards >}} {{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} {{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/visualize-a-motion-plan/" noimage="true" %}} +{{% card link="/motion-planning/visualize-a-motion-plan/" noimage="true" %}} {{< /cards >}} ## Viam Visualization From 504fe027d346c64e3accd1d4f461318b2a136700 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Thu, 2 Jul 2026 09:49:18 -0600 Subject: [PATCH 37/46] Create a 3D scene section in Visualization - Rename '3D Scene Tools' to a '3D scene' section (visualization/3d-scene/) and move Measuring between frames, Editing frames visually, 3D Scene Widgets into it - Make 'Visualizing with the 3D scene' the section landing (_index) - Move the old motion-planning '3D scene tab' reference in as 'The 3D scene interface', stripping the concept intro now on the landing and dropping its cross-section how-to cards - Aliases from all old URLs (including /motion-planning/3d-scene/); repoint links --- docs/motion-planning/_index.md | 2 +- .../frame-system/camera-calibration.md | 4 +-- .../frame-system/end-effector-frames.md | 4 +-- docs/motion-planning/frame-system/overview.md | 4 +-- docs/visualization/3d-scene-tools/_index.md | 18 ------------- .../3d-scene-widgets.md | 4 ++- .../_index.md} | 15 +++++++---- .../editing-frames-visually.md | 3 ++- .../measuring-between-frames.md | 3 ++- .../3d-scene/the-3d-scene-interface.md} | 27 +++++++------------ .../verify-point-cloud-alignment.md | 2 +- 11 files changed, 34 insertions(+), 52 deletions(-) delete mode 100644 docs/visualization/3d-scene-tools/_index.md rename docs/visualization/{3d-scene-tools => 3d-scene}/3d-scene-widgets.md (95%) rename docs/visualization/{visualizing-with-the-3d-scene.md => 3d-scene/_index.md} (88%) rename docs/visualization/{3d-scene-tools => 3d-scene}/editing-frames-visually.md (98%) rename docs/visualization/{3d-scene-tools => 3d-scene}/measuring-between-frames.md (98%) rename docs/{motion-planning/3d-scene/_index.md => visualization/3d-scene/the-3d-scene-interface.md} (82%) diff --git a/docs/motion-planning/_index.md b/docs/motion-planning/_index.md index 1f35e6c94f..915fd0b515 100644 --- a/docs/motion-planning/_index.md +++ b/docs/motion-planning/_index.md @@ -80,7 +80,7 @@ returns a collision-free path from the current pose to your target. {{< cards >}} {{% card link="/motion-planning/frame-system/" noimage="true" %}} {{% card link="/motion-planning/obstacles/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/" noimage="true" %}} +{{% card link="/visualization/3d-scene/" noimage="true" %}} {{% card link="/motion-planning/move-an-arm/" noimage="true" %}} {{< /cards >}} diff --git a/docs/motion-planning/frame-system/camera-calibration.md b/docs/motion-planning/frame-system/camera-calibration.md index 145a10e290..187ef75774 100644 --- a/docs/motion-planning/frame-system/camera-calibration.md +++ b/docs/motion-planning/frame-system/camera-calibration.md @@ -233,10 +233,10 @@ fmt.Printf(" z=%.1f mm\n", pt.Z) If the computed position is within 10-20 mm of the measured position at a working distance of 500-1000 mm, your calibration is good. -For a visual sanity check, open the [3D SCENE tab](/motion-planning/3d-scene/). +For a visual sanity check, open the [3D SCENE tab](/visualization/3d-scene/). The camera frame should sit in the correct position and orientation relative to the arm, and any visible obstacles should appear in plausible locations. -See [Measuring between frames](/visualization/3d-scene-tools/measuring-between-frames/) +See [Measuring between frames](/visualization/3d-scene/measuring-between-frames/) for the full workflow. ## Troubleshooting diff --git a/docs/motion-planning/frame-system/end-effector-frames.md b/docs/motion-planning/frame-system/end-effector-frames.md index ad2c73ca1f..4ce305ec07 100644 --- a/docs/motion-planning/frame-system/end-effector-frames.md +++ b/docs/motion-planning/frame-system/end-effector-frames.md @@ -258,7 +258,7 @@ different things, and only one affects a `Move` call. plan this motion, and they are gone when the call returns. The transform in the previous section is a `WorldState` transform, so it shapes the plan. - The world state store service is a separate service that holds transforms for - client-side visualization in the [3D scene](/motion-planning/3d-scene/). + client-side visualization in the [3D scene](/visualization/3d-scene/). To change where the arm goes, put the transform in the `WorldState` you pass to `Move`, as shown above. To render a custom visual that leaves planning unchanged, @@ -274,7 +274,7 @@ same axes drawn in the diagrams on this page. A frame you add in code with a `WorldState` transform lasts only for that `Move` call, so it stays out of the 3D scene by default. To draw a code-defined frame in -the [3D scene](/motion-planning/3d-scene/), publish it through a world state store +the [3D scene](/visualization/3d-scene/), publish it through a world state store service. A module that implements this service holds your transforms and streams them to the scene, which renders a frame with no geometry as a set of coordinate axes alongside the static frames. diff --git a/docs/motion-planning/frame-system/overview.md b/docs/motion-planning/frame-system/overview.md index 20fc3c9720..fb6a2bd9b4 100644 --- a/docs/motion-planning/frame-system/overview.md +++ b/docs/motion-planning/frame-system/overview.md @@ -195,7 +195,7 @@ A typical frame configuration looks like this: } ``` -After saving, the [3D SCENE tab](/motion-planning/3d-scene/) renders the +After saving, the [3D SCENE tab](/visualization/3d-scene/) renders the frame at its computed world pose so you can verify visually. For hardware-specific walkthroughs (table-mounted arm with gripper and camera, mobile manipulator, mobile base with sensors), see the cards at @@ -271,7 +271,7 @@ Common uses: ## Verify visually with the 3D SCENE tab -The [3D SCENE tab](/motion-planning/3d-scene/) renders your frame hierarchy in +The [3D SCENE tab](/visualization/3d-scene/) renders your frame hierarchy in 3D: every configured component appears at its computed world pose, with geometry and axes drawn. Open it after a configuration change to confirm the arm is mounted on the table, the camera is above the arm, and any obstacles diff --git a/docs/visualization/3d-scene-tools/_index.md b/docs/visualization/3d-scene-tools/_index.md deleted file mode 100644 index 914431250c..0000000000 --- a/docs/visualization/3d-scene-tools/_index.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -linkTitle: "3D Scene Tools" -title: "3D scene tools" -weight: 40 -layout: "docs" -type: "docs" -no_list: true -description: "Tools built into the 3D scene: overlay widgets for live data, measurement, and visual frame editing." ---- - -The 3D scene includes tools for reading live data and adjusting your frame system without -leaving the view. - -{{< cards >}} -{{% card link="/visualization/3d-scene-tools/measuring-between-frames/" noimage="true" %}} -{{% card link="/visualization/3d-scene-tools/editing-frames-visually/" noimage="true" %}} -{{% card link="/visualization/3d-scene-tools/3d-scene-widgets/" noimage="true" %}} -{{< /cards >}} diff --git a/docs/visualization/3d-scene-tools/3d-scene-widgets.md b/docs/visualization/3d-scene/3d-scene-widgets.md similarity index 95% rename from docs/visualization/3d-scene-tools/3d-scene-widgets.md rename to docs/visualization/3d-scene/3d-scene-widgets.md index 8553a3251e..f5f8969267 100644 --- a/docs/visualization/3d-scene-tools/3d-scene-widgets.md +++ b/docs/visualization/3d-scene/3d-scene-widgets.md @@ -1,10 +1,12 @@ --- linkTitle: "3D Scene Widgets" title: "3D scene widgets" -weight: 30 +weight: 40 layout: "docs" type: "docs" description: "Overlay live data on the 3D scene with widgets: an arm's joint positions and a camera's live feed." +aliases: + - /visualization/3d-scene-tools/3d-scene-widgets/ --- Widgets are floating panels you overlay on the 3D scene to read live data next to the 3D diff --git a/docs/visualization/visualizing-with-the-3d-scene.md b/docs/visualization/3d-scene/_index.md similarity index 88% rename from docs/visualization/visualizing-with-the-3d-scene.md rename to docs/visualization/3d-scene/_index.md index 1a3414930e..8d435c1583 100644 --- a/docs/visualization/visualizing-with-the-3d-scene.md +++ b/docs/visualization/3d-scene/_index.md @@ -1,10 +1,13 @@ --- -linkTitle: "Visualizing with the 3D scene" +linkTitle: "3D scene" title: "Visualizing with the 3D scene" weight: 5 layout: "docs" type: "docs" description: "What the 3D scene renders, where each element comes from, and how built-in configuration content differs from custom visuals a module publishes at runtime." +aliases: + - /visualization/visualizing-with-the-3d-scene/ + - /motion-planning/3d-scene/ --- The **3D SCENE** tab renders your machine in an interactive 3D view: the frames of every @@ -14,8 +17,8 @@ otherwise read as JSON numbers into a picture you can inspect, so you can confir sits where the arm expects it or watch a motion plan against the obstacles around it. This page describes what the scene shows, where each element comes from, and how the scene -stays current. For the tab's panels, navigation, and settings, see the -[3D scene tab reference](/motion-planning/3d-scene/). +stays current. For the tab's panels, navigation, and settings, see +[The 3D scene interface](/visualization/3d-scene/the-3d-scene-interface/). ## What the scene renders @@ -75,8 +78,10 @@ To publish your own custom visuals this way, see ## What's next {{< cards >}} +{{% card link="/visualization/3d-scene/the-3d-scene-interface/" noimage="true" %}} +{{% card link="/visualization/3d-scene/measuring-between-frames/" noimage="true" %}} +{{% card link="/visualization/3d-scene/editing-frames-visually/" noimage="true" %}} +{{% card link="/visualization/3d-scene/3d-scene-widgets/" noimage="true" %}} {{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} {{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} -{{% card link="/visualization/drawing-library/" noimage="true" %}} -{{% card link="/motion-planning/3d-scene/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/3d-scene-tools/editing-frames-visually.md b/docs/visualization/3d-scene/editing-frames-visually.md similarity index 98% rename from docs/visualization/3d-scene-tools/editing-frames-visually.md rename to docs/visualization/3d-scene/editing-frames-visually.md index e060939a6b..ce0c1a5ffd 100644 --- a/docs/visualization/3d-scene-tools/editing-frames-visually.md +++ b/docs/visualization/3d-scene/editing-frames-visually.md @@ -1,12 +1,13 @@ --- linkTitle: "Editing frames visually" title: "Editing frames visually" -weight: 20 +weight: 30 layout: "docs" type: "docs" description: "Add, edit, and attach geometry to frames directly in the 3D scene instead of editing JSON configuration." aliases: - /motion-planning/3d-scene/edit-frames/ + - /visualization/3d-scene-tools/editing-frames-visually/ --- The **3D SCENE** tab can serve as a configuration editor: you can add, move, re-parent, and reshape frames without writing JSON. diff --git a/docs/visualization/3d-scene-tools/measuring-between-frames.md b/docs/visualization/3d-scene/measuring-between-frames.md similarity index 98% rename from docs/visualization/3d-scene-tools/measuring-between-frames.md rename to docs/visualization/3d-scene/measuring-between-frames.md index 91cde182cc..6117b42171 100644 --- a/docs/visualization/3d-scene-tools/measuring-between-frames.md +++ b/docs/visualization/3d-scene/measuring-between-frames.md @@ -1,12 +1,13 @@ --- linkTitle: "Measuring between frames" title: "Measuring between frames" -weight: 10 +weight: 20 layout: "docs" type: "docs" description: "Verify and adjust the spatial relationship between components using the 3D scene and measurement tool." aliases: - /motion-planning/3d-scene/calibrate-frame-offsets/ + - /visualization/3d-scene-tools/measuring-between-frames/ --- When you configure a camera on an arm, or a sensor on a base, the frame system needs the exact translation and orientation between the two components. A 15 mm error in a camera offset places a detected object 15 mm off; the arm then reaches for the wrong spot, or the point cloud sits behind the table instead of on it. The **3D SCENE** tab lets you verify offsets visually and measure distances directly, so you can catch these errors before they produce bad motion. diff --git a/docs/motion-planning/3d-scene/_index.md b/docs/visualization/3d-scene/the-3d-scene-interface.md similarity index 82% rename from docs/motion-planning/3d-scene/_index.md rename to docs/visualization/3d-scene/the-3d-scene-interface.md index 4a9669278e..f3ebe21cea 100644 --- a/docs/motion-planning/3d-scene/_index.md +++ b/docs/visualization/3d-scene/the-3d-scene-interface.md @@ -1,17 +1,16 @@ --- -linkTitle: "3D scene tab" -title: "3D scene tab" -weight: 30 +linkTitle: "The 3D scene interface" +title: "The 3D scene interface" +weight: 10 layout: "docs" type: "docs" -no_list: true -description: "Visualize your machine's frame system, geometries, and point clouds in an interactive 3D view." +description: "The panels, toolbar, navigation, and settings of the 3D SCENE tab in the Viam app." --- -The **3D SCENE** tab renders your machine's frame system as an interactive 3D visualization on your machine's page in the [Viam app](https://app.viam.com). -Frame configuration is otherwise invisible: a JSON translation of `{x: 50, y: 0, z: 110}` tells you nothing about whether the gripper actually sits where the arm needs it. The **3D SCENE** tab makes that spatial relationship visible so you can catch misconfigurations before a motion plan fails. - -The tab reads your machine's configuration and, when the machine is online, connects for live data. Each component's frame appears as a set of coordinate axes positioned by its translation and orientation relative to its parent frame. Attached geometries render as translucent shapes, and point clouds from depth cameras render as colored point sets. +The **3D SCENE** tab on your machine's page in the [Viam app](https://app.viam.com) is where +you inspect the 3D scene. This page is a reference to its panels, toolbar, navigation, and +settings. For what the scene renders and where each element comes from, see +[Visualizing with the 3D scene](/visualization/3d-scene/). ## The interface @@ -40,7 +39,7 @@ Entities that can be removed (for example, dropped PCD files) also show a **Remo **Dashboard toolbar** (top-center): Visible buttons, left to right: - **Orthographic / Perspective** toggle — switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. -- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene-tools/editing-frames-visually/). +- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene/editing-frames-visually/). - **Measurement** (ruler icon) — activate to measure distance between two points you pick in the viewport. Click the icon again to clear. - **Measurement settings** (sliders icon next to the ruler) — toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. - **Logs** — shows a count badge for errors/warnings from the scene renderer. @@ -96,11 +95,3 @@ To add a HoverLink: After the link is added, hovering a point in the source entity highlights the matching point in the target entity (and updates the hover tooltip with both points' positions). Existing links appear under **Relationships** in the Details panel and have per-link remove buttons. This is useful for comparing point clouds that should align (a registered scan against a transformed version, ground-truth points against predicted points) without flipping back and forth between separate views. - -## How-to guides - -{{< cards >}} -{{% card link="/motion-planning/obstacles/verify-obstacles/" noimage="true" %}} -{{% card link="/motion-planning/debug-motion-plan/" noimage="true" %}} -{{% card link="/motion-planning/visualize-a-motion-plan/" noimage="true" %}} -{{< /cards >}} diff --git a/docs/visualization/perception/verify-point-cloud-alignment.md b/docs/visualization/perception/verify-point-cloud-alignment.md index 63dc6b9098..8c6647edae 100644 --- a/docs/visualization/perception/verify-point-cloud-alignment.md +++ b/docs/visualization/perception/verify-point-cloud-alignment.md @@ -47,7 +47,7 @@ The point cloud sits wherever the camera's frame puts it. If the frame configura - The floor appears at the expected height relative to the world frame. If the point cloud appears shifted, rotated, or in an unexpected location, the camera's frame offset or orientation is likely wrong. -See [Measuring between frames](/visualization/3d-scene-tools/measuring-between-frames/). +See [Measuring between frames](/visualization/3d-scene/measuring-between-frames/). ### 4. Check data quality From 661e68fdb8ec84adbb7f244a2314284a9591ee15 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 7 Jul 2026 10:04:57 -0600 Subject: [PATCH 38/46] Review fixes: drop em-dashes, personified planner, and banned 'lands' - Convert em-dash-as-punctuation to colons in the 3D scene interface and editing-frames pages (carried in from the old motion-planning pages) - Reword 'the planner knows about them' to avoid personification - 'lands in the wrong place' -> 'appears in the wrong place' --- docs/motion-planning/obstacles/verify-obstacles.md | 2 +- .../3d-scene/editing-frames-visually.md | 6 +++--- .../visualization/3d-scene/the-3d-scene-interface.md | 12 ++++++------ docs/visualization/perception/cameras.md | 2 +- docs/visualization/perception/point-clouds.md | 2 +- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/motion-planning/obstacles/verify-obstacles.md b/docs/motion-planning/obstacles/verify-obstacles.md index 2230395865..a821a42230 100644 --- a/docs/motion-planning/obstacles/verify-obstacles.md +++ b/docs/motion-planning/obstacles/verify-obstacles.md @@ -9,7 +9,7 @@ aliases: - /motion-planning/3d-scene/set-up-obstacle-avoidance/ --- -The motion planner avoids obstacles only if it knows about them, and it knows about them only as geometries you define: boxes, spheres, capsules, or cylinders positioned in the frame system. That definition is invisible in JSON. A box specified as `{x: 800, y: 1200, z: 20}` at some parent-relative translation either covers the table or it doesn't, and you can't tell which from the numbers. The **3D SCENE** tab lets you see what the planner sees, so you can check coverage before running a plan. +The motion planner routes around an obstacle only when a geometry represents it: a box, sphere, capsule, or cylinder positioned in the frame system. An obstacle you do not define is invisible to the planner, and the definition itself is hard to read in JSON. A box specified as `{x: 800, y: 1200, z: 20}` at some parent-relative translation either covers the table or it doesn't, and you can't tell which from the numbers. The **3D SCENE** tab lets you see what the planner sees, so you can check coverage before running a plan. ## Prerequisites diff --git a/docs/visualization/3d-scene/editing-frames-visually.md b/docs/visualization/3d-scene/editing-frames-visually.md index ce0c1a5ffd..aed6a499f9 100644 --- a/docs/visualization/3d-scene/editing-frames-visually.md +++ b/docs/visualization/3d-scene/editing-frames-visually.md @@ -72,10 +72,10 @@ To delete a frame, remove it from the component's configuration on the CONFIGURE Visual editing covers most cases, but a few are faster in JSON: -- **Bulk changes** (renaming many frames, regenerating a layout) — JSON +- **Bulk changes** (renaming many frames, regenerating a layout): JSON edits are easier in a text editor. -- **Frames that reference components on a different machine part** — +- **Frames that reference components on a different machine part**: the visual editor's parent dropdown only shows local frames. - **Complex orientations** (rotations expressed in `axis_angles` or - `quaternion` rather than `ov_degrees`) — the visual editor surfaces + `quaternion` rather than `ov_degrees`): the visual editor surfaces only the orientation vector form. diff --git a/docs/visualization/3d-scene/the-3d-scene-interface.md b/docs/visualization/3d-scene/the-3d-scene-interface.md index f3ebe21cea..3fe10ff397 100644 --- a/docs/visualization/3d-scene/the-3d-scene-interface.md +++ b/docs/visualization/3d-scene/the-3d-scene-interface.md @@ -38,12 +38,12 @@ Entities that can be removed (for example, dropped PCD files) also show a **Remo **Dashboard toolbar** (top-center): Visible buttons, left to right: -- **Orthographic / Perspective** toggle — switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. -- **Add frames** — opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene/editing-frames-visually/). -- **Measurement** (ruler icon) — activate to measure distance between two points you pick in the viewport. Click the icon again to clear. -- **Measurement settings** (sliders icon next to the ruler) — toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. -- **Logs** — shows a count badge for errors/warnings from the scene renderer. -- **Settings** (gear icon) — opens the Settings panel. +- **Orthographic / Perspective** toggle: switch between an orthographic view (no foreshortening) and a perspective view. Keyboard: `C`. +- **Add frames**: opens a floating panel listing components that do not yet have a frame; click a component and then **Add frame** (singular) to attach a default frame to it. See [Editing frames visually](/visualization/3d-scene/editing-frames-visually/). +- **Measurement** (ruler icon): activate to measure distance between two points you pick in the viewport. Click the icon again to clear. +- **Measurement settings** (sliders icon next to the ruler): toggle `x`, `y`, or `z` under **Enabled axes** to constrain the second point to the enabled axes of the first. +- **Logs**: shows a count badge for errors/warnings from the scene renderer. +- **Settings** (gear icon): opens the Settings panel. ## Navigation controls diff --git a/docs/visualization/perception/cameras.md b/docs/visualization/perception/cameras.md index 8c9635933c..f1e542b0b6 100644 --- a/docs/visualization/perception/cameras.md +++ b/docs/visualization/perception/cameras.md @@ -29,5 +29,5 @@ a point cloud. For how point clouds render, display, and compare, see ## Camera frames A camera appears in the scene as a coordinate frame, and its point cloud renders at that -frame. If a camera's point cloud lands in the wrong place, check the camera's frame +frame. If a camera's point cloud appears in the wrong place, check the camera's frame configuration rather than the camera itself. diff --git a/docs/visualization/perception/point-clouds.md b/docs/visualization/perception/point-clouds.md index 51a03f44fd..01e986fb4d 100644 --- a/docs/visualization/perception/point-clouds.md +++ b/docs/visualization/perception/point-clouds.md @@ -15,7 +15,7 @@ camera perceives in the same space as the rest of your machine. When your machine is online, the scene streams point clouds from your depth cameras and draws each as a set of points at the camera's frame. Because each point cloud sits at its -camera's frame, a point cloud that lands in the wrong place usually points to a wrong camera +camera's frame, a point cloud that appears in the wrong place usually points to a wrong camera frame, not wrong perception. Vision services can also contribute point-cloud entities, such as the points behind a From 1cf833e9902bdab33a1d612753ad43a660747525 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 7 Jul 2026 10:27:13 -0600 Subject: [PATCH 39/46] Add redirect for the removed 3d-scene-tools section root The Netlify no-more-404 plugin failed the deploy: /visualization/3d-scene-tools/ existed in this branch's earlier deploys (before the section was renamed to '3d-scene') and 404'd with no redirect. Alias it to /visualization/3d-scene/. --- docs/visualization/3d-scene/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/visualization/3d-scene/_index.md b/docs/visualization/3d-scene/_index.md index 8d435c1583..b53c7e2b10 100644 --- a/docs/visualization/3d-scene/_index.md +++ b/docs/visualization/3d-scene/_index.md @@ -7,6 +7,7 @@ type: "docs" description: "What the 3D scene renders, where each element comes from, and how built-in configuration content differs from custom visuals a module publishes at runtime." aliases: - /visualization/visualizing-with-the-3d-scene/ + - /visualization/3d-scene-tools/ - /motion-planning/3d-scene/ --- From a2ddd35daeca4f7ee5fbbcd32936dfbe833e04cd Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 09:37:15 -0600 Subject: [PATCH 40/46] Fix API field name, GetPose deprecation target, and style review findings - Transform's geometry field is physical_object, not Transform.Geometry (verified against go.viam.com/api common.proto and the Python SDK) - Motion.GetPose is deprecated in favor of the frame system service's GetPose, not Robot.GetPose (rdk services/motion/motion.go) - Rewrite negation-heavy openings positively (verify-obstacles, visualize-a-motion-plan), drop personification (arm "thinks"/"expects") - Replace banned words: surface(s) as verb, ships, guarantees - Define stepColor/goalColor and commonpb import in plan snippets; replace placeholder loop body with the stepMarker call - Note commonpb/pb import paths in module snippets - Consistent linkTitle casing; WorldState page heading and tightening - Sphere: name both the JSON r key and proto radius_mm field --- docs/motion-planning/debug-motion-plan.md | 13 +++++----- .../obstacles/verify-obstacles.md | 2 +- .../visualize-a-motion-plan.md | 24 ++++++++++++++----- .../3d-scene/3d-scene-widgets.md | 2 +- docs/visualization/3d-scene/_index.md | 3 ++- .../3d-scene/editing-frames-visually.md | 4 ++-- docs/visualization/drawing-library.md | 10 ++++---- docs/visualization/overview.md | 5 ++-- docs/visualization/perception/point-clouds.md | 2 +- .../verify-point-cloud-alignment.md | 2 +- .../publish-visuals-from-a-module.md | 4 +++- docs/visualization/reference/world-state.md | 12 +++++----- docs/visualization/visuals-and-collisions.md | 12 ++++++---- 13 files changed, 56 insertions(+), 39 deletions(-) diff --git a/docs/motion-planning/debug-motion-plan.md b/docs/motion-planning/debug-motion-plan.md index edb0794525..889250cf4f 100644 --- a/docs/motion-planning/debug-motion-plan.md +++ b/docs/motion-planning/debug-motion-plan.md @@ -100,9 +100,9 @@ reference, see [Motion CLI commands](/motion-planning/reference/cli-commands/). ### Frames are in the wrong place -The arm reaches where it thinks is `x=300, y=200` and physically ends up elsewhere, or the -scene shows the gripper off to the side of the arm. The frame configuration does not match -what the machine does. Dump the configured frame tree: +The arm reports reaching `x=300, y=200` but physically sits elsewhere, or the +scene shows the gripper off to the side of the arm. Both symptoms point to a frame +configuration that disagrees with the physical setup. Dump the configured frame tree: ```sh viam machines part motion print-config --part "my-machine-main" @@ -123,7 +123,7 @@ viam machines part motion print-status --part "my-machine-main" `print-status` prints one line per frame part with its computed world-frame pose: ```text - my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 + my-arm : X: 0.00 Y: 0.00 Z: 0.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 0.00 my-gripper : X: 0.00 Y: 0.00 Z: 110.00 OX: 0.00 OY: 0.00 OZ: 1.00 Theta: 90.00 ``` @@ -177,8 +177,9 @@ commanded, the pose may have been interpreted in an unexpected reference frame. `reference_frame` on the target `PoseInFrame`: the same `(x, y, z)` in the arm's frame and in the world frame describes two different places. -The CLI commands `print-status`, `get-pose`, and `set-pose` call `Motion.GetPose`, which is -deprecated in favor of `Robot.GetPose`; the commands and their output format are stable. +The CLI commands `print-status`, `get-pose`, and `set-pose` call the motion service's +`GetPose`, which is deprecated in favor of the frame system service's `GetPose`; the +commands and their output format are stable. `set-pose` calls the motion service's `Move` and blocks until the motion finishes or fails, returning a non-zero exit status with the error message on failure. diff --git a/docs/motion-planning/obstacles/verify-obstacles.md b/docs/motion-planning/obstacles/verify-obstacles.md index a821a42230..92e6e9d6fe 100644 --- a/docs/motion-planning/obstacles/verify-obstacles.md +++ b/docs/motion-planning/obstacles/verify-obstacles.md @@ -9,7 +9,7 @@ aliases: - /motion-planning/3d-scene/set-up-obstacle-avoidance/ --- -The motion planner routes around an obstacle only when a geometry represents it: a box, sphere, capsule, or cylinder positioned in the frame system. An obstacle you do not define is invisible to the planner, and the definition itself is hard to read in JSON. A box specified as `{x: 800, y: 1200, z: 20}` at some parent-relative translation either covers the table or it doesn't, and you can't tell which from the numbers. The **3D SCENE** tab lets you see what the planner sees, so you can check coverage before running a plan. +The motion planner routes around an obstacle only when a geometry represents it: a box, sphere, capsule, or cylinder positioned in the frame system. The **3D SCENE** tab renders each configured geometry in place, so you can confirm that a box defined as `{x: 800, y: 1200, z: 20}` at a parent-relative translation actually covers the table, and check coverage of the whole workspace, before you run a plan. ## Prerequisites diff --git a/docs/motion-planning/visualize-a-motion-plan.md b/docs/motion-planning/visualize-a-motion-plan.md index 230f367f23..d66c52130f 100644 --- a/docs/motion-planning/visualize-a-motion-plan.md +++ b/docs/motion-planning/visualize-a-motion-plan.md @@ -20,9 +20,9 @@ use the rendered path to debug a plan that failed or moved unexpectedly. ## Why publish the plan as custom visuals -A plan is a sequence of joint configurations. Read as numbers it tells you -little; rendered in the scene it tells you immediately whether the path clips an -obstacle, swings wide, or aims at a target outside the arm's reach. Publishing +A plan is a sequence of joint configurations. Rendered in the scene, the path +shows immediately whether it clips an obstacle, swings wide, or aims at a +target outside the arm's reach. Publishing the plan as world state store transforms puts the trajectory and goals in the same 3D view as the frames and obstacle geometry the planner used, so you can see the path and the world together. @@ -35,16 +35,21 @@ convert each step into end-effector poses with the frame system. `ComputePoses` takes a configuration and returns the pose of each frame: ```go +var transforms []*commonpb.Transform for i, step := range plan.Trajectory() { poses, err := step.ComputePoses(fs) if err != nil { return err } gripperPose := poses["my-gripper"].Pose() - // Place a marker for this step at gripperPose (next section). - _ = i - _ = gripperPose + // stepMarker (next section) turns this pose into a visual. + transform, err := stepMarker(i, gripperPose) + if err != nil { + return err + } + transforms = append(transforms, transform) } +// Serve transforms through a world state store service (last section). ``` Each `step` is the arm's configuration at that point in the trajectory, and @@ -59,9 +64,16 @@ can update them when you re-plan. ```go import ( "github.com/viam-labs/motion-tools/draw" + commonpb "go.viam.com/api/common/v1" "go.viam.com/rdk/spatialmath" ) +// Distinct colors keep the trajectory and its goals apart in the scene. +var ( + stepColor = draw.NewColor(draw.WithName("blue")) + goalColor = draw.NewColor(draw.WithName("green")) +) + func stepMarker(i int, pose spatialmath.Pose) (*commonpb.Transform, error) { id := fmt.Sprintf("step-%d", i) // Build the sphere at the origin; WithPose below places it at the step pose. diff --git a/docs/visualization/3d-scene/3d-scene-widgets.md b/docs/visualization/3d-scene/3d-scene-widgets.md index f5f8969267..2f7ecd3b02 100644 --- a/docs/visualization/3d-scene/3d-scene-widgets.md +++ b/docs/visualization/3d-scene/3d-scene-widgets.md @@ -1,5 +1,5 @@ --- -linkTitle: "3D Scene Widgets" +linkTitle: "3D scene widgets" title: "3D scene widgets" weight: 40 layout: "docs" diff --git a/docs/visualization/3d-scene/_index.md b/docs/visualization/3d-scene/_index.md index b53c7e2b10..cb8c9ab896 100644 --- a/docs/visualization/3d-scene/_index.md +++ b/docs/visualization/3d-scene/_index.md @@ -15,7 +15,8 @@ The **3D SCENE** tab renders your machine in an interactive 3D view: the frames component, the geometry attached to them, point clouds from depth cameras, and any custom visuals a module publishes while the machine runs. It turns configuration you would otherwise read as JSON numbers into a picture you can inspect, so you can confirm a gripper -sits where the arm expects it or watch a motion plan against the obstacles around it. +sits where its frame configuration places it or watch a motion plan against the obstacles +around it. This page describes what the scene shows, where each element comes from, and how the scene stays current. For the tab's panels, navigation, and settings, see diff --git a/docs/visualization/3d-scene/editing-frames-visually.md b/docs/visualization/3d-scene/editing-frames-visually.md index aed6a499f9..9df67ad1bc 100644 --- a/docs/visualization/3d-scene/editing-frames-visually.md +++ b/docs/visualization/3d-scene/editing-frames-visually.md @@ -12,7 +12,7 @@ aliases: The **3D SCENE** tab can serve as a configuration editor: you can add, move, re-parent, and reshape frames without writing JSON. -Visual editing is most useful while you are still figuring out where things go. Typing coordinates into JSON and reloading the 3D view to check them is slow; editing in the viewport and seeing the result immediately is faster. The trade-off is that the visual editor writes the same JSON fields through a smaller surface area, so it is less suited to bulk changes or cross-machine-part frames. Changes flow back to the machine configuration, and the app surfaces an unsaved-changes banner on the CONFIGURE tab where you save them with **Save** or `⌘/Ctrl+S`. +Visual editing is most useful while you are still figuring out where things go. Typing coordinates into JSON and reloading the 3D view to check them is slow; editing in the viewport and seeing the result immediately is faster. The trade-off is that the visual editor edits fewer fields than the JSON editor, so it is less suited to bulk changes or cross-machine-part frames. Changes flow back to the machine configuration, and the app shows an unsaved-changes banner on the CONFIGURE tab where you save them with **Save** or `⌘/Ctrl+S`. ## Prerequisites @@ -77,5 +77,5 @@ Visual editing covers most cases, but a few are faster in JSON: - **Frames that reference components on a different machine part**: the visual editor's parent dropdown only shows local frames. - **Complex orientations** (rotations expressed in `axis_angles` or - `quaternion` rather than `ov_degrees`): the visual editor surfaces + `quaternion` rather than `ov_degrees`): the visual editor offers only the orientation vector form. diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md index c803c01bd1..9e5dec65f4 100644 --- a/docs/visualization/drawing-library.md +++ b/docs/visualization/drawing-library.md @@ -7,11 +7,9 @@ type: "docs" description: "Use the draw library to build visuals, and run the standalone Viam visualizer to preview spatial data from a Go client." --- -Viam Visualization is a standalone 3D visualizer you run yourself. Unlike the **3D -SCENE** tab in the Viam app, which renders a configured machine's frames, -geometry, point clouds, and published visuals, Viam Visualization is a separate tool for -monitoring, testing, and debugging spatial data: you start it locally and push -visuals to it from your own Go code. +Viam Visualization is a standalone 3D visualizer for monitoring, testing, and +debugging spatial data: you start it locally and push visuals to it from your +own Go code while you develop. It shares the same `draw` library used to build world state store transforms, so the visuals you construct are the same either way. @@ -60,7 +58,7 @@ and approach directions as arrows. ## Construct visuals with the library Building visuals with the library rather than by hand keeps producer code -readable and guarantees the identifiers and metadata are correct. You construct +readable and fills in the identifiers and metadata for you. You construct a shape with a styling option and let the library assemble the entity: ```go diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index 5bf5214c2c..204c4b9195 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -7,8 +7,9 @@ type: "docs" description: "Ways to visualize a Viam machine: the 3D scene, the standalone Viam visualizer, custom apps you build, and time-series dashboards." --- -There are several approaches to visualizing information in Viam. For things like motion, -frames, robot state, collisions, and live perception data, you can use the 3D scene view in +A machine produces two kinds of data you can watch: spatial state (frames, geometries, +collisions, point clouds, motion plans) and time series readings (temperatures, speeds, +counts). For spatial state, use the 3D scene view in the Viam app, or Viam Visualization, a standalone visualizer you run yourself. You can also read this data from components with a Viam SDK and present it in a custom user interface you build, which runs in a browser, on a phone, or on a server. For time series data, services diff --git a/docs/visualization/perception/point-clouds.md b/docs/visualization/perception/point-clouds.md index 01e986fb4d..4af7602a7a 100644 --- a/docs/visualization/perception/point-clouds.md +++ b/docs/visualization/perception/point-clouds.md @@ -1,5 +1,5 @@ --- -linkTitle: "Point Clouds" +linkTitle: "Point clouds" title: "Point clouds in the 3D scene" weight: 10 layout: "docs" diff --git a/docs/visualization/perception/verify-point-cloud-alignment.md b/docs/visualization/perception/verify-point-cloud-alignment.md index 8c6647edae..c7870115ee 100644 --- a/docs/visualization/perception/verify-point-cloud-alignment.md +++ b/docs/visualization/perception/verify-point-cloud-alignment.md @@ -12,7 +12,7 @@ aliases: Depth cameras produce point clouds: sets of 3D points that represent the surfaces the camera sees. The **3D SCENE** tab renders those points in your frame system, so you can check two things at once: the camera is producing usable data, and the data lines up with the rest of the workspace. -Misalignment usually means one of two things: the camera's frame offset is wrong, or the camera itself has a problem. Finding that out now, before a motion plan runs or an ML detector ships, costs minutes; finding it out later costs a day of debugging a downstream pipeline. +Misalignment usually means one of two things: the camera's frame offset is wrong, or the camera itself has a problem. Finding that out now, before a motion plan runs or you deploy an ML detector, costs minutes; finding it out later costs a day of debugging a downstream pipeline. ## Prerequisites diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 84df27cbd7..5bce79944e 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -39,7 +39,9 @@ calls to discover and follow your visuals: A typical implementation keeps the current transforms in a map keyed by UUID, serves `ListUUIDs` and `GetTransform` from that map, and fans out change events -to subscribers from `StreamTransformChanges`. +to subscribers from `StreamTransformChanges`. In the snippets below, `commonpb` +is `go.viam.com/api/common/v1` and `pb` is +`go.viam.com/api/service/worldstatestore/v1`. ```go func (s *visualizer) ListUUIDs( diff --git a/docs/visualization/reference/world-state.md b/docs/visualization/reference/world-state.md index 1ffb2b7b05..4adbc20b48 100644 --- a/docs/visualization/reference/world-state.md +++ b/docs/visualization/reference/world-state.md @@ -1,5 +1,5 @@ --- -linkTitle: "World State" +linkTitle: "WorldState" title: "WorldState" weight: 10 layout: "docs" @@ -8,8 +8,8 @@ description: "What a WorldState is: the per-request set of obstacles and frame t --- A `WorldState` is the argument you pass to a single `Move` call. It carries the obstacles -and frame transforms that the motion planner should account for on that one request, and -nothing else uses it. When the call returns, the `WorldState` is gone. +and frame transforms that the motion planner should account for on that one request. When +the call returns, the `WorldState` is gone. ## What a WorldState carries @@ -22,11 +22,11 @@ A `WorldState` holds two kinds of item: reposition where the arm moves (for example, a frame at a grasped object's tip) or carry a geometry that travels with a component. -## It applies to one request +## A WorldState applies to one request A `WorldState` is per-request. The motion service uses it to plan the single motion you -attach it to, then discards it. It is not stored, so a later `Move` call sees none of it -unless you pass a `WorldState` again. To build obstacles and transforms and attach them to a +attach it to, then discards it. To carry the same obstacles into a later `Move` call, pass +a `WorldState` again. To build obstacles and transforms and attach them to a `Move` call, see [Define obstacles](/motion-planning/obstacles/) and [Arm and end effector frames](/motion-planning/frame-system/end-effector-frames/). diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index 70142f3c88..6869bcf07b 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -11,7 +11,8 @@ A geometry is a simple shape, such as a box, sphere, or capsule, that represents object's physical extent. Viam uses one set of geometry types for two jobs: drawing a custom visual in the 3D scene, and telling the motion planner about an obstacle. -To create a custom visual, you attach a geometry to `Transform.Geometry` and provide +To create a custom visual, you attach a geometry to a transform's `physical_object` +field and provide that transform to a [world state store service](/reference/apis/services/world-state-store/). This service holds the transforms you publish and streams them to the 3D scene. This page covers what a transform contains, the difference between planner geometry and @@ -21,9 +22,9 @@ visualization geometry, and how to build each geometry type. A `Transform` carries four things that together place and style a visual: -- **Reference frame and pose**: This defines the visual's origin. -- **Geometry**: the shape to draw (a box, sphere, capsule, mesh, or point - cloud). +- **Reference frame and pose**: the visual's origin. +- **Geometry** (the `physical_object` field): the shape to draw (a box, sphere, + capsule, mesh, or point cloud). - **Metadata**: styling such as color and opacity. - **UUID**: a stable identifier for this specific visual. @@ -149,7 +150,8 @@ box := &commonpb.Geometry{ ### Sphere -A sphere takes a radius `radius_mm` in millimeters. +A sphere takes a radius in millimeters: the `r` key in JSON, the `radius_mm` +field in the proto. {{< tabs >}} {{% tab name="JSON" %}} From 947935ea841206a902f7b57d38f82f6342d642b9 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 09:45:35 -0600 Subject: [PATCH 41/46] Trim over-length page descriptions and split perception intro sentence Hugo warned on the two new descriptions (171 and 159 chars, limit 158); the perception index opener was one 31-word sentence. --- docs/motion-planning/debug-motion-plan.md | 2 +- docs/motion-planning/visualize-a-motion-plan.md | 2 +- docs/visualization/perception/_index.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/motion-planning/debug-motion-plan.md b/docs/motion-planning/debug-motion-plan.md index 889250cf4f..553622e7df 100644 --- a/docs/motion-planning/debug-motion-plan.md +++ b/docs/motion-planning/debug-motion-plan.md @@ -4,7 +4,7 @@ title: "Debug a motion plan" weight: 84 layout: "docs" type: "docs" -description: "Find why a motion plan failed or moved unexpectedly by inspecting frames, obstacles, and reach, either visually in the 3D scene or from the command line with the Viam CLI." +description: "Find why a motion plan failed or moved unexpectedly by checking frames, obstacles, and reach in the 3D scene or from the Viam CLI." aliases: - /motion-planning/3d-scene/debug-motion-plan/ - /motion-planning/debug-motion-with-cli/ diff --git a/docs/motion-planning/visualize-a-motion-plan.md b/docs/motion-planning/visualize-a-motion-plan.md index d66c52130f..fa4fb90ed5 100644 --- a/docs/motion-planning/visualize-a-motion-plan.md +++ b/docs/motion-planning/visualize-a-motion-plan.md @@ -4,7 +4,7 @@ title: "Visualize a motion plan" weight: 86 layout: "docs" type: "docs" -description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path, then compare it against obstacles and reach to debug failures." +description: "Publish a motion plan's trajectory and goals as custom visuals so the 3D scene renders the path against obstacles and reach." aliases: - /motion-planning/3d-scene/visualize-a-motion-plan/ --- diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md index 352794c80a..b30fc2b609 100644 --- a/docs/visualization/perception/_index.md +++ b/docs/visualization/perception/_index.md @@ -8,8 +8,8 @@ no_list: true description: "View and compare live perception data, such as depth-camera point clouds, in the 3D scene." --- -The 3D scene renders live perception data alongside your frame system, so you can see what -your machine senses in the same space as the components that sense it. +The 3D scene renders live perception data alongside your frame system. You see what your +machine senses in the same space as the components that sense it. {{< cards >}} {{% card link="/visualization/perception/point-clouds/" noimage="true" %}} From 0b99ba498f73b3864045034744d702d70bb699ae Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 10:12:47 -0600 Subject: [PATCH 42/46] Document the metadata keys the renderer implements The scene's metadata parser recognizes colors, color_format, opacities, show_axes_helper, invisible, chunks, and relationships, and ignores unknown keys. collision_allowed appears only in a proto comment; neither the draw library writes it nor the scene reads it, so drop it. --- docs/visualization/visuals-and-collisions.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index 6869bcf07b..fb30c3601b 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -45,14 +45,14 @@ incremental add, update, and remove operations to individual visuals. The metadata is a set of rendering attributes the scene reads when it draws the geometry: -- `color`: the fill color -- `opacity`: how transparent the shape is -- per-point colors: for point cloud geometry -- `collision_allowed`: a rendering hint that marks the visual as a permitted - contact, for display only - -These are all **visualization attributes**: they control how the visual looks, -`collision_allowed` included. The planner reads its solid geometry from the frame +- `colors`: the fill color, or one color per point for point cloud geometry +- `opacities`: how transparent the shape is +- `show_axes_helper`: draws a coordinate triad at the visual's origin +- `invisible`: hides the visual by default + +The scene reads only the keys it recognizes and ignores the rest. These are all +**visualization attributes**: they control how the visual looks. +The planner reads its solid geometry from the frame system and the [`WorldState`](/motion-planning/obstacles/) you pass to `Move`, so metadata changes what you see without changing what the planner plans around. From a98a8a7c0177882e555030394b9696486cb82f3f Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 10:28:13 -0600 Subject: [PATCH 43/46] Add concept-gap pages from the visualization coverage review - Transform metadata reference: the seven keys the scene reads (colors, color_format, opacities, show_axes_helper, invisible, chunks, relationships) with wire formats, verified against draw/transform.go and the scene's metadata.ts parser - Vision services in the 3D scene: display GetObjectPointClouds output, per-service toggles and polling rate, camera-frame vs detector attribution, tuning loop, empty-layer checks - Troubleshoot the 3D scene: classify symptoms by layer (configuration, module, connection) and read the scene's own signals first - Rename drawing-library.md to viam-visualization.md (alias kept): the page is the standalone-visualizer how-to; adds the snapshot save/load flow with the visualization_snapshot filename prefix - Python note on publish-visuals: viam.services.worldstatestore implements the same three methods; draw helper is Go-only - Cards for the new pages on overview, perception, and reference indexes --- docs/visualization/drawing-library.md | 114 ----------------- docs/visualization/overview.md | 3 +- docs/visualization/perception/_index.md | 1 + .../perception/vision-services.md | 78 ++++++++++++ .../publish-visuals-from-a-module.md | 13 +- docs/visualization/reference/_index.md | 1 + .../reference/transform-metadata.md | 65 ++++++++++ .../troubleshoot-the-3d-scene.md | 90 ++++++++++++++ docs/visualization/viam-visualization.md | 115 ++++++++++++++++++ 9 files changed, 363 insertions(+), 117 deletions(-) delete mode 100644 docs/visualization/drawing-library.md create mode 100644 docs/visualization/perception/vision-services.md create mode 100644 docs/visualization/reference/transform-metadata.md create mode 100644 docs/visualization/troubleshoot-the-3d-scene.md create mode 100644 docs/visualization/viam-visualization.md diff --git a/docs/visualization/drawing-library.md b/docs/visualization/drawing-library.md deleted file mode 100644 index 9e5dec65f4..0000000000 --- a/docs/visualization/drawing-library.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -linkTitle: "Drawing library" -title: "The drawing library and Viam Visualization" -weight: 30 -layout: "docs" -type: "docs" -description: "Use the draw library to build visuals, and run the standalone Viam visualizer to preview spatial data from a Go client." ---- - -Viam Visualization is a standalone 3D visualizer for monitoring, testing, and -debugging spatial data: you start it locally and push visuals to it from your -own Go code while you develop. -It shares the same `draw` library used to build world state store transforms, so -the visuals you construct are the same either way. - -This page covers what the drawing library is, the primitives it exposes, and how -to run the visualizer and drive it from a Go client. - -## Viam Visualization versus the 3D scene tab - -The two render 3D visuals, but they are different tools for different moments: - -- The **3D SCENE tab** lives in the Viam app and renders a configured machine: - its frames, geometry, point clouds, and the custom visuals published to its - world state store service. It is the in-app view of a machine. -- **Viam Visualization** is a standalone visualizer you run on your own machine and push - to from a client. It is for previewing and debugging spatial data while you - develop, without deploying a module or opening the Viam app. - -Reach for the 3D scene tab to inspect a running machine; reach for Viam -Visualization to iterate on spatial data from a script or test. - -## What the drawing library is - -The `draw` library turns geometry, pose, and styling metadata into the entities -the renderer displays. Instead of assembling proto structs by hand, you describe -a shape and its appearance and the library produces a correct, fully-formed -visual with the right identifiers and metadata. The same library backs both -render targets: it builds the `commonpb.Transform` values a world state store -module serves, and the entities you push to Viam Visualization. - -## Drawing primitives and styling - -The library exposes a set of drawable shapes and styling options. Pick the -primitive that matches what you are drawing: - -- **arrows**: directions, normals, or vectors -- **lines**: paths, segments, or connections -- **points**: sampled data or markers -- **models**: detailed 3D meshes -- **NURBS**: smooth curves and surfaces - -Each visual takes styling options: color, opacity, per-point colors for point -data, and a label. Choosing the right primitive and styling keeps a busy scene -readable, for example drawing a trajectory as a line, its waypoints as points, -and approach directions as arrows. - -## Construct visuals with the library - -Building visuals with the library rather than by hand keeps producer code -readable and fills in the identifiers and metadata for you. You construct -a shape with a styling option and let the library assemble the entity: - -```go -import "github.com/viam-labs/motion-tools/draw" - -shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) -``` - -Because the library owns the proto details, you work in terms of shapes and -styles, not field-by-field struct assembly. - -## Run the visualizer and push from a Go client - -Viam Visualization runs locally and renders in your browser. From the -[motion-tools repository](https://github.com/viam-labs/motion-tools), run `make setup` -once, then `make up` to start the app at `http://localhost:5173`. With the app running, -push visuals to it from Go with the client API. Reusing an entity ID updates that visual -in place; a new ID adds another: - -```go -import ( - "github.com/viam-labs/motion-tools/client/api" - "github.com/viam-labs/motion-tools/draw" -) - -// Reuse an ID to update that visual in place; change or omit it to add another. -_, err := api.DrawGeometry(api.DrawGeometryOptions{ - ID: "obstacle-1", - Geometry: box, - Color: draw.NewColor(draw.WithName("red")), -}) -``` - -This lets you preview spatial data, a point cloud, a set of detections, a planned -path, straight from a script or test, without deploying a module or connecting -through the Viam app. For setup, the local server, and the full client API, see the -[Viam Visualization documentation](https://viamrobotics.github.io/visualization/). - -## How updates reach the browser - -The app runs a **draw service** that the client API calls. Each push becomes an -`AddEntity`, `UpdateEntity`, or `RemoveEntity` operation, and the service fans that -single change out over a `StreamEntity` stream the browser subscribes to. The browser -applies the one change instead of re-rendering the scene. This is the same add, -update, and remove model the world state store service uses to feed the in-app 3D -scene, so a busy scene stays in sync as your data changes. - -## What's next - -- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): - use the same `draw` library to serve transforms to the in-app 3D scene. -- [Visuals and collisions](/visualization/visuals-and-collisions/): - what a transform contains and how the scene renders it. diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index 204c4b9195..c0f57c3be4 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -31,6 +31,7 @@ point clouds, and custom visuals a module publishes at runtime. {{% card link="/visualization/visuals-and-collisions/" noimage="true" %}} {{% card link="/visualization/publish-visuals-from-a-module/" noimage="true" %}} {{% card link="/motion-planning/visualize-a-motion-plan/" noimage="true" %}} +{{% card link="/visualization/troubleshoot-the-3d-scene/" noimage="true" %}} {{< /cards >}} ## Viam Visualization @@ -45,7 +46,7 @@ the visuals you build work either way. Viam app. {{< cards >}} -{{% card link="/visualization/drawing-library/" noimage="true" %}} +{{% card link="/visualization/viam-visualization/" noimage="true" %}} {{< /cards >}} ## Custom apps diff --git a/docs/visualization/perception/_index.md b/docs/visualization/perception/_index.md index b30fc2b609..e6a8ec2664 100644 --- a/docs/visualization/perception/_index.md +++ b/docs/visualization/perception/_index.md @@ -14,5 +14,6 @@ machine senses in the same space as the components that sense it. {{< cards >}} {{% card link="/visualization/perception/point-clouds/" noimage="true" %}} {{% card link="/visualization/perception/cameras/" noimage="true" %}} +{{% card link="/visualization/perception/vision-services/" noimage="true" %}} {{% card link="/visualization/perception/verify-point-cloud-alignment/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/perception/vision-services.md b/docs/visualization/perception/vision-services.md new file mode 100644 index 0000000000..5cd2a75801 --- /dev/null +++ b/docs/visualization/perception/vision-services.md @@ -0,0 +1,78 @@ +--- +linkTitle: "Vision services" +title: "Vision services in the 3D scene" +weight: 30 +layout: "docs" +type: "docs" +description: "Render a vision service's segmented objects in the 3D scene and tune detection parameters against the live view." +--- + +A vision service with a 3D segmenter returns the objects it finds as point clouds, one +per object, through `GetObjectPointClouds`. The 3D scene polls each vision service on +your machine and renders those objects in place, so you see what the detector found in +the same space as the camera, the arm, and the workspace geometry. + +## Prerequisites + +- A vision service configured with a 3D segmenter, such as the + [`obstacles_pointcloud` module](https://app.viam.com/module/viam/obstacles-pointcloud). +- A depth camera that supports point clouds, with its frame configured. + +## Display a vision service's objects + +Open the **3D SCENE** tab on your machine's page. When the machine is online, the scene +calls each vision service's `GetObjectPointClouds` and draws the returned objects as +point-cloud entities. Each object renders at the pose the service reports, placed in the +scene by the camera's frame. + +To control which services render, open the **Settings** panel (gear icon) and select +**Vision**: each vision service has its own toggle. The polling rate lives under +**Connection**, with the scene's other data-stream rates; slow it down when a busy scene +lags, or turn it off while you work on something else. + +## Read an object's placement + +An object's position in the scene combines two sources: the segmenter's output (the +object's points and pose relative to the camera) and the camera's frame configuration +(where the camera sits in the frame system). When an object renders in the wrong place, +this split tells you where to look: + +- **Every object is offset the same way**: the camera's frame configuration is wrong. + Verify it with [Measuring between frames](/visualization/3d-scene/measuring-between-frames/) + and [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). +- **One object is wrong or split in two**: the segmenter's parameters group the points + incorrectly. Tune the service configuration. + +## Tune a detector against the live view + +The scene gives you a visual feedback loop for segmenter parameters: + +1. Place representative objects in the camera's view. +2. Watch the rendered objects in the 3D scene next to the raw point cloud (enable the + camera under **Settings** > **Pointclouds**). +3. Adjust the vision service's configuration, for example the minimum points per object + or the plane-removal threshold, and save. +4. Compare the new segmentation against the physical objects, then repeat. + +Segmentation quality shows up directly: an over-aggressive threshold drops small +objects, and a loose one merges neighbors into one blob. + +## When no objects appear + +Work through these checks in order: + +1. The vision service's toggle is on under **Settings** > **Vision**. +2. The vision polling rate under **Settings** > **Connection** is not off. +3. The service returns objects at all: call `GetObjectPointClouds` from the + [vision service API](/reference/apis/services/vision/) and check the count. +4. The camera behind the service supports point clouds: cameras that report + `supports_pcd=false` produce nothing for a 3D segmenter to segment. + +## What's next + +- [Point clouds](/visualization/perception/point-clouds/): how raw depth-camera data + renders and displays. +- [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/): + confirm the camera's frame before you tune a detector. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): draw + custom annotations the built-in vision layer does not cover. diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 5bce79944e..89007de799 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -43,6 +43,13 @@ to subscribers from `StreamTransformChanges`. In the snippets below, `commonpb` is `go.viam.com/api/common/v1` and `pb` is `go.viam.com/api/service/worldstatestore/v1`. +{{% alert title="Implementing in Python" color="tip" %}} +The Python SDK also includes the service, in `viam.services.worldstatestore`: implement +the same three methods there. The `draw` helper library below is Go-only; a Python +module builds its `Transform` protos directly, with the geometry constructors shown in +[Visuals and collisions](/visualization/visuals-and-collisions/). +{{% /alert %}} + ```go func (s *visualizer) ListUUIDs( ctx context.Context, extra map[string]any, @@ -208,7 +215,9 @@ func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstate - [Visuals and collisions](/visualization/visuals-and-collisions/): what a transform contains, and which geometry the planner collision-checks. -- [The drawing library and Viam Visualization](/visualization/drawing-library/): - the `draw` primitives and the standalone visualizer. +- [Viam Visualization](/visualization/viam-visualization/): + preview the same visuals from a script with the standalone visualizer. +- [Transform metadata](/visualization/reference/transform-metadata/): + the styling keys the scene reads and their wire formats. - [Frame system](/motion-planning/frame-system/): position the transforms you publish. diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md index 19cf55cab9..ab36b68d40 100644 --- a/docs/visualization/reference/_index.md +++ b/docs/visualization/reference/_index.md @@ -13,4 +13,5 @@ Reference pages for the types and services that back the 3D scene and custom vis {{< cards >}} {{% card link="/reference/apis/services/world-state-store/" noimage="true" %}} {{% card link="/visualization/reference/world-state/" noimage="true" %}} +{{% card link="/visualization/reference/transform-metadata/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/reference/transform-metadata.md b/docs/visualization/reference/transform-metadata.md new file mode 100644 index 0000000000..c45bec7a68 --- /dev/null +++ b/docs/visualization/reference/transform-metadata.md @@ -0,0 +1,65 @@ +--- +linkTitle: "Transform metadata" +title: "Transform metadata" +weight: 20 +layout: "docs" +type: "docs" +description: "The metadata keys the 3D scene reads from a transform, with the wire format for each: colors, opacities, axes helper, and visibility." +--- + +A transform's `metadata` field is a protobuf `Struct` of rendering attributes. The 3D +scene reads seven keys from it and ignores every other key. If a visual renders with +default styling, check the key names and formats on this page first: an unrecognized +key fails silently. + +Metadata affects rendering only. The motion planner reads its geometry from the frame +system and the [`WorldState`](/visualization/reference/world-state/) you pass to `Move`, +so no metadata key changes what the planner plans around. + +## The keys the scene reads + +| Key | Type | What it controls | +| ------------------ | --------------- | --------------------------------------------------------------------------------- | +| `colors` | string (base64) | Fill color, or one color per point for point cloud geometry. | +| `color_format` | number | How the color bytes are laid out. The `draw` library writes the RGB format. | +| `opacities` | string (base64) | Transparency, `0` (invisible) to `255` (opaque). One byte, or one byte per point. | +| `show_axes_helper` | bool | Draws a coordinate triad at the visual's origin. | +| `invisible` | bool | Hides the visual by default; the viewer can re-enable it in the World panel. | +| `chunks` | object | Streams a large entity in pieces: `chunk_size`, `total`, and `stride`. | +| `relationships` | list | Links this entity to others, which powers features such as HoverLink. | + +## Color and opacity wire format + +The binary values travel as base64-encoded strings, because a protobuf `Struct` has no +bytes type: + +- `colors` packs 3 bytes per color, in R, G, B order. A single color styles the whole + shape; for point cloud geometry, supply one color per point. +- `opacities` packs 1 byte per value. A single byte applies one opacity to the whole + visual; a byte per point sets per-point transparency. + +## Produce metadata with the draw library + +The [`draw` library](https://github.com/viam-labs/motion-tools) encodes these formats +for you, so producer code sets options instead of packing bytes: + +```go +import "github.com/viam-labs/motion-tools/draw" + +// Red at half opacity. WithName, WithHex, WithRGB, and WithHSV also build colors. +drawn, err := draw.NewDrawnGeometry( + box, + draw.WithGeometryColor(draw.NewColor(draw.WithRGBA(255, 0, 0, 128))), +) +``` + +`WithMetadataColors`, `WithMetadataAxesHelper`, and `WithMetadataInvisible` set the +corresponding keys on a drawing. When you assemble the `Struct` in another language, +follow the wire formats in the table above. + +## What's next + +- [Visuals and collisions](/visualization/visuals-and-collisions/): the transform the + metadata belongs to, and which geometry the planner collision-checks. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): serve + styled transforms to the 3D scene. diff --git a/docs/visualization/troubleshoot-the-3d-scene.md b/docs/visualization/troubleshoot-the-3d-scene.md new file mode 100644 index 0000000000..ec4cc149b7 --- /dev/null +++ b/docs/visualization/troubleshoot-the-3d-scene.md @@ -0,0 +1,90 @@ +--- +linkTitle: "Troubleshoot the 3D scene" +title: "Troubleshoot the 3D scene" +weight: 80 +layout: "docs" +type: "docs" +description: "Trace a wrong or missing element in the 3D scene to its source: the machine configuration, a module, or the connection." +--- + +Every element in the 3D scene comes from one of three layers: the machine's saved +configuration (frames and geometry), a module publishing at runtime (custom visuals and +point clouds), or the live connection that streams poses and data. When something looks +wrong, identify the layer first; the fix lives there, and edits to the wrong layer waste +time. + +## Read the scene's own signals first + +Three signals tell you what the scene is receiving before you change anything: + +- **Online state.** An offline machine renders the saved frame configuration only, with + every component at its configured pose. Live poses, point clouds, and custom visuals + need the machine online and connected. +- **The Logs button** in the top-center toolbar shows a count of renderer errors and + warnings. A nonzero badge means the scene received data it could not draw; open it and + read the message before assuming data is missing. +- **Polling rates** under **Settings** > **Connection** control how often the scene + fetches each data stream. A stream set to off explains a layer that never updates. + +## A frame or geometry is in the wrong place + +Frames and configured geometry come from the machine configuration, so a misplaced one +is a configuration problem. Select the entity in the **World** panel and compare its +**local position** and **parent frame** to your physical measurements. Then fix the +values in the configuration or +[edit the frame in the scene](/visualization/3d-scene/editing-frames-visually/). +For the full checklist of frame and obstacle checks, see +[Debug a motion plan](/motion-planning/debug-motion-plan/). + +## A point cloud is missing + +Work down the camera pipeline: + +1. The camera's toggle is on under **Settings** > **Pointclouds** > **Enabled cameras**. +2. The camera supports point clouds. A camera that reports `supports_pcd=false` is + auto-disabled in that list; check its module documentation. +3. The camera's frame is configured. A point cloud renders at its camera's frame, so a + camera with no frame has nowhere to draw. + +A point cloud that renders in the wrong place, rather than missing, is a frame problem: +see [Verify point cloud alignment](/visualization/perception/verify-point-cloud-alignment/). + +## A custom visual is missing or stale + +Custom visuals come from a module through a +[world state store service](/reference/apis/services/world-state-store/), so a missing or +stale one is a module problem. Check the module's output at the service boundary: + +1. `ListUUIDs` returns the visual's UUID. An empty list means the module publishes + nothing; check the module's logs and its dependencies. +2. The module emits a change event when the data changes. A visual that appears once and + never moves usually means the poll loop stopped or emits no `TransformChange` events. +3. The UUID is stable across updates. A module that generates a fresh UUID each tick + piles up duplicates instead of updating one visual. + +The implementation side of these checks is covered in +[Publish visuals from a module](/visualization/publish-visuals-from-a-module/). + +## Know what the scene can tell you + +Match your conclusion to what the scene renders: + +- **Machine offline**: trust the frame tree and configured geometry; treat every pose as + the configured default, and expect point clouds, live poses, and custom visuals to be + absent. +- **Machine online**: poses, point clouds, and custom visuals are live, subject to the + polling rates you set. + +A plan's obstacles are a separate case: `WorldState` obstacles passed to a single `Move` +call are planner input for that request and are never drawn. To see them, publish the +same geometry as a custom visual; see +[Visuals and collisions](/visualization/visuals-and-collisions/). + +## What's next + +- [Debug a motion plan](/motion-planning/debug-motion-plan/): the frame, obstacle, and + reach checks for planning failures. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the + service methods and poll loop behind custom visuals. +- [The 3D scene interface](/visualization/3d-scene/the-3d-scene-interface/): every + panel, setting, and toolbar button. diff --git a/docs/visualization/viam-visualization.md b/docs/visualization/viam-visualization.md new file mode 100644 index 0000000000..df299ca7a7 --- /dev/null +++ b/docs/visualization/viam-visualization.md @@ -0,0 +1,115 @@ +--- +linkTitle: "Viam Visualization" +title: "Viam Visualization" +weight: 30 +layout: "docs" +type: "docs" +description: "Run the standalone Viam Visualization app locally and push geometries, point clouds, and frame systems to it from a Go client." +aliases: + - /visualization/drawing-library/ +--- + +Viam Visualization is a standalone 3D visualizer for monitoring, testing, and debugging +spatial data: you start it locally and push visuals to it from your own Go code while +you develop. It renders the same entities the in-app **3D SCENE** tab renders, built +with the same `draw` library, so the visuals you construct work in either place. + +This page covers running the app, pushing visuals from a Go client, and saving scenes +as snapshots. + +## Viam Visualization versus the 3D scene tab + +The two render 3D visuals, but they are different tools for different moments: + +- The **3D SCENE tab** lives in the Viam app and renders a configured machine: + its frames, geometry, point clouds, and the custom visuals published to its + world state store service. It is the in-app view of a machine. +- **Viam Visualization** is a standalone visualizer you run on your own machine and push + to from a client. It is for previewing and debugging spatial data while you + develop, without deploying a module or opening the Viam app. + +Reach for the 3D scene tab to inspect a running machine; reach for Viam +Visualization to iterate on spatial data from a script or test. + +## Run the app + +From the [motion-tools repository](https://github.com/viam-labs/motion-tools), run +`make setup` once, then `make up` to start the app at `http://localhost:5173`. It +renders in your browser. + +## Push visuals from a Go client + +With the app running, push visuals to it with the client API from +`github.com/viam-labs/motion-tools/client/api`. Reusing an entity ID updates that +visual in place, so an iterating script animates state instead of piling up duplicates; +a new or empty ID adds another entity: + +```go +import ( + "github.com/viam-labs/motion-tools/client/api" + "github.com/viam-labs/motion-tools/draw" +) + +// Reuse an ID to update that visual in place; change or omit it to add another. +_, err := api.DrawGeometry(api.DrawGeometryOptions{ + ID: "obstacle-1", + Geometry: box, + Color: draw.NewColor(draw.WithName("red")), +}) +``` + +The client API has a call per data kind: geometries, point clouds, lines, NURBS, GLTF +models, single frames, and whole frame systems. Beyond plain shapes, the `draw` package +supplies the primitives you push, arrows for directions and normals, lines for paths, +points for sampled data, and styling options for each: + +```go +import "github.com/viam-labs/motion-tools/draw" + +shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) +``` + +This lets you preview spatial data, a point cloud, a set of detections, a planned +path, straight from a script or test. For setup, the local server, and the full client +API, see the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). + +## Save and load scene snapshots + +A snapshot captures a scene as a JSON file you can share, commit as a test fixture, or +reload later. Build one in Go with the `draw` package, then drag the file onto either +viewer's viewport to load it: + +```go +import "github.com/viam-labs/motion-tools/draw" + +snapshot := draw.NewSnapshot( + draw.WithSceneCamera(camera), // where the scene camera starts + draw.WithGrid(true), +) +if err := snapshot.DrawGeometry(box, boxPose, "world", draw.ColorFromName("dodgerblue")); err != nil { + return err +} +data, err := snapshot.MarshalJSON() +``` + +Name the output file with a `visualization_snapshot` prefix, for example +visualization_snapshot_grasp_test.json: the drag-and-drop loader accepts +only files that match that prefix. + +## How updates reach the browser + +The app runs a **draw service** that the client API calls. Each push becomes an +`AddEntity`, `UpdateEntity`, or `RemoveEntity` operation, and the service fans that +single change out over a `StreamEntity` stream the browser subscribes to. The browser +applies the one change instead of re-rendering the scene. This is the same add, +update, and remove model the world state store service uses to feed the in-app 3D +scene, so a busy scene stays in sync as your data changes. + +## What's next + +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): + use the same `draw` library to serve transforms to the in-app 3D scene. +- [Transform metadata](/visualization/reference/transform-metadata/): + the styling attributes behind colors, opacity, and visibility. +- [Visuals and collisions](/visualization/visuals-and-collisions/): + what a transform contains and how the scene renders it. From fc9db9f5a3b8e5657d4eaedd056f25071ee1f71d Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 10:35:31 -0600 Subject: [PATCH 44/46] Close the remaining coverage-ledger items - Document the frame POV widget (opens from the Details panel's View from this frame button; verified ungated, WEBLABS_EXPERIMENTS is empty) and move camera-widget mechanics to the widgets page - Map the axis colors to X, Y, Z on the 3D scene index - Link the embedding guide from the overview's custom apps section - Link the geometry types table from visuals-and-collisions - World state store API page: replace the stale VISUALIZE tab name, describe what the API does, and link the implementing how-to --- .../apis/services/world-state-store.md | 9 ++++--- .../3d-scene/3d-scene-widgets.md | 25 ++++++++++++++----- docs/visualization/3d-scene/_index.md | 4 +-- .../3d-scene/the-3d-scene-interface.md | 2 +- docs/visualization/overview.md | 4 ++- docs/visualization/perception/cameras.md | 10 +++----- docs/visualization/visuals-and-collisions.md | 3 ++- 7 files changed, 37 insertions(+), 20 deletions(-) diff --git a/docs/reference/apis/services/world-state-store.md b/docs/reference/apis/services/world-state-store.md index 8edcd13159..f5d4868c3c 100644 --- a/docs/reference/apis/services/world-state-store.md +++ b/docs/reference/apis/services/world-state-store.md @@ -4,7 +4,7 @@ linkTitle: "World state store" weight: 90 type: "docs" tags: ["world_state_store", "services"] -description: "Retrieve a list of world objects." +description: "List, get, and stream the transforms a world state store service publishes for the 3D scene to draw." icon: true images: ["/icons/components/generic.svg"] date: "2025-09-12" @@ -13,8 +13,11 @@ aliases: - /dev/reference/apis/services/world-state-store/ --- -The world state store service API allows you to retrieve a list of world objects. -You can use this list to create custom visualizers to render spatial data related to a machine on the machine's **VISUALIZE** tab. +The world state store service API lets a client list, get, and stream the transforms a +world state store service publishes. The **3D SCENE** tab uses this API to render a +machine's [custom visuals](/visualization/visuals-and-collisions/), and a custom +visualizer you build can consume it the same way. To implement the service in a module, +see [Publish visuals from a module](/visualization/publish-visuals-from-a-module/). The world state store service supports the following methods: diff --git a/docs/visualization/3d-scene/3d-scene-widgets.md b/docs/visualization/3d-scene/3d-scene-widgets.md index 2f7ecd3b02..c5abb41317 100644 --- a/docs/visualization/3d-scene/3d-scene-widgets.md +++ b/docs/visualization/3d-scene/3d-scene-widgets.md @@ -10,15 +10,15 @@ aliases: --- Widgets are floating panels you overlay on the 3D scene to read live data next to the 3D -view. Each widget is draggable and resizable, and you turn them on and off in the scene's -settings. +view. Each widget is draggable and resizable. ## Enable a widget Open the **Settings** panel (the gear icon in the scene toolbar) and select **Widgets**. Toggle a widget on and it appears floating over the viewport; toggle it off to remove it. The panel lists two kinds of widget: **Arm positions**, and a **Camera widgets** section with one -toggle per configured camera. +toggle per configured camera. A third widget, the [frame POV](#frame-pov), opens from the +Details panel instead. ## Arm positions @@ -38,6 +38,19 @@ while you inspect its pose in the 3D view. ## Camera widgets -A camera widget shows a camera's live feed as a panel over the 3D scene. Each configured -camera has its own toggle under **Camera widgets**. For the feed's resolution and frame-rate -controls, see [Cameras](/visualization/perception/cameras/). +A camera widget shows a camera's live feed as a panel over the 3D scene, with its current +frame rate, for example `20.0fps`. Each configured camera has its own toggle under +**Camera widgets**. + +A resolution dropdown on the widget sets the feed size: **Default**, `1280x720`, `640x360`, +`320x180`, `160x90`, or `80x44`. A lower resolution costs less bandwidth, which helps when +you watch several cameras at once or work over a slow connection. For how a camera's data +appears in the scene itself, see [Cameras](/visualization/perception/cameras/). + +## Frame POV + +The frame POV widget renders the scene from a selected frame's perspective, so you can +check what a camera's view covers, or what an end effector approaches, from its configured +pose. Select an entity in the **World** panel, then click the camera icon (**View from this +frame**) in the Details panel. A panel titled **POV** opens for that frame; open one per +frame you want to watch. diff --git a/docs/visualization/3d-scene/_index.md b/docs/visualization/3d-scene/_index.md index cb8c9ab896..0606c0d61a 100644 --- a/docs/visualization/3d-scene/_index.md +++ b/docs/visualization/3d-scene/_index.md @@ -26,8 +26,8 @@ stays current. For the tab's panels, navigation, and settings, see The scene draws four kinds of element, each with its own appearance: -- **Component frames** render as sets of red, green, and blue coordinate axes, one per - component, positioned by each frame's translation and orientation. +- **Component frames** render as sets of red, green, and blue coordinate axes (X, Y, and + Z), one per component, positioned by each frame's translation and orientation. - **Geometries** render as translucent shapes (a box, sphere, or capsule) at the frame they are attached to. - **Point clouds** from depth cameras render as colored point sets. diff --git a/docs/visualization/3d-scene/the-3d-scene-interface.md b/docs/visualization/3d-scene/the-3d-scene-interface.md index 3fe10ff397..86bb7dc577 100644 --- a/docs/visualization/3d-scene/the-3d-scene-interface.md +++ b/docs/visualization/3d-scene/the-3d-scene-interface.md @@ -33,7 +33,7 @@ It includes: - **local position** (mm) and **local orientation** (deg): pose relative to the parent frame. Editable for configurable frames; these correspond to the `translation` and `orientation` in your frame configuration. - **geometry**: four buttons (`None` / `Box` / `Sphere` / `Capsule`) plus **dimensions** (`x / y / z` for Box, `r / l` for Capsule, `r` for Sphere, all in mm). -The panel header includes a **Zoom to object** button (centers the camera on the selected entity) and a copy-to-clipboard button next to the `Details` heading that exports the entity's pose and geometry as JSON. +The panel header includes a **Zoom to object** button (centers the camera on the selected entity), a **View from this frame** button (camera icon) that opens a [frame POV widget](/visualization/3d-scene/3d-scene-widgets/#frame-pov), and a copy-to-clipboard button next to the `Details` heading that exports the entity's pose and geometry as JSON. Entities that can be removed (for example, dropped PCD files) also show a **Remove from scene** button in the header. **Dashboard toolbar** (top-center): Visible buttons, left to right: diff --git a/docs/visualization/overview.md b/docs/visualization/overview.md index c0f57c3be4..acd618faf7 100644 --- a/docs/visualization/overview.md +++ b/docs/visualization/overview.md @@ -53,7 +53,9 @@ the visuals you build work either way. A custom app uses a Viam SDK to read your machine's data and present it however you design. It runs outside the machine, in a browser, on a phone, or on a server, so you can build a -custom dashboard, an operator console, or a fleet view. +custom dashboard, an operator console, or a fleet view. A Svelte app can also +[embed the 3D scene's own renderer](https://viamrobotics.github.io/visualization/guides/embedding/) +as a component, so a custom UI gets the full 3D view alongside your own controls. - Use it when operators or stakeholders need a tailored UI beyond the built-in scene. - Best when you want to choose exactly which data to show and how, for one machine or a diff --git a/docs/visualization/perception/cameras.md b/docs/visualization/perception/cameras.md index f1e542b0b6..f0512cccd5 100644 --- a/docs/visualization/perception/cameras.md +++ b/docs/visualization/perception/cameras.md @@ -13,12 +13,10 @@ its live feed shows what it currently sees, and a depth camera also produces a p ## See a camera's live feed To watch a camera's feed next to the 3D view, open the **Settings** panel (gear icon), select -**Widgets**, and toggle the camera on under **Camera widgets**. A floating panel then shows -the live feed with its current frame rate, for example `20.0fps`. - -A resolution dropdown on the widget sets the feed size: **Default**, `1280x720`, `640x360`, -`320x180`, `160x90`, or `80x44`. A lower resolution costs less bandwidth, which helps when you -watch several cameras at once or work over a slow connection. +**Widgets**, and toggle the camera on under **Camera widgets**. For the widget's resolution +and frame-rate controls, see [3D scene widgets](/visualization/3d-scene/3d-scene-widgets/). +To view the scene from the camera's own perspective, open its +[frame POV](/visualization/3d-scene/3d-scene-widgets/#frame-pov). ## Point clouds from depth cameras diff --git a/docs/visualization/visuals-and-collisions.md b/docs/visualization/visuals-and-collisions.md index fb30c3601b..ecdb29ba08 100644 --- a/docs/visualization/visuals-and-collisions.md +++ b/docs/visualization/visuals-and-collisions.md @@ -98,7 +98,8 @@ The supported types are: Choose the type that matches what you are representing: a box or capsule to approximate a physical object, a mesh for a precise model, a point cloud for -sensor data. +sensor data. For guidance on matching a geometry type to a physical object, see the +[geometry types table](/motion-planning/obstacles/overview/#geometry-types). You build a geometry as a `Geometry` proto, the same type a world state store transform and a `WorldState` obstacle both carry. The Python SDK and the Go SDK From 4afb5310eda3ab7a657f6b002838f5eb0c936dc0 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 11:02:02 -0600 Subject: [PATCH 45/46] Add a draw library reference and the service config stanza - draw library reference: placement/identity options, shape constructors, color builders, metadata options, snapshot options, each verified against motion-tools main; pkg.go.dev linked as the current inventory since the library moves faster than the RDK - Publish visuals: show the machine-config services entry (api rdk:service:world_state_store, model, depends_on) so the implemented module actually lands on a machine --- .../publish-visuals-from-a-module.md | 22 +++++ docs/visualization/reference/_index.md | 1 + docs/visualization/reference/draw-library.md | 89 +++++++++++++++++++ 3 files changed, 112 insertions(+) create mode 100644 docs/visualization/reference/draw-library.md diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 89007de799..28fd44b77f 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -211,6 +211,28 @@ func newVisualizer(deps resource.Dependencies, conf resource.Config) (worldstate } ``` +## Add the service to your machine + +The module registers a model against the world state store API, so you configure it like +any other modular service: add the module to your machine, then add a service with +`"api": "rdk:service:world_state_store"` and your model. Name the resources the module +pulls from in `depends_on`, so the module starts after them: + +```json +"services": [ + { + "name": "visualizer", + "api": "rdk:service:world_state_store", + "model": "::", + "depends_on": ["obstacle-sensor"] + } +] +``` + +Once the machine is online with the service running, the **3D SCENE** tab discovers the +service and streams its transforms. For packaging and deploying the module itself, see +[Build modules](/build-modules/). + ## What's next - [Visuals and collisions](/visualization/visuals-and-collisions/): diff --git a/docs/visualization/reference/_index.md b/docs/visualization/reference/_index.md index ab36b68d40..8c1998801f 100644 --- a/docs/visualization/reference/_index.md +++ b/docs/visualization/reference/_index.md @@ -14,4 +14,5 @@ Reference pages for the types and services that back the 3D scene and custom vis {{% card link="/reference/apis/services/world-state-store/" noimage="true" %}} {{% card link="/visualization/reference/world-state/" noimage="true" %}} {{% card link="/visualization/reference/transform-metadata/" noimage="true" %}} +{{% card link="/visualization/reference/draw-library/" noimage="true" %}} {{< /cards >}} diff --git a/docs/visualization/reference/draw-library.md b/docs/visualization/reference/draw-library.md new file mode 100644 index 0000000000..bc84ada42f --- /dev/null +++ b/docs/visualization/reference/draw-library.md @@ -0,0 +1,89 @@ +--- +linkTitle: "draw library" +title: "draw library" +weight: 30 +layout: "docs" +type: "docs" +description: "Lookup tables for the draw library: placement and identity options, shape constructors, colors, metadata, and snapshot options." +--- + +The [`draw` library](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) +(`github.com/viam-labs/motion-tools/draw`) builds the transforms and entities that the 3D +scene and [Viam Visualization](/visualization/viam-visualization/) render. This page lists +the options you reach for while writing a producer. The library lives in `viam-labs` and +moves faster than the RDK, so treat +[pkg.go.dev](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) as the full and +current inventory. + +## Placement and identity + +Options for `Draw` calls and `NewDrawConfig`, all of type `DrawableOption`: + +| Option | What it sets | +| ---------------- | ------------------------------------------------------------------------------------------ | +| `WithParent` | The reference frame the pose is expressed in. Defaults to `world`. | +| `WithPose` | The entity's pose in the parent frame. | +| `WithCenter` | An offset applied at the entity's own center. | +| `WithID` | A string identity; the library derives a stable UUID from it, so re-sends update in place. | +| `WithUUID` | An explicit UUID, when you manage identity yourself. | +| `WithAxesHelper` | Draws a coordinate triad at the entity's origin. | +| `WithInvisible` | Hides the entity by default; the viewer can re-enable it. | + +## Shapes + +| Constructor | Builds | +| ------------------------------------- | ----------------------------------------------------------------------------------- | +| `NewDrawnGeometry` | A styled `spatialmath.Geometry`; its `Draw` method returns a `*commonpb.Transform`. | +| `NewShape` + `WithArrows` | Arrows, for directions, normals, or vectors. | +| `NewShape` + `WithLine` / `NewLine` | A line, for paths, segments, or connections. | +| `NewShape` + `WithPoints` | Points, for sampled data or markers. | +| `NewShape` + `WithModel` | A detailed 3D mesh model. | +| `NewShape` + `WithNurbs` / `NewNurbs` | A smooth NURBS curve or surface. | + +Style a geometry with `WithGeometryColor` (one color) or `WithGeometryColors` (per-point +colors) when you call `NewDrawnGeometry`. + +## Colors + +Build a `Color` with `NewColor` plus one option, or use the one-call helpers: + +| With `NewColor` | Helper | Input | +| --------------- | --------------- | -------------------------------------- | +| `WithRGB` | `ColorFromRGB` | `r, g, b` as 0 to 255 | +| `WithRGBA` | `ColorFromRGBA` | `r, g, b` plus alpha for opacity | +| `WithName` | `ColorFromName` | A CSS color name, such as `dodgerblue` | +| `WithHex` | `ColorFromHex` | A hex string | +| `WithHSV` | `ColorFromHSV` | Hue, saturation, value | + +## Metadata + +Options for `NewDrawing` and `NewTransform`, all of type `DrawMetadataOption`. Each writes +one of the [metadata keys](/visualization/reference/transform-metadata/) the scene reads: + +| Option | Metadata it writes | +| --------------------------- | -------------------------------------------- | +| `WithMetadataColors` | `colors` (and `opacities` from alpha) | +| `WithMetadataAxesHelper` | `show_axes_helper` | +| `WithMetadataInvisible` | `invisible` | +| `WithMetadataRelationships` | `relationships`, for links such as HoverLink | + +## Snapshots + +Options for `NewSnapshot`, which builds a loadable +[scene snapshot](/visualization/viam-visualization/#save-and-load-scene-snapshots): + +| Option | What it sets | +| ------------------------------------------- | -------------------------------------------------- | +| `WithSceneCamera` | Where the scene camera starts. | +| `WithGrid`, `WithGridCellSize` | The reference grid and its cell size. | +| `WithScenePointSize`, `WithScenePointColor` | Default point rendering. | +| `WithRenderArmModels` | Whether arms render as colliders, models, or both. | + +## What's next + +- [Transform metadata](/visualization/reference/transform-metadata/): the wire formats + behind the metadata options. +- [Publish visuals from a module](/visualization/publish-visuals-from-a-module/): the + library in a world state store module. +- [Viam Visualization](/visualization/viam-visualization/): the library from a script, + pushed to the standalone visualizer. From 123ade4f82c3bb5b9bd15cba3e91712934ac0692 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Tue, 14 Jul 2026 11:16:27 -0600 Subject: [PATCH 46/46] Link the Viam Visualization site where it adds value - Viam Visualization page: hosted playground (try without installing), the Running locally guide, the generated client API reference, and a new Connect to a live machine section (.env.local credentials + machine picker), which the upstream guide documents and our page previously omitted - Publish visuals: link the upstream Implementing WorldStateStoreService guide and its complete runnable example module - draw reference: link the project's generated API docs alongside pkg.go.dev --- .../publish-visuals-from-a-module.md | 3 +++ docs/visualization/reference/draw-library.md | 3 ++- docs/visualization/viam-visualization.md | 19 ++++++++++++++++--- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/visualization/publish-visuals-from-a-module.md b/docs/visualization/publish-visuals-from-a-module.md index 28fd44b77f..7cf731ff7b 100644 --- a/docs/visualization/publish-visuals-from-a-module.md +++ b/docs/visualization/publish-visuals-from-a-module.md @@ -235,6 +235,9 @@ service and streams its transforms. For packaging and deploying the module itsel ## What's next +- [Implementing WorldStateStoreService](https://viamrobotics.github.io/visualization/guides/worldstatestore/): + the upstream guide this pattern follows, with a + [complete runnable module](https://github.com/viam-labs/motion-tools/tree/main/docs/examples/worldstatestore). - [Visuals and collisions](/visualization/visuals-and-collisions/): what a transform contains, and which geometry the planner collision-checks. - [Viam Visualization](/visualization/viam-visualization/): diff --git a/docs/visualization/reference/draw-library.md b/docs/visualization/reference/draw-library.md index bc84ada42f..1a0f654125 100644 --- a/docs/visualization/reference/draw-library.md +++ b/docs/visualization/reference/draw-library.md @@ -11,7 +11,8 @@ The [`draw` library](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) (`github.com/viam-labs/motion-tools/draw`) builds the transforms and entities that the 3D scene and [Viam Visualization](/visualization/viam-visualization/) render. This page lists the options you reach for while writing a producer. The library lives in `viam-labs` and -moves faster than the RDK, so treat +moves faster than the RDK, so treat the project's +[generated API docs](https://viamrobotics.github.io/visualization/api/draw/) and [pkg.go.dev](https://pkg.go.dev/github.com/viam-labs/motion-tools/draw) as the full and current inventory. diff --git a/docs/visualization/viam-visualization.md b/docs/visualization/viam-visualization.md index df299ca7a7..4429d6fd26 100644 --- a/docs/visualization/viam-visualization.md +++ b/docs/visualization/viam-visualization.md @@ -35,7 +35,12 @@ Visualization to iterate on spatial data from a script or test. From the [motion-tools repository](https://github.com/viam-labs/motion-tools), run `make setup` once, then `make up` to start the app at `http://localhost:5173`. It -renders in your browser. +renders in your browser. For prerequisites and incremental-rebuild details, see the +[Running locally guide](https://viamrobotics.github.io/visualization/guides/local-usage/). + +To try the visualizer before installing anything, open the hosted +[playground](https://viamrobotics.github.io/visualization/playground/snapshot), which +renders a sample scene snapshot in your browser. ## Push visuals from a Go client @@ -70,8 +75,16 @@ shape := draw.NewShape(center, "approach", draw.WithArrows(arrows)) ``` This lets you preview spatial data, a point cloud, a set of detections, a planned -path, straight from a script or test. For setup, the local server, and the full client -API, see the [Viam Visualization documentation](https://viamrobotics.github.io/visualization/). +path, straight from a script or test. For every call and option, see the generated +[client API reference](https://viamrobotics.github.io/visualization/api/client-api/). + +## Connect to a live machine + +The visualizer can also connect to a Viam machine and render its frame system, arms, and +cameras, the way the in-app **3D SCENE** tab does. Put the machine's credentials in a +.env.local file at the repository root, run `make up`, then pick the machine +in the machine config panel (lower right). For the credential format, see the +[Running locally guide](https://viamrobotics.github.io/visualization/guides/local-usage/#connecting-to-a-viam-machine). ## Save and load scene snapshots