From f3e03f29708b6a2a56d8295fdc18c01c85b48766 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 15 Jul 2026 10:04:17 -0600 Subject: [PATCH 1/4] Document per-SDK OpenTelemetry request tracing for Python and C++ --- docs/build-apps/setup/python.md | 8 +++++ docs/data/overview.md | 59 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/docs/build-apps/setup/python.md b/docs/build-apps/setup/python.md index bb73e2d52f..91e43c0fc4 100644 --- a/docs/build-apps/setup/python.md +++ b/docs/build-apps/setup/python.md @@ -43,6 +43,14 @@ If you need the ML model service, install with the optional dependency: pip install 'viam-sdk[mlmodel]' ``` +For OpenTelemetry request tracing in a Python module, install the `tracing` extra: + +```sh {class="command-line" data-prompt="$"} +pip install 'viam-sdk[tracing]' +``` + +See [OpenTelemetry distributed tracing](/data/overview/#opentelemetry-distributed-tracing) for how to enable it. + ## Configure credentials Create a `.env` file in your project root: diff --git a/docs/data/overview.md b/docs/data/overview.md index 27a0262e66..71ad498204 100644 --- a/docs/data/overview.md +++ b/docs/data/overview.md @@ -121,6 +121,65 @@ For local development, see the [Jaeger getting started guide](https://www.jaeger You can combine multiple destinations (for example, `"disk": true` and `"otlpendpoint": ":4317"` together). +#### Trace your module's methods + +When tracing is enabled, each incoming component or service method call in your module opens its own span, nested under the RPC that triggered it. +Reading a camera, for example, produces a `GetImage` span under the caller's request span, so you can see how long that method takes as part of the whole request. + +The Go, Python, and C++ SDKs all create these per-method spans. +viam-server sets the `VIAM_MODULE_TRACING` environment variable on each module process when tracing is enabled in the machine config. +The SDK inside the module reads that variable to decide whether to record spans. +You enable tracing once in the `tracing` config block above, and individual modules pick it up. + +Go modules record per-method spans automatically. +Python and C++ modules each need one more step, described below. + +#### Enable tracing in a Python module + +Python module tracing is an opt-in extra. +Install `viam-sdk` with the `tracing` extra so the module has the OpenTelemetry packages it needs: + +```sh {class="command-line" data-prompt="$"} +pip install 'viam-sdk[tracing]' +``` + +Add this extra to your module's dependencies (for example, in `requirements.txt`) so it is present wherever the module runs. + +The extra is the only change your module needs. +With the extra installed and tracing enabled in the machine config, each component and service method your module implements emits a span, and the module sends those spans to viam-server. +Without the extra installed, tracing stays off and your module runs unchanged. + +#### Enable tracing in a C++ module + +The C++ SDK creates a per-method span the same way, but only when you build the SDK with OpenTelemetry support (the `VIAMCPPSDK_OPENTELEMETRY_TRACING` build option). +A build without that option compiles the tracing calls as no-ops. +Build the C++ SDK with OpenTelemetry support to record spans from a C++ module. + +To trace a section of your own code inside a method, create a child span with `TracingSpan` from ``: + +```cpp +#include + +ProtoStruct MySensor::get_readings(const ProtoStruct&) { + TracingSpan span("readings implementation"); + span.add_event("computing signal"); + // your logic here + return {{"signal", multiplier_}}; +} +``` + +This span nests under the automatic `GetReadings` span for the call. +If the SDK is built without OpenTelemetry support, `TracingSpan` is a no-op and the code runs unchanged. + +#### Use traces to find latency and failures + +Because each method span nests under the request that called it, a trace shows where time goes across service boundaries. +If a `Move` request is slow, expand its trace to see whether the delay is in the arm's `GetJointPositions` span, a vision service's `GetDetections` span, or the network between them. +A method that returns an error marks its span with an error status, so a failed span points to the method and machine that returned the failure. + +Retrieve and inspect traces with the `viam traces` CLI commands. +See [Traces](/monitor/troubleshoot/#traces) and [Work with traces](/cli/manage-your-fleet/#work-with-traces). + See [Trigger on data events](/data/trigger-on-data/) and [Visualize data](/data/visualize-data/). ## Manage data volume From e036ff2fce95ca67774a5b0975d949c4c85ac50d Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 15 Jul 2026 10:13:00 -0600 Subject: [PATCH 2/4] Style viam-server consistently as code in tracing prose --- docs/data/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/data/overview.md b/docs/data/overview.md index 71ad498204..15efbd9ab0 100644 --- a/docs/data/overview.md +++ b/docs/data/overview.md @@ -127,7 +127,7 @@ When tracing is enabled, each incoming component or service method call in your Reading a camera, for example, produces a `GetImage` span under the caller's request span, so you can see how long that method takes as part of the whole request. The Go, Python, and C++ SDKs all create these per-method spans. -viam-server sets the `VIAM_MODULE_TRACING` environment variable on each module process when tracing is enabled in the machine config. +`viam-server` sets the `VIAM_MODULE_TRACING` environment variable on each module process when tracing is enabled in the machine config. The SDK inside the module reads that variable to decide whether to record spans. You enable tracing once in the `tracing` config block above, and individual modules pick it up. @@ -146,7 +146,7 @@ pip install 'viam-sdk[tracing]' Add this extra to your module's dependencies (for example, in `requirements.txt`) so it is present wherever the module runs. The extra is the only change your module needs. -With the extra installed and tracing enabled in the machine config, each component and service method your module implements emits a span, and the module sends those spans to viam-server. +With the extra installed and tracing enabled in the machine config, each component and service method your module implements emits a span, and the module sends those spans to `viam-server`. Without the extra installed, tracing stays off and your module runs unchanged. #### Enable tracing in a C++ module From 7a52595c75846ea0cf0ce8b5c825a78ebedeba58 Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 15 Jul 2026 10:20:30 -0600 Subject: [PATCH 3/4] Reword tracing enablement to avoid personifying the SDK --- docs/data/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data/overview.md b/docs/data/overview.md index 15efbd9ab0..9926d263fe 100644 --- a/docs/data/overview.md +++ b/docs/data/overview.md @@ -128,7 +128,7 @@ Reading a camera, for example, produces a `GetImage` span under the caller's req The Go, Python, and C++ SDKs all create these per-method spans. `viam-server` sets the `VIAM_MODULE_TRACING` environment variable on each module process when tracing is enabled in the machine config. -The SDK inside the module reads that variable to decide whether to record spans. +The SDK inside the module records spans only when that variable is set. You enable tracing once in the `tracing` config block above, and individual modules pick it up. Go modules record per-method spans automatically. From b332bc6dd431389b99a23152f2f9b71c485a2e9b Mon Sep 17 00:00:00 2001 From: Brandon Shrewsbury Date: Wed, 15 Jul 2026 11:57:41 -0600 Subject: [PATCH 4/4] Expand C++ tracing: build recipe, runtime gate, fuller TracingSpan API, source-build caveat --- docs/data/overview.md | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/data/overview.md b/docs/data/overview.md index 9926d263fe..15569a70fe 100644 --- a/docs/data/overview.md +++ b/docs/data/overview.md @@ -151,25 +151,41 @@ Without the extra installed, tracing stays off and your module runs unchanged. #### Enable tracing in a C++ module -The C++ SDK creates a per-method span the same way, but only when you build the SDK with OpenTelemetry support (the `VIAMCPPSDK_OPENTELEMETRY_TRACING` build option). -A build without that option compiles the tracing calls as no-ops. -Build the C++ SDK with OpenTelemetry support to record spans from a C++ module. +C++ tracing has two requirements: you build the SDK with OpenTelemetry support, and, as with Go and Python, tracing is enabled on `viam-server` in the machine config. +Compiling with OpenTelemetry support alone records nothing until the module runs under a `viam-server` that has tracing enabled; a build without OpenTelemetry support compiles the tracing calls as no-ops. -To trace a section of your own code inside a method, create a child span with `TracingSpan` from ``: +How you enable the build support depends on how you build the SDK: + +- **With Conan:** the `opentelemetry_tracing` option is on by default. + It pulls in `opentelemetry-cpp` with the OTLP gRPC exporter and compiles tracing into the SDK. + To build without tracing, set the option off (`-o opentelemetry_tracing=False`). +- **With CMake directly:** the `VIAMCPPSDK_OPENTELEMETRY_TRACING` option is off by default. + Turn it on with `-DVIAMCPPSDK_OPENTELEMETRY_TRACING=ON`, and provide `opentelemetry-cpp` (built with the OTLP gRPC exporter) as a dependency. + +Because tracing must be compiled in, prebuilt SDK packages and the [cloud build](/cli/build-and-deploy-modules/#cloud-builds) image do not include it. +Build the SDK from source with the option above to record spans from a C++ module. + +To trace a section of your own code inside a method, create a child span with `TracingSpan` from ``. +The span begins when you construct it and ends when it leaves scope, so scope it to the region you want to measure: ```cpp #include ProtoStruct MySensor::get_readings(const ProtoStruct&) { TracingSpan span("readings implementation"); + span.set_attribute("multiplier", multiplier_); // attach structured context span.add_event("computing signal"); // your logic here + span.set_status_ok(); return {{"signal", multiplier_}}; } ``` -This span nests under the automatic `GetReadings` span for the call. -If the SDK is built without OpenTelemetry support, `TracingSpan` is a no-op and the code runs unchanged. +`TracingSpan` also provides `set_status_error()`, `record_exception()` and `record_unknown_exception()` to mark a failure, and `end()` to end the span before it leaves scope. +You cannot copy or move a `TracingSpan`. + +This span nests under the automatic `GetReadings` span for the call, and the SDK exports it over OTLP gRPC to the destination you set in the `tracing` config block above. +If the SDK is built without OpenTelemetry support, every `TracingSpan` call is a no-op and the code runs unchanged. #### Use traces to find latency and failures