diff --git a/docs/build-modules/deploy-a-module.md b/docs/build-modules/deploy-a-module.md index fe00d1ba53..e54c26ab73 100644 --- a/docs/build-modules/deploy-a-module.md +++ b/docs/build-modules/deploy-a-module.md @@ -357,6 +357,15 @@ This means the binary was compiled for the wrong architecture. For example, you {{< /expand >}} +{{< expand "Module fails to start or keeps restarting" >}} + +- Read the module's lifecycle state from [`GetMachineStatus`](/reference/apis/robot/#getmachinestatus). A module in `STATE_UNHEALTHY` reports the cause in its `error` field and the time it entered that state in `last_updated`. +- Check `consecutive_failures`. A count that keeps climbing indicates a restart loop, such as a module failing repeatedly on a syntax error. +- For the state list and field details, see [Module status states](/build-modules/module-reference/#module-status-states). +- Check the module logs in the **LOGS** tab for the full traceback. + +{{< /expand >}} + {{< expand "Module works locally but fails after deployment" >}} - Check for hard-coded paths, missing environment variables, or dependencies installed on your machine but not in the build environment. diff --git a/docs/build-modules/module-reference.md b/docs/build-modules/module-reference.md index 74ea0b4ed6..45bbad0c5f 100644 --- a/docs/build-modules/module-reference.md +++ b/docs/build-modules/module-reference.md @@ -84,6 +84,49 @@ If a module process crashes, `viam-server` automatically restarts it: If the module keeps crashing, `viam-server` retries indefinitely. Check the **LOGS** tab for crash tracebacks. +### Module status states + +While the machine runs, `viam-server` tracks every configured module through a +fixed set of lifecycle states, from download through running. The +[`GetMachineStatus`](/reference/apis/robot/#getmachinestatus) method reports the +current state of each module in a `ModuleStatus` message, so you can tell whether +a module is still downloading, starting up, running, or failing. + +`ModuleStatus.state` is one of the following values. Each maps to what the module +process is doing at that point: + +| State | What the module is doing | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `STATE_UNSPECIFIED` | The state is unknown or unset. | +| `STATE_PENDING` | The module is configured, but `viam-server` has not yet started it, for example because a required package is still downloading. Next state: `STATE_STARTING`. | +| `STATE_STARTING` | The module process has spawned, and `viam-server` is waiting for it to send a `Ready` response and register its models. Next state: `STATE_READY`. | +| `STATE_READY` | The module has sent its `Ready` response, registered its models, and is running. If it exits with an error, it moves to `STATE_UNHEALTHY`. | +| `STATE_UNHEALTHY` | The module failed to start, exited unexpectedly, or is restarting after a crash. Any failure in another state moves to this state. | +| `STATE_CLOSING` | The module process is shutting down, because you removed the module or because it is restarting during a reconfigure. On a restart it moves to `STATE_STARTING`. | + +This per-module state is separate from the machine-level state (`initializing` or +`running`) that `GetMachineStatus` also reports. For how the machine-level state +changes when a module crashes, see [Crash recovery](#crash-recovery). + +Alongside `state`, each `ModuleStatus` reports: + +- `module_name`: the configured name of the module. +- `last_updated`: the time of the module's most recent state transition. Use it to + see how long the module has been in its current state. +- `error`: the error that caused the module to enter `STATE_UNHEALTHY`. The error + persists until the module returns to `STATE_READY`, where it is empty. +- `consecutive_failures`: the number of times the module has entered + `STATE_UNHEALTHY` since it was last ready. This count resets to zero when the + module reaches `STATE_READY` or you reconfigure it. A count that keeps climbing + indicates a module stuck in a restart loop, such as a Python module with a syntax + error. + +To diagnose a module that failed to start, read its `ModuleStatus` from +`GetMachineStatus`. A module in `STATE_UNHEALTHY` reports the cause in `error` and +the time it entered that state in `last_updated`. A rising `consecutive_failures` +count indicates a restart loop, where the module fails and retries repeatedly. For +the full crash traceback, check the **LOGS** tab. + ### Communication By default, modules communicate with `viam-server` over a Unix domain socket