TDOAK

TouchDesigner extension for OAK cameras via depthai v3.

Contributing or digging into the internals? Read docs/ARCHITECTURE.md — component map, threading model, data flow, lifecycle state machine, and the crash-safety invariants behind the device teardown path.

Supports USB OAK and PoE OAK including OAK-D SR PoE (ToF depth + aligned color) with automatic camera-type detection, plus neural pipelines (object detection, human-pose/skeleton tracking). Pipeline scripts are reloadable DATs. Runtime camera controls (exposure, stereo config, ToF config) can be sent from the main thread at any time while the pipeline is running.

Stream data is decoded off the main thread and pushed into Script Operators. Common streams (rgb, depth, amplitude, detections, pose, imu…) are handled out of the box; any other model's output can be decoded and shaped with a per-stream Handler DAT — see Script OP Integration.

Neural models declared by a pipeline are pre-fetched on a subprocess before the device pipeline is built, so a Model Zoo download never freezes the UI — progress is reported live in the Models sequence. Custom local models (NNArchive/blob on disk) are detected and loaded directly without a download. See Model Pre-fetch.

Requirements

TouchDesigner 2025.33xxx

Quick Start

Install the package into a Python virtual environment used by TouchDesigner (e.g. one managed by TD's Python Environment Manager). The package is not on a public index, so install straight from GitHub (or from a local clone with pip install ):
```
pip install git+https://github.com/TouchDesigner/TDOAK
```
depthai and depthai-nodes are installed as dependencies.
The packaged component (TDOAK.tox) ships inside the package. To load it, create a Base COMP and set its External .tox parameter to the packaged file using an expression:
```
mod.TDOAK.toxPath('TDOAK')
```
Click List Devices to populate the Device menu.
Select a device (or leave as -- Select Device -- to use whichever camera connects first).
Click Start.

The component's Connected, Fps, Activestreams, and Lasterror parameters update every cook. Reference Script TOPs, CHOPs, SOPs, or DATs on the Streams sequential block to receive frame data.

With no pipeline specified, a default pipeline is run with a rgb stream. To see the stream, create a Script TOP, toggle on the Script TOP's Modify Outside of Cook parameter and empty out the onCook callback. Now reference the Script TOP in the TDOAK's Stream parameters. This pushing action is not quite optimal in ways that it happens outside TouchDesigner's cook engine with possible sideeffects. So alternatively, in a Script OP, make use of the TDOAK COMP's FillScriptOp() function to pull frame data into Script OPs.

The examples folder has a range of pipeline scripts and handler scripts.

Component Parameters

Parameter	Type	Description
`Device`	Menu	Target device MXID or IP. `-- Select Device --` uses the first available device. Populated by List Devices.
`Log Level`	Menu	depthai verbosity (`trace`…`critical`), applied to the device on Start. Default `warn` silences the periodic `[info]` chatter (temperatures, cpu usage).
`Start`	Pulse	Start the camera pipeline. Resets the retry counter.
`Stop`	Pulse	Stop the pipeline cleanly.
`Rebuild`	Pulse	Stop and restart (useful after changing parameters).
`List Devices`	Pulse	Enumerate all visible OAK devices and update the Device menu.
`Reload Handlers`	Pulse	Hot-reload the Streams blocks' Handler DATs (`parse`/`fill`) into the running pipeline — applies handler/binding edits without a restart. See Script OP Integration.
`Pipeline DAT`	DAT ref	Optional custom pipeline script. See Pipeline Scripts. Leave empty for the built-in auto-detect pipeline.
`Auto Reload`	Toggle	When enabled, automatically rebuilds when the Pipeline DAT changes.
`Enable Depth`	Toggle	Enable stereo depth output on RGB cameras (ignored for ToF devices).
`Enable IMU`	Toggle	Enable IMU output (accelerometer + gyroscope) on RGB cameras.
`Fps`	Float (read-only)	Measured frame rate, updated once per second.
`Connected`	Toggle (read-only)	`1` while the pipeline is running and connected.
`Active Streams`	String (read-only)	Comma-separated list of active stream names.
`Last Error`	String (read-only)	Most recent error message, or empty if no error.
`Streams` (seq)	Sequential block	Bind stream names to Script Operators. Each block has a `Stream` menu, a `Scriptop` OP reference, and an optional `Handler` DAT reference (custom `parse`/`fill` — see Script OP Integration).
`Models` (seq)	Sequential block (read-only)	Status readout, one block per model the pipeline declares. `Model` is the slug or local path; `Progress` (0–1) tracks the pre-fetch. Populated automatically on Start — see Model Pre-fetch.

`OAKManagerExt` API Reference

All methods listed here are safe to call from the main thread unless otherwise noted.

Lifecycle

`Start() -> None`

Start the camera pipeline. Resets the retry counter to zero (so errors before this call are forgotten). If the pipeline fails, it retries up to 3 times with a 10-second delay between attempts. After 3 failures, Lasterror shows the final error and the pipeline stops retrying — call Start() again to reset.

`Stop() -> None`

Signal the background pump to exit cleanly. The pipeline and device are closed on the background thread (the pump's finally), not in a main-thread hook — device.close()/pipeline.stop() are blocking depthai calls, so keeping them off the main thread avoids freezing the UI for several frames on each stop or retry.

`Rebuild() -> None`

Stop the pipeline and restart it after 5 frames. Equivalent to Stop() followed by a delayed Start(). Triggered automatically when Auto Reload is on and the Pipeline DAT changes.

`ReloadHandlers() -> None`

Re-load the Streams blocks' Handler DATs and hot-swap their parse/fill into the running pipeline — no device or pipeline restart. Call after adding or editing a Streams block, its Scriptop, or its Handler DAT. The worker reads the new parse on its next message; the main thread picks up the new fill on the next cook. Adding a brand-new physical stream still needs a Rebuild() (that's a build() change). Safe to call while stopped — it just primes state for the next Start. Triggered automatically when Auto Reload is on and a Streams block/binding/Handler changes, or manually via the Reload Handlers pulse.

Device Discovery

`ListDevices(callback=None) -> None`

Enumerate all visible OAK devices in a background thread and update the Device parameter menu. Optionally accepts a callback that receives the device list when enumeration completes.

def on_devices(devices):
    for d in devices:
        print(d['name'], d['mxid'], d['protocol'])

oak.ListDevices(callback=on_devices)

`GetDevices() -> list[DeviceInfo]`

Return the last enumerated device list (no network call). Returns an empty list if ListDevices has not been called yet.

DeviceInfo fields:

Key	Type	Description
`mxid`	`str`	Unique device identifier (also used as the menu name).
`name`	`str`	Human-readable device name.
`state`	`str`	Connection state, e.g. `UNBOOTED`, `BOOTLOADER`, `FLASH_BOOTED`.
`protocol`	`str`	Transport protocol, e.g. `USB`, `TCP`.

Frame Data

Frames are written by the background pump thread and read-locked for safe access from cook callbacks.

Streams are decoded on demand: the worker skips decoding a stream that has no Streams-block/registry binding and has never been pulled via GetFrame (its queue is still drained, so the device never backs up; rgb is always decoded — it drives the Fps readout). The first GetFrame(name) on such a stream registers interest and may return None until the next frame arrives.

`GetFrame(name: str) -> FrameData | None`

Return the latest frame for the given stream name, or None if no frame has arrived yet.

FrameData is one of:

numpy.ndarray — image, mask, or point-cloud data (see format notes below)
list[DetectionEntry] — NN object-detection results (dai.ImgDetections)
list[PoseEntry] — NN pose / skeleton results (dai.ImgDetections with keypoints)
list[ExtendedDetectionEntry] — depthai-nodes ImgDetectionsExtended (rotated box + keypoints)
list[ClassificationEntry] — depthai-nodes Classifications
list[list[float]] — keypoints [[x, y, score], …] or lines [[x1, y1, x2, y2, conf], …]
list[list[list[float]]] — clusters [[[x, y], …], …]
ImuData — accelerometer + gyroscope packet
any value returned by a custom Handler parse (see Script OP Integration)

`GetFrameNames() -> list[str]`

Return the stream names for which at least one frame has been received.

Convenience getters

oak.GetRgb()      # → ndarray (H, W, 3) uint8 BGR, or None
oak.GetDepth()    # → ndarray (H, W) uint16, or None
oak.GetIrLeft()   # → ndarray (H, W) uint8, or None
oak.GetIrRight()  # → ndarray (H, W) uint8, or None
oak.GetImu()      # → ImuData, or None
oak.GetPose()     # → list[PoseEntry], or None

Frame formats by stream:

Stream	dtype	Shape	Notes
`rgb`	`uint8`	`(H, W, 3)`	BGR channel order, vertically flipped to TD convention
`depth`	`uint16`	`(H, W)`	Millimetres
`amplitude`	`uint16`	`(H, W)`	ToF amplitude (signal strength)
`color`	`uint8`	`(H, W, 3)`	BGR, aligned to ToF depth perspective
`ir_left` / `ir_right`	`uint8`	`(H, W)`	Rectified infrared frames
`pointcloud`	`float32`	`(H, W, 3)`	Organized XYZ in mm (R=X, G=Y, B=Z) — drops into a 32-bit float TOP
`segmentation`	`uint16`	`(H, W)`	depthai-nodes `SegmentationMask` class indices (background → 0), flipped to TD convention — drops into a TOP
`imu`	—	—	`ImuData` dict, not an ndarray
`detections`	—	—	`list[DetectionEntry]`, not an ndarray
`pose`	—	—	`list[PoseEntry]`, not an ndarray

The depth stream is uint16 millimetres for the stereo and ToF pipelines, but float32 relative inverse depth ([0,1], larger = nearer) for mono_depth_pipeline.py, since that comes from a neural Map2D rather than a metric sensor.

Stream names above are conventions, not requirements — GetFrame(name) keys off whatever names your pipeline's build() returns. The dtype/shape is determined by the decoded message type, not the name.

DetectionEntry fields:

Key	Type
`label`	`int`
`confidence`	`float` (0–1)
`xmin` / `ymin` / `xmax` / `ymax`	`float` (0–1, normalised)

PoseEntry fields (one per detected person):

Key	Type	Description
`confidence`	`float` (0–1)	Detection confidence
`box`	`list[float]`	`[xmin, ymin, xmax, ymax]`, normalised 0–1
`keypoints`	`list[list[float]]`	Joint positions `[[x, y], …]`, normalised 0–1 (COCO-17 order for the bundled pose model)
`edges`	`list[list[int]]`	Skeleton bone connections `[[i, j], …]` indexing into `keypoints`

ExtendedDetectionEntry fields (depthai-nodes ImgDetectionsExtended, one per detection):

Key	Type	Description
`label`	`int`	Class index
`confidence`	`float` (0–1)	Detection confidence
`box`	`list[float]`	Axis-aligned `[xmin, ymin, xmax, ymax]` outer bounds of the rotated box, normalised 0–1
`angle`	`float`	Rotated-box rotation in degrees (`0` for upright boxes)
`keypoints`	`list[list[float]]`	`[[x, y, score], …]`, normalised 0–1; empty when the model emits none

ImgDetectionsExtended may also carry a message-level segmentation mask. It isn't part of the default payload — read it in a custom parse if you need it.

ClassificationEntry fields (depthai-nodes Classifications, one per class):

Key	Type	Description
`label`	`str`	Class name
`score`	`float` (0–1)	Class probability

Bare lists — Keypoints → [[x, y, score], …]; Lines → [[x1, y1, x2, y2, confidence], …]; Clusters → [[[x, y], …], …] (all normalised 0–1).

ImuData fields:

Key	Type	Description
`accel`	`list[float]`	`[x, y, z]` m/s²
`gyro`	`list[float]`	`[x, y, z]` rad/s

Camera Intrinsics

Intrinsics are read from the device calibration after pipeline.start() and cached for the lifetime of the connection. They are cleared when the pipeline stops.

`GetIntrinsics(socket: str = 'CAM_A') -> dict[str, float] | None`

Return the intrinsic matrix coefficients for the given camera socket at its native sensor resolution. Returns None if not yet connected or if the socket has no calibration.

k = oak.GetIntrinsics('CAM_A')
# → {'fx': 461.3, 'fy': 461.3, 'cx': 320.0, 'cy': 240.0, 'width': 640.0, 'height': 480.0}

k = oak.GetIntrinsics('CAM_B')
# → {'fx': 855.1, 'fy': 856.2, 'cx': 638.4, 'cy': 403.7, 'width': 1280.0, 'height': 800.0}

width/height are the native sensor resolution the coefficients refer to. When reprojecting a stream delivered at a different resolution, scale fx/cx by frameW / width and fy/cy by frameH / height.

Socket names match dai.CameraBoardSocket member names: 'CAM_A', 'CAM_B', 'CAM_C'.

Point cloud reconstruction from ToF depth:

import numpy as np

k     = oak.GetIntrinsics('CAM_A')
depth = oak.GetDepth()          # (H, W) uint16, mm

if k and depth is not None:
    h, w = depth.shape
    u, v = np.meshgrid(np.arange(w), np.arange(h))
    z = depth.astype(np.float32) / 1000.0       # metres
    x = (u - k['cx']) * z / k['fx']
    y = (v - k['cy']) * z / k['fy']
    points = np.stack([x, y, z], axis=-1)       # (H, W, 3)

Runtime Controls

Pipeline nodes expose named input queues that accept depthai message objects. These queues are non-blocking (maxSize=1): if a new message arrives before the device has consumed the previous one, the previous message is silently dropped. This is intentional — for real-time parameter control you always want the latest value, not a stale queue.

InputQueue.send() is thread-safe; SendControl may be called from TD's main thread, an Execute DAT, or a panel callback without coordination.

`SendControl(name: str, message: object) -> bool`

Send a depthai control or config message to the named input queue. Returns True on success, False if the queue name is not registered for the active pipeline (e.g. the pipeline is not running, or that control was not declared by the pipeline script).

import depthai as dai

# Manual exposure
ctrl = dai.CameraControl()
ctrl.setManualExposure(10000, 400)          # exposure_us, iso
oak.SendControl('cam_ctrl', ctrl)

# Auto exposure
ctrl = dai.CameraControl()
ctrl.setAutoExposureEnable()
oak.SendControl('cam_ctrl', ctrl)

# Auto white balance
ctrl = dai.CameraControl()
ctrl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.AUTO)
oak.SendControl('cam_ctrl', ctrl)

`SendFrame(name: str, source: TOP | np.ndarray, imgType=None) -> bool`

Send a host image into a named input queue, so the device runs a pipeline on TD-supplied frames instead of an OAK camera. source is any TOP (read via .numpyArray()) or an (H, W, 3|4) numpy array; it's coerced to a top-down uint8 BGR dai.ImgFrame (float TOPs are treated as normalized 0–1). Returns False if the queue isn't registered. Main-thread only — call it once per cook, e.g. from an Execute/CHOP-Execute callback. See src/TDOAK/examples/detection_input_pipeline.py, which runs YOLO object detection on whatever TOP you feed it (an on-device ImageManip letterboxes/repacks to the model input, so the source can be any size).

# in an Execute DAT's onFrameStart, or a CHOP Execute callback:
op('TDOAK').SendFrame('image_in', op('moviefilein1'))

The matching pipeline declares the queue with node.someInput.createInputQueue(...) and returns it in the controls dict — SendFrame and SendControl target the same queue registry.

`GetControlNames() -> list[str]`

Return the names of all registered input queues for the currently running pipeline. Returns an empty list when no pipeline is active.

print(oak.GetControlNames())
# e.g. → ['cam_ctrl', 'stereo_cfg']

Built-in control queues by pipeline:

Pipeline	Control name	Message type	Notes
Built-in RGB	`cam_ctrl`	`dai.CameraControl`	Exposure, focus, WB, etc.
Built-in RGB + depth	`cam_ctrl`	`dai.CameraControl`
Built-in RGB + depth	`stereo_cfg`	`dai.StereoDepthConfig`	LR-check, subpixel, etc.
Built-in ToF	`tof_cfg`	`dai.ToFConfig`	ToF base configuration
`src/TDOAK/examples/pipeline.py`	`cam_ctrl`, `stereo_cfg`	same as above
`src/TDOAK/examples/tof_showcase_pipeline.py`	`tof_cfg`	`ToFConfig`
`src/TDOAK/examples/detection_input_pipeline.py`	`image_in`	`ImgFrame` (via `SendFrame`)	host-frame input, no camera

Script OP Integration

Stream data reaches a Script Operator in two stages, split across threads:

decode — turn the (model-specific) depthai message into plain, thread-safe data. Runs on the background pump thread, so it never blocks cooking/rendering.
fill — write that data into a Script TOP/CHOP/SOP/DAT. Runs on the main thread, where TD object access is legal.

Built-in decoders cover images (rgb, depth, amplitude, color, ir_left/right), pointcloud, imu, dai.ImgDetections (→ detections/pose), and the depthai-nodes message family — Map2D (monocular depth / heatmaps → float32 (H, W)), ImgDetectionsExtended, Keypoints, SegmentationMask, Classifications, Lines, and Clusters — so common streams need no code at all. Decoding keys off the message type, not the stream name, so these work whatever you name the stream. For any other model output, attach a per-stream Handler DAT (its parse overrides the default decode).

These are zero-config defaults, not a fixed contract. A model-specific format (different keypoint order, custom tensor layout, etc.) is still handled the normal way: a Handler parse returns whatever structure you like and a fill shapes it into the Script OP.

Per-stream Handler DAT

Point a Streams block's Handler parameter at a DAT. It may define either or both of:

# `dai`, `np`, and `oak` (the OAK component) are already in scope. Use `oak` to reach
# the promoted API from a fill (e.g. oak.GetIntrinsics(...)) — fill runs inside the
# extension, where a bare op('...') would resolve inside the component itself.

def parse(msg):
    # BACKGROUND THREAD — no op()/me/TD access. depthai message -> plain data
    # (numpy array, or list/dict of primitives). Return None to skip this frame.
    return msg.getTensor('output')

def fill(scriptOp, data):
    # MAIN THREAD — full TD access. plain data -> the bound operator.
    scriptOp.clear()
    ...

parse overrides the built-in decoder for that stream; omit it to keep the default decode.
fill overrides the default push; omit it to use the default (numpy → Script TOP).
Handlers are loaded on Start, and hot-reload into a running pipeline when you add/edit a Streams block, its Scriptop, or its Handler DAT — automatically while Auto Reload is on, or on demand via the Reload Handlers pulse / ReloadHandlers(). No device restart is needed; only adding a brand-new physical stream (a build() change) requires a Rebuild. A failing handler reports to Lasterror and is skipped — it never stalls the pump.

Binding an existing ndarray stream to a plain Script TOP (no Handler) updates live on its own — the sequence is read every cook. Anything needing a custom parse/fill (Script CHOP/SOP/DAT, or a non-default decode) is what the hot-reload applies.

Push vs pull, by operator family

Script POPs, TOPs, CHOPs, and DATs support Modify Outside of Cook, so the extension writes into them directly each cook. Script SOPs build geometry inside their cook, so they have to pull instead:

Operator	How it's filled	Cook callback needed
Script TOP	pushed — default `copyNumpyArray`, or your `fill`	No
Script CHOP	pushed — your `fill` (e.g. `copyNumpyArray((chans, samples) float32)`)	No
DAT	pushed — your `fill` (e.g. `clear()` + `appendRows([...])`)	No
Script POP	pushed — your `fill` (staging API works outside the cook)	No
Script SOP	pulled — your `fill` runs inside the cook	Yes — `FillScriptOp(me)`

For a Script SOP, add a one-line cook callback:

def onCook(scriptOp):
    op('TDOAK').FillScriptOp(scriptOp)

Any Script OP can also pull without any binding at all — pass the stream (and optionally a Handler DAT) directly; see the FillScriptOp reference below:

def onCook(scriptOp):
    op('TDOAK').FillScriptOp(scriptOp, stream='rgb')

Binding streams to operators

Two ways to bind, both honouring the Handler/push/pull rules above:

Sequential block (UI) — add a Streams block, set Stream, point Scriptop at the operator, and (optionally) Handler at a DAT.

Code registry — bind from a script instead of the UI:

oak.RegisterScriptop('depth', op('script_depth'))   # multiple OPs per stream OK
oak.UnregisterScriptop('depth', op('script_depth')) # remove one
oak.UnregisterScriptop('rgb')                        # remove all for the stream

Examples

Default Script TOP (no Handler — automatic): just bind rgb to a Script TOP; the extension pushes it. No cook code required.

Pose skeleton into a Script SOP — a fill-only Handler DAT on the pose block (decode stays the built-in PoseEntry):

def fill(scriptOp, people):          # people: list[PoseEntry]
    scriptOp.clear()
    if not people:
        return
    for person in people:
        pts = [scriptOp.appendPoint() for _ in person['keypoints']]
        for p, (x, y) in zip(pts, person['keypoints']):
            p.x, p.y, p.z = x, 1.0 - y, 0.0     # y-flip: rgb is flipud'd for display
        for i, j in person['edges']:
            prim = scriptOp.appendPoly(2, closed=False, addPoints=False)
            prim[0].point, prim[1].point = pts[i], pts[j]

…with op('TDOAK').FillScriptOp(me) in the SOP's onCook.

`RegisterScriptop(streamName: str, scriptop: OP) -> None`

Bind a Script Operator to receive frames for streamName. Multiple OPs can be registered for the same stream.

`UnregisterScriptop(streamName: str, scriptop: OP | None = None) -> None`

Remove a registered binding. Pass scriptop=None to remove all bindings for the stream.

`RegisterComponent(scriptop: OP, streamName: str | None = None, handlerDat: OP | None = None) -> None`

One-call wiring for a self-contained stream component: binds the scriptop to its stream and ingests its Handler DAT. streamName/handlerDat default to Stream/Handler custom parameters on the scriptop's parent COMP, so a wrapper component wires itself with RegisterComponent(op('script1')). Call on init and whenever the component's Stream/Handler parameter or the handler text changes — not per cook. Re-binds cleanly across a Stream change (the op is cleared from any prior stream first).

`UnregisterComponent(scriptop: OP) -> None`

Tear-down counterpart (call e.g. on component delete): unbinds the op from every stream and drops its fill. A stream's parse is left in place, since it's shared per physical queue — Rebuild/ReloadHandlers re-derives it.

`RegisterHandler(scriptop: OP, streamName: str, handlerDat: OP | None) -> None` / `UnregisterHandler(scriptop: OP, streamName: str | None = None) -> None`

Lower-level halves of RegisterComponent: ingest (or drop) a Handler DAT for one operator — its fill is keyed by the op's path, its parse installed per stream (last registration wins; parse is shared across all consumers of that stream). Usually you want RegisterComponent instead.

`FillScriptOp(scriptop: OP, stream: str | None = None, handler: OP | str | None = None) -> bool`

Pull entry point: call from a Script Operator's cook callback to fill it inside its cook. Three ways to use it, from most to least wiring:

def onCook(scriptOp):
    op('TDOAK').FillScriptOp(scriptOp)                    # 1. registered binding
    op('TDOAK').FillScriptOp(scriptOp, stream='rgb')      # 2. explicit stream
    op('TDOAK').FillScriptOp(scriptOp, stream='depth',    # 3. explicit stream + handler
                             handler=op('text_pointcloudHandler'))

No arguments — the operator's registered binding is used (a Streams block or RegisterScriptop; the fill is keyed by the operator's path). This is the classic Script-SOP form, since SOPs can't be written outside their cook.
stream='name' — pull that stream explicitly with zero registration: no Streams block, no registry entry. The first call marks the stream as wanted, so the worker starts decoding it even if nothing else is bound to it (data then arrives on the next frames). Works for any Script OP family, not just SOPs — useful for ad-hoc consumers that shouldn't appear in the component's UI.
handler= — a Handler DAT (or its path) to apply on top of an explicit stream: its fill(scriptOp, data) runs against the stream's latest frame, and if it defines a parse(msg) that parse is installed for the stream's worker-side decode. The DAT is re-exec'd only when its text changes, so calling this every cook is cheap.

When no fill is available (neither registered nor passed), built-in fallbacks apply: an ndarray frame into a Script TOP (copyNumpyArray), a float point array into a Script POP. Returns True if the operator was filled; False means the stream is unknown/unbound, no frame has arrived yet (normal during the first cooks after Start), or no fill/fallback matched. A fill that raises reports to Lasterror and returns False — it never stalls the cook.

`GetFrameForScriptop(scriptop: OP) -> FrameData | None`

Look up the raw frame for the given Script OP (registry + sequential block). A lower-level pull alternative to FillScriptOp when you want the data rather than auto-fill.

def cook(scriptOp):
    data = op('TDOAK').GetFrameForScriptop(me)

Status

`GetStatus() -> StatusInfo`

Return a snapshot of the current pipeline state.

StatusInfo fields:

Key	Type	Description
`connected`	`bool`	Whether the pipeline is running.
`fps`	`float`	Measured frame rate (updated once per second, driven by the `rgb` stream).
`latency_ms`	`float`	Frame latency in milliseconds (device timestamp vs host receipt).
`error`	`str`	Last error message, or empty string.
`streams`	`list[str]`	Active stream names.

Pipeline Scripts

Drop a Python DAT into the Pipeline DAT parameter to replace the built-in auto-detect pipeline with a custom one. Enable Auto Reload to have the pipeline rebuild automatically whenever the DAT changes.

Contract

The DAT must define a build function:

def build(pipeline: dai.Pipeline) -> tuple[
    dict[str, dai.DataOutputQueue],   # output streams  (required)
    list[str],                         # stream names    (required)
    dict[str, dai.InputQueue],         # control inputs  (optional)
]:

The third element is optional — returning a 2-tuple is also accepted (no control queues will be registered).

Output queues: keys become the stream names available via GetFrame(name).
Stream names: the ordered list used to populate the Stream menus on sequential blocks.
Control queues: keys become the names accepted by SendControl(name, msg). Create these with node.someInput.createInputQueue(maxSize=1, blocking=False).

Minimal example

import depthai as dai

def build(pipeline):
    cam = pipeline.create(dai.node.Camera).build()
    out = cam.requestOutput((1280, 720), type=dai.ImgFrame.Type.BGR888p)

    queues   = {'rgb': out.createOutputQueue()}
    controls = {'cam_ctrl': cam.inputControl.createInputQueue(maxSize=1, blocking=False)}

    return queues, ['rgb'], controls

Model Pre-fetch

Resolving a neural model inline (the Model Zoo's online freshness check + download, or loading an archive) is largely Python-level work that holds the GIL — done on the camera pump it still froze TouchDesigner for several seconds, because the worker and TD share one interpreter. So declared models are fetched in a separate process (its own GIL) before the device pipeline is built, and progress streams back into the read-only Models sequence while the UI stays responsive.

Declaring models

A pipeline script declares the models it uses with a module-level MODELS list (a bare MODEL = '...' is also accepted). This is read on Start by statically parsing the script (ast) — it is never executed, so the scan has no side effects. The declaration must be a literal or reference other module-level literal assignments (MODELS = [MODEL_A, MODEL_B] works); a computed value is skipped and build() then resolves it inline.

MODEL  = 'luxonis/yolov8-nano-pose-estimation:coco-512x384'
MODELS = [MODEL]            # one entry per model build() resolves

Each entry is classified automatically:

HubAI zoo slug (anything that isn't a local file) → downloaded via getModelFromZoo in the pre-fetch subprocess, into the shared cache (.depthai_cached_models/, set with DEPTHAI_ZOO_CACHE_PATH). Progress (depthai's JSON download bytes) is mirrored into the block's Progress 0→1.
Local file (a path that resolves, absolute or relative to project.folder) → an NNArchive/blob already on disk. No download; the block shows 1.0 (ready) immediately.

After a successful pre-fetch of a zoo slug, the in-process build() runs with DEPTHAI_ZOO_INTERNET_CHECK=0 so it loads the cached archive instead of repeating the online check. An all-local pipeline leaves the check on, so any incidental zoo resolve still works.

The subprocess uses the project's .venv interpreter (/.venv/Scripts/python.exe, falling back to TD's Python); that environment must have depthai installed.

Declaring MODELS is optional. With none declared, build() resolves models itself inline — simplest, but that's the path that briefly hangs the UI.

Custom & converted models

Pass a local model straight to the parser node — ParsingNeuralNetwork.build() accepts a dai.NNArchive and skips the zoo entirely:

MODELS = ['models/mymodel.tar.xz']     # resolves to a file → classified local, no download

def build(pipeline):
    import depthai as dai
    from depthai_nodes.node import ParsingNeuralNetwork
    cam     = pipeline.create(dai.node.Camera).build()
    archive = dai.NNArchive(f'{project.folder}/models/mymodel.tar.xz')
    nn      = pipeline.create(ParsingNeuralNetwork).build(cam, archive)
    return ({'rgb': nn.passthrough.createOutputQueue(maxSize=1, blocking=False),
             'out': nn.out.createOutputQueue(maxSize=1, blocking=False)},
            ['rgb', 'out'])

A bare blob without parser metadata won't work with ParsingNeuralNetwork (it needs the NNArchive config to pick parsers) — use a plain dai.node.NeuralNetwork for those.

Conversion of a raw model (ONNX/OpenVINO/TFLite → NNArchive for RVC2/RVC4, or a legacy MyriadX blob via blobconverter for RVC2 only) is a one-time offline step done with Luxonis tooling (Hub / modelconverter). It can take minutes — never run it inside build() on the Start path. Commit the resulting NNArchive and reference it as a local entry in MODELS.

Bundled pipeline scripts

All paths relative to src/TDOAK/examples/ (resolve at runtime with mod.TDOAK.examplesPath()).

File	Device	Streams	Controls
`pipeline.py`	OAK-D (RGB + stereo depth, synced)	`rgb`, `depth`	`cam_ctrl`, `stereo_cfg`
`tof_showcase_pipeline.py`	OAK-D SR PoE (ToF depth + amplitude; `rgb` when a color socket exists; organized point cloud opt-in via `STREAM_POINTCLOUD`)	`rgb`, `depth`, `amplitude` (+ `pointcloud`)	`tof_cfg`
`yolo_pipeline.py`	Any RGB OAK (YOLOv6-nano object detection)	`rgb`, `detections`	—
`detection_input_pipeline.py`	Any OAK (YOLO detection on host frames via `SendFrame` — no camera)	`rgb`, `detections`	`image_in`
`pose_pipeline.py`	Any RGB OAK (YOLOv8-nano pose / skeleton — needs `depthai-nodes`)	`rgb`, `pose`	—
`mono_depth_pipeline.py`	Any RGB OAK (MiDaS v2.1 monocular depth — needs `depthai-nodes`)	`rgb`, `depth`	—
`face_pipeline.py`	Any RGB OAK (YuNet face detection + 5 landmarks — needs `depthai-nodes`)	`rgb`, `faces`	—
`hand_pipeline.py`	Any RGB OAK (MediaPipe two-stage hand tracking — needs `depthai-nodes`; pair with `hand_handler.py`)	`rgb`, `hands`	—
`classification_pipeline.py`	Any RGB OAK (EfficientNet-Lite0 classification — needs `depthai-nodes`)	`rgb`, `classes`	—
`segmentation_pipeline.py`	Any RGB OAK (DeepLabV3+ person segmentation — needs `depthai-nodes`)	`rgb`, `segmentation`	—

Bundled Handler DATs

File	Stream	What it does
`pointcloud_handler.py`	`depth`	fill-only — reprojects the depth map through the device calibration (`GetIntrinsics`) into a metric point cloud in a Script POP. Pairs with `pipeline.py`.
`face_handler.py`	`faces`	fill-only — draws the detected face landmark points into a Script POP.
`hand_handler.py`	`hands`	`parse` + `fill` — decodes the GatheredData message, back-maps each hand's crop-local keypoints onto the full frame, and draws 21-point skeletons into a Script SOP. Required by `hand_pipeline.py`.
`classification_handler.py`	`classes`	fill-only — writes the top results into a Table DAT (`label \| score` rows, sorted).
`segmentation_handler.py`	`segmentation`	fill-only — shapes the class mask into silhouette points in a Script POP.
`pose_handler.py`, `skeleton_handler.py`	—	blank `parse`/`fill` templates to start a custom Handler from.

Threading Model

The camera pump runs in a background thread managed by TD's ThreadManager. Frame data is transferred to the main thread via a threading.Lock-protected dictionary.

Background thread (cameraPump): opens the device, pre-fetches declared models on a subprocess (streaming download progress into shared state — see Model Pre-fetch), builds and starts the pipeline, polls output queues in a tight loop, decodes each message (built-in decoder or a Handler's parse), and writes the plain result to the shared dict. On exit (any path) it closes the device in its finally, keeping that blocking call off the main thread.
Main thread (onRefresh): called by the Thread Manager each cook, reads shared state, updates TD parameters, mirrors per-model download progress into the Models sequence, and fills Script OPs (push for TOP/CHOP/DAT, pull for SOP via FillScriptOp).
Model pre-fetch subprocess: spawned from the background thread with its own interpreter, so the GIL-holding zoo download/check runs without blocking either TD thread. It reports progress as JSON on stdout, which the background thread parses and the main thread reflects.
SendControl: calls InputQueue.send() directly from the main thread. depthai input queues are designed for cross-thread use; no intermediate queue is needed.

The background thread must not access any TD objects (op(), me, run(), parameters, etc.) — this is why a Handler's parse (worker thread) must stay pure-Python/numpy while fill (main thread) does the operator I/O. Handler DAT text is read on the main thread (it's TD access) and the resulting parse/fill callables are stored on the extension; the worker reads self._parsers fresh per message rather than capturing it, so ReloadHandlers() can swap the whole dict on the main thread (an atomic reference reassignment) and the running worker picks it up with no restart and no lock. All TD access happens exclusively in the Thread Manager hooks.

Releasing

The package version lives in one place: __version__ in src/TDOAK/__init__.py (pyproject.toml reads it dynamically). To publish a release:

Bump __version__ (e.g. "0.1.2"), commit, push.
Tag and push the tag:
```
git tag v0.1.2
git push origin v0.1.2
```

The Release GitHub workflow triggers on v* tags, verifies the tag matches __version__ (mismatch fails the run), builds sdist + wheel, and publishes them to a GitHub Release with generated notes. CI (ci.yml) additionally builds and smoke-tests the wheel on every push.

Name		Name	Last commit message	Last commit date
Latest commit History 28 Commits
.github/workflows		.github/workflows
docs		docs
src/TDOAK		src/TDOAK
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
TDOAK.toe		TDOAK.toe
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

TDOAK

Requirements

Quick Start

Component Parameters

OAKManagerExt API Reference

Lifecycle

Start() -> None

Stop() -> None

Rebuild() -> None

ReloadHandlers() -> None

Device Discovery

ListDevices(callback=None) -> None

GetDevices() -> list[DeviceInfo]

Frame Data

GetFrame(name: str) -> FrameData | None

GetFrameNames() -> list[str]

Convenience getters

Camera Intrinsics

GetIntrinsics(socket: str = 'CAM_A') -> dict[str, float] | None

Runtime Controls

SendControl(name: str, message: object) -> bool

SendFrame(name: str, source: TOP | np.ndarray, imgType=None) -> bool

GetControlNames() -> list[str]

Script OP Integration

Per-stream Handler DAT

Push vs pull, by operator family

Binding streams to operators

Examples

RegisterScriptop(streamName: str, scriptop: OP) -> None

UnregisterScriptop(streamName: str, scriptop: OP | None = None) -> None

RegisterComponent(scriptop: OP, streamName: str | None = None, handlerDat: OP | None = None) -> None

UnregisterComponent(scriptop: OP) -> None

RegisterHandler(scriptop: OP, streamName: str, handlerDat: OP | None) -> None / UnregisterHandler(scriptop: OP, streamName: str | None = None) -> None

FillScriptOp(scriptop: OP, stream: str | None = None, handler: OP | str | None = None) -> bool

GetFrameForScriptop(scriptop: OP) -> FrameData | None