Skip to content

fix(evaluator-sdk): align Fabric runtime calls to current nemo-fabric API (enable_relay, adapter ids)#844

Draft
SandyChapman wants to merge 1 commit into
fabric-published-wheels/schapmanfrom
fabric-runtime-api-alignment/schapman
Draft

fix(evaluator-sdk): align Fabric runtime calls to current nemo-fabric API (enable_relay, adapter ids)#844
SandyChapman wants to merge 1 commit into
fabric-published-wheels/schapmanfrom
fabric-runtime-api-alignment/schapman

Conversation

@SandyChapman

Copy link
Copy Markdown
Contributor

Summary

Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis). They were masked because every fabric unit test fakes nemo_fabric, so the fakes kept encoding the old API.

L7 — enable_relay dropped a keyword and changed types

runtime.py called cfg.enable_relay(output_dir=..., config=self._relay_config(...)), but current Fabric's signature is enable_relay(*, project=None, output_dir=None, observability=..., components=..., policy=None) — there is no config=, and it wants Fabric's own RelayObservabilityConfig / RelayAtifConfig / RelayAtofConfig.

_relay_config now builds RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...)) from Fabric's typed models and passes it via enable_relay(observability=...) — the SDK only configures ATIF/ATOF observability, so observability= is the semantically clean target. This mirrors nemo-fabric's own Harbor integration (Fabric owns the schema).

L8 — harness adapter ids dropped their .cli/.sdk suffixes

Current nemo-fabric registers bare nvidia.fabric.codex / nvidia.fabric.hermes (transport moved to runtime.transport, which the code already sets). Swept the retired suffixes across the SDK runtime docstring, examples, the notebook, skill-eval, and the hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests (regenerated openapi.yaml).

Why the drift wasn't caught + the guard

The hermetic tests fake nemo_fabric, so they can't catch upstream API drift. #778's fabric-wheel-smoke job installs the real wheels but only did a bare import — it never touched enable_relay or adapter resolution, the exact blind spot.

  • Added test_fabric_surface.py: a real-nemo_fabric contract test (importorskip everywhere Fabric isn't installed) that drives the runtime's own call sites — _compose_configenable_relay(observability=...), and bare-id resolution through Fabric().plan().
  • Wired the fabric-wheel-smoke job to run it, and added a fabric path filter so the job also fires on runtime/test edits (not just dependency bumps — though a Fabric wheel bump is itself a uv.lock/deps change).

Validated locally against a real Fabric build (script/dev-install-fabric.sh): both assertions pass, and the pre-fix forms fail as expected (old config=TypeError, old nvidia.fabric.codex.cliFabricConfigError).

Stacking / gate

Out of scope (flagged)

_common.py (the containerized runtime) still emits the old component-wrapped relay telemetry for the Fabric CLI profile path — a separate mechanism gated behind its own image's Fabric version, not the enable_relay path. Worth a follow-up to confirm newer Fabric's CLI relay parser still accepts that shape.

Test plan

  • pytest packages/nemo_evaluator_sdk/tests/agent_eval/ -k fabric — 79 passed, 1 skipped
  • pytest plugins/nemo-evaluator/tests/test_agent_evaluate.py — 38 passed
  • test_fabric_surface.py against a real Fabric build — 2 passed
  • ruff + CI ty + lint-sdk-vendored (mirror in sync)

… API (enable_relay, adapter ids)

Two call sites in the Fabric agent-eval runtime drifted against a newer
nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked
because every fabric unit test fakes nemo_fabric.

L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants
its own typed relay models. `_relay_config` now builds
`RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))`
from nemo_fabric's own types and passes it via `enable_relay(observability=...)`,
mirroring Fabric's Harbor integration (Fabric owns the schema).

L8 — harness adapter ids: current nemo-fabric registers bare
`nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to
runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK
runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the
nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec.

Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric
contract test (importorskip elsewhere) that exercises the runtime's own call
sites — enable_relay's observability shape and bare-id resolution via the
planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import
could not catch either drift). A new `fabric` path filter triggers that job on
runtime/test edits as well as wheel bumps.

Stacks on the Fabric-packaging work (#778); shares the same non-prerelease
Fabric gate. The fabric unit tests are hermetic and validate the call shape;
live ATIF relay capture needs a real Fabric + nemo-relay gateway and was not
CI-verified (the smoke job installs wheels but runs no agent).

Signed-off-by: Sandy Chapman <[email protected]>
@github-actions github-actions Bot added the fix label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant