docs(evaluator): showcase custom AgentStreamTranslator hook on the Hermes example by ngoncharenko · Pull Request #750 · NVIDIA-NeMo/nemo-platform

ngoncharenko · 2026-07-17T03:38:26Z

Summary

add a runnable Hermes /v1/responses SSE evaluation using GenericAgent
translate the terminal response.completed frame into ATIF-v1.7 with token metrics
score the extracted response with a deterministic keyword metric
document where response_path is applied to SSE data: frames and how custom translators consume captured frames
cover the request contract, authentication, ATIF evidence, metric pass/fail behavior, and malformed terminal streams with a recorded transcript

Why

Customers evaluating custom streaming agents need a concrete example of keeping the shared HTTP/SSE transport generic while translating application-specific frames into canonical trajectory evidence.

The example validates API_SERVER_KEY before transport, points to the Hermes API-server setup documentation when it is missing, and never embeds or prints the credential.

Validation

example.py is tested on a local Hermes deployment
uv run --frozen pytest packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py -q — 7 passed
scoped ruff check and ruff format --check
scoped ty check
make docs-check
git diff --check origin/main...HEAD

Live Hermes validation was skipped because API_SERVER_KEY is not exported in this environment; the recorded-SSE path validates the complete request and response flow.

Linear

AALGO-353 — GenericAgent Custom Stream Parsing Docs

Summary by CodeRabbit

New Features
- Added customer-facing support for streaming SSE agent evaluations, including response-frame extraction and ATIF-v1.7 translation for candidate outputs.
- Introduced a Hermes Responses SSE example with an async keyword-containment metric and evidence capture.
Documentation
- Updated evaluator architecture and added guidance for “Streaming SSE agents” behavior and configuration.
Bug Fixes
- Improved demo server logging suppression and made trial output reporting resilient to missing outputs.
Tests
- Added end-to-end Hermes SSE tests, metric and translator validation, incomplete-stream error handling, and missing-token failure checks.

github-actions · 2026-07-17T03:49:46Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	27239/34958	77.9%	62.2%
Integration Tests	15999/33670	47.5%	19.9%

coderabbitai · 2026-07-17T04:02:55Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 34335518-b3db-497a-a351-7d591569b701

📥 Commits

Reviewing files that changed from the base of the PR and between 1094538 and 479e9c2.

📒 Files selected for processing (4)

docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
packages/nemo_evaluator_sdk/examples/hermes/example.py
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md
packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py

🚧 Files skipped from review as they are similar to previous changes (3)

packages/nemo_evaluator_sdk/examples/hermes/example.py
docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md

📝 Walkthrough

Walkthrough

Adds Hermes Responses SSE evaluation through GenericAgent, ATIF-v1.7 translation, keyword scoring, streaming documentation, and tests covering successful, incomplete, malformed, and credential-missing flows.

Changes

Hermes SSE evaluation

Layer / File(s)	Summary
Streaming architecture and Hermes flow `packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md`	Documents Hermes SSE parsing, event handling, evidence retention, translation, reporting, and evaluator integration.
Hermes example implementation `packages/nemo_evaluator_sdk/examples/hermes/example.py`	Adds Hermes response translation, keyword scoring, token validation, streaming evaluator setup, and an async entry point.
Example replay and failure validation `packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py`	Tests request configuration, output, scoring, evidence, translation validation, incomplete streams, and missing credentials.
Evaluation usage documentation `docs/evaluator/agent-eval/evaluate-deployed-agent.mdx`	Updates async evaluation examples and adds streaming SSE and Hermes configuration guidance.

Sequence Diagram(s)

sequenceDiagram
  participant GenericAgent
  participant HermesEndpoint
  participant AgentEvaluator
  participant HermesStreamTranslator
  GenericAgent->>HermesEndpoint: Send streaming Responses request
  HermesEndpoint-->>AgentEvaluator: Return SSE frames
  AgentEvaluator->>HermesStreamTranslator: Translate completed response
  HermesStreamTranslator-->>AgentEvaluator: Return trajectory and metrics

Possibly related PRs

NVIDIA-NeMo/nemo-platform#710: Updates the same deployed-agent evaluation documentation and GenericAgent response extraction flow.

Suggested labels: feat

Suggested reviewers: sandychapman

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: showcasing a custom AgentStreamTranslator hook in the Hermes example.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch ngoncharenko/aalgo-353-hermes-custom-stream-parsing

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/evaluator/agent-eval/evaluate-deployed-agent.mdx`:
- Around line 113-160: Add a Fern tab-set around the Streaming SSE agents
example with separate Python SDK and CLI tabs. Keep the existing Python example,
and add the equivalent CLI configuration and invocation for the SSE workflow; if
the CLI cannot accept custom stream translators, explicitly state that
limitation and direct users to the Python example.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5f1a9679-ec94-46b9-9497-b002670a9700

📥 Commits

Reviewing files that changed from the base of the PR and between 2412ab7 and 3c9b387.

📒 Files selected for processing (4)

docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
packages/nemo_evaluator_sdk/examples/hermes/example.py
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md
packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/examples/hermes/example.py`:
- Line 39: Restore the API_SERVER_KEY credential contract by defining and using
API_SERVER_KEY_ENV_NAME instead of HERMES_TOKEN_ENV_NAME. Update
_require_api_server_key() and SecretRef, including the related references, so
they read the API_SERVER_KEY environment variable and expose the expected
constant for missing-key tests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 791674b0-8547-4c74-b3d3-7287a4d87143

📥 Commits

Reviewing files that changed from the base of the PR and between 3c9b387 and 3c671e9.

📒 Files selected for processing (2)

packages/nemo_evaluator_sdk/examples/hermes/example.py
packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md`:
- Line 2: Update the architecture document heading so it begins with the
required H1 title, then retain “GenericAgent streaming with optional
translation” as an H2 with blank lines before and after it to satisfy MD041 and
MD022.
- Around line 467-470: Add a tab set in the architecture documentation section
containing both the existing validated Python SDK configuration and an
equivalent validated CLI configuration for the GenericAgent setup. Preserve the
current Python example, add the required CLI alternative with matching URL and
agent settings, and use the documentation’s established tab-set syntax.
- Around line 525-529: Append a “Next Steps” section to the end of
architecture.md, adding cross-links to related streaming configuration and
reference documentation. Keep the existing architecture content unchanged and
use the repository’s established documentation link style and relevant
destination pages.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 229b67c9-42a7-4f81-add4-1e9ee5da1b4e

📥 Commits

Reviewing files that changed from the base of the PR and between cb8030a and 582e20a.

📒 Files selected for processing (4)

docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
packages/nemo_evaluator_sdk/examples/hermes/example.py
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/docs/architecture.md
packages/nemo_evaluator_sdk/tests/agent_eval/test_hermes_example.py

🚧 Files skipped from review as they are similar to previous changes (2)

docs/evaluator/agent-eval/evaluate-deployed-agent.mdx
packages/nemo_evaluator_sdk/examples/hermes/example.py

arpitsardhana

minor nits.

Signed-off-by: Nick Goncharenko <[email protected]>

…parsing

github-actions Bot added the docs label Jul 17, 2026

ngoncharenko marked this pull request as ready for review July 17, 2026 03:44

ngoncharenko requested review from a team as code owners July 17, 2026 03:44

coderabbitai Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread docs/evaluator/agent-eval/evaluate-deployed-agent.mdx

coderabbitai Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread packages/nemo_evaluator_sdk/examples/hermes/example.py

ngoncharenko force-pushed the ngoncharenko/aalgo-353-hermes-custom-stream-parsing branch from cb8030a to 582e20a Compare July 17, 2026 05:01

ngoncharenko changed the title ~~docs(evaluator): add Hermes SSE agent evaluation example~~ docs(evaluator): showcase custom StreamTranslator hook with Hermes example Jul 17, 2026

coderabbitai Bot reviewed Jul 17, 2026

View reviewed changes

ngoncharenko changed the title ~~docs(evaluator): showcase custom StreamTranslator hook with Hermes example~~ docs(evaluator): showcase custom AgentStreamTranslator hook on the Hermes example Jul 17, 2026

ngoncharenko self-assigned this Jul 17, 2026

ngoncharenko requested review from SandyChapman and arpitsardhana July 17, 2026 18:27

arpitsardhana approved these changes Jul 20, 2026

View reviewed changes

ngoncharenko added 7 commits July 20, 2026 17:04

docs: add hermes eval + stream parsing example

3937117

Signed-off-by: Nick Goncharenko <[email protected]>

fix: align Hermes API key environment variable

0d71a99

Signed-off-by: Nick Goncharenko <[email protected]>

chore: use more realistic example

24d41c8

Signed-off-by: Nick Goncharenko <[email protected]>

fix: fix test

650d3c8

Signed-off-by: Nick Goncharenko <[email protected]>

chore: update architecture

ab0f966

Signed-off-by: Nick Goncharenko <[email protected]>

doc: udpate doc

b5c1e25

Signed-off-by: Nick Goncharenko <[email protected]>

chore: fixes

479e9c2

Signed-off-by: Nick Goncharenko <[email protected]>

ngoncharenko force-pushed the ngoncharenko/aalgo-353-hermes-custom-stream-parsing branch from 3e2fba3 to 479e9c2 Compare July 21, 2026 00:07

ngoncharenko added 2 commits July 24, 2026 15:31

Merge branch 'main' into ngoncharenko/aalgo-353-hermes-custom-stream-…

17ae316

…parsing

Merge branch 'main' into ngoncharenko/aalgo-353-hermes-custom-stream-…

1e733f4

…parsing

ngoncharenko added this pull request to the merge queue Jul 25, 2026

Merged via the queue into main with commit 4496535 Jul 25, 2026
106 of 109 checks passed

ngoncharenko deleted the ngoncharenko/aalgo-353-hermes-custom-stream-parsing branch July 25, 2026 00:32

Uh oh!

Conversation

ngoncharenko commented Jul 17, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Validation

Linear

Summary by CodeRabbit

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arpitsardhana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ngoncharenko commented Jul 17, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading