Skip to content

fix(evaluator): preserve non-string stream values#824

Draft
ngoncharenko wants to merge 1 commit into
mainfrom
6486650-fix-nat-stream-token-truncation/ngoncharenko-patch
Draft

fix(evaluator): preserve non-string stream values#824
ngoncharenko wants to merge 1 commit into
mainfrom
6486650-fix-nat-stream-token-truncation/ngoncharenko-patch

Conversation

@ngoncharenko

Copy link
Copy Markdown
Contributor

Summary

  • preserve a lone non-string SSE value under concat instead of coercing it with str()
  • continue concatenating string token deltas in arrival order
  • reject mixed string/non-string or multi-frame non-string concat streams as partial results with an actionable error
  • regenerate the vendored Python SDK mirror

Root cause

PR #812 changed NAT stream aggregation to concat. Its accumulator converted every matched non-string value to a Python string representation.

For example:

SSE:      data: {"value": {"answer": 7}}
Before:   "{'answer': 7}"
After:    {"answer": 7}

A single structured result is not a token delta, so this patch preserves its original JSON type. Text deltas such as ["15", ".", "0"] still become "15.0".

Validation

  • uv run --frozen pytest packages/nemo_evaluator_sdk/tests -q: 1125 passed, 2 skipped
  • post-rebase focused inference suite: 75 passed
  • staged uv run --frozen pre-commit run: passed
  • scoped Ruff, Ruff format, and ty checks: passed
  • tools/lint/lint-sdk-vendored.sh: passed
  • git diff --check origin/main...HEAD: passed

@github-actions github-actions Bot added the fix label Jul 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 26404/34057 77.5% 61.8%
Integration Tests 15162/32682 46.4% 18.6%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant