docs(experiments): document Experiments ingest flow + add nemo-experiments-upload skill#694
Merged
Merged
Conversation
shanaiabuggy
changed the title
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughChangesExperiments upload workflow
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
services/intake/README.md (1)
26-40: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftKeep the README focused on one documentation quadrant.
The new section adds a multi-step how-to to an API/reference and local-development README. Keep a short workflow summary and link here; move the procedural details to the dedicated skill or a separate how-to page.
As per coding guidelines, each documentation page should fit one Diátaxis quadrant.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@services/intake/README.md` around lines 26 - 40, Shorten the “Logging experiment data (end-to-end)” section in the README to a brief workflow summary and link to the dedicated nemo-experiments-upload skill for procedural details. Remove the multi-step instructions and endpoint-specific guidance from this README while preserving the concise navigation link.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md`:
- Around line 9-14: Make both labeled JSON examples executable: in
ingest-formats.md, wrap the evaluation_context fragment in a complete object and
remove inline comments; in harbor-quickstart.md, replace the comment placeholder
with valid steps JSON. Ensure both examples parse as-is without altering their
documented meaning.
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md`:
- Around line 35-55: Make the setup workflow fail closed by enabling strict Bash
behavior with set -euo pipefail, changing the readiness check to exit nonzero
when /health/ready fails, and validating that GROUP_ID is non-empty after the
experiment-group creation before proceeding to create the Evaluation.
- Around line 79-87: Update the “Log traces + evaluator results” guidance in
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md
(lines 79-87) to apply evaluation_context only to ATIF and chat-completions, and
instruct OTLP users to provide identity through root-span attributes. Update
services/intake/README.md (lines 33-34) to document the OTLP attributes
nemo.experiment.id and nemo.test_case.id.
---
Nitpick comments:
In `@services/intake/README.md`:
- Around line 26-40: Shorten the “Logging experiment data (end-to-end)” section
in the README to a brief workflow summary and link to the dedicated
nemo-experiments-upload skill for procedural details. Remove the multi-step
instructions and endpoint-specific guidance from this README while preserving
the concise navigation link.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 34e15efb-0920-43b8-9982-fe56c6281276
⛔ Files ignored due to path filters (4)
sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/harbor-quickstart.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.mdis excluded by!sdk/**
📒 Files selected for processing (6)
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.jsonservices/intake/README.md
shanaiabuggy
force-pushed
the
sbuggy/experiments-agent-skill
branch
from
July 15, 2026 18:40
2831663 to
5a4321c
Compare
Contributor
|
shanaiabuggy
force-pushed
the
sbuggy/experiments-agent-skill
branch
from
July 15, 2026 21:14
5a4321c to
3ef003f
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md`:
- Around line 146-151: Add the OTLP traces protocol configuration next to the
endpoint example in the ingest-formats documentation, setting
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL or the global OTEL_EXPORTER_OTLP_PROTOCOL to
http/protobuf so SDKs use the required HTTP protobuf transport.
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md`:
- Line 3: Update the SKILL.md frontmatter and shared ingest/troubleshooting
guidance to distinguish ATIF and chat-completions, which use evaluation_context,
from OTLP, which requires nemo.experiment.id and nemo.test_case.id on the root
span. Remove any wording that implies evaluation_context applies to every
endpoint, and ensure the OTLP troubleshooting entry explains these root-span
attributes for associating runs.
- Around line 105-107: Update the evaluator-results documentation near the
explicit POST payload to distinguish fields by data_type: use value for numeric
results, and document string_value for CATEGORICAL and TEXT results. Preserve
the existing span_id, session_id, name, and data_type fields and clarify that
payloads must use the field matching the declared type.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9efb7f59-8fe3-49dc-98b6-18b34e67410d
⛔ Files ignored due to path filters (4)
sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/harbor-quickstart.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.mdis excluded by!sdk/**
📒 Files selected for processing (6)
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.jsonservices/intake/README.md
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md
- services/intake/README.md
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.md
shanaiabuggy
force-pushed
the
sbuggy/experiments-agent-skill
branch
from
July 15, 2026 21:31
3ef003f to
ecd8938
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md`:
- Around line 57-65: Update the group creation flow that assigns GROUP_ID so the
GET fallback runs only when the POST returns HTTP 409. Capture and inspect the
POST status separately, preserve the created group ID on success, invoke the
existing-name GET lookup for 409, and propagate validation, transport, and 5xx
failures instead of masking them.
- Around line 35-45: Make each executable workflow self-contained by
initializing or validating NMP_BASE_URL and WORKSPACE before use. In
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md
lines 35-45, update the readiness-check block; apply the same initialization or
validation in
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md
lines 149-152 before exporting the OTLP endpoint, preserving valid standalone
copy/paste execution.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3e6a9b39-8923-4e3c-ae7e-1cd764243d7a
⛔ Files ignored due to path filters (4)
sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/harbor-quickstart.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.mdis excluded by!sdk/**
📒 Files selected for processing (6)
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.jsonservices/intake/README.md
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json
- services/intake/README.md
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.md
BrianNewsom
approved these changes
Jul 15, 2026
…ments-upload skill Signed-off-by: shanaiabuggy <[email protected]>
shanaiabuggy
force-pushed
the
sbuggy/experiments-agent-skill
branch
from
July 15, 2026 23:12
ecd8938 to
917922c
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md`:
- Around line 3-7: Revise the ingest-formats documentation so it no longer
claims evaluation_context is shared by all endpoints. Move the
evaluation_context section under the ATIF/chat-completions guidance, and state
before the endpoint schemas that OTLP does not accept this JSON field and
instead requires evaluation identity in root-span attributes.
In
`@packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md`:
- Around line 83-91: Update the evaluation curl snippet to be independently
executable by initializing defaults for NMP_BASE_URL and WORKSPACE, then
explicitly validate that GROUP_ID is set before issuing the request. Keep the
existing payload and endpoint behavior unchanged, and fail with a clear message
when GROUP_ID is missing.
- Around line 102-103: Update the OTLP guidance in SKILL.md at lines 102-103 to
describe nemo.experiment.id and nemo.test_case.id as attributes on the root
span, not root-span resource attributes. In references/ingest-formats.md lines
3-7, remove the “shared by all” framing and explicitly scope evaluation_context
to ATIF and chat-completions JSON payloads only; keep OTLP identity represented
by root-span attributes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d6ac2502-7489-41b5-bc38-4f37f6090ce9
⛔ Files ignored due to path filters (4)
sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/harbor-quickstart.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.mdis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.mdis excluded by!sdk/**
📒 Files selected for processing (6)
packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.mdpackages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.jsonservices/intake/README.md
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json
- packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/harbor-quickstart.md
- services/intake/README.md
Signed-off-by: shanaiabuggy <[email protected]>
shanaiabuggy
enabled auto-merge
July 16, 2026 03:01
Signed-off-by: shanaiabuggy <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
nemo-experiments-uploadskill guide (curl workflow) to create experiment groups/evaluations, ingest ATIF/chat-completions/OTLP, verify sessions/rollups, and view results in Studio.nemo-experiments-uploadskill (positive and negative control prompts).