fix(agents): make deployed agent inference URL container-reachable#899
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (8)
🚧 Files skipped from review as they are similar to previous changes (5)
📝 WalkthroughWalkthroughAgent deployments now resolve container-reachable gateway URLs, rewrite embedded NAT model URLs, expose Kubernetes internal URLs, and fail fast for unreachable targets. Documentation and tests cover Docker, Kubernetes, overrides, validation, and environment precedence. ChangesContainer-Reachable Agent Deployments
Sequence Diagram(s)sequenceDiagram
participant DeploymentsRunnerBackend
participant resolve_agent_gateway_url
participant rewrite_config_base_urls
participant build_deployment_config
DeploymentsRunnerBackend->>resolve_agent_gateway_url: resolve container-reachable gateway
resolve_agent_gateway_url-->>DeploymentsRunnerBackend: return URL or fail
DeploymentsRunnerBackend->>rewrite_config_base_urls: rewrite inference-gateway NAT URLs
DeploymentsRunnerBackend->>build_deployment_config: build configuration from rewritten NAT data
build_deployment_config-->>DeploymentsRunnerBackend: deployment configuration
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`:
- Around line 103-140: The docker branch of container_gateway_url must rewrite
bracketed IPv6 loopback URLs such as http://[::1]:8080. Replace the raw
host-string substitution with parsed URL reconstruction that preserves the
scheme, port, path, query, and fragment while setting the hostname to
host.docker.internal; add coverage for the ::1 case.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a26e4eb5-8eb8-4478-8bc8-afaa3f627a59
📒 Files selected for processing (8)
docs/agents/deploy-agents.mdxk8s/helm/templates/api/api-deployment.yamlk8s/helm/templates/core/controller-deployment.yamlplugins/nemo-agents/src/nemo_agents_plugin/config.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.pyplugins/nemo-agents/src/nemo_agents_plugin/utils.pyplugins/nemo-agents/tests/unit/test_runner_deployments.pyplugins/nemo-agents/tests/unit/test_utils.py
ec0b8ef to
407d787
Compare
|
Deployed agents baked the API pod's own base URL into their NAT config llms.*.base_url via inject_gateway_url() at API-create time. Under embedded-PDP auth the API pod's NMP_BASE_URL is a loopback address (http://localhost:8080), and in k8s an agent pod resolves loopback to itself, so agent model calls never reach the platform. If the platform base URL instead pointed at an auth front-door proxy, agents looped forever on 503s and could DoS it (and everything behind it). The only mode-aware rewrite (container_gateway_url) fed a dead NMP_GATEWAY_BASE_URL env var that nothing reads, so neither the docker loopback rewrite nor any k8s fix ever reached the config the agent runs. This fix hands an agent only a base URL we know is container-reachable, never the raw platform base URL. - resolve_agent_gateway_url() returns a known-good target per mode: k8s uses the in-cluster API Service DNS; docker rewrites loopback (including IPv6 [::1]) to host.docker.internal and passes other hosts through. It raises for unsupported modes or when k8s has no internal URL, rather than deploying an agent that cannot reach the platform. - rewrite_config_base_urls() rebases each Inference Gateway llms.*.base_url onto that reachable address, preserving the path. - get_internal_base_url() reads NEMO_INTERNAL_BASE_URL then NMP_INTERNAL_BASE_URL; also agents.deployments.k8s_internal_base_url. - Removes the dead NMP_GATEWAY_BASE_URL env var. - Helm sets NMP_INTERNAL_BASE_URL to the internal API Service DNS on the api and controller pods. - Updates deploy-agents docs; adds unit tests for docker/k8s resolution, the fail-fast path, and config rebasing. Signed-off-by: Ben McCown <[email protected]>
407d787 to
663e76d
Compare
|
Reviewed and satisfied with the fix — approving. Root cause and the "only ever hand a container-reachable URL" invariant look right, fail-fast + no orphaned config is clean, and the Not blockers for the release, just some adjacent edge cases worth tracking as follow-ups:
|
3 participants
TL;DR
Deployed agents were calling the wrong URL for inference. Fixed by only ever handing an agent a URL we know is reachable from inside its container.
The bug
inject_gateway_url()bakes the API pod's own base URL into each agent'sllms.*.base_urlat create time. That URL is wrong from inside an agent container:NMP_BASE_URLishttp://localhost:8080(correct for its own PDP self-call). Baked into an agent,localhostis the agent itself → calls never reach the platform.The one mode-aware rewrite that existed only wrote
NMP_GATEWAY_BASE_URL— a dead env var nothing reads — so it never reached the config the agent runs.The fix
Enforce one invariant: an agent is only ever handed a known-good, container-reachable URL — never the raw platform base URL.
resolve_agent_gateway_url()emits only known-good values per mode:NMP_INTERNAL_BASE_URL, set by Helm), or an explicit override. No internal URL → fail fast instead of propagating something unverified.host.docker.internal; other hosts pass through.llms.*.base_urlonto that URL (preserving the IGW path).NEMO_INTERNAL_BASE_URL/NMP_INTERNAL_BASE_URL(andagents.deployments.k8s_internal_base_url); Helm sets it on the api + controller pods.NMP_GATEWAY_BASE_URL.Verify
helm template --set platformConfig.auth.enabled=trueon the API pod:ruff + ty +
make docs-checkclean; unit tests for docker/k8s resolution, the fail-fast path, and config rebasing all pass.Notes
gateway_url_override(documented).nemo_fabric, CLI table assertions) reproduce onmain.Summary by CodeRabbit
New Features
Documentation