2.7.0 on Agent Engine: worker dies without a traceback on the second model call after a tool round-trip, truncating the stream

[TakumaKurosawa]

🔴 Required Information

Bug description:

After upgrading google-adk 2.6.2 → 2.7.0 (with google-cloud-aiplatform 1.163.0 → 1.164.0), every invocation that involves a tool call fails on Vertex AI Agent Engine. The uvicorn worker process handling the request disappears without emitting a Python traceback, and the parent process starts a replacement worker a few seconds later. The :streamQuery response ends after two events, so the client sees a well-formed HTTP 200 stream that simply contains no model text.

The invocation dies after the tool round-trip completes successfully — the failure is in the second model call of the turn.

Reverting only the two dependencies (no application code change) makes the same question complete normally.

Steps to Reproduce:

1. Build an LlmAgent as the root agent (coordinator) with:
   • a function tool that performs network I/O (in our case a Workspace search returning ~5 results),
   • seven sub_agents, all mode="single_turn" (five domain specialists, one artifact reader, one grounding judge),
   • before_model_callback that sets tool_config (FunctionCallingConfigMode.ANY) on the first LLM call of a turn,
   • after_agent_callback returning types.Content when the root produced no visible text,
   • BigQueryAgentAnalyticsPlugin registered via AdkApp(agent=..., plugins=[...]).
2. Deploy to Agent Engine (vertexai.agent_engines, agent_framework="google-adk"), region asia-northeast1, Python 3.13.
3. Send a question through :streamQuery that requires the tool.

Expected Behavior:

The turn proceeds: model → tool call → tool result → model → delegation to a sub-agent → final text. This is exactly what 2.6.2 does (14 events, final answer with citations).

Observed Behavior:

The stream ends after 2 events:

// event 1 — coordinator, function_call + thought_signature
{"author":"coordinator","content":{"parts":[{"function_call":{"name":"search_workspace_documents", ...},
 "thought_signature":"..."}],"role":"model"},"finish_reason":"STOP", ...}

// event 2 — coordinator, function_response (tool succeeded, 5 results)
{"author":"coordinator","content":{"parts":[{"function_response":{"name":"search_workspace_documents",
 "response":{"status":"success","results":[ ... 5 items ... ]}}}],"role":"user"}, ...}

// nothing further

Runtime logs for the same invocation:

03:40:04.415  [26]  POST /api/stream_reasoning_engine HTTP/1.1" 200 OK
03:40:06.514  [26]  Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:07.692  [26]  Response received from the model.
              (tool executes successfully, ~5.4 s)
03:40:13.100  [26]  Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:17.518  [413] gcsfs experimental features enabled ...      <- different PID: a new worker booted
03:40:17.529  [413] Plugin 'bigquery_agent_analytics' registered.
03:40:17.532  [413] Application startup complete.

Worker [26] never logs Response received from the model. for the second call. No traceback, no ERROR-level entry appears anywhere in the invocation window, which points to the process being killed by a signal rather than raising.

What we ruled out:

Hypothesis	Result
Model generation	Reproduced with both gemini-2.5-flash and gemini-3.5-flash
The tool itself	Tool completed in 5.4 s and returned status: success with 5 results
Auth / credentials	status: success, no auth_required
Application code	Unchanged across the working and failing deploys; only the two dependencies differ
tool_config forcing not being released	Verified with a real Runner on 2.7.0: the second call has tool_config=None as expected
Dependency set growth	uv export --only-group runtime diff is 3 lines (adk, aiplatform, and google-cloud-iam dropped); no new packages

Possibly related observation (not a proven cause):

Container memory allocation for the same question roughly doubled: ~0.8 GiB on 2.6.2 vs ~1.6–2.0 GiB on 2.7.0 (Agent Engine console metric).

We also measured that LlmAgent._resolved_model (new in 2.7.0) keeps one resolved BaseLlm per agent. With 8 agents that all pass the model as a string, this retains 8 distinct Gemini instances and 8 distinct genai.Client objects (+18.5 MB RSS per worker in our measurement). That alone is far too small to explain a kill, but it does mean 2.7.0 retains state where 2.6.2 let it be collected.

Environment Details:

• ADK version: 2.7.0 (working: 2.6.2)
• google-cloud-aiplatform: 1.164.0 (working: 1.163.0)
• Python: 3.13
• Runtime: Vertex AI Agent Engine (reasoningEngines), region asia-northeast1, 11 uvicorn workers
• Model: gemini-2.5-flash / gemini-3.5-flash, stream: False, Vertex backend
• Telemetry: GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true
• Plugins: BigQueryAgentAnalyticsPlugin

Happy to provide the full NDJSON stream, additional log windows, or to run a targeted experiment (e.g. with telemetry disabled) if that would help narrow it down.

[llalitkumarrr]

Hello @TakumaKurosawa,

We tried to reproduce the issue using google-adk==1.27.1 and google-cloud-aiplatform==1.165.0. We used 7 sub-agents and 1 tool (with no plugins), but we were unable to replicate the problem. Please let us know if we should adjust any configuration details to better match your setup.
Could you please try using above mentioned versions and let us know if the issue still occurs? If you are still experiencing the issue please share a reproducible code snippet so we can investigate further.

[TakumaKurosawa]

@llalitkumarrr
Thank you for investigating the issue!

using google-adk==1.27.1 and google-cloud-aiplatform==1.165.0

I would like you to reproduce with google-adk==2.7.0.
Could you re-try with the above version, please? 🙏

[llalitkumarrr]

Hello @TakumaKurosawa,

We can test it but since we have an upgraded version, would it be possible for you to try it and let us know your feedback?

[TakumaKurosawa]

Thanks for following up, @llalitkumarrr. We ran the bisects on our side — here is what we found.

On the 2.7.0 symptom: #6780 identifies LlmAgent._resolved_model memoization as the cause, and that analysis matches what we observed. Pinning back to 2.6.2 resolved it for us, so we're treating that part as explained pending the fix landing.

A second, independent finding: while moving to Gemini 3.x models we hit the same symptom (worker dies without a traceback mid-invocation, stream truncates) on 2.6.2 — where the memoization doesn't exist. Single-variable bisects on the same Agent Engine deployment, all on google-adk==2.6.2 + google-cloud-aiplatform==1.164.0 with identical app code:

Model	Endpoint	Result
gemini-2.5-flash	regional (asia-northeast1)	✅ full run, final answer
gemini-3.7-flash	global (GOOGLE_CLOUD_LOCATION=global)	❌ worker killed
gemini-2.5-flash	global (GOOGLE_CLOUD_LOCATION=global)	❌ worker killed

The only difference between the first and last row is the endpoint, so our measurements suggest the global endpoint is somehow involved — though we can't say definitively why. The kill point varies between runs (during the model call, right after the response, ~5s after the response), which to us looks more like resource exhaustion than a specific code path. We were using the global endpoint because Gemini 3.x models are only served from global / us / eu, not from our deployment region.

Since this second one reproduces on 2.6.2 with no memoization involved, it may well be an Agent Engine / environment matter rather than ADK itself — sharing it here mainly in case others hit the same symptom while trying Gemini 3 models on a regional Agent Engine deployment.

Current state on our side: 2.6.2 + gemini-2.5-flash + regional endpoint is stable in production. Happy to test a build containing #6780 when it lands.