[loop-streaming] Ephemeral hook injections break prompt-cache prefix continuity — conversation history is re-written (1.25x) on every provider call

[Joi]

Summary

loop-streaming appends ephemeral hook injections (skills-visibility list ~5k tokens, status-context with a seconds-precision timestamp, todo reminder, etc.) by string-concatenating them into the last message's content on every provider request, without storing them in history (amplifier_module_loop_streaming/__init__.py:350 and :388, verified on main 2026-07-20).

Request N therefore caches a prefix ending in [...last_msg + injection_N], while request N+1 sends [...last_msg] (no injection) + new messages + injection_N+1. The prefix diverges exactly where the previous request's injection was baked in, so with Anthropic prompt caching conversation history never cache-hits beyond the initial system+tools prefix — it is re-written at cache-write rates (1.25x input) on every single call, growing each time.

Evidence (real session, claude-fable-5, from events.jsonl usage fields)

call 1:  cache_write=263,338  cache_read=0          <- session start
call 2:  cache_write=9,658    cache_read=254,421
call 3:  cache_write=10,750   cache_read=254,421
call 5:  cache_write=13,046   cache_read=254,421
...
call 12: cache_write=25,305   cache_read=254,421
(>5min gap) cache_write=281,993 cache_read=null     <- full re-write after TTL expiry
call 14: cache_write=28,944   cache_read=254,421

cache_read is frozen at the initial prefix forever; cache_write grows monotonically. 1.7M tokens cache-written across 27 calls in one session. Org-wide July 1–20: 83% of total API cost ($6,191 of $7,474) was input_cache_write_5m, with a cache read/write ratio of 2.0 (a healthy agentic workload is 5–10x+).

Suggested fix (two small coordinated changes)

1. loop-streaming: append the ephemeral injection as a separate trailing message instead of concatenating into the last real message.
2. provider-anthropic: place cache breakpoints on the last TWO messages (Anthropic allows 4 breakpoints; only 3 are used today — _apply_message_cache_control at __init__.py:3722). The last real message then remains a stable cache hit-point across requests, and the ephemeral tail becomes cheap uncached input (~5–7k tokens at 1x) instead of forcing a full-history re-write.

With both in place, conversation history caches incrementally as designed; per-call cache_write drops from tens of thousands of tokens (growing) to just the actual increment.

Happy to submit PRs for both if given the nod — I have the fix sketched and the measurement harness to verify before/after.

Environment

• Amplifier via uv tool install, macOS arm64
• amplifier-module-loop-streaming @ main (2e7232a-era cache), amplifier-module-provider-anthropic @ main
• Related observation: skills-visibility hook injects the full skills list (~5k tokens) into every provider request by default; capping visibility.max_skills_visible mitigates the tail cost but not the prefix-busting.

[Joi]

Update — both fixes now implemented and running in production; two corrections to the report above

Nine days on, some of what I wrote above turned out to be incomplete, and I've since built and deployed both halves. Posting the corrections and the implementation hazards, because anyone working from the original text alone would fix only half the bug and then hit two 400-level traps.

Correction 1 — there are TWO injection paths, and the issue only describes one

The report says injections are string-concatenated into the last message's content (:350, :388). That's the append_to_last_tool_result=True path, taken when the last message is a tool result. But the default path (:369, :404) appends the injection as a new trailing message — and that breaks caching too, because the appended ephemeral message then becomes the message the provider breakpoints.

So "append as a separate trailing message" (my own suggested fix #1) is not sufficient on its own. Either path leaves the breakpoint sitting on volatile content — the status-context injection carries a seconds-resolution timestamp, so the cached prefix includes bytes that change on the next turn.

Correction 2 — one consolidated trailing message, not N separate ones

Up to five hooks can inject per request (skills-visibility, status-context, todo, routing, mcp). If each appends its own message, a "breakpoint the last two messages" provider fix can still land the penultimate marker on an ephemeral message. The two changes have to be specified together:

1. loop-streaming: consolidate ALL pending ephemeral injections into exactly one trailing message, and drop the concat-into-tool-result path entirely so no stored message is ever mutated.
2. provider-anthropic: breakpoint the last two messages, so the penultimate marker lands on the last stored message — whose bytes are stable across turns — while the single ephemeral tail rides as ordinary uncached input.

Two implementation hazards in the provider half (both found in review, both cause hard 400s)

a) cache_control on a thinking block. The new penultimate breakpoint can land on an assistant turn whose converted content ends in a thinking / redacted_thinking block. The API rejects cache_control on those → 400 on every thinking-enabled request. This is introduced by the change: upstream only marks the final message, which is always a user/tool_result turn and therefore never thinking-terminated. Fix is to walk the content blocks backwards and stamp the last block whose type is not thinking/redacted_thinking; if a message has no eligible block, leave it unmarked and degrade to marking only the final message rather than emitting an invalid payload.

b) Breakpoint budget exhaustion. Anthropic allows 4 cache_control breakpoints. Marking the last two messages spends all four (system + tools + 2 messages), leaving zero headroom — so a message arriving with a pre-existing cache_control, or any future marking site, silently makes it 5 and hard-400s. Worth adding a budget check that drops provider-added message markers (penultimate first) until within budget. Removing a breakpoint only reduces caching; it can never 400.

Correction 3 — the org-level "83%" number is real, but I cannot yet show a before/after

Being explicit since the original post leaned on it. The session-level evidence above (cache_read frozen at 254,421 while cache_write grows) is solid and reproducible. The org-level improvement claim does not exist yet, and I found out why today: I rolled the fix out to one machine on 2026-07-27 and the org number didn't move, because a different machine was carrying effectively all the traffic and had neither half. That machine's own telemetry:

day          calls     cache_read   cache_write  ratio   usd/day
2026-07-25     988     59,144,250    32,252,211   1.83    368.23
2026-07-26    1035     62,083,540    33,838,419   1.83    386.29
2026-07-27    1085     65,296,557    35,056,747   1.86    400.70   <- fix lands on the OTHER machine
2026-07-28    1222     83,970,286    50,888,082   1.65    484.62

Ratio pinned at ~1.83 for 14 consecutive days, zero response. Both halves went live there 2026-07-29; that is the real before/after baseline and I'll report the measured delta here once there are a few days of post-fix traffic. Treat the mechanism as well-evidenced and the savings estimate as not yet validated.

State of the patches

Both are implemented as local forks and running in production on my fleet:

• loop-streaming: forked from current main (a9a2ddc8) — main hasn't been touched since 2026-07-13, so a PR would apply cleanly. Consolidation + removal of the concat path, with unit tests asserting stored history stays byte-identical across requests (including a trailing tool-result).
• provider-anthropic: forked from 2e7232a; main has since moved to 99b3df31, so this one would need a rebase. Includes the thinking-block guard and the breakpoint-budget enforcer described above, with unit tests for each of the four thinking-block shapes.

(Test counts as recorded at implementation time; I did not re-run the suites for this comment.)

On submitting PRs

I offered above to submit both. No response in nine days, and both module READMEs say the projects are "not currently accepting external contributions." So I'll leave it here rather than open PRs that can't be merged — but the patches exist, they're running, and I'm happy to hand them over in whatever form is useful. Say the word.

Related: #337 (1h prompt-cache TTL) touches the same code path in provider-anthropic.

[Joi]

Patch inventory — what exists, where it applies, and how to split it

Retracting the last paragraph of my previous comment: I'd read the "not currently accepting external contributions" line in the module READMEs plus the silence here as "don't send patches." I now understand this issue is being tracked, so here is everything needed to act on it — in whatever form is easiest for you. PRs, branches, or plain patch files: say which and I'll produce it.

loop-streaming — applies cleanly to current main

Forked from a9a2ddc8, which is still main HEAD (untouched since 2026-07-13). No rebase needed.

file	change
amplifier_module_loop_streaming/__init__.py	+44 / −67 (net −23)
tests/test_prompt_cache_ephemeral_consolidation.py	new

It is a net deletion, which is the encouraging part: the append_to_last_tool_result concat branch goes away entirely, and the _pending_ephemeral_injections loop that previously emitted one message per injection collapses into a single appended message. Nothing in the module grows a new concept. Tests assert that stored history stays byte-identical across successive requests, including when the last stored message is a tool result.

This is the whole of the loop-streaming half of #336.

provider-anthropic — needs a rebase, and carries three separable concerns

Forked from 2e7232a8; main has since moved to 99b3df31 (3 commits). More importantly, only one of the three things in this fork belongs to #336 — please don't take it as a unit:

concern	where	issue
1. Last-two-message cache breakpoints — _mark_message_cache_control, _count_cache_breakpoints, _enforce_cache_breakpoint_budget	__init__.py	this issue (#336)
2. cache_ttl config ("5m" / "1h") + _cache_control() builder	__init__.py, _cost.py	#337
3. Unrelated _RATES key bug (below)	_cost.py	not filed

Totals across all three: __init__.py +172/−26, _cost.py +27/−8, tests/test_reasoning_effort.py +5/−1, new tests/test_prompt_cache_ttl.py. Concern 1 alone is a small fraction of that.

Concern 1 is where the two 400-level hazards from my previous comment live — the thinking-block guard and the breakpoint-budget enforcer. Both are necessary; neither is obvious until it bites.

Unrelated bug found in the same audit — Haiku 3.5 cost is silently untracked

Surfaced while working in _cost.py, nothing to do with caching, not filed separately — flagging it here rather than losing it:

_RATES key on main : "claude-haiku-3-5-20250929"   <- family/version order reversed,
                                                       and that date is Sonnet 4.5's
actual snapshot id : "claude-3-5-haiku-20241022"

compute_cost() does an exact _RATES.get(), so the key matches nothing and every Haiku 3.5 call is costed at zero. Silent — no warning, no fallback. Happy to file this on its own if you'd prefer it tracked separately.

One note for anyone reproducing the measurement

Settings are read at session start, so only sessions started after the module swap exercise the new code path. A long-running session spans the change and keeps using the old modules, which will contaminate a naive before/after cut on wall-clock time. I lost my first post-deploy datapoint to exactly that. Cut on session start time, not call timestamp.

Still no clean org-level before/after from me yet — as noted above, my first rollout landed on the wrong machine. Clean baseline started 2026-07-29; I'll post the measured delta here rather than leave the savings estimate hanging.

[Joi]

Branches are up — off current main, one commit each, zero conflicts

#	repo	branch	concern
1	amplifier-module-loop-streaming	fix/prompt-cache-ephemeral-consolidation	#336 loop-streaming half
2	amplifier-module-provider-anthropic	fix/prompt-cache-message-breakpoints	#336 provider half
3	amplifier-module-provider-anthropic	fix/haiku-3-5-rates-key	unrelated rates bug
4	amplifier-module-provider-anthropic	feat/prompt-cache-1h-ttl	#337

Each branched from the main HEAD as of today (loop-streaming a9a2ddc8, provider-anthropic 99b3df31): 1 commit ahead, 0 behind. Review or merge in any order, independently.

1  amplifier_module_loop_streaming/__init__.py        | 113 +/-      2 files, +391/-67
   tests/test_prompt_cache_ephemeral_consolidation.py | 345 +

2  amplifier_module_provider_anthropic/__init__.py    | 149 +/-      2 files, +400/-25
   tests/test_prompt_cache_breakpoints.py             | 276 +

3  amplifier_module_provider_anthropic/_cost.py       |   7 +/-      2 files, +30/-1
   tests/test_cost.py                                 |  24 +

4  amplifier_module_provider_anthropic/__init__.py    |  43 +/-      3 files, +206/-6
   amplifier_module_provider_anthropic/_cost.py       |  21 +/-
   tests/test_prompt_cache_ttl.py                     | 148 +

Branch 2 touches only _apply_message_cache_control plus three new private helpers (_mark_message_cache_control, _count_cache_breakpoints, _enforce_cache_breakpoint_budget) and one call site. Nothing else in the file.

The #336 / #337 coupling I warned about turns out not to exist upstream

Worth correcting since I raised it. On current main, _format_system_with_cache and _apply_tool_cache_control both inline a bare {"type": "ephemeral"} — there is no _cache_control() helper yet. That helper exists only in my local fork, where it arrived as part of the #337 TTL work. So the entanglement was an artifact of my fork having both fixes merged, not a property of your code.

Consequence: branch 2 emits {"type": "ephemeral"} directly and never references cache_ttl. Branch 4 introduces _cache_control() and rewires only the two pre-existing bare-literal sites, deliberately leaving _apply_message_cache_control alone. If you take both, _mark_message_cache_control could later swap its literal for self._cache_control() for consistency — but it is not required, since that helper returns byte-identical output for the default "5m" tier.

Test status — honest, including a pre-existing failure

branch	result
loop-streaming #336	79 passed, 0 failed
provider-anthropic main (baseline, untouched)	540 passed, 3 failed
provider-anthropic #336	554 passed, 3 failed — 14 new tests all pass
provider-anthropic rates fix	542 passed, 3 failed — 2 new tests pass
provider-anthropic #337	558 passed, 3 failed — 18 new tests pass

The 3 failures are identical on unmodified main — tests/test_tool_repair.py::test_streaming_*, failing with 'async for' requires an object with __aiter__ method, got MockStreamManager, i.e. a test mock that no longer matches the pinned anthropic==0.96.0 streaming interface. I ran the baseline on main before writing anything specifically so this could be attributed. None of the four branches introduces a new failure. Flagging it because it is presumably green in your CI, which would mean CI pins a different anthropic.

Two things that got in the way, in case they matter to you

1. amplifier-module-provider-anthropic's pyproject.toml does not declare amplifier-core at all — dev deps are only pytest/pytest-asyncio. So uv sync alone does not produce a runnable test environment for that repo; it needs a separate uv pip install amplifier-core@<commit>. amplifier-module-loop-streaming does declare it, and uv sync there just works.
2. amplifier-core@main HEAD did not build locally — its wasmtime dependency wants rustc 1.92 and I had 1.91.1. Pinned to 47fd2adf to get a build. Purely environmental, but it means a contributor on a slightly older toolchain cannot build HEAD.

No PRs opened — these are branches on my forks, per the contribution note in the READMEs. Say the word and I will open all four, or any subset, against whichever base you prefer.

[Joi]

Retraction — do not merge the loop-streaming branch. The fix does not do what I claimed.

I put these branches up this morning and said they were ready. An independent review plus my own source verification says the central one is wrong. Correcting it before anyone spends review time on it.

The loop-streaming branch does not move the volatile bytes out of the cached prefix

Three facts, all verified in source rather than inferred:

1. provider-anthropic/__init__.py:2203 splits the request by role — system_msgs = [m for m in request.messages if m.role == "system"], while conversation keeps only ("user","assistant","tool"). System messages are pulled out of messages[] and folded into the top-level system block by _format_system_with_cache(). They never reach _apply_message_cache_control().
2. loop-streaming/__init__.py:356 — the consolidated ephemeral tail defaults to ephemeral_role: str = "system".
3. amplifier_core/models.py:231-232 — context_injection_role is Literal["system","user","assistant"] with default="system". And real hooks use it: hooks-iteration-spiral sets context_injection_role="system" explicitly.

So for the default role — which is also the role actually in use — the "separate trailing message" never becomes a trailing message. It gets folded into the system block, which is exactly where _format_system_with_cache places a cache breakpoint. That is the worst available destination for per-request volatile content: rather than fixing prefix divergence, it plausibly is prefix divergence, which is the symptom this issue opens with (cache_read frozen at the initial prefix while cache_write grows).

At best the branch is inert for system-role injections. At worst it is the bug wearing the fix's clothes. Either way, my suggested fix #1 in the original report is incomplete: consolidating into one trailing message is necessary but not sufficient — the injection also has to land in messages[] as a non-system role, or the provider has to stop hoisting it.

Two further defects in the same branch

• Per-injection role semantics are flattened. Consolidation collapses every pending fragment under a single role (__init__.py:365). If a provider-request hook injects system and a prompt-submit hook injects user, one of them is silently sent under the other's role. context_injection_role is a per-injection contract; the branch quietly makes it per-batch.
• append_to_last_tool_result is now silently ignored while amplifier_core/models.py:314-322 still documents it as active placement control. The orchestrator continues to store the flag from hook results and then discards it. That is a contract break with no corresponding contract change.

The new tests exercise only all-system injections, so they catch none of the above. They pass, and they prove nothing about the cache shape in a real Anthropic request.

The rates branch is also wrong — worse than the bug it fixes

I offered fix/haiku-3-5-rates-key as a clean standalone win. It isn't. Renaming the _RATES key to the canonical snapshot id breaks _detect_version(), which only understands family-MAJOR-MINOR ordering. Executed, not reasoned about:

claude-haiku-3-5-20250929   (the "invalid" key)  -> (3, 5)         parses correctly
claude-3-5-haiku-20241022   (canonical id)       -> (20241022, 0)  major-only fallback

So the id that supposedly matched nothing was in fact parsing correctly for capability detection, and my corrected id yields a bogus major version that will compare as newer than every real Claude model in any version gate. The underlying _RATES miss is real, but the fix needs the parser taught this ordering (or a family/snapshot alias map) plus a test, and the existing Haiku 3.5 capability tests still assert against the old id. Don't take it as-is either.

What I got wrong in my own reasoning, since it is on this thread

Yesterday I explained the missing org-level improvement entirely as "the rollout landed on the low-volume machine." That was one live hypothesis. The other — that the fork's premise was broken — was also live, I noted it once, and then stopped testing it and published anyway. The measurement confound was real; it was not the whole story, and I presented it as though it were.

Standing

• fix/prompt-cache-ephemeral-consolidation — do not merge. Needs a different approach for system-role injections.
• fix/prompt-cache-message-breakpoints — the provider-side last-two-message change, thinking-block guard and breakpoint-budget enforcer are not implicated by any of the above and I still believe they are correct. But given the above I would want them re-reviewed rather than taken on my word.
• fix/haiku-3-5-rates-key — do not merge. Fixes cost lookup, breaks version detection.
• feat/prompt-cache-1h-ttl — separate defect, see [provider-anthropic] Support 1h prompt-cache TTL — verified working against the live API, no beta header required #337.

The diagnosis in the original report stands: ephemeral injections break prefix continuity, and the session-level evidence is reproducible. The remedy I proposed is not right yet.

[Joi]

Consolidated status — and one more retraction

This thread now has several corrections layered on it. Since a maintainer arriving cold shouldn't have to reconstruct the sequence, here is the whole state in one place, plus a final correction.

Branch standing

branch	verdict
loop-streaming fix/prompt-cache-ephemeral-consolidation	do not merge — wrong approach, see below
provider-anthropic fix/prompt-cache-message-breakpoints	plausible, needs independent review — do not take on my word
provider-anthropic fix/haiku-3-5-rates-key	do not merge — fixes cost lookup, breaks version detection
provider-anthropic feat/prompt-cache-1h-ttl	do not merge — see #337, TTL never reaches the message breakpoint

Retraction: the two #336 branches are NOT independent

I said earlier they could be "reviewed or merged in any order, independently." That was wrong, and the branch's own test file says so out loud:

The companion provider fix (modules/provider-anthropic) marks the last TWO
messages with cache_control; it only works robustly if exactly ONE volatile
ephemeral message sits after stable stored history.
      -- tests/test_prompt_cache_ephemeral_consolidation.py, FORK NOTE

Meanwhile provider-anthropic main still documents the singular behavior:

"""Add cache_control to last content block of last message."""
      -- _apply_message_cache_control, main

So the loop-streaming change is written against a provider behavior that exists only on my other, unmerged branch. Merging it alone lands a consolidation whose stated precondition isn't met.

Why the loop-streaming approach is wrong, precisely

Established earlier in this thread and worth restating compactly, because the mechanism is what rules the approach out rather than merely weakening it:

_format_system_with_cache() concatenates stable and volatile system text first (:2165-2168), builds a single text block (:2173), then attaches cache_control to that block (:2175-2178). The breakpoint therefore sits after the volatile text, so the system checkpoint changes on every request.

Combined with the provider hoisting role="system" messages out of messages[] (:2203) and the consolidated tail defaulting to role="system" (loop-streaming:356, matching the core default at models.py:231-232), the "trailing message" is not trailing at all — it lands inside the cached system prefix. For injections that were already separate system messages this is merely no better; for injections that were previously appended to a tool result, moving them into the system block is worse than the status quo.

The approach that looks right

Not blanket role coercion. Have the known volatile hook producers explicitly opt in:

1. the cache-sensitive producers (skills-visibility, status-context, todo, routing, mcp) set context_injection_role="user" and append_to_last_tool_result=False
2. loop-streaming consolidates those into one trailing user message, so it stays in messages[]
3. provider-anthropic marks both the penultimate stable message and the final volatile one, so the stable-history prefix still hits when the volatile envelope changes

Why the alternatives lose:

• Force all ephemeral injections to a non-system role — breaks the explicit per-injection contract. models.py:231-237 doesn't merely default to system, it recommends it ("System role recommended for most use cases"), so silently overriding a producer's stated role is a contract violation, not a tuning change.
• Keep system role but exclude ephemeral content from the system block — either drops content or needs the same undocumented role conversion by another route.
• Split the system block so volatile text sits after the last breakpoint — protects only an earlier system checkpoint. Anthropic orders system before conversation, so every later conversation breakpoint still carries the volatile system bytes in its prefix. This one looked promising to me and doesn't survive the ordering.

Producer opt-in uses the existing role API, needs no contract change, and is small.

Two defects I previously left as open questions — both are blocking

• Role flattening. Consolidation collapses every pending fragment under a single role (loop-streaming:365), so a user injection can be emitted as system or vice versa depending on arrival order. context_injection_role is per-injection; this silently makes it per-batch.
• append_to_last_tool_result is ignored while models.py:314-322 still documents it as active placement control. The flag is still stored from hook results (:241-246, :1224-1229) and then dropped.

I had flagged both as "possibly acceptable collateral." They aren't — they are observable semantic regressions independent of whether the cache fix is correct.

What still stands from the original report

The diagnosis. Ephemeral injections do break prefix continuity, the session-level evidence (cache_read frozen while cache_write grows) is reproducible, and the cost impact is real. What I got wrong was the remedy, twice, and the confidence with which I published it. I'd suggest treating the branches as a reproduction case and a set of hazards to avoid rather than as a patch to apply.