test(eve): cover progress preservation across compaction

[ruiconti]

Why

Issue #749 records a production task that ran for 442 model turns, made 2,026 tool calls, consumed 23,737,121 input tokens, and cost $59.0342 before cancellation. After the task had both a source conclusion and runtime evidence, it still made another 1,641 calls; 844 were exact repeats of earlier tool-and-input combinations.

An earlier version of these evals required the checkpoint to use pi's exact headings and place the completion marker under ### Done. Run 29298027726 then produced 60 calls instead of 6. That result measured compliance with the required format. It did not prove that eve's free-form checkpoint had lost the completion evidence.

This draft adds fixture-owned E2E evals only. It does not change eve runtime behavior.

What

The corrected task model checks the actual checkpoint message pair. The assistant checkpoint immediately after Summary of our conversation so far: must contain the completion marker. Section names and formatting do not matter.

Case	Compaction model input	Passing result
Exact redundant tool calls	One successful inspect-repository result with REPOSITORY_INSPECTION_COMPLETE	1 call
Redundant work from a stale todo	One completed source-analysis result plus a pending todo for the same work	1 work unit

The compaction model is real and static per fixture. The task model is deterministic and shared across fixtures. It stops when the generated checkpoint contains the completion marker, or at a hard ceiling of 10 successful calls.

Compaction triggers at 640 tokens, or 2% of a synthetic 32,000-token context window. Each eval has a 100,000-input-token session limit, a 180-second timeout, and concurrency of 1.

Current result

Corrected run 29303880312 passes all six cases on eve's existing free-form checkpoint prompt:

Summary family	Exact-call result	Stale-todo result
GPT-5.6	1 call	1 work unit
Claude Opus 4.8	1 call	1 work unit
Claude Sonnet 5	1 call	1 work unit

These fixtures now cover progress preservation across the three model families, but they do not independently reproduce the production loop from #749.

Tracks #749.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eve-docs	Ready	Preview, Comment, Open in v0	Jul 14, 2026 8:38pm

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
eve-docs-1644	Skipped	Open in v0	Jul 14, 2026 8:38pm
eve-docs-4759	Skipped	Open in v0	Jul 14, 2026 8:38pm

[github-actions]

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

• No notable deltas vs main (d810570).

Delta vs main (d810570)

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	3.65 MB	3.65 MB	+925 B ⚠️
Package	Unpacked publish size	13.26 MB	13.26 MB	+3.3 kB ⚠️
Package	Installed footprint	52.52 MB	52.52 MB	+3.3 kB ⚠️
Package	Published files	2561	2563	+2
Package	Installed files	5801	5803	+2
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	9.35 MB	9.35 MB	+2.0 kB ⚠️
Runtime	Public routes	9	9	0

Changed function payloads vs main (d810570) (2)

Function	Status	Baseline	Current	Delta	Route changes
functions/.well-known/workflow/v1/flow.func	changed	5.87 MB	5.87 MB	+2.0 kB ⚠️	none
functions/__server.func	changed	3.48 MB	3.48 MB	-32 B ✅	none

eve init install

Metric	Baseline	Current	Delta
Installed footprint	90.78 MB	90.78 MB	+3.3 kB ⚠️
Installed packages	122	122	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	27.46 MB	27.46 MB	+3.3 kB ⚠️
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

• Preset: vercel
• Nitro: [email protected]
• Output directory: apps/fixtures/weather-agent/.vercel/output
• Build metadata timestamp: 2026-07-14T20:40:05.897Z
• Route aliases: 9 public, 1 internal (10 total aliases)
• Vercel routes in config: 10
• Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

• Package: [email protected]
• Package directory: packages/eve
• Tarball: 3.65 MB (eve-0.24.1.tgz)
• Unpacked payload: 13.26 MB across 2563 published files
• Installed footprint: 52.52 MB across 5803 installed files
• Installed root package: 11.98 MB
• Installed dependencies: 40.54 MB
• Runtime dependencies: 1
• Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

• @rolldown/binding-linux-x64-gnu: 19.13 MB (36.4%)
• eve: 11.98 MB (22.8%)
• ai: 6.39 MB (12.2%)
• zod: 5.04 MB (9.6%)
• nitro: 2.41 MB (4.6%)

Publish payload breakdown

Published file size
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [####....................] 1.51 MB 11.4%
🟡 dist/src/compiled/@vercel/sandbox/index.js       [##......................] 632.2 kB 4.8%
🟡 dist/src/compiled/_chunks/workflow/undici-fb5... [#.......................] 502.4 kB 3.8%
🟡 dist/src/compiled/@chat-adapter/slack/index.js   [#.......................] 433.8 kB 3.3%
🟡 dist/src/compiled/_chunks/workflow/dist-Blxny... [#.......................] 277.7 kB 2.1%
🔴 Other published files                            [########################] 9.91 MB 74.7%

Installed footprint breakdown

Installed package size
🔴 @rolldown/binding-linux-x64-gnu [########################] 19.13 MB 36.4%
🔴 eve                             [###############.........] 11.98 MB 22.8%
🔴 ai                              [########................] 6.39 MB 12.2%
🔴 zod                             [######..................] 5.04 MB 9.6%
🟠 nitro                           [###.....................] 2.41 MB 4.6%
🟡 @ai-sdk/provider-utils          [#.......................] 847.4 kB 1.6%
🔴 Other installed packages        [########................] 6.73 MB 12.8%

Runtime dependencies (1)

Package	Range	Notes
nitro	3.0.260610-beta

Peer dependencies (5)

Package	Range	Notes
@opentelemetry/api	^1.0.0	optional peer
ai	catalog:
braintrust	^3.0.0	optional peer
just-bash	^3.0.0	optional peer
microsandbox	^0.5.0	optional peer

eve init install drill-down

eve init install details

• Command: eve init my-agent
• Package manager: npm
• Installed footprint: 90.78 MB across 7621 installed files
• Installed packages: 122 total (116 transitive-only)
• dependencies: 4 direct packages totaling 27.46 MB
• devDependencies: 2 direct packages totaling 5.04 MB
• Other transitive package files: 58.28 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

• @typescript/typescript-linux-x64: 27.95 MB (30.8%)
• @rolldown/binding-linux-x64-gnu: 19.13 MB (21.1%)
• eve: 11.98 MB (13.2%)
• zod: 9.00 MB (9.9%)
• ai: 6.39 MB (7.0%)

Installed footprint breakdown

Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 30.8%
🔴 @rolldown/binding-linux-x64-gnu  [################........] 19.13 MB 21.1%
🔴 eve                              [##########..............] 11.98 MB 13.2%
🔴 zod                              [########................] 9.00 MB 9.9%
🔴 ai                               [#####...................] 6.39 MB 7.0%
🟠 @types/node                      [##......................] 2.54 MB 2.8%
🔴 Other installed packages         [############............] 13.80 MB 15.2%

dependencies (4)

Package	Range	Installed size	Share
@vercel/connect	0.2.2	98.6 kB	0.1%
ai	^7.0.26	6.39 MB	7.0%
eve	file:eve-0.24.1.tgz	11.98 MB	13.2%
zod	4.4.3	9.00 MB	9.9%

devDependencies (2)

Package	Range	Installed size	Share
@types/node	24.x	2.54 MB	2.8%
typescript	7.0.2	2.50 MB	2.8%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 5.87 MB 62.8%
🟠 functions/__server.func                         [##############..........] 3.48 MB 37.2%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 5.87 MB

Metric	Value
Public routes	/.well-known/workflow/v1/flow
Runtime	nodejs24.x
Handler	index.mjs
Payload	5.87 MB
Function files	5.87 MB across 33 files
Traced dependencies	0 B
Signal	🟠 Bundled file __eve_nitro_handler__.mjs is 2.18 MB (37.2%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 __eve_nitro_handler__.mjs       [########################] 2.18 MB 37.2%
🟡 _chunks/world-vercel.mjs        [#########...............] 826.3 kB 14.1%
🟡 _chunks/sandbox.mjs             [########................] 766.3 kB 13.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [#####...................] 430.4 kB 7.3%
🟡 _chunks/dist-Blxnyb7-.mjs       [####....................] 349.1 kB 5.9%
🟠 Other bundled files             [##############..........] 1.32 MB 22.4%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "environment": {
    "NODE_OPTIONS": "--experimental-require-module"
  },
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ]
}

---

🟠 functions/__server.func • 8 public routes, 1 internal alias • 3.48 MB

Metric	Value
Public routes	/ /eve/v1/callback/[token] /eve/v1/connections/[name]/callback/[token] /eve/v1/health /eve/v1/info /eve/v1/session /eve/v1/session/[sessionId] /eve/v1/session/[sessionId]/stream
Internal aliases	/__server
Runtime	nodejs24.x
Handler	index.mjs
Payload	3.48 MB
Function files	3.48 MB across 23 files
Traced dependencies	0 B
Signal	🟠 Bundled file _chunks/runtime-artifacts.mjs is 942.0 kB (27.1%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/runtime-artifacts.mjs [########################] 942.0 kB 27.1%
🟠 _chunks/world-vercel.mjs      [#####################...] 826.3 kB 23.7%
🟠 _chunks/sandbox.mjs           [####################....] 766.3 kB 22.0%
🟡 _chunks/dist-Blxnyb7-.mjs     [#########...............] 348.1 kB 10.0%
⚪ _libs/zod.mjs                 [###.....................] 114.2 kB 3.3%
🟡 Other bundled files           [############............] 485.3 kB 13.9%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}