digest: 2026-06-15

_meta/claims-ledger.md

@@ -1,5 +1,12 @@
 # Claims Ledger
 
+- **2026-06-15** [arxiv:2606.14470] CLAIM: GitOfThoughts (Shekar / Abhishek / Krishnan) — version-controlled reasoning treats agent thoughts as git commits/tags/annotations; **pre-registered replications across 2 benchmarks, 2 model scales, 5 memory substrates (none/markdown/vector/graph/git)**: memory does not reliably help on novel problems; accuracy improves sharply **only at ≥0.8 cosine similarity** (near-duplicates); 4.5× larger model doubles near-duplicate payoff but **still cannot extract a transferable method from a worked example**; only test-time sampling reliably improves; paper documents a retracted result + refuted hypothesis | source: https://arxiv.org/abs/2606.14470 | status: open ; STRONGEST RECENT COUNTER-EVIDENCE to memory-architecture cluster — directly tensions with [arxiv:2606.06036] MRAgent (+23%), [arxiv:2606.04315] AutoMEM, [arxiv:2606.03463] DMF, [arxiv:2606.06090] MAGE, [arxiv:2606.05894] EMBER, [arxiv:2606.11680] HORMA, [arxiv:2606.06337] TokenMizer ; reconciliation hypothesis: cluster benchmarks (LoCoMo / LongMemEval / MemoryAgentBench) may contain enough near-duplicate cases to meet 0.8-similarity bar — open empirical question is the realized similarity distribution between training memory and test problems on those evals
+- **2026-06-15** [arxiv:2606.14571] CLAIM: StreamMemBench (Liu / Ren / Gu / Zhang / Wang / Liu / Gu / Lu) — streaming-evaluation benchmark for personal-assistant agent memory; egocentric video streams structured as **two-task sequences** where evidence/feedback from task 1 must inform task 2; four diagnostic metrics (evidence recall, evidence application, feedback integration, follow-up reuse) decompose where memory pipelines fail; 8 memory systems × 2 backbones tested; finding: agents store and locally apply information but fail to translate to consistent future behavior | source: https://arxiv.org/abs/2606.14571 | status: open ; second independent data point alongside [arxiv:2606.14470] GitOfThoughts that **bottleneck is use/transfer not storage** ; complements [arxiv:2605.12493] LongMemEval-V2 (5 abilities, 500 trajectories, 115M tokens) at the streaming-input → future-assistance trajectory
+- **2026-06-15** [arxiv:2606.14106] CLAIM: AGMem / "Naive Visual Memory is Not Enough" (Choi / Ko / Lee / Kim / Song / Jeon / Shin, ICML 2026 workshop) — failure-mode study of GUI agents identifies four failure modes (cognitive, visual state misunderstanding, hidden operation blindness, grounding error); **prepending full-image memory has a divergent effect on the failure distribution: reduces state-level failures but worsens action-level ones**; Action-Grounded Visual Memory stores cropped image regions of successful actions only; **+33.3% task success on OSWorld vs full-image memory** | source: https://arxiv.org/abs/2606.14106 | status: open ; directly actionable design pattern for computer-use stacks; locally implementable independent of model
+- **2026-06-15** [arxiv:2606.14672] CLAIM: Parallel-Synthesis (Liu / Li / Fu / Wang / Xia / Li / Yan / Li) — synthesizer directly consumes KV caches from parallel worker agent branches instead of concatenated text; Cache Mapper aligns independently generated caches + Synthesizer Adapter fine-tuned for non-sequential cache interface; training mixes parallel-cache exposure + cross-branch aggregation + distillation from text-concat baselines; **matches/beats text synthesis on 7 of 9 downstream datasets (math, science QA, code, GAIA, multi-agent DB diagnosis) at 2.5×–11× lower time-to-first-token** | source: https://arxiv.org/abs/2606.14672 | status: open ; second KV-cache-as-communication paper in two days alongside [arxiv:2606.13594] Dense Latent Communication (yesterday, sequential heterogeneous-agent handoff); both papers converging on "text is not the right inter-agent currency when both sides share tokenizer/hidden-state geometry"; open question whether cache handoff generalizes across model families (Qwen-to-Llama, MoE routing layers)
+- **2026-06-15** [arxiv:2606.14249] CLAIM: HarnessX (Chen / Lu / Zhao + 11 collaborators) — composable / adaptive / evolvable agent harness foundry; typed harness primitives assembled via substitution algebra + AEGIS engine bridges symbolic adaptation + RL via execution traces; closed loop converts trajectories into harness improvements + model training signals; **+14.5% average / +44.0% max across ALFWorld + GAIA + WebShop + τ³-Bench + SWE-Bench Verified**; largest gains where baseline weakest | source: https://arxiv.org/abs/2606.14249 | status: open ; third harness-layer paper in five days alongside [arxiv:2606.13643] RAH (parent-spawns-sub-harness structure) + [arxiv:2606.12882] HarnessBridge (learnable observation/action controller); typed-primitives + substitution-algebra is the most engineering-grade framing of the three; complements [arxiv:2606.13598] OrchRM (orchestrator RM); none of the four publishes head-to-head on shared benchmark yet
+- **2026-06-15** [arxiv:2606.13715] CLAIM: WorkBench Revisited (Styles) — two-year longitudinal re-run of workplace-agent benchmark (original [arxiv:2405.00823], March 2024); GPT-4 March 2024: 43% task completion + 26% unintended harmful side-effects; **Claude Opus 4.8 June 2026: 89% completion + 2.5% harmful side-effects**; capability and safety improved together rather than trading off on this benchmark; frontier models still occasionally commit irreversible errors; open-weight models closed cost-per-task gap substantially; updated benchmark + code released | source: https://arxiv.org/abs/2606.13715 | status: open ; cleanest single-benchmark longitudinal frontier-progress data point yet; 2.5% irreversible-harm rate is the operational ceiling for agents with write access
+- **2026-06-15** [arxiv:2606.14179] CLAIM: CacheRL (Islam / Thakur / Chen / Park / Wang / Kim) — small (4B) tool-calling agent training: hybrid thinking trajectory pipeline distills reasoning from large LLMs + CacheAgentLoop (three-tier fuzzy cache for RL over cached environments instead of live tools, token-level masking) + cache-tier-aware reward; **Qwen3-4B-Thinking validation reward 0.43 → 0.78; 92% process accuracy on multi-step tool calling vs GPT-5 94% at ~100× less compute**; ablation: removing knowledge transfer drops 41%, cache-aware reward adds 17%; **RL improved training stability but contributed little beyond strong SFT** | source: https://arxiv.org/abs/2606.14179 | status: open ; consistent with [arxiv:2606.06079] SkillComposer "data engineering not RL complexity is the lever" for sub-7B tool-calling agents; Qwen3-4B-Thinking is local-fit on M5 Max (FP16 ~8 GB / Q4 ~2.5 GB)
 - **2026-06-11** [arxiv:2606.12329] CLAIM: PROJECTMEM (Malo / Qiu, github.com/riponcm/projectmem) — **local-first, offline-first, event-sourced memory layer for AI coding agents**; append-only plain-text event log (issues / attempts / fixes / decisions / notes) deterministically projected into compact AI-readable summaries via MCP (14 MCP tools + 19 CLI commands + 37 automated tests, 3-dependency Python package); **memory-as-governance** pre-action gate that warns the agent before repeating a failed fix or editing a known-fragile file; 2-month / 10-project / 207-event self-study reports **5,000–20,000 tokens saved per session**; CC BY 4.0 | source: https://arxiv.org/abs/2606.12329 | status: open ; most directly deployable artifact in the **file-based-memory-for-coding-agents** wave that Fable 5's persistent-memory release implied an app-layer ecosystem would fill; structurally an MCP-served per-project journal — closest analog to the AI research vault's own append-only design; novel piece is the **deterministic pre-action gate** (not just retrieval but rejection of known-bad actions); selection effect severe (authors' own use), open question whether the 5–20k tokens/session number reproduces under a third-party benchmarked workload (e.g., SWE-Bench Verified with/without the MCP layer)
 - **2026-06-11** [arxiv:2606.11680] CLAIM: HORMA / Organize then Retrieve (Hsu / Kuang / Liu / Yao / He, DeepSpeed-era infra leads) — two-stage memory framework: (1) structured construction distinguishing **information gaps vs misleading/overloaded context** before storing, (2) **navigation-based retrieval via a lightweight RL-trained navigator** that traverses a file-system-like hierarchy linking summaries to detailed trajectories; summaries are *pointers* into raw trajectory blocks (no lossy compression at storage, only at retrieval selection); **token usage reduced to at most 22.17% of baseline** on long conversations; improves ALFWorld / LoCoMo / LongMemEval; generalizes to unseen tasks | source: https://arxiv.org/abs/2606.11680 | status: open ; fourth distinct tree/file-system memory architecture in two weeks alongside [arxiv:2606.06090] MAGE (state tree), [arxiv:2606.04555] SegTreeMem (temporal tree), [arxiv:2606.04780] PersonaTree (lifecycle tree); HORMA's **summary-as-pointer + RL navigator** combo is the distinct contribution — no information loss at storage tier, separates retrieval *selection* from compression; open question whether HORMA / MAGE / EMBER / MRAgent show consistent relative ordering when run head-to-head on a unified eval (each currently posts on overlapping but non-identical benchmark subsets)
 - **2026-06-11** [arxiv:2606.11926] CLAIM: Arbor (Jin et al., 18-author RUC NLPIR + collaborators) — autonomous-research agent framework with **long-lived coordinator + short-lived executors in isolated environments + Hypothesis Tree Refinement (HTR)** linking hypotheses ↔ artifacts ↔ evidence ↔ distilled insights across time; introduces an **Autonomous Optimization (AO) evaluation setting** (artifact-improvement-under-iteration); **>2.5× the average relative held-out gain over Codex and Claude Code** across six research tasks (model training, harness engineering, data synthesis); **86.36% Any Medal on MLE-Bench Lite with GPT-5.5** (strongest in their comparison) | source: https://arxiv.org/abs/2606.11926 | status: open ; *fourth* recent agent-improves-its-own-stack result beating a named frontier baseline on a non-SWE-Bench workload (alongside [arxiv:2606.06453] Vortex agents-author-sparse-attention, [arxiv:2606.06473] MLEvolve beats AlphaEvolve on math opt, [arxiv:2606.05922] RHO 59→78% SWE-Bench Pro); cluster is now too large to dismiss as cherry-picked; does *not* contradict the Fable 5 SWE-Bench Pro 80.3% claim ([anthropic-news-2026-06-09-claude-fable-5-mythos-5]) — Arbor wins on **open-ended research/optimization workloads**, Fable 5 wins on **bug-fix / terminal / SWE-Bench-style** workloads; open question whether the gap holds when the AO setting is restricted to localized code-change subtasks

_meta/processed.json

@@ -1,4 +1,46 @@
 [
+  {
+    "id": "arxiv:2606.14470",
+    "url": "https://arxiv.org/abs/2606.14470",
+    "title": "GitOfThoughts — Version-Controlled Reasoning + Agent Memory You Can Replay/Diff/Merge; Treats Reasoning as Git Repo (Thoughts=Commits, Scores=Annotations, Outcomes=Tags); Pre-Registered Replications Across 2 Benchmarks, 2 Model Scales, 5 Memory Substrates (None/Markdown/Vector/Graph/Git); KEY FINDING: Memory Does Not Reliably Help on Novel Problems — Accuracy Improves Sharply Only at ≥0.8 Similarity (Near-Duplicates); 4.5× Larger Model Doubles Near-Duplicate Payoff But Cannot Extract Transferable Methods; Only Test-Time Sampling Reliably Improves; Documents Retracted Result + Refuted Hypothesis",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.14571",
+    "url": "https://arxiv.org/abs/2606.14571",
+    "title": "StreamMemBench — Streaming Evaluation of Agent Memory for Future-Oriented Assistance; Two-Task Sequences from Egocentric Video Streams; Four Diagnostic Metrics (Evidence Recall, Evidence Application, Feedback Integration, Follow-Up Reuse); 8 Memory Systems × 2 Backbones; Finds Agents Can Store/Process Locally But Fail to Translate to Consistent Future Behavior; Open Code/Data",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.14106",
+    "url": "https://arxiv.org/abs/2606.14106",
+    "title": "AGMem (Action-Grounded Visual Memory) — Failure-Mode Study of GUI Agents Shows Naive Visual Memory is Not Enough; Identifies 4 Failure Modes (Cognitive, Visual State Misunderstanding, Hidden Operation Blindness, Grounding Error); Full-Image Memory Reduces State-Level Failures But Worsens Action-Level Failures; AGMem Stores Cropped Image Regions of Successful Actions Only; +33.3% Task Success on OSWorld vs Full-Image Memory; ICML 2026 Workshop",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.14672",
+    "url": "https://arxiv.org/abs/2606.14672",
+    "title": "Parallel-Synthesis — Synthesizer Directly Consumes KV Caches from Parallel Worker Agent Branches Instead of Concatenated Text; Cache Mapper Aligns Independently Generated Caches + Synthesizer Adapter Fine-Tuned for Non-Sequential Cache Interface; Training Mixes Parallel-Cache Exposure + Cross-Branch Aggregation + Distillation from Text-Concatenation Baselines; Matches/Beats Text Synthesis on 7 of 9 Downstream Datasets (Math, Science QA, Code, GAIA, Multi-Agent DB Diagnosis); 2.5×–11× Lower Time-to-First-Token",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.14249",
+    "url": "https://arxiv.org/abs/2606.14249",
+    "title": "HarnessX — Composable / Adaptive / Evolvable Agent Harness Foundry; Typed Harness Primitives Assembled via Substitution Algebra + AEGIS Engine Bridges Symbolic Adaptation + RL via Execution Traces; Closed Loop Converts Trajectories Into Harness Improvements + Model Training Signals; +14.5% Average / +44.0% Max Across ALFWorld + GAIA + WebShop + τ³-Bench + SWE-Bench Verified; Largest Gains Where Baseline Was Weakest; Codebase Release Promised",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.13715",
+    "url": "https://arxiv.org/abs/2606.13715",
+    "title": "WorkBench Revisited — Two-Year Longitudinal Re-Run of Workplace Agent Benchmark (Original arXiv:2405.00823); GPT-4 March 2024: 43% Task Completion + 26% Unintended Harmful Side-Effects; Claude Opus 4.8 June 2026: 89% Completion + 2.5% Harmful Side-Effects; Capability + Safety Improved Together Rather Than Trading Off; Frontier Models Still Occasionally Commit Irreversible Errors; Open-Weight Models Closed Cost-Per-Task Gap Substantially; Updated Benchmark + Code Released",
+    "date_covered": "2026-06-15"
+  },
+  {
+    "id": "arxiv:2606.14179",
+    "url": "https://arxiv.org/abs/2606.14179",
+    "title": "CacheRL — Small (4B) Tool-Calling Agent Training; Hybrid Thinking Trajectory Pipeline (Distills Reasoning from Large LLMs) + CacheAgentLoop (Three-Tier Fuzzy Cache for RL Over Cached Environments Instead of Live Tools, Token-Level Masking) + Cache-Tier-Aware Reward; Qwen3-4B-Thinking Validation Reward 0.43 → 0.78; 92% Process Accuracy on Multi-Step Tool Calling vs GPT-5 94% at ~100× Less Compute; Removing Knowledge Transfer Drops Performance 41%, Cache-Aware Reward Adds 17%; RL Improved Training Stability But Limited Gains Beyond Strong SFT",
+    "date_covered": "2026-06-15"
+  },
   {
     "id": "nvidia-blog-2026-06-12-aa-agentperf",
     "url": "https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/",