diff --git a/_meta/claims-ledger.md b/_meta/claims-ledger.md index e312da1..c429ac8 100644 --- a/_meta/claims-ledger.md +++ b/_meta/claims-ledger.md @@ -117,3 +117,13 @@ - **2026-05-23** [arxiv:2605.22106] CLAIM: ArborKV — structure-aware KV cache for tree-of-thoughts reasoning: tree-aware allocation policy + lightweight value estimator drive token-extractive eviction with lazy rehydration for branch back-tracking; **~4× peak KV memory reduction with near-full-retention accuracy**, enabling larger search configurations at fixed hardware | source: https://arxiv.org/abs/2605.22106 | status: open ; KV codec specifically for tree-shaped inference-time-compute (OpenDeepThink Bradley-Terry / MAP / InsightReplay); composes orthogonally with [arxiv:2605.19660] OScaR (per-token codec) and [arxiv:2605.18643] ZEDA (active-expert reduction) on the M5 Max long-context-inference budget - **2026-05-23** [arxiv:2605.22535] CLAIM: TerminalWorld (UCL/Meta/Tsinghua — Peter O'Hearn, Earl T. Barr, Mark Harman, Federica Sarro) — reverse-engineers benchmark from **80,870 real terminal recordings** → 1,530 validated tasks across 18 categories / 1,280 unique commands, 200-task Verified subset; **max pass rate 62.5%** across 8 frontier models × 6 agents; **Pearson r = 0.20 vs. Terminal-Bench** (the two benchmarks measure different things) | source: https://arxiv.org/abs/2605.22535 | status: open ; second "real recordings, not synthetic sandboxes" benchmark in 2 weeks after [arxiv:2605.10912] WildClawBench (62.2% Claude Opus 4.7 ceiling); r=0.20 means existing Terminal-Bench-based model rankings need re-validation - **2026-05-23** [hf-blog-2026-05-23-nvidia-nemotron-labs-diffusion] CLAIM: NVIDIA Nemotron-Labs Diffusion — open diffusion language models at 3B / 8B / 14B (text) + 8B VLM, base + instruction-tuned variants; generates 32-token blocks via parallel denoising; three modes in one model (AR / diffusion / self-speculation = diffusion drafts + AR verification); **8B +1.2% accuracy vs. Qwen3-8B; diffusion mode 2.6× faster than AR; self-speculation quadratic 6.4× faster (~865 tok/s on B200, ~4× AR baseline)**; fully open weights + Megatron Bridge training code + SGLang inference; 1.3T pretraining + 45B fine-tune tokens; commercially-friendly NVIDIA Nemotron Open Model License (text) | source: https://huggingface.co/blog/nvidia/nemotron-labs-diffusion | status: open ; first credible open diffusion LM at production scale; self-speculation pattern structurally aligned with [arxiv:2605.15077] AsyncFC and [arxiv:2605.22154] IdleSpec — three same-week orthogonal latency optimizations (one in decoder, two in harness) +- **2026-05-24** [deepmind-blog-2026-05-19-gemini-3-5-flash] CLAIM: Gemini 3.5 Flash — Google DeepMind frontier-fast model, claimed to **outperform Gemini 3.1 Pro on challenging coding and agentic benchmarks**; **76.2% Terminal-Bench 2.1**, **84.2% multimodal understanding**, **~4× faster than other frontier models** | source: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/ | status: open ; vendor-self-reported; direct `compare` against [arxiv:2605.22535] TerminalWorld (62.5% frontier ceiling on real-recording-derived tasks; Pearson r = 0.20 vs. Terminal-Bench) — open empirical question whether the 76.2% headline transfers to TerminalWorld +- **2026-05-24** [arxiv:2605.22794] CLAIM: MOSS — autonomous-agent self-evolution via **source-level code rewriting** (vs. text-layer state changes — prompts, configs, skills); deterministic multi-stage pipeline with code modifications delegated to external coding agents, replay testing against production failure batches, user-consent-gated deployment, post-deploy health monitoring; **OpenClaw four-task mean grader score 0.25 → 0.61 in a single cycle without human intervention**; code on GitHub | source: https://arxiv.org/abs/2605.22794 | status: open ; 6th layer in the self-evolution stack above [arxiv:2605.06614] SkillOS, [arxiv:2605.18401] SkillsVote, [arxiv:2605.13941] EvolveMem, [arxiv:2605.22148] Ratchet, [arxiv:2605.09998] Continual Harness — attacks structural failures the others cannot reach +- **2026-05-24** [arxiv:2605.22733] CLAIM: HarnessAPI — single typed Python + Pydantic source of truth auto-derives both streaming HTTP/SSE endpoints and MCP tool registrations with dynamic Pydantic type propagation; dual-mode content negotiation (SSE + JSON identical); **74% reduction in framework-facing boilerplate vs. manually maintained FastAPI + FastMCP dual-stack** across 6 representative skills; Apache 2.0, `pip install harnessapi` | source: https://arxiv.org/abs/2605.22733 | status: open ; production-side companion to [arxiv:2605.10787] ComplexMCP MCP-at-scale benchmark; immediately try-now-able for MCP companion stacks +- **2026-05-24** [arxiv:2605.22791] CLAIM: Gated DeltaNet-2 (NVIDIA Labs — Hatamizadeh, Yejin Choi, Jan Kautz) — linear attention with **channel-wise decoupled erase gate b_t + write gate w_t** (vs. single scalar gate in Mamba-2/Gated DeltaNet); **1.3B parameters, 100B FineWeb-Edu tokens**; strongest overall results among Mamba-2, Gated DeltaNet, KDA, Mamba-3 on language modeling, commonsense reasoning, retrieval; specific wins on **long-context RULER NIAH multi-key retrieval**; code released as NVlabs/GatedDeltaNet-2 | source: https://arxiv.org/abs/2605.22791 | status: open ; sub-quadratic-attention path orthogonal to KV-cache compression line ([arxiv:2605.22106] ArborKV, [arxiv:2605.09649] learnable KV eviction, [arxiv:2605.19660] OScaR) — relevant local-fit precursor for any future >7B linear-attention agent backbone +- **2026-05-24** [arxiv:2605.22786] CLAIM: LCGuard (RPI / IBM Research) — first paper to define **KV-cache leakage as a security boundary in multi-agent systems**; safety metric is "cache unsafe iff adversarial decoder can recover agent-specific sensitive inputs from it"; adversarial training framework learns representation-level transformations on shared caches; consistent reductions in reconstruction-based leakage and attack success while maintaining task performance vs. standard KV-sharing baselines | source: https://arxiv.org/abs/2605.22786 | status: open ; fourth angle on agent-safety surface alongside [arxiv:2605.04785] AgentTrust (call-level), [arxiv:2605.06455] PrefixGuard (trajectory-level), [arxiv:2605.14271] HarnessAudit-Bench (resource-access seam) — LCGuard attacks the *latent communication channel* others don't touch +- **2026-05-24** [arxiv:2605.22643] CLAIM: Boiling the Frog — stateful multi-turn benchmark for agentic safety with persistent workspace state; incremental benign→risky escalation across turns; three-level operational risk taxonomy aligned with EU AI Act; **9 frontier models tested**; **44.4% aggregate attack success rate**; range **Claude Haiku 4.5 lowest 20.5% → Gemini 3.1 Flash Lite 92.9%** (4.5× spread); **Seed 2.0 Lite >80%**; **loss-of-control scenarios average 93.3% across all models** | source: https://arxiv.org/abs/2605.22643 | status: open ; temporal-attack dimension on the agent-eval-credibility thread joining [arxiv:2605.10448] Evidence-Supported Bounds / [arxiv:2605.12131] Rollout Cards / [arxiv:2605.11946] Counterfactual Trace Auditing / [arxiv:2605.12673] BenchJack / [arxiv:2605.14271] HarnessAudit-Bench +- **2026-05-24** [arxiv:2605.22672] CLAIM: "Is Capability a Liability?" (Forecasting Research Institute — Merrill, Lee, Ezra Karger) — inverse scaling: **more capable LLMs produce worse distributional forecasts** on tasks involving superlinear growth and regime-change risk; failure concentrates in upper-tail (aggressive upward extrapolation, lower tail unchanged); tested on ForecastBench-Sim contamination-free + real-world COVID-19, measles, housing, hyperinflation; within-family Llama-3.1 analysis isolates both scale and post-training as contributors; **single-threshold scoring at conventional cutoffs reverses sign of capability-accuracy relationship vs. tail-inclusive scoring on the same outputs** | source: https://arxiv.org/abs/2605.22672 | status: open ; extends scoring-rule credibility thread from [arxiv:2605.04624] AuditRepairBench, [arxiv:2605.12131] Rollout Cards (agent-bench score swings under reporting rules) to forecasting benchmarks +- **2026-05-24** [arxiv:2605.22511] CLAIM: Search-E1 — minimal recipe (**vanilla GRPO interleaved with offline self-distillation, no external supervision or auxiliary modules**) for search-augmented reasoning; after each GRPO iteration policy rolls out on training questions; token-level forward KL aligns inference-time distribution to model's privileged-context distribution that reveals more efficient sibling trajectories; **Qwen2.5-3B reaches 0.440 avg EM across 7 QA benchmarks**, claimed to surpass all open-source baselines at the same scale | source: https://arxiv.org/abs/2605.22511 | status: open ; minimal-pipeline counterpoint to [arxiv:2605.06326] TIR (4-stage SFT+RLVR), [arxiv:2605.06642] StraTA, [arxiv:2605.06078] BEACON, [arxiv:2605.15155] SDAR on agentic-RL training recipes +- **2026-05-24** [arxiv:2605.22505] CLAIM: Shor (Yonsei/KAIST cluster — Kai Tzu-iunn Ong et al.) — **priority-ranking evaluation of harness optimizers**: optimizers rank which components most improve agent performance, evaluated step-level without full rollouts; **182 human-verified optimization scenarios across domains, designs, time stages**; priority-ranking performance **correlates with actual multi-step harness-optimization gains**, i.e. cheap step-level metric is valid proxy for expensive end-to-end measurement; open data + code | source: https://arxiv.org/abs/2605.22505 | status: open ; seventh paper in the agent-eval credibility thread; attacks evaluator *cost* where [arxiv:2605.10448] Evidence-Supported Bounds, [arxiv:2605.12131] Rollout Cards, [arxiv:2605.04624] AuditRepairBench, [arxiv:2605.12673] BenchJack, [arxiv:2605.11946] Counterfactual Trace Auditing, [arxiv:2605.14271] HarnessAudit-Bench attack evaluator *fidelity* +- **2026-05-24** [arxiv:2605.22817] CLAIM: VPO (Vector Policy Optimization, MIT cluster — Bahlous-Boldi, Puri, Shenfeld, Damani, Risi, **Omar Khattab**, Hong, **Pulkit Agrawal**) — drop-in replacement for GRPO advantage estimator that trains policies to anticipate **vector-valued rewards** (e.g., per-test-case correctness in code generation) instead of scalar; preserves response-distribution diversity required for inference-time search; **matches or beats strongest scalar RL baselines on test-time search across 4 tasks**; **performance gap widens as search budget increases**; for evolutionary search (AlphaEvolve-style), **VPO models solve problems GRPO models cannot solve at all** | source: https://arxiv.org/abs/2605.22817 | status: open ; third axis with [arxiv:2605.08083] AutoTTS (controller synthesis), [arxiv:2605.15177] OpenDeepThink (Bradley-Terry aggregation of parallel reasoning) on "test-time search needs to be co-designed with training" diff --git a/_meta/processed.json b/_meta/processed.json index ba3d5b3..8bbc1a2 100644 --- a/_meta/processed.json +++ b/_meta/processed.json @@ -1,4 +1,14 @@ [ + { "id": "deepmind-blog-2026-05-19-gemini-3-5-flash", "url": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/", "title": "Gemini 3.5: frontier intelligence with action", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22794", "url": "https://arxiv.org/abs/2605.22794", "title": "MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22733", "url": "https://arxiv.org/abs/2605.22733", "title": "HarnessAPI: A Skill-First Framework for Unified Streaming APIs and MCP Tools", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22791", "url": "https://arxiv.org/abs/2605.22791", "title": "Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22786", "url": "https://arxiv.org/abs/2605.22786", "title": "LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22643", "url": "https://arxiv.org/abs/2605.22643", "title": "Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22672", "url": "https://arxiv.org/abs/2605.22672", "title": "Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22511", "url": "https://arxiv.org/abs/2605.22511", "title": "Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22505", "url": "https://arxiv.org/abs/2605.22505", "title": "Towards Direct Evaluation of Harness Optimizers via Priority Ranking (Shor)", "date_covered": "2026-05-24" }, + { "id": "arxiv:2605.22817", "url": "https://arxiv.org/abs/2605.22817", "title": "Vector Policy Optimization: Training for Diversity Improves Test-Time Search", "date_covered": "2026-05-24" }, { "id": "microsoft-research-2026-05-21-magenticlite-fara15", "url": "https://www.microsoft.com/en-us/research/blog/magenticlite-magenticbrain-fara1-5-an-agentic-experience-optimized-for-small-models/", "title": "MagenticLite, MagenticBrain, Fara1.5: An agentic experience optimized for small models", "date_covered": "2026-05-23" }, { "id": "anthropic-research-2026-05-22-glasswing-initial-update", "url": "https://www.anthropic.com/research/glasswing-initial-update", "title": "Project Glasswing: An initial update", "date_covered": "2026-05-23" }, { "id": "arxiv:2605.21850", "url": "https://arxiv.org/abs/2605.21850", "title": "ACC: Compiling Agent Trajectories for Long-Context Training", "date_covered": "2026-05-23" }, diff --git a/inbox/2026-05-24.md b/inbox/2026-05-24.md new file mode 100644 index 0000000..75b8315 --- /dev/null +++ b/inbox/2026-05-24.md @@ -0,0 +1,141 @@ +--- +date: 2026-05-24 +type: research-digest +tags: [ai-research, daily-digest] +sources_checked: [arxiv-cs.AI, arxiv-cs.CL, arxiv-cs.LG, hf-daily-papers, hf-blog, anthropic-news, anthropic-research, deepmind-blog, google-research-blog, mistral-news, microsoft-research-blog, meta-ai-research, nvidia-developer-blog] +items_count: 10 +conflicts_flagged: 1 +--- + +# AI Research Digest — Sunday, May 24, 2026 + +## TL;DR +- **Gemini 3.5 Flash (May 19) — frontier release missed by prior digests.** Reports **76.2% Terminal-Bench 2.1** and 84.2% multimodal understanding while running ~4× faster than other frontier models; Google claims it **outperforms Gemini 3.1 Pro on coding and agentic benchmarks**. That Terminal-Bench number is the headline — sits well above [arxiv:2605.22535] TerminalWorld's 62.5% frontier ceiling on in-the-wild tasks, which is a `compare` flag (curated bench vs. real recordings showed Pearson r = 0.20 last week). +- **Two papers same week attack agent self-evolution at different layers.** [arxiv:2605.22148] Ratchet (yesterday) optimized the *skill library*; today **MOSS (arxiv:2605.22794)** lets agents rewrite their own *source code* with deterministic multi-stage replay testing — reports **OpenClaw four-task mean grader score 0.25 → 0.61 in a single cycle without human intervention**. Sits above SkillOS / SkillsVote / EvolveMem / Ratchet on the self-evolution stack. +- **HarnessAPI (arxiv:2605.22733) ships a single-source-of-truth framework for MCP tools + streaming HTTP endpoints** generated from typed Python + Pydantic — **74% boilerplate reduction vs. manually maintained FastAPI + FastMCP dual stack** on 6 representative skills. `try-now` for Justin's MCP companion stack; `pip install harnessapi`, Apache 2.0. +- **NVIDIA Labs ships Gated DeltaNet-2** (arxiv:2605.22791) — linear attention with **channel-wise decoupled erase + write gates** beats Mamba-2, Gated DeltaNet, KDA, and Mamba-3 at 1.3B / 100B FineWeb-Edu tokens; specific wins on **long-context RULER needle-in-a-haystack multi-key retrieval**. Code released; relevant pattern for sub-quadratic agent backbones. +- **Multi-turn safety degrades sharply from "boiling the frog" incremental attacks.** [arxiv:2605.22643] Boiling the Frog reports **44.4% aggregate attack success rate** across 9 frontier models — **Claude Haiku 4.5 lowest at 20.5%, Gemini 3.1 Flash Lite 92.9%, Seed 2.0 Lite >80%; loss-of-control scenarios average 93.3%**. Joins the eval-credibility cluster from a multi-turn angle. +- **"More capable models forecast worse on tail-risk problems."** [arxiv:2605.22672] documents inverse scaling on COVID-19, measles, housing, hyperinflation forecasting; **single-threshold scoring at conventional cutoffs reverses sign vs. tail-inclusive scoring on the same outputs** — methodological warning that touches every benchmark using thresholded accuracy. + +## Items + +### 1. Gemini 3.5 Flash — Frontier Intelligence with Action +- **Source:** Google blog / DeepMind (May 19, 2026 — missed in prior digests) +- **Link:** https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/ +- **Authors / org:** Google DeepMind +- **Focus area:** Frontier models / Coding agents / Evaluation +- **What it is:** Gemini 3.5 Flash, positioned as a fast frontier model "for agents and coding." Reports **76.2% on Terminal-Bench 2.1**, **84.2% on multimodal understanding**, and claims to **outperform Gemini 3.1 Pro on challenging coding and agentic benchmarks** while running **~4× faster than other frontier models**. Marketed as the new default for agentic loops where latency matters. +- **Why it matters for Justin's agents:** (a) Direct competitor to Claude Haiku 4.5 / Sonnet 4.6 in the "fast frontier" tier that companion-app loops actually depend on — most MCP runs are bottlenecked on latency × token budget, not peak capability. If the 4× speed claim holds, this changes the cost-shape for any agent product that runs many short turns. (b) **76.2% Terminal-Bench 2.1** is materially above what's been seen at this scale, and sits as a `compare` against [arxiv:2605.22535] TerminalWorld's **62.5% maximum on in-the-wild terminal tasks** — the open question from last week (whether Terminal-Bench scores transfer to TerminalWorld at **Pearson r = 0.20**) now applies directly to Gemini 3.5 Flash. (c) Closed model, not local-fit; relevant only if Justin pipes through Google AI Studio / Vertex. +- **Action signal:** `track` `compare` — wait for independent third-party Terminal-Bench reproduction and for a TerminalWorld score; rank against Claude Haiku 4.5 on whichever benchmark matches Justin's actual task mix. + +### 2. MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems +- **Source:** arXiv (cs.AI, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22794 +- **Authors / org:** Qianshu Cai, Yonggang Zhang, Xianzhang Jia, Wei Xue, Jun Song, Xinmei Tian, Yike Guo +- **Focus area:** Long-running / autonomous loops / Agent memory +- **What it is:** Self-evolving agent that **rewrites its own source code** rather than just text-layer state (prompts, skills, configs) — argued to be the only mechanism that can reach structural failure modes. Architecture is a **deterministic multi-stage pipeline**: code modifications are delegated to external coding agents while MOSS owns stage ordering and verification; candidates pass through replay testing against production failure batches before user-consent-gated deployment with post-deploy health monitoring. Headline: **OpenClaw four-task mean grader score 0.25 → 0.61 in a single cycle without human intervention**. Code on GitHub. +- **Why it matters for Justin's agents:** Adds a sixth layer to the ledger's self-evolution stack: [arxiv:2605.06614] SkillOS (curator/executor split), [arxiv:2605.18401] SkillsVote (lifecycle gating), [arxiv:2605.13941] EvolveMem (architecture-level evolution), [arxiv:2605.22148] Ratchet (outcome-driven skill retirement), [arxiv:2605.09998] Continual Harness (reset-free game loops). MOSS sits above all of them — its primitive of change is the *agent code itself*, not skills or prompts. Concrete take for Justin: (a) the deterministic-multi-stage-with-replay-testing pattern is the same architectural shape his MCP companion stack would need before letting Claude Code edit its own harness; (b) **0.25 → 0.61 in a single cycle** is a large step but on a narrow OpenClaw subset — open empirical question whether it survives broader task distributions; (c) MOSS pairs as `compare` with item #3 below — opposite directions on the same axis (code-rewriting vs. spec-driven code generation). +- **Action signal:** `track` `compare` — review the code release; instructive pattern even if Justin doesn't deploy self-rewriting in production. The replay-testing + consent-gated deploy primitives are reusable independently. + +### 3. HarnessAPI: A Skill-First Framework for Unified Streaming APIs and MCP Tools +- **Source:** arXiv (cs.AI / cs.SE, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22733 +- **Authors / org:** Edwin Jose (solo author) +- **Focus area:** Tools & connectors +- **What it is:** Framework that lets developers maintain a **single typed Python function + Pydantic schema as the source of truth**, then auto-derives both an HTTP/SSE streaming endpoint and an MCP tool registration. Dual-mode content negotiation serves SSE-streaming and JSON clients identically; OpenAPI/Swagger UI generated automatically; dynamic code-generation propagates Pydantic types through to MCP. Headline: **74% reduction in framework-facing boilerplate vs. manually maintained FastAPI + FastMCP dual-stack** across 6 representative skills. Apache 2.0; `pip install harnessapi`. +- **Why it matters for Justin's agents:** Directly addresses the duplication problem in any MCP companion stack — every tool typically gets implemented twice (once as an HTTP endpoint for the web app, once as an MCP tool for the agent). HarnessAPI collapses the two into one source of truth, which is the exact pain point Justin's stack already shadows. Pairs as production-side counterpart to research benchmarks in the MCP cluster — [arxiv:2605.10787] ComplexMCP (300+ tool benchmark), [arxiv:2605.12481] ToolCUA (in-model tool routing), [arxiv:2605.07112] Switchcraft (cross-model dispatch). Solo-author release with shipped package + GitHub is the unusually-actionable side of arXiv — most papers in this cluster need code-release waits. +- **Action signal:** `try-now` — `pip install harnessapi`, port one existing dual-stack skill, measure boilerplate-LOC delta and check that SSE + MCP serve identically. Low-risk drop-in. + +### 4. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention +- **Source:** arXiv (cs.AI, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22791 +- **Authors / org:** Ali Hatamizadeh, **Yejin Choi**, **Jan Kautz** (NVIDIA Labs) +- **Focus area:** Context engineering +- **What it is:** Linear-attention architecture that splits memory updates into **two independent channel-wise gates** — an erase gate b_t and a write gate w_t — where prior Mamba/DeltaNet variants used a single scalar coupling the two. Trained at **1.3B parameters on 100B FineWeb-Edu tokens**. Headlines: **strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants** on language modeling, commonsense reasoning, and retrieval; specific wins on **long-context RULER needle-in-a-haystack multi-key retrieval**. Code released as NVlabs/GatedDeltaNet-2. +- **Why it matters for Justin's agents:** Linear-attention architectures are the credible alternative path to the transformer KV cluster (item-2/3/4 of last week — OScaR / OCTOPUS / Mix-Quant — were all KV-cache *compression*; this is a different attention substrate entirely). NVIDIA shipping at 1.3B + 100B tokens with code is the operating-system signal that sub-quadratic agent backbones are landing. Local-fit angle: a 1.3B model isn't itself useful, but the architecture is what would back any 8B–30B linear-attention model that could run with **vastly larger context windows** on M5 Max's bandwidth-limited 64 GB unified memory (KV cache, not weights, is the binding constraint at long context). Pairs with [arxiv:2605.22106] ArborKV (tree-aware KV) and [arxiv:2605.09649] learnable KV eviction as three complementary attention-substrate optimizations. +- **Action signal:** `track` `local-fit` — track for a >7B Gated DeltaNet-2 scale-up; if and when a Qwen3-30B-class linear-attention release drops, this is the architecture pattern to benchmark against. + +### 5. LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems +- **Source:** arXiv (cs.AI / cs.MA / cs.LG, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22786 +- **Authors / org:** Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas, Prasanna Sattigeri, Karthikeyan Natesan Ramamurthy (RPI / IBM Research cluster) +- **Focus area:** Agent orchestration / Context engineering +- **What it is:** First paper to define **KV-cache leakage as a security boundary in multi-agent systems**. Setup: when agents share KV caches to save recomputation, an adversarial decoder can reconstruct one agent's sensitive inputs from another agent's cache view. **LCGuard** is an adversarial training framework that learns representation-level transformations on shared caches — safety metric explicitly is "a cache is unsafe if an adversarial decoder can recover agent-specific sensitive inputs." Reports consistent reductions in reconstruction-based leakage and attack success while maintaining task performance vs. standard KV-sharing baselines. +- **Why it matters for Justin's agents:** Multi-agent stacks that share inference state for efficiency (the production pattern Anthropic Skills, Microsoft MagenticBrain, Maestro all assume) inherit a leakage surface that no current open framework defends against. For a federal data-engineering context where one agent might handle PII and another doesn't, this is the kind of finding that becomes a compliance gate before any subagent-sharing architecture ships. Joins [arxiv:2605.04785] AgentTrust (call-level interceptor), [arxiv:2605.06455] PrefixGuard (trajectory-level monitor), [arxiv:2605.14271] HarnessAudit-Bench (resource-access seam audits) — fourth angle on the agent-safety surface, attacking the *latent communication channel* the other three don't touch. +- **Action signal:** `track` — wait for code release; flag as a known unmodeled risk in any current multi-agent KV-sharing design. + +### 6. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety +- **Source:** arXiv (cs.CL, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22643 +- **Authors / org:** 14 researchers led by Piercosma Bisconti (AI safety / EU-AI-Act-aligned) +- **Focus area:** Evaluation / Safety +- **What it is:** Stateful multi-turn benchmark where attacks progress incrementally from benign to risky requests across conversation turns ("boiling the frog"). Evaluates 9 frontier models with a persistent workspace state and a three-level operational risk taxonomy aligned with the AI Act's high-risk contexts. Headlines: **44.4% aggregate attack success rate**; range **Claude Haiku 4.5 lowest at 20.5%** to **Gemini 3.1 Flash Lite 92.9%**; **Seed 2.0 Lite >80%**; **loss-of-control scenarios average 93.3% across all models**. +- **Why it matters for Justin's agents:** Reframes "safety" from single-shot refusal classification to a **temporal property of the conversation as a whole** — the same property that matters in any long-horizon companion-app session. The Claude Haiku 4.5 / Gemini 3.1 Flash Lite spread (4.5×) is the kind of selection signal that should bias a fast-tier choice for Justin's loops. **93.3% loss-of-control rate** is the more important number — every model in scope is highly susceptible to incremental compromise on loss-of-control scenarios specifically, regardless of provider. Joins the agent-eval-credibility thread ([arxiv:2605.10448] Evidence-Supported Bounds / [arxiv:2605.12131] Rollout Cards / [arxiv:2605.11946] Counterfactual Trace Auditing / [arxiv:2605.12673] BenchJack / [arxiv:2605.14271] HarnessAudit-Bench) with a specifically *temporal-attack* dimension the others don't cover. +- **Action signal:** `track` — track for code release; the 93.3% loss-of-control finding is the kind of empirical anchor to cite when arguing for fresh-session resets or trajectory-length caps in any companion product. + +### 7. Is Capability a Liability? More Capable Language Models Make Worse Forecasts +- **Source:** arXiv (cs.AI, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22672 +- **Authors / org:** Nick Merrill, Jaeho Lee, **Ezra Karger** (Forecasting Research Institute) +- **Focus area:** Evaluation / Frontier models +- **What it is:** Inverse-scaling finding — more capable LLMs produce **worse distributional forecasts** on tasks involving superlinear growth and regime-change risk (the structures common in finance, epidemiology, energy demand). Failure concentrates in upper-tail predictions: larger models aggressively extrapolate growth trends while leaving lower-tail estimates unchanged. Tested on **ForecastBench-Sim (FBSim)** contamination-free simulated benchmark + real-world validation across COVID-19, measles, housing markets, hyperinflation; within-family Llama-3.1 analysis isolates both scale and post-training as contributors. **Methodologically critical claim: single-threshold scoring at conventional cutoffs reverses the sign of the capability-accuracy relationship on the same outputs**, i.e. tail-inclusive scoring shows degradation that threshold scoring hides. +- **Why it matters for Justin's agents:** Two pulls. (a) Direct datapoint for Justin's federal-data-engineering work where forecasting is structurally similar — never assume frontier capability transfers to tail-risk regimes; always validate at the threshold separately from the tail. (b) **The threshold-scoring sign flip is the same class of methodological problem** the ledger has been tracking on agent benchmarks ([arxiv:2605.04624] AuditRepairBench: ranking instability under evaluator reconfiguration; [arxiv:2605.12131] Rollout Cards: 20.9-pp score swings on identical rollouts under different reporting rules). This paper extends that thread from agent-benchmark scoring to *forecasting-benchmark scoring* — the credibility-of-scoring-rules problem is general, not agent-specific. +- **Action signal:** `track` `compare` — open empirical question whether the threshold-scoring sign flip also surfaces on Claude / Gemini coding-benchmark thresholded pass@k; this should be tested before quoting any threshold-headline benchmark number for tail-risk applications. + +### 8. Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning +- **Source:** arXiv (cs.AI, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22511 +- **Authors / org:** Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Xuxin Zhang, Huangyu Dai, Lingtao Mao +- **Focus area:** Reasoning & inference-time compute / Tools & connectors +- **What it is:** Minimal recipe — **vanilla GRPO interleaved with offline self-distillation (OFSD)**, no external supervision or auxiliary modules. After each GRPO iteration the policy rolls out on training questions; a token-level forward KL objective aligns the inference-time distribution to the model's distribution under privileged context that reveals more efficient sibling trajectories. Headline: **Qwen2.5-3B reaches 0.440 average EM across 7 QA benchmarks**, claimed to surpass all open-source baselines at the same scale. +- **Why it matters for Justin's agents:** Joins the agentic-RL recipe line — [arxiv:2605.06326] TIR (4-stage SFT+RLVR), [arxiv:2605.06642] StraTA (strategy abstraction + GRPO), [arxiv:2605.06078] BEACON (milestone-segmented reward shaping), [arxiv:2605.15155] SDAR (self-distilled agentic RL). Search-E1's selling point is **minimality**: no synthetic data, no auxiliary teacher, no reward shaping past GRPO. For Justin, the implication is the *training pipeline complexity floor* for getting search-agent gains keeps dropping; if his MCP harness ever generates enough trajectory data, a single GRPO+OFSD pass is plausibly all the post-training infrastructure he needs. +- **Action signal:** `track` — wait for code release; competitive only at 3B scale in the reported result so the local-fit story is "minimum-viable training recipe" rather than a deployable model. + +### 9. Shor: Direct Evaluation of Harness Optimizers via Priority Ranking +- **Source:** arXiv (cs.AI, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22505 +- **Authors / org:** 12 researchers led by Kai Tzu-iunn Ong, including Minseok Kang, Dongwook Choi (Yonsei / KAIST cluster — author overlap with prior optimizer papers) +- **Focus area:** Evaluation +- **What it is:** Methodology paper attacking a specific gap in the agent-eval credibility thread — **harness-optimizer evaluation**. Current practice measures only end-to-end performance lift, which is expensive (full rollouts × multiple seeds) and hides intermediate optimization decisions. Shor proposes **priority ranking**: harness optimizers rank which components (tools, scaffolding pieces) would most improve agent performance, evaluated at step-level without rollouts. Releases **Shor — 182 human-verified optimization scenarios** spanning domains, designs, and time stages. Claim: priority-ranking performance **correlates with actual multi-step harness-optimization gains**, i.e. the cheap step-level metric is a valid proxy. +- **Why it matters for Justin's agents:** Operational implication is that harness-optimization experiments don't need full-rollout cost. For Justin's stack where harness tuning is the actual budget-eating loop (running a candidate scaffold across 100s of tasks × multiple seeds), a validated step-level proxy is the difference between "test 5 scaffolds" and "test 50." Sits next to [arxiv:2605.10448] Evidence-Supported Bounds (outcome-evidence reporting layer) and [arxiv:2605.12131] Rollout Cards (failure/error/skip accounting) as the **seventh paper in the agent-eval credibility thread** alongside [arxiv:2605.04624] AuditRepairBench, [arxiv:2605.12673] BenchJack, [arxiv:2605.11946] Counterfactual Trace Auditing, [arxiv:2605.14271] HarnessAudit-Bench — Shor attacks evaluator *cost* where the others attack evaluator *fidelity*. +- **Action signal:** `track` — open data + code; useful pattern even without adoption — measure correlation between any cheap proxy and the expensive ground truth before trusting it. + +### 10. Vector Policy Optimization: Training for Diversity Improves Test-Time Search +- **Source:** arXiv (cs.LG, submitted May 21, 2026) +- **Link:** https://arxiv.org/abs/2605.22817 +- **Authors / org:** Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, **Omar Khattab**, Zhang-Wei Hong, **Pulkit Agrawal** (MIT cluster — Khattab is the DSPy author) +- **Focus area:** Reasoning & inference-time compute +- **What it is:** Standard RL post-training on scalar rewards collapses to low-entropy response distributions that **can't be searched at inference time**. **VPO** trains policies to anticipate **vector-valued rewards** (e.g., per-test-case correctness in code generation) — a drop-in replacement for the GRPO advantage estimator. Headlines: **matches or beats the strongest scalar RL baselines on test-time search across 4 tasks**; **performance gap widens as search budget increases**; for evolutionary search (AlphaEvolve-style), **VPO models solve problems GRPO models cannot solve at all**. +- **Why it matters for Justin's agents:** Names and fixes a specific training-inference mismatch — post-training that produces a single-mode output distribution is incompatible with any inference-time-search method (best-of-N, beam, evolutionary, Monte-Carlo). For Justin's stack, the operationally-relevant version is: any subagent that will be used inside a parallel search (Anthropic-style multi-shot pass@k, multi-rollout planner) needs to be trained with diversity-preserving rewards, not scalar pass-rate. Pairs as a third axis with [arxiv:2605.08083] AutoTTS (controller synthesis over cached trajectories) and [arxiv:2605.15177] OpenDeepThink (Bradley-Terry aggregation of parallel reasoning) — three independent papers in three weeks on "test-time search needs to be co-designed with training." Khattab/Agrawal author signature lends the implementation credibility. +- **Action signal:** `track` `compare` — wait for code; the GRPO-cannot-solve-at-all framing is strong and reproducible-against; will likely become standard if it holds up at larger scale. + +## Local-model corner +Slow day for purely local releases. Three indirect items: + +- **HarnessAPI (item #3)** — not a model, but the most directly actionable Mac-local item today. `pip install harnessapi` works on macOS; the framework runs anywhere Python + FastAPI run. For Justin's MCP companion stack, this is a same-day install candidate. + +- **Gated DeltaNet-2 (item #4)** — 1.3B too small to be a usable local model directly, but the architecture is the relevant pattern for any future >7B sub-quadratic-attention release. Track NVlabs/GatedDeltaNet-2 for scale-ups; if a Qwen3-30B-class Gated-DeltaNet variant drops, the long-context RULER NIAH gains would be a key local-fit signal on the M5 Max 64 GB memory budget where KV cache (not weights) is binding. + +- **No usable open-weight local model in today's batch.** Yesterday's Microsoft MagenticBrain (Qwen3-14B) + Fara 1.5 (4B/9B/27B) cluster remains the active local-fit reference; ACC-tuned Qwen3-30B-A3B (yesterday item #3) remains the active training-recipe candidate when code lands. + +## Conflicts & open questions +Cross-referenced against `_meta/claims-ledger.md`: + +- **`compare` — Gemini 3.5 Flash 76.2% Terminal-Bench 2.1 vs. TerminalWorld 62.5% frontier ceiling.** Item #1 (Gemini 3.5 Flash) reports 76.2% on the curated Terminal-Bench. Item from yesterday [arxiv:2605.22535] TerminalWorld reports 62.5% maximum across 8 frontier models × 6 agents on **in-the-wild terminal recordings**, with **Pearson r = 0.20 between Terminal-Bench and TerminalWorld-Verified scores on the same systems**. Open empirical question: where does Gemini 3.5 Flash actually rank on TerminalWorld? The r = 0.20 finding implies the curated-bench ranking does not transfer — until a TerminalWorld run is published for Gemini 3.5 Flash, the 76.2% headline number is unverified as a real-world capability signal. This is the standing "compare" cluster the ledger has been tracking on agent-benchmark credibility ([arxiv:2605.04624] AuditRepairBench, [arxiv:2605.12131] Rollout Cards, [arxiv:2605.10448] Evidence-Supported Bounds, [arxiv:2605.12673] BenchJack, [arxiv:2605.14271] HarnessAudit-Bench, [arxiv:2605.11946] Counterfactual Trace Auditing, now [arxiv:2605.22505] Shor) — now extended to a frontier-model release announcement. + +No direct contradictions of prior ledger claims today. + +## Sources checked +- arXiv cs.AI / cs.CL / cs.LG recent listings: ~50 new papers considered across all three, 8 kept after yesterday's overlap removed +- Hugging Face Daily Papers: ~10 considered, 0 distinct kept (overlap with arXiv already covered; DelTA RLVR credit-assignment paper is interesting but at 191 upvotes was likely batched yesterday — no benchmark-shifting claim worth covering separately today) +- Hugging Face Blog: 1 new post in window (LeRobot Humanoid — off-priority for Justin's stack) +- Anthropic news / research: only Project Glasswing (covered yesterday) is in the 48-hour window; "Widening the conversation on frontier AI" (May 19) has one interesting empirical claim (a tool-call mid-task that reminds Claude of its ethical commitments markedly lowers misaligned-behavior rates on internal evals) but no quantitative benchmark numbers — `track` only, not picked up as a standalone item +- DeepMind / Google blog: Gemini 3.5 Flash (May 19) picked up as item #1 — missed by prior digests +- OpenAI news: HTTP 403 on both `/news/` and `/research/` — could not access — retry next run +- Google Research blog: ERA / Empirical Research Assistance posts already overlap upstream coverage +- Microsoft Research blog: only Vega (zero-knowledge proofs for AI identity, May 21) in window — off-priority for agent stack +- Mistral news: no new posts since 2026-04-29 Vibe / Medium 3.5 +- Meta AI research: no new posts since 2026-04-08 +- NVIDIA Developer Blog (AI section): 4 posts in window (Agent Evaluation, Verified Agent Skills, Deep Research Skill, Agent Customization) — all marketing-shaped pieces without benchmark numbers, `track`-tier only +- Allen AI blog: no new posts +- Hacker News / r/LocalLLaMA discovery layer: skipped — already at 10 high-signal items with strong primary-source verification