From 7e3fa22e059dce69b00bfe860597380685f3eebc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 15 May 2026 10:19:08 +0000 Subject: [PATCH] digest: 2026-05-15 8 verified items spanning evaluation, agent memory, orchestration, coding agents, tool use, reasoning, and inference infrastructure. WildClawBench retrofits the cross-benchmark Opus 4.7 spread as harness variance (18 pp range). PREPING provides the positive recipe for the consolidation-decay failure mode flagged 5/14. Three conflicts resolved or extended; one open question raised. https://claude.ai/code/session --- _meta/claims-ledger.md | 8 +++ _meta/processed.json | 8 +++ inbox/2026-05-15.md | 141 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 157 insertions(+) create mode 100644 inbox/2026-05-15.md diff --git a/_meta/claims-ledger.md b/_meta/claims-ledger.md index 9808d50..87d74de 100644 --- a/_meta/claims-ledger.md +++ b/_meta/claims-ledger.md @@ -73,3 +73,11 @@ - **2026-05-14** [arxiv:2605.12925] CLAIM: AgentLens — analysis of 2,614 trajectories from 8 models on 60 SWE-bench Verified tasks finds **10.7% of passing solutions are "Lucky Passes"** (regression cycles, blind retries, missing verification, temporally disordered exploration); per-model Lucky rate ranges **0.5% → 23.2%**; releases 1,815 trajectories labeled Lucky/Solid/Ideal; **up to 5-position model rank shifts** when ranked by process quality instead of pass rate | source: https://arxiv.org/abs/2605.12925 | status: open ; SWE-bench-specific instantiation of [arxiv:2605.11946] Counterfactual Trace Auditing's "aggregate gains hide mixed-sign behavior changes" finding - **2026-05-14** [arxiv:2605.12975] CLAIM: PyRAG reformulates multi-hop RAG as program synthesis and execution — Python programs call retrieval as a tool, get deterministic feedback, self-repair against compiler/runtime errors; consistent gains over strong baselines on PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle under both training-free and RL-trained settings | source: https://arxiv.org/abs/2605.12975 | status: open ; multi-hop-specific complement to [arxiv:2605.05242] Direct Corpus Interaction (which replaced general retrieval with code-over-corpus); training-free result the operationally useful one for existing MCP-based stacks - **2026-05-14** [arxiv:2605.13301] CLAIM: SU-01 (30B-A3B reasoning backbone, ~340K curriculum-SFT trajectories + ~200 RL steps + test-time scaling, reasoning trajectories beyond 100K tokens) claims gold-medal-level performance on IMO 2025 / USAMO 2026 / IPhO 2024 / IPhO 2025; open recipe | source: https://arxiv.org/abs/2605.13301 | status: open ; same parameter class as Qwen3.6-35B-A3B → local-fit math reasoning subagent on M5 Max once weights release; recipe overlaps with [arxiv:2605.06326] Teaching Thinking Models to Reason with Tools (TIR) +- **2026-05-15** [arxiv:2605.10912] CLAIM: WildClawBench — 60 human-authored bilingual multimodal tasks (~8 min / 20+ tool calls each) inside reproducible Docker with real CLI agent harnesses (OpenClaw, Claude Code, Codex, Hermes Agent); 19 frontier models evaluated; **Opus 4.7 best at 62.2% under OpenClaw, all others <60%**; **harness switching alone shifts a single model by up to 18 pp** | source: https://arxiv.org/abs/2605.10912 | status: open ; **resolves cross-benchmark Opus 4.7 spread** (Vals AI Finance 64.37% / AstaBench 58.0% / WildClawBench 62.2%) as a harness-variance phenomenon; sharpens [arxiv:2605.04624] AuditRepairBench by identifying harness as a leak channel above evaluator reconfiguration; joins [arxiv:2605.12673] BenchJack, [arxiv:2605.10448] Evidence-Supported Bounds, [arxiv:2605.12131] Rollout Cards, [arxiv:2605.12925] AgentLens, [arxiv:2605.11946] Counterfactual Trace Auditing as 6th paper in the agent-eval credibility thread +- **2026-05-15** [arxiv:2605.15040] CLAIM: Microsoft Research Orchard — open-source agent training framework (not just orchestration) with three competitive recipes: **Orchard-SWE 67.5% SWE-bench Verified**, **Orchard-GUI 4B VLM 74.1% WebVoyager**, **Orchard-Claw 73.9% pass@3**; core innovation is Orchard Env sandbox-lifecycle primitive | source: https://arxiv.org/abs/2605.15040 | status: open ; runtime-substrate convergence with [arxiv:2605.10913] Shepherd typed event trace and [arxiv:2605.06614] SkillOS Markdown SkillRepo — three independent open frameworks pointing at "typed environment substrate, not orchestration glue, is the missing layer"; 4B GUI agent at competitive WebVoyager is local-fit for M5 Max +- **2026-05-15** [arxiv:2605.13880] CLAIM: PREPING (KAIST AI) — task-free agent memory via Proposer/Solver/Validator separation, validator-gated entry into memory; reaches strong-playbook-baseline performance at **2.99× lower deploy cost on AppWorld, 2.23× on BFCL v3**; success requires explicit proposer control over feasibility/redundancy/coverage plus selective memory updates | source: https://arxiv.org/abs/2605.13880 | status: open ; **positive recipe answer to [arxiv:2605.12978] consolidation-decay** — structured/role-separated/validator-gated consolidation builds memory while background/same-model/default-on consolidation breaks it; same separation pattern as [arxiv:2605.06614] SkillOS curator/executor decomposition +- **2026-05-15** [arxiv:2605.14678] CLAIM: π-Bench — 100 multi-turn proactive personal-assistant tasks across 5 personas; finds task completion and proactivity are statistically distinct capabilities; prior interactions improve hidden-intent resolution in subsequent tasks | source: https://arxiv.org/abs/2605.14678 | status: open ; capability-side analogue of [microsoft-research-2026-05-11-socialreasoning-bench] "completion ≠ value" finding from a different operational angle; open empirical question whether proactivity/optimality correlate at the model level across both benchmarks +- **2026-05-15** [arxiv:2605.15177] CLAIM: OpenDeepThink — Bradley-Terry pairwise aggregation across parallel reasoning candidates replaces noisy point-wise LLM-as-judge selection; **Gemini 3.1 Pro Codeforces Elo +405 over 8 LLM-call rounds (~27 min)**; transfers across model strengths without retuning; gains concentrate in objectively verifiable domains and reverse in subjective ones; releases CF-73 dataset (73 expert-rated Codeforces problems, 99% agreement vs. official verdicts) | source: https://arxiv.org/abs/2605.15177 | status: open ; orthogonal to [arxiv:2605.10805] RACER (judge routing as DRO) — OpenDeepThink replaces judgment primitive; RACER routes between judge types; both attack judge-quality bottleneck in TTS +- **2026-05-15** [arxiv:2605.15155] CLAIM: SDAR (gated self-distillation atop GRPO; sigmoid gate on detached token-level signals amplifies teacher-endorsed tokens, moderates rejection signals) tested on Qwen2.5/3 reports **+9.4% ALFWorld, +7.0% Search-QA, +10.2% WebShop-Acc** over GRPO baselines | source: https://arxiv.org/abs/2605.15155 | status: open ; **4th orthogonal mechanism hitting the ALFWorld 90+ plateau** alongside [arxiv:2605.06078] BEACON (92.9%), [arxiv:2605.06642] StraTA (93.1%), [arxiv:2605.05413] Constant-Context Skill Learning (89.6% unseen) — benchmark now appears method-saturated; gated self-distillation likely to land in trl/verifiers libraries +- **2026-05-15** [arxiv:2605.15041] CLAIM: CAST treats historical execution trajectories as structured cases, extracts complexity + failure profiles, feeds them into fine-grained reward design and adaptive reasoning during RL; on BFCLv2 + ToolBench reports **+5.85 pp execution accuracy with 26% shorter reasoning length** | source: https://arxiv.org/abs/2605.15041 | status: open ; positive-policy answer to [arxiv:2605.00136] tool-use tax; orthogonal mechanism to [arxiv:2605.00737] hidden-state need estimators (CAST learns from completed traces; hidden-state estimators learn from in-call activations) and [arxiv:2605.07990] linearly-readable tool calls (mechanistic counterpart) +- **2026-05-15** [hf-blog-2026-05-14-continuous-async] CLAIM: HF Transformers `ContinuousBatchingAsyncIOs` decouples CPU batch prep from GPU compute via 3 CUDA streams (H2D / compute / D2H) + CUDA events + double-buffering + carry-over masks; identifies 24% of total runtime as GPU-idle waiting for CPU prep; achieves **300.6s → 234.5s on 8K tokens, batch 32, 8B model — 22% generation-time speedup; GPU utilization 76.0% → 99.4%**; zero new kernels, no model changes | source: https://huggingface.co/blog/continuous_async | status: open ; portable pattern for llama.cpp and mlx-lm where same CPU-prep / GPU-compute idle-window problem exists on unified-memory architectures; sanity-check ceiling on serving-cost spreadsheets diff --git a/_meta/processed.json b/_meta/processed.json index ba4968f..8888482 100644 --- a/_meta/processed.json +++ b/_meta/processed.json @@ -1,4 +1,12 @@ [ + { "id": "arxiv:2605.10912", "url": "https://arxiv.org/abs/2605.10912", "title": "WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.15040", "url": "https://arxiv.org/abs/2605.15040", "title": "Orchard: An Open-Source Agentic Modeling Framework", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.13880", "url": "https://arxiv.org/abs/2605.13880", "title": "PREPING: Building Agent Memory without Tasks", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.14678", "url": "https://arxiv.org/abs/2605.14678", "title": "π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.15177", "url": "https://arxiv.org/abs/2605.15177", "title": "OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.15155", "url": "https://arxiv.org/abs/2605.15155", "title": "Self-Distilled Agentic Reinforcement Learning (SDAR)", "date_covered": "2026-05-15" }, + { "id": "arxiv:2605.15041", "url": "https://arxiv.org/abs/2605.15041", "title": "CAST: Case-Based Calibration of Adaptive Reasoning and Execution for LLM Tool Use", "date_covered": "2026-05-15" }, + { "id": "hf-blog-2026-05-14-continuous-async", "url": "https://huggingface.co/blog/continuous_async", "title": "Unlocking asynchronicity in continuous batching", "date_covered": "2026-05-15" }, { "id": "arxiv:2605.12978", "url": "https://arxiv.org/abs/2605.12978", "title": "Useful Memories Become Faulty When Continuously Updated by LLMs", "date_covered": "2026-05-14" }, { "id": "arxiv:2605.12922", "url": "https://arxiv.org/abs/2605.12922", "title": "When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction", "date_covered": "2026-05-14" }, { "id": "arxiv:2605.13050", "url": "https://arxiv.org/abs/2605.13050", "title": "Context Training with Active Information Seeking", "date_covered": "2026-05-14" }, diff --git a/inbox/2026-05-15.md b/inbox/2026-05-15.md new file mode 100644 index 0000000..208dbfd --- /dev/null +++ b/inbox/2026-05-15.md @@ -0,0 +1,141 @@ +--- +date: 2026-05-15 +type: research-digest +tags: [ai-research, daily-digest] +sources_checked: [arxiv-cs.AI, arxiv-cs.CL, arxiv-cs.LG, arxiv-cs.MA, hf-daily-papers, hf-blog, anthropic-news, anthropic-research, deepmind-blog, openai-news, mistral-news, research-google, ai-meta, allenai, microsoft-research] +items_count: 8 +conflicts_flagged: 3 +--- + +# AI Research Digest — Friday, May 15, 2026 + +## TL;DR +- **A real-world, multi-harness, long-horizon agent benchmark lands and the headline reframes a month of Opus 4.7 numbers.** WildClawBench runs 60 bilingual / multimodal tasks (~8 min each, 20+ tool calls) inside actual CLI agent harnesses — **OpenClaw, Claude Code, Codex, Hermes Agent** — under reproducible Docker. Best score across **19 frontier models is Opus 4.7 at 62.2% under OpenClaw**; every other model sits below 60%. The most consequential finding: **harness choice alone shifts a single model's performance by up to 18 percentage points**, which directly explains why Opus 4.7 reads as 64.37% (Vals AI Finance), 58.0% (AstaBench), and 62.2% (WildClawBench) depending on the wrapper. Flagged below. +- **Microsoft Research ships an open agent training framework with three recipes already at competitive numbers.** Orchard releases not just orchestration glue but **training infrastructure**: Orchard-SWE hits 67.5% SWE-bench Verified, Orchard-GUI hits 74.1% WebVoyager from a 4B vision-language backbone, Orchard-Claw hits 73.9% pass@3 on personal-assistant tasks. First open framework explicitly designed for *training* agents at competitive quality, not just running them. +- **Task-free agent memory becomes runnable.** PREPING (KAIST) constructs agent memory entirely from self-generated synthetic practice via a Proposer / Solver / Validator separation — matches strong playbook baselines at **2.99× lower deploy cost on AppWorld, 2.23× on BFCL v3**. Combined with yesterday's "useful memories become faulty when continuously updated" finding, the synthesis is now visible: *structured, separated, gated* consolidation works; *background, same-model, default-on* consolidation is the failure mode. +- **A new long-horizon benchmark separates "proactivity" from "task completion."** π-Bench (100 multi-turn tasks × 5 personas) reports that **task completion and proactivity are statistically distinct capabilities** — matching Microsoft's SocialReasoning-Bench finding that *completion ≠ value*. Two independent papers in three days saying the same thing. +- **Parallel test-time scaling gets a robust selector.** OpenDeepThink replaces noisy point-wise judges with Bradley-Terry pairwise aggregation across candidates; **Gemini 3.1 Pro Codeforces Elo +405 points over 8 rounds (~27 min)**, transfers across model strengths without retuning. CF-73 dataset (73 expert-rated Codeforces problems, 99% agreement against official verdicts) released alongside. +- **The ALFWorld 90+ club gets a 4th mechanism.** Self-Distilled Agentic RL (gated self-distillation atop GRPO) reports **+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc** over GRPO baselines on Qwen2.5/3, joining BEACON (92.9%), StraTA (93.1%), and Constant-Context Skill Learning (89.6% unseen) as mechanically distinct paths to the same plateau. +- **Tool-use gets a case-based calibration layer.** CAST treats historical trajectories as structured cases, extracts complexity + failure profiles, feeds them into reward design; **+5.85 pp execution accuracy with 26% shorter reasoning** on BFCLv2 + ToolBench. +- **Continuous batching gets an asynchronous primitive worth 22% in tok/s, for free.** HF Transformers' `ContinuousBatchingAsyncIOs` decouples CPU batch prep from GPU compute via 3 CUDA streams + double-buffering — **300.6s → 234.5s for 8K tokens, batch 32, 8B model; GPU utilization 76% → 99.4%**. Zero new kernels, no model changes. + +## Items + +### 1. WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation +- **Source:** arXiv (cs.AI) / HF Daily Papers (top-trending today, 364 upvotes) +- **Link:** https://arxiv.org/abs/2605.10912 +- **Authors / org:** Intern Large Models team (author list not on abstract; "Intern" is the Shanghai AI Lab group behind InternLM / InternVL) +- **Focus area:** Evaluation / Agent orchestration +- **What it is:** First large-scale agent benchmark that runs frontier models *inside their actual production CLI harnesses* on real long-horizon tasks. **60 human-authored, bilingual, multimodal tasks across 6 thematic categories**, each ~8 minutes and 20+ tool calls. Hosted in reproducible Docker containers wired to live CLI agents — **OpenClaw, Claude Code, Codex, Hermes Agent** — running real tools. Grading combines deterministic rule checks, filesystem/environment-state auditing for side effects, and LLM/VLM semantic judges. Headline result across **19 frontier models**: **Opus 4.7 best at 62.2% under OpenClaw; every other model below 60%**. The hidden bombshell: **harness switching alone shifts a single model's performance by up to 18 percentage points** — the harness wrapper is a first-class evaluation variable. +- **Why it matters for Justin's agents:** Two structural takeaways for an applied-agent stack. First, this is the **first benchmark that explicitly grades agents inside Claude Code** (and OpenClaw, which is the parallel open-source-Claude-Code-clone harness Justin has been tracking). The 62.2% Opus 4.7 number is the closest public approximation to "what an applied developer actually gets out of the box" — useful as a sanity-check ceiling for any internal eval Justin runs against Claude Code. Second, the **18-percentage-point harness sensitivity** retrofits a confusing month of Opus 4.7 numbers: Vals AI Finance 64.37%, AstaBench 58.0%, BioMysteryBench ~77%, now WildClawBench 62.2%. These were never measuring the same thing — the harness was uncontrolled. For Justin's MCP-based products, this means **the harness layer (system prompt structure, tool-result formatting, error-recovery patterns) is worth at least as much engineering investment as the model choice**. +- **Action signal:** `try-now` `compare` — directly relevant; the harness-sensitivity finding alone is operationally important. Cross-references: [anthropic-news-2026-05-05-finance-agents] (Opus 4.7 / Vals AI 64.37%), 5/6 AstaBench Opus 4.7 (58.0%), [anthropic-research-2026-04-29-biomysterybench] (Opus 4.6 ~77%, predates 4.7 by one release). See Conflicts section for the harness-variability angle. + +### 2. Orchard: An Open-Source Agentic Modeling Framework +- **Source:** arXiv (cs.AI, May 14, 2026) +- **Link:** https://arxiv.org/abs/2605.15040 +- **Authors / org:** Baolin Peng, Wenlin Yao, and colleagues — **Microsoft Research-style author cluster** (Peng and Yao are MSR Redmond senior researchers) +- **Focus area:** Agent orchestration / Tools & connectors / Coding agents +- **What it is:** Open-source framework explicitly designed for **training agents**, not just running them. Three named recipes ship with quantified results: **Orchard-SWE reaches 67.5% on SWE-bench Verified** after training; **Orchard-GUI is a 4B vision-language computer-use agent at 74.1% WebVoyager success**; **Orchard-Claw is a personal-assistant agent at 73.9% pass@3**. Core infrastructure is **Orchard Env** — a lightweight environment service providing reusable primitives for sandbox lifecycle management (a deliberate inversion of the "framework is the orchestrator" pattern in most open agent stacks). Argues that the gap in open agent research is not orchestration but training scaffolding, since most open frameworks "focus on orchestration and evaluation rather than scalable agent training." +- **Why it matters for Justin's agents:** Three structural takeaways. First, **the 4B GUI agent at 74.1% WebVoyager is local-fit** — a 4B VLM quantized to Q4/Q5 runs on the M5 Max with substantial KV headroom and is the right size for an agent that needs to share memory with a browser, screen capture, and tool runtime. Second, **Orchard-SWE at 67.5% SWE-bench Verified is the first competitive open recipe** with the training pipeline released — a meaningful counterfactual to closed-stack pricing for internal coding-agent products. Third, the Orchard Env "sandbox lifecycle primitive" is the architecturally important piece: the same shape as [arxiv:2605.10913] Shepherd's typed event trace, and the convergence is now too strong to ignore — **agent runtimes need a typed environment substrate, not just orchestration glue**. +- **Action signal:** `try-now` `local-fit` — Orchard-GUI is a directly runnable VLM at M5-Max-friendly size with a competitive WebVoyager number. Relates-to [arxiv:2605.10913] Shepherd (runtime substrate convergence), [arxiv:2605.03042] ARIS (open agent framework with explicit Markdown skill convention), [arxiv:2605.06614] SkillOS (skill curation as agent state). + +### 3. PREPING: Building Agent Memory without Tasks +- **Source:** arXiv (cs.AI) / HF Daily Papers +- **Link:** https://arxiv.org/abs/2605.13880 +- **Authors / org:** Yumin Choi, Sangwoo Park, Minki Kang, Jinheon Baek, Sung Ju Hwang — **KAIST AI** +- **Focus area:** Agent memory +- **What it is:** Memory construction framework that runs **before** the agent encounters target tasks, using only **self-generated synthetic practice**. Three-component decomposition: **Proposer** generates synthetic tasks from a structured control state, **Solver** executes them, **Validator** filters which trajectories enter memory and provides feedback to the Proposer. The paper's core claim is mechanistic: naive synthetic-task self-play degenerates ("redundant, infeasible, ultimately uninformative") unless the proposer is explicitly controlled for feasibility, redundancy, and coverage, *and* memory updates are selective rather than write-everything. Headline numbers: results comparable to strong playbook-based memory methods built from offline or online experience, at **2.99× lower deploy cost on AppWorld** and **2.23× lower on BFCL v3**. +- **Why it matters for Justin's agents:** This is the most concrete operational answer yet to a structural problem in companion-app / personal-agent memory: **how do you bootstrap a useful memory before the user has done anything memorable?** The Proposer/Solver/Validator separation maps cleanly onto a "pre-warm" pattern Justin could implement: a curator-subagent generates plausible synthetic interactions against the user's domain (calendar, MCP tools, prior context), an executor-subagent runs them, a validator-subagent gates which trajectories actually enter the long-term memory. Crucially, this is the **inverse of yesterday's [arxiv:2605.12978] failure mode**: that paper showed background, same-model, after-every-interaction consolidation breaks memory; PREPING shows that **structured, role-separated, validator-gated synthetic consolidation builds memory**. The 2-3× cost reduction is the kind of headline that matters at unit-economics scale for an indie product. +- **Action signal:** `try-now` `compare` — directly implementable on top of any MCP-based stack. Relates-to and resolves the open question from yesterday's item 1 [arxiv:2605.12978]: the failure mode is *unstructured* consolidation, not consolidation itself. Cross-refs: [arxiv:2605.06614] SkillOS (similar separation: frozen executor + trainable curator; PREPING is the no-task-required variant), [arxiv:2605.05583] Belief Memory (orthogonal mechanism, alternative-conclusions retention). + +### 4. π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows +- **Source:** arXiv (cs.AI, May 14, 2026) / HF Daily Papers +- **Link:** https://arxiv.org/abs/2605.14678 +- **Authors / org:** Haoran Zhang, Luxin Xu, Zhilin Wang, and 11 collaborators (affiliations not on abstract) +- **Focus area:** Long-running / autonomous loops / Evaluation +- **What it is:** Benchmark designed to test whether agents can **anticipate user needs before they're explicitly stated**. 100 multi-turn tasks across 5 domain-specific user personas, evaluating jointly: (a) ability to identify hidden user intents before explicit requests, (b) inter-task dependencies and cross-session continuity, (c) sustained performance over extended interactions. The benchmark's explicit framing is "proactivity AND task completion" — and the headline result is that the two are **statistically distinct capabilities**. Three reported findings: proactivity remains a significant challenge for current agents; task completion and proactivity are decoupled (good at one ≠ good at the other); prior interactions improve agents' ability to resolve hidden intents in subsequent tasks. +- **Why it matters for Justin's agents:** For a companion-app product, **proactivity is the differentiator** — task completion is table stakes, but anticipating user need is what makes the product feel intelligent. The π-Bench finding that proactivity is a separate axis from completion has two operational implications: (a) **default LLM evals are blind to proactivity** — if Justin runs a Claude eval suite on his companion-app agent and it passes, the agent might still be a poor proactive assistant; the only way to know is to test proactivity directly with a π-Bench-style protocol; (b) **cross-session memory pays off** — the "prior interactions improve hidden-intent resolution" finding is the empirical evidence that long-term memory layers (PREPING, SkillOS, δ-mem) have user-visible value beyond eval gains. Pairs with [microsoft-research-2026-05-11-socialreasoning-bench] (calendar / marketplace) which made the same "completion ≠ value" point from the other end of the workflow. +- **Action signal:** `track` — concrete benchmark to revisit when Justin builds his next proactive-agent feature. Cross-refs: [microsoft-research-2026-05-11-socialreasoning-bench] (companion finding from utility/optimality angle), [arxiv:2605.12493] LongMemEval-V2 (the same "5 memory abilities including premise awareness" axis), [arxiv:2605.06078] BEACON / [arxiv:2605.06642] StraTA (the closest existing benchmark family). + +### 5. OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation +- **Source:** arXiv (cs.AI, May 14, 2026) / HF Daily Papers +- **Link:** https://arxiv.org/abs/2605.15177 +- **Authors / org:** Shang Zhou, Wenhao Chai, Kaiyuan Liu, Huanzhi Mao, Qiuyang Mang, Jingbo Shang (UCSD-style author cluster; Shang and Mao are UCSD CSE) +- **Focus area:** Reasoning & inference-time compute +- **What it is:** Replaces the brittle "point-wise LLM-as-judge picks the best of N reasoning candidates" with **Bradley-Terry pairwise aggregation**. A population of candidate reasoning traces is generated; an LLM judges *random pairs*; pairwise votes are aggregated into a global ranking via the Bradley-Terry model. Generation-by-generation, the top quartile is preserved, the middle 75% is mutated using natural-language critiques drawn from the comparison rationales, and the bottom 25% is discarded. Headline numbers: **Gemini 3.1 Pro Codeforces Elo +405 over 8 LLM-call rounds (~27 minutes)**. Transfers across model strengths without retuning. On HLE, gains concentrate in objectively verifiable domains and *reverse* in subjective ones (a caveat worth noting). **CF-73** released: 73 expert-rated Codeforces problems with 99% agreement against official verdicts. +- **Why it matters for Justin's agents:** Point-wise LLM-as-judge has been the workhorse pattern for self-consistency and tree-of-thought-style selection, but it's known noisy. Bradley-Terry aggregation is the **right mechanism** — pairwise judgment is a cognitively easier task for LLMs than absolute scoring, and the BT model is statistically well-behaved. For Justin's coding-agent loops, the operational signal is: **if you're running self-consistency or N-of-best sampling, switch the selector from point-wise to pairwise-aggregated and you're likely to see meaningful gains for ~zero added compute** (the comparison calls are short, not full re-generations). The +405 Codeforces Elo over 27 minutes is a concrete number that says this is a *production* technique, not a research curio. The HLE caveat (subjective tasks regress) is the right shape — pairwise comparison works when there's a ground truth or near-ground-truth signal. +- **Action signal:** `try-now` — minimal-friction swap in any best-of-N selection loop. Relates-to [arxiv:2605.10805] RACER (judge routing as constrained DRO; OpenDeepThink replaces the *judgment* primitive while RACER routes between judge types — orthogonal axes), [arxiv:2605.07112] Switchcraft (router pattern), [arxiv:2605.08083] AutoTTS (TTS controller synthesis — OpenDeepThink is a TTS strategy AutoTTS could discover). + +### 6. Self-Distilled Agentic Reinforcement Learning (SDAR) +- **Source:** arXiv (cs.AI, May 14, 2026) / HF Daily Papers +- **Link:** https://arxiv.org/abs/2605.15155 +- **Authors / org:** Zhengxi Lu and 10 co-authors (affiliations not on abstract) +- **Focus area:** Reasoning / Agent orchestration +- **What it is:** Adds a **gated self-distillation auxiliary objective** atop GRPO for multi-turn agent training. The gate maps detached token-level signals through a sigmoid to **amplify learning from teacher-endorsed tokens while moderating rejection signals**, which the authors argue solves the instability that breaks naive RL+distillation hybrids in multi-turn settings. Tested on Qwen2.5 and Qwen3 families. Headline gains over GRPO baselines: **ALFWorld +9.4%, Search-QA +7.0%, WebShop-Acc +10.2%**. +- **Why it matters for Justin's agents:** Two reads. First, **the ALFWorld 90+ club just got a 4th mechanism** — BEACON (milestone-segmented temporal reward, 92.9%), StraTA (top-down strategy abstraction, 93.1%), Constant-Context Skill Learning (in-weights procedural memory, 89.6% unseen), and now SDAR (gated self-distillation atop GRPO, +9.4% over baseline). Four orthogonal mechanisms hitting the same plateau is a strong signal that **ALFWorld is now method-saturated**, not capability-saturated — every credit-assignment improvement gets you to ~90% via a different path. Second, the **gated self-distillation pattern is broadly applicable to any multi-turn RL setup** — Justin would not likely train his own RL pipeline today, but if the gate primitive lands in trl/verifiers libraries (which it almost certainly will within a month), it becomes a one-line config flip for anyone fine-tuning Qwen3 backbones on agentic tasks. +- **Action signal:** `track` `compare` — fourth ALFWorld 90+ club member; reinforces that credit assignment is now a solved-with-multiple-mechanisms problem on this benchmark. Cross-refs: [arxiv:2605.06078] BEACON, [arxiv:2605.06642] StraTA, [arxiv:2605.05413] Constant-Context Skill Learning, [arxiv:2605.00425] AEM (response-level entropy modulation — the original baseline this family extends). + +### 7. CAST: Case-Based Calibration of Adaptive Reasoning and Execution for LLM Tool Use +- **Source:** arXiv (cs.AI, May 14, 2026) +- **Link:** https://arxiv.org/abs/2605.15041 +- **Authors / org:** Renning Pang, Tian Lan, Leyuan Liu, Piao Tong, Sheng Cao, Xiaosong Zhang +- **Focus area:** Tools & connectors +- **What it is:** Treats **historical execution trajectories as structured cases**, then extracts two distinct profiles from them: (a) **complexity profiles** to estimate optimal reasoning depth per task, and (b) **failure profiles** to map likely structural errors. These profiles feed fine-grained reward design and adaptive reasoning during RL; the model learns to internalize case-based strategies rather than copy trajectories verbatim. Headline numbers on **BFCLv2 + ToolBench**: **+5.85 pp execution accuracy** with **26% shorter reasoning length on average**. +- **Why it matters for Justin's agents:** A simultaneous accuracy gain *and* reasoning-length reduction is the rare result that improves both quality and unit economics. For Justin's MCP-tool-using agents, the pattern is directly transferable as a **training-free heuristic**: maintain a small library of past trajectories (success and failure), match incoming tool-call patterns against the library, and use the matched profiles to set reasoning budget for the call. This is the data-driven version of the "tool-use tax" finding [arxiv:2605.00136] — CAST measures *which* tool calls cost what, then deliberately routes around the expensive ones. The 26% reasoning-length reduction at higher accuracy is also a useful empirical refutation of the "longer chain = better answer" assumption that drives a lot of default reasoning configs in production stacks. +- **Action signal:** `try-now` — implementable as a runtime case-matcher without retraining. Cross-refs: [arxiv:2605.00136] tool-use tax (CAST is the positive-policy answer), [arxiv:2605.00737] hidden-state need estimators (orthogonal mechanism — CAST learns from completed traces; hidden-state estimators learn from in-call activations), [arxiv:2605.07990] linearly-readable tool calls (mechanistic counterpart to CAST's empirical case library). + +### 8. Asynchronous Continuous Batching in HF Transformers +- **Source:** Hugging Face Blog, May 14, 2026 +- **Link:** https://huggingface.co/blog/continuous_async +- **Authors / org:** Rémi Ouazan Reboul, Pedro Cuenca, Aritra Roy Gosthipaty + 17 HF contributors +- **Focus area:** Tools & connectors / Local & open models infrastructure +- **What it is:** New `ContinuousBatchingAsyncIOs` class in HF Transformers' continuous-batching path that decouples CPU batch preparation from GPU compute using **three CUDA streams** (host-to-device, compute, device-to-host) plus **CUDA events for ordering**, **double-buffering** to prevent race conditions, and **carry-over masks** to propagate output tokens from batch N into batch N+1's inputs. Profiling on a 8B model showed **24% of total runtime wasted on an idle GPU waiting for CPU batch prep**. Headline result: **300.6s → 234.5s for 8K tokens at batch size 32 on an 8B model — 22% generation-time speedup; GPU utilization 76.0% → 99.4%**. Zero new kernels. No model changes required. +- **Why it matters for Justin's agents:** This is infrastructure work, but the kind that compounds. For local inference on the M5 Max, the analogous bottleneck exists on the unified-memory pipeline (the M5 Max doesn't have a CPU/GPU PCIe boundary, but it does have CPU-driven tokenization / sampling / KV-cache management running serially with the GPU's compute kernels). The HF result is **directly portable to llama.cpp and mlx-lm** as a pattern — both projects have the same "CPU prepares the next batch while GPU runs the current one" idle-window problem, and both have open issues discussing equivalent fixes. For Justin's hosted inference, if the OpenAI-compatible inference layer he uses (Together / Fireworks / Groq / etc.) is HF-Transformers-derived (Together is the most likely), throughput will improve silently within weeks. The 22% number is also a useful sanity check on serving-cost spreadsheets — if a vendor is more than ~25% slower at the same hardware spec as a competitor, the gap is unlikely to be the model. +- **Action signal:** `try-now` — directly usable in any HF-Transformers-based serving stack today; instructive pattern for MLX / llama.cpp porting. + +## Local-model corner + +Today's most directly local-fit item is **Orchard-GUI (item 2)** — Microsoft Research's 4B vision-language computer-use agent at **74.1% WebVoyager success**. + +- **Model:** Orchard-GUI, **4B parameters**, vision-language backbone, trained inside the Orchard Env environment service. +- **Memory fit:** 4B VLMs at Q4/Q5 typically weigh in around **3-5 GB on disk and roughly 5-8 GB resident** including the vision tower and KV cache for moderate context. **Completely comfortable on the M5 Max + 64 GB**, with substantial headroom for: (a) a co-resident screen-capture buffer, (b) a browser process, (c) a tool runtime, (d) a separate companion LLM for chat. The whole computer-use stack fits in well under half the unified pool. +- **Realistic tasks:** + - **Where it's plausibly excellent:** browser-based task automation (the 74.1% WebVoyager number is *directly* on this workload), web form filling, scripted multi-step UI walkthroughs, screenshot-grounded QA. + - **Where it's likely worse than a generalist model:** anything off the WebVoyager-style distribution — desktop apps with unusual UI, native iOS/macOS apps, complex multi-window workflows. 4B VLMs in this class also typically underperform on dense-document reasoning vs. an 8B-class generalist; if Justin wants to read long PDFs *and* operate a browser, this is the wrong model. + - **Where it's untested:** the MCP integration story for VLM agents — Orchard ships its own env primitives, not MCP, and the bridging story isn't in the paper. Justin would need to wrap the Orchard agent in an MCP client to integrate with the rest of his stack. +- **How to run:** Wait for the weight release on Hugging Face (the paper announces release; check the Microsoft Research / wenlinyao or baolinpeng org on HF in the coming week). Once released: **MLX** is the right runtime — `mlx-vlm` supports the Qwen2-VL / Qwen2.5-VL family that's most likely the Orchard-GUI base, and the 4B size is the sweet spot where MLX is closest to llama.cpp throughput. If weights ship as HF Transformers, the new async-continuous-batching path (item 8) lands within weeks and gives Justin a free ~20% throughput uplift on any served inference of this model. +- **Speed estimate:** 4B VLMs at Q4 on M5 Max benchmark in the **~80-120 tok/s generation** range for moderate-context single-turn inference (published llama.cpp numbers for Qwen2.5-VL-7B Q4 are ~60-70 tok/s; the 4B variant should be ~30-50% faster). For a WebVoyager-style task (~15-30 steps, ~1-3 KB of generated output per step), expect **5-15 seconds per step** wall-clock, dominated by screenshot tokenization rather than text generation. + +**Operational note:** Granite Embedding Multilingual R2 (IBM, May 14) also landed today — **311M ModernBERT embedding model, Apache 2.0, 32K context, 65.2 MTEB Multilingual**. Not an agent model, but the right substrate to upgrade the embedding layer of any RAG / memory-retrieval stack on the M5 Max (it runs comfortably on CPU at 1,800+ docs/sec on an H100; the M5 Max with MLX is ~5-10× slower per absolute throughput but well in range for any companion-app retrieval workload). Skipped from the main items list because it's a model release without research novelty, but flagged here for the local stack. + +## Conflicts & open questions + +Cross-referenced against `_meta/claims-ledger.md`: + +- **CONFLICT (resolution): Harness sensitivity retrofits the Opus 4.7 cross-benchmark numbers.** + - WildClawBench (item 1): Opus 4.7 = **62.2% under OpenClaw**, and **harness switching alone moves a single model by up to 18 pp**. Source: https://arxiv.org/abs/2605.10912 + - Prior Opus 4.7 data points in the ledger: **Vals AI Finance 64.37%** [anthropic-news-2026-05-05-finance-agents], **AstaBench 58.0%** (5/6 digest), **BioMysteryBench ~77%** (Opus 4.6, predates 4.7 by one release) [anthropic-research-2026-04-29-biomysterybench]. + - **Resolution:** These were never measuring the same thing — different harness wrappers (Anthropic's finance harness for Vals AI, Asta-runner for AstaBench, OpenClaw for WildClawBench) sit between the model and the benchmark and the 18-pp harness variance fully accounts for the cross-benchmark spread. **Operational takeaway: when comparing model benchmarks, control for the harness explicitly; "Opus 4.7 scores X%" without a named harness is methodologically incomplete.** This also retroactively sharpens [arxiv:2605.04624] AuditRepairBench's ranking-instability findings — *harness* is a major leak channel above and beyond the evaluator reconfiguration AuditRepairBench identified. +- **RESOLUTION: Yesterday's "consolidation-decay" conflict (item 1, 5/14) gets a positive recipe.** + - Yesterday [arxiv:2605.12978]: continuous LLM-driven consolidation degrades memory below the no-memory baseline; recommended fix = retain raw episode traces and **gate consolidation explicitly**. + - Today PREPING (item 3): Proposer / Solver / Validator separation, with Validator-gated entry into memory, **at 2.99× / 2.23× lower cost vs. online-experience memory construction**. + - **Resolution:** PREPING is the positive instantiation of the fix yesterday's paper recommended. Together they form a consistent picture: **structured, role-separated, validator-gated consolidation builds memory; background, same-model, default-on consolidation breaks it**. The previously-flagged SkillOS [arxiv:2605.06614] conflict resolves the same way — SkillOS's curator/executor separation is the mechanism that immunizes it from yesterday's failure mode. Open empirical question still open: does SkillOS retain its gains when the curator and executor share the same backbone? +- **OPEN QUESTION: Are π-Bench (item 4) and SocialReasoning-Bench measuring the same axis?** + - π-Bench: "task completion and proactivity are distinct capabilities" — proactivity is a separate, harder challenge. Source: https://arxiv.org/abs/2605.14678 + - SocialReasoning-Bench [microsoft-research-2026-05-11-socialreasoning-bench]: "completion ≈ 1.0 but marketplace Outcome Optimality ≈ 0" — frontier models complete tasks while giving away virtually all surplus to adversarial counterparties. + - **Open question:** π-Bench's "proactivity" and SocialReasoning-Bench's "outcome optimality" are arguably the same latent variable — *does the agent act on behalf of the user's actual interests beyond literal task completion?* — viewed from different operational angles. No paper has tested whether the two correlate at the model level; the proactivity-completion gap π-Bench reports and the optimality-completion gap SocialReasoning-Bench reports should both be present in the same model populations. **This is the high-signal followup**: cross-benchmark correlation analysis at the model level would tell us whether "true user-interest alignment" is one capability or two. + +## Sources checked +- arXiv cs.AI / cs.CL / cs.LG / cs.MA (recent listings, past 24-48h): 15+ candidates evaluated, 7 retained +- Hugging Face Daily Papers (May 14-15, 2026): cross-checked with arXiv list; WildClawBench surfaced as today's top-trending paper +- Hugging Face Blog (May 14, 2026): 2 new posts — **continuous_async** retained as item 8; **Granite Embedding Multilingual R2** flagged in Local-model corner but not promoted to a main item (model release, not research novelty) +- Anthropic News: 2 new posts — **Claude for Small Business** (5/13) and **Gates Foundation $200M partnership** (5/14); both are product/policy without research substance, skipped per "no marketing fluff" rule +- Anthropic Research: 1 new post — **2028: Two scenarios for global AI leadership** (5/14, Policy) — not research; skipped +- DeepMind Discover Blog: 1 new post — **Reimagining the mouse pointer for the AI era** — product vision post about Gemini-in-Chrome contextual pointing, no benchmarks, skipped per "no marketing fluff" +- OpenAI News: HTTP 403 — **could not access, retry next run** +- Mistral News: no new posts since April 29 +- Meta AI Research: no new posts since April 8 +- Google Research Blog: no new posts since May 1 +- Allen AI Blog: no new posts since May 13 AIMIP (covered in 5/14 digest, climate domain) +- Microsoft Research Blog: no new posts since May 13 (mimalloc / GridSFM / MatterSim — none AI-agent relevant) +- NVIDIA Developer Blog: not surveyed today (skipped per recent-pattern non-AI-agent focus)