digest: 2026-05-15

[JustinPerea]

Summary

Daily AI research digest for 2026-05-15. 8 verified items, 3 conflict flags (2 resolved, 1 open).

Focus areas covered

• Evaluation (2 items): WildClawBench (real-world long-horizon, 19 frontier models inside real CLI harnesses; Opus 4.7 best at 62.2%; harness switching alone moves a single model up to 18 pp), π-Bench (proactive personal-assistant agents; completion ≠ proactivity)
• Agent memory (1 item): PREPING (KAIST AI) — task-free memory via Proposer/Solver/Validator separation, 2.99× / 2.23× lower deploy cost
• Agent orchestration / coding agents / computer-use (1 item): Microsoft Research Orchard — open agent training framework with three competitive recipes (Orchard-SWE 67.5% SWE-bench Verified, Orchard-GUI 4B VLM 74.1% WebVoyager, Orchard-Claw 73.9% pass@3)
• Reasoning / inference-time compute (2 items): OpenDeepThink (Bradley-Terry pairwise aggregation; Gemini 3.1 Pro Codeforces Elo +405 over ~27 min), SDAR (gated self-distillation atop GRPO; +9.4% / +7.0% / +10.2% on ALFWorld / Search-QA / WebShop)
• Tools & connectors (1 item): CAST (case-based calibration; +5.85 pp accuracy, -26% reasoning length on BFCLv2 / ToolBench)
• Local & open models infrastructure (1 item): HF Transformers async continuous batching (22% generation-time speedup, GPU util 76% → 99.4%, zero new kernels)

Conflicts flagged

1. RESOLUTION — Harness sensitivity retrofits Opus 4.7 cross-benchmark numbers. WildClawBench's 18-pp harness-variance finding fully accounts for the spread across Vals AI Finance 64.37%, AstaBench 58.0%, WildClawBench 62.2%. Operational takeaway: "Opus 4.7 scores X%" without a named harness is methodologically incomplete. Sharpens [arxiv:2605.04624] AuditRepairBench by identifying harness as a leak channel above evaluator reconfiguration.
2. RESOLUTION — Yesterday's consolidation-decay finding gets a positive recipe. PREPING (item 3) is the structured/role-separated/validator-gated consolidation pattern that [arxiv:2605.12978] recommended as the fix. Consistent picture now: structured consolidation builds memory; background same-model consolidation breaks it.
3. OPEN — Are π-Bench (item 4) and SocialReasoning-Bench measuring the same axis? Both report task-completion / actual-user-interest gaps. Cross-benchmark correlation at the model level is the high-signal followup.

Local-model corner

• Orchard-GUI 4B VLM at 74.1% WebVoyager is the local-fit pick — comfortably runs on M5 Max + 64 GB with substantial headroom; MLX-via-mlx-vlm once weights ship.
• Granite Embedding Multilingual R2 (IBM, 311M ModernBERT, Apache 2.0, 32K context, 65.2 MTEB Multilingual) also landed today; flagged as the RAG-substrate upgrade.

Sources status

• Surveyed: arXiv cs.AI / cs.CL / cs.LG / cs.MA, HF Daily Papers, HF Blog, Anthropic News, Anthropic Research, DeepMind Discover Blog, Mistral News, Google Research, Meta AI, Allen AI, Microsoft Research
• Could not access (retry next run): OpenAI News (HTTP 403)
• No new posts in window: Mistral, Meta AI, Google Research, Allen AI (since 5/13 climate post), Microsoft Research (since 5/13 mimalloc/GridSFM/MatterSim)
• Skipped per "no marketing fluff" rule: Anthropic Claude for Small Business (5/13), Anthropic Gates Foundation $200M partnership (5/14), Anthropic "2028: Two scenarios" (5/14 policy), DeepMind "Reimagining the mouse pointer" (5/14 product vision, no benchmarks)

Files

• inbox/2026-05-15.md — digest
• _meta/processed.json — 8 new IDs prepended
• _meta/claims-ledger.md — 8 new tracked claims appended

https://claude.ai/code/session

---

Generated by Claude Code