digest: 2026-06-14

_meta/claims-ledger.md

@@ -296,3 +296,10 @@
 - **2026-06-13** [arxiv:2606.13473] CLAIM: MaxProof (Jiacheng Chen et al., MiniMax) — **population-level test-time scaling** for competition-level mathematical proof on **MiniMax-M3**; single model trained on **three proof capabilities — generation + verification (engineered for low false-positive rate) + critique-conditioned proof repair** — via defense-in-depth generative-verifier RL; same model serves as **generator + verifier + refiner + ranker** at test time, searches over candidate-proof population, tournament-selects final proof; **IMO 2025 35/42 and USAMO 2026 36/42**, both above human gold-medal thresholds | source: https://arxiv.org/abs/2606.13473 | status: open ; **role-multiplexing primitive** (one model, four heads, switch at inference) avoids multi-model orchestration plumbing for inference-time scaling; reusable recipe for any tool whose output can be verified (compiled code, schema-valid JSON, tests-pass-or-not) — single model, four roles, tournament across N candidates; currently demonstrated only on math proof; watch for code/data adaptations
 - **2026-06-13** [arxiv:2606.12087] CLAIM: FORT-Searcher (Jia Deng, Yimeng Chen et al., Renmin / collaborators inferred from Ji-Rong Wen authorship) — **shortcut-resistant training-data synthesis** for deep search agents; formalizes gap between *structural difficulty* (graph nodes/hops) and *realized search difficulty*: prior synthesis methods inflate structural complexity but intended search collapses through cheaper identifying routes; names **four shortcut risks — evidence co-coverage, single-clue selectivity, exposed constants, prior-knowledge binding** — and **trajectory signatures (solving cost, answer hit time, prior-shortcut rate)** to diagnose realized effects; claims best-overall performance among comparable-size open-source search agents on deep-search benchmarks (numbers deferred to PDF) | source: https://arxiv.org/abs/2606.12087 | status: open ; **methodology lens** — the four shortcut categories are a reusable checklist for evaluating whether any agent benchmark actually tests what its authors think it tests, not just for search; general lesson **structural complexity ≠ realized task difficulty** applies to any synthetic agent-training data
 - **2026-06-13** [deepmind-blog-2026-06-11-multi-agent-safety-funding] CLAIM: Google DeepMind + Schmidt Sciences + Cooperative AI Foundation + ARIA + Google.org — **up-to-$10M multi-agent AI safety research funding call** across four areas: (1) sandboxes/testbeds for multi-agent systems, (2) agent network science, (3) infrastructure security for inter-agent protocols, (4) oversight systems for deployed agent populations; deadline **August 8, 2026**, awards autumn 2026; framing: "Soon, millions of AI agents — built by different organizations — will interact across digital environments, communicating, negotiating and transacting with one another" | source: https://deepmind.google/blog/investing-in-multi-agent-ai-safety-research/ | status: open ; funding-flow signal more than research signal; **infrastructure for inter-agent protocols** now explicitly funded alongside MCP momentum — broader than tool-use, includes negotiation/transaction, connects to agent-payments / agentic-commerce thread; credible non-dilutive funding path for indie work in oversight, observability, or multi-agent testbeds
+- **2026-06-14** [nvidia-blog-2026-06-12-aa-agentperf] CLAIM: NVIDIA + Artificial Analysis introduce **AA-AgentPerf** — "industry's first multi-vendor open benchmark profiling trajectories representative of real-world AI agent coding tasks"; measures **concurrent agents per accelerator and per megawatt under model-specific SLO tiers** rather than single-turn token throughput; inputs are **prerecorded agentic coding trajectories with interleaved reasoning + tool use across 12+ programming languages, sequence lengths 5K–131K tokens (~27K avg)**, with non-deterministic sequencing and tool-call latencies; model under test **DeepSeek-V4-Pro**; **GB300 NVL72 = 61.4K concurrent agents/MW vs H200 = 2.6K under SLO #1 (~20×)** | source: https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/ | status: open ; methodology is the durable artifact (the H200/GB300 number is product framing); **agents-per-MW-under-SLO with realistic trajectory shapes** is the inference economic unit agent serving has been missing; complementary not competing with WeaveBench [arxiv:2606.09426] (PassRate 41.2%) — both belong in the eval stack measuring different axes (capability vs throughput); open question whether AA-AgentPerf trajectory ranking transfers to WeaveBench-style task-completion ranking, or high-throughput models lose disproportionately on long-horizon hybrid tasks
+- **2026-06-14** [allenai-blog-2026-06-12-olmo-eval] CLAIM: Allen AI releases **olmo-eval**, OLMES successor — evaluation workbench for the model development loop with **modular task / suite / harness architecture** decoupling benchmark logic from runtime policy (same benchmark runs under lightweight direct eval *or* containerized sandbox for tool-using / code-execution / agentic); **pairwise checkpoint comparison view with standard error and minimum detectable effect (MDE)**; multi-turn tool-using evals integrated from day one; normalized experiment schema; sandbox + capability-routing layer with async planning | source: https://huggingface.co/blog/allenai/olmo-eval | status: open ; **task/suite/harness separation** is the right abstraction (cheap-indicative score under lightweight, expensive-definitive score under full agent harness, same benchmark); **pairwise + MDE** is the right report shape for "is checkpoint B actually better than checkpoint A" (separates real gain from noise); reusable workbench for any fine-tuning loop even without OLMo
+- **2026-06-14** [arxiv:2606.13594] CLAIM: "See What I See, Know What I Think" (Siyi Chen et al., NVIDIA + academic) — **dense latent communication across heterogeneous agents via KV-cache exchange instead of text tokens**; lightweight cache transformation + two-phase training (reconstruction → generation) aligns hidden states across **Qwen3-4B / 8B / 14B**; demonstrated **in all six directional pairings**; reports **matching or surpassing text-based communication in context-aware scenarios at ~2–3× lower computational cost**; effective on in-domain and out-of-domain benchmarks; beats prior heterogeneous-communication baselines especially in context-unaware transfer where text approaches fail | source: https://arxiv.org/abs/2606.13594 | status: open ; **structurally novel multi-agent primitive** — replaces tokenize/detokenize/context-rebuild on every cross-agent handoff; direct read-through to **local heterogeneous ensembles on M5 Max** (small fast router + large slower executor, both Qwen-family or both MLX-supported, sharing memory pool); MCP-style protocols may eventually grow a "cache-handoff" verb if cross-family generalization reproduces; open empirical question: does this work across model *families* (not just Qwen3 sizes)
+- **2026-06-14** [arxiv:2606.13598] CLAIM: OrchRM (King Yeung Tsang, Zihao Zhao, Vishal Venkataramani et al., 5 co-authors) — **self-supervised reward modeling for the orchestrator role in multi-agent LLM systems**; trains orchestrator without expensive sub-agent rollouts and without human annotations by constructing Bradley-Terry training pairs **directly from execution artifacts at the orchestration-decision level** (RM scores orchestration choices, not terminal outcomes); **~10× training-token efficiency vs sub-agent-rollout baselines**; **+8% accuracy at multi-agent test-time scaling** across math reasoning, web QA, multi-hop QA | source: https://arxiv.org/abs/2606.13598 | status: open ; composes naturally with **RAH [arxiv:2606.13643]** (parent-script-spawns-sub-harnesses pattern) and **HarnessBridge [arxiv:2606.12882]** (learnable observation/action projection); together: RAH = architecture, HarnessBridge = harness layer, OrchRM = training signal for the parent's dispatch policy; **10× training-token efficiency** is the bigger headline for indie devs (orchestrator improvements no longer require full agent-system rollouts)
+- **2026-06-14** [arxiv:2606.13608] CLAIM: AgentBeats — **agentified agent assessment (AAA)**: judge agents evaluate subject agents over **standardized A2A (task delivery) + MCP (tool access) protocols** rather than fixed scaffolded benchmarks; validated by 5-month open competition with **298 judge agents and 467 subject agents across 12 categories** (includes coding-agent case study); framing problem — fixed benchmarks suffer "test-production mismatch" + prevent fair comparison across diverse agent designs | source: https://arxiv.org/abs/2606.13608 | status: open ; **MCP-as-evaluation-substrate** signal: if eval frameworks normalize on MCP for the tool surface, entrenches MCP as the agent interoperability layer (favorable for any MCP-tool product); **judge-agent design** is portable to internal QA — instead of hand-graded eval rubrics for an MCP tool, write a judge agent that interacts with the subject agent through the tool; competition scale (765 agents) is a real reproducibility claim
+- **2026-06-14** [arxiv:2606.13106] CLAIM: SWITCH — **boundary-tokenized switchable latent reasoning compatible with on-policy RL**; explicit `<swi>` / `</swi>` boundary tokens mark entry/exit of latent-reasoning mode, making **policy ratio well-defined at decision points** (unlocks GRPO-style on-policy RL training for latent CoT, which prior hidden-state-recurrence approaches couldn't access); adds visible-to-latent curriculum + Switch-GRPO objective; mechanistic probing shows boundary tokens implement **learned switching (not stylistic markers)**, latent steps perform problem-specific computation, and computation concentrates at a **single hidden-state transition upon entry**; outperforms prior hidden-state-recurrence latent-reasoning at similar scale (numbers in PDF) | source: https://arxiv.org/abs/2606.13106 | status: open ; cleanest published path to on-policy RL for latent reasoning; practical implication for agent products: **reasoning compute that doesn't blow up output token budgets** (model thinks in hidden state, only emits when ready) — directly cost-relevant for API-tier MCP-tool products if a frontier provider ships a SWITCH-style model
+- **2026-06-14** [arxiv:2606.13669] CLAIM: Agents-K1 — **agent-native scientific knowledge orchestration**; three components: (a) **multimodal parser** capturing entities/evidence/citations/relationships across full papers (not just abstracts), (b) **4B information-extraction backbone trained with GRPO + rule-based rewards**, (c) **GraphAnything CLI** providing agents with web search + multimodal graph retrieval + cross-document traversal; releases **Scholar-KG** dataset (1M-paper subset of full 2.46M papers across 6 subjects); claims superior performance on scientific IE, KG construction, and multi-hop scientific reasoning (numbers in PDF) | source: https://arxiv.org/abs/2606.13669 | status: open ; design-pattern lesson: ship a CLI + curated KG + small (4B) domain-specific IE model rather than dump raw documents into frontier-model context; **4B IE backbone is local-runnable on M5 Max** at FP16 (~8 GB) with headroom for KG + agent model alongside; pattern reusable for any domain Justin would otherwise raw-document into context

_meta/processed.json

@@ -1,4 +1,46 @@
 [
+  {
+    "id": "nvidia-blog-2026-06-12-aa-agentperf",
+    "url": "https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/",
+    "title": "NVIDIA + Artificial Analysis — AA-AgentPerf: Industry's First Multi-Vendor Open Benchmark Profiling Real-World AI Agent Coding Trajectories; Measures Concurrent Agents Per Accelerator + Per Megawatt Under Model-Specific SLO Tiers; Prerecorded Trajectories Across 12+ Languages, 5K-131K Tokens (~27K Avg) with Interleaved Reasoning + Tool Use; DeepSeek-V4-Pro on GB300 NVL72 = 61.4K Agents/MW vs H200 = 2.6K (~20×)",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "allenai-blog-2026-06-12-olmo-eval",
+    "url": "https://huggingface.co/blog/allenai/olmo-eval",
+    "title": "Allen AI olmo-eval — OLMES Successor Evaluation Workbench for the Model Development Loop; Modular Task / Suite / Harness Architecture Decoupling Benchmark Logic from Runtime Policy; Pairwise Checkpoint Comparison View with Standard Error + Minimum Detectable Effect; Multi-Turn Tool-Using Evals Integrated from Day One; Containerized Sandbox + Capability-Routing Layer; Normalized Experiment Schema",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "arxiv:2606.13594",
+    "url": "https://arxiv.org/abs/2606.13594",
+    "title": "See What I See, Know What I Think — Dense Latent Communication Across Heterogeneous Agents via KV-Cache Exchange Instead of Text Tokens; Lightweight Cache Transformation + Two-Phase Training (Reconstruction → Generation) Aligns Hidden States Across Qwen3-4B/8B/14B in All Six Directional Pairings; Matches/Surpasses Text Communication at ~2-3× Lower Compute; Effective In-Domain + Out-of-Domain; Beats Heterogeneous-Comm Baselines Especially in Context-Unaware Transfer",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "arxiv:2606.13598",
+    "url": "https://arxiv.org/abs/2606.13598",
+    "title": "OrchRM — Self-Supervised Reward Modeling for the Orchestrator Role in Multi-Agent LLM Systems; Trains Orchestrator Without Sub-Agent Rollouts + Without Human Annotation via Bradley-Terry Pairs Directly at Orchestration-Decision Level Over Execution Artifacts; ~10× Training-Token Efficiency vs Sub-Agent-Rollout Baselines; +8% Accuracy at Multi-Agent Test-Time Scaling Across Math/Web-QA/Multi-Hop-QA",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "arxiv:2606.13608",
+    "url": "https://arxiv.org/abs/2606.13608",
+    "title": "AgentBeats — Agentified Agent Assessment (AAA) Where Judge Agents Evaluate Subject Agents Over Standardized A2A (Task Delivery) + MCP (Tool Access) Protocols Rather than Fixed Scaffolded Benchmarks; 5-Month Open Competition with 298 Judge Agents + 467 Subject Agents Across 12 Categories Including Coding-Agent Case Study; Frames Fixed Benchmarks as Suffering 'Test-Production Mismatch' Preventing Fair Cross-Design Comparison",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "arxiv:2606.13106",
+    "url": "https://arxiv.org/abs/2606.13106",
+    "title": "SWITCH — Boundary-Tokenized Switchable Latent Reasoning Compatible with On-Policy RL; Explicit <swi></swi> Boundary Tokens Make Policy Ratio Well-Defined at Decision Points (Unlocks GRPO-Style On-Policy RL for Latent CoT); Visible-to-Latent Curriculum + Switch-GRPO Objective; Mechanistic Probing Shows Boundary Tokens Implement Learned Switching (Not Stylistic Markers) + Latent Steps Perform Problem-Specific Computation Concentrated at Single Hidden-State Transition Upon Entry",
+    "date_covered": "2026-06-14"
+  },
+  {
+    "id": "arxiv:2606.13669",
+    "url": "https://arxiv.org/abs/2606.13669",
+    "title": "Agents-K1 — Agent-Native Scientific Knowledge Orchestration; Multimodal Parser (Entities + Evidence + Citations + Relationships Across Full Papers) + 4B IE Backbone Trained with GRPO + Rule-Based Rewards + GraphAnything CLI (Web Search + Multimodal Graph Retrieval + Cross-Document Traversal); Releases Scholar-KG Dataset (1M-Paper Subset of Full 2.46M Across 6 Subjects); Claims Superior Performance on Scientific IE + KG Construction + Multi-Hop Scientific Reasoning",
+    "date_covered": "2026-06-14"
+  },
   {
     "id": "arxiv:2606.13643",
     "url": "https://arxiv.org/abs/2606.13643",