From 23fcf09d6bb2aaea158f7ac94831d4e8cb635d6d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 7 May 2026 10:12:56 +0000 Subject: [PATCH] digest: 2026-05-07 8 verified items covering context engineering (LongSeeker, Impossibility Triangle, When Context Hurts), agent orchestration (Uno-Orchestra), tools/safety (AgentTrust), autonomous loops (Design Conductor 2.0), RL training infra (vLLM V0->V1), and a frontier-model data point (Opus 4.7 / Vals AI). Two cross-ledger flags: Uno-Orchestra extends Web2BigTable (learned vs. hand-designed orchestrator); AgentTrust adjacent to Tool-Use Tax (both intervene at the tool-call protocol). https://claude.ai/code/session_01KR8SxiAQ6wJwc47SZYn8ZZ --- _meta/claims-ledger.md | 8 +++ _meta/processed.json | 8 +++ inbox/2026-05-07.md | 120 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 136 insertions(+) create mode 100644 inbox/2026-05-07.md diff --git a/_meta/claims-ledger.md b/_meta/claims-ledger.md index 62ec505..92ab115 100644 --- a/_meta/claims-ledger.md +++ b/_meta/claims-ledger.md @@ -9,3 +9,11 @@ - **2026-05-04** [hf-blog-2026-04-29-ibm-granite-4-1] CLAIM: Granite 4.1-8B "consistently matches or outperforms" Granite 4.0-H-Small (32B MoE / 9B active); 30B reports BFCL v3 73.7, RULER@128K 76.7, GSM8K 94.2, HumanEval 89.6 | source: https://huggingface.co/blog/ibm-granite/granite-4-1 | status: open - **2026-05-04** [hf-blog-2026-04-29-evaleval-eval-costs-bottleneck] CLAIM: agent benchmarks compress only 2–3.5× vs. 100–200× for static; single GAIA frontier-model run ~$2,829; HAL k=8 reliability rerun ~$320K; pass^k drops from 60% (k=1) to 25% (k=8) | source: https://huggingface.co/blog/evaleval/eval-costs-bottleneck | status: open - **2026-05-04** [anthropic-research-2026-04-29-biomysterybench] CLAIM: Opus 4.6 ~77% / Sonnet 4.6 ~66% on 76 human-solvable BioMysteryBench tasks; "Claude Mythos Preview" 30% on 23 human-difficult tasks (vs. Opus 4.6 ~24%); many hard-set wins are "brittle" (inconsistent across 5 attempts) | source: https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench | status: open ; first public reference to "Claude Mythos Preview" model handle +- **2026-05-07** [arxiv:2605.05191] CLAIM: LongSeeker (Qwen3-30B-A3B fine-tune) with five context-orchestration ops (Skip/Compress/Rollback/Snippet/Delete) achieves 61.5% BrowseComp / 62.5% BrowseComp-ZH, vs. Tongyi DeepResearch 43.2/46.7 and AgentFold 36.2/47.3; Compress operator proven expressively complete | source: https://arxiv.org/abs/2605.05191 | status: open +- **2026-05-07** [arxiv:2605.05066] CLAIM: No architecture can simultaneously achieve length-independent per-step compute, length-independent state, and length-proportional recall; models satisfying first two recall at most O(poly(d)/log V) KV pairs; verified across 52 architectures pre-March 2026 | source: https://arxiv.org/abs/2605.05066 | status: open ; foundational theoretical claim, expect citations rather than refutations +- **2026-05-07** [arxiv:2605.05007] CLAIM: Uno-Orchestra (learned joint policy over decomposition + worker selection + budget) hits 77.0% macro pass@1 across 13 benchmarks, +16 pp over strongest workflow baseline at ~10× lower per-query cost | source: https://arxiv.org/abs/2605.05007 | status: open ; relates-to [arxiv:2604.27221] (learned vs. hand-designed orchestrator, both report large gains) +- **2026-05-07** [arxiv:2605.04785] CLAIM: AgentTrust pre-execution tool-call gate achieves 95.0% verdict accuracy / 73.7% risk-level accuracy on 300-scenario internal benchmark; 96.7% / ~93% on 630-scenario adversarial benchmark; ms-latency; ships as MCP server (AGPL-3.0) | source: https://arxiv.org/abs/2605.04785 | status: open ; relates-to [arxiv:2605.00136] tool-use tax (both intervene at protocol layer; safety vs. performance angles) +- **2026-05-07** [arxiv:2605.04361] CLAIM: Same context artifacts produce up to 20× speedup or 46% degradation across 10 tasks × 7 conditions × 2,700+ runs; baseline (no-context) exploration predicts outcome with Pearson r = -0.82 (p < 0.001); two regimes — training-data-driven (disrupted by artifacts) vs. explicit-instruction-driven (not) | source: https://arxiv.org/abs/2605.04361 | status: open +- **2026-05-07** [arxiv:2605.05170] CLAIM: Design Conductor 2.0 multi-agent system (frontier models, April 2026) autonomously designed FPGA-mapped TurboQuant inference accelerator (240-cycle pipeline, 5,129 FP16/32 units, 5.7 mm² in TSMC 16FF) in 80 hours; claims 80× larger task scope than prior 12h RISC-V CPU result | source: https://arxiv.org/abs/2605.05170 | status: open ; vendor-self-reported, autonomy claim awaits independent reproduction +- **2026-05-07** [hf-blog-2026-05-06-vllm-correctness-before-corrections] CLAIM: When migrating PipelineRL from vLLM 0.8.5 → 0.18.1, logprob mismatches between inference and training masquerade as RL-objective divergence; required fixes (logprobs_mode=processed, disable prefix caching, disable async scheduling, fp32 lm_head, specific weight-update sequence) restore V0 parity on policy ratio, KL, entropy, reward, weight lag | source: https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections | status: open +- **2026-05-07** [anthropic-news-2026-05-05-finance-agents] CLAIM: Claude Opus 4.7 scores 64.37% on Vals AI Finance Agent benchmark | source: https://www.anthropic.com/news/finance-agents | status: open ; vendor-self-reported, first public Opus 4.7 number tracked in this ledger diff --git a/_meta/processed.json b/_meta/processed.json index e10596d..e83813a 100644 --- a/_meta/processed.json +++ b/_meta/processed.json @@ -1,4 +1,12 @@ [ + { "id": "arxiv:2605.05191", "url": "https://arxiv.org/abs/2605.05191", "title": "LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents", "date_covered": "2026-05-07" }, + { "id": "arxiv:2605.05066", "url": "https://arxiv.org/abs/2605.05066", "title": "The Impossibility Triangle of Long-Context Modeling", "date_covered": "2026-05-07" }, + { "id": "arxiv:2605.05007", "url": "https://arxiv.org/abs/2605.05007", "title": "Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation", "date_covered": "2026-05-07" }, + { "id": "arxiv:2605.04785", "url": "https://arxiv.org/abs/2605.04785", "title": "AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use", "date_covered": "2026-05-07" }, + { "id": "arxiv:2605.04361", "url": "https://arxiv.org/abs/2605.04361", "title": "When Context Hurts: The Crossover Effect of Knowledge Transfer on Multi-Agent Design Exploration", "date_covered": "2026-05-07" }, + { "id": "arxiv:2605.05170", "url": "https://arxiv.org/abs/2605.05170", "title": "Design Conductor 2.0: An agent builds a TurboQuant inference accelerator in 80 hours", "date_covered": "2026-05-07" }, + { "id": "hf-blog-2026-05-06-vllm-correctness-before-corrections", "url": "https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections", "title": "vLLM V0 to V1: Correctness Before Corrections in RL", "date_covered": "2026-05-07" }, + { "id": "anthropic-news-2026-05-05-finance-agents", "url": "https://www.anthropic.com/news/finance-agents", "title": "Agents for financial services", "date_covered": "2026-05-07" }, { "id": "arxiv:2605.00737", "url": "https://arxiv.org/abs/2605.00737", "title": "To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling", "date_covered": "2026-05-04" }, { "id": "arxiv:2604.27221", "url": "https://arxiv.org/abs/2604.27221", "title": "Web2BigTable: A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction", "date_covered": "2026-05-04" }, { "id": "arxiv:2605.00425", "url": "https://arxiv.org/abs/2605.00425", "title": "AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning", "date_covered": "2026-05-04" }, diff --git a/inbox/2026-05-07.md b/inbox/2026-05-07.md new file mode 100644 index 0000000..5cd704e --- /dev/null +++ b/inbox/2026-05-07.md @@ -0,0 +1,120 @@ +--- +date: 2026-05-07 +type: research-digest +tags: [ai-research, daily-digest] +sources_checked: [arxiv-cs.AI, hf-daily-papers, hf-blog, anthropic-news, anthropic-research, deepmind-blog, mistral-news, openai-news, research-google, ai-meta] +items_count: 8 +conflicts_flagged: 2 +--- + +# AI Research Digest — Thursday, May 07, 2026 + +## TL;DR +- **Context engineering hit a theoretical ceiling**: a 41-page proof (arXiv:2605.05066) shows no architecture can simultaneously have length-independent compute, length-independent state, and length-proportional recall — 52 architectures empirically lie strictly below the bound. +- **Long-horizon search agents got a concrete recipe**: LongSeeker (arXiv:2605.05191) introduces five atomic context operations (Skip, Compress, Rollback, Snippet, Delete) and posts **61.5% on BrowseComp / 62.5% on BrowseComp-ZH**, well past Tongyi DeepResearch and AgentFold. Backbone is Qwen3-30B-A3B — locally runnable. +- **Multi-agent orchestration is converging on learned routers**: Uno-Orchestra (arXiv:2605.05007) hits **77.0% macro pass@1** across 13 benchmarks at ~10× lower per-query cost than workflow baselines — a different lineage from last week's bi-level Web2BigTable but the same supervisor-delegation thesis. +- **Context-injection wisdom is wrong as a default**: "When Context Hurts" (arXiv:2605.04361) finds the same artifacts give up to **20× speedup or 46% degradation** depending on task; one no-context trial predicts which (Pearson r = -0.82). Cheap to add to any agent harness. +- **Autonomous engineering loops crossed 80 hours**: Design Conductor 2.0 (arXiv:2605.05170) drove an LLM-based agent fully autonomously through the design of an FPGA-mapped TurboQuant inference accelerator — 80h end-to-end, claiming 80× larger tasks than its 12h RISC-V CPU predecessor. + +## Items + +### 1. LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents +- **Source:** arXiv (cs.AI, May 6, 2026) +- **Link:** https://arxiv.org/abs/2605.05191 +- **Authors / org:** Yijun Lu, Rui Ye, Yuwen Du, Jiajun Wang, Songhua Liu, Siheng Chen +- **Focus area:** Context engineering / Long-running loops / Tools +- **What it is:** Proposes **Context-ReAct**, a paradigm with five atomic context operations — `Skip`, `Compress`, `Rollback`, `Snippet`, `Delete` — that the agent invokes to reshape its working window during a run. Trained on 10,000 synthesized trajectories on top of **Qwen3-30B-A3B**. Reports **61.5% BrowseComp / 62.5% BrowseComp-ZH**, vs. Tongyi DeepResearch (43.2% / 46.7%) and AgentFold (36.2% / 47.3%). Includes a proof that the `Compress` operator is expressively complete. +- **Why it matters for Justin's agents:** This is the closest thing yet to a drop-in **context-management op set** for any long-running MCP agent — explicit primitives map cleanly onto compaction, scratchpads, and the kind of self-pruning Claude Code already does informally. Backbone is local-fit (30B-A3B is the same MoE class Justin tracks). +- **Action signal:** `try-now` for the op-set design pattern — even without their weights, the five-op vocabulary is portable into a Claude Code-style harness today. + +### 2. The Impossibility Triangle of Long-Context Modeling +- **Source:** arXiv (cs.AI, May 6, 2026) +- **Link:** https://arxiv.org/abs/2605.05066 +- **Authors / org:** Yan Zhou +- **Focus area:** Context engineering / Frontier models (theory) +- **What it is:** Proves that **no architecture** can simultaneously satisfy (i) per-step compute independent of sequence length, (ii) state size independent of sequence length, and (iii) recall proportional to sequence length. Uses Data Processing Inequality + Fano's Inequality; bounds Efficiency-and-Compactness models to `O(poly(d) / log V)` recallable key-value pairs. Empirically surveys **52 architectures** (Transformers, SSMs, linear RNNs, hybrids) — every one lies strictly below the bound, and each achieves at most two of the three. +- **Why it matters for Justin's agents:** A clean north star for picking inference stacks. If a vendor markets "infinite context, constant cost, lossless recall," one of those three is a marketing claim. Frames why context-offloading-to-files (Claude's current pattern) is not a hack — it's the only way to escape the triangle by punting recall to external storage. +- **Action signal:** `track` — useful framing to cite when justifying retrieval-augmented or file-based memory designs over end-to-end long-context approaches. + +### 3. Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation +- **Source:** arXiv (cs.AI, May 6, 2026) +- **Link:** https://arxiv.org/abs/2605.05007 +- **Authors / org:** Cui, Xie, Yuan, Yang, Wang, Wu, Wu, Zhong, Yu, Guo, Zhang, Yu, Ren, Naseem +- **Focus area:** Agent orchestration +- **What it is:** Learns a unified policy that **jointly** decides decomposition depth, worker selection, and compute budget — trained via RL on real worker-interaction trajectories. Reports **77.0% macro pass@1** across 13 benchmarks (math, code, knowledge, long-context, tool-use), **+16 pp over the strongest workflow baseline**, at **~10× lower per-query cost**. +- **Why it matters for Justin's agents:** Direct prior art for the recurring "supervisor on Claude, sub-tasks on local Granite/Qwen" cost-routing question. The "joint optimization" framing is the right level of abstraction — current implementations (LangGraph, autogen) treat routing and decomposition as separable, which Uno-Orchestra shows is a mistake. +- **Action signal:** `compare` — extends the bi-level Web2BigTable pattern (ledger 2026-05-04, arxiv:2604.27221) by learning the orchestrator instead of hand-designing it; both report large gains, different mechanisms. + +### 4. AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use +- **Source:** arXiv (cs.AI, May 6, 2026) +- **Link:** https://arxiv.org/abs/2605.04785 +- **Authors / org:** Chenglin Yang +- **Focus area:** Tools & connectors / Agent orchestration +- **What it is:** Pre-execution gate for tool calls returning `allow / warn / block / review` verdicts. Combines shell deobfuscation, safer-alternative suggestion, multi-step attack-chain detection, and an LLM judge. Ships an **MCP server** (AGPL-3.0). Reports **95.0% verdict accuracy / 73.7% risk-level accuracy** on a 300-scenario internal benchmark and **96.7% / ~93%** on a 630-scenario adversarial benchmark, at millisecond latency. +- **Why it matters for Justin's agents:** Plugs straight into a Claude-Code-style MCP-tool stack as a sidecar gate. The shell-deobfuscation and chain-detection pieces are exactly what's missing when wiring a long-running agent against a real shell. AGPL-3.0 limits redistribution — fine for personal infra, awkward for shipping into a companion app. +- **Action signal:** `try-now` for personal infra; license-aware for product. + +### 5. When Context Hurts: The Crossover Effect of Knowledge Transfer on Multi-Agent Design +- **Source:** arXiv (cs.AI, May 5, 2026) +- **Link:** https://arxiv.org/abs/2605.04361 +- **Authors / org:** Saranyan Vigraham +- **Focus area:** Context engineering / Agent orchestration +- **What it is:** **2,700+ runs across 10 tasks × 7 context-injection conditions**. Same artifact gives **up to 20× improvement** on some tasks, **up to 46% degradation** on others. Irrelevant docs sometimes match relevant ones. The decisive predictor is **baseline (no-context) exploration performance**: Pearson **r = -0.82, p < 0.001** — high baseline → injection hurts; low baseline → injection helps. Two regimes identified: training-data-driven convergence (disrupted by artifacts) vs. explicit-instruction-driven (not disrupted). +- **Why it matters for Justin's agents:** Cheap, directly actionable: **run one no-context trial as a diagnostic** before committing to RAG/system-prompt heavy designs. Justin's companion-app domain (chat, recall) likely sits in the "training-data-driven" regime where artifact injection is risky. +- **Action signal:** `try-now` — the diagnostic protocol is one extra eval run; flag it as a checklist item for new agent features. + +### 6. Design Conductor 2.0: An Agent Builds a TurboQuant Inference Accelerator in 80 Hours +- **Source:** arXiv (cs.AI, May 6, 2026) +- **Link:** https://arxiv.org/abs/2605.05170 +- **Authors / org:** Verkor Team — Ravi Krishna, Suresh Krishna, David Chin +- **Focus area:** Long-running / autonomous loops / Coding agents +- **What it is:** Multi-agent system "powered by frontier models released in April 2026" autonomously designs **VerTQ**, an LLM inference accelerator (240-cycle pipeline, 5,129 FP16/32 units, FPGA at 125 MHz, 5.7 mm² in TSMC 16FF). Authors claim **80× larger task scope** than their prior 12-hour RISC-V CPU result. Fully autonomous **80-hour** run. +- **Why it matters for Justin's agents:** A new high-water mark for autonomous multi-day loops on a hard, verifiable domain (silicon design has a closed-loop verifier). The pattern — frontier model + scripted scaffolding + verification programs over many days — is the same archetype Justin tracks for durable agents. +- **Action signal:** `track` — domain-specific (HDL/EDA); the orchestration pattern is what's portable, not the tooling. + +### 7. vLLM V0 to V1: Correctness Before Corrections in RL +- **Source:** Hugging Face Blog — ServiceNow-AI (May 6, 2026) +- **Link:** https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections +- **Authors / org:** Rafael Pardinas, Ehsan Kamalloo et al. (ServiceNow-AI) +- **Focus area:** Reasoning & inference-time compute / Coding agents (training infra) +- **What it is:** Migration retrospective from vLLM 0.8.5 → 0.18.1 inside PipelineRL. Found that **logprob mismatches between inference and training** look like RL-objective divergence but are actually backend bugs. Required fixes: `logprobs_mode: processed_logprobs`, disable `enable_prefix_caching` during online RL, disable `async_scheduling`, **fp32 lm_head**, and a specific pause-update-resume sequence for inflight weight updates. After fixes, V1 matches V0 on policy ratio (centered at 1.0), KL, entropy, reward, and weight-lag curves. +- **Why it matters for Justin's agents:** If/when training a small local agent (e.g., GRPO on Qwen3-30B-A3B for tool use), this is the first list to check when curves go sideways. Saves the temptation to "fix" the loss with importance-sampling tricks that are masking infrastructure errors. +- **Action signal:** `track` — file under "RL training infra checklist." + +### 8. Agents for Financial Services + Opus 4.7 Vals AI Score +- **Source:** Anthropic News (May 5, 2026) +- **Link:** https://www.anthropic.com/news/finance-agents +- **Authors / org:** Anthropic +- **Focus area:** Frontier models / Agent orchestration (product release) +- **What it is:** Ten agent templates (pitch builder, earnings reviewer, valuation reviewer, KYC screener, etc.), shipped as Cowork/Code plugins and Managed Agent cookbooks; native Excel/PowerPoint/Word add-ins; new MCP connectors from Moody's, FactSet/S&P/MSCI, D&B, IBISWorld, etc. Single technical claim: **Claude Opus 4.7 scores 64.37% on Vals AI's Finance Agent benchmark**. +- **Why it matters for Justin's agents:** Two takeaways past the marketing. (1) The "skill + connector + subagent" packaging Anthropic is shipping is a template for how to bundle MCP-rich workflows for distribution. (2) Opus 4.7 has a published frontier number on a real-domain agent benchmark — first such number tracked for that model handle. +- **Action signal:** `track` — copy the bundling pattern, watch for Vals AI third-party benchmark replication. + +## Local-model corner + +LongSeeker is the only directly-runnable item today. + +- **Model:** LongSeeker (Qwen3-30B-A3B fine-tune, MoE, ~3B active per token) +- **Memory fit:** Base Qwen3-30B-A3B at Q4_K_M lands ~17–18 GB; Q5/Q6 ~21–24 GB. Comfortable on 64 GB unified memory with **35+ GB headroom** for KV cache at 128K context. Long-context inference at 256K+ would push KV close to model size — viable but watch tok/s. +- **Realistic tasks:** Long-horizon web research and multi-step browse loops where the agent needs to actively prune its own scratchpad. Not a general chat model — the fine-tune is search-trajectory-specific, so expect strong BrowseComp performance and ordinary general-purpose behavior. +- **How to run:** Wait for HF release; until then, the **op-set design** (Skip / Compress / Rollback / Snippet / Delete) is the portable artifact — wire it into a Claude Code or local-MCP harness as scratchpad operators the agent can invoke explicitly. +- **Speed estimate:** No M5 Max-specific number from the paper. Qwen3-30B-A3B base on M5 Max under MLX has been reported in the **~70–110 tok/s range at Q4** for short context; expect ~30–50 tok/s with 128K KV loaded. + +## Conflicts & open questions +Cross-referenced against `_meta/claims-ledger.md`: + +- **Uno-Orchestra (arxiv:2605.05007) vs. Web2BigTable (arxiv:2604.27221, ledger 2026-05-04)** — Both report large gains from supervisor + worker decomposition; Web2BigTable hand-designs the orchestrator (7.5× on WideSearch), Uno-Orchestra **learns** it (+16 pp, 10× cheaper across 13 benchmarks). No direct contradiction — orthogonal mechanisms — but worth running side-by-side on the same benchmark suite. Open question: does the learned router degrade as the worker pool changes (production agent stacks swap models often)? +- **AgentTrust (arxiv:2605.04785) vs. Tool-Use Tax (arxiv:2605.00136, ledger 2026-05-04)** — Both intervene at the tool-call protocol layer. Tool-Use Tax measures a **performance** cost of the protocol itself; AgentTrust adds a **safety** gate on top. Open question: stacking them — does the safety gate add to the tool-use tax, or do gating decisions overlap with the "should I have called this at all" question from arxiv:2605.00737? +- **LongSeeker (arxiv:2605.05191) and "When Context Hurts" (arxiv:2605.04361) and the Impossibility Triangle (arxiv:2605.05066)** — Three different angles on the same problem this week. Triangle says you can't have it all; When Context Hurts says even when you can have more, it sometimes makes things worse; LongSeeker offers an explicit op set for managing what you do keep. Together they argue against "just buy more context" as a design strategy. No flagged conflicts — these compose. + +## Sources checked +- **arXiv cs.AI (recent, May 5–7):** ~20 candidates surveyed, 6 selected. +- **Hugging Face Daily Papers:** today's top trending heavily skewed to video generation / multimodal — only OpenSearch-VL and the search/retrieval items overlapped with focus areas; OpenSearch-VL noted but not selected (multimodal-specific). +- **Hugging Face Blog:** 1 selected (vLLM correctness), 1 skipped (Open ASR Leaderboard — out of scope). +- **Anthropic News:** 2 posts (May 5, May 6) — 1 selected (Finance Agents), 1 skipped (SpaceX compute deal — no technical content). +- **Anthropic Research:** no new posts since April 30. +- **OpenAI News:** 403 (blocked); could not access — retry next run. +- **DeepMind Blog:** 503 (transient); could not access — retry next run. +- **Mistral News:** no new posts since April 29 (Mistral Medium 3.5). +- **Google Research Blog:** no posts in window (most recent May 1). +- **Meta AI Blog:** no posts in window (most recent April 8).