EPIC (0.7.6 release gate): output-based model-behavior detection — profile-driven adaptation, no name gates

[hartsock]

Summary

Replace model-name gates with output-based behavioral detection +
adaptation: newt observes what a model actually emits (reasoning channel,
tool-call channel, budget signatures) and drives its existing recovery
machinery (tool_recovery.rs, suspicious-empty retry, tenacity) from one
learned BehaviorProfile — continuously, including across hot model swaps
behind a router. Design: docs/decisions/output_based_model_behavior_detection.md
(ADR PR to follow; revised under a 33-finding adversarial design review).
This is approach A; #1492 (psyche/cognition) is approach B.

Goal gate (A + B together, 0.7.6 methodology: tb-30 confined, honesty-classified, digest-pinned)

• Lift nemotron-3-nano_30b: from quarantined (16/30 real attempts) → scoreable → scored.
• Lift gpt-4.1-mini: above its 3.3% record.
• No harm to ornith-1.0-35b-q8: holds ≥ 36.7%, plus an A/A control
  (adaptation force-disabled vs enabled) — rung 1's signature is known to occur
  on the champion, so the A/A run is what proves engagement ≠ harm.

Honesty note: both lift targets probed clean on every axis the detector
sees single-turn, so A's lift contribution is a hypothesis until W0 traces
attribute the losses — B may carry the lift. No-harm is the hard constraint
regardless.

Evidence (fixtures captured; limits stated)

• 0.7.6 quarantines: nano 16/30, canonical 20/30, granite 22/30 real attempts —
  non-engagement, not transport noise (Benchmark validity: disambiguate chat-template / tool-call-parse artifacts from genuine model weakness #1500).
• 2026-07-31 raw-output probes (six models, request params recorded): all six
  emit clean structured tool_calls on trivial single-turn prompts; reasoning
  arrives server-split in reasoning_content (non-streaming bodies; streaming
  shape unverified — W1 obligation); qwen3.6 + ornith return empty content
  with budget spent in reasoning at the probe's 350-token cap.
• emits_leading_reasoning name list (reasoning.rs) + the model-card override
  = two label-keyed sources of truth; the one shape the gate really protects is
  the lone-leading-closer (reasoning</think>answer), which needs an
  explicit start-of-content detection rule (mid-content tags are quotable).
• newt ALREADY ships content recovery: agentic/tool_recovery.rs (engaged only
  when the native array is empty; weak-model findings) — this epic widens
  it; it does not duplicate it.

Workstreams

• W0 — Raw-output observability. Trace/contract carries
  reasoning_overflow, recovered_tool_call{dialect}, no_parseable_tool_call
  • raw-output capture, folding into the observability-contract emission work
    (with model_digest). Grounds Benchmark validity: disambiguate chat-template / tool-call-parse artifacts from genuine model weakness #1500's artifact-vs-weakness split AND
    attributes the nano losses (prerequisite for believing A's lift).
• W1 — Fixture corpus. Real captures only: probe bodies (+ params),
  streaming SSE fixtures for the six probed models, weak-model dialect traces,
  0.7.6 quarantine traces (post-W0), a vLLM/template-mismatch inline-<think>
  capture, swap-mid-session fixture, and adversarial fixtures (per-dialect
  echo attacks, unknown-name, schema-mismatch, MCP-namespaced — each proving
  fail-closed). A rung with no real firing fixture does not ship.
• W2 — BehaviorProfile. Per (session, backend, request-slot);
  positive-evidence axes incl. LeadingCloser; absence = decay, never
  contradiction; N-consecutive hysteresis; ambiguity ratchets to default.
  Cards + (endpoint, digest) cache = warm-start priors that may only pre-set
  read-side expectations and never arm recovery/filter modes. New leaf module
  beside tool_recovery.rs (e.g. agentic/behavior.rs); mod.rs wires only.
• W3 — Profile-driven ladder (widen, don't duplicate). Rung 1 =
  suspicious-empty retry widened: full budget-signature trigger
  (finish=length + empty + reasoning + no calls), one bounded auto-continue
  per turn, counted against round limits, operator-surfaced; finish=stop
  empty+reasoning is a separate non-mutating case. Rung 2 = tool_recovery.rs
  → data-driven registry (operator-scope config only; workspace may narrow,
  never extend) + profile guard on Structured-proven backends. Rung 3 =
  tenacity, unchanged. Single nudge ownership: rungs emit signals the
  existing tenacity accounting consumes.
• W4 — Kill the name gate, in order. Streaming fixtures first (W1);
  filter defaults OFF, arms only via the multi-turn ratchet; bounded first-turn
  hold-back for the lone-closer scan; delete the list; card field becomes
  prior-only; fix stale ornith cards. Scoped law: no recognition/recovery
  behavior keyed on model name (operator disposition + evidence-overridable
  priors exempt).
• W5 — Security boundary. Echo rule (verbatim-substring rejection
  against untrusted transcript regions), declared-set rule (no __ exemption
  for recovered calls; schema-validate at recovery time; headless MCP recovery
  fails closed), recovery provenance rendered on the permission prompt + TUI
  line.
• W6 — A/B verification. The goal gate above + the ornith A/A control,
  run per the release ceremony (digest-pinned record in docs/releases/).

Acceptance

• No recognition or recovery behavior keyed on model name (grep-provable in
  the parse/dialect/reasoning paths; operator disposition + priors exempt).
• Every rung: a real firing fixture, a must-NOT-fire fixture (incl. ornith's
  healthy finish=stop reasoning-only bodies), and its adversarial fixture —
  all green in the fully-mocked tier.
• Ornith A/A: adaptation-enabled run ≥ adaptation-disabled run within noise.
• Goal-gate bench table committed to the release record.

Refs #1500, #1492, #384 (model-card retirement — subsumed: cards become
evidence-overridable priors). Related: the external-evaluator contract work
(model_digest, outcome taxonomy).

[hartsock]

Release re-scope (operator, 2026-07-31): the v0.7.6 tag is HELD — 0.7.6 ships with this epic. The ADR is ACCEPTED and merged (#1507). Release prep already on main (version bump, changelog cut, scoreboard bank, staging build green) stands; the changelog date and release record will be refreshed when this epic lands and the tag ships.

[hartsock]

Scope expansion (operator, 2026-07-31): the harness must support multiple backends running simultaneously, detecting the model behind each and compensating — and the hosted-model survey grows to OpenAI's newer gpt-5.6 family (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna — confirmed live on the account) plus grok-4.5 (model ID confirmed; x.ai access pending team credits).

Multi-backend-simultaneous is the BehaviorProfile's home turf by construction — profiles key per (session, backend, request-slot) (W2), so each concurrent backend carries its own observed behavior; nothing new is needed beyond honoring that keying in the ladder.

New probe evidence (fixtures captured, both directions):

• The gpt-5.6 family rejects function tools on /v1/chat/completions while reasoning is active — HTTP 400: "use /v1/responses or set reasoning_effort to 'none'". This is a new behavior class for the corpus: a transport-level tools×reasoning constraint, structurally detectable (a typed 400, not a parse failure — maps to model_error in the contract taxonomy, and is exactly the kind of thing the profile should learn and compensate for).
• Compensation verified: with reasoning_effort: "none", all three emit clean structured tool_calls (finish=tool_calls). So there are two routes: (a) chat-completions + effort-none — works today, sacrifices reasoning; (b) the Responses API — which is what Nemotron qualification framework (bench/psyche-ab-matrix) — DRAFT until live plan runs #1492 (approach B) is building. The two approaches of this epic's goal gate literally require each other for these models.
• Third reasoning-channel shape — Hidden: no reasoning_content, no <think>; reasoning is visible only as usage.completion_tokens_details.reasoning_tokens (luna spent 33 on a trivial prompt; sol/terra 0). W2 refinement: for this family the budget signature has a cleaner observable — finish=length with reasoning_tokens ≈ budget.

W1 corpus additions: 3× tools-with-reasoning 400 fixtures, 3× effort-none success fixtures, 3× reason-probe bodies. Bench-roster additions (gpt-5.6 trio now; grok-4.5 when credits land) fold into W6's goal-gate runs.