An open benchmark for AI agent governance. Mapped to NIST AI RMF. Vendor-neutral.
Live scorecard · All 48 scenarios · Methodology · Architecture-is-governance · agenticcontrolplane.com
Existing benchmarks (HarmBench, InjecAgent, AgentDAM, AgentLeak) test the model — does the LLM refuse harmful prompts, resist injection, protect PII. AgentGovBench tests the governance layer around the model — the part responsible for who can call which tool, whose identity rides along with each call, how rate limits cascade across delegated subagents, and what the audit record contains after the fact.
What other benchmarks test What AgentGovBench tests
───────────────────────── ───────────────────────────
The model's behavior The system around the model
(refuses bad prompts?) (enforces the policy?)
(attributes the call?)
(logs enough to reconstruct?)
Eight categories, each mapped to one or more NIST AI RMF 1.0 controls:
| # | Category | What breaks if this fails | NIST |
|---|---|---|---|
| 1 | Identity propagation | End user's identity doesn't reach the tool; audit attributes actions to the agent, not the human | MAP-2.1, MEASURE-2.6, GOVERN-1.4 |
| 2 | Per-user policy enforcement | User X's subagent performs actions X was forbidden from | GOVERN-1.2 |
| 3 | Delegation provenance | Cannot trace a tool call back to the originating user through the delegation chain | MEASURE-2.3 |
| 4 | Scope inheritance | Child agent inherits parent's broader scope instead of being narrowed to its task | MAP-4.1, MEASURE-2.7 |
| 5 | Rate-limit cascade | User bypasses a rate limit by spawning N subagents | MANAGE-2.1 |
| 6 | Audit completeness | Actions happen without logs, or logs lack detail for forensic reconstruction | MEASURE-2.3 |
| 7 | Fail-mode discipline | Gateway failure → system defaults to fail-open when policy says fail-closed (or vice versa) | GOVERN-1.1 |
| 8 | Cross-tenant isolation | Tenant A's agent observes or affects tenant B's data | GOVERN-1.2 |
Deeper rationale and threat model: METHODOLOGY.md. Full control mapping: NIST_MAPPING.md. All 48 scenarios with expected outcomes: scenarios/.
Works on a fresh clone with no credentials. Shows what a framework scores when governance is not in place — the scorecard floor.
git clone https://github.com/agentic-control-plane/agentgovbench
cd agentgovbench
python -m venv .venv && source .venv/bin/activate
pip install -e .
agentgovbench run --runner vanillaExpected: 13/48 (full vanilla scorecard →). Shows the harness, scorer, and scenario library are working.
Hits a live ACP deployment using only an API key. No Firebase Admin SDK, no service-account JSON. You'll need a gsk_ API key minted on the target ACP deployment with bench.impersonate and admin.audit.read scopes.
Mint the key with the right scopes. The ACP dashboard issues empty-scope keys by default. Use the "AgentGovBench testing (24h, impersonation)" preset on the API Keys page — it pre-fills the right scopes and hard-caps expiry to 24h. If you mint a key without the preset, leave the scopes field blank or pass
*; an explicitly-narrow key without the bench scopes will silently fail audit-related scenarios with messages like0 matching audit entries.
pip install -e '.[acp]' # adds firebase-admin; optional for this runner, required for --runner acp
export ACP_API_KEY=gsk_your-tenant-slug_...
export ACP_BASE_URL=https://api.agenticcontrolplane.com # or your deployment
export ACP_TENANT_SLUG=your-tenant-slug
agentgovbench run --runner acp_api --out results/acp-api.jsonExpected: 46/48 against api.agenticcontrolplane.com, with 5 declinations documented in the runner manifest (results/acp_api-v0.1.0-live.json) — three of those still pass their checkable criteria; the two that don't are the cross-tenant scenarios, which require multi-tenant deployment mode. Different number? Either you're on an older ACP version, your tenant has custom policy that changes outcomes, or you've found a governance gap we haven't seen. File an issue.
Scoring much lower than expected (e.g. 18/48)? The most common cause is a missing scope on your API key. Symptoms:
audit_completeness 0/6, lots oftool_allowed: some calls were denied, and a singlerunner_errors_empty: ['audit GET 403: api key lacks admin.audit.read scope']. Mint a new key with the scopes above and rerun.
agentgovbench run --runner crewai_native # CrewAI without governance — baseline
agentgovbench run --runner crewai_acp # CrewAI + ACP @governed decorator
agentgovbench run --runner langgraph_native
agentgovbench run --runner langgraph_acp
agentgovbench run --runner claude_code_acp # via hook protocol
agentgovbench run --runner codex_acp
agentgovbench run --runner openai_agents_acp # via base_url proxy
agentgovbench run --runner anthropic_agent_sdk_acp # via governHandlers
agentgovbench run --runner cursor_acp # via MCP server
# Limit to one category for quick iteration
agentgovbench run --runner acp_api --category identity_propagationEach framework runner requires the respective SDK. Install with pip install -e '.[crewai]' / .[langchain] / etc.
We ran every runner against the same backend and published every scorecard. The nine-point spread tells the story:
| Integration pattern | Frameworks | Score |
|---|---|---|
| Decorator at orchestration boundary | Anthropic Agent SDK (governHandlers) |
46 / 48 |
| Proxy | OpenAI Agents SDK (base_url swap) |
45 / 48 |
| Hook | Claude Code · Codex CLI | 43 / 48 each |
| Decorator below orchestration | CrewAI · LangGraph (@governed) |
40 / 48 each |
| MCP | Cursor | 37 / 48 |
Same gateway. Same scenarios. Same scorer. The spread is architectural, not product-quality. Full walkthrough →
- Deterministic — no LLM in the hot path. Scenarios fully describe the agent action sequence; governance is tested on what it does with those actions. Reproducible byte-for-byte across runs.
- Framework-agnostic — scenarios don't assume CrewAI, LangGraph, Claude, etc. They describe actions and expected governance outcomes.
- Pluggable — any governance product implements the
BaseRunnerinterface. No ACP assumptions in the scenarios. - Versioned — each scenario carries a version. Old results remain comparable; new scenarios extend the set without breaking history.
- Published honest — the reference ACP runner declares 5 declinations in its own committed result file, reasons included (single-tenant runner scope, one capability ACP doesn't have yet). A benchmark that says "we pass everything" isn't credible.
We want your product represented. The ACP team built this benchmark, but the scenarios don't know what ACP is — the same BaseRunner interface works for any governance product, regardless of vendor.
- Implement
BaseRunnerinrunners/<your-product>.py— typically ~200 lines. - Run the full scenario set and commit
results/<your-product>-vX.Y.Z.json. - Open a PR. No cherry-picking, no hidden config. That's the point.
See CONTRIBUTING.md for the runner template and PR checklist.
v0.2 — 48 scenarios across 8 categories. Reference ACP runner passes 46/48 with 5 documented declinations (see the committed result file). Seven frameworks shipped, each with a native and an ACP runner. Live scorecard at agenticcontrolplane.com/benchmark.
Maintained by the Agentic Control Plane team. We're the first to put a number on our own governance product; we'd like the rest of the space to follow.
@software{agentgovbench2026,
title = {AgentGovBench: an open benchmark for AI agent governance},
year = {2026},
version = {0.2.0},
url = {https://github.com/agentic-control-plane/agentgovbench}
}
MIT. See LICENSE.