Add policy guardrails for statistical evaluation artifacts

[dgenio]

Context

agent-kernel provides capability-based authorization, policy enforcement, context firewalling and audit for agent tool ecosystems.

A useful cross-repo scenario is an agent consuming a statistical/model-evaluation artifact, such as an offline policy evaluation report from skdr-eval. These artifacts can be misused if an agent treats a headline value estimate as deployment evidence while ignoring support diagnostics, uncertainty or warnings.

Goal

Add a policy pattern for gating agent actions based on structured evaluation artifacts.

Example principle:

An agent may summarize a high-risk evaluation artifact, but it must not recommend deployment or automatic rollout when support diagnostics are high_risk.

Suggested capabilities / policy checks

Support a generic artifact policy layer that can inspect fields such as:

• artifact_type
• support_health
• warnings
• uncertainty
• decision_stable
• recommendation.intent
• limitations

Potential decisions:

• allow_summary
• allow_manual_review_recommendation
• require_human_review
• deny_deployment_recommendation
• deny_automatic_rollout

Example scenario

An agent receives an EvaluationArtifact with:

• candidate appears better than baseline;
• support_health = high_risk;
• warnings include low ESS or poor overlap.

Expected behavior:

• allowed: summarize the result and explain the caveats;
• allowed: recommend improving logs/support;
• denied: recommend deployment, rollout, or automatic A/B promotion as if the result were reliable.

Acceptance criteria

• Add a documented policy example for evaluation artifacts.
• Add tests for at least ok, caution, and high_risk support states.
• The policy is generic enough to support non-skdr-eval producers.
• Audit traces record why an action was denied or downgraded.
• Docs explain the distinction between summarizing evidence and acting on evidence.
• Align with weaver-spec EvaluationArtifact contract if/when available.

Non-goals

• Do not implement OPE/statistical estimation in agent-kernel.
• Do not hard-code a dependency on skdr-eval.
• Do not make policy decisions based only on a single numeric metric.

AI agent notes

This is a policy-safety example. Keep it small, generic and testable. Prefer fixture artifacts rather than external package dependencies.

[dgenio]

Closing as completed — this is fully implemented on main.

The work landed in commit affac5e ("docs: add ChainWeaver + evaluation-artifact integration cookbooks (#95, #96)", merged via #112) and is recorded in the CHANGELOG [Unreleased] section. Every acceptance-criterion box is met:

• Documented policy example — docs/integrations/evaluation_artifacts.md + runnable companion examples/evaluation_artifact_policy.py.
• Tests for ok / caution / high_risk — tests/test_evaluation_artifact_policy.py covers all three states (test_ok_artifact_allows_deployment, test_caution_artifact_downgrades_without_human_review, test_high_risk_artifact_requires_human_review).
• Producer-agnostic — assess_artifact() inspects a plain dict (support_health, warnings, uncertainty, decision_stable, recommendation.intent, limitations); no skdr-eval dependency, fixtures only (test_unknown_support_health_normalised_to_safest, test_missing_fields_default_to_safest).
• Not a single-metric gate — verified by test_gate_is_not_single_metric (stable-but-unhealthy and healthy-but-unstable both block).
• Audit traces record the reason for a denial/downgrade — test_act_on_artifact_audits_decision asserts the downgraded action records reason and downgraded_from in ActionTrace.args via Kernel.explain.
• Summarize-vs-act distinction documented in the walkthrough.

The "align with weaver-spec EvaluationArtifact contract" item was explicitly conditional ("if/when available"); the implementation is deliberately contract-agnostic, so it does not block closure. If a canonical spec contract later lands, please track that as a separate follow-up. Reopen if anything above is incomplete.

---

Generated by Claude Code