Skip to content

STANDARDS_AND_COMPLIANCE.md

Latest commit

 

History

History
212 lines (159 loc) · 27 KB

STANDARDS_AND_COMPLIANCE.md

File metadata and controls

212 lines (159 loc) · 27 KB
Raw

AI Security Standards, Compliance & Regulatory Frameworks

As AI systems move from experimental prototypes to mission-critical enterprise production, security auditing and compliance have become primary requirements. Security architects and compliance officers must navigate a rapidly evolving landscape of AI-specific threat matrices and international regulations.

This document serves as the definitive reference for AI security standards, risk frameworks, and regulatory compliance.

🗺️ MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems)

MITRE ATLAS is the absolute industry-standard framework for threat modeling AI systems. Modeled after the classic MITRE ATT&CK framework, ATLAS maps the specific tactics, techniques, and procedures (TTPs) used by real-world adversaries to compromise Machine Learning systems.

The ATLAS Tactical Matrix

ATLAS organizes attack techniques into 12 core tactics representing the logical progression of an AI compromise:

  1. Reconnaissance: Gathering information on target model architectures, data sources, API endpoints, and training pipelines.
  2. Resource Development: Acquiring infrastructure, poisoning datasets, or building specialized adversarial attack tools.
  3. Initial Access: Gaining a foothold through compromised Hugging Face tokens, supply chain poisoning, or prompt injection endpoints.
  4. Execution: Launching adversarial prompts, executing code via malicious model serialization (e.g., pickle exploits), or invoking unauthorized agent tools.
  5. Persistence: Injecting persistent backdoors in training data, fine-tuning checkpoints, or system prompts.
  6. Privilege Escalation: Escalating from an unauthenticated LLM user to invoking administrator-level OS commands via tool calling.
  7. Defense Evasion: Evading detection by adding subtle perturbations to inputs (evasion attacks) or formatting malicious payloads in Base64/multilingual scripts.
  8. Credential Access: Extracting API keys, Hugging Face tokens, or developer credentials stored in model registries or environment variables.
  9. Discovery: Mapping the internal network, discovering model endpoints, scanning connected vector databases, and listing active agent tools.
  10. Lateral Movement: Moving from a compromised LLM orchestration layer to the underlying cloud Kubernetes clusters or corporate databases.
  11. Collection: Gathering sensitive training data, system prompts, model weights, or user conversation logs.
  12. Exfiltration: Exfiltrating stolen model weights or proprietary data via model output leaks (steganography) or traditional network channels.
  13. Impact: Denying service by flooding model endpoints (AI-specific DoS), corrupting model weights, or manipulating outputs to cause reputational damage.

💡 Tactical Application: How to Use MITRE ATLAS for Threat Modeling

  • Step 1: Map the AI Pipeline Asset Surface: Define where data is ingested, where model weights are stored (e.g., MLflow, Hugging Face), where inference occurs, and what external tools/APIs the model can invoke.
  • Step 2: Walk Through the Matrix: For each asset, ask: "How could an attacker exploit this technique at this stage?" (e.g., T1566: Poison Training Data during the data ingestion phase).
  • Step 3: Document Mitigations: Map each identified threat to specific defensive controls (e.g., cryptographic hashing of datasets, runtime prompt firewalls).

🛡️ OWASP Top 10 for LLM Applications

The OWASP Top 10 for LLM Applications represents the foundational vulnerabilities that application security teams must mitigate when building LLM-powered applications.

Vulnerability Class Core Threat Primary Defense
LLM01: Prompt Injection Adversarial inputs manipulate the LLM into executing unauthorized actions or leaking system rules (Direct/Indirect). Enforce strict input sanitization, separate data from instruction context, and use runtime LLM firewalls.
LLM02: Insecure Output Handling Accepting LLM-generated output without validation, leading to XSS, CSRF, or Remote Code Execution (RCE) in downstream systems. Treat all LLM outputs as untrusted user inputs. Encode outputs before rendering and validate prior to execution.
LLM03: Training Data Poisoning Malicious corruption of pre-training or fine-tuning datasets to introduce backdoors, bias, or security bypasses. Verify data sources, cryptographically sign datasets, audit labels, and run adversarial evaluations post-training.
LLM04: Model Denial of Service Resource-heavy queries (e.g., massive context windows, recursive tool calling) that exhaust GPU memory and drive up API billing. Enforce strict rate limits, restrict maximum prompt token size, cap agent loop execution counts, and monitor GPU utilization.
LLM05: Supply Chain Vulnerabilities Compromised base models, malicious third-party datasets, compromised PyPI packages, or hijacked model registries. Scan model weights for serialization exploits (e.g., using safetensors), pin library dependencies, and sign model artifacts.
LLM06: Sensitive Information Disclosure The LLM revealing proprietary data, PII, system prompts, or trade secrets in its generated responses. Sanitize pre-training data, use system prompt defenses, and deploy active PII detection/scrubbing layers on outputs.
LLM07: Insecure Plugin Design LLM plugins/tools accepting raw text inputs from the model without validation, allowing attackers to hijack underlying APIs. Design plugins to require structured parameters (e.g., strict JSON schema), avoid dynamic shell execution, and enforce OAuth.
LLM08: Excessive Agency Granting LLMs broad permissions, high privileges, or autonomous execution rights without human-in-the-loop checks. Follow the principle of least privilege. Do not allow LLMs to write code and execute it directly without user confirmation.
LLM09: Overreliance Blindly trusting LLM outputs (code, security advice, facts) without validation, leading to software supply chain vulnerabilities. Implement automated static/dynamic code analysis (SAST/DAST) on all LLM-generated code before deployment.
LLM10: Model Theft Exfiltrating proprietary model weights or copying model functionality through recursive API querying (distillation). Enforce API rate-limiting, monitor output query patterns for extraction signatures, and secure model storage registries.

🤖 OWASP Top 10 for Agentic Applications (2026)

The LLM Top 10 above is the model-era baseline. Once a model can plan, delegate, use tools, and act autonomously, a new class of risk appears that a prompt-level list never captured — so the OWASP GenAI Security Project shipped a dedicated agentic list. Released December 2025 (peer-reviewed with input from 100+ contributors), the OWASP Top 10 for Agentic Applications 2026 uses the ASI prefix (Agentic Security / Agentic Systems Initiative) and each entry maps back to one or more foundational LLM risks while adding attack vectors that only exist under autonomy, tool integration, and multi-agent coordination.

If you build agents, this — not the LLM Top 10 — is your baseline. Each row below links the agentic risk to the LLM-era risk it extends and to the real-world incident in INCIDENTS_AND_GUIDANCE_2026.md that demonstrates it.

ID Risk What's new once the model can act Extends / maps to
ASI01 Agent Goal / Behavior Hijack An attacker redirects the agent's objective mid-task, not just a single response — turning your asset into a weapon that keeps pursuing the hijacked goal across steps. LLM01 (Prompt Injection); realized by EchoLeak, CometJacking
ASI02 Tool Misuse & Exploitation Abusing tools the agent was legitimately granted (over-broad scope, missing arg validation) to reach systems the model itself never could. LLM07 (Insecure Plugin Design), LLM08 (Excessive Agency); the MCP tool-poisoning cluster, Semantic Kernel tool→RCE
ASI03 Agent Identity & Privilege Abuse One broad token or shared identity across an agent fleet means a single compromise inherits everything the agent can do. LLM08; LiteLLM gateway breach, ServiceNow Now Assist privilege reuse
ASI04 Agentic Supply Chain Compromise Poisoned MCP servers, malicious tool definitions, or trojaned sub-agents entering via the agent's extension surface, not the model. LLM05 (Supply Chain); first in-the-wild malicious MCP (postmark-mcp), IDEsaster
ASI05 Unexpected Code Execution The agent reaching a code/eval/shell sink — often the framework's own — with attacker-influenced input. LLM02 (Insecure Output Handling); Semantic Kernel eval() (CVE-2026-26030), MCP Inspector RCE
ASI06 Memory & Context Poisoning Corrupting persistent memory or retrieved context so the attack survives across sessions — the model looks fine each turn, the state is compromised. LLM03 (Data Poisoning); ATLAS Memory Manipulation / Context Poisoning techniques
ASI07 Insecure Inter-Agent Communication In multi-agent systems, one agent injecting/deceiving another over an untrusted A2A channel — agent-to-agent recruitment. new to agentic; ServiceNow Now Assist agent-to-agent injection
ASI08 Cascading Agent Failures A single bad output propagating through a chain of agents, each amplifying the last — a failure mode with no single-agent analogue. new to agentic (systemic/emergent)
ASI09 Human-Agent Trust Exploitation Abusing the human's trust in the agent (e.g. an agent confidently requesting an approval it was tricked into) to get high-impact actions rubber-stamped. LLM09 (Overreliance)
ASI10 Rogue Agents An agent that drifts from its intended purpose without active external manipulation — misaligned reward/governance rather than a discrete exploit. new to agentic (alignment/governance)

How to use it: run the OWASP GenAI Red Teaming Guide four-phase method (below); the agentic risks land almost entirely in Phase 4 (runtime behavior), which is exactly where a model-only red team goes blind. For a matrix (tactic→technique) view of the same execution-layer threats, pivot to the MITRE ATLAS agentic expansion — ASI06 ≈ ATLAS Context Poisoning + Memory Manipulation, ASI07 ≈ Modify AI Agent Configuration.

Verification note (zero-fabrication): The list, ASI01–ASI10 IDs, and December 2025 release are corroborated across multiple independent write-ups of the OWASP publication (the OWASP GenAI project announcement, plus DeepTeam / Promptfoo / vendor guides). The OWASP primary resource + PDF (the resource page) currently blocks automated fetching, and some coverage renders ASI01 as either "Agent Goal Hijack" or "Agent Behavior Hijacking" — confirm exact wording, ordering, and any point-release renames against the official OWASP PDF before quoting in formal work. Incident cross-references point to entries already verified in INCIDENTS_AND_GUIDANCE_2026.md.

🎯 OWASP GenAI Red Teaming Guide

The Top 10 tells you what can go wrong; the OWASP GenAI Red Teaming Guide (first version released January 2025) is the OWASP Gen AI Security Project's structured, risk-based methodology for actively finding it. Its core contribution is refusing to treat "red teaming an LLM" as "type jailbreak prompts until one works." Instead it frames the exercise as a holistic, four-phase evaluation — because a model that passes prompt-level testing can still be trivially exploited through its deployment pipeline, its surrounding infrastructure, or its emergent behavior once tools and memory are attached.

Phase What it evaluates Why a single-layer test misses it
1. Model evaluation The model itself — alignment gaps, jailbreak susceptibility, bias, unsafe generation, training-data leakage. This is where most "red teaming" stops. Necessary, but blind to everything the model is wired into.
2. Implementation testing The application layer wrapped around the model — system-prompt defenses, guardrails, input/output handling, integration logic. Maps directly to OWASP LLM01/LLM02/LLM07: the model can be fine, but the glue code leaks or executes untrusted output.
3. Infrastructure assessment The deployment stack — model registries, serving infra, supply chain, secrets, access control around the inference endpoint. Covers LLM05/LLM10: weight exfiltration and supply-chain compromise never show up in a prompt-only test.
4. Runtime behavior analysis The live, in-production system — agentic tool use, memory, multi-turn drift, and emergent behavior under real traffic. This is where agentic risk lives (excessive agency, indirect injection via tools, context poisoning) — invisible to any static, pre-deployment check.

Why it belongs in this repo: it is the connective tissue between the taxonomy (OWASP Top 10, above) and the tooling (TOOLS.md: garak, PyRIT, promptfoo) — a phase model for deciding which tool to point at which layer. It also aligns cleanly with the MITRE ATLAS tactics above (phases 1–2 ≈ model/ML-attack tactics; phases 3–4 ≈ the agentic and infrastructure techniques) and with the real-world failures in INCIDENTS_AND_GUIDANCE_2026.md — EchoLeak and CometJacking are precisely the phase-4 runtime/agentic failures a model-only red team would never surface.

Audience: the guide explicitly targets a broad set of roles — security engineers, AI/ML engineers, red-team practitioners, risk managers, and business leaders — so it doubles as a shared-vocabulary document for getting a whole org aligned on what "we red-teamed it" actually means. Confirm the current version and any phase-naming updates on the primary resource page before quoting it in formal work.

🧭 OWASP GenAI Security Solutions Landscapes (Q2 2026) — including the first Red Teaming Landscape

The Red Teaming Guide (above) tells you how to test; the Solutions Landscapes tell you what to test with — they map the actual tooling ecosystem, open-source and commercial, onto the AI lifecycle so you can see which stage each tool covers and where the gaps are. Ahead of RSA 2026, the OWASP Gen AI Security Project expanded this line of work into a trio of Q2 2026 landscapes — and, notably, its first dedicated Red Teaming Landscape:

Landscape What it maps Why it matters here
LLM & GenAI Security Solutions Landscape The tooling ecosystem across the full LLM/GenAI lifecycle — development, testing, deployment, governance. The "what exists to defend this" companion to the OWASP Top 10's "what can go wrong."
Agentic AI Security Solutions Landscape (Q2 2026) Solutions across the agentic lifecycle, framed at the DevOps–SecOps intersection and tied to the Agentic AI Threats & Mitigations guide and SecOps tasks. Agentic risk (privilege escalation, tool abuse, memory/context poisoning) needs agent-aware tooling — this is the map of it.
AI & Agentic Red Teaming Landscape (Q2 2026) — the first of its kind A structured, lifecycle-wide agentic red teaming taxonomy for identifying, measuring, mitigating, and governing AI risk via coordinated adversarial testing, defensive validation, and continuous feedback loops. Turns "red teaming" from a one-off prompt exercise into a continuous, lifecycle discipline with a shared taxonomy — the missing structured vocabulary for agentic adversarial testing.

Why it belongs in this repo: these landscapes are the connective tissue between the methodology (the four-phase Red Teaming Guide above) and the operational tooling (TOOLS.md: garak, PyRIT, promptfoo, mcp-scan) — a way to check that your chosen tools actually cover every lifecycle stage rather than clustering on the easy (model-prompt) layer. The agentic red teaming taxonomy also maps directly onto the runtime/agentic failures cataloged in INCIDENTS_AND_GUIDANCE_2026.md (EchoLeak, CometJacking, ServiceNow Now Assist) and complements the MITRE ATLAS agent-technique vocabulary.

Verification / confirm-live: The existence of the three Q2 2026 landscapes and the launch of a first dedicated Red Teaming Landscape are corroborated by OWASP's own resource pages (linked above), the OWASP GenAI project's pre-RSA-2026 announcement (March 2026), and independent third-party analysis. The Solutions Landscapes are living documents — vendor/tool inclusion, the exact taxonomy, and version labels change each release, and OWASP-published landscapes do not imply endorsement of any listed vendor. Confirm the current edition and scope on the primary resource pages before citing specifics in formal work.

🏛️ NIST Artificial Intelligence Risk Management Framework (NIST AI RMF 1.0)

Released by the U.S. National Institute of Standards and Technology, the NIST AI RMF is a voluntary organizational framework designed to help enterprises design, deploy, and govern trustworthy and secure AI systems.

The Four Pillars of AI Governance

The framework is structured around four continuous core functions:

                  ┌───────────────┐
                  │    GOVERN     │
                  └───────┬───────┘
            ┌─────────────┼─────────────┐
            ▼             ▼             ▼
      ┌───────────┐ ┌───────────┐ ┌───────────┐
      │    MAP    │ │  MEASURE  │ │  MANAGE   │
      └───────────┘ └───────────┘ └───────────┘
  1. GOVERN: Build a culture of AI risk management. Establish clear internal policies, assign accountability, define ethical guidelines, and train teams on security and compliance mandates.
  2. MAP: Categorize the AI systems, identify the target users, document the system's operational boundaries, and map out the potential security and compliance risks.
  3. MEASURE: Conduct rigorous evaluations. Run quantitative testing (e.g., automated red teaming, bias assessment, latency tests) to verify the system's security, privacy, and safety posture.
  4. MANAGE: Allocate resources to continuously mitigate identified risks. Deploy runtime guardrails, implement fallback architectures, and set up continuous monitoring to handle failures.

🌐 ISO/IEC 42001:2023 (Artificial Intelligence Management System - AIMS)

Released in late 2023, ISO/IEC 42001 is the world's first international standard for certifying AI Management Systems. Similar to ISO 27001 (Information Security), it provides a structured process for organizations to manage AI risks throughout the product lifecycle.

Core Areas of ISO 42001 Compliance

  • System Integrity: Organizations must demonstrate controls to protect the confidentiality, integrity, and availability of AI assets (data, weights, models, code).
  • Data Governance: Strict requirements for verifying training data provenance, ensuring fair collection practices, and protecting data privacy.
  • Transparency and Explainability: Documented processes to ensure that AI model outputs can be traced, explained, and audited when errors or security incidents occur.
  • Lifecycle Management: Defining risk assessments at every stage—from feasibility research and data acquisition to model development, testing, deployment, and eventual retirement.

🇪🇺 The EU AI Act Compliance Playbook (2025–2026)

The European Union Artificial Intelligence Act (EU AI Act) is the world's first comprehensive legal framework regulating AI. It operates on a strict, risk-based classification system, imposing severe penalties (up to €35M or 7% of global annual turnover) for non-compliance.

Risk Categorization & Obligations

 ┌─────────────────────────────────────────────────────────────┐
 │  UNACCEPTABLE RISK (BANNED)                                 │
 │  e.g., Social scoring, biometric categorization, dark-patterns│
 ├─────────────────────────────────────────────────────────────┤
 │  HIGH RISK (STRICT AUDITS)                                  │
 │  e.g., Critical infrastructure, hiring, biometrics, healthcare│
 ├─────────────────────────────────────────────────────────────┤
 │  GENERAL PURPOSE AI (GPAI / FRONTIER LABS)                 │
 │  e.g., Base LLMs (Llama 3, GPT-4, Claude 3)                 │
 ├─────────────────────────────────────────────────────────────┤
 │  MINIMAL RISK (LOW REGULATION)                              │
 │  e.g., Spam filters, video games, basic classification       │
 └─────────────────────────────────────────────────────────────┘

🚫 Unacceptable Risk (Banned)

  • Systems: Real-time remote biometric identification in public spaces, cognitive behavioral manipulation (e.g., voice-activated toys encouraging dangerous behavior), social scoring, and untargeted scraping of facial images from the internet.
  • Compliance Timeline: Banned entirely within 6 months of entry into force.

⚠️ High Risk (Strict Compliance Required)

  • Systems: AI used in critical infrastructure (water, electricity), CV sorting software for hiring, biometric identification, grading exams, credit scoring, and law enforcement.
  • Compliance Requirements:
    1. Rigorous Risk Management: Establish a continuous, documented risk management system.
    2. High-Quality Datasets: Training, validation, and testing datasets must be checked for bias, poisoning, and data-governance violations.
    3. Technical Documentation: Maintain detailed system logs and architecture documents to prove compliance.
    4. Logging Capability: Automatically log events (audit trails) throughout the system's operational lifetime.
    5. Human Oversight: Design systems with fail-safes and manual intervention overrides ("Human-in-the-Loop").
    6. Accuracy, Robustness, and Cybersecurity: Systems must achieve high levels of adversarial robustness against inputs designed to alter performance (jailbreaks, poisoning, data exploitation).

🧠 General Purpose AI (GPAI / Foundation Models)

  • Systems: Base foundation models trained on massive data (like GPT-4, Llama 3, Claude 3).
  • Compliance Requirements:
    • Draw up technical documentation (e.g., training processes, evaluation results).
    • Publish a detailed summary of the content used to train the model.
    • Respect EU copyright law.
  • Models with Systemic Risk (Models trained on > $10^{25}$ FLOPs, e.g., frontier models):
    • Must conduct rigorous model evaluations (red teaming).
    • Must perform adversarial risk assessments.
    • Must report energy efficiency metrics.
    • Must implement state-of-the-art cybersecurity protections throughout model training and deployment.

📝 Enterprise AI Security Hardening Checklist

To protect and prepare your AI systems for compliance, implement this technical hardening baseline:

  • Weight Protection: Store model weights in a secure, encrypted object registry (e.g., AWS S3 with KMS). Restrict access using strict IAM roles. Never store model registry API tokens in plaintext in code.
  • Malicious Serialization Defense: Disable the loading of unverified models via pickle. Enforce the use of safe file formats (like safetensors) which prevent arbitrary code execution during loading.
  • Data Provenance Checks: Cryptographically sign raw training data (sha256 or GPG). Run outlier-detection algorithms on all ingested datasets to screen for data poisoning and backdoors.
  • Downstream Output Isolation: Treat all model responses as untrusted. Enforce HTML-encoding, output sanitization, and strictly typed JSON schemas before passing outputs to downstream applications, databases, or APIs (prevents indirect SQLi/XSS).
  • Excessive Agency Restriction: Set maximum loop boundaries on all autonomous agent runs. Run tool execution in isolated, read-only Docker containers or serverless runtimes. Implement explicit human approval prompts for high-risk actions (e.g., database writes, emails).
  • Pipeline Dependency Pinning: Lock all package dependencies using a package-lock system (requirements.txt.lock, pnpm-lock.yaml). Run automated software composition analysis (SCA) to detect compromised Python packages before builds.
  • Runtime Guardrails: Deploy a runtime guardrail framework (e.g., Llama Guard, NeMo Guardrails) to intercept prompts and output generations, filtering for PII, toxic content, and prompt injection signatures.
  • Strict Prompt Logging & Auditing: Maintain a secure, tamper-proof audit trail of all prompts, outputs, agent decisions, and tool executions to comply with high-risk system obligations under international law.