Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ BAWBEL_MAX_FILE_SIZE_MB=10
# Default: 30
BAWBEL_SCAN_TIMEOUT_SEC=30

# Minimum confidence score for a finding to pass FP-3 threshold check.
# The effective threshold depends on file profile (skill=0.60, mcp_manifest=0.55,
# documentation=0.85, unknown=0.60). This var overrides only the fallback threshold
# for files that don't match any named profile.
# Default: 0.80
# BAWBEL_CONFIDENCE_THRESHOLD=0.80


# ── Stage 2: LLM Semantic Analysis ───────────────────────────────────────────
# Requires: pip install "bawbel-scanner[llm]"
Expand Down Expand Up @@ -69,6 +76,23 @@ BAWBEL_SCAN_TIMEOUT_SEC=30
# BAWBEL_LLM_ENABLED=true


# ── FP-4: Meta-Analyzer ───────────────────────────────────────────────────────
# Requires: pip install "bawbel-scanner[llm]" (same dependency as Stage 2)
# Reviews medium-confidence findings (one LLM call per file) and reclassifies
# each as real (+0.15 confidence), needs_review (-0.05), or false_positive (suppressed).
# Skips automatically when no provider key is configured.

# Set to "false" to disable the LLM FP filter entirely
# Default: true
# BAWBEL_META_ANALYZER_ENABLED=true

# Confidence window — only findings within [MIN, MAX] are sent to the LLM.
# Findings above MAX are trusted as-is; findings below MIN are already suppressed.
# Default: 0.35 (min), 0.80 (max)
# BAWBEL_META_MIN_CONFIDENCE=0.35
# BAWBEL_META_MAX_CONFIDENCE=0.80


# ── Stage 3: Behavioral Sandbox ──────────────────────────────────────────────
# Requires: Docker Desktop or Docker Engine running
# Disabled by default — set BAWBEL_SANDBOX_ENABLED=true to enable.
Expand Down
139 changes: 69 additions & 70 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,73 @@ Versioning follows [Semantic Versioning](https://semver.org/).

---

## [1.3.0] - 2026-06-28

### Added

**Confidence and evidence metadata on every finding (Issue #69)**

Every `Finding` now carries four evidence fields populated at detection time:

- `confidence` — AVE-baseline confidence (0.0–1.0). Seeds from
`AVE_META[ave_id].confidence_baseline` when the engine fires, then adjusted by
FP-3 (score_confidence) and FP-4 (meta-analyzer). Always present in JSON output.
- `evidence_kind` — how evidence was gathered: `multi_engine`,
`tool_description_pattern`, `behavioral_pattern`, `semantic_inference`,
`file_type_mismatch`, or `config_schema`.
- `detection_stage` — `static_detection` or `runtime_observed`.
- `detection_layer` — `content`, `server_card`, `runtime`, or `registry_metadata`.

`ToxicFlow` gains a `confidence` field: `min(confidence_baseline)` across all
contributing findings, computed before the FP pipeline runs. Reflects raw detection
strength of the weakest link in the chain.

`scanner/ave_meta.py` is a new module with 51 `EveMeta` records (one per AVE ID
plus engine-specific overrides) providing `confidence_baseline`, `evidence_kind`,
`detection_stage`, and `detection_layer` for every known attack class. Engines that
produce no AVE match fall back to engine-type defaults.

**Golden JSON fixture for output contract stability (Issue #70)**

`tests/fixtures/golden/malicious_scan.json` is a committed snapshot of
`ScanResult.to_dict()` for the malicious fixture file. `TestGoldenSnapshot` in
`tests/unit/test_output_contracts.py` compares each live scan against the snapshot
to catch unintended schema changes. Regenerate the snapshot after intentional changes
with the command in the test module docstring.

25 schema contract tests lock: top-level field names, all 18 finding fields (including
the four new evidence fields), all 10 toxic flow fields, evidence kinds, detection
stages, confidence values, and risk score. YARA-dependent tests skip gracefully when
YARA is not installed.

**`confidence_band()` utility and evidence lifecycle documentation (Issue #71)**

`confidence_band(score)` in `scanner.core.fp_pipeline` maps a confidence score to
`"high"` (≥0.80), `"medium"` (0.55–0.79), or `"low"` (<0.55). Bands align with the
meta-analyzer window and profile thresholds.

`docs/guides/evidence-lifecycle.md` documents the full confidence pipeline: AVE
baseline → FP-2 negation → FP-3 scoring → threshold → FP-4 meta-analyzer →
`Finding.confidence`. Includes the ASCII flow diagram, FP-3 adjustment table,
`confidence_band()` tier table, pipeline ordering note, and lifecycle state reference.

34 evidence lifecycle tests cover: `confidence_band()` boundaries, all FP-3 context
adjustments (negation, table, heading, docs path, line number, cross-engine agreement,
skill name, AIVSS boost), threshold suppression per profile, all four meta-analyzer
verdict paths (`real`, `false_positive`, `needs_review`, no-provider skip), and
ToxicFlow confidence derivation.

### Fixed

- **LiteLLM Bedrock/SageMaker pre-load warnings** — two `WARNING` lines appeared on
every invocation (`could not pre-load bedrock-runtime response stream shape`) because
`logging.getLogger("LiteLLM").setLevel(logging.ERROR)` was called in `llm_engine.py`
*after* `import litellm`, giving litellm's module-level init code a window to log.
Fixed by setting the level at `meta_analyzer.py` module scope (runs at scanner import
time, before any lazy litellm import) and reordering the calls in `llm_engine.py`.

---

## [1.2.2] - 2026-05-20

### Fixed
Expand Down Expand Up @@ -113,75 +180,6 @@ Pattern engine: 37 rules -> 40 rules.

---

## [1.2.0] - 2026-05-16

### Added

**Justified suppression and false positive feedback (Part 14)**

Two new suppression keywords on top of the existing `bawbel-ignore` system:

- `bawbel-ignore` with metadata fields (`reason`, `reviewer`, `reviewed`) declares a
false positive permanently. The reason is recorded in the audit trail.
- `bawbel-accept` with an `expires` field declares an accepted risk. When the expiry
date passes, the finding resurfaces automatically as an active finding on the next scan.

`bawbel accept` CLI command inserts justified suppression comments directly into source
files. `bawbel accept --list` shows all accepted findings. `bawbel accept --expiring-soon`
shows findings expiring within a configurable window and exits 1 for CI use.

Anonymous FP signals can be sent to PiranhaDB via `--report`. Only AVE ID, engine,
confidence score, and a hash of the match context are sent. No file content.

`ScanResult.accepted_findings` is a new field in JSON output containing full metadata
for each justified suppression.

**New detection rules**

Three new AVE records and pattern rules:

- `bawbel-hook-hijack` (AVE-2026-00046): MCP tool hook hijacking. CRITICAL, AIVSS 9.1.
Detects skill files that register hooks to intercept or redirect tool execution calls.
- `bawbel-hardcoded-credential` (AVE-2026-00047): Hardcoded credentials. HIGH, AIVSS 7.8.
Detects API keys, tokens, passwords, private keys, and URL-embedded credentials.
- `bawbel-unsafe-delegation` (AVE-2026-00048): Unsafe agent delegation chain. HIGH, AIVSS 8.2.
Detects sub-agent spawning with inherited permissions and no trust boundary.

Pattern engine: 37 rules -> 40 rules.

**New commands**

- `bawbel creds <path>`: credential-focused scan, filters to AVE-2026-00047 and related
rules. Same output format as `bawbel scan`. Supports `--recursive`, `--no-ignore`,
`--fail-on-any`, `--format json`.
- `bawbel chain <path>`: delegation chain scanner, filters to AVE-2026-00048 and related
rules. Same flags as `bawbel creds`.

**`bawbel report` improvements**

- Added `--recursive` / `-r` flag. `bawbel report ./skills/ --recursive` generates
a full remediation report for every file in the directory.
- Added `--no-ignore` flag matching `bawbel scan`.

### Changed

- `scanner.py` Step 10 added: justified suppression runs after Step 9 (inline suppression).
Expired accepted risks are re-surfaced as active findings at this stage.
- Pattern engine rule count: 37 -> 40.

### Fixed

- `pr-review.yml` regression-check job: missing `pip install -e .` caused scan import
failures on clean repos.
- `ci.yml` test job: missing `pip install -e .` caused import failures.
- `ci.yml` Docker verify step: `python3 -c "..."` with f-strings caused shell brace
expansion to mangle the script before Python saw it. Replaced with single-line
assertion using no f-strings.
- `ci.yml` Docker verify step: wrong `aivss` field name (should be `aivss_score`),
wrong threshold (9.0 should be 7.0 to match actual fixture score).

---

## [1.1.1] - 2026-05-07

### Fixed
Expand Down Expand Up @@ -509,7 +507,8 @@ First public release.

---

[Unreleased]: https://github.com/bawbel/scanner/compare/v1.2.2...HEAD
[Unreleased]: https://github.com/bawbel/scanner/compare/v1.3.0...HEAD
[1.3.0]: https://github.com/bawbel/scanner/releases/tag/v1.3.0
[1.2.2]: https://github.com/bawbel/scanner/releases/tag/v1.2.2
[1.2.1]: https://github.com/bawbel/scanner/releases/tag/v1.2.1
[1.2.0]: https://github.com/bawbel/scanner/releases/tag/v1.2.0
Expand Down
10 changes: 5 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
# docker run --rm bawbel/scanner:test
#
# production - minimal runtime image, non-root user, read-only fs
# docker build --target production -t bawbel/scanner:1.2.3 .
# docker run --rm -v $(pwd)/skills:/scan:ro bawbel/scanner:1.2.3 scan /scan
# docker build --target production -t bawbel/scanner:1.3.0 .
# docker run --rm -v $(pwd)/skills:/scan:ro bawbel/scanner:1.3.0 scan /scan
#
# Build args:
#
Expand All @@ -24,7 +24,7 @@
# ─────────────────────────────────────────────────────────────────────────────

ARG PYTHON_VERSION=3.12
ARG VERSION=1.2.3
ARG VERSION=1.3.0


# ── Base: shared system dependencies ──────────────────────────────────────────
Expand Down Expand Up @@ -101,7 +101,7 @@ CMD ["python", "-m", "pytest", "tests/", "-v", "--tb=short"]
# ── Production: minimal runtime image ─────────────────────────────────────────
FROM python:${PYTHON_VERSION}-slim AS production

ARG VERSION=1.2.3
ARG VERSION=1.3.0
ARG WITH_YARA=false
ARG WITH_SEMGREP=false
ARG WITH_LLM=false
Expand All @@ -117,7 +117,7 @@ LABEL org.opencontainers.image.title="Bawbel Scanner" \
org.opencontainers.image.vendor="Bawbel" \
org.opencontainers.image.documentation="https://bawbel.io/docs" \
bawbel.aivss.spec="0.8" \
bawbel.ave.records="45"
bawbel.ave.records="51"

WORKDIR /app

Expand Down
116 changes: 111 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://pypi.org/project/bawbel-scanner/)
[![AIVSS aligned](https://img.shields.io/badge/AIVSS-v0.8-teal.svg)](https://aivss.owasp.org)
[![AVE Records](https://img.shields.io/badge/AVE_Records-48-green.svg)](https://github.com/bawbel/ave)
[![AVE Records](https://img.shields.io/badge/AVE_Records-51-green.svg)](https://ave.bawbel.io)
[![MCP Registry](https://img.shields.io/badge/MCP_Registry-listed-purple.svg)](https://registry.modelcontextprotocol.io)

<!-- [![Star History Chart](https://api.star-history.com/svg?repos=bawbel/scanner&type=Date)](https://star-history.com/#bawbel/scanner&Date) -->
Expand Down Expand Up @@ -193,10 +193,83 @@ Eight layers run automatically before a finding is reported:
| FP-5c | `.bawbelignore` patterns | per-file |
| FP-6 | Justified suppression with audit trail | per-finding |

Every active finding carries a `confidence` field (0.0–1.0) that starts from the
AVE-class baseline and is adjusted by FP-2 through FP-4 before appearing in output.
`confidence_band()` maps it to `"high"` / `"medium"` / `"low"` for human display.
See [Evidence Lifecycle](docs/guides/evidence-lifecycle.md) for the full pipeline.

See [Suppression Guide](docs/guides/suppression.md) for full details.

---

## Toxic flow detection

A single `fetch()` call is a finding. A `fetch()` that retrieves credentials and then
sends them to an external endpoint is an attack chain — and the two findings together
are far more dangerous than either alone.

Bawbel is the only open-source scanner that detects these **toxic flows**: compound
attack sequences where two or more findings combine into a higher-severity threat.
After deduplication, every finding is mapped to a capability tag. Bawbel then checks
all pairs against 12 built-in chain definitions and raises a `ToxicFlow` when a
dangerous combination is found.

```
skill.md findings:
AVE-2026-00003 credential-read (READ ~/.ssh/id_rsa) AIVSS 6.8 MEDIUM
AVE-2026-00026 data-exfil (POST to external endpoint) AIVSS 6.8 MEDIUM
│ │
└───── chain ────┘
ToxicFlow: credential-exfiltration AIVSS 9.8 CRITICAL
confidence: 0.83 (min of contributing baselines)
```

The toxic flow AIVSS (9.8) is higher than either individual finding (6.8), because
the chain represents a complete, end-to-end exploit — not just a capability.

**12 built-in chains:**

| Flow | Capabilities required | AIVSS |
|---|---|---|
| Credential Exfiltration | credential-read + data-exfil | 9.8 |
| Remote Code Execution | code-exec + external-fetch | 9.7 |
| Supply Chain RCE | supply-chain + code-exec | 9.6 |
| Goal Override + Execution | goal-hijack + code-exec | 9.5 |
| Lateral Movement + Execution | lateral-movement + code-exec | 9.4 |
| Tool Poisoning + Exfiltration | tool-poison + data-exfil | 9.3 |
| Identity Spoof + Escalation | identity-spoof + privilege-escalation | 9.2 |
| Persistence + Exfiltration | persistence + data-exfil | 9.1 |
| Context Inject + Memory Write | context-inject + memory-write | 8.9 |
| Goal Override + Exfiltration | goal-hijack + data-exfil | 8.8 |
| Scope Expansion + Exfiltration | scope-expansion + data-exfil | 8.7 |
| Covert Channel + Persistence | covert-channel + persistence | 8.6 |

**Toxic flow in JSON output:**

```json
{
"flow_id": "credential-exfiltration",
"title": "Credential Exfiltration Chain",
"severity": "CRITICAL",
"aivss_score": 9.8,
"confidence": 0.83,
"ave_ids": ["AVE-2026-00003", "AVE-2026-00026"],
"capabilities": ["credential-read", "data-exfil"],
"owasp_mcp": ["MCP06", "MCP07"],
"remediation": "Remove credential access. Block egress to untrusted endpoints."
}
```

`confidence` is `min(baseline confidence)` across the contributing findings —
the weakest link in the chain. A chain is only as confident as its least certain component.

Adding a new flow requires one entry in `scanner/core/toxic_flows/flows.py`. No other
files need to change.

---

## Install

**pip**
Expand All @@ -215,8 +288,8 @@ Requires Python 3.10+. No other system dependencies for core install.

| Image | Engines | Best for |
|---|---|---|
| [`bawbel/scanner:latest`](https://hub.docker.com/r/bawbel/scanner) · `1.2.3` | Pattern | Lightweight CI pipelines |
| [`bawbel/scanner:full`](https://hub.docker.com/r/bawbel/scanner) · `1.2.3-full` | Pattern + YARA | Recommended for most users |
| [`bawbel/scanner:latest`](https://hub.docker.com/r/bawbel/scanner) · `1.3.0` | Pattern | Lightweight CI pipelines |
| [`bawbel/scanner:full`](https://hub.docker.com/r/bawbel/scanner) · `1.3.0-full` | Pattern + YARA | Recommended for most users |

```bash
# Scan a local directory (recommended image)
Expand Down Expand Up @@ -363,13 +436,46 @@ AARS is the sum of 10 Agentic Risk Amplification Factors scored per the
"aivss_severity": "HIGH",
"spec_version": "0.8"
},
"confidence": 0.98,
"evidence_kind": "multi_engine",
"detection_stage": "static_detection",
"detection_layer": "content",
"owasp_mcp": ["MCP03", "MCP04"],
"piranha_url": "https://api.piranha.bawbel.io/records/AVE-2026-00001"
}
```

---

## AVE — the taxonomy behind every finding

Bawbel Scanner implements [**AVE** (Agentic Vulnerability Enumeration)](https://ave.bawbel.io),
the behavioral classification standard for agentic AI components.

AVE assigns stable identifiers to distinct attack classes — each with an AIVSS score,
a behavioral fingerprint, OWASP MCP Top 10 and MITRE ATLAS mappings, and indicators of
compromise. Every finding Bawbel produces maps to an AVE ID so teams using different
scanners speak the same language.

```
AVE-2026-00001 Metamorphic Payload via External Config Fetch AIVSS 8.0 HIGH
AVE-2026-00002 Tool Poisoning via Description Manipulation AIVSS 7.3 HIGH
AVE-2026-00046 MCP Tool Hook Hijacking AIVSS 9.2 CRITICAL
... 51 records total
```

| | |
|---|---|
| Records | 51 (AVE-2026-00001 → 00051) |
| Schema | v1.0.0 — validates at [ave.bawbel.io/schema.html](https://ave.bawbel.io/schema.html) |
| AIVSS | v0.8 — every record scored |
| Crosswalks | OWASP MCP Top 10 · MITRE ATLAS · NIST AI RMF · OWASP AST10 |

Any scanner can emit AVE IDs — see [ave.bawbel.io](https://ave.bawbel.io) for the
implementer guide and record index.

---

## Detection engines

| Engine | What it does | Install |
Expand Down Expand Up @@ -487,8 +593,8 @@ bawbel scan ./skills/ --format sarif # GitHub Security / GHAS

| | |
|---|---|
| [github.com/bawbel/ave](https://github.com/bawbel/ave) | AVE vulnerability database - 48 records |
| [api.piranha.bawbel.io](https://api.piranha.bawbel.io) | PiranhaDB - public threat intel API |
| [ave.bawbel.io](https://ave.bawbel.io) | AVE — Agentic Vulnerability Enumeration standard (51 records, schema, crosswalks) |
| [api.piranha.bawbel.io](https://api.piranha.bawbel.io) | PiranhaDB public threat intel API |
| [aivss.owasp.org](https://aivss.owasp.org) | OWASP AIVSS v0.8 scoring standard |
| [bawbel.io/docs](https://bawbel.io/docs) | Full documentation |

Expand Down
Loading
Loading