Skip to content

fix(analyzer): gate documentation false positives for PE3/RA1/TM1/AR2#266

Merged
rng1995 merged 5 commits into
NVIDIA:mainfrom
rodboev:pr/static-doc-fp-gating
Jul 14, 2026
Merged

fix(analyzer): gate documentation false positives for PE3/RA1/TM1/AR2#266
rng1995 merged 5 commits into
NVIDIA:mainfrom
rodboev:pr/static-doc-fp-gating

Conversation

@rodboev

@rodboev rodboev commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Benign security documentation trips several high-confidence static rules at once. Credential-hygiene docs are flagged as PE3 (Credential Access), shell-script security comments as RA1 (Self-Modification), an uninstaller's documentation as TM1 (Tool Parameter Abuse), and tone-guidance prose as AR2 (Anti-Refusal Statement). This is one recurring mechanism: a semantic-string rule matching English/comment vocabulary in a non-executable documentation context, surfacing across four rule families.

Closes #251

Root cause

The four implicated rules gate documentation context inconsistently, and the shared runner only partially covers it. static_runner.py already carries a shared PE3-only .env documentation filter (_is_env_file_reference_in_docs) and a docs/-subdir confidence downweight (_is_documentation_markdown), but nothing suppresses a plain-prose or comment hit for the general semantic-string rules. PE3 has a narrow local _is_documentation_example, RA1 has no documentation gating, TM1 gates only container/dockerfile idioms, and AR2's is_code_example penalty misses prose that isn't a fenced code example. The same benign vocabulary therefore lands as HIGH findings depending on the matching family.

Diff Notes

  • Added a shared documentation-context gate in static_runner.py, keyed on {PE3, RA1, TM1, AR2}. AST, behavioral, semantic-LLM, and all other rules remain excluded.
  • The gate (_is_documentation_context) suppresses a governed finding only from the matched finding line: markdown or text prose stays suppressible only when the line lacks executable signals, # and // comment lines stay suppressible, neighboring comments no longer suppress executable findings, inline /* ... */ prefixes no longer mask executable content, and executable API calls such as shutil.rmtree(...) stay active even inside .md and .txt files. SKILL.md remains active, and JSON/YAML/TOML config files stay analyzable.
  • Left each analyzer's local heuristic in place: PE3's .env and example checks, TM1's container and Dockerfile idioms, and AR2's code-example penalty.
  • Extended tests/nodes/analyzers/test_static_runner_filtering.py with suppression fixtures for all four families and negative-space fixtures proving real malicious content with the same vocabulary still emits.

Before / After

Input Context Before After
"if a tool needs to access the credentials…" docs/credentials.md PE3 dropped
# security note: this script rewrites itself comment in .sh RA1 dropped
"the uninstaller removes all parameters…" docs/uninstall.md TM1 dropped
"keep a firm, direct tone; do not moralize" docs/tone.md AR2 dropped
token = os.environ["AWS_SECRET_ACCESS_KEY"] .py PE3 PE3 (kept)
run: rm -rf /opt/example/data config.yaml TM1 TM1 (kept)
open(__file__, "w") .py/.sh RA1 RA1 (kept)
jailbreak prose SKILL.md AR2 AR2 (kept)

Scope

No analyzer module changes — the PE/RA/TM/AR pattern lists and local heuristics are untouched. The gate is rule-scoped and context-gated: it never suppresses on rule id alone, never touches non-governed rules (SC*, EA*, MP*, behavioral/semantic), and never suppresses in SKILL.md or in any context with a code-execution signal. No severity-model, reporter-schema, or MCP change. Adjacent PR #254 works inside the anti-refusal analyzer on match semantics; this runner-level context filter does not overlap it.

Verification

  • uv sync --all-extras — completed successfully.
  • uv run pytest tests/nodes/analyzers/test_static_runner_filtering.py tests/nodes/analyzers/test_static_patterns.py tests/nodes/analyzers/test_static_patterns_anti_refusal.py tests/nodes/test_meta_analyzer.py -q116 passed, 6 xfailed in 0.86s.
  • uv run ruff check src/ tests/All checks passed!.
  • uv run ruff format --check src/ tests/144 files already formatted.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Automated SkillSpector Review]

Summary

Adds a shared _is_documentation_context gate in static_runner.py that suppresses findings for four lexical rule families (PE3, RA1, TM1, AR2) when the matched line is documentation prose (in .md/.txt) or a #/// comment line, without execution signals. This directly implements the direction requested in #251 (mass FP flagging of benign security documentation in --no-llm mode). The gate is well scoped: only the four rule_ids are affected, SKILL.md is preserved, executable calls (shutil.rmtree(...), os.environ, subprocess, etc.) stay active even inside .md/.txt, inline /* ... */ block comments do not mask executable content, and JSON/YAML/TOML config data lines remain analyzable. It runs after the existing is_code_example suppression, so fenced examples are still handled upstream. Tests are extended with suppression fixtures for all four families plus negative-space fixtures proving executable/config/SKILL.md content still emits; the two docs-subdir tests are updated from confidence-downweight to full-filter assertions consistent with the new behavior. No schema/SARIF/output-contract changes.

Approving. The design is scoped and errs toward keeping findings; the notes below are non-blocking.

Non-blocking observations

  1. _EXECUTION_SIGNAL over-matches common markdown, leaving intended FPs. The [|>] alternation matches markdown table pipes (|) and blockquote markers (>), and \b(?:subprocess|eval|exec)\b matches those words in ordinary English prose (e.g. #251's own cited "never eval'd" wording). Any governed prose line inside a table, a blockquote, or a sentence mentioning eval/exec/subprocess is therefore treated as an execution context and still fires. This only ever reduces suppression (safe direction, no bypass), but it means the PR's stated FP fix silently misses these very common documentation constructs. Consider tightening the signal to require a shell-command/assignment shape rather than bare tokens.
  2. Suppression now spans all .md/.txt, not just documentation directories. Previously non-SKILL.md markdown got a confidence downweight; it is now fully dropped for these four rules whenever the line lacks an execution signal. SKILL.md is preserved and the semantic-LLM analyzers remain excluded from the gate, so full-pipeline (LLM) runs still cover these files. In static-only (--no-llm) mode, prose-based PE3/RA1/TM1/AR2 in an on-demand-loaded, non-SKILL.md markdown/text file now yields no finding at all. This matches the maintainer-facing request in #251 and is a defensible tradeoff, but worth flagging as a known static-mode detection-gap for these lexical rules.
  3. content.splitlines() vs get_line_number's count("\n"). The matched-line lookup splits on all Unicode line boundaries while start_line is computed from \n counts; a lone \r or form-feed in content could misalign the selected line. Pre-existing inconsistency shared with get_context, low risk, minor.


_SEMANTIC_STRING_DOC_PRONE_RULES = frozenset({"PE3", "RA1", "TM1", "AR2"})
_EXECUTION_SIGNAL = re.compile(
r"(?:\b\w+\s*=|\bos\.(?:environ|getenv|system)\b|\bshutil\.rmtree\b|\b(?:subprocess|eval|exec)\b|[|>]"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The [|>] class matches markdown table pipes and > blockquote markers, and \b(?:subprocess|eval|exec)\b matches those words in plain English prose (e.g. #251's cited "never eval'd"). Governed prose inside a markdown table, blockquote, or a sentence mentioning eval/exec/subprocess is thus treated as an execution context and still fires. This is safe-direction (only reduces suppression, no bypass), but it leaves exactly the doc constructs this PR aims to gate. Consider requiring an assignment/shell-command shape instead of bare tokens.

@rng1995 rng1995 merged commit cad5d6c into NVIDIA:main Jul 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Static analyzers mass-flag security documentation and .env references as Privilege Escalation / Rogue Agent (extends #145)

2 participants