feat(scan): add opt-in transitive reference scanning

[rodboev]

Summary

Adds opt-in transitive scanning for source references inside scanned skill content while keeping directly requested roots complete, preserving baseline compatibility for direct findings, and surfacing child-budget truncation as incomplete analysis instead of silent coverage loss.

Closes #97

Root cause

The transitive traversal budget was attached too early in the scan pipeline. Direct roots could spend the shared child target, byte, and time budget during input resolution, context loading, graph execution, and recursive entrypoint setup. When a child budget later truncated reads, build_context() reduced the component inventory to the cached file subset, and report() treated truncation as display metadata instead of a completeness or verdict input. That combination let incomplete scans still look complete and sometimes report SAFE.

Diff Notes

• Add a shared transitive traversal module that extracts source-like external references from file_cache, filters out adjacent non-source URLs, canonicalizes source identities with dot-segment and unreserved percent-decoding normalization, enforces allow or deny prefix checks on normalized path boundaries, and owns per-root visited-state mutation.
• Add --transitive, --transitive-depth, and repeated allow or deny prefix controls to skillspector scan, then route both single-skill and recursive multi-skill entrypoints through a cached traversal helper.
• Keep directly requested roots outside the shared child traversal budget. The shared target, byte, and time limits now start with the first transitive child scan and stay shared across recursive roots.
• Preserve the full discovered component list even when a child scan hits the byte cap, keep file_cache partial, and compute completeness from discovered components instead of only cached content.
• Treat traversal truncation as an incomplete-analysis signal in every report format, add truncation reasons to report metadata, and floor SAFE to CAUTION without changing score or severity.
• Reuse InputHandler for approved external targets so existing host allowlists, SSRF checks, clone or download handling, and archive protections stay authoritative, and collapse redirect validation plus final streaming into one deadline-aware request path.
• Preserve cached child llm_call_log records and merge them into the final report, so degraded transitive semantic calls stay visible even when a dependency result is reused.
• Count transitive findings from active post-baseline report findings, so baseline-suppressed dependency findings do not inflate summaries.
• Add transitive_depth and source_url to Finding, preserve direct baseline fingerprints by adding provenance only for transitive findings, and keep source-aware component coverage so same-named dependency files do not collapse or appear skipped.
• Remove the dead cached-byte estimator path and other unused helpers left behind by the earlier budget work, and validate allow or deny prefixes before the root scan starts.
• Replay the branch onto current main.

Scope

This change stays on source types already supported by InputHandler, normalized trust-prefix checks, traversal depth plus hard fan-out limits, allow or deny prefix controls, cache reuse across recursive siblings, and provenance in reports. It does not add a web crawler, new allowed hosts, MCP behavior, or any default behavior change when --transitive is absent.

Verification

• pytest tests\\unit\\test_cli.py tests\\unit\\test_transitive.py tests\\unit\\test_input_handler_ssrf.py - pass, 71 passed
• pytest tests\\nodes\\test_analysis_completeness.py tests\\nodes\\test_report.py tests\\nodes\\test_sarif_rules_and_empty_findings.py - pass, 86 passed
• pytest tests\\unit\\test_suppression.py tests\\nodes\\test_llm_analyzer_base.py tests\\unit\\test_llm_utils.py - pass, 170 passed
• ruff check src tests
• ruff format --check src tests
• Invariant enumeration gate - pass, 69 rows, state domains checked
• CI Lint & Test (Python 3.12), Lint & Test (Python 3.13), and DCO Check - pending after push

[rng1995]

Requesting changes. Transitive prefix controls can be escaped through non-normalized URL paths, traversal has no fan-out budget, and shared visited state causes sibling skills to omit shared dependency findings. The merged report also misreports dependency coverage, several report tests are no longer collected, and the branch conflicts with current main. Please address the inline blockers and rebase while preserving current CLI, input-handler, and reporting behavior.

[rodboev]

Addressed the blockers on the transitive traversal path rather than patching each edge case in isolation. The branch now normalizes target identities before allow or deny checks, applies an explicit traversal budget, reuses cached child results per referring root instead of suppressing sibling findings behind one shared visited set, keeps dependency coverage source-aware in reporting, restores the report tests that had fallen out of pytest collection, and replays the whole change onto current main.

[rng1995]

[Automated SkillSpector Review]

Re-review: still requesting changes. The prior path-normalization, per-root cache/visited, source-aware coverage, and test-collection issues are addressed. Remaining blockers are that byte/time limits are checked only after a child scan, recursive roots do not share a command-level budget, child LLM degradation is dropped, baseline-suppressed transitive findings are overcounted, and the branch still conflicts with current main.

[rodboev]

Rebased onto current main as cb5ffa1. The update preserves the review-thread fixes already described above and removes the remaining merge conflict.

[rng1995]

[Automated SkillSpector Review]

Re-review of cb5ffa1 (rebased single commit). The PR adds opt-in --transitive scanning with canonicalized allow/deny prefix controls, a command-level target/byte/time budget shared across recursive roots, per-root visited state with a shared dependency result cache, cached child LLM telemetry, and source-aware reporting.

Prior-issue resolution checklist

Prior blocker (review 2026-07-09)	Status	Evidence
Byte/time limits checked only after a child scan	Resolved	Downloads are now streamed against remaining_bytes with mid-stream abort; git clones get --filter=blob:limit=N, a remaining_seconds subprocess timeout, and a post-clone tree-size check; file reads in build_context._read_file_cache are capped; LLM calls take per-call dynamic timeouts (LLMAnalyzerBase._model_for_call). Tests: test_input_handler_rejects_oversized_transitive_git_clone, TestDynamicTimeout, test_dispatches_timeout_to_cli_provider_complete.
Recursive roots do not share a command-level budget	Resolved	_scan_multi_skill creates one _TransitiveTraversalState (shared cache + counters + deadline) passed to every root; visited sets stay per-root. Tests: test_recursive_transitive_root_graphs_share_one_traversal, test_scan_transitive_root_graph_uses_shared_traversal.
Cached child llm_call_log dropped	Resolved	_CachedTransitiveResult.llm_call_log is stored and merged into the final report state; degraded metadata verified by test_scan_transitive_preserves_cached_child_llm_telemetry.
transitive_finding_count counted pre-baseline	Resolved	Count now derives from report_result["active_findings"] after baseline partitioning; test_scan_transitive_counts_only_active_post_baseline_findings covers suppression.
Branch conflicts with main	Resolved	Rebased to a single commit cb5ffa1; GitHub reports the PR mergeable.

New blockers

1. Root-scan byte-budget truncation silently zeroes coverage. When --transitive is set, the root skill's file reads consume the shared 10 MB budget (build_context._read_file_cache breaks on remaining_bytes), and every unread file is dropped from components/component_metadata entirely. _build_analysis_completeness then computes totals from component_metadata (i.e., only the files that were read), so a truncated scan reports 100% coverage, the dropped files never reach any analyzer, risk stays LOW, and the CLI exits 0 — only a truncation string in metadata hints anything was skipped. A skill padded with one >10 MB file that sorts first produces an empty root scan (file_cache={}, 0 findings) under --transitive; in recursive mode, sibling roots after budget exhaustion are scanned as empty too. Enabling the deeper-scan flag must not weaken the direct scan: exempt the directly requested root(s) from the byte cap (budget only transitive ingest), and/or keep unread files in the completeness totals as unscanned.
2. Budget truncation does not fail closed. Semantic analyzers skipped on deadline (semantic_*, meta_analyzer, TP4) record a truncation reason but no llm_call_record, so _llm_runtime_status sees no failures and the degraded-scan CAUTION floor never engages; combined with (1), a truncated deep scan can still yield a SAFE recommendation. Mirror the existing degraded-LLM behavior: floor risk_recommendation at CAUTION whenever transitive_truncated is true (or when analyzers were budget-skipped), with a regression test.

Non-blocking

• finding_fingerprint now unconditionally appends source_url/transitive_depth, which changes the hash for every existing baseline entry (root findings now hash ...||0) — all previously recorded baselines stop matching after upgrade. Provenance scoping is right for transitive findings, but append the extra fields only when source_url is set to keep existing baselines valid.
• _download_with_redirect_validation runs before the budget checks and uses self._remaining_seconds() or 30, so an expired deadline (0.0) becomes a 30 s timeout across up to 5 validation GETs; it also fetches the final URL twice (validation GET + content GET).
• Dead code: _CachedTransitiveResult.bytes_scanned / _estimate_scan_bytes are computed but never read; _URL_PATTERN and _EXCLUDED_PATH_MARKERS in transitive.py are unused.
• _scan_skill's try/except Exception: raise is a no-op, and _scan_state_with_baseline is a pure pass-through wrapper.
• Budgets are hardcoded (32 targets / 10 MB / 60 s) with no CLI override, and the 60 s clock starts before the root scan's own LLM calls — --transitive plus LLM will routinely truncate on realistic scans. Consider CLI-configurable budgets or scoping the time budget to transitive work.
• An invalid --transitive-allow-prefix/--transitive-deny-prefix value raises ValueError from _normalize_prefix only after the root scan has already run; validate prefixes at CLI parse time.
• The transitive CLI parameter shadows the transitive module import inside scan() (style).

Test coverage for the transitive machinery itself is thorough (traversal sharing, prefix normalization/boundaries, cache reuse, truncation reporting, SARIF provenance).

[rodboev]

The remaining failures came from letting the transitive budget leak into the direct root scan.

1. Directly requested roots now run without traversal budget state. The shared target, byte, and time limits start with the first transitive child scan and stay shared across recursive roots, so --transitive no longer reduces root coverage.

2. build_context() now keeps the full discovered component list even when a child hits the byte cap. file_cache still stays partial, so unread files show up as skipped instead of disappearing from coverage.

3. Any traversal truncation now marks the scan incomplete and lifts SAFE to CAUTION, even when the skipped analyzer never ran long enough to record its own LLM call.

4. Direct finding fingerprints stay backward compatible. Source and depth are added only for transitive findings, and redirect validation now stops immediately on an exhausted deadline while streaming the final response once.

5. The dead cached-byte plumbing and other unused helpers are gone, and allow or deny prefixes are validated before the root scan starts.