trailmark: update skills to use v0.4.0 features#183
Conversation
|
@claude review once |
Redraws the version boundary to match the actual Trailmark release history, verified against the v0.2.0/v0.2.1/v0.2.2/v0.3.1/v0.4.0 tags of trailofbits/trailmark: P1 (misclassified APIs, fixed): - ancestors_of(), reachable_from(), entrypoint_paths_to(), nodes_with_annotation(), clear_annotations() and the diff/entrypoints CLI exist since v0.2.0 — moved from the v0.4+ list to the v0.2-safe baseline; removed needless hasattr() gates and degraded fallbacks - supported_languages()/detect_languages() (trailmark.parse) are 0.3+ modules, not v0.2-safe — annotated as such - CLI --version/version were added in 0.2.2, not 0.4 — documented as 0.2.2+ and version-probe failure semantics clarified - graph-evolution/SKILL.md reverted to main: native diff has existed since v0.2.0 with identical args, so the 0.2.x-fallback rewrite was built on a false premise (also resolves the quality-checklist contradiction flagged in review) P3 (fixed): lexical version comparison hazard noted; stale diagramming-code checklist label; README baseline list now defers to the SKILL.md Version Gate instead of keeping a second divergent copy Dismissed: 'diff CLI signatures contradict' (both forms valid — before/ after positionals accept paths or git refs, --repo/--json exist since 0.2.0); 'subgraph_edges has no edge_kinds param' (v0.4.0 signature has edge_kinds keyword); 'diagram.py fallback broken on 0.2.x' (trailmark.diagram module exists in 0.2.x; only the CLI subcommand is new); 'phantom diff_against()' (real, v0.2-safe, now in baseline) Verified: check_claude_loadability.py, check_codex_loadability.py, pre-commit hooks pass; all SKILL.md files under 500 lines Co-Authored-By: Claude Fable 5 <[email protected]>
Review SummaryReview ran four passes (pr-review-toolkit code-reviewer + comment-analyzer, Codex The core issue: the PR drew the v0.2/v0.4 boundary in the wrong place. Several APIs labeled v0.4+ have existed since v0.2.0, and one API labeled v0.2-safe doesn't exist until 0.3.x. Runtime behavior was mostly protected by the Findings
Claims verified as correct and kept: proxy node format Verification
Commitbe058c7 |
|
Re-verified after be058c7. The review feedback's version-boundary fixes are present on |
Interesting failure mode with the newlines there, Codex. |
|
@claude review once |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| Runs `trailmark analyze --language auto --summary` on a target directory. | ||
| This is a v0.2-safe workflow; do not require Trailmark 0.4.0 just to produce a | ||
| summary. |
There was a problem hiding this comment.
🔴 Two skills added in this PR claim v0.2-safe workflows (trailmark-summary/SKILL.md:9-11 and trailmark-structural/SKILL.md:9-11) but their operational Step 2 (and Step 3 in trailmark-structural) unconditionally execute from trailmark.parse import detect_languages. Per this PR's own Version Gate in trailmark/SKILL.md, trailmark.parse was added in 0.3.x — on a true 0.2.x install (which the README explicitly claims to support), these imports raise ModuleNotFoundError before any analysis runs, and the documented fallback (uv run python) hits the same install and fails identically. Fix: either drop the parse-API call and rely on QueryEngine.from_directory(language="auto") plus trailmark analyze --language auto --summary (both v0.2-safe per the baseline), or restate the compatibility floor for these two skills as 0.3+.
Extended reasoning...
Contradiction
This PR introduces a direct internal contradiction in two skill files:
plugins/trailmark/skills/trailmark-summary/SKILL.md— line 10 (added in this diff): "This is a v0.2-safe workflow; do not require Trailmark 0.4.0 just to produce a summary." But Step 2 of the mandatory Execution section runsfrom trailmark.parse import detect_languages.plugins/trailmark/skills/trailmark-structural/SKILL.md— line 10 (added in this diff): "The core workflow is v0.2-safe; v0.4-only details are included only after checking method availability." But Steps 2 and 3 both runfrom trailmark.parse import detect_languages.
The PRs own Version Gate (in plugins/trailmark/skills/trailmark/SKILL.md) states authoritatively: "Added in 0.3.x: the trailmark.parse module with module-level detect_languages() and supported_languages()." The README also flags trailmark.parse.detect_languages() as 0.3+.
Why the fallback does not save it
Both skills document the same fallback: "If the import fails, rerun the same snippet with uv run python - {args}." This addresses a different failure mode — Python interpreter not seeing the project virtualenv. On a true 0.2.x install, python3 and uv run python both resolve to the same installed trailmark package, where the trailmark.parse module simply does not exist. Both invocations raise ModuleNotFoundError identically.
Step-by-step proof on a 0.2.x install
- User has
trailmark==0.2.2installed (within the PRs claimed support window — README: "these skills support Trailmark 0.2.x through the 0.4.0 release line"). - Vivisect or galvanize invokes
trailmark-summary. The skill announces "This is a v0.2-safe workflow" on line 10. - Step 1 passes —
trailmark analyze --helpworks on 0.2.x. - Step 2 executes the heredoc:
python3 - "{args}" <<PY ... from trailmark.parse import detect_languages ... PY. - Python raises
ModuleNotFoundError: No module named trailmark.parse. - The skills documented fallback runs the same heredoc under
uv run python. Same trailmark install, same module-not-found, same error. - The skill cannot proceed. Step 4s success criteria mandates "Detected languages from Step 2" — which never produced output. A v0.2.x user cannot complete the documented "v0.2-safe" workflow.
The same trace applies to trailmark-structural (with two failure points — Steps 2 and 3 — rather than one).
Why the existing fix was incomplete
This is the same class of issue as Finding #2 in dguidos timeline comment, which was supposedly fixed in be058c7 by "documenting as 0.3+; import sites annotated." That commit did add the guard comment # trailmark.parse is a 0.3+ module; on 0.2.x pass language="auto" instead at illustrative import sites in trailmark/SKILL.md:119 and references/query-patterns.md:158 — but it missed the operational import sites in trailmark-summary/SKILL.md (Step 2) and trailmark-structural/SKILL.md (Steps 2 and 3), which are the sites the agent actually executes.
Impact
The "v0.2-safe" claim is load-bearing: it tells downstream skills (vivisect, galvanize) that they may invoke this workflow on a 0.2.x install. The contradiction silently breaks both skills on a configuration the PR explicitly promises to support.
Suggested fix
Either:
- Replace the parse-API call with
QueryEngine.from_directory(target, language="auto")(v0.2-safe per the Version Gate baseline) and rely ontrailmark analyze --language auto --summary(which the summary skill already invokes in Step 3) for language detection. This keeps the "v0.2-safe" claim true. - Or relabel both skills as requiring Trailmark 0.3+ if the parse-API call is intentional.
2 participants
Summary
Updates the
trailmarkplugin skills to document and use Trailmark v0.4.0 features while remaining backward-compatible with v0.2.x installs. Bumps the plugin version 0.8.2 → 0.8.3.This is a documentation-only change to the skill content; no scripts or executable behavior change.
What changed
trailmark --version) or runtimehasattr()probe, with a documented v0.2-safe fallback. The skills never assume v0.4 APIs exist.trailmark: Adds a "Version Gate" section listing the v0.2-safe baseline vs. v0.4+ features (proxy nodes, node origins, transitive slices,entrypoint_paths_to(),subgraph_edges(),type_references(), binary augmentation, nativeversion/entrypoints/diff/diagramCLI). Expanded graph model docs (proxy nodes, new origins and edge kinds, expanded parser list).audit-augmentation: Documentsengine.augment_binary()for importing external binary-analysis graph exports (v0.4+), behind a version gate, with the expected JSON shape.diagramming-code: Notes the nativetrailmark diagramcommand (v0.4+) with a version check, falling back to the bundled script otherwise.graph-evolution: Gates nativetrailmark diffbehind a version check; falls back tograph_diff.pyover JSON exports on v0.2.x.trailmark-structural/trailmark-summary: Optional version recording andhasattr()-probed v0.4 fields (proxy counts, subgraph edges, type/reference summaries).marketplace.json,plugin.json, and README to reflect the expanded (binary + source) capabilities.Verification
uv run python3 .github/scripts/check_claude_loadability.py✅uv run python3 .github/scripts/check_codex_loadability.py✅plugins/trailmark/.claude-plugin/plugin.jsonand the rootmarketplace.json(both 0.8.3).🤖 Generated with Claude Code