WIP: feat(audit): effort-profit improvement backlog (generate mode) - #148
Open
AlexanderMakarov wants to merge 27 commits into
Open
WIP: feat(audit): effort-profit improvement backlog (generate mode)#148AlexanderMakarov wants to merge 27 commits into
AlexanderMakarov wants to merge 27 commits into
Conversation
Argument-gated generate action for /awos:ai-readiness-audit: LLM-clustered tickets validated and rendered by a new generate-backlog engine verb — interactive backlog.html dependency graph, per-ticket Jira-style files, org-mode aggregation. Source: AppDev Practice Effort-Profit matrix item.
…lacement, filename notation
Rebuild minified bundle (dist/cli.js) with generate-backlog verb from tasks 1-6. Version unchanged at 2.2.0 per user feedback (release-drafter manages versioning).
…backlog drafts, correct spec wording
- backlog_render.ts: recompute() now updates rb-coverage-tip with the
live Σ coverage_delta formula, matching rb-effort-tip/rb-duration-tip
instead of leaving it static.
- backlog.ts/backlog_render.ts: org repos[] only carries backlog_href
for repos with a stamped backlog; audit-only fallback repos render
as plain text in the bottom repo list instead of a 404 link.
- backlog.ts: buildBacklog/buildOrgBacklog reject an empty
tickets/org_tickets array ("draft has no tickets" / "draft has no
org_tickets") instead of silently writing an empty backlog.
- docs/specs: correct backlog.html's self-contained description to
match the report.html convention (inline CSS/JS plus the same
Google Fonts <link>s, not fully self-contained).
- rebuild dist/cli.js from the updated engine sources.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR adds generate-mode audit dispatch, deterministic single-repository and organization backlog generation, ticket and HTML rendering, CLI support, combined audit-generation flows, and QA compliance checks. ChangesAudit backlog generation
Estimated code review effort: 5 (Critical) | ~90 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant AuditSkill
participant QAHarness
participant BacklogCLI
participant BacklogEngine
User->>AuditSkill: request generate or combined audit
AuditSkill->>QAHarness: start generation session
QAHarness->>BacklogCLI: execute generate-backlog with draft
BacklogCLI->>BacklogEngine: validate and build backlog
BacklogEngine-->>BacklogCLI: write backlog.json, tickets, and HTML
BacklogCLI-->>QAHarness: return generation summary
QAHarness-->>User: report artifacts and compliance
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
plugins/awos/skills/ai-readiness-audit/backlog_render.ts (1)
78-93: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSingle-repo and org
backlog.htmlrenderers duplicate most of the page shell.
renderBacklogHtml(480-578) andrenderOrgBacklogHtml(346-464) share near-identical header/ribbon/ribbon-warning/legend/graph-container/script markup, differing only in a handful of strings and the repos/member-table sections.layerTickets(78-93) andlayerOrgTickets(286-301) are the same depth-computation duplicated by key (slugvsid). Any future edit to shared markup (ribbon copy, legend wording, warning row) has to be made twice and can silently drift.Consider extracting a shared page-shell/ribbon/legend builder parameterized by the small org-specific bits (repos section, member table, extra warning line), and a single depth-layering helper keyed by a caller-supplied id-accessor.
Also applies to: 286-301, 346-578
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/awos/skills/ai-readiness-audit/backlog_render.ts` around lines 78 - 93, Refactor the duplicated backlog rendering logic by extracting a shared page-shell builder for the common header, ribbon, warning, legend, graph container, and script markup, parameterized by the org-specific sections, strings, and extra warning line. Replace the separate layerTickets and layerOrgTickets implementations with one depth-layering helper that accepts a caller-supplied ticket key accessor, and update both renderBacklogHtml and renderOrgBacklogHtml to use these shared helpers without changing their rendered content.plugins/awos/skills/ai-readiness-audit/backlog.ts (2)
897-920: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
scanPerRepo(orgDir)is run twice per org generate call.
generateOrgBacklog→buildOrgBacklogscansper-repo/*(reading every repo'sbacklog.json/audit.json) at Line 624, thencollectUnlinkedTicketWarnings(orgDir, draft)(Line 919) scans the exact same tree again at Line 877. For an org with many repos this doubles the JSON reads for no functional reason.♻️ Proposed fix — thread the already-scanned entries through
export function generateOrgBacklog( orgDir: string, draftRaw: unknown ): GenerateSummary { const draft = shapeCheckOrgDraft(draftRaw); - const backlog = buildOrgBacklog(orgDir, draft); + const entries = scanPerRepo(orgDir); + const backlog = buildOrgBacklog(orgDir, draft, entries); ... - warnings: collectUnlinkedTicketWarnings(orgDir, draft), + warnings: collectUnlinkedTicketWarnings(entries, draft), }; }(with
buildOrgBacklog/collectUnlinkedTicketWarningsacceptingentries: PerRepoScan[]instead of re-deriving it internally).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/awos/skills/ai-readiness-audit/backlog.ts` around lines 897 - 920, Update generateOrgBacklog, buildOrgBacklog, and collectUnlinkedTicketWarnings to share one PerRepoScan[] result: scan per-repo entries once in generateOrgBacklog, pass those entries into both consumers, and remove their internal scanPerRepo(orgDir) calls while preserving existing backlog and warning behavior.
305-348: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKahn topological-sort duplicated verbatim in
buildOrgBacklog.This block (ready-queue build, draft-order tie-break via
idOrder.indexOf, cycle detection viaremaining) is structurally identical to the one at Lines 754-795 inbuildOrgBacklog, differing only in the ticket type it closes over. Extracting a sharedtopoSort(ids, dependsOnById, validIds, dupIds)helper would remove ~40 duplicated lines and the risk of the two copies drifting apart on a future fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/awos/skills/ai-readiness-audit/backlog.ts` around lines 305 - 348, Extract the duplicated Kahn topological-sort logic from the current function and buildOrgBacklog into a shared topoSort(ids, dependsOnById, validIds, dupIds) helper. Update both call sites to use it while preserving draft-order tie-breaking, duplicate/invalid dependency handling, returned ordering, and cycle violation reporting.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/awos/skills/ai-readiness-audit/backlog.ts`:
- Around line 226-283: Update backlog validation to explicitly require arrays
before checking or iterating t.checks and t.depends_on, recording a validation
violation for truthy non-array values instead of allowing iteration to throw.
Apply the same Array.isArray guards to t.members and t.depends_on in
buildOrgBacklog, preserving normal validation for valid arrays and ensuring
malformed drafts raise BacklogValidationError rather than TypeError.
In `@tools/ai-readiness-audit/qa/run_audit_test.ts`:
- Around line 335-377: Make runGeneratePhase resilient when run2.jsonl is
absent: treat reading the generate log as best-effort, using empty lines or
equivalent fallback when fs.readFileSync fails, so the generate phase can still
return its compliance result and performRun can write run-meta.json. Preserve
normal parsing when the log exists and keep the existing artifact/compliance
flow unchanged.
---
Nitpick comments:
In `@plugins/awos/skills/ai-readiness-audit/backlog_render.ts`:
- Around line 78-93: Refactor the duplicated backlog rendering logic by
extracting a shared page-shell builder for the common header, ribbon, warning,
legend, graph container, and script markup, parameterized by the org-specific
sections, strings, and extra warning line. Replace the separate layerTickets and
layerOrgTickets implementations with one depth-layering helper that accepts a
caller-supplied ticket key accessor, and update both renderBacklogHtml and
renderOrgBacklogHtml to use these shared helpers without changing their rendered
content.
In `@plugins/awos/skills/ai-readiness-audit/backlog.ts`:
- Around line 897-920: Update generateOrgBacklog, buildOrgBacklog, and
collectUnlinkedTicketWarnings to share one PerRepoScan[] result: scan per-repo
entries once in generateOrgBacklog, pass those entries into both consumers, and
remove their internal scanPerRepo(orgDir) calls while preserving existing
backlog and warning behavior.
- Around line 305-348: Extract the duplicated Kahn topological-sort logic from
the current function and buildOrgBacklog into a shared topoSort(ids,
dependsOnById, validIds, dupIds) helper. Update both call sites to use it while
preserving draft-order tie-breaking, duplicate/invalid dependency handling,
returned ordering, and cycle violation reporting.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f02c2e80-0ad9-4a11-b27c-852ae670fb77
⛔ Files ignored due to path filters (1)
plugins/awos/skills/ai-readiness-audit/dist/cli.jsis excluded by!**/dist/**
📒 Files selected for processing (14)
docs/specs/2026-07-15-audit-improvement-backlog-design.mdplugins/awos/skills/ai-readiness-audit/SKILL.mdplugins/awos/skills/ai-readiness-audit/audit_patch.tsplugins/awos/skills/ai-readiness-audit/backlog.tsplugins/awos/skills/ai-readiness-audit/backlog_render.tsplugins/awos/skills/ai-readiness-audit/cli.tsplugins/awos/skills/ai-readiness-audit/render.tsplugins/awos/skills/ai-readiness-audit/tests/backlog-render.test.tsplugins/awos/skills/ai-readiness-audit/tests/generate-backlog.test.tstests/lint-prompts.test.jstools/ai-readiness-audit/qa/README.mdtools/ai-readiness-audit/qa/harness.test.tstools/ai-readiness-audit/qa/harness_lib.tstools/ai-readiness-audit/qa/run_audit_test.ts
…t name Extend OrgBacklogJson.repos so the org backlog can render a real repositories table: each repo entry now carries its current coverage, ticket count, and Σ ticket effort (engine-computed from the per-repo backlog.json, or coverage-only for an audit-only fallback repo). Also resolve the org backlog's project name properly. basename(orgDir) yields the run timestamp for the canonical context/audits/<timestamp> layout; now an explicit project field in org-portfolio.json is preferred, falling back to the project root's name three levels up when orgDir is a timestamped audits directory.
…bon, legend, repos table Address the reviewed look-and-feel issues on the interactive backlog: - Resolve org repo links and member ticket links from the org page location (../per-repo/…); the page lives one level below the audit dir. - Document foundation-at-top layer orientation and pin it with tests for both variants (depended-upon nodes render above their dependents). - Constrain node width and clamp titles to three lines so boxes stay squarish instead of stretching to one long line. - Draw filled endpoint markers at both ends of each edge (bottom-center of the upper node to top-center of the lower one) so connections read. - Blur a graph node after clicking and lift the tooltip's z-index so the hover tip no longer pins open (focus) or clips under later nodes. - Trim the org node tooltip to the node-box headline plus the member table (drop goal/description, which can differ across member repos). - Style the member table for the dark tooltip (light text, subtle borders, visible links) instead of inheriting dark-on-dark report cells. - Give disabled nodes a distinct background, not only reduced opacity. - Replace the prose legend with hardcoded structured legends per variant: interaction bullets plus a field/description/formula table. - Restructure the ribbon into three centered metric cards (Effort, Coverage gain, Duration) with the developer-count input inside the Duration card; keep the enable-all button, warning row, and live tips. - Replace the bottom repo list with a repositories table (linked name, current coverage with applicable-weight on hover, ticket count, effort to close identified gaps).
AlexanderMakarov
changed the title
The backlog graph node click handler blurred the node unconditionally to drop the hover tooltip. Keyboard activation (Enter/Space on the <button>) also fires a click, so keyboard users lost their tab position. Gate the blur on pointer interaction — e.detail > 0 — and keep focus for keyboard activation (e.detail === 0). Rebuild the committed dist bundle.
Contributor
|
one question regarding running audits , i see https://code.claude.com/docs/en/workflows#ask-for-a-workflow-in-your-prompt, CC documentation they recommend like audit and all kind of long running tasks to be run as a workflow , what do you think @AlexanderMakarov should we have the the whole audit as a workflow ? |
Contributor
Author
|
@Mgrdich - Good question — we actually went the other way on purpose. The slow part of an audit used to be per-dimension agent fan-out. It was retired for exactly the reliability reasons the workflow docs describe. But instead of orchestrating it, we replaced it with code: one deterministic engine command ( What remains for the LLM is small: judgment checks, connector fetches, report/backlog authoring. Two parts are interactive (scope confirmation, audit picker) — workflow scripts can't ask questions mid-run. One place does fit the workflow shape: org mode's per-repo fan-out (N repo-auditors → rollup). A workflow there would give deterministic sequencing and resume-with-cache when a run dies. Worth a targeted experiment as an optional path, not a rewrite of the whole audit. Tracking it as a follow-up idea rather than in this PR. |
- Group dependency-connected tickets into their own dashed component boxes (chains stacked vertically, edges short and unoccluded) and collect edge-less tickets under a labelled "Independent tickets" grid — the old single depth-0 layer wrapped into multiple rows, hiding edges behind node cards and scattering connected nodes. - Clamp graph-node tooltips to the viewport: flip above the node when space below runs out, shift horizontally, cap height with scroll. - Widen the org tooltip (tipbox-org) so the member table fits, and let the slug column wrap instead of overflowing the box. - Org tooltip member links now deep-link to the ticket's node in the repo backlog graph (#node-<slug>, scrolled/highlighted/focus-pinned via revealFromHash) instead of opening the raw ticket markdown. - Org rollup re-renders each per-repo backlog.html with a "← Back to org backlog" backlink once the org page exists. - Redraw edges after web fonts load so they track the final layout.
…aveat tooltip - Lift a hovered or focus-pinned node's whole stacking context (.gcomps/.glayer get z-index 30 on :hover and :focus-within) so an open tooltip paints over sibling graph sections instead of under the independent-tickets grid. - placeTip now measures available space from the sticky ribbon's bottom edge, not the viewport top, and height-caps the tooltip to the chosen side (scrolling inside) so it can never slide under the ribbon or off screen. - Edges are directed: arrowhead marker at the dependent end, dot at the dependency end; both legends name the arrow direction. - The Amdahl sublinear-scaling warning band moved into a hover tooltip on the Developers control in the Duration card. - Org repos table degrades missing per-repo fields (pre-upgrade backlog.json) to em dashes instead of "undefined"/NaN.
- Move "Enable all nodes" out of the ribbon into a new graph toolbar above the graph: the collapsible legend on the left, enable/disable buttons on the right. Add "Disable all nodes", which turns every ticket off so a plan can be built up node by node from zero. - Repositories table: cell values are left-aligned, and a new "Current score" column (the repo audit's achieved weighted points, with an out-of-N-applicable tooltip) sits between coverage and tickets. Org repo entries carry the new audit_total field — from the per-repo backlog when one exists, else from the stamped audit.json fallback — and pre-upgrade org backlog.json files degrade to an em dash.
Runs only the second (generate) claude session against the newest pre-existing audit dir in the target, skipping the audit phase and its engine-compliance retry loop entirely — for exercising backlog generation end-to-end without re-computing metrics. All other scaffolding (run lock, worktree deploy, target snapshot, MCP discovery, archive, run-meta, restore) is unchanged; exit code comes from the generate phase's compliance gate alone. Requires --generate; dies with a usage error otherwise, and with exit 2 when the target has no existing audit. The generated backlog/ stays in the target's audit dir afterward (that dir predates the run, so cleanup leaves it alone).
Non-array checks/depends_on/members in a generated backlog draft now fail validation with a BacklogValidationError violation instead of escaping as an uncaught TypeError from the for...of iteration. A defined-but-non-array depends_on records its own violation rather than being coerced to empty, which would otherwise pass validation and crash in the post-validation build step. Regression tests cover all four shapes alongside the existing unknown-member-repo TypeError case.
# Conflicts: # plugins/awos/skills/ai-readiness-audit/dist/cli.js
# Conflicts: # plugins/awos/skills/ai-readiness-audit/dist/cli.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an argument-gated generate mode to
/awos:ai-readiness-audit: turn an audit into a prioritized, dependency-aware ticket backlog — an interactive effort-profit graph (backlog.html), one Jira-style ticket file per work item, and org-mode aggregation across a portfolio. Implements the "Nudge SDLC automation" item from the AppDev Practice Effort-Profit matrix.Spec:
docs/specs/2026-07-15-audit-improvement-backlog-design.md(in this PR).How it works
/awos:ai-readiness-audit generate <freeform request>consumes an existing audit (picker; headless default = newest). Combined intent (audit and generate improvement backlog) runs a fresh audit and flows straight into generation.tickets-draft.json(titles, business-language goals, effort d/dev, dependencies, per-check coverageshares); the new engine verbgenerate-backlogvalidates everything (check ids, Σshare ≤ 1 per check, dependency cycles, provenance) and computes every number — coverage deltas fromweight_max − weight_awarded, topologicalA00N-slugs, org aggregation. The model never hand-computes a score; unstamped inputs are refused (same circuit-breaker aspatch-judgment).backlog.html: sticky ribbon (devs input with Amdahl-law sublinear duration scaling + always-visible warning, live formula tooltips), vertical dependency graph with click-to-disable subtree cascade, collapsible legend, Provectus report styling. Org variant shows per-repo aggregation tables, repo spread ("3/8 repositories"), and links to per-repo graphs./awos:spec-convertible (footer note); spec generation itself is out of scope.--generateflag runs the generate phase headlessly after the audit and gates compliance (realgenerate-backloginvocation + stamped backlog + tickets on disk, org-level backlog required).Validation
Notes
audit_core.ts,detectors/,metrics/) untouched; the verb only reads scored artifacts and never writesaudit.json.dist/rebuilt and committed.Summary by CodeRabbit
backlog.html, engine-stampedbacklog.json, and per-ticket Markdown (single-repo) plus org rollups with cross-repo linking (combined runs).generate-backlogCLI command to build backlogs from audit inputs and draft JSON.--generateand--generate-only.