Skip to content

RLCR: add a convergence/stopping rule and tiered review surface (run overran cap 5 to 9; post-acceptance effort all on non-deliverable tooling) #195

@chang-l

Description

@chang-l

Retrospective on one RLCR run: 1 execution round plus 9 review rounds vs a configured max of 5. The deliverable headline conclusion was correct and fully evidenced in round 0 and never changed; the first review round returned a terminal accept verdict. The loop still ran 8 more rounds, each fixing real-but-minor correctness issues in auxiliary, non-deliverable tooling. Fully sanitized retrospective (no project specifics).

Findings

  1. No effective stopping rule after the deliverable converged. The accept verdict was not latched; every later round restated the mainline was unchanged.
  2. Review effort inversely correlated with importance. Post-acceptance findings were all in supporting/throwaway tooling, never the deliverable.
  3. Recurring bug CLASS patched instance-by-instance across 3 rounds before a root/class-level fix; tails (stale docs, helper off-by-one, un-routed scripts) kept recurring after.
  4. Most consequential finding surfaced last: a tool printing a verdict contradicting the documented headline appeared only at round 8 of 9; review was not severity/consistency-first.
  5. Iteration cap overrun (9 vs 5) with no proportionality check.
  6. Loop could not honor a stop decision: cancellation was operator-only/out-of-band, so requested stops still produced more rounds.
  7. Throwaway exploration lived in the review surface; rounds were spent deciding what was even reviewable and fixing superseded scripts.
  8. First-round verdict conflated substantive vs housekeeping completeness (a whole round on bookkeeping).
  9. Plan-to-execution: strong on mainline (never drifted), but budget migrated to supporting artifacts with no scope-exhaustion trigger.

Highest-leverage changes

  1. Latch the accept verdict: after convergence, stop by default or require explicit opt-in into a separately-budgeted hardening mode.
  2. Tier the review surface: quarantine throwaway exploration; weight findings by artifact tier.
  3. Severity- and consistency-first review: hunt artifact-vs-headline contradictions first; escalate a recurring finding to a class-level sweep immediately.
  4. Enforce the cap with a proportionality gate plus an agent-honorable park/drain stop state so a requested cancellation takes effect immediately.
  5. Split the readiness verdict into substantive vs housekeeping axes.

What worked

Evidence discipline (early correct conclusion, defended by direct measurement, never drifted); clear auditable round summaries; specific, actionable findings with near-zero false positives; a reused lessons log.

(Filed from an automated, fully-sanitized RLCR methodology retrospective.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions