Skip to content

fix(fixtures): scorer regex accepts digits + first validated baselines (6 scenarios, avg 99/100)#17

Merged
ptengelmann merged 1 commit into
mainfrom
fix/scorer-regex-and-bench-baselines
Jun 27, 2026
Merged

fix(fixtures): scorer regex accepts digits + first validated baselines (6 scenarios, avg 99/100)#17
ptengelmann merged 1 commit into
mainfrom
fix/scorer-regex-and-bench-baselines

Conversation

@ptengelmann

Copy link
Copy Markdown
Owner

Summary

Real-API bench run across the five new scenarios from #15, plus a regex bug fix the run surfaced, plus the first validated marketing claims that survive the truth-check.

Bench results

All six scenarios scored EXCELLENT (90+):

Scenario Score Signal Code Unexpected
freehold-enforcement-and-undisclosed-occupier 97.0 4/4 5/5 1
freehold-clean-with-satisfied-restriction 97.0 3/3 1/1 1
freehold-resolved-boundary-dispute 100.0 2/2 1/1 0
freehold-missing-building-regs-cert 100.0 3/3 2/2 0
freehold-disclosure-inconsistency-flooding 100.0 4/4 4/4 0
leasehold-short-lease-and-escalation 100.0 5/5 5/5 0

Average: 99.0 / 100. 21/21 signals across all six. 18/18 codes across all six. 0/8 over-flags across the three scenarios that have antipatterns.

The two scenarios at 97 lose 3 points each to a single "unexpected" code emission: PLANNING_BUILDING_REGS_MISSING (defensible inference from the enforcement notice) and TITLE_RESTRICTION_CONSENT (model emitted the code for the Form A restriction but correctly placed it at LOW severity, so the code emission is observation not over-flagging).

Scorer regex bug discovered + fixed

The leasehold scenario initially reported 4/5 codes hit. Investigation showed the pipeline emitted the right code ([CODE: LEASE_SECTION_20_MAJOR_WORKS]) in the right place. The scorer's extractEmittedIssueCodes() used [A-Z_]+ which silently dropped any code containing a digit - so LEASE_SECTION_20_MAJOR_WORKS (with "20") was invisible. Fixed by adding 0-9 to the character class. Leasehold scenario re-scored at 100/100 after the fix.

What this changes

  • packages/fixtures/src/score.ts: regex character class [A-Z_]+ -> [A-Z0-9_]+ in extractEmittedIssueCodes. With a comment explaining why so the same bug doesn't get reintroduced.
  • packages/fixtures/BASELINES.md: six validated baselines (refreshed enforcement scenario + five new ones) with metrics, regression rules, and a summary table.
  • apps/marketing/src/sections/trust.tsx: "Severity calibrated" card body now states "Validated against six adversarial conveyancing scenarios with an average Matter Quality Score of 99 out of 100" (both numbers grep-verifiable in BASELINES.md).
  • apps/marketing/src/sections/faq.tsx: wrapper answer now references the six validated scenarios with the concrete average score and the "every planted issue detected / zero over-flagging" claim.

Test plan

  • pnpm typecheck - 9/9 workspaces pass
  • pnpm check - 0 errors, 3 pre-existing warnings
  • pnpm build - 3/3 tasks pass
  • Real-API bench run on all five new scenarios + re-score of existing scenario
  • All six scenarios verdict EXCELLENT
  • No new em-dashes; no §; UK English throughout

Cost of this PR

~$4.50 of Anthropic API calls (5 fresh scenario benches at ~$0.90 each).

Honest gaps in the corpus

  • 3 legacy scenarios (leasehold-flat-with-issues, freehold-house-clean, freehold-house-edge-cases) have NOT been re-benched in this session. They are in the scenarios registry but not in BASELINES.md. Status: TODO for a follow-up run.
  • 5 of 21 taxonomy codes are still not exercised by any scenario (TITLE_OVERRIDING_INTERESTS, TITLE_DEFECTIVE_INDEMNITY_REQUIRED, LEASE_ALTERATIONS_CONSENT, SEARCH_CONTAMINATED_LAND, OCCUPIER_TENANCY_UNDISCLOSED, MORTGAGE_OFFER_EXPIRY_RISK). Growth should follow pilot patterns, not speculation.

Brand and liability check

  • All marketing claims verifiable in BASELINES.md
  • No "the product is proven" claim - the language is "validated against six scenarios", not "proven across all conveyancing"
  • UK English, no emoji, no hype

Moat level

L5 / L7 - the proprietary bench corpus crosses from 1 measured to 6 measured, and the marketing now has measured numbers behind it for the first time. Each baseline metric is a regression guard for future prompt or model changes.

…s for 6 scenarios

Two related changes from a real-API bench run across the five new
scenarios added in PR #15:

1. Scorer regex bug fix. extractEmittedIssueCodes() used [A-Z_]+ as the
   character class for code names, which silently dropped any code with
   a digit. LEASE_SECTION_20_MAJOR_WORKS contains "20" and was therefore
   invisible to the scorer, producing a false "expected code missing"
   failure on the leasehold scenario. Fixed by adding 0-9 to the class.
   The leasehold scenario's Matter Quality Score moves from 94.0 -> 100
   purely from this fix.

2. First validated baselines for the five new scenarios plus a refreshed
   baseline for freehold-enforcement-and-undisclosed-occupier. All six
   captured in packages/fixtures/BASELINES.md with full metrics and
   regression rules. Summary:

     freehold-enforcement-and-undisclosed-occupier   97.0 / 100
     freehold-clean-with-satisfied-restriction       97.0 / 100
     freehold-resolved-boundary-dispute             100.0 / 100
     freehold-missing-building-regs-cert            100.0 / 100
     freehold-disclosure-inconsistency-flooding     100.0 / 100
     leasehold-short-lease-and-escalation           100.0 / 100

     Average: 99.0 / 100
     Pattern-signal hit rate across all six:  21 / 21 (100%)
     Taxonomy code hit rate across all six:   18 / 18 (100%)
     Over-flagging across the 3 scenarios with antipatterns: 0 / 8

3. Marketing updates reflecting validated truth:
   - Trust card "Severity calibrated" now states the corpus size, code
     count, and the average score with the actual number.
   - FAQ wrapper now references the six validated scenarios with the
     concrete average score and the "every planted issue detected /
     zero over-flagging" claim.

   Both numbers (six scenarios, 99/100 average) are verifiable in the
   BASELINES.md table. If the bench gets re-run and the average shifts
   beyond a small margin, the marketing copy must be re-validated.

Refs: GPT external review of v2 bench, "Matter Quality Score" + scenario
breadth; feedback_marketing_must_reflect_truth.md.
@ptengelmann
ptengelmann merged commit 8b4509e into main Jun 27, 2026
1 check passed
@ptengelmann
ptengelmann deleted the fix/scorer-regex-and-bench-baselines branch June 27, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants