Skip to content

feat(fixtures): 9/9 baselined - Track 1 complete (legacy scenarios re-benched)#19

Merged
ptengelmann merged 1 commit into
mainfrom
feat/track-1-legacy-baselines
Jun 27, 2026
Merged

feat(fixtures): 9/9 baselined - Track 1 complete (legacy scenarios re-benched)#19
ptengelmann merged 1 commit into
mainfrom
feat/track-1-legacy-baselines

Conversation

@ptengelmann

Copy link
Copy Markdown
Owner

Summary

Real Anthropic API runs on the three legacy scenarios from the pre-taxonomy era. Gets the corpus to 9/9 baselined per GPT's Track 1.

The 3 legacy results

Scenario Score Verdict Notes
leasehold-flat-with-issues 94.0 EXCELLENT Expected-codes list refreshed (see below)
freehold-house-clean 100.0 EXCELLENT Clean control, no surprises
freehold-house-edge-cases 97.0 EXCELLENT 0/6 over-flag suppression - the adversarial test works

Honest disclosure: leasehold-flat-with-issues expected-codes refresh

Initially scored 82.0/100 with the original 7-code expected list. Inspection: the pipeline correctly emitted 5 additional codes that didn't exist when the scenario was originally written. Specifically the scenario plants issues that map to LEASE_ALTERATIONS_CONSENT, TITLE_CHARGE_DISCHARGE_EVIDENCE, SEARCH_DRAINAGE_DISCREPANCY, and SELLER_DISCLOSURE_INCONSISTENCY - all of which are now in the expected list.

Each addition was verified against the actual scenario source (the seeded issue genuinely exists in the pack). Not overfitting; the list was simply stale. The only remaining unexpected code is PLANNING_BREACH_ENFORCEMENT - a model emission that may be borderline (the pack has an unauthorised kitchen extension, but no CON29 enforcement notice). Flagged as a regression-rule watch item.

After the refresh: 94.0/100 EXCELLENT.

Corpus state after this PR

Metric Value
Scenarios baselined 9 / 9
Pattern signals across all 9 40 / 40 (100%)
Expected codes across all 9 33 / 33 (100%)
Over-flagging (across 3 antipattern scenarios) 0 / 8
Average Matter Quality Score 98.3 / 100
Codes exercised by expected-codes 17 / 21

What this changes

  • packages/fixtures/src/scenarios/expected-codes.ts: leasehold-flat-with-issues list refreshed (7 -> 11 codes), with a comment explaining the staleness.
  • packages/fixtures/BASELINES.md: three new baseline entries (94, 100, 97), refreshed summary table, refreshed honest-gaps section. "3 legacy scenarios not re-benched" line removed (resolved).
  • apps/marketing/src/sections/trust.tsx: "six" -> "nine" scenarios, "99" -> "98" average score.
  • apps/marketing/src/sections/faq.tsx: same numeric updates, two-sentence framing preserved.

Both numbers (9, 98) remain grep-verifiable in BASELINES.md.

Test plan

  • pnpm typecheck - 9/9 workspaces pass
  • pnpm check - 0 errors, 3 pre-existing warnings
  • pnpm build - 3/3 tasks pass
  • Three real-API bench runs at ~$0.80 each
  • Re-score of leasehold scenario after list refresh produces 94.0/100
  • No new em-dashes; no §; UK English throughout

Cost of this PR

~$2.50 of Anthropic API calls.

Honest disclosure

  • PLANNING_BREACH_ENFORCEMENT was emitted on leasehold-flat-with-issues despite the pack having no enforcement notice. The model appears to have conflated "unauthorised works" with "enforcement breach". Not a critical failure (it's the only unexpected code and the over-flag detection is n/a for this scenario), but worth watching if it recurs.
  • PLANNING_TA6_SEARCH_DISCREPANCY was emitted on freehold-house-edge-cases for the unexercised 2024 permission. Borderline emission; not over-flagging.
  • The corpus is now 9 internal benchmark scenarios. This is NOT a claim that the product is proven across the market - it is a claim that the pipeline behaves correctly on the specific seeded issues across these specific packs.

Brand and liability check

Moat level

L5 / L7 - the proprietary bench corpus is now fully baselined for the first time. Marketing has a defensible composite claim. The next strategic move per GPT is one real conveyancer reviewing one real-looking pack; Track 3 (a /validation page) follows in a separate PR to support that conversation.

…-benched)

Real Anthropic API runs on the three legacy scenarios from the
pre-taxonomy era, getting the corpus to 9 / 9 baselined per the
GPT review's Track 1.

Three legacy results:

  leasehold-flat-with-issues         94.0 / 100   EXCELLENT
  freehold-house-clean              100.0 / 100   EXCELLENT
  freehold-house-edge-cases          97.0 / 100   EXCELLENT

The leasehold scenario initially scored 82.0 with a stale expected-codes
list (it pre-dated SELLER_DISCLOSURE_INCONSISTENCY, the
LEASE_ALTERATIONS_CONSENT split, and SEARCH_DRAINAGE_DISCREPANCY
emission for the build-over agreement issue). The pipeline correctly
emitted those codes; the list just had not kept up. Refreshed the list
honestly (every added code corresponds to an issue actually planted in
the pack, verified against the scenario source) and re-scored: 94.0
EXCELLENT.

Corpus summary across all 9 scenarios:
  Pattern signals:    40 / 40 (100%)
  Expected codes:     33 / 33 (100%)
  Over-flagging:       0 / 8  (across the 3 scenarios with antipatterns)
  Average score:      98.3 / 100

Marketing copy updated to match:
  - Trust card: "six" -> "nine" scenarios, "99" -> "98" average score.
  - FAQ wrapper: same numeric updates, same two-sentence framing.

Both numbers (9, 98) remain grep-verifiable in BASELINES.md.

Honest gaps section refreshed:
  - Removed the "3 legacy scenarios not re-benched" item (now resolved).
  - Uncovered-codes count updated: 17 of 21 exercised across the corpus,
    4 strictly uncovered (TITLE_OVERRIDING_INTERESTS,
    TITLE_DEFECTIVE_INDEMNITY_REQUIRED, SEARCH_CONTAMINATED_LAND,
    MORTGAGE_OFFER_EXPIRY_RISK) plus 3 adjacent gaps documented.

Per the GPT review, the next milestone is not more synthetic scenarios.
It is one real conveyancer reviewing one real-looking pack. Track 3
(a /validation marketing page) follows in a separate PR.

Cost: ~$2.50 of Anthropic API calls (3 scenarios at ~$0.80 each).

Refs: GPT external review of v3 bench, Track 1; feedback_marketing_
must_reflect_truth.md.
@ptengelmann
ptengelmann merged commit 5b4547c into main Jun 27, 2026
1 check passed
@ptengelmann
ptengelmann deleted the feat/track-1-legacy-baselines branch June 27, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants