feat(fixtures): 9/9 baselined - Track 1 complete (legacy scenarios re-benched)#19
Merged
Merged
Conversation
…-benched)
Real Anthropic API runs on the three legacy scenarios from the
pre-taxonomy era, getting the corpus to 9 / 9 baselined per the
GPT review's Track 1.
Three legacy results:
leasehold-flat-with-issues 94.0 / 100 EXCELLENT
freehold-house-clean 100.0 / 100 EXCELLENT
freehold-house-edge-cases 97.0 / 100 EXCELLENT
The leasehold scenario initially scored 82.0 with a stale expected-codes
list (it pre-dated SELLER_DISCLOSURE_INCONSISTENCY, the
LEASE_ALTERATIONS_CONSENT split, and SEARCH_DRAINAGE_DISCREPANCY
emission for the build-over agreement issue). The pipeline correctly
emitted those codes; the list just had not kept up. Refreshed the list
honestly (every added code corresponds to an issue actually planted in
the pack, verified against the scenario source) and re-scored: 94.0
EXCELLENT.
Corpus summary across all 9 scenarios:
Pattern signals: 40 / 40 (100%)
Expected codes: 33 / 33 (100%)
Over-flagging: 0 / 8 (across the 3 scenarios with antipatterns)
Average score: 98.3 / 100
Marketing copy updated to match:
- Trust card: "six" -> "nine" scenarios, "99" -> "98" average score.
- FAQ wrapper: same numeric updates, same two-sentence framing.
Both numbers (9, 98) remain grep-verifiable in BASELINES.md.
Honest gaps section refreshed:
- Removed the "3 legacy scenarios not re-benched" item (now resolved).
- Uncovered-codes count updated: 17 of 21 exercised across the corpus,
4 strictly uncovered (TITLE_OVERRIDING_INTERESTS,
TITLE_DEFECTIVE_INDEMNITY_REQUIRED, SEARCH_CONTAMINATED_LAND,
MORTGAGE_OFFER_EXPIRY_RISK) plus 3 adjacent gaps documented.
Per the GPT review, the next milestone is not more synthetic scenarios.
It is one real conveyancer reviewing one real-looking pack. Track 3
(a /validation marketing page) follows in a separate PR.
Cost: ~$2.50 of Anthropic API calls (3 scenarios at ~$0.80 each).
Refs: GPT external review of v3 bench, Track 1; feedback_marketing_
must_reflect_truth.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Real Anthropic API runs on the three legacy scenarios from the pre-taxonomy era. Gets the corpus to 9/9 baselined per GPT's Track 1.
The 3 legacy results
leasehold-flat-with-issuesfreehold-house-cleanfreehold-house-edge-casesHonest disclosure: leasehold-flat-with-issues expected-codes refresh
Initially scored 82.0/100 with the original 7-code expected list. Inspection: the pipeline correctly emitted 5 additional codes that didn't exist when the scenario was originally written. Specifically the scenario plants issues that map to
LEASE_ALTERATIONS_CONSENT,TITLE_CHARGE_DISCHARGE_EVIDENCE,SEARCH_DRAINAGE_DISCREPANCY, andSELLER_DISCLOSURE_INCONSISTENCY- all of which are now in the expected list.Each addition was verified against the actual scenario source (the seeded issue genuinely exists in the pack). Not overfitting; the list was simply stale. The only remaining unexpected code is
PLANNING_BREACH_ENFORCEMENT- a model emission that may be borderline (the pack has an unauthorised kitchen extension, but no CON29 enforcement notice). Flagged as a regression-rule watch item.After the refresh: 94.0/100 EXCELLENT.
Corpus state after this PR
What this changes
packages/fixtures/src/scenarios/expected-codes.ts:leasehold-flat-with-issueslist refreshed (7 -> 11 codes), with a comment explaining the staleness.packages/fixtures/BASELINES.md: three new baseline entries (94, 100, 97), refreshed summary table, refreshed honest-gaps section. "3 legacy scenarios not re-benched" line removed (resolved).apps/marketing/src/sections/trust.tsx: "six" -> "nine" scenarios, "99" -> "98" average score.apps/marketing/src/sections/faq.tsx: same numeric updates, two-sentence framing preserved.Both numbers (9, 98) remain grep-verifiable in BASELINES.md.
Test plan
pnpm typecheck- 9/9 workspaces passpnpm check- 0 errors, 3 pre-existing warningspnpm build- 3/3 tasks passCost of this PR
~$2.50 of Anthropic API calls.
Honest disclosure
PLANNING_BREACH_ENFORCEMENTwas emitted onleasehold-flat-with-issuesdespite the pack having no enforcement notice. The model appears to have conflated "unauthorised works" with "enforcement breach". Not a critical failure (it's the only unexpected code and the over-flag detection is n/a for this scenario), but worth watching if it recurs.PLANNING_TA6_SEARCH_DISCREPANCYwas emitted onfreehold-house-edge-casesfor the unexercised 2024 permission. Borderline emission; not over-flagging.Brand and liability check
Moat level
L5 / L7 - the proprietary bench corpus is now fully baselined for the first time. Marketing has a defensible composite claim. The next strategic move per GPT is one real conveyancer reviewing one real-looking pack; Track 3 (a
/validationpage) follows in a separate PR to support that conversation.