Skip to content

Commit 181fe2e

Browse files
dcramerclaudeCopilot
authored
feat: add StructuredOutputScorer for validating JSON outputs (#23)
## Summary - Adds a new `StructuredOutputScorer` for evaluating structured JSON outputs from language models - Supports both strict and fuzzy matching modes for flexible validation - Includes comprehensive test coverage with 483 lines of tests ## Changes - **New scorer implementation**: `src/scorers/structuredOutputScorer.ts` (362 lines) - Validates JSON structure and field values - Supports strict equality checking and fuzzy matching - Handles nested objects, arrays, and complex data structures - Configurable matching modes and validation options - **Comprehensive test suite**: `src/scorers/structuredOutputScorer.test.ts` (483 lines) - Tests strict and fuzzy matching modes - Covers edge cases like null values, empty objects, and arrays - Tests partial matching and extra field handling - Validates custom matcher functionality - **Integration updates**: - Exported from `src/scorers/index.ts` - Added to main exports in `src/index.ts` ## Features - **Matching modes**: - `strict`: Exact equality required (default) - `fuzzy`: Case-insensitive strings, numeric tolerance, regex patterns, subset matching - Custom function support for specialized validation logic - **Configuration options**: - `requireAll`: Whether all expected fields must be present - `allowExtras`: Whether to allow additional fields beyond expected - `debug`: Enable detailed logging for troubleshooting ## Type of change - ✨ New feature (non-breaking change which adds functionality) ## Testing - [x] Comprehensive unit tests added (100% coverage of new code) - [x] All existing tests pass - [x] TypeScript type checking passes - [x] Linting passes ## Related issues - N/A 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Copilot <[email protected]>
1 parent c0945ef commit 181fe2e

13 files changed

Lines changed: 4766 additions & 546 deletions

.gitignore

Lines changed: 167 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,176 @@
1-
/node_modules
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
yarn-debug.log*
6+
yarn-error.log*
7+
lerna-debug.log*
8+
pnpm-debug.log*
9+
10+
# Diagnostic reports (https://nodejs.org/api/report.html)
11+
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
12+
13+
# Runtime data
14+
pids
15+
*.pid
16+
*.seed
17+
*.pid.lock
218

3-
# Ignore test-related files
19+
# Directory for instrumented libs generated by jscoverage/JSCover
20+
lib-cov
21+
22+
# Coverage directory used by tools like istanbul
23+
coverage
24+
*.lcov
425
/coverage.data
526
/coverage/
627

28+
# nyc test coverage
29+
.nyc_output
30+
31+
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
32+
.grunt
33+
34+
# Bower dependency directory (https://bower.io/)
35+
bower_components
36+
37+
# node-waf configuration
38+
.lock-wscript
39+
40+
# Compiled binary addons (https://nodejs.org/api/addons.html)
41+
build/Release
42+
43+
# Dependency directories
44+
/node_modules
45+
jspm_packages/
46+
47+
# Snowpack dependency directory (https://snowpack.dev/)
48+
web_modules/
49+
50+
# TypeScript cache
51+
*.tsbuildinfo
52+
53+
# Optional npm cache directory
54+
.npm
55+
56+
# Optional eslint cache
57+
.eslintcache
58+
59+
# Optional stylelint cache
60+
.stylelintcache
61+
62+
# Optional REPL history
63+
.node_repl_history
64+
65+
# Output of 'npm pack'
66+
*.tgz
67+
68+
# Yarn Integrity file
69+
.yarn-integrity
70+
71+
# dotenv environment variable files
72+
.env
73+
.env.*
74+
!.env.example
75+
76+
# parcel-bundler cache (https://parceljs.org/)
77+
.cache
78+
.parcel-cache
79+
80+
# Next.js build output
81+
.next
82+
out
83+
84+
# Nuxt.js build / generate output
85+
.nuxt
86+
dist
87+
788
# Build files
889
/dist
990

91+
# Gatsby files
92+
.cache/
93+
# Comment in the public line in if your project uses Gatsby and not Next.js
94+
# https://nextjs.org/blog/next-9-1#public-directory-support
95+
# public
96+
97+
# vuepress build output
98+
.vuepress/dist
99+
100+
# vuepress v2.x temp and cache directory
101+
.temp
102+
103+
# Sveltekit cache directory
104+
.svelte-kit/
105+
106+
# vitepress build output
107+
**/.vitepress/dist
108+
109+
# vitepress cache directory
110+
**/.vitepress/cache
111+
112+
# Docusaurus cache and generated files
113+
.docusaurus
114+
115+
# Serverless directories
116+
.serverless/
117+
118+
# FuseBox cache
119+
.fusebox/
120+
121+
# DynamoDB Local files
122+
.dynamodb/
123+
124+
# Firebase cache directory
125+
.firebase/
126+
127+
# TernJS port file
128+
.tern-port
129+
130+
# Stores VSCode versions used for testing VSCode extensions
131+
.vscode-test
132+
133+
# yarn v3
134+
.pnp.*
135+
.yarn/*
136+
!.yarn/patches
137+
!.yarn/plugins
138+
!.yarn/releases
139+
!.yarn/sdks
140+
!.yarn/versions
141+
142+
# Vite logs files
143+
vite.config.js.timestamp-*
144+
vite.config.ts.timestamp-*
145+
146+
# Lock files (excluding others, keeping pnpm for this project)
10147
/package-lock.json
11-
/pnpm-lock.yaml
148+
/yarn.lock
12149

13-
.env
150+
# Test results
151+
*.junit.xml
152+
153+
# OS generated files
154+
.DS_Store
155+
.DS_Store?
156+
._*
157+
.Spotlight-V100
158+
.Trashes
159+
ehthumbs.db
160+
Thumbs.db
161+
162+
# IDE files
163+
.vscode/
164+
!.vscode/settings.json
165+
!.vscode/tasks.json
166+
!.vscode/launch.json
167+
!.vscode/extensions.json
168+
.idea/
169+
*.swp
170+
*.swo
171+
*~
172+
173+
# Temporary files
174+
*.tmp
175+
*.temp
176+
temp/

.vscode/settings.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,19 @@
11
{
2-
"editor.formatOnSave": true,
32
"editor.codeActionsOnSave": {
43
"source.fixAll.biome": "explicit"
54
},
5+
"editor.formatOnSave": false,
66
"files.trimTrailingWhitespace": false,
77
"files.trimFinalNewlines": false,
88
"files.insertFinalNewline": true,
99
"cursor.general.enableShadowWorkspace": true,
1010
"[typescript]": {
11+
"editor.formatOnSave": true,
1112
"editor.tabSize": 2,
1213
"editor.defaultFormatter": "biomejs.biome"
1314
},
1415
"[json]": {
16+
"editor.formatOnSave": true,
1517
"editor.tabSize": 2,
1618
"editor.defaultFormatter": "biomejs.biome"
1719
}

0 commit comments

Comments
 (0)