feat: add automated skill evaluation workflow and reporting#115

Open

yongsinp wants to merge 133 commits into

uw-ssec:mainfrom

Commits on Apr 20, 2026

Commits on Apr 21, 2026

Commits on Apr 27, 2026

feat(ci): fetch all available models and run evaluation for each model
yongsinp
committed
fix(ci): point skill-eval-action to forked repository for testing
yongsinp
committed
fix(ci): hardcode test models instead of fetching from LiteLLM proxy
yongsinp
committed
refactor(ci): move skill and model name to top
yongsinp
committed
feat(ci): add cross-model summary
yongsinp
committed
refactor(ci): split review and evaluate into separate jobs and consolidate summary
yongsinp
committed
chore(ci): bump artifact actions to node 24 compatible versions
yongsinp
committed
feat(ci): suppress redundant per-job eval summaries
yongsinp
committed
style(ci): reorganize summary layout with eval results table and compact model headings
yongsinp
committed
feat(ci): add token usage to eval results summary table
yongsinp
committed
style(ci): add chart emoji to eval results section header
yongsinp
committed
fix(ci): add pass/fail emoji to eval results table rows
yongsinp
committed
refactor(ci): define pass threshold once at workflow level
yongsinp
committed
fix(evaluate-all-skills): read eval outputs via step context instead of GITHUB_OUTPUT
yongsinp
committed
feat(evaluate-all-skills): filter evaluate matrix to skills with evals only
yongsinp
committed
style(ci): rename tokens column to "Tokens Used" in eval summary
yongsinp
committed
feat(ci): add baseline mode to measure value-add of the skills
yongsinp
committed
style(ci): use pp (percentage points) for the delta column
yongsinp
committed
style(ci): replace empty table with plain message when no evals exist
yongsinp
committed
feat(ci): re-add per-case breakdown table under each model in summary
yongsinp
committed
feat(ci): add baseline per-case breakdown table under each model heading
yongsinp
committed
style(ci): remove emoji from per-model heading
yongsinp
committed
perf(ci): batch review jobs into chunks of 8 to reduce runner overhead
yongsinp
committed
Revert "perf(ci): batch review jobs into chunks of 8 to reduce runner overhead"
yongsinp
committed

Commits on May 2, 2026

Commits on May 5, 2026

Commits on May 18, 2026

feat(ci): add configurable sleep before evaluation to avoid rate limits
yongsinp
committed

Commits on May 22, 2026

ci: change sleep duration to 5 mins
yongsinp
committed

Commits on May 26, 2026

Commits on May 29, 2026

fix(ci): fix hardcoded testcases repo
yongsinp
committed

Commits on May 31, 2026

refactor(ci): move sleep to after evaluation
yongsinp
committed

Commits on Jun 2, 2026