Skip to content

Add mitata benchmark for recommended config linting#2632

Merged
NullVoxPopuli merged 8 commits intoember-cli:masterfrom
NullVoxPopuli-ai-agent:nvp/add-benchmark
Mar 24, 2026
Merged

Add mitata benchmark for recommended config linting#2632
NullVoxPopuli merged 8 commits intoember-cli:masterfrom
NullVoxPopuli-ai-agent:nvp/add-benchmark

Conversation

@NullVoxPopuli-ai-agent
Copy link
Copy Markdown

Summary

  • Adds a benchmark suite using mitata that measures linting performance of the recommended config across small/medium/large .js, .gjs, and .gts fixture files
  • Includes a CI workflow that runs on every PR, comparing the branch against master and posting a performance comparison comment with summary table and boxplots
  • Provides local scripts (pnpm bench, pnpm bench:compare, pnpm bench:summary) for running benchmarks during development

Modeled after the benchmark infrastructure in ember-eslint-parser.

New files

Path Purpose
tests/lint.bench.mjs Main benchmark script using ESLint's Linter API + mitata
tests/bench/*.{js,gjs,gts} Fixture files (small/medium/large) with realistic Ember code
scripts/bench-compare.mjs Orchestrates control vs experiment comparison via git archive
scripts/bench-utils.mjs Shared formatting utilities
scripts/format-bench-comment.mjs Generates GitHub PR comment markdown
scripts/format-bench-cli.mjs CLI-friendly summary table
scripts/run-bench.sh CPU-pinned wrapper for local runs
scripts/local-bench-summary.sh Full local comparison with CPU tuning tips
.github/workflows/bench-compare.yml CI workflow for automatic PR benchmark comments

Test plan

  • pnpm bench runs successfully and produces output
  • Verify pnpm bench:compare works against master
  • Open a PR and confirm the CI workflow posts a benchmark comment

🤖 Generated with Claude Code

@NullVoxPopuli
Copy link
Copy Markdown
Contributor

looking good so far:


━━━ Summary ━━━

   Benchmark    Control (p50)   Experiment (p50)       Δ
   ──────────   ─────────────   ────────────────   ─────
⚪ js small           4.00 ms            3.95 ms   -1.3%
⚪ js medium          2.00 ms            2.02 ms   +1.0%
⚪ js large         822.04 µs          820.92 µs   -0.1%
⚪ gjs small        466.50 µs          461.96 µs   -1.0%
⚪ gjs medium       229.13 µs          230.92 µs   +0.8%
⚪ gjs large         92.67 µs           92.63 µs   -0.0%
⚪ gts small        456.54 µs          459.54 µs   +0.7%
⚪ gts medium       231.63 µs          229.13 µs   -1.1%
⚪ gts large         91.33 µs           92.08 µs   +0.8%

🟢 faster · 🔴 slower · 🟠 slightly slower · ⚪ within 2%

━━━ Tips for more reliable results ━━━

NullVoxPopuli and others added 7 commits March 24, 2026 12:57
Introduces a benchmark suite that measures how fast the recommended config
lints files of various sizes (.js, .gjs, .gts), with automatic PR comparison
comments so performance regressions are visible on every PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Fixes bench:compare failure caused by pnpm lockfile config mismatch
when .npmrc (which sets auto-install-peers=false) was missing.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The frozen lockfile install can fail when the control branch's lockfile
was generated with different pnpm settings. Fall back gracefully since
exact reproducibility of the control's node_modules is less critical
than being able to run the benchmark at all.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Medium fixtures: ~400-470 lines (was ~85-110)
Large fixtures: ~690-980 lines (was ~180-335)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The bench fixtures are plain Ember source files, not test files.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The bench fixtures are Ember app source files (ES modules) that aren't
meant to be linted by the project's own eslint config (which uses
sourceType: 'script').

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Move bench scripts to alphabetical position in package.json
  (required by lint:package-json)
- Set cwd on the type-aware linting test's ESLint instance so its
  **/*.gts glob doesn't accidentally pick up bench fixture files

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants