feat: add public-safe story regression lane (#1701)#1712
Open
prompt-driven-github[bot] wants to merge 1 commit into
Open
feat: add public-safe story regression lane (#1701)#1712prompt-driven-github[bot] wants to merge 1 commit into
prompt-driven-github[bot] wants to merge 1 commit into
Conversation
Make the story-backed regression tests a first-class CI lane: - Makefile: new `regression-stories` target running `pytest -m story` via the offline tests/story_regression.sh harness (+ .PHONY/help). - tests/story_regression.sh: deterministic/offline prelude (scratch HOME, stripped credentials, PDD_RUN_LLM_TESTS=0); auto-discovers story-marked tests; treats "no tests collected" (exit 5) as a green no-op lane. - pytest.ini: register the `story` marker. - tests/conftest.py: per-story pass/fail summary plugin (pytest_runtest_makereport tally + pytest_terminal_summary). - .github/workflows/unit-tests.yml: non-blocking `story-regression` PR job (draft guard, timeout, no secrets). - docs/ci.md, CONTRIBUTING.md: document the lane. Closes #1701 Co-Authored-By: Claude Opus 4 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a first-class, public-safe story regression lane that runs
pytest -m storywith a deterministic/offline harness — locally viamake regression-storiesand in CI as a non-blockingstory-regressionPR job that needs no LLM/cloud credentials. Story-marked tests are auto-discovered (no manual registration) and a per-story pass/fail summary is reported.Closes #1701
Changes Made
This issue is build/CI/test-harness infrastructure. Per the Step 8 analysis, no
.promptfiles were modified: the only prompt-backed candidate (prompts/regression_bash.prompt) generates a credential-requiring live-LLM exerciser — the opposite of this offline lane — and the liveMakefileis hand-maintained (its staleprompts/Makefile_makefile.promptdoes not back the real targets). All edits are additive and direct.Build / Harness
Makefile— new.PHONY: regression-storiestarget running the offlinetests/story_regression.sh; added tohelp.tests/story_regression.sh(new) — public-safe prelude (scratchHOME, stripped API/cloud credentials,PDD_RUN_LLM_TESTS=0); runspytest -m storysingle-process; treats "no tests collected" (exit 5) as a green no-op so the lane can't silently false-green during the story backfill window.pytest.ini— register thestorymarker.tests/conftest.py— per-story summary plugin (pytest_runtest_makereporttally +pytest_terminal_summary) reporting story count, total story tests, and per-story PASS/FAIL. Renders only when story tests actually ran, so the normal unit run is unaffected.CI
.github/workflows/unit-tests.yml— non-blockingstory-regressionPR job modeled onpublic-cli-regression(draft-PR guard,timeout-minutes: 20, no secrets) runningmake regression-stories.Documentation Updated
docs/ci.md— documentstory-regressionas the third default PR job; broaden the fork-safety statement to covermake regression-stories.CONTRIBUTING.md— add a "Story Regression Lane" subsection.Review Checklist
Makefileuntouched at the prompt layer)make regression-storiesrunspytest -m storyoffline with no API keysstory-regressionjob is non-blocking and credential-freeNext Steps After Merge
.promptchanges) —./sync_order.shis a no-op for this PR.make regression-storiesto verify the lane is green.storymarker; the merge-blocking coverage gate is tracked separately (CI gate: every user story must have a passing, non-stale regression test #1702) and is out of scope here.Created by pdd change workflow