GH-3547: Add semi-automated release pipeline for Apache Parquet Java#3548
GH-3547: Add semi-automated release pipeline for Apache Parquet Java#3548RussellSpitzer wants to merge 6 commits intoapache:masterfrom
Conversation
| paths: | ||
| - 'release/**' | ||
|
|
||
| jobs: |
There was a problem hiding this comment.
We can rip out all of these tests if folks don't find them useful, it's just a bunch of Bash Unit Testing to try to make sure at least our helper functions do what they are supposed to. They use mocked output though so they have limited utility for testing the real true pathway.
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 |
There was a problem hiding this comment.
Parquet isn't using zimor yet, but in the future we should switch this and all other actions to hard coded sha's
|
|
||
| - name: Cancel Release Candidate | ||
| env: | ||
| DRY_RUN: ${{ inputs.dry_run && '1' || '0' }} |
There was a problem hiding this comment.
anything in "secrets" is automatically redacted by Github Actions so we don't have to worry about any of this stuff being exposed
f722749 to
967152b
Compare
… Java Adds a release automation framework modeled after Apache Polaris, adapted for Parquet's Maven-based build. Replaces the manual maven-release-plugin workflow with explicit, scriptable steps that support both CI (GitHub Actions) and local execution, with dry-run by default. Scripts (release/bin/): - prepare-rc.sh: full pre-vote flow (branch, version, tag, Nexus, SVN, GitHub pre-release, vote email) - publish-release.sh: full post-vote flow (SVN promotion, final tag, Nexus release, GitHub release, version bump, announce email) - cancel-rc.sh: rollback a failed RC (Nexus drop, SVN cleanup) Shared libraries (release/libs/): - _constants.sh, _log.sh, _exec.sh, _version.sh - _github.sh, _nexus.sh, _maven.sh GitHub Actions workflows: - release-prepare-rc.yml, release-publish.yml, release-cancel-rc.yml - ci-release-scripts.yml (bats unit tests on PR/push) Includes 85 bats unit tests covering all shared libraries.
967152b to
81d2bdd
Compare
Remove the next_dev_version input from publish-release.sh and the workflow. The next version is always the current patch incremented by one (e.g. 1.18.0 -> 1.18.1-SNAPSHOT), since the release branch only produces patches for that major.minor.
| source "$LIBS_DIR/_constants.sh" | ||
| source "$LIBS_DIR/_log.sh" | ||
|
|
||
| function _redact_secrets { |
There was a problem hiding this comment.
This is me being paranoid for local runs, Github will do this automatically but incase someone wants to run this locally and copies and pastes, this will protect their secrets.
The settings.xml now contains ${env.NEXUS_USERNAME} and
${env.NEXUS_PASSWORD} instead of the actual secret values. Maven
resolves these from environment variables at build time, so the
file itself contains no secrets and cannot be exfiltrated.
The glob-based git tag -l "...-rc*" could match malformed tags like "-rc10extra" or "-rc-foo". Add a _filter_rc_tags helper that applies a strict ^...-rc[0-9]+$ regex after the glob, so only well-formed RC tags are considered when auto-detecting RC numbers.
| return 0 | ||
| } | ||
|
|
||
| function _filter_rc_tags { |
There was a problem hiding this comment.
Another paranoia from me, stops us from finding a 1.9.0-rcFoo being considered
Previously, a missing GITHUB_TOKEN silently skipped CI verification and returned success, allowing a release to proceed even if CI was red. Now it fails unless running in dry-run mode.
The find_latest_rc_number tests create temporary git repos and run git commit, which requires user.name and user.email to be set. The GitHub Actions runner has no default git identity.
Includes 85 bats unit tests covering all shared libraries.
Rationale for this change
Adds a release automation framework modeled after Apache Polaris, adapted for Parquet's Maven-based build. Replaces the manual maven-release-plugin workflow with explicit, scriptable steps that support both CI (GitHub Actions) and local execution, with dry-run by default.
What changes are included in this PR?
Scripts (release/bin/):
Shared libraries (release/libs/):
GitHub Actions workflows:
Are these changes tested?
Only locally and with fake commands. To actually make this work we also have to raise an Infra ticket to get appropriate secrets applied to the parquet-java repo.
Are there any user-facing changes?
No, just for contributors
Release Workflow
This PR replaces the manual release process (documented in
How to Release
and the existing
dev/scripts:prepare-release.sh,source-release.sh,finalize-release) with three GitHub Actions workflows backed by locally-runnableBash scripts. All workflows default to dry-run mode.
1. Prepare RC (Pre-Vote)
The release manager launches the "Prepare Release Candidate" workflow
(
release-prepare-rc.yml) viaworkflow_dispatchwith:1.18.0)true; set tofalsefor real executionprepare-rc.shgpg/svn/mvnwgit branch parquet-X.Y.xdev/prepare-release.sh(mvn release:prepare)dev/prepare-release.sh(mvn release:prepare)mvn release:perform(prompted byprepare-release.sh)dev/source-release.sh(git archive,gpg,shasum)dev/source-release.sh(svn co,svn add,svn ci)The script also generates a
[VOTE]email template with all links andhashes. The release manager copies this and manually sends it to
[email protected].2. Vote
The release manager waits 72 hours for the community vote and tallies the
results manually.
If the vote fails → Cancel RC
The release manager launches the "Cancel Release Candidate" workflow
(
release-cancel-rc.yml) viaworkflow_dispatchwith:1.18.0)1)orgapacheparquet-1234)truecancel-rc.shsvn rmThe script also generates a
[RESULT][VOTE]failure email template.The release manager fills in the failure reason and manually sends it.
If the vote passes → Publish Release
The release manager launches the "Publish Release" workflow
(
release-publish.yml) viaworkflow_dispatchwith:1.18.0)orgapacheparquet-1234)-SNAPSHOT(e.g.1.18.1)truepublish-release.shsvn mvfromdist/devtodist/releasesvn rmof prior versionsdev/finalize-release(git tag)dev/finalize-release(mvn release:update-versions,versions:set-property,git commit)3. Manual Follow-ups
After the publish workflow completes, the release manager must:
[ANNOUNCE]email (generated by step 7 ofpublish-release.sh) to[email protected]and[email protected]apache/parquet-siteon thestagingbranch with the blog post template (generated by step 8 ofpublish-release.sh)