[CI] Add nightly build/release and test workflow#2950
Conversation
Add a scheduled GitHub Actions workflow that runs every night at 00:00 Beijing Time (16:00 UTC) with the following capabilities: - Nightly wheel builds for 3 variants: - CUDA 12.8 x86_64 (with EP support) - CUDA 12.8 arm64 (SBSA) - CUDA 13 x86_64 (with EP support) - Publishes dev wheels to TestPyPI - Comprehensive nightly test suite: - CTest unit tests (Release mode) - Rust smoke tests and benchmarks - Go binding integration tests - Full Python integration tests with optional tests enabled - Failure notification via GitHub issue creation - Manual trigger support with skip_publish option Requires TESTPYPI_API_TOKEN secret to be configured for publishing.
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
|
@qoder Review it |
|
On it — I’ll review this PR and follow up here with findings. 🤖 Generated by Qoder • View workflow run |
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed .github/workflows/nightly.yml for CI/YAML correctness and alignment with existing build/test patterns.
Summary:
- The workflow is structurally valid and matches the described nightly build + test behavior, including CUDA variants, TestPyPI publishing, and full integration suites.
- Permissions, concurrency, and failure notification are sensibly configured.
- Left a few minor, non-blocking suggestions around guarding
github.event.inputs, clarifying use ofnightly_version, and considering CI resource usage trade-offs.
From a CI design perspective I don’t see blockers to merging; my comments are mostly about robustness and maintainability over time. ✅
🤖 Generated by Qoder • View workflow run
| publish-testpypi: | ||
| needs: [version-stamp, build-cuda128-x86, build-cuda128-arm64, build-cuda13-x86] | ||
| if: ${{ github.event.inputs.skip_publish != 'true' }} |
There was a problem hiding this comment.
Thanks for the PR — this looks well-structured and aligned with the existing CI patterns.
A few minor suggestions:
-
publish-testpypiconditional:- Consider guarding against the
scheduleevent (wheregithub.event.inputsis undefined) to avoid surprises if the workflow is ever triggered in other ways. For example, using a boolean input likedo_publishwith a default, or checking for the presence ofgithub.event_name == 'workflow_dispatch'when accessing inputs.
- Consider guarding against the
-
Artifacts and versioning:
- You compute
nightly_versioninversion-stampbut currently only log it; if you intend to embed this into wheel metadata or artifact names, a brief inline comment or follow-up usage would help future maintainers. If logging only is intentional, no change needed.
- You compute
-
Resource usage on
nightly-test:- The
Free up disk spaceand CUDA install steps are quite heavy (which is probably necessary), but if you hit CI limits in practice, consider a separate lighter-weight workflow for smoke tests vs. full nightly to keep overall runtime predictable.
- The
Overall, functionally this looks solid and I don’t see blockers from a CI/YAML correctness perspective. ✅
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
Pull request overview
Adds a new scheduled GitHub Actions workflow to run nightly builds, publish wheels to TestPyPI, execute a broad Release-mode test suite (CTest + Rust/Go/Python integration), and create an issue on failure.
Changes:
- Introduces
.github/workflows/nightly.ymlwith a daily schedule (16:00 UTC / 00:00 Beijing) andworkflow_dispatchsupport. - Reuses the existing reusable wheel build workflow to build multiple CUDA wheel variants and publishes artifacts to TestPyPI.
- Runs a full nightly integration pipeline (etcd + metadata server + CTest + Rust/Go/Python tests) and files an issue when the nightly gate fails.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| version-stamp: | ||
| runs-on: ubuntu-22.04 | ||
| outputs: | ||
| nightly_version: ${{ steps.version.outputs.nightly_version }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Compute nightly version | ||
| id: version | ||
| run: | | ||
| BASE_VERSION=$(grep -Po '(?<=^version = ")[^"]+' mooncake-wheel/pyproject.toml) | ||
| NIGHTLY_VERSION="${BASE_VERSION}.dev$(date -u +%Y%m%d)" | ||
| echo "nightly_version=$NIGHTLY_VERSION" >> "$GITHUB_OUTPUT" |
| etcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 & | ||
| sleep 3 | ||
| etcdctl --endpoints=http://127.0.0.1:2379 endpoint health |
| MC_METADATA_SERVER=http://127.0.0.1:8080/metadata \ | ||
| DEFAULT_KV_LEASE_TTL=500 \ | ||
| ctest -j --output-on-failure |
| - name: Install twine | ||
| run: pip install twine | ||
|
|
||
| - name: Validate wheels | ||
| run: twine check dist-publish/*.whl |
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| steps: |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Aionw
left a comment
Aionw
left a comment
There was a problem hiding this comment.
Two additional issues in the failure-reporting path:
|
|
||
| nightly-gate: | ||
| if: always() | ||
| needs: [publish-testpypi, nightly-test] |
There was a problem hiding this comment.
Build failures can bypass this gate. Because publish-testpypi depends on all three wheel-build jobs, a failure in any build makes the publish job skipped. This gate does not depend on the build jobs directly and only rejects a publish result of failure, so it can report success and notify-failure will not run. Please include the build jobs in needs and validate their results, while distinguishing an intentional skip_publish from a dependency-induced skip.
| repo: context.repo.repo, | ||
| title, | ||
| body, | ||
| labels: ['nightly-failure'], |
There was a problem hiding this comment.
The repository does not currently have a nightly-failure label. Passing a nonexistent label to the create-issue API can cause the notification step itself to fail with a validation error. Please create the label before enabling this workflow, use an existing label, or add a fallback that creates the issue without the label.
Description
Add a scheduled GitHub Actions workflow (
.github/workflows/nightly.yml) that provides daily automated builds, releases, and comprehensive testing.Module(s) affected
Type of change
Schedule
workflow_dispatchWorkflow Jobs
Build (Nightly Wheels)
Publish
skip_publishworkflow inputtwine checkbefore uploadNightly Test (Full Suite)
mooncake-store/rust)mooncake-store/go)scripts/run_tests.shFailure Notification
nightly-failureRequired Secrets
TESTPYPI_API_TOKEN: API token for TestPyPI publishing (must be configured in repo settings)Checklist
_build-wheel.yamlfor consistent builds