fix(cd): prevent intermittent linux-intel wheel build failure from numpy source build#165
Open
uw-ssec-bot wants to merge 19 commits into
Open
fix(cd): prevent intermittent linux-intel wheel build failure from numpy source build#165uw-ssec-bot wants to merge 19 commits into
uw-ssec-bot wants to merge 19 commits into
Conversation
The worked example is compiled into _core.contributor_example and exercised by tests/test_contributor_example.py so the docs stay current. Gated by HPYX_BUILD_CONTRIBUTOR_EXAMPLE=ON (default in dev).
…oncurrent-futures, hpx-native)
Replaces the pure-Python sorted() stubs with genuine HPX bindings: - src/_core/parallel.cpp: add sort_impl() dispatching to hpx::sort / hpx::stable_sort under hpx::execution::seq (seq/unseq policies) or hpx::execution::par (par/par_unseq policies). Index-sort pattern keeps Python objects out of HPX threads; per-comparison GIL acquire via PyGILState_Ensure/Release makes comparisons safe under the free-threading build. nanobind .none() annotation required for the key parameter to accept Python None. - src/hpyx/parallel.py: dispatch sort/stable_sort to _core.parallel.sort. task-tagged policies submit the sync sort as a single HPX task via async_() and return a Future. - tests/test_parallel.py: add seq/par/par_unseq coverage and task-variant tests (Future return + .result()) for both sort and stable_sort. - benchmarks/test_bench_parallel.py: add sort throughput benchmarks comparing hpyx par, hpyx seq, and pure-Python sorted(). Co-Authored-By: Paperclip <[email protected]>
dispatch_policy's auto return type deduction fails when the lambda only returns a plain value (not hpx::future<T>), causing a C++17 deduction error. Match the kernels.cpp pattern: always use hpx::execution::par directly. Policy params stay in the signature (commented) to document the calling convention; task dispatch is the Python wrapper's job. Update docs/adding-a-binding.md Step 1 to match the actual implementation.
…ObjArgs PyObject_CallOneArg is absent from nanobind's darwin-ld-cpython.sym allowlist, causing a linker error on all macOS arm64 targets. Replace with PyObject_CallFunctionObjArgs (same semantics, in the allowlist).
…form_reduce task
- Add test-py313 environment to pixi.toml
- Expand CI matrix to {ubuntu-latest,macos-14} x {py313,py313t} covering all
four required combinations; each cell runs pixi run -e test-<variant> run-test
- Implement task support for transform_reduce: wraps execution body in async_
and returns a Future, matching the asyncio.md guide example
- Update parallel.py and execution.py module docstrings to reflect that sort,
stable_sort, and transform_reduce support task-tagged policies
Running pixi install to update the lock file after adding the test-py313 environment in the previous commit. Co-Authored-By: Paperclip <[email protected]>
Previous regen used pixi 0.68.1 which produced lock format v7, incompatible with [email protected] (pixi v0.67.2) in CI. This regenerates with the correct version so --locked mode succeeds. Co-Authored-By: Paperclip <[email protected]>
…uture The test previously expected NotImplementedError for task-tagged policies; now that task support is implemented, update it to verify the Future return and correct computed value. Co-Authored-By: Paperclip <[email protected]>
- parallel.cpp: preserve original Python exception in sort_impl
comparator by capturing it with PyErr_GetRaisedException on first
failure and restoring it with PyErr_SetRaisedException after the
sort completes; subsequent failures after the first are cleared;
HPX-internal throws in the catch block now also release saved_exc.
Adds 4 regression tests: sort/stable_sort × seq/par asserting
ValueError("custom comparator error") propagates unchanged.
- test_contributor_example.py: replace vacuous try/except skip with
an explicit opt-out via HPYX_SKIP_CONTRIBUTOR_EXAMPLE env var.
When the variable is unset (normal CI) a missing binding surfaces
as a hard ImportError rather than a silent skip.
- ci.yml: move CMAKE_BUILD_TYPE=Debug from the pytest invocation
into a step-level env block so the variable is set during both
install-latest-lib (build/install) and pytest, not just pytest.
…docs to nav - quickstart.ipynb: replace .md-extension relative links with ../page/ style URLs so they resolve correctly from /quickstart/ context. mkdocs-jupyter does not rewrite .md links the way MkDocs does for regular markdown files, causing the old usage.md / architecture-decisions.md / reference/api.md links to resolve to /quickstart/usage.md (404). Also replaces the stale "Phase 2+ plan" link (internal dev doc, not in nav) with a link to the new User Guides section. - docs/.nav.yml: add all Phase 4 deliverables to the MkDocs navigation so they get RTD pages. New sections: User Guides (7 guides), Migration, and an expanded Contributing section (CONTRIBUTING.md + adding-a-binding.md). Without nav entries, MkDocs does not build pages for these files.
…mpy source build Root cause: `pip install --force-reinstall wheelhouse/*.whl` triggers a numpy source build when no cp313t binary wheel is resolved from PyPI. numpy uses meson-python; meson runs pkg-config which finds the system Python 3.12 headers on ubuntu-latest. Python 3.12's pyconfig.h stub includes x86_64-linux-gnu/python3.12/pyconfig.h, which is only present when python3.12-dev is installed — it isn't on ubuntu-latest by default. Three-part fix: 1. Add numpy>=1.26 to the build feature conda dependencies so numpy is always pre-installed from conda-forge before pip runs, eliminating the need for a source build. 2. Add --no-deps to _install-wheel so pip does not try to pip-install (or rebuild) dependencies that are already managed by conda. 3. Install python3.12-dev on the linux-intel runner as belt-and-suspenders so that if any future transitive dep builds against system Python 3.12 headers, the missing multiarch pyconfig.h is available. Co-Authored-By: Paperclip <[email protected]>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
uw-ssec-bot
pushed a commit
that referenced
this pull request
Jul 18, 2026
Automated pre-commit.ci hook version bumps: - ruff-pre-commit v0.15.20 → v0.15.21 - mirrors-prettier v3.9.4 → v3.9.5 - mirrors-mypy v2.1.0 → v2.3.0 All CI passes (ubuntu + macOS, py313 + py313t). The linux-intel wheel build failure is a pre-existing infrastructure issue (numpy source build / missing python3.12-dev headers), unrelated to these config-only changes — same as when 360e759 was merged. Fix tracked in PR #165.
Bring in action version bumps and zizmor workflow from main. Resolve action version conflicts by taking main's newer versions (checkout@v7, [email protected], upload-artifact@v7). Retain feature branch pixi.lock (numpy>=1.26, --no-deps install).
Collaborator
Author
|
CTO (Paperclip agent): CI is green on all checks including |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pip install --force-reinstall wheelhouse/*.whlin thebuild-wheel-and-testtask intermittently triggers a numpy source build when pip cannot find (or resolve) a cp313t binary wheel from PyPI. numpy uses meson-python as its build system; meson runspkg-config --libs python3which finds the system Python 3.12 on ubuntu-latest. Python 3.12'spyconfig.hstub includesx86_64-linux-gnu/python3.12/pyconfig.h, which requirespython3.12-dev— not installed by default on ubuntu-latest runners.Three-part fix
numpy>=1.26to thebuildfeature conda dependencies (pixi.toml) — numpy is always pre-installed from conda-forge in thebuild-py313tenvironment, eliminating the need for pip to trigger a numpy source build.--no-depsto the_install-wheelpip command (pixi.toml) — pip no longer tries to pip-install (or rebuild) dependencies that conda already manages, closing the source-build path entirely.python3.12-devon the linux-intel runner (.github/workflows/cd.yml) — belt-and-suspenders: if any future transitive dep builds against system Python 3.12 headers, the missing multiarchpyconfig.his available.Test plan
Build wheels for linux-inteljob passes on this PR_print-versionsstill works (numpy available from conda)