Refactor: rewrite in rust#26
Merged
Merged
Conversation
Reimplement the seqfold algorithm and Tm calculation in Rust, exposed to
Python via PyO3 and packaged with maturin. Stays a drop-in `pip install`:
the public API (fold/dg/dg_cache/dot_bracket/tm/tm_cache/gc_cache + Struct)
and the `seqfold` CLI are unchanged, and outputs are byte-for-byte identical
to the previous Python implementation.
What changed:
- src/core/: pure-Rust engine (no Python dep), unit-tested via `cargo test`
- fold.rs: line-by-line port of the Zuker 1981 DP (_w/_v/_stack/_hairpin/
_bulge/_internal_loop/_multi_branch/_traceback). Indices kept as i64 to
preserve the -1 dangling-end sentinels.
- tm.rs: port of tm/tm_cache/gc_cache incl. the Owczarzy salt-correction tree
- pyfloat.rs: Python-compatible round() (ties-to-even) and str(float) so
textual output matches exactly
- data.rs: energy tables codegen'd from the original Python modules
(codegen/gen_data.py) for identical parameters
- src/python.rs: PyO3 `seqfold._core` module; bad sequences map to
RuntimeError/ValueError as before
- Packaging: pyproject.toml (maturin backend, mixed layout), Cargo.toml,
thin python/seqfold/{__init__,main}.py. CLI keeps the original argparse
main.py (exact --help/--version/error text) and calls into Rust. Removed
setup.py/requirements.txt and the old seqfold/*.py implementation.
Verification (all exact):
- cargo test: ported internal tests (_pair/_stack/_bulge/_hairpin/
_internal_loop/_w) pass
- Python public-API suite passes
- examples/dna.csv + rna.csv regenerate byte-for-byte (incl. 118bp)
- CLI parity (--celcius, -d -r, -r, --help, --version) byte-identical
- 200bp sequence matches original Python (-21.7), ~6.5s -> ~0.84s
- Differential fuzz: 120 random DNA/RNA seqs (dg + dot-bracket + every
struct's e/desc/ij) and 80 random tm/tm_cache/gc_cache: 0 mismatches
- Clean wheel installed in a fresh venv: import + CLI + suite all pass
Note: the CLI argument parsing stays in Python (argparse) because argparse's
exact help/usage/error text is impractical to reproduce in clap; all compute
is in Rust.
… output) Optimize the folding hot path without changing any output. Verified byte-for-byte identical via the differential fuzz suite (120 + 150 fresh random DNA/RNA sequences vs the original Python), the examples CSVs, and both test suites. Changes (all output-preserving): - Dense integer-keyed energy tables: pack a stack's 4 nucleotides into a 12-bit code (energies::encode4) and index a flat array, replacing String-key construction + SipHash HashMap lookups in stack/hairpin/bulge/ internal-loop/interior-search. Eliminates millions of heap allocations and string hashes. String maps are retained for the cold Tm path and the variable-length tri/tetra-loop keys. - Complement lookup table ([u8;256]) instead of HashMap<u8,u8>. - Hoist loop-invariants out of the E2 interior-loop search: STACK_DE-ness and the left pair's NN membership depend only on (i, j). - Defer desc-string formatting: build the description only for the winning candidate (after the search), not for every candidate. - Energy-only cache accessors (v_energy/w_energy) for the two hot sites that read just `.e`, avoiding a full Struct clone (desc String + ij Vec) on every memoization hit. Benchmark (examples/bench: 200bp DNA + 118bp RNA + 98bp + 76bp, release): ~1.117s -> ~0.424s (checksum unchanged)
- Version 0.8.0, single-sourced from Cargo.toml. pyproject.toml uses `dynamic = ["version"]` so the version is no longer duplicated; bumpversion updates only Cargo.toml. - Enable PyO3 abi3-py38: one stable-ABI wheel (cp38-abi3) per platform works on CPython 3.8+, instead of a wheel per Python minor version. Output verified byte-for-byte identical via the differential fuzz suite. - Add .github/workflows/CI.yml (maturin generate-ci): builds + tests wheels for Linux manylinux & musllinux (x86_64/x86/aarch64/armv7/s390x/ppc64le), Windows (x64/x86/arm64), macOS (x86_64/arm64), plus sdist, and publishes to PyPI on tag push. This is where cross-platform wheels are produced. - Scope pytest to tests/ (suite is unittest-based) for a deterministic CI run. - README: drop the redundant API/CLI note and the from-source build section.
PyO3 0.22 doesn't support the free-threaded/no-GIL CPython build
("the Python interpreter was built with the GIL disabled, which is not yet
supported by PyO3"), so the maturin-generated 'Build free-threaded wheels'
steps failed every job. The standard abi3 (cp38-abi3) wheels build fine; remove
the 3.14t steps. Release job remains gated on tag pushes only (no PyPI publish
on PRs).
The armv7 (and other exotic) wheels build fine but uraimo/run-on-arch-action has no armv7/ubuntu24.04 image, so the emulated pytest step can't run there. Rather than ship untested wheels, scope linux/musllinux to mainstream, test-emulatable arches (x86_64, aarch64); s390x/ppc64le/32-bit install from the sdist. fail-fast: false so one arch never cancels its siblings.
run-on-arch-action@v2 has no ubuntu24.04 image (the maturin-generated default), which broke the emulated pytest step on aarch64. Pin to ubuntu22.04, which the action provides. Build always succeeded; this fixes only the test step.
- Rename .github/workflows/CI.yml -> release.yml (name: Release); it builds + tests wheels on push/PR and publishes to PyPI on tag. - tests/cli_test.py: exercise the installed `seqfold` console entry point (dg, --celcius, -d dot-bracket, -r substructures, --version) as a subprocess, so CI now covers the CLI too. Collected by the existing pytest step. - README: add a Performance section benchmarking CPython vs the Rust engine across lengths (~16-25x, identical output); make pip the recommended install and drop the now-obsolete pypy speed note.
Replace the recursive memoized fill with a bottom-up fill by anti-diagonal (span d = j - i). Every cell on a diagonal depends only on strictly smaller spans, so each diagonal's cells are computed in parallel with rayon (into_par_iter) for sequences >= 64 nt; shorter ones run sequentially. Each cell is a pure function of already-finalized smaller cells, so the result is independent of evaluation order -> output is byte-for-byte identical and deterministic. The recursive w/v/multi_branch are replaced by pure compute_v/compute_w/mb that read the (immutable) caches; compute_cell does v then w per cell; fill() drives the diagonals and writes results back after each parallel batch. Verified identical (vs the original Python) on the real abi3 artifact: - cargo test (9) + Python suite incl. CLI (14) - differential fuzz: 120 short + 40 large (64-300 nt, the parallel path) + 80 tm/tm_cache/gc_cache -> 0 mismatches - examples/dna.csv + rna.csv regenerate byte-for-byte - determinism: repeated folds identical Parallel speedup (10 cores), dg unchanged: 120 nt 3.4x | 200 nt 4.3x | 300 nt 5.1x vs single-threaded -> up to ~130x vs the original CPython implementation. Bump version to 0.10.0 (minor): API/output unchanged, but the packaging/runtime model changed (native extension, min Python 3.5->3.8, sdist needs a Rust toolchain). Single-sourced from Cargo.toml.
Stop building "BIFURCATION:/STACK:/HAIRPIN:/..." label strings during the
matrix fill. Cache cells now carry a compact `Desc` tag (None/Hairpin/Pair/
Bifurcation{unpaired,count}); the human-readable label is reconstructed only
for the handful of structures on the traceback path (render_desc), from the
cell coordinates + the stored inner pair.
The bifurcation label was previously `format!`-ed in mb() on the order of n^3
times (once per multibranch candidate, nearly all discarded), and those heap
allocations also serialized across rayon worker threads on the global
allocator. Removing them helps both single-thread time and parallel scaling.
Output is byte-for-byte identical (labels reconstructed exactly): verified vs
the original Python on 120 short + 40 large (64-300 nt) + 80 tm cases, the
examples CSVs, and the 14-test Python suite (incl. CLI) + 9 cargo tests.
Speedup (10-core box), dg unchanged:
4-seq bench single-thread 0.398s -> 0.175s (2.3x) | parallel 0.106s -> 0.059s (1.8x)
python 200nt 70ms -> 37ms | 300nt 219ms -> 116ms
-> ~115x (120nt) to ~250x (300nt) vs the original CPython implementation.
Store each cell's basepair list as SmallVec<[(i32,i32);1]> instead of Vec<(i64,i64)>: inline (no heap allocation) for the common 0-1 pair case, and 32-bit since sequences are far shorter than 2^31. The multi-branch working list in mb() likewise becomes SmallVec<[(i64,i64);4]> (inline for the typical fan-out; its arithmetic stays i64 and is unchanged). This removes two O(n^3) allocation sources from the hot path: the temporary branch list built in every mb() call, and the per-cell list cloned every time a finalized cell is read by a larger span. Indices flow to Python as ints (converted to i64 at the PyO3 boundary), so the API is unchanged. Output byte-for-byte identical: differential vs original Python (120 + 40 large + 80 tm), examples CSVs, 14 Python tests, 9 cargo tests. Speedup (10-core box), dg unchanged: 4-seq bench single-thread 0.175s -> 0.091s (1.9x) | parallel 0.059s -> 0.041s (1.4x) Cumulative vs the faithful Rust port: ~27x (parallel).
Random sequences barely fold, so they under-represent the folding search and make poor examples. Add examples/known_structures.fasta: real sequences with known secondary structure (yeast tRNA-Phe, E. coli 5S rRNA, tRNAs, and structured RNA/DNA oligos from the regression set, each with hairpins/stems/ multi-branch junctions and a reference free energy). Point examples/bench.rs at these real sequences and add examples/README.md. On these real structured sequences the Rust engine is ~80-180x faster than the original CPython implementation (identical dg).
Releasing is now just: bump the version in Cargo.toml, commit, and push a matching git tag (the tag triggers the build+publish workflow). No separate bump tool needed; pyproject.toml derives its version from Cargo.toml.
setup-python can't reliably provide 32-bit CPython 3.13 on the x86 runner
("Version 3.13 was not found in the local cache"), failing the job before it
builds. 32-bit Windows is a legacy target with effectively no users for this
tool; keep windows x64 + arm64 and let x86 install from the sdist.
- Release trigger is now `release: published` (a GitHub Release), not a bare tag push. The version is taken from the release tag and injected into Cargo.toml before each wheel/sdist build, so the tag is the single source of truth and no version file is edited by hand. Cargo.toml's version is just a dev default (noted in a comment). - Switch publishing to PyPI Trusted Publishing over GitHub OIDC: no API token (removed UV_PUBLISH_TOKEN / PYPI_API_TOKEN), `uv publish --trusted-publishing always`, and a dedicated `environment: pypi` on the release job so publish rights can be gated with required reviewers. - Add DEVELOPMENT.md: build/test/bench, codegen, and the full release + trusted-publishing setup.
Self-contained check of the public API + CLI against known-good values; works against a local maturin build, a CI wheel, or a published release. Document it in DEVELOPMENT.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
This is a rewrite with Claude Code with Claude 4.8, 1M context, High.
cons:
pros:
I am inspired to do this because of, well, how code gets written now. And also:
Overview
Reimplements the seqfold algorithm and Tm calculation in Rust, exposed to Python via PyO3 and packaged with maturin. It stays a drop-in
pip install: the public Python API (fold,dg,dg_cache,dot_bracket,tm,tm_cache,gc_cache,Struct) and theseqfoldCLI are unchanged, and outputs are byte-for-byte identical to the previous Python implementation.Motivation: speed. The 200bp profiling sequence goes from ~6.5s (CPython) to ~0.84s on this machine, same result.
dg()on real, structured sequences (examples/known_structures.fasta— yeast tRNA-Phe, E. coli 5S rRNA, tRNAs, structured oligos; these actually fold, unlike random sequences), original pure-Pythonseqfoldvs. the Rust engine (10-core Apple M-series, CPython 3.9; dg identical):0.10.0Several optimizations stack (each verified output-identical): moving off CPython; dense integer-keyed energy tables instead of string-keyed hash lookups; deferring all human-readable label rendering out of the matrix fill (the
BIFURCATION:label alone wasformat!-ed ~n³ times); inline 32-bit per-cell basepair lists (SmallVec, no heap alloc for the common case); and a parallel anti-diagonal fill via rayon for sequences ≥64 nt (the speedup grows with length — ~2× at 120 nt, more on longer inputs). Reproduce withcargo run --release --example bench.Compatibility / breaking changes
requires-python = ">=3.8", enforced by thecp38-abi3wheels), up from the oldsetup.py'spython_requires=">=3.5". In strict semver, raising the floor is breaking — but in practice it restricts no working install: the previous release declared>=3.5yet its__init__.pyimportsimportlib.metadata, which only entered the stdlib in 3.8, soimport seqfoldalready failed on 3.5–3.7. This just makes the declared floor honest.pypy3 (recommended)install guidance is gone: the engine is now a native extension, so the interpreter no longer affects folding speed.Verification (all exact)
cargo test— ported internal tests (_pair/_stack/_bulge/_hairpin/_internal_loop/_w)tests/)examples/dna.csv+examples/rna.csvregenerated--celcius,-d -r,-r,--help,--version)-21.7=-21.7, ~6.5s → ~0.84se/desc/ij)tm/tm_cache/gc_cachepip install)Reviewer notes
main.pyis ~10 lines of glue; all computation is Rust. Happy to switch to a clap-based Rust CLI if preferred (would add an output-parity test).maturin develop --release, orpip install .via the maturin backend). Prebuilt wheels bundle the compiled extension, so end users installing from PyPI need no Rust.src/core/data.rsis generated; regenerating viacodegen/gen_data.pyrequires the originalseqfold/dna.py/rna.py(available in git history).