Skip to content

Pepsickle / torch libomp clash crashes vaxrank on macOS — robust fix needed #266

Description

@iskandr

Repro (macOS, vaxrank with --input-lens)

```
vaxrank --input-lens HugoLo_IPRES_2016-Pt02....tsv --output-peptide x.fasta
```

Process aborts with:

```
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
[1] PID abort vaxrank --input-lens
```

OR (with `KMP_DUPLICATE_LIB_OK=TRUE` set) silently exits with code 139 (SIGSEGV) without running pepsickle / construct writers.

Cause

Pepsickle pulls in torch. macOS torch ships its own `libomp.dylib`. By the time `_load_default_predictor` is called, pandas / numpy / pyarrow have already loaded a different `libomp` via OpenBLAS / MKL. The second OpenMP runtime to initialize either:

  • aborts the process at init time (the loud failure), OR
  • continues with two OpenMP runtimes coexisting and segfaults later when their internal state collides during inference (the silent failure).

`KMP_DUPLICATE_LIB_OK=TRUE` (the LLVM-documented workaround) silences the abort but doesn't prevent the segfault.

Current state (PR #262, post-this-fix)

Pepsickle credibility tagging is now off by default (`--processing-aware-annotation=False`). Users opt in when their environment has a single OpenMP runtime. Linux installs typically work; macOS users can try `MKL_THREADING_LAYER=GNU` or run pepsickle separately.

Robust fix options

  1. Subprocess isolation: spawn a fresh Python process for pepsickle inference (no pandas / numpy in the child). Pickle source sequences in, pickle cleavage probabilities out. Adds ~1s subprocess-launch overhead per run; fully solves the libomp clash. Probably the right move.

  2. Use ONNX runtime instead of torch: pepsickle's epitope model could be exported to ONNX and run via `onnxruntime` (single OpenMP, no torch). Requires upstream work in pepsickle.

  3. Pin a single libomp: ship vaxrank with a wheel that links every OpenMP-using dep against the same libomp. Brittle; couples vaxrank to torch's libomp version.

  4. Use a non-OpenMP backend for inference: torch.set_num_threads(1) + force CPU mode + disable parallelism. Doesn't fully solve the dual-runtime issue but mitigates some failure modes.

Acceptance

  • Pepsickle credibility tagging reliable on macOS without env-var dancing.
  • Default returns to on (currently off as a defensive measure).
  • CI test that exercises the LENS + pepsickle path end-to-end on macOS — currently we opt out via `--no-processing-aware-annotation` because of this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions