INTERSECT_VARIANTS intermittently crashes with KeyError: 'CHR:POS:A0:A1' in pgscatalog-intersect (pgsc_calc:v2-blob, --run_ancestry)

### Description of the bug

## Description of the bug

When running ancestry projection (`--run_ancestry` with the HGDP+1kGP v1 reference), the `ANCESTRY_PROJECT:INTERSECT_VARIANTS` process intermittently fails with exit status 1. `pgscatalog-intersect` processes all ~84.3M reference variants, logs `Outputting REFERNCE variants -> reference_variants.txt.gz`, then crashes in the `heapq.merge` key lambda because a row is missing the `CHR:POS:A0:A1` column:

```
pgscatalog.match.cli.intersect_cli: INFO Processed 84325213 REFERENCE variants
pgscatalog.match.cli.intersect_cli: INFO Outputting REFERNCE variants -> reference_variants.txt.gz
Traceback (most recent call last):
  File ".../bin/pgscatalog-intersect", line 10, in <module>
    sys.exit(run_intersect())
  File ".../pgscatalog/match/cli/intersect_cli.py", line 85, in run_intersect
    for v in heapq.merge(
  File ".../python3.12/heapq.py", line 376, in merge
    h_append([key(value), order * direction, value, next])
  File ".../pgscatalog/match/cli/intersect_cli.py", line 87, in <lambda>
    key=lambda v: (v["CHR:POS:A0:A1"], v["ID_REF"], v["REF_REF"]),
KeyError: 'CHR:POS:A0:A1'
```

The upstream steps (`COMBINE_SCOREFILES`, `RELABEL`, `EXTRACT_DATABASE`) complete successfully; scoring (`PLINK2_SCORE`) never runs.

## Environment

- pgsc_calc **v2.0.0**, run via Nextflow
- `pgscatalog.match` running on Python 3.12
- Reference: `pgsc_HGDP+1kGP_v1` (HGDP+1kGP, ~84,325,213 reference variants)
- Target: single sample, GRCh38, `--target_build GRCh38`, `--min_overlap 0.5`

## Intermittency

A single failing log looks deterministic (a `KeyError` on a missing column, not a transient I/O error). **In practice, however, the failure is intermittent**: roughly 1 in 20 samples fails on a given batch, and re-running the *same* sample later succeeds with no code or input changes.

Because the crash happens right after the reference variants are written out and the merge begins, the re-run success suggests the failing input to `heapq.merge` is occasionally malformed — pointing at a nondeterministic intermediate (variant ordering, or a row written without the expected header/columns) rather than a property of any specific sample.

We encounter this bug on a regular basis.

## Command executed

```
pgscatalog-intersect \
  --ref GRCh38_HGDP+1kGP_ALL.pvar.zst \
  --target GRCh38_SAMPLE_ALL.pvar.zst \
  --chrom ALL \
  --maf_target 0 \
  --geno_miss 0.1 \
  --outdir . \
  -v
```

## Expected behavior

`INTERSECT_VARIANTS` should either complete the intersection, or fail with an actionable error if the merge input is malformed — rather than raising a raw `KeyError` on a missing column key.

## Notes / open questions

- Has this `KeyError: 'CHR:POS:A0:A1'` signature been seen before? Existing `INTERSECT_VARIANTS` failures (#273, #319) appear to have different root causes, so this may be distinct.
- Given the intermittency, is there a known source of nondeterminism in how the intersect intermediate / reference variants file is written or ordered before the `heapq.merge`?
- What artifacts would be most useful for diagnosing this if it recurs? (Our environment does not retain the Nextflow work dir on failure, so we currently only have the captured stdout/stderr above — happy to capture more on the next occurrence.)

### Command used and terminal output

```console

```

### Relevant files

2026-06-02T01:30:56.034Z
[-        ] PGS…PROJECT:PLINK2_MAKEBED_REF -
[- ] PGS…PROJECT:PLINK2_MAKEBED_REF -
2026-06-02T01:30:56.034Z
[-        ] PGS…_PROJECT:INTERSECT_THINNED -
[- ] PGS…_PROJECT:INTERSECT_THINNED -
2026-06-02T01:30:56.034Z
[-        ] PGS…CESTRY_PROJECT:RELABEL_IDS -
[- ] PGS…CESTRY_PROJECT:RELABEL_IDS -
2026-06-02T01:30:56.034Z
[-        ] PGS…JECT:PLINK2_MAKEBED_TARGET -
[- ] PGS…JECT:PLINK2_MAKEBED_TARGET -
2026-06-02T01:30:56.034Z
[-        ] PGS…STRY_PROJECT:PLINK2_ORIENT -
[- ] PGS…STRY_PROJECT:PLINK2_ORIENT -
2026-06-02T01:30:56.034Z
[-        ] PGS…CESTRY_PROJECT:FRAPOSA_PCA -
[- ] PGS…CESTRY_PROJECT:FRAPOSA_PCA -
2026-06-02T01:30:56.034Z
[-        ] PGS…RY_PROJECT:FRAPOSA_PROJECT -
[- ] PGS…RY_PROJECT:FRAPOSA_PROJECT -
2026-06-02T01:30:56.034Z
[97/e4dcd7] PGS…TS (SAMPLE chromosome ALL) | 1 of 1 ✔

[97/e4dcd7] PGS…TS (SAMPLE chromosome ALL) | 1 of 1 ✔
2026-06-02T01:30:56.034Z
[-        ] PGS…SCCALC:MATCH:MATCH_COMBINE -

[- ] PGS…SCCALC:MATCH:MATCH_COMBINE -
2026-06-02T01:30:56.034Z
[-        ] PGS…Y_SCORE:RELABEL_SCOREFILES -

[- ] PGS…Y_SCORE:RELABEL_SCOREFILES -
2026-06-02T01:30:56.034Z
[-        ] PGS…:APPLY_SCORE:RELABEL_AFREQ -

[- ] PGS…:APPLY_SCORE:RELABEL_AFREQ -
2026-06-02T01:30:56.034Z
[-        ] PGS…C:APPLY_SCORE:PLINK2_SCORE -

[- ] PGS…C:APPLY_SCORE:PLINK2_SCORE -
2026-06-02T01:30:56.034Z
[-        ] PGS…PPLY_SCORE:SCORE_AGGREGATE -

[- ] PGS…PPLY_SCORE:SCORE_AGGREGATE -
2026-06-02T01:30:56.034Z
Plus 3 more processes waiting for tasks…

Plus 3 more processes waiting for tasks…
2026-06-02T01:30:56.034Z
Execution cancelled -- Finishing pending tasks before exit

Execution cancelled -- Finishing pending tasks before exit
2026-06-02T01:30:56.034Z
-[pgscatalog/pgsc_calc] Pipeline completed with errors-

-[pgscatalog/pgsc_calc] Pipeline completed with errors-
2026-06-02T01:30:56.034Z
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)'

ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)'
2026-06-02T01:30:56.034Z
Caused by:

Caused by:
2026-06-02T01:30:56.034Z
  Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)` terminated with an error exit status (1)

Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)` terminated with an error exit status (1)
2026-06-02T01:30:56.034Z
Command executed:

Command executed:
2026-06-02T01:30:56.034Z
  pgscatalog-intersect --ref GRCh38_HGDP+1kGP_ALL.pvar.zst         --target GRCh38_SAMPLE_ALL.pvar.zst         --chrom ALL         --maf_target 0         --geno_miss 0.1         --outdir .         -v

pgscatalog-intersect --ref GRCh38_HGDP+1kGP_ALL.pvar.zst --target GRCh38_SAMPLE_ALL.pvar.zst --chrom ALL --maf_target 0 --geno_miss 0.1 --outdir . -v
2026-06-02T01:30:56.034Z
  

2026-06-02T01:30:56.034Z
  n_matched=$(sed -n '3p' intersect_counts_ALL.txt)

n_matched=$(sed -n '3p' intersect_counts_ALL.txt)
2026-06-02T01:30:56.034Z
  

2026-06-02T01:30:56.034Z
  if [ $n_matched == "0" ]

if [ $n_matched == "0" ]
2026-06-02T01:30:56.034Z
  then

then
2026-06-02T01:30:56.034Z
      echo "ERROR: No variants in intersection"

echo "ERROR: No variants in intersection"
2026-06-02T01:30:56.034Z
      exit 1

exit 1
2026-06-02T01:30:56.034Z
  else

else
2026-06-02T01:30:56.034Z
      mv matched_variants.txt.gz SAMPLE_ALL_matched.txt.gz

mv matched_variants.txt.gz SAMPLE_ALL_matched.txt.gz
2026-06-02T01:30:56.034Z
  fi

fi
2026-06-02T01:30:56.034Z
  
2026-06-02T01:30:56.034Z
  cat <<-END_VERSIONS > versions.yml
cat <<-END_VERSIONS > versions.yml
2026-06-02T01:30:56.034Z
  INTERSECT_VARIANTS:
INTERSECT_VARIANTS:
2026-06-02T01:30:56.034Z
      pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
2026-06-02T01:30:56.034Z
  END_VERSIONS
END_VERSIONS
2026-06-02T01:30:56.034Z
Command exit status:
Command exit status:
2026-06-02T01:30:56.034Z
  1
1
2026-06-02T01:30:56.034Z
Command output:
Command output:
2026-06-02T01:30:56.034Z
  (empty)
(empty)
2026-06-02T01:30:56.034Z
Command error:
Command error:
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:19 INFO     Processed 69000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:19 INFO Processed 69000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:22 INFO     Processed 69500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:22 INFO Processed 69500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:25 INFO     Processed 70000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:25 INFO Processed 70000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:29 INFO     Processed 70500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:29 INFO Processed 70500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:32 INFO     Processed 71000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:32 INFO Processed 71000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:35 INFO     Processed 71500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:35 INFO Processed 71500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:38 INFO     Processed 72000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:38 INFO Processed 72000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:41 INFO     Processed 72500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:41 INFO Processed 72500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:45 INFO     Processed 73000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:45 INFO Processed 73000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:48 INFO     Processed 73500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:48 INFO Processed 73500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:51 INFO     Processed 74000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:51 INFO Processed 74000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:54 INFO     Processed 74500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:54 INFO Processed 74500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:57 INFO     Processed 75000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:57 INFO Processed 75000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:00 INFO     Processed 75500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:00 INFO Processed 75500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:03 INFO     Processed 76000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:03 INFO Processed 76000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:06 INFO     Processed 76500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:06 INFO Processed 76500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:09 INFO     Processed 77000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:09 INFO Processed 77000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:12 INFO     Processed 77500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:12 INFO Processed 77500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:15 INFO     Processed 78000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:15 INFO Processed 78000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:18 INFO     Processed 78500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:18 INFO Processed 78500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:22 INFO     Processed 79000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:22 INFO Processed 79000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:25 INFO     Processed 79500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:25 INFO Processed 79500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:28 INFO     Processed 80000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:28 INFO Processed 80000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:31 INFO     Processed 80500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:31 INFO Processed 80500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:34 INFO     Processed 81000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:34 INFO Processed 81000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:38 INFO     Processed 81500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:38 INFO Processed 81500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:41 INFO     Processed 82000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:41 INFO Processed 82000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:44 INFO     Processed 82500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:44 INFO Processed 82500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:47 INFO     Processed 83000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:47 INFO Processed 83000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:50 INFO     Processed 83500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:50 INFO Processed 83500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:53 INFO     Processed 84000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:53 INFO Processed 84000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO     Processed 84325213 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO Processed 84325213 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO     Outputting REFERNCE variants -> reference_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO Outputting REFERNCE variants -> reference_variants.txt.gz
2026-06-02T01:30:56.034Z
  Traceback (most recent call last):
Traceback (most recent call last):
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/bin/pgscatalog-intersect", line 10, in <module>
File "/opt/pgscatalog_utils/bin/pgscatalog-intersect", line 10, in <module>
2026-06-02T01:30:56.034Z
      sys.exit(run_intersect())
sys.exit(run_intersect())
2026-06-02T01:30:56.034Z
               ^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 85, in run_intersect
File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 85, in run_intersect
2026-06-02T01:30:56.034Z
      for v in heapq.merge(
for v in heapq.merge(
2026-06-02T01:30:56.034Z
               ^^^^^^^^^^^^
^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/heapq.py", line 376, in merge
File "/opt/pgscatalog_utils/lib/python3.12/heapq.py", line 376, in merge
2026-06-02T01:30:56.034Z
      h_append([key(value), order * direction, value, next])
h_append([key(value), order * direction, value, next])
2026-06-02T01:30:56.034Z
                ^^^^^^^^^^
^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 87, in <lambda>
File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 87, in <lambda>
2026-06-02T01:30:56.034Z
      key=lambda v: (v["CHR:POS:A0:A1"], v["ID_REF"], v["REF_REF"]),
key=lambda v: (v["CHR:POS:A0:A1"], v["ID_REF"], v["REF_REF"]),
2026-06-02T01:30:56.034Z
                     ~^^^^^^^^^^^^^^^^^
~^^^^^^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
  KeyError: 'CHR:POS:A0:A1'
KeyError: 'CHR:POS:A0:A1'
2026-06-02T01:30:56.034Z
  cp: '.command.out' and '.command.out' are the same file
cp: '.command.out' and '.command.out' are the same file
2026-06-02T01:30:56.034Z
  cp: '.command.err' and '.command.err' are the same file
cp: '.command.err' and '.command.err' are the same file
2026-06-02T01:30:56.034Z
  cp: '.command.trace' and '.command.trace' are the same file
cp: '.command.trace' and '.command.trace' are the same file
2026-06-02T01:30:56.034Z
Work dir:
Work dir:
2026-06-02T01:30:56.034Z
  /data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b/nextflow-work/cf/2e529cd82cf8223659cd1aeb4b9072
/data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b/nextflow-work/cf/2e529cd82cf8223659cd1aeb4b9072
2026-06-02T01:30:56.034Z
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
2026-06-02T01:30:56.034Z
 -- Check '.nextflow.log' file for details
-- Check '.nextflow.log' file for details
2026-06-02T01:30:56.034Z
executor >  local (5)
executor > local (5)
2026-06-02T01:30:56.034Z
[e9/96caeb] PGS…ECK:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[e9/96caeb] PGS…ECK:COMBINE_SCOREFILES (1) | 1 of 1 ✔
2026-06-02T01:30:56.034Z
[-        ] PGS…MPATIBLE:PLINK2_RELABELBIM -
[- ] PGS…MPATIBLE:PLINK2_RELABELBIM -
2026-06-02T01:30:56.034Z
[7e/6d4abc] PGS…AR (SAMPLE chromosome ALL) | 1 of 1 ✔
[7e/6d4abc] PGS…AR (SAMPLE chromosome ALL) | 1 of 1 ✔
2026-06-02T01:30:56.034Z
[-        ] PGS…MAKE_COMPATIBLE:PLINK2_VCF -
[- ] PGS…MAKE_COMPATIBLE:PLINK2_VCF -
2026-06-02T01:30:56.034Z
[76/44e2db] PGS…OJECT:EXTRACT_DATABASE (1) | 1 of 1 ✔
[76/44e2db] PGS…OJECT:EXTRACT_DATABASE (1) | 1 of 1 ✔
2026-06-02T01:30:56.034Z
[cf/2e529c] PGS…TS (SAMPLE chromosome ALL) | 1 of 1, failed: 1 ✘
[cf/2e529c] PGS…TS (SAMPLE chromosome ALL) | 1 of 1, failed: 1 ✘
2026-06-02T01:30:56.034Z
[-        ] PGS…RY_PROJECT:FILTER_VARIANTS -
[- ] PGS…RY_PROJECT:FILTER_VARIANTS -
2026-06-02T01:30:56.034Z
[-        ] PGS…PROJECT:PLINK2_MAKEBED_REF -
[- ] PGS…PROJECT:PLINK2_MAKEBED_REF -
2026-06-02T01:30:56.034Z
[-        ] PGS…_PROJECT:INTERSECT_THINNED -
[- ] PGS…_PROJECT:INTERSECT_THINNED -
2026-06-02T01:30:56.034Z
[-        ] PGS…CESTRY_PROJECT:RELABEL_IDS -
[- ] PGS…CESTRY_PROJECT:RELABEL_IDS -
2026-06-02T01:30:56.034Z
[-        ] PGS…JECT:PLINK2_MAKEBED_TARGET -
[- ] PGS…JECT:PLINK2_MAKEBED_TARGET -
2026-06-02T01:30:56.034Z
[-        ] PGS…STRY_PROJECT:PLINK2_ORIENT -
[- ] PGS…STRY_PROJECT:PLINK2_ORIENT -
2026-06-02T01:30:56.034Z
[-        ] PGS…CESTRY_PROJECT:FRAPOSA_PCA -
[- ] PGS…CESTRY_PROJECT:FRAPOSA_PCA -
2026-06-02T01:30:56.034Z
[-        ] PGS…RY_PROJECT:FRAPOSA_PROJECT -
[- ] PGS…RY_PROJECT:FRAPOSA_PROJECT -
2026-06-02T01:30:56.034Z
[97/e4dcd7] PGS…TS (SAMPLE chromosome ALL) | 1 of 1 ✔
[97/e4dcd7] PGS…TS (SAMPLE chromosome ALL) | 1 of 1 ✔
2026-06-02T01:30:56.034Z
[-        ] PGS…SCCALC:MATCH:MATCH_COMBINE -
[- ] PGS…SCCALC:MATCH:MATCH_COMBINE -
2026-06-02T01:30:56.034Z
[-        ] PGS…Y_SCORE:RELABEL_SCOREFILES -
[- ] PGS…Y_SCORE:RELABEL_SCOREFILES -
2026-06-02T01:30:56.034Z
[-        ] PGS…:APPLY_SCORE:RELABEL_AFREQ -
[- ] PGS…:APPLY_SCORE:RELABEL_AFREQ -
2026-06-02T01:30:56.034Z
[-        ] PGS…C:APPLY_SCORE:PLINK2_SCORE -
[- ] PGS…C:APPLY_SCORE:PLINK2_SCORE -
2026-06-02T01:30:56.034Z
[-        ] PGS…PPLY_SCORE:SCORE_AGGREGATE -
[- ] PGS…PPLY_SCORE:SCORE_AGGREGATE -
2026-06-02T01:30:56.034Z
Plus 3 more processes waiting for tasks…
Plus 3 more processes waiting for tasks…
2026-06-02T01:30:56.034Z
Execution cancelled -- Finishing pending tasks before exit
Execution cancelled -- Finishing pending tasks before exit
2026-06-02T01:30:56.034Z
-[pgscatalog/pgsc_calc] Pipeline completed with errors-
-[pgscatalog/pgsc_calc] Pipeline completed with errors-
2026-06-02T01:30:56.034Z
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)'
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)'
2026-06-02T01:30:56.034Z
Caused by:
Caused by:
2026-06-02T01:30:56.034Z
  Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)` terminated with an error exit status (1)
Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (SAMPLE chromosome ALL)` terminated with an error exit status (1)
2026-06-02T01:30:56.034Z
Command executed:
Command executed:
2026-06-02T01:30:56.034Z
  pgscatalog-intersect --ref GRCh38_HGDP+1kGP_ALL.pvar.zst         --target GRCh38_SAMPLE_ALL.pvar.zst         --chrom ALL         --maf_target 0         --geno_miss 0.1         --outdir .         -v
pgscatalog-intersect --ref GRCh38_HGDP+1kGP_ALL.pvar.zst --target GRCh38_SAMPLE_ALL.pvar.zst --chrom ALL --maf_target 0 --geno_miss 0.1 --outdir . -v
2026-06-02T01:30:56.034Z
  
2026-06-02T01:30:56.034Z
  n_matched=$(sed -n '3p' intersect_counts_ALL.txt)
n_matched=$(sed -n '3p' intersect_counts_ALL.txt)
2026-06-02T01:30:56.034Z
  
2026-06-02T01:30:56.034Z
  if [ $n_matched == "0" ]
if [ $n_matched == "0" ]
2026-06-02T01:30:56.034Z
  then
then
2026-06-02T01:30:56.034Z
      echo "ERROR: No variants in intersection"
echo "ERROR: No variants in intersection"
2026-06-02T01:30:56.034Z
      exit 1
exit 1
2026-06-02T01:30:56.034Z
  else
else
2026-06-02T01:30:56.034Z
      mv matched_variants.txt.gz SAMPLE_ALL_matched.txt.gz
mv matched_variants.txt.gz SAMPLE_ALL_matched.txt.gz
2026-06-02T01:30:56.034Z
  fi
fi
2026-06-02T01:30:56.034Z
  
2026-06-02T01:30:56.034Z
  cat <<-END_VERSIONS > versions.yml
cat <<-END_VERSIONS > versions.yml
2026-06-02T01:30:56.034Z
  INTERSECT_VARIANTS:
INTERSECT_VARIANTS:
2026-06-02T01:30:56.034Z
      pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
2026-06-02T01:30:56.034Z
  END_VERSIONS
END_VERSIONS
2026-06-02T01:30:56.034Z
Command exit status:
Command exit status:
2026-06-02T01:30:56.034Z
  1
1
2026-06-02T01:30:56.034Z
Command output:
Command output:
2026-06-02T01:30:56.034Z
  (empty)
(empty)
2026-06-02T01:30:56.034Z
Command error:
Command error:
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:19 INFO     Processed 69000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:19 INFO Processed 69000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:22 INFO     Processed 69500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:22 INFO Processed 69500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:25 INFO     Processed 70000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:25 INFO Processed 70000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:29 INFO     Processed 70500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:29 INFO Processed 70500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:32 INFO     Processed 71000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:32 INFO Processed 71000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:35 INFO     Processed 71500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:35 INFO Processed 71500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:38 INFO     Processed 72000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:38 INFO Processed 72000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:41 INFO     Processed 72500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:41 INFO Processed 72500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:45 INFO     Processed 73000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:45 INFO Processed 73000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:48 INFO     Processed 73500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:48 INFO Processed 73500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:51 INFO     Processed 74000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:51 INFO Processed 74000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:54 INFO     Processed 74500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:54 INFO Processed 74500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:57 INFO     Processed 75000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:29:57 INFO Processed 75000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:00 INFO     Processed 75500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:00 INFO Processed 75500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:03 INFO     Processed 76000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:03 INFO Processed 76000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:06 INFO     Processed 76500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:06 INFO Processed 76500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:09 INFO     Processed 77000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:09 INFO Processed 77000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:12 INFO     Processed 77500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:12 INFO Processed 77500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:15 INFO     Processed 78000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:15 INFO Processed 78000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:18 INFO     Processed 78500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:18 INFO Processed 78500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:22 INFO     Processed 79000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:22 INFO Processed 79000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:25 INFO     Processed 79500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:25 INFO Processed 79500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:28 INFO     Processed 80000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:28 INFO Processed 80000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:31 INFO     Processed 80500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:31 INFO Processed 80500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:34 INFO     Processed 81000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:34 INFO Processed 81000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:38 INFO     Processed 81500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:38 INFO Processed 81500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:41 INFO     Processed 82000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:41 INFO Processed 82000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:44 INFO     Processed 82500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:44 INFO Processed 82500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:47 INFO     Processed 83000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:47 INFO Processed 83000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:50 INFO     Processed 83500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:50 INFO Processed 83500000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:53 INFO     Processed 84000000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:53 INFO Processed 84000000 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO     Processed 84325213 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO Processed 84325213 REFERENCE variants
2026-06-02T01:30:56.034Z
  pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO     Outputting REFERNCE variants -> reference_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2026-06-02 01:30:54 INFO Outputting REFERNCE variants -> reference_variants.txt.gz
2026-06-02T01:30:56.034Z
  Traceback (most recent call last):
Traceback (most recent call last):
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/bin/pgscatalog-intersect", line 10, in <module>
File "/opt/pgscatalog_utils/bin/pgscatalog-intersect", line 10, in <module>
2026-06-02T01:30:56.034Z
      sys.exit(run_intersect())
sys.exit(run_intersect())
2026-06-02T01:30:56.034Z
               ^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 85, in run_intersect
File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 85, in run_intersect
2026-06-02T01:30:56.034Z
      for v in heapq.merge(
for v in heapq.merge(
2026-06-02T01:30:56.034Z
               ^^^^^^^^^^^^
^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/heapq.py", line 376, in merge
File "/opt/pgscatalog_utils/lib/python3.12/heapq.py", line 376, in merge
2026-06-02T01:30:56.034Z
      h_append([key(value), order * direction, value, next])
h_append([key(value), order * direction, value, next])
2026-06-02T01:30:56.034Z
                ^^^^^^^^^^
^^^^^^^^^^
2026-06-02T01:30:56.034Z
    File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 87, in <lambda>
File "/opt/pgscatalog_utils/lib/python3.12/site-packages/pgscatalog/match/cli/intersect_cli.py", line 87, in <lambda>
2026-06-02T01:30:56.034Z
      key=lambda v: (v["CHR:POS:A0:A1"], v["ID_REF"], v["REF_REF"]),
key=lambda v: (v["CHR:POS:A0:A1"], v["ID_REF"], v["REF_REF"]),
2026-06-02T01:30:56.034Z
                     ~^^^^^^^^^^^^^^^^^
~^^^^^^^^^^^^^^^^^
2026-06-02T01:30:56.034Z
  KeyError: 'CHR:POS:A0:A1'
KeyError: 'CHR:POS:A0:A1'
2026-06-02T01:30:56.034Z
  cp: '.command.out' and '.command.out' are the same file
cp: '.command.out' and '.command.out' are the same file
2026-06-02T01:30:56.034Z
  cp: '.command.err' and '.command.err' are the same file
cp: '.command.err' and '.command.err' are the same file
2026-06-02T01:30:56.034Z
  cp: '.command.trace' and '.command.trace' are the same file
cp: '.command.trace' and '.command.trace' are the same file
2026-06-02T01:30:56.034Z
Work dir:
Work dir:
2026-06-02T01:30:56.034Z
  /data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b/nextflow-work/cf/2e529cd82cf8223659cd1aeb4b9072
/data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b/nextflow-work/cf/2e529cd82cf8223659cd1aeb4b9072
2026-06-02T01:30:56.034Z
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
2026-06-02T01:30:56.034Z
 -- Check '.nextflow.log' file for details
-- Check '.nextflow.log' file for details
2026-06-02T01:30:56.584Z
Cleaned up output directory: /data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b
Cleaned up output directory: /data/results/38979e5c-e866-447f-aef2-bf19e0a5dc4b
2026-06-02T01:30:56.875Z
Cleaned up data directory: /work/data
Cleaned up data directory: /work/data
No newer events at this moment. 
Auto retry paused.
 
Resume
 
202 events loaded

## System information

* **Nextflow version:** 24.10.2
* **Hardware:** Cloud (AWS)
* **Executor:** Nextflow runs inside a single container orchestrated by AWS Step Functions, so each process runs with the `local` executor rather than `awsbatch`
* **Container engine:** Docker (PGScatalog prebuilt image)
* **OS:** Linux
* **Version of `pgsc_calc`:** 2.0.0

Navigation Menu

Uh oh!

INTERSECT_VARIANTS intermittently crashes with KeyError: 'CHR:POS:A0:A1' in pgscatalog-intersect (pgsc_calc:v2-blob, --run_ancestry) #502

Description of the bug

Description of the bug

Environment

Intermittency

Command executed

Expected behavior

Notes / open questions

Command used and terminal output

Relevant files

System information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

INTERSECT_VARIANTS intermittently crashes with KeyError: 'CHR:POS:A0:A1' in pgscatalog-intersect (pgsc_calc:v2-blob, --run_ancestry) #502

Description

Description of the bug

Description of the bug

Environment

Intermittency

Command executed

Expected behavior

Notes / open questions

Command used and terminal output

Relevant files

System information

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions