PDC integration: one downloader for both data commons

[paulocilasjr]

Replaces the GDC-only downloader with a tool that takes a manifest from either the
Genomic or Proteomic Data Commons, detects which produced it, and returns a
collection that is usable by downstream tools without manual conversion.

tools/gdc-downloader/ → tools/gacdi-downloader/; tool id gdc_downloader →
gacdi_downloader.

What it does

Detection (download/detect.py) reads the manifest header and picks GDC vs
PDC from column names. Matching neither, or both, fails with the observed and
expected columns rather than guessing.

PDC transfers (download/pdc.py) stream to a .partial path, verify size and
MD5 before moving into place, and skip rows already present with the right
checksum — PDC allows only 10 downloads of a file per 24h per IP. Expired signed
URLs (7-day lifetime) and rate limits get actionable messages. Duplicate names are
disambiguated by file ID; path separators and .. are rejected before any write.

GDC transfers keep the existing gdc-client wrapper and its mode-0600 named
pipe for controlled-access tokens.

Gzipped payloads are expanded after checksum verification, so integrity is
checked against the bytes the manifest describes. This is what makes the output
usable — Galaxy's mzml/mzid datatypes describe uncompressed XML, so .mzML.gz
can't reach msconvert, Comet or OpenMS without a conversion step. Expansion uses
an allow-list of inner extensions, not magic bytes: BAM, BGZF VCF and tabix
indexes are all gzip but meaningless expanded. --keep-compressed exists at the
CLI, not as a tool parameter.

Discovery, and two silent bugs

Unregistered extensions gave a collection that opened empty. __name_and_ext__
hands the trailing filename token to the datatype registry, but commons files are
named .mzML, .mzid, .psm, .raw where Galaxy wants mzml, mzid,
tabular, thermo.raw. Galaxy set element_count while the elements stayed
unresolvable: "a list with 5 datasets" that opened empty.

Datatypes are now declared explicitly, one rule per format, with a catch-all
keeping unknowns as data instead of dropping them. Sniffing was rejected — these
files reach hundreds of MB and several are binary. Patterns are anchored and the
catch-all uses a negative lookahead, so every file matches exactly one rule.

Sorting on an uncaptured group errored the collection. The rules sorted on
name, which no pattern captures. Galaxy sorts with attrgetter(sort_by) and
JsonCollectedDatasetMatch.name returns None, so sorting two matches raised
TypeError. The tool had already exited 0, so the only symptom was an errored
collection with clean stderr — and it needed two files matching one rule to fire,
so a manifest with one file per datatype passed. Now sorts on designation.

Tests

Transfer logic is covered offline: checksum and size mismatches, partial cleanup,
retry budget, expired URLs, rate limits, path traversal, duplicate names, resume,
and that integrity is checked against the compressed bytes.

Two contract tests need a reviewer's judgement, since they encode Galaxy behaviour
in our suite. test_wrapper_discovery_contract.py replicates what
output_collect requires of these patterns; reintroducing the sort_by bug fails
it in 14 places. test_version_contract.py ties each wrapper's @TOOL_VERSION@ to
the package __version__. Trade-off: if Galaxy changes how it sorts, these model
the old behaviour and could pass while the wrapper breaks. Both bugs above were
invisible to the tool test and to planemo lint, which is why I took that.

The GDC tool test hits the live API, so it only runs under planemo test. PDC has
no live test on purpose: signed URLs expire in 7 days and each run burns quota.

Versioning

__version__ → 0.3.0, both wrappers with it. containers.yml tags both images
from that version and refuses to overwrite, so a suffix-only bump would have failed
the build — gacdi-downloader:0.2.0 is already published. manifest_gdc was
pinned at 0.1.0 against a 0.2.0 package, orphaning a tag on every build;
test_version_contract.py prevents that recurring.

Expect one red CI run after merge

Both jobs now verify the image is publicly pullable, querying Quay without
credentials — the workflow is logged in, so an authenticated check would pass on a
private repo and prove nothing.

goeckslab/gacdi-downloader doesn't exist on Quay yet, and Quay creates
repositories private, so this step fails the first time it runs. That's the
check working: otherwise the push succeeds and every Galaxy job dies at pull time
with an auth error that says nothing about visibility. Fix once after merge: let
it push, set the repo public in Quay settings, re-run.

Review checklist

• Set gacdi-downloader public on Quay after the first push
• Datatype mapping matches what this instance's proteomics tools expect
• Decompression allow-list is right for the GDC formats you use (expands
  .txt.gz/.maf.gz, protects .bam/.vcf.gz/.tbi)
• Whether the contract tests are worth their coupling to Galaxy internals
• Tool id change breaks saved workflows referencing gdc_downloader

[luke-c-sargent]

functionality added via #5