Skip to content

Add resumable download reports and Galaxy-safe recovery collections #9

Description

@paulocilasjr

Why this is important

mcdi download currently reports per-file results only in the job log. The Galaxy wrapper exposes only the downloaded-files collection.

If a Galaxy execution is interrupted:

  • A new execution receives a new working directory.
  • Previously downloaded files are not available to the new job.
  • The same manifest is downloaded again from the beginning.
  • Users do not receive a structured list of successful, failed, incomplete, or checksum-invalid files.
  • Large GDC, PDC, or IDC cohorts can waste substantial time and bandwidth.

Expected behavior

The tool should produce:

  • A visible collection containing successfully downloaded samples.
  • A machine-readable download-state file, hidden in the Galaxy history by default.

Supported statuses should include:

  • pending
  • downloaded
  • skipped
  • error
  • checksum_mismatch
  • extract_error

Galaxy constraint

  • Creating a hidden status report and retrying only unresolved files is feasible.
  • Appending datasets directly to an existing Galaxy history collection by collection ID is not a normal or safe Galaxy tool operation.

The Galaxy-safe approach is:

  1. Preserve the successful collection from the first run.
  2. Download unresolved files into a new recovery collection.
  3. Use Galaxy’s built-in Merge Collections operation to create a new combined collection containing the old and recovered datasets.
  4. Leave the original collection unchanged.

The report should use a portable run_id and manifest fingerprint to associate related outputs. A Galaxy collection ID may be recorded as informational metadata if Galaxy can safely provide it, but retry logic must not depend on it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions