Native cvmfsexec support for CVMFS-hosted Singularity/Apptainer containers#475
Open
natefoo wants to merge 10 commits into
Open
Native cvmfsexec support for CVMFS-hosted Singularity/Apptainer containers#475natefoo wants to merge 10 commits into
natefoo wants to merge 10 commits into
Conversation
Introduce pulsar.managers.util.cvmfsexec, a pure (I/O-free) helper that generates the shell needed to run jobs under cvmfsexec. It supports two modes: - namespace: run the command under `cvmfsexec <repos> -- ...` so the real /cvmfs is available (no path rewriting). - mountrepo: for hosts without unprivileged user namespaces, generate the mount preamble and rewrite host-side container image paths from /cvmfs/<repo> to $CVMFSEXEC_DIR/.cvmfsexec/dist/cvmfs/<repo>. Only image repositories (default: names starting with "singularity") are rewritten, so in-container /cvmfs references that resolve via the singularity bind mount are left intact. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add an optional $cvmfsexec_setup placeholder to the default job file template, rendered after $prepare_dirs_statement and before the working directory change. It defaults to an empty string, so ordinary jobs are unaffected. This gives managers a first-class place to inject cvmfsexec mount setup instead of smuggling it through the env mechanism. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Read a `cvmfsexec` manager option (app.yml) into BaseManager, and in DirectoryBaseManager._setup_job_file render the mount preamble into the new $cvmfsexec_setup template slot and rewrite/wrap the command line according to the configured mode. A per-job override supplied in setup_params takes precedence over the manager default. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Let a Galaxy job destination set a `cvmfsexec` param that overrides the Pulsar manager's default. The client carries it in the launch message (HTTP and message queue), the submit endpoint accepts it, and the server merges it into setup_params after the setup decision so it reaches the manager without triggering job setup or changing any launch() signature. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a commented cvmfsexec example to the managers section of app.yml.sample covering mode, cvmfsexec path, repositories, and image_repositories, and noting it can be overridden per destination. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Remove rewrite_command and the image_repositories config from the cvmfsexec helper. In mountrepo mode the host-side container image path is now remapped by an ordinary Galaxy file_actions `rewrite` rule (which for rewrite_parameters destinations relies on the companion galaxyproject/galaxy change routing the resolved image path through the compute environment), rather than a bespoke string replace on the Pulsar server. Pulsar now owns only the cvmfsexec runtime setup: the mountrepo preamble and, for namespace mode, wrapping the command. This and the Galaxy change are deployed together; the two must not both rewrite the image path. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replace the __PULSAR_JOB_DIRECTORY__ token with the absolute per-job directory server-side (mirroring the existing __PULSAR_JOBS_DIRECTORY__ handling; fulfils the adjacent TODO). This lets the client emit a path relative to the runtime-unknown job directory without embedding a shell variable - important for tokens that pass through shlex.quote on the client, e.g. a container image path rewritten for cvmfsexec. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generate the mountrepo preamble against the absolute job directory (Pulsar knows it at job-script generation time) rather than a $CVMFSEXEC_DIR shell variable computed from $(dirname "$_GALAXY_JOB_DIR"). The container image path is rewritten (Galaxy side) to __PULSAR_JOB_DIRECTORY__/.cvmfsexec/dist/ cvmfs/... which the server substitutes to the same absolute path; a $VAR there would be single-quoted by the client's shlex.quote of the image and never expand. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a Containers overview framing Docker (batch/co-execution backends) vs Singularity/Apptainer, and a "Singularity / Apptainer and CVMFS" section covering the singularity.galaxyproject.org CVMFS image repo and cvmfsexec (namespace vs mountrepo modes, building the self-contained distribution, the galaxyproject.cvmfsexec role, and a usegalaxy.org example). Document the mountrepo container-image rewrite rule in galaxy_conf, and trim the app.yml.sample cvmfsexec commentary to point at these docs. Co-Authored-By: Claude Opus 4.8 <[email protected]>
4 tasks
Introduce a first-class `container` path type in the file action mapper so a resolved container image path (e.g. a Singularity/Apptainer image on CVMFS) can be rewritten for the compute node via a `file_actions` `rewrite` rule with `path_types: container`, rather than overloading the `unstructured` tool-parameter path type. Like `unstructured`, `container` is not an ACTION_DEFAULT_PATH_TYPE, so its default action is `none` (no-op) and it is never staged -- container images live on CVMFS or in a registry. `PathMapper.check_for_container_rewrite()` applies a matching `rewrite` action and is a no-op otherwise. Consumed by Galaxy's PulsarComputeEnvironment.container_path_rewrite (dependent Galaxy PR galaxyproject/galaxy#23129). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds first-class support for running jobs under cvmfsexec, so Singularity/Apptainer containers (and Galaxy reference data) hosted on CVMFS can be used on compute nodes that cannot mount
/cvmfssystem-wide, without root or special privileges. I've already been using this with great effect by hackingenvexecutestatements in job destinations, but this is much better, plus it documents it and doesn't need a dirty hack for rewriting the container path whenmountrepomode is required.How it works
A
cvmfsexecoption on the job manager (app.yml), overridable per Galaxy destination:Two modes:
/cvmfsinside a mount namespace and the job runs inside it. No path rewriting needed.<job_directory>/.cvmfsexec/dist/cvmfs. The container image path (resolved by Galaxy against the real/cvmfs) is rewritten to the mounted location via an ordinary Galaxyfile_actionsrewriterule.The mount preamble is injected into the job script template (after dir prep); namespace mode wraps the command.
Dependent Galaxy PR
mountrepo mode's image-path rewrite depends on a companion Galaxy change (galaxyproject/galaxy#23129) that routes the resolved container image path through the compute environment's
unstructured_path_rewrite(so afile_actionsrewriterule applies to it, the same way it applies to tool parameters).Docs & tests
galaxyproject.cvmfsexecrole, a usegalaxy.org example), plus the mountrepo rewrite example in the Galaxy Configuration docs.setup_paramsoverride.