Framework for building IEDB-style command-line scientific tools — standardized argument parsing, dependency wiring, and reproducible tarball builds.
After installing the framework, you get a cli command and a Python class (NGArgumentParser) that together produce well-shaped scientific CLI apps:
- A standardized
preprocess → predict → postprocessworkflow shared across every app - A reproducible build that emits a self-contained tarball for HPC distribution
- Two-tier dependency management:
uvfor Python packages,cli deps+paths.pyfor external binaries / HPC modules / library paths - A clean ownership boundary: framework files live in
core/subdirectories and get sync-managed; everything else is yours
ngargparser uses uv to manage Python versions and virtual environments. Install it first:
curl -LsSf https://astral.sh/uv/install.sh | shYou don't need to create a Python venv yourself — uv creates a .venv/ for each scaffolded app and downloads the right Python version on demand. Skip the usual python -m venv step.
# Best option: install as an isolated uv tool, `cli` on PATH globally.
# Using SSH (recommended for IEDB devs with SSH keys configured):
uv tool install 'git+ssh://[email protected]/iedb/tools/tools-redesign/global-dependencies/ngargparser.git'
# Using HTTPS (requires token-based auth for private repos):
uv tool install 'git+https://gitlab.lji.org/iedb/tools/tools-redesign/global-dependencies/ngargparser.git'
# Pin to a tag for reproducibility (SSH form shown; HTTPS works the same way):
uv tool install 'git+ssh://[email protected]/iedb/tools/tools-redesign/global-dependencies/[email protected]'
# Upgrade later (either works)
cli upgrade # self-update to the latest release tag on GitLab
uv tool upgrade ngargparseruv pip install -e .
pip install .cli generate my-app # scaffold
cd my-app
uv sync # install Python deps
uv run python src/run_my_app.py predict --help # see what your app does
make build # build/IEDB_NG_MY-APP-local.tar.gzWhat just happened:
cli generatescaffolded a fresh project tree atmy-app/. Non-interactive: no prompts, scriptable.uv syncreadpyproject.toml, locked deps intouv.lock, installed them into.venv/.uv run python …ran the entry script through the project venv. Out of the box your app understandspreprocess,predict, andpostprocesssubcommands.make buildran the build pipeline (copy + symlink, run hooks, tarball). Output ships to HPC.
When you need an external tool dep:
cli deps add mhci-predictor pepx # writes stub blocks to paths.py
$EDITOR paths.py # fill in actual binary paths
./configure # generate per-tool setup_<name>_env.sh + .envmy-app/
├── Makefile # framework-owned (sync overwrites)
├── configure # framework-owned executable
├── pyproject.toml # Python deps + project metadata + scaffold_version stamp
├── uv.lock # locked deps (commit this)
├── README # your app's user-facing README
├── license-LJI.txt
├── paths.py # declares external tool deps (yours)
├── src/
│ ├── core/ # framework-owned (sync overwrites) — DO NOT EDIT
│ │ ├── NGArgumentParser.py
│ │ ├── core_validators.py
│ │ ├── configure.py
│ │ └── set_pythonpath.py
│ ├── run_my_app.py # entry script (yours)
│ ├── MyAppArgumentParser.py # subclass of NGArgumentParser (yours)
│ ├── preprocess.py # your input prep logic
│ ├── postprocess.py # your aggregation logic
│ └── validators.py # your custom validators
├── scripts/
│ ├── core/ # framework-owned (sync overwrites)
│ │ └── build.sh # build pipeline
│ ├── build.conf # build knobs (yours)
│ ├── hooks.sh # imperative build hook (yours)
│ └── do-not-distribute.txt # exclusion list (yours)
└── deploy/
└── install.sh # imperative deploy hook (yours, runs on target host)
Ownership rule: anything inside a core/ subdirectory is framework-owned — cli sync overwrites those files when you upgrade. Everything else is yours; sync never touches it.
Every generated app supports three subcommands out of the box:
python src/run_my_app.py preprocess -j input.json -o output-dir/
python src/run_my_app.py predict -j input.json -o output-file
python src/run_my_app.py postprocess -j job_descriptions.json -p final-results/| Stage | Role | Where you customize |
|---|---|---|
preprocess |
Validate inputs, split work into job units, emit job_descriptions.json |
src/preprocess.py |
predict |
Run the core analysis on a single job unit | src/MyAppArgumentParser.py (args) + src/run_my_app.py (logic) |
postprocess |
Merge per-job results into a single output | src/postprocess.py |
All three subparsers come with built-in arguments so you don't redefine them: preprocess/postprocess bring input/output paths and validation flags, and predict brings the output arguments (--output-prefix/-o, --output-format/-f). predict is where you add your tool-specific input options.
The argument parser for your app lives in src/MyAppArgumentParser.py and subclasses NGArgumentParser:
import textwrap
import argparse
from core.NGArgumentParser import NGArgumentParser
class MyAppArgumentParser(NGArgumentParser):
def __init__(self):
super().__init__()
self.description = textwrap.dedent("""
One-line description of your app.
""")
self.parser_predict = self.add_predict_subparser(
help='Run a single prediction.',
description='Multi-line help that survives formatting.',
formatter_class=argparse.RawDescriptionHelpFormatter,
)
# --output-prefix/-o and --output-format/-f are already provided by the
# base class — add your tool-specific *input* args here instead.
self.parser_predict.add_argument(
"--threshold", "-t",
dest="threshold",
help="Score threshold",
metavar="THRESHOLD",
group="prediction options", # same group label clusters args in --help
)Common patterns:
- Argument grouping — pass
group="<label>"to anyadd_argumentcall. - Custom validators — define a function in
src/validators.pythat raisesargparse.ArgumentTypeErroron bad input, then pass it astype=...onadd_argument. - Updating inherited arguments — to change a default or help text on an argument from the parent class, call
parser.update_arguments(...)instead of redefining.
The full parser API is in Reference at the bottom.
ngargparser separates two kinds of dependencies and uses different tools for each:
| Kind | Manage with | Manifest |
|---|---|---|
| Python packages (numpy, pandas, …) | uv |
pyproject.toml + uv.lock |
External binaries / HPC modules / LD_LIBRARY_PATH |
cli deps + manual paths.py edits |
paths.py |
uv sync # install locked deps into .venv/
uv add pandas # add a dep, update pyproject.toml + uv.lock
uv remove pandas # remove
uv run python src/run_my_app.py --help # run inside the project venvCommit uv.lock so deploys are reproducible.
cli deps add mhci-predictor pepx # write stub blocks to paths.py
cli deps remove pepx # remove (alias: rm)
cli deps list # show declared deps + which paths are filled in (alias: ls)
cli deps # bare → interactive add/remove menucli deps add writes a stub: <name>_path = None, <name>_venv = None, <name>_module = None, <name>_lib_path = None. You then edit paths.py to fill in actual paths and run ./configure to regenerate setup_<name>_env.sh shell scripts and the project's .env.
cli deps remove accepts the original name (mhci-predictor), the display form (Mhci Predictor), or the var-name form (mhci_predictor).
make build # produce build/IEDB_NG_<TOOL>-<VERSION>.tar.gz
make build-verbose # same, with full build output (no progress bar)
make clean # remove build/Pipeline (under the hood, scripts/core/build.sh):
- Set up
build/<TOOL>-<VERSION>/ - Copy or symlink the source tree into the build dir (symlink by default; copy items listed in
EXCLUDE_FROM_BUILD_SYMLINK— defaults tolibs run_*.py) - Run
scripts/hooks.sh(your imperative hook) - Tar it up
This is your imperative escape hatch. It runs after the source tree is in place but before the tarball is created. Working dir: build/<TOOL>/libs/. Available env vars: BUILD_DIR, PROJECT_ROOT, APP_NAME, TOOL_NAME, TOOL_VERSION, TOOL_DIR, SRC_DIR.
Use it for: vendoring git deps, downloading archives, copying local binaries, code generation, patching files, signing, validation — anything bash can do.
# Items to COPY instead of symlink (glob-matched). Default: "libs run_*.py".
EXCLUDE_FROM_BUILD_SYMLINK="libs run_*.py"
# Tarball filename prefix.
TARBALL_PREFIX="IEDB_"build.conf is user-owned — sync never touches it.
make build produces a tarball; nxg-tools-deployments (an internal IEDB orchestrator at gitlab.lji.org/iedb/tools/tools-redesign/nxg-tools-deployments) takes it from there. The full lifecycle:
developer GitLab CI target server
───────── ───────── ─────────────
git push ──────────────► make build ──────► (tarball lands here)
tarball as
CI artifact
│
│ nxg-tools-deployments fetches
▼
(extract tarball) ──► bash deploy/install.sh
(flip `current` symlink on success)
make build produces build/IEDB_NG_<TOOL>-<VERSION>.tar.gz. The orchestrator's only contract: the tarball top-level must contain deploy/install.sh and a README file. Both are scaffolded by cli generate and pass through the build pipeline unchanged.
Runs on the target host after the tarball is extracted. Working dir is the extracted tarball. The default scaffold installs uv if needed, runs uv sync to materialize the .venv, then runs ./configure if present.
The orchestrator passes five env vars (IEDB_STANDALONE_NAME, IEDB_VERSION, IEDB_VERSION_DIR, IEDB_STANDALONE_DIR, IEDB_PREVIOUS_VERSION_DIR) — most install scripts ignore them and resolve everything from ${BASH_SOURCE[0]}. The phbr install.sh is the exception: it uses IEDB_STANDALONE_DIR to validate and wire in peer tools (tcell_mhci, tcell_mhcii).
Do not hardcode UV_PYTHON_INSTALL_DIR / UV_CACHE_DIR / UV_PYTHON_PREFERENCE in install.sh. Those come from the target's install_shell_init snippet (sourced by the orchestrator before invoking install.sh) and let the cluster share a single /apps/uv/ cache across users. Hardcoding them locks cluster paths into every release tarball and breaks dev-laptop installs.
The two *.sh user-owned files are easy to confuse. They run at different times in different places:
scripts/hooks.sh |
deploy/install.sh |
|
|---|---|---|
| When | Build time (developer / CI) | Deploy time (target server) |
| Invoked by | scripts/core/build.sh |
nxg-tools-deployments orchestrator |
| Purpose | Pull stuff INTO the tarball (vendor git deps, codegen, patch source) | Set up the extracted tarball ON the server (uv sync, configure, edit paths.py) |
Two commands, two different targets — cli upgrade updates the tool; cli sync updates a project.
| You want to… | Run | Where |
|---|---|---|
Update the cli tool itself (to get the latest, or before cli generate-ing a new project) |
cli upgrade |
anywhere |
| Bring one project's framework files up to the installed tool's version | cli sync |
inside that project |
They overlap in exactly one spot: cli sync self-upgrades the tool first (unless --no-upgrade), so the common "get my project on the latest framework" flow is a single command. That convenience is the only reason the two look similar — cli upgrade is still the only way to update the tool without being in (and rewriting) a project, and it's also what --check / --ref <version> hang off of.
After a successful cli upgrade, it reminds you to run cli sync — because cli upgrade updates the CLI, not your projects. Run from inside a project that's behind, it names both versions ("You upgraded the CLI to Y, but this project is still on X — run cli sync here to update this project to Y"), even when the tool itself was already current. Run from anywhere else, it prints a generic reminder to cli sync your projects.
-
cli upgrade— self-updates the installedngargparsertool (thecliitself) to the latest release tag on GitLab. This is the one command to update the CLI (0.2.4+) — it auto-detects how ngargparser was installed (uv tool, pip venv, or a pip-less uv venv) and runs the right installer, so you never type a longgit+https://…command. Works from any directory and touches no project files. Add--checkto see installed-vs-latest without installing.cli upgrade --check # is a newer release available? cli upgrade # update the cli to the latest tag (alias: u)
Upgrading from a pre-0.2.4 version (one-time bootstrap).
cli upgradedidn't exist before 0.2.4, so to get onto it the first time, reinstall the package directly with whatever installed it (you only do this once — after that it's justcli upgrade):# uv tool install: uv tool install --force --reinstall 'git+https://gitlab.lji.org/iedb/tools/tools-redesign/global-dependencies/[email protected]' # pip venv: pip install --upgrade --force-reinstall 'git+https://gitlab.lji.org/iedb/tools/tools-redesign/global-dependencies/[email protected]' # pip-less uv venv: uv pip install --reinstall 'git+https://gitlab.lji.org/iedb/tools/tools-redesign/global-dependencies/[email protected]'
Update notifier. You don't have to remember to check — when you run
cli, it occasionally prints a one-line "a newer version is available" notice (throttled to one remote check per day, shown only in an interactive terminal, never in CI/pipelines). Silence it withNGARGPARSER_NO_UPDATE_CHECK=1.Install from a local checkout (contributors).
cli upgrade --ref .installs ngargparser from your current working tree instead of a release tag — handy while developing the framework itself. Pass.(run from the repo) or a path to a checkout;--checkdry-runs it:cli upgrade --ref . # install the working tree (run from the ngargparser repo) cli upgrade --ref /path/to/repo # same, from anywhere cli upgrade --ref . --check # show what it would install, install nothing
-
cli sync— refreshes the framework-owned files inside a project to match the installed version (and self-upgrades first, unless--no-upgrade). Run it from inside an existing project:
cd path/to/your-app
cli synccli sync only touches framework-owned files. It refreshes:
src/core/*.pyscripts/core/build.sh- the root
Makefile - the
[tool.ngargparser] scaffold_versionstamp inpyproject.toml - the README badge (flips to green to mark "synced")
It never overwrites validators.py, <App>ArgumentParser.py, paths.py, hooks.sh, build.conf, or anything else in user-owned space. It also migrates legacy filenames from older framework versions (e.g., scripts/build.sh → scripts/core/build.sh, scripts/dependencies.sh → scripts/hooks.sh) on first run.
For breaking changes — file layouts shifting, schema changes, removed APIs — sync keeps your core files current but won't transform user-edited content. The right move there: read the release notes, scaffold a fresh project with the new framework, and port your application code over.
cli --version
cli --help
cli generate <name> # scaffold a new project (alias: g)
cli generate example # scaffold the aa-counter example app
cli deps # interactive add/remove menu (alias: d)
cli deps add <name> ... # add stub blocks to paths.py
cli deps remove <name> ... # remove blocks (alias: rm)
cli deps list # show declared deps + status (alias: ls)
cli sync # pull latest framework files into a project (alias: s)
cli upgrade # self-update the cli to the latest release tag (alias: u)
cli upgrade --check # report installed vs latest without installing
NGArgumentParser auto-creates three subparsers: preprocess, predict, postprocess. All three come with built-in arguments. predict's output arguments (--output-prefix/-o, --output-format/-f) are framework-owned; you add the tool-specific input arguments in your subclass.
--input-json / -j JSON_FILE (required)
--output-dir / -o DIR (required)
--params-dir DIR (default: $OUTPUT_DIR/predict-inputs/params)
--inputs-dir DIR (default: $OUTPUT_DIR/predict-inputs/data)
--assume-valid (skip validation)
--output-prefix / -o STR
--output-format / -f {tsv,json} (default: tsv)
Predict output is serialized by core.result_writer.write_results (see Result output below): tsv to stdout when no -o is given, otherwise <prefix>.<ext>. These two arguments come from the base class — add your tool-specific input arguments in the subclass.
--job-desc-file / -j JSON ─┐ pick exactly one
--input-results-dir / -i DIR ─┘
--postprocessed-results-dir / -p DIR (required)
--output-prefix / -o STR
--output-format / -f {tsv,json} (default: json)
Every subparser exposes .help and .description as plain attributes:
self.parser_preprocess.help = 'Custom preprocess help'
self.parser_preprocess.description = 'Detailed preprocess instructions'self.parser_predict.add_argument(
"--threshold", "-t",
help="Score threshold",
group="prediction options", # args sharing this label cluster in --help
)Some arguments are provided by the base class — notably predict's --output-prefix/-o
and --output-format/-f. To tweak one for your tool (change its default, help text,
choices, or group), call update_arguments(...) — it edits the inherited argument in
place:
# e.g. make this tool default to json instead of the framework-wide tsv:
self.parser_predict.update_arguments(
"--output-format", "-f",
default="json",
help="Updated output format description",
group="modified options",
)The change is local to your tool and doesn't affect others. An explicit flag on the
command line (-f tsv) still overrides whatever default you set.
⚠️ Useupdate_argumentsfor this — do not re-declare an inherited argument with a freshadd_argument("--output-format", ...). That raisesconflicting option strings. (To replace one entirely,remove_argument(...)first, thenadd_argument(...).)
Every tool emits the same result envelope and serializes it through the shared,
framework-owned helper core.result_writer.write_results (synced into src/core/result_writer.py):
from core.result_writer import write_results
result = {
"warnings": [], "errors": [],
"results": [
{"type": "peptide_table",
"table_columns": ["peptide", "score"],
"table_data": [["ADMGHLKY", 0.5]]},
# ... more tables ...
],
}
write_results(result, args.output_prefix, args.output_format)- Format (
-f):tsv(default) orjson. tsv is flat and pipeable; json preserves the full envelope (warnings/errors and any extra per-table metadata). - Destination: no
-o→ stdout;-o <prefix>→ file(s). - Multiple tables: on stdout each table is prefixed with a
--- <type> ---banner; to file each is written as<prefix>.<type>.tsv. A single table gets no banner and a single<prefix>.tsv. - json always writes the whole envelope verbatim (
<prefix>.jsonor stdout).
predict defaults to tsv; postprocess defaults to json because the aggregated
envelope carries metadata that tsv can't represent. warnings/errors are echoed to
stderr for tsv so stdout stays a clean data stream.
from core.core_validators import (
validate_file, # file exists + readable
validate_directory, # directory exists or can be created
validate_directory_given_filename, # parent dir of a path
validate_preprocess_dir, # special preprocessing dir setup
)# src/validators.py
def validate_peptide_length(value):
try:
length = int(value)
except ValueError:
raise argparse.ArgumentTypeError("peptide length must be a number")
if not 1 <= length <= 50:
raise argparse.ArgumentTypeError("peptide length must be 1–50")
return length
# src/MyAppArgumentParser.py
self.parser_predict.add_argument(
"--peptide-length",
type=validate_peptide_length,
)| Path | Ownership |
|---|---|
src/core/* |
Framework — cli sync overwrites |
scripts/core/* |
Framework — cli sync overwrites |
Makefile |
Framework — cli sync overwrites |
pyproject.toml [tool.ngargparser] scaffold_version |
Framework — sync stamps |
pyproject.toml (everything else) |
You |
uv.lock |
You (regenerated by uv commands) |
paths.py |
You (cli deps writes stubs; you fill in values) |
scripts/hooks.sh |
You |
scripts/build.conf |
You |
scripts/do-not-distribute.txt |
You |
deploy/install.sh |
You — sync creates from template if missing, never overwrites |
src/run_*.py, src/<App>ArgumentParser.py, src/preprocess.py, src/postprocess.py, src/validators.py |
You |