v0.19.0 — Runtime Modernization, fmx In-Container Utility, SSL Overhaul & Zero-Downtime Migrations

[Xieyt]

This major release spans the full stack: updates to the in-container fmx utility, replacement of certbot with acme.sh for SSL, replacement of pyenv/nvm with uv/fnm for Python/Node runtime management, Docker performance improvements, a completely reworked FM migration system, and an overhauled documentation site.

---

⚠️ Breaking Changes

1. Migration window: v0.18.0 minimum - v0.19.0 only migrates from v0.18.0+. Users on older versions must upgrade to v0.18.0 first.

2. Runtime: pyenv/nvm → uv/fnm - Old runtime directories are cleaned up during migration. Python versions are now managed by uv and stored per-bench at .uv/python/. Node versions are managed by fnm and stored per-bench at .fnm/.

3. SSL: certbot → acme.sh - All certificate operations now use acme.sh. No email required for Let's Encrypt. The certbot CLI dependency is removed entirely.

4. [ssl] config → [[ssl_certificates]] array - The old single [ssl] table is migrated to an array of [[ssl_certificates]] entries, supporting multiple certs per bench. preferred_challenge is renamed to challenge_type; credentials move to [dns_providers.cloudflare].

5. fm ssl delete → fm ssl remove - The delete subcommand is renamed.

6. --frappe-branch removed - Use --apps frappe:version-16 instead.

7. --ssl, --letsencrypt-email, --letsencrypt-preferred-challenge removed from create/update - SSL is now managed post-creation via fm ssl add.

8. --sync-config flag removed - Configuration sync is handled automatically.

9. Docker images updated to v0.19.0 - Global nginx-proxy updated from jwilder/nginx-proxy:1.6 to 1.11.

10. nginx SITENAME env var → SITE_MAPPINGS - Now a JSON mapping ({"sitename": "sitename"}) instead of a plain string.

11. Removed: Redis Queue Dashboard from admin tools; DisplayManager module from the host CLI (inlined into RichOutputHandler).

---

What's New

fmx - In-Container Utility (Updated)

fmx (originally fm-helper) is the existing in-container CLI that manages bench processes from within the Frappe Docker image. v0.19.0 upgrades it to Python 3.10, migrates its build system to Hatchling, locks dependencies, and ships several RQ fixes.

v0.19.0 changes to fmx:

• Upgraded runtime to Python 3.10 with locked dependencies (uv.lock)
• Migrated build system to Hatchling (pyproject.toml)
• Simplified unresolvable RQ job name extraction
• Fixed RQ job deserialization edge cases
• Improved restart behavior and process state handling

Commands (for reference):

Command	Description
fmx status [services...]	Supervisor process tree + RQ worker status (queue depths, worker states, suspend flag)
fmx start [services...]	Start services/processes via supervisor
fmx stop [services...]	Stop services with optional worker draining (--drain-workers, --worker-kill-timeout, --skip-stale-workers)
fmx restart [services...]	Restart services with optional zero-downtime Frappe app migration (see below)
fmx rq suspend	Set Redis rq:suspended flag - workers stop dequeuing new jobs
fmx rq resume	Clear the Redis suspend flag
fmx rq status [-v]	Show RQ suspend state, queue depths, per-worker details

How it works:

• Supervisor integration via XML-RPC over Unix sockets (/fm-sockets/*.sock)
• Parallel execution across services using ThreadPoolExecutor
• Worker draining: suspends RQ workers via Redis flag, polls until all workers are idle (configurable timeout + stale-worker detection)
• Warm shutdown: sends SIGUSR1 to worker processes for graceful job completion
• Fault handler: graceful recovery from supervisor SPAWN_ERROR and connection failures
• bench-wrapper.sh: intercepts bench restart/status/stop → delegates to fmx; wraps bench worker with a SIGUSR1 trap for graceful shutdown; bare bench with no args shows fmx helper commands

Zero-Downtime Frappe App Deployment (fmx restart --migrate)

fmx restart --migrate is a standalone production deployment tool inside the container. It is completely independent of fm migrate (FM infrastructure migration). Use it whenever you update a Frappe app and need to run Frappe's own database migrations (bench migrate) without dropping in-flight jobs or taking the site down hard.

Flow (fmx restart --migrate):

1. Optionally enable maintenance mode (--maintenance-mode)
2. Suspend RQ workers via Redis rq:suspended flag
3. Wait for all in-flight jobs to complete (configurable --worker-kill-timeout, --skip-stale-workers)
4. Run bench migrate (Frappe app DB schema migrations - not FM)
5. Restart all services
6. Resume RQ workers
7. Disable maintenance mode

Relevant flags:

Flag	Description
--migrate	Run bench migrate before restarting
--migrate-command	Override the migrate command (e.g. bench --site mysite migrate)
--drain-workers	Suspend + drain RQ before restart (implied by --migrate)
--worker-kill-timeout	Max seconds to wait for workers to go idle
--skip-stale-workers	Skip workers that appear stuck/stale
--maintenance-mode	Toggle maintenance_mode=1 in common_site_config.json during the flow

Note: This runs bench migrate (Frappe framework's app schema migration) - a completely different operation from fm migrate (Frappe Manager infrastructure upgrade). The two systems have zero overlap.

---

SSL Management Overhaul

New fm ssl subcommand tree:

Subcommand	Description
fm ssl add <bench> <domain>	Issue certificate for a bench domain
fm ssl add --standalone <domain>	Issue certificate for an external Docker project
fm ssl remove <bench> <domain>	Remove a bench certificate
fm ssl list [bench] / --all / --standalone	List certificates
fm ssl renew [bench] [domain]	Renew with --dry-run, --force, --all, --standalone
fm ssl acme-sh <args>	Direct passthrough to acme.sh for advanced operations
fm ssl dns-config	Configure DNS provider credentials (Cloudflare)

Challenge types:

• HTTP-01 (--challenge http01): Webroot via nginx-proxy. Requires domain to resolve to the server.
• DNS-01 (--challenge dns01): Cloudflare API. Supports --cname for CNAME delegation.

Standalone mode (--standalone): Manages SSL for any Docker project using FM's nginx-proxy via the fm-global-frontend-network. Configurations stored in external_domains.toml.

Other:

• Staging/dry-run support via FM_LETSENCRYPT_STAGING=1
• Pre-flight DNS validation (--skip-dns-check, --wait-for-dns)
• Renewal failures surfaced as errors; orphaned certs cleaned up on HTTPS config or reload failure

---

CLI / UX Improvements

New global flags:

Flag	Description
--non-interactive / -n	Disable all prompts (CI/CD mode)
--log-level debug|info|warning|error	Set log level (default: warnings/errors only)

fm create - new flags:

Flag	Description
--python	Pin Python version (e.g. 3.11)
--node	Pin Node version (e.g. 20)
--restart	Docker restart policy
--alias-domains	Comma-separated alias domains
--github-token / -t	GitHub token for private repos
--newrelic / --no-newrelic	Enable/disable NewRelic APM

fm update - new flags:

Flag	Description
--python	Update Python version
--node	Update Node version
--add-alias	Add alias domains (comma-separated)
--remove-alias	Remove alias domains (comma-separated)
--upload-limit	Set max upload size (e.g. 500M)
--restart	Update Docker restart policy
--newrelic / --no-newrelic	Toggle NewRelic APM

fm restart - new flags:

Flag	Description
--nginx	Restart nginx service
--container	Full container restart
--supervisor	In-container supervisor restart (faster, default)
--force	Force immediate restart

fm migrate - new command:

Flag	Description
--all-benches	Migrate all benches in one run
--auto-proceed	Skip confirmation prompts
--rerun	Re-apply migration steps
--on-failure prompt|archive|rollback	Action on failure
--skip-all-backup	Skip all backups (dangerous)
--skip-backup-for <bench>	Skip backup for a specific bench
--exclude-bench <bench>	Exclude from --all-benches run

fm shell - new flags:

Flag	Description
-c / --command	Execute command and exit
--bench-console	Open Frappe IPython console

Other:

• Interactive bench selection (fuzzy prompt when no bench name is given)
• Structured CLI examples on every --help via typer-examples
• fm info now shows Python and Node versions
• fm self compose - debug compose configuration
• fm services shell - shell into global service containers

---

Architecture & Configuration

• Commands reorganized into frappe_manager/commands/{ssl,self,services}/ module structure
• Global output handler registry: get_global_output_handler / set_global_output_handler with LoggingOutputHandler wrapper
• Bench class modularized into app_manager, certificate_manager, docker_ops, supervisor, ssl, orchestrator sub-objects
• App cloning: monorepo support, parallel cloning, multi-auth fallback (SSH → HTTPS → token)
• Docker client: new BenchOrchestrator, auto-streaming, os.execvp for direct docker compose execution (removes subprocess overhead)
• Unique database names per bench (fm_{name}_{token_hex(8)})
• Version source of truth moved to frappe_manager/__about__.py
• pyproject.toml migrated to PEP 621 format with hatchling build system
• Alias domains: primary domain + multiple aliases, each with automatic SSL

---

FM Migration System (Infrastructure Upgrade)

This is FM's own upgrade mechanism - it transforms config files, Docker images, and runtime tooling when upgrading FM from v0.18.0 → v0.19.0. It has nothing to do with bench migrate (Frappe app DB migrations) or fmx.

Two-tier architecture:

• Infrastructure migration (global services, CLI config) - checked on every fm invocation (except whitelisted commands: list, self compose, self update-images, migrate); user is prompted to update inline
• Per-bench migration - opt-in via fm migrate <benchname>

What fm migrate does per-bench (all performed outside the container via Docker Compose and config file edits):

Step	What happens
Config transform	[ssl] → [[ssl_certificates]], rename fields, move DNS creds, add alias_domains / upload_limit / restart_policy / use_uv
Docker Compose update	Image tags v0.18.0 → v0.19.0, SITENAME → SITE_MAPPINGS, add restart policies
Image pull	Pull updated Docker images if tags changed
Upload limit	Apply client_max_body_size across vhostd, site_config.json, nginx custom config
Runtime rebuild	pyenv/nvm → uv/fnm (auto-detects Python/Node versions from the running container)
Supervisor regen	Regenerate supervisor config from FM template
Container restart	Force-recreate containers via docker compose up --force-recreate
Backup & rollback	Per-bench backup of config, compose, DB, supervisor, nginx, env, common_site_config.json, and per-site site_config.json before any change

fm migrate flags:

Flag	Description
--all-benches	Migrate all benches in one run
--auto-proceed	Skip confirmation prompts
--rerun	Re-apply migration steps (re-runs config transforms and supervisor regen; runtime rebuild only skipped if env is already current)
--on-failure prompt|archive|rollback	What to do when a bench fails
--skip-all-backup	Skip all backups (dangerous)
--skip-backup-for <bench>	Skip backup for a specific bench
--exclude-bench <bench>	Exclude from --all-benches run

---

Docker & Performance

• exec-entrypoint.sh: lightweight entrypoint for one-off commands - skips full init, uses numeric gosu UID/GID to eliminate the ~16s usermod/groupmod delay
• Removed recursive /opt chown - was a major slow path on every startup
• wkhtmltopdf moved to base Docker stage - available in all image variants
• Build cache per architecture - faster multi-arch CI builds
• .uv / .fnm relocated to frappe-bench/ (per-bench isolation, cleaner home dir)
• Multi-stage Dockerfile: base (Ubuntu 22.04 + system deps + wkhtmltopdf) → tooling (fnm, uv, fmx) → frappe (final)
• Pre-baked runtimes: Python 3.14.2 + Node 24.11.0 in the Docker image for faster first-site creation
• PyPI publish workflow: automated publish on v* tag push

---

Documentation Overhaul

• Migrated to GitHub Pages with versioned docs (MkDocs + mike)
• New guides: fmx.md, ssl.md, backup-restore.md, environments.md, python-node-versions.md, upload-limits.md, admin-tools.md, app-management.md, external-database.md
• Auto-generated CLI reference from live Typer app via scripts/update_cli_docs.py
• SCSS-based styling with rtCamp branding
• CLI examples standardized across all commands using typer-examples

---

Tests Added

• Comprehensive SSL manager unit tests (add/renew/remove lifecycle)
• Migration flow integration tests (--rerun, --on-failure flags)
• fmx worker kill parameter tests
• Global output handler autouse fixture in conftest.py
• Docker Compose config tests
• Contextual logging tests

---

Bug Fixes

• Heavyweight ctx.obj assignments (services, fm_config_manager, Docker daemon check, migration checks, image pulling) guarded behind help_called check - fixes crashes on --help
• export_to_toml raises on write failure instead of silently returning False
• Empty FRAPPE_MANAGER_HOME treated as unset
• Venv backup directories get timestamps (prevents name collisions)
• RQ queue status filtering fixed (was reporting incorrect idle/active state)
• All services (not just workers) restarted on migration failure
• Mirror selection curl calls now have timeouts; validates HTTP 200 before use
• Subprocess output decoding hardened (PYTHONUNBUFFERED=1, UTF-8 fallback)
• Orphaned certs cleaned up on HTTPS config or nginx reload failure
• Renewal failures surfaced as errors (not warnings)
• Spinner suppression in non-interactive mode
• New LiveAwareRichHandler wraps RichHandler to prevent log output colliding with Rich Live displays
• SocketIO uses fnm-managed node path instead of system node

---

Upgrading from v0.18.0

# 1. Upgrade fm CLI
uv tool install frappe-manager==0.19.0

# 2. Run infrastructure migration (global services, nginx-proxy, MariaDB)
fm migrate --auto-proceed

# 3. Migrate each bench (rebuilds runtime, transforms config)
fm migrate mybench --auto-proceed

# Or migrate all benches at once with automatic rollback on failure
fm migrate --all-benches --auto-proceed --on-failure=rollback

Notes: Migration creates automatic backups (config, compose, DB, supervisor, nginx). Runtime rebuild takes ~5–10 min per bench. SSL certificates and database data are preserved.

To re-apply migration steps without a full runtime rebuild: fm migrate mybench --rerun

---

Full changelog

See docs/changelog.md for the complete commit history.

Compare: v0.18.0...v0.19.0