diff --git a/docs/src/experimental-composefs.md b/docs/src/experimental-composefs.md index dd8d1427f..7c844d4b4 100644 --- a/docs/src/experimental-composefs.md +++ b/docs/src/experimental-composefs.md @@ -5,14 +5,34 @@ do provide feedback on them. ## Overview -The composefs backend is an experimental alternative storage backend that uses [composefs-rs](https://github.com/containers/composefs-rs) instead of ostree for storing and managing bootc system deployments. +The composefs backend is an experimental alternative storage backend that uses [composefs-rs](https://github.com/composefs/composefs-rs) instead of ostree for storing and managing bootc system deployments. -**Status**: Experimental. The composefs backend is under active development and not yet suitable for production use. The feature is always compiled in as of bootc v1.10.1. +**Status**: Experimental, but close to stabilization! We are committed to in-place upgrades from all systems deployed since bootc 1.16.0. -A key goal is custom "sealed" images, signed with your own Secure Boot keys. -This is based on [Unified Kernel Images](https://uapi-group.org/specifications/specs/unified_kernel_image/) -that embed a digest of the target container root filesystem, typically alongside a bootloader (such -as systemd-boot) also signed with your key. +The composefs backend supports two distinct levels of integrity guarantee, controlled by whether fsverity is strictly enforced on the root filesystem (i.e. whether the image was built with `--allow-missing-verity`): + +- **Sealed**: The composefs digest is baked into the kernel command line of a UKI and *required* to match at boot. +- **Unsealed**: fsverity enforcement is optional, so composefs still provides content-addressed, deduplicated storage and garbage collection, but without a guarantee that the root filesystem matches what was signed. Unsealed composefs most commonly boots via a traditional `vmlinuz`/`initramfs.img` and a BLS boot entry, but a UKI built with `--allow-missing-verity` is *also* unsealed in this sense — packaging as a UKI is a boot convenience here, not by itself a security boundary. See [Bootloader Support](#bootloader-support) below. + +## Storage and repository structure + +Unlike the ostree backend, which keeps its repository at `/ostree/repo`, the composefs backend splits its on-disk state across two top-level directories in the physical sysroot: + +- `/composefs`: The [composefs-rs repository](https://github.com/composefs/composefs-rs/blob/main/crates/composefs/src/repository_format.rs) (mode `0700`), containing: + - `objects/`: content-addressed file storage, keyed by SHA-512 fsverity digest and shared via reflink (`FICLONE`) where the filesystem supports it + - `images/`: EROFS images describing each deployment's root filesystem metadata + - `streams/`: OCI manifest, config, and layer splitstreams captured during image pulls + - `bootc/storage/`: the `containers-storage:` instance backing logically bound images, reflink-shared with the composefs object store +- `/state/deploy//`: Persistent per-deployment state, one directory per deployment (named after its composefs digest): + - `etc/`: a writable copy of the deployment's `/etc`, bind-mounted onto the booted root's `/etc` + - `var`: a symlink to the shared `/state/os/default/var`, bind-mounted onto the booted root's `/var` + - `.origin`: an INI file recording the image reference, boot type (BLS or UKI) and digest, and the OCI manifest digest (the latter is what keeps a deployment's objects alive across garbage collection) + +Although composefs-rs supports other fsverity hash algorithms, bootc currently hardcodes `SHA-512` for the repository (see `Algorithm::SHA512` at every repository init/open call site, and the `ComposefsRepository` type alias in `crates/lib/src/store/mod.rs`). This is why deployment and object identifiers throughout this document (and in `bootc status`) are 128-character hex strings. + +There is no `/ostree/repo`; the composefs backend doesn't use the ostree repository at all. A minimal `/ostree` directory is still created, but only to hold a compatibility symlink (`ostree/bootc -> ../composefs/bootc`) so that existing tooling expecting `/usr/lib/bootc/storage` to resolve through `ostree/bootc` keeps working. + +Transient, not-yet-finalized deployment state (used while staging an update before reboot) lives under `/run/composefs/staged-deployment` and is never persisted to disk. ## How Sealed Images Work @@ -61,51 +81,78 @@ valid use case is to temporarily disable it in order to test a change locally on e.g. one machine, then re-enable it later. However at the current time it is not yet streamlined to regenerate the UKI locally. -### Build Pattern: Compute Digest and Generate UKI in One Stage +Note this is a different, independent weakening from `--allow-missing-verity` +(see [Overview](#overview) above): disabling Secure Boot only removes firmware +verification of the UKI's own signature, while the fsverity digest of the root +filesystem is still enforced. Building with `--allow-missing-verity` instead +disables that digest enforcement itself, which is what actually makes an image +unsealed regardless of whether Secure Boot is enabled. + +### Build Pattern: Split the Kernel, Then Generate the UKI in a Separate Stage -The key to building sealed images is using a multi-stage Dockerfile where a separate stage mounts the target rootfs, computes its composefs digest, and generates the signed UKI in one step: +Building a sealed image involves three stages: build the rootfs, split the kernel and initramfs out of it, and generate the signed UKI from the split rootfs in a tools stage: ```dockerfile # Build your rootfs with all packages and configuration FROM as rootfs RUN apt|dnf|zypper install ... && bootc container lint --fatal-warnings +# Split the kernel and initramfs out of the rootfs. This moves +# /usr/lib/modules//{vmlinuz,initramfs.img} into /kernel//, +# since for a sealed image they end up embedded in the UKI instead. +FROM rootfs as split +RUN mkdir /kernel && bootc container split-kernel-and-rootfs --rootfs / --output /kernel + # Generate the sealed UKI in a tools stage FROM as sealed-uki -RUN --mount=type=bind,from=rootfs,target=/target \ +RUN --mount=type=bind,from=split,target=/target \ + --mount=type=bind,from=split,source=/kernel,target=/kernel \ --mount=type=secret,id=secureboot_key \ --mount=type=secret,id=secureboot_cert <`) -4. The final stage copies the signed UKI into the rootfs without modifying any files used in the digest calculation +1. `bootc container split-kernel-and-rootfs` removes the raw kernel and initramfs from the rootfs ahead of time, so the final image never carries a duplicate copy of them (they end up embedded in the UKI instead) +2. `bootc container ukify` handles computing the composefs digest and assembling the kernel command line, so you only need to pass `ukify`-specific options (like signing) after `--` +3. The final stage copies the signed UKI into the already-split rootfs + +### The `bootc container ukify` Command + +```bash +bootc container ukify --rootfs [OPTIONS] -- [UKIFY_ARGS...] +``` + +This is the recommended way to build a UKI for a bootc image. It computes the composefs digest of `--rootfs` (using the lower-level `compute-composefs-digest` primitive described below), reads extra kernel arguments from `/usr/lib/bootc/kargs.d`, and invokes the system `ukify` binary with the resulting cmdline. Anything after `--` is forwarded to `ukify` unchanged (e.g. `--output`, `--signtool`, signing key/cert options). + +**Options:** + +- `--rootfs `: Root filesystem to operate on (default: `/`) +- `--kernel-dir `: Directory containing `vmlinuz`/`initramfs.img`, named `/parent/`. Needed when the kernel has already been split out of `--rootfs`, e.g. via `split-kernel-and-rootfs` +- `--allow-missing-verity`: Make fsverity validation optional, for filesystems that don't support it (e.g. XFS) +- `--write-dumpfile-to `: Write a composefs dumpfile for debugging ### The `bootc container compute-composefs-digest` Command @@ -113,7 +160,7 @@ This pattern works because: bootc container compute-composefs-digest [PATH] ``` -Computes the composefs digest for a filesystem. The digest is a 128-character SHA-512 hex string that uniquely identifies the filesystem contents. +A lower-level primitive, used internally by `ukify` above, that computes just the composefs digest for a filesystem without building a UKI. The digest is a 128-character SHA-512 hex string that uniquely identifies the filesystem contents. Useful for scripting or debugging outside of the UKI build flow. **Options:** @@ -146,7 +193,9 @@ See [CONTRIBUTING.md](https://github.com/bootc-dev/bootc/blob/main/CONTRIBUTING. ## Bootloader Support -To use sealed images, the container image must have a UKI and systemd-boot installed (and not have `bootupd`). If these conditions are met, bootc will automatically detect and use the composefs backend during installation. +Whenever the container image has a UKI, bootc automatically selects the composefs backend during installation (see [Prerequisites](#prerequisites) above for the currently-supported UKI + systemd-boot configuration for building sealed images). Note that having a UKI does not by itself make an install sealed — that also depends on whether fsverity enforcement is on, per [Overview](#overview) above. + +Composefs installs using a traditional `vmlinuz`/`initramfs.img` layout instead of a UKI are always unsealed, and can use either `bootupd` (GRUB) or systemd-boot, the same as the ostree backend. See [bootloaders.md](bootloaders.md) for the general bootloader selection rules. Under the hood, bootc writes standard BLS boot entries for both UKI and traditional kernels; see the [composefs boot module documentation](https://github.com/bootc-dev/bootc/blob/main/crates/lib/src/bootc_composefs/boot.rs) for details on how entry filenames and sort-keys are chosen to sort correctly on both GRUB and systemd-boot. ## Installation @@ -156,33 +205,24 @@ There is a `--composefs-backend` option for `bootc install` to explicitly select The composefs backend is experimental; on-disk formats are subject to change. -### Deployment blockers +### Important -- [Garbage collection](https://github.com/bootc-dev/bootc/pull/2040): In progress - Extended install APIs: Ability to cleanly implement anaconda %post and osbuild post mutations and general post-install pre-reboot; right now some tools just mount the deployment directory (note this one also relates to [APIs in general](https://github.com/bootc-dev/bootc/issues/522)) -- [OCI registry install](https://github.com/bootc-dev/bootc/issues/1703): Installing from registry can fail due to config mismatch (suggestion: just clean reject v2s2) -- [composefs-rs repository finalization](https://github.com/bootc-dev/bootc/issues/1320) +- [Dual EROFS v1/v2 generation](https://github.com/bootc-dev/bootc/pull/2248): `bootc install` currently generates both EROFS v1 and v2 images for every deployment, since v1 is required on older kernels (e.g. RHEL 9/CentOS Stream 9) while v2 is the modern native format. The overhead is small since EROFS images only hold metadata, but this dual generation is a stopgap until we can drop v1 support. -### Important +## Related issues -- Extended test suite: Right now we're not covering upgrades well, need to build upgrade image in sealed cases -- Full workflow test - add composefs into https://gitlab.com/fedora/bootc/tests/bootc-workflow-test for example - - Workflow upgrades especially "from old systems" - [Unified storage](https://github.com/bootc-dev/bootc/issues/20): Not strictly a blocker but a really nice to have - [Sealed image build UX](https://github.com/bootc-dev/bootc/issues/1498): Streamlined tooling for building sealed images - In place transitions: - First: support [factory reset](https://github.com/bootc-dev/bootc/issues/404) from ostree to composefs - Next: Support copying /etc and /var -- A lot more practical level docs for using this - -### Minor - -- Remove `/usr/lib/bootc/kargs.d` as part of UKI creation (also `bootc container inspect` should show UKI kargs) ## Additional Resources - See [filesystem.md](filesystem.md) for information about composefs in the standard ostree backend - See [bootloaders.md](bootloaders.md) for bootloader configuration details -- [composefs-rs](https://github.com/containers/composefs-rs) - The underlying composefs implementation +- [composefs-rs](https://github.com/composefs/composefs-rs) - The underlying composefs implementation +- [composefs-rs repository format](https://github.com/composefs/composefs-rs/blob/main/crates/composefs/src/repository_format.rs) - Detailed on-disk layout of the `/composefs` repository - [Unified Kernel Images specification](https://uapi-group.org/specifications/specs/unified_kernel_image/) - [ukify documentation](https://www.freedesktop.org/software/systemd/man/latest/ukify.html) - Tool for building UKIs