install: Add `to-filesystem --[re]assemble-root`

[cgwalters]

xref #542 and https://bootc.dev/bootc/bootc-install.html#postprocessing-after-to-filesystem

---

Following is LLM-generated design doc:

Details

# Design: `bootc install` root-assembly flags (`--assemble-root` / `--reassemble-root`)

0) Brief rationale

Installer tooling (primarily Anaconda) needs to run kickstart-style %post
customization against a freshly-installed-but-unbooted bootc system, without
rebooting first. That requires a fully assembled root (usr+etc+var
merged, correctly labeled) mounted somewhere chroot()-able.

Today this only sort of works, and only for the ostree backend, via
Anaconda's own fragile heuristics (scanning the physical root for the
deployment checkout directory). It doesn't work at all for the composefs
backend, which has no checkout tree — just an EROFS blob plus a small state
directory — so there's nothing for that kind of scanning to find, and /usr
would be entirely empty under the installer's chroot target.

The fix is to have bootc itself expose verbs that do this assembly
correctly, for both backends, using exactly the same logic that a real boot
would use (honoring /usr/lib/composefs/setup-root-conf.toml for composefs)
rather than reimplementing divergent policy in each installer.

Two variants are needed because callers have two different shapes:

• Callers that keep the physical root and the assembled root as separate
  directories (e.g. Anaconda's physroot/sysroot split) want the
  assembled root placed at a path of their choosing: --assemble-root <PATH>.
• Callers that only have one directory — the ROOT_PATH given to
  to-filesystem/to-existing-root — want that same path to become the
  assembled root in place, exactly like a real boot turns /sysroot (in the
  initramfs) into / (post switch-root): --reassemble-root (no argument).

1) CLI (exposed API) changes

Two new, mutually exclusive flags on bootc install to-filesystem and
bootc install to-existing-root:

• --assemble-root <PATH> — mount the assembled root at <PATH>,
  leaving the physical root's existing mount at ROOT_PATH untouched.
• --reassemble-root (boolean, no value) — reuse ROOT_PATH itself as
  the assembled root: the physical root mount currently at ROOT_PATH is
  relocated to <ROOT_PATH>/sysroot, and the assembled root (usr+etc+var)
  takes its place at ROOT_PATH. No path argument is needed because the
  target is implied by the existing ROOT_PATH.

Common properties of both flags:

• Available on to-filesystem (InstallTargetFilesystemOpts) and
  to-existing-root (InstallToExistingRootOpts). Not available on
  to-disk — it partitions/formats a new block device and doesn't have a
  pre-existing single ROOT_PATH mount lifecycle this composes with
  cleanly.
  • to-existing-root doesn't currently flatten InstallTargetFilesystemOpts
    on the CLI side (it manually constructs one internally, always with
    skip_finalize: true already hardcoded — see install_to_existing_root,
    crates/lib/src/install.rs:2733). Minimal-diff plan: add the two new
    fields directly to InstallToExistingRootOpts and pass them through in
    that manual construction. (A larger, optional cleanup would be to
    de-duplicate by having it flatten InstallTargetFilesystemOpts
    entirely — not required for this feature.)
• Valid for both the ostree backend and the composefs backend
  (--composefs-backend).
• Implies/forces --skip-finalize. The existing finalize_filesystem()
  does fstrim + mount -o remount,ro + fsfreeze on the physical root,
  which would conflict with a live bind-mounted /etc//var sharing that
  filesystem. Caller remains responsible for finalization after it's done
  with the assembled root and has unmounted it — same contract
  --skip-finalize already documents today. (For to-existing-root this is
  a no-op constraint, since skip_finalize: true is already unconditional
  there.)
• No requirement that the target be a subpath of ROOT_PATH, and no
  container-boundary/mount-propagation concerns (see §3 for why — the
  primary consumer invokes bootc as a same-namespace host subprocess, not
  inside a container).
• Target-path handling:
  • For --assemble-root <PATH>: created (create_dir_all) if it doesn't
    exist; fails clearly, up front, if it already exists non-empty or is
    already a mountpoint.
  • For --reassemble-root: ROOT_PATH is by definition already mounted
    (it's the install target); the pre-check instead verifies ROOT_PATH
    isn't already in a "reassembled" state from a previous run (e.g.
    <ROOT_PATH>/sysroot doesn't already look like a relocated mount).
• On partial failure mid-assembly, the caller is responsible for umount -R <PATH-or-ROOT_PATH> before retrying (documented, not auto-recovered).
• No new teardown verb is needed (see §2.3) — everything mounted is ordinary
  kernel mounts, unmountable with a plain recursive unmount.

Contract exposed to callers, identical for both backends and both flags:
after success, the target (<PATH>, or ROOT_PATH itself for
--reassemble-root) contains usr+etc+var populated and
SELinux-labeled, ready to chroot(), with the original physical root
accessible underneath at <target>/sysroot (mirroring exactly what a real
booted system looks like — this nested /sysroot is not unique to
--reassemble-root; see §2.2, it comes for free from the same underlying
mechanism for --assemble-root too). Neither flag mounts /proc, /sys,
/dev, /run, /boot, or arbitrary user-configured extra mountpoints
(/home, /srv, ...) — that remains the caller's responsibility, unchanged
from what installers already have to do today for any deployment.

Caveat callers must know about

Composefs images configured with [etc] mount = "transient" or [root] transient = true (real, documented options in
bootc-setup-root-conf.5.md, backed by a tmpfs overlay) will silently
discard any writes made to /etc via either flag at the very next real
boot — the transient overlay always starts fresh from the pristine image.
This isn't a bug (we're required to honor the image's declared policy
exactly, not override it), but it's a sharp edge for kickstart-style
%post customization. The assembly code should log a warning when it
detects this policy is active. There is no ostree-backend equivalent (no
analogous configurable transient-etc policy exists in that codepath).

2) Architecture details

2.1 Prerequisite refactor (lands as its own PR first)

crates/initramfs/src/lib.rs::setup_root() currently reads
/usr/lib/composefs/setup-root-conf.toml via a bare host-path
std::fs::read_to_string(args.config) before the composefs image is
mounted (mount_composefs_image happens later in the same function). This
only produces correct results today because crates/initramfs/dracut/ module-setup.sh bakes a copy of the target image's config file into the
initramfs at dracut-build time (inst_simple, documented in
docs/src/man/bootc-setup-root-conf.5.md).

That bake-in mechanism breaks for:

• Soft-reboot (crates/lib/src/bootc_composefs/soft_reboot.rs): a soft
  reboot does not re-run the initramfs at all (kernel/initramfs state is
  unchanged by definition), so there's no "new" baked-in config available.
  Today's code passes Args { config: Default::default(), ... } (an empty
  PathBuf, since Args is a Rust struct literal there, not
  clap-parsed — default_value never applies), which silently falls back
  to Config::default(), ignoring the newly-staged image's actual
  [etc]/[var]/[root] policy. This is a live bug today, independent
  of this feature.
• Install-time assembly (either flag): no initramfs is involved at all;
  the live/host environment running bootc install has no baked-in copy of
  the target image's config, and could even pick up an unrelated file
  belonging to the host/live environment instead.

Fix: reorder setup_root() to read config via openat relative to the
freshly-mounted composefs root fd (new_root, from mount_composefs_image),
after mounting instead of before. Verified safe: config content isn't
actually consulted until after new_root is obtained (config.root .transient first read at line ~508; mount_subdir calls for etc/var
later still). The one early use — the systemd.volatile cmdline check
(lines ~455-468) — only sets a default based on kernel cmdline, never
reads image-specific config content, so it's unaffected by the reorder as
long as it stays between the config-read and the first real config-driven
branch.

Also as part of this refactor:

• Remove the now-redundant dracut bake-in step in module-setup.sh.
• Update docs/src/man/bootc-setup-root-conf.5.md (currently documents the
  bake-in mechanism explicitly).
• Keep Args.config as an explicit test-only override, paired with the
  existing --root-fs test-only override.
• Factor the Args/synthetic-Cmdline construction that
  soft_reboot.rs::prepare_soft_reboot_composefs() already does into a
  shared helper, reused by soft-reboot and both install-time assembly modes
  (same shape: different target path + sysroot fd source).
• Other call sites verified unaffected: crates/initramfs/src/main.rs (real
  initramfs binary gets the desired new behavior automatically);
  crates/lib/src/generator.rs's config_has_transient_submounts() is a
  separate function reading the booted root's own file post-switch-root —
  already correct, not touched by this refactor.

2.2 How setup_root() already implements the "reassemble" shape

This is the key discovery that makes --reassemble-root low-risk: the
composefs backend doesn't need new mount-arrangement logic at all —
setup_root() already does precisely this dance for the ordinary real-boot
case, and we just need to invoke the existing logic with a different
parameter.

Tracing setup_root() (crates/initramfs/src/lib.rs):

• Line 485: sysroot_clone = bind_mount(&sysroot, "") — an
  OPEN_TREE_CLONE detached clone of the physical root's fd, taken
  before anything shadows the path it's mounted at. This clone remains
  valid regardless of what happens to that path afterward.
• Line 489: mount_target = args.target.unwrap_or(args.sysroot.clone()) —
  this is the fork point. Real cold boot passes target: None, so
  mount_target defaults to the same path the physical root is already
  mounted at (/sysroot). Soft-reboot (and the new --assemble-root <PATH>) pass an explicit different target.
• Line 541: composefs::mount::mount_at(&sysroot_clone, visible_root, "sysroot") — the cloned physical-root fd is attached at
  <assembled-root>/sysroot, unconditionally, regardless of which branch
  set mount_target. This is why both --assemble-root <PATH> and
  --reassemble-root get a working nested /sysroot for free — it's not
  reassemble-specific, it's just always part of composefs assembly (needed
  so bootc status and friends can find the physical root after
  switch-root).
• Lines 559-565 (kernel 6.15+): unmount(&args.sysroot, DETACH) followed by
  mounting the assembled tree at mount_target. When mount_target == args.sysroot (the default/reassemble case), this is exactly "detach the
  old mount from that path, put the new one there instead." Pre-6.15
  achieves the identical end state earlier (lines 494-496) by stacking the
  new mount on top of the old one at the same path instead (kernel allows
  multiple mounts stacked at one path; the buried one stays reachable via
  the already-open sysroot/sysroot_clone fds).

Conclusion: for the composefs backend, --assemble-root <PATH> and
--reassemble-root are the same underlying helper call, differing only in
whether target is Some(<PATH>) or None (defaults to ROOT_PATH). No
new composefs-side mount logic is needed beyond what --assemble-root
already requires — --reassemble-root for composefs is essentially "pass
target: None."

2.3 Ostree backend: same shape, new (symmetric) low-level logic

Ostree has no existing code path that does this swap, so it needs new logic
— but it should mirror the composefs mechanism exactly, using the same
low-level primitives (open_tree(OPEN_TREE_CLONE), mount_at/move_mount,
kernel-version-conditional stack-vs-detach), so both backends share one
mental model. Concretely, for --reassemble-root:

1. Clone the physical root at ROOT_PATH via open_tree(..., OPEN_TREE_CLONE)
   before touching anything (mirrors sysroot_clone).
2. Bind-mount deployment_checkout (usr+etc) onto ROOT_PATH itself
   (stack-on-top pre-6.15, detach-then-move on 6.15+, matching the
   composefs approach for consistency) — then separately bind-mount
   physical_stateroot_var onto <ROOT_PATH>/var (see §2.4 below for why
   this second step is required).
3. Attach the cloned physical-root fd at <ROOT_PATH>/sysroot.

For --assemble-root <PATH> (ostree, explicit separate target), the same
three steps apply, just targeting <PATH> instead of ROOT_PATH — and per
the composefs precedent, this means <PATH>/sysroot should also be
populated with the physical root, for consistency between the two flags and
because callers may reasonably expect bootc status-style tooling to work
against either assembly target.

Worth factoring as one shared, backend-agnostic low-level primitive (e.g.
shadow_mount_with_sysroot(physical_root_fd, new_root_source, target_path))
used by the composefs path (where new_root_source is the fully-assembled
visible_root) and the ostree path (where it's the deployment-checkout
bind, with var already attached) — rather than reimplementing the
stack-vs-detach kernel-version branching twice.

2.4 The ostree /var symlink-under-chroot problem

An ostree deployment checkout (/ostree/deploy/<stateroot>/deploy/ <checksum>.<serial>/) has real files for usr and etc, but var is a
relative symlink (var -> ../../var, pointing up to the
stateroot-shared var directory). A plain bind of the deployment dir carries
that symlink as-is. Once something chroot()s into the target, ..
traversal can't escape the chroot root, so the symlink resolves back into
the target itself rather than the real shared var directory — this is why
Anaconda's own PrepareOSTreeMountTargetsTask always bind-mounts /var
explicitly as a separate step rather than relying on the symlink, and why
both assemble_root and the reassemble logic above need the same explicit
second bind for var. Composefs's equivalent relative-symlink var handling
is directly confirmed in crates/lib/src/bootc_composefs/state.rs around
SHARED_VAR_PATH.

2.5 install_to_filesystem_impl control-flow changes

Current structure (crates/lib/src/install.rs, install_to_filesystem_impl
~line 1993): branches early on state.composefs_options.composefs_backend
into a composefs-specific block (initialize_composefs_repository ->
setup_composefs_boot -> SELinux object relabel) or ostree_install(...)
(-> initialize_ostree_root -> install_with_sysroot -> install_container
-> deploy, then unconditionally install_finalize(&rootfs.physical_root_path)
— an ostree-only deployment-existence sanity check, distinct from and
unaffected by skip_finalize/these flags). Both branches rejoin into one
shared tail: full SELinux relabel of the physical root
(ensure_dir_labeled_recurse, ~2094-2104), then conditionally
(!rootfs.skip_finalize) finalize_filesystem() (fstrim + ro-remount +
fsfreeze) at ~2106-2112.

Note on naming collision: there are two unrelated "finalize" concepts in
this codebase — finalize_filesystem() (fstrim/ro-remount, gated by
skip_finalize) and install_finalize() (ostree-only sanity check, always
runs inside ostree_install, untouched by this feature). Document this
distinction clearly to avoid confusion.

Changes needed:

• setup_composefs_boot (crates/lib/src/bootc_composefs/boot.rs) changes
  from Result<()> to return the Sha512HashValue digest it already
  computes internally (Copy type, no lifetime concerns).
• install_container already returns Result<(ostree::Deployment, InstallAleph)>, but install_with_sysroot and ostree_install currently
  discard the Deployment and return Result<()>. Rather than threading
  the ostree::Deployment GObject itself out past the sysroot's scope,
  compute the concrete paths needed —
  sysroot.deployment_dirpath(&deployment) for the checkout dir, plus the
  physical stateroot var path — while the Sysroot is still alive,
  before the // We must drop the sysroot here in order to close any open file descriptors point in ostree_install, and thread those paths out
  instead. (This avoids relying on assumptions about libostree GObject
  lifetime past the sysroot drop point.)

Shared type:

enum DeployedRoot {
    Composefs { digest: Sha512HashValue },
    Ostree { deployment_checkout: Utf8PathBuf, physical_stateroot_var: Utf8PathBuf },
}

enum RootAssemblyTarget {
    /// --assemble-root <PATH>
    SeparatePath(Utf8PathBuf),
    /// --reassemble-root
    InPlace,
}

In the shared tail, after the SELinux relabel and before the (now-skipped
when either flag is set) finalize block:

if let Some(mode) = rootfs.assembly_target.as_ref() {
    assemble_root(rootfs, &deployed_root, mode)?;
}

assemble_root resolves the concrete target path from mode
(rootfs.physical_root_path for InPlace, the given path for
SeparatePath) and dispatches per-DeployedRoot-variant:

• Composefs: reuse the shared setup_root()-based helper from the
  prerequisite refactor, passing target: None for InPlace or
  target: Some(path) for SeparatePath — see §2.2, this is the only
  difference between the two flags for this backend. Thread
  allow_missing_fsverity through explicitly.
• Ostree: run the logic in §2.3/§2.4 — bind deployment_checkout (+
  explicit var bind) onto the resolved target path, then attach the
  physical-root clone at <target>/sysroot.

Sequencing after the SELinux relabel step is required: the assembled root's
content must already carry final labels before anything writes into it or
reads from it via the target.

2.6 Teardown

No bootc install finalize extension needed — everything mounted by either
flag is ordinary kernel mounts (bind mounts, EROFS, overlayfs) nested under
the target, so a plain recursive unmount by the caller before final
physroot unmount/trim suffices. For --reassemble-root, this recursive
unmount naturally also restores ROOT_PATH to its original physical-root
mount (the original mount is still there underneath, just buried/detached
from the path — an umount -R on the assembled tree ends with the
original physical mount either still directly accessible via its retained
fd, or the caller can re-derive it if needed).

2.7 Testing strategy

• Unit test on the prerequisite refactor: setup_root() correctly reads
  setup-root-conf.toml from the mounted composefs root fd (via the
  existing --root-fs test override mechanism), not a host path. Also
  covers the soft-reboot bug fix.
• Integration (tmt) test building on the existing pattern in
  tmt/test-install-to-filesystem-var-mount.sh: for both backends and both
  flags, run bootc install to-filesystem [--assemble-root <path> | --reassemble-root], assert usr/etc/var are populated, assert
  chroot <target> /bin/true succeeds, assert <target>/sysroot correctly
  exposes the physical root, assert writes to <target>/etc and
  <target>/var land in the correct persistent backing storage.
• A to-existing-root variant of the above, given the shared
  install_to_filesystem_impl plumbing.
• Test with non-default --stateroot, with --disable-selinux, and with
  allow_missing_fsverity to cover edge cases identified in review.

2.8 Review history

Two independent architect-subagent reviews (architect-g, architect-c46) were
run against the original --assemble-root-only design. Both converged on
the same assessment: the setup_root() reorder is safe and independently
valuable, the ostree /var symlink-under-chroot reasoning is sound, the
enum-based dispatch is the right structural pattern, and the one concrete
correction needed was to thread deployment paths rather than the
ostree::Deployment GObject through DeployedRoot. That correction is
incorporated above. Both reviews recommended landing the prerequisite
refactor as its own PR before starting on the assembly flags themselves.
--reassemble-root and to-existing-root support were added to the design
afterward and have not yet been through a separate review pass.

3) Relationship with Anaconda

Anaconda (rhinstaller/anaconda) is the primary real-world consumer this
design targets, and its actual invocation pattern shaped several decisions
above.

3.1 Ground truth (verified by reading Anaconda source)

• bootc install to-filesystem is invoked as a direct host subprocess
  (execReadlines("bootc", ...) in DeployBootcTask,
  pyanaconda/modules/payloads/payload/rpm_ostree/installation.py), not
  inside a container/podman. Anaconda runs in the same mount namespace as
  systemd PID 1 (no unshare, no PrivateMounts=). So mounts bootc creates
  are immediately visible to the rest of Anaconda's process tree — this is
  why the design has no mount-propagation/container-boundary requirements
  (an earlier draft of this design assumed bootc always ran inside a
  privileged container and required bind-propagation=rshared-style
  flags; that assumption was wrong for the primary consumer and was
  dropped).
• Current ostree flow: DeployBootcTask -> SetSystemRootTask (scans
  physical_root for the ostree deployment checkout dir via the heuristic
  get_ostree_deployment_path(), then mount --rbind <deployment_dir> /mnt/sysroot) -> PrepareBootcMountTargetsTask (bind-mounts /proc,
  /sys, /var, /boot on top; missing /dev, /run, /sysroot
  self-bind, and arbitrary user mountpoints compared to the ostree-native
  PrepareOSTreeMountTargetsTask, which is a pre-existing gap independent
  of this work). Anaconda's physroot (/mnt/sysimage) and sysroot
  (/mnt/sysroot) are two separate directories — this maps directly onto
  --assemble-root <sysroot> (with ROOT_PATH = physroot), not
  --reassemble-root.
• %post scripts run via a real os.chroot() into conf.target .system_root (/mnt/sysroot), after the full install task queue
  completes.
• Composefs is not Anaconda's default today (composefs_backend is only
  auto-enabled for UKI target images per crates/lib/src/install.rs
  ~line 1646-1660; Anaconda never passes --composefs-backend). For
  composefs, none of the ostree-era mechanism works: no checkout tree
  exists (write_composefs_state only produces state/deploy/<digest>/ {etc, var->symlink}), so /usr is empty and there's nothing for
  get_ostree_deployment_path-style discovery to find.
• Documented workarounds/gaps referenced directly in Anaconda source
  comments: Don't look for host policy with --source-imageref #1438 (SELinux must be present even if
  disabled), to-filesystem with --source-imgref #1400/Defer reading prepare-root config when using `--source-imageref #1410 (/etc/ostree/prepare-root.conf expected even for
  non-ostree-native flows), Regression: bootc incorrectly detecting missing bootupd #1778 (--bootloader grub arg broken on
  s390x). These are adjacent friction points, not directly fixed by this
  design, but worth being aware of.

3.2 Scope decisions

Explicit decisions from the project owner:

• Cover both backends, not just composefs. The end goal is for Anaconda
  to detect the new verb(s) and use them uniformly for ostree and
  composefs, retiring its own backend-specific mount logic.
• Support both --assemble-root (two-directory callers like Anaconda)
  and --reassemble-root (single-directory callers).
• Support both flags on to-existing-root in addition to to-filesystem.
  Not on to-disk.

3.3 Anaconda-side migration (their work, not ours, but shapes this contract)

• DeployBootcTask gains a capability check (e.g. scan bootc install to-filesystem --help for --assemble-root) and, when available, passes
  --assemble-root <sysroot> directly, for either backend (Anaconda's
  existing two-directory model is the natural fit for --assemble-root
  rather than --reassemble-root).
• SetSystemRootTask (today ostree-only, fragile heuristic scan) becomes
  unnecessary when the flag is used — bootc already knows the deployment
  path/digest it just created, no scanning required.
• PrepareBootcMountTargetsTask becomes backend-agnostic: proc/sys/dev/run/
  boot/extra-mountpoints only, dropping its current ad hoc /var handling
  since --assemble-root now owns usr+etc+var for both backends.
• Teardown on Anaconda's side: a plain recursive unmount of <sysroot>
  before final physroot unmount/trim, same as today.

[cgwalters]

@ckyrouac can you review this?