Skip to content

fix(nix-daemon): fix linux-builder feature scheduling for nixosTest#91

Merged
emaiax merged 2 commits into
mainfrom
fix/linux-builder-kvm-feature
Jul 16, 2026
Merged

fix(nix-daemon): fix linux-builder feature scheduling for nixosTest#91
emaiax merged 2 commits into
mainfrom
fix/linux-builder-kvm-feature

Conversation

@emaiax

@emaiax emaiax commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Removes kvm from nix.linux-builder.config.supportedFeatures. The linux-builder VM boots via QEMU+HVF (accel=hvf:tcg), which never exposes /dev/kvm to the guest — nested-virtualization builds (nixosTest with a VM backend) cannot run inside it, no matter what the config claims.
  • Declaring kvm as supported let Nix schedule such derivations onto this machine anyway, where they'd crash with a garbled Undefined error: 0 exec failure instead of a clean scheduling rejection.
  • Follow-up (same root problem, wider fix): most prometheus-exporters nixosTests (and others) use the container backend (systemd-nspawn), not a VM, and don't need KVM at all. They were still being rejected at scheduling because this builder didn't advertise nixos-test/uid-range, and even once scheduled, systemd-nspawn needs a real root-mapped UID range inside the sandbox to start. Added nixos-test/uid-range to supportedFeatures, plus auto-allocate-uids and the cgroups/auto-allocate-uids experimental Nix features on the guest, so container-backed nixosTest derivations can actually run here. NixOS's own default for nix.settings.system-features is a static list that predates auto-allocate-uids and doesn't add uid-range for it automatically (unlike plain Nix's dynamic default), hence the explicit extra-system-features.

Verification

  • /dev/kvm is absent inside the guest (ls: cannot access '/dev/kvm': No such file or directory), confirmed via a real dispatched build.
  • Before the fix, nixosTests.login and nixosTests.prometheus-exporters.speedtest-ookla were already rejected at scheduling time (missing system features), just for the wrong/incomplete reason (nixos-test also missing from supportedFeatures).
  • After darwin-rebuild switch, /etc/nix/machines no longer advertises kvm for the builder, and now advertises nixos-test/uid-range.
  • Re-tested nixosTests.prometheus-exporters.speedtest-ookla end to end (via overrideTestDerivation to drop the still-required kvm from that one test, since it's a VM-backend leftover default unrelated to this test's container backend): it schedules, boots the container, the exporter starts, and curl -sSf http://localhost:9876 succeeds inside the test.
  • Re-tested nixosTest scenarios post-fix: VM-backed nixosTest builds are still cleanly rejected (now for the honest reason, missing kvm), and plain aarch64-linux builds still dispatch and complete normally on the builder.
  • Confirmed the guest kernel itself supports unprivileged user namespaces fine (unshare -Ur id → uid 0) — the earlier uid-range rejection was pure Nix/NixOS configuration, not a kernel limitation.

Test plan

  • just build evaluates cleanly
  • just switch applied without errors
  • /etc/nix/machines reflects the change post-switch
  • nixosTests.login still rejected cleanly at scheduling time
  • plain aarch64-linux build dispatches and completes on the builder
  • nixosTests.prometheus-exporters.speedtest-ookla (container-backed) schedules and passes on the builder

The VM boots via QEMU+HVF (accel=hvf:tcg), which never exposes
/dev/kvm to the guest, so nested-virtualization builds cannot
actually run inside it. Declaring kvm as supported let Nix schedule
nixosTest derivations onto this machine anyway, where they crashed
with a garbled "Undefined error: 0" exec failure instead of a clean
scheduling rejection.

Confirmed live: /dev/kvm is absent in the guest, and after this
change the generated /etc/nix/machines no longer advertises kvm.
nixosTest builds are now rejected cleanly at scheduling time, and
plain aarch64-linux/x86_64-linux builds still dispatch and complete
normally on the builder.

See dotfiles#82.
nixosTests using the container backend (systemd-nspawn) require the nixos-test and uid-range builder features, plus the cgroups and auto-allocate-uids experimental Nix features, to get a real root-mapped UID range inside the build sandbox. Without it nspawn refuses to start ("systemd-nspawn requires root to work").

NixOS's own default for nix.settings.system-features (nixos/modules/config/nix.nix) is a static list that predates auto-allocate-uids and does not add uid-range automatically the way plain Nix's dynamic default does, hence extra-system-features here instead of relying on the built-in default.

Verified against nixosTests.prometheus-exporters.speedtest-ookla from nixpkgs, which now schedules and passes on this builder. kvm still needs to be stripped per-test via overrideTestDerivation, since VM-backed nixosTest derivations genuinely can't run without hardware acceleration in this VM.
@emaiax emaiax changed the title fix(nix-daemon): drop kvm from linux-builder supportedFeatures fix(nix-daemon): fix linux-builder feature scheduling for nixosTest Jul 16, 2026
@emaiax
emaiax merged commit 746945c into main Jul 16, 2026
1 of 2 checks passed
@emaiax
emaiax deleted the fix/linux-builder-kvm-feature branch July 16, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant