fix(nix-daemon): fix linux-builder feature scheduling for nixosTest#91
Merged
Conversation
The VM boots via QEMU+HVF (accel=hvf:tcg), which never exposes /dev/kvm to the guest, so nested-virtualization builds cannot actually run inside it. Declaring kvm as supported let Nix schedule nixosTest derivations onto this machine anyway, where they crashed with a garbled "Undefined error: 0" exec failure instead of a clean scheduling rejection. Confirmed live: /dev/kvm is absent in the guest, and after this change the generated /etc/nix/machines no longer advertises kvm. nixosTest builds are now rejected cleanly at scheduling time, and plain aarch64-linux/x86_64-linux builds still dispatch and complete normally on the builder. See dotfiles#82.
3 tasks
nixosTests using the container backend (systemd-nspawn) require the nixos-test and uid-range builder features, plus the cgroups and auto-allocate-uids experimental Nix features, to get a real root-mapped UID range inside the build sandbox. Without it nspawn refuses to start ("systemd-nspawn requires root to work").
NixOS's own default for nix.settings.system-features (nixos/modules/config/nix.nix) is a static list that predates auto-allocate-uids and does not add uid-range automatically the way plain Nix's dynamic default does, hence extra-system-features here instead of relying on the built-in default.
Verified against nixosTests.prometheus-exporters.speedtest-ookla from nixpkgs, which now schedules and passes on this builder. kvm still needs to be stripped per-test via overrideTestDerivation, since VM-backed nixosTest derivations genuinely can't run without hardware acceleration in this VM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kvmfromnix.linux-builder.config.supportedFeatures. The linux-builder VM boots via QEMU+HVF (accel=hvf:tcg), which never exposes/dev/kvmto the guest — nested-virtualization builds (nixosTestwith a VM backend) cannot run inside it, no matter what the config claims.kvmas supported let Nix schedule such derivations onto this machine anyway, where they'd crash with a garbledUndefined error: 0exec failure instead of a clean scheduling rejection.prometheus-exportersnixosTests (and others) use the container backend (systemd-nspawn), not a VM, and don't need KVM at all. They were still being rejected at scheduling because this builder didn't advertisenixos-test/uid-range, and even once scheduled,systemd-nspawnneeds a real root-mapped UID range inside the sandbox to start. Addednixos-test/uid-rangetosupportedFeatures, plusauto-allocate-uidsand thecgroups/auto-allocate-uidsexperimental Nix features on the guest, so container-backed nixosTest derivations can actually run here. NixOS's own default fornix.settings.system-featuresis a static list that predatesauto-allocate-uidsand doesn't adduid-rangefor it automatically (unlike plain Nix's dynamic default), hence the explicitextra-system-features.Verification
/dev/kvmis absent inside the guest (ls: cannot access '/dev/kvm': No such file or directory), confirmed via a real dispatched build.nixosTests.loginandnixosTests.prometheus-exporters.speedtest-ooklawere already rejected at scheduling time (missing system features), just for the wrong/incomplete reason (nixos-testalso missing fromsupportedFeatures).darwin-rebuild switch,/etc/nix/machinesno longer advertiseskvmfor the builder, and now advertisesnixos-test/uid-range.nixosTests.prometheus-exporters.speedtest-ooklaend to end (viaoverrideTestDerivationto drop the still-requiredkvmfrom that one test, since it's a VM-backend leftover default unrelated to this test's container backend): it schedules, boots the container, the exporter starts, andcurl -sSf http://localhost:9876succeeds inside the test.nixosTestscenarios post-fix: VM-backednixosTestbuilds are still cleanly rejected (now for the honest reason, missingkvm), and plainaarch64-linuxbuilds still dispatch and complete normally on the builder.unshare -Ur id→ uid 0) — the earlieruid-rangerejection was pure Nix/NixOS configuration, not a kernel limitation.Test plan
just buildevaluates cleanlyjust switchapplied without errors/etc/nix/machinesreflects the change post-switchnixosTests.loginstill rejected cleanly at scheduling timeaarch64-linuxbuild dispatches and completes on the buildernixosTests.prometheus-exporters.speedtest-ookla(container-backed) schedules and passes on the builder