Add temproots in copyPaths, plus all the associated stuff#15719
Open
dramforever wants to merge 8 commits into
Open
Add temproots in copyPaths, plus all the associated stuff#15719dramforever wants to merge 8 commits into
dramforever wants to merge 8 commits into
Conversation
github-actions
Bot
added
new-cli
dramforever
force-pushed
the
batch-addtemproots
branch
from
f236067 to
67d061f
Compare
Move the writing of the temproots file to before checking the GC lock. This closes a race, however unlikely, where yet another GC may start again after we sent the temproots to the old running GC, but before writing the temproots file.
Introduce LocalStore::addTempRoots, which adds all temproots in a batch for LocalStore. Make this available from the base class Store by making a virtual method Store::addTempRoots, overriding it in LocalStore and RemoteStore (it is just a loop for each path because the benefit of having this be batched is unclear), and making the single-path Store::addTempRoot a wrapper around that.
This avoids some system calls in case of duplicate temproots.
Use the new batched addTempRoots for both convenience and better performance for LocalStore.
Use the new batched addTempRoots for both convenience and better performance for LocalStore.
Add AddTempRootsFlag for optionally adding all queried paths as temproots. This will be used to close a GC race in copyPaths. The existing LegacySSHStore::queryValidPaths subclass overload with the lock parameter is absorbed. LegacySSHStore::Connection::queryValidPaths (i.e. ServeProto::BasicClientConnection::queryValidPaths) remains unchanged.
This closes a race where paths in the result of queryValidPaths can become invalid if a GC happens later that deletes the path, leaving paths invalid after return from copyPaths.
This asks the QueryValidPaths operation to add temproots on the remote, saving roundtrips in RemoteStore::queryValidPaths.
dramforever
force-pushed
the
batch-addtemproots
branch
from
67d061f to
3a2bd29
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
GC "race" whack-a-mole.
Context
Fixes, hopefully, #1970. Supersedes #15613. Originally discussed in #15616 but split out.
Make
copyPathsadd the copied paths as temproots, but hopefully in a better manner than individually rooting every single path:Store::addTempRootvirtual method to a batchedStore::addTempRootsmethod, with the original one remaining as a non-virtual helper. ImplementLocalStore::addTempRootsas writing out temproots in a batch. Also add an LRU cache forLocalStore::addTempRoots.RemoteStore::addTempRootsis just a loop.AddTempRootsFlagtoStore::queryValidPathscopyPaths, setAddTempRootsFlagwhile callingqueryValidPathsAddTempRootsFlagflag worker protocol operationQueryValidPaths, and use that inRemoteStore::queryValidPaths. This requires a new daemon on the remote side, in exchange for (hopefully) better performance.Adding a flag to worker protocol
QueryValidPathsis somewhat unorthogonal, but it was how it's done in the legacy SSH store, and I do think it still makes sense to do inQueryValidPathsin the worker protocol since it closes a TOCTOU for the result of list of valid paths, if needed by the caller. I haven't added a batchAddTempRootsworker protocol operation since it doesn't seem to be needed.Also note that
LocalStore::addTempRootscode. Including it makes the later commit make more sense.Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.