storage: Add bucket GC pacing options#660
Open
andrejbranch wants to merge 1 commit into
Open
Conversation
VShard limits concurrent bucket transfers, but bucket garbage collection keeps deleting data until all garbage buckets are empty. Transaction chunking bounds a single TX-thread burst, but it does not let operators reserve TX capacity for foreground requests. Add bucket_gc_batch_size to control the number of tuples deleted in one GC transaction and bucket_gc_batch_delay to pause between non-empty GC delete transactions. Keep the defaults at 1000 tuples and zero delay to preserve the existing behavior. The settings are dynamic and apply to both the automatic collector and vshard.storage.bucket_delete_garbage(). Delays happen after commit and bucket state is checked again after each pacing yield. @TarantoolBot document Title: Document bucket GC pacing options Two new root configuration options control bucket garbage collection: * `bucket_gc_batch_size` is the maximum number of tuples deleted in one bucket GC transaction. It defaults to 1000. Smaller values reduce the longest uninterrupted TX-thread burst, but create more transactions. * `bucket_gc_batch_delay` is the delay in seconds between non-empty bucket GC delete transactions. It defaults to 0, which disables the delay. A positive value reduces sustained GC pressure on the TX thread, but makes removal of transferred bucket data take longer. Both options can be changed dynamically and affect automatic bucket GC as well as `vshard.storage.bucket_delete_garbage()`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add independent pacing controls for bucket garbage collection
Summary
This change adds independent controls for bucket garbage-collection transaction size and inter-batch delay. It allows operators to reduce sustained garbage-collection pressure on Tarantool's transaction thread without reducing bucket transfer concurrency or changing transfer transaction sizing.
Closes #659
Motivation
Bucket garbage collection already processes tuples in bounded transactions, but it immediately begins another transaction after each commit. The existing chunking therefore limits individual transaction duration without limiting sustained cleanup pressure. The chunk-size constant is also shared with bucket sending and receiving, so it cannot be tuned specifically for source cleanup.
Changes
bucket_gc_batch_size, a positive integer that controls the maximum number of tuples deleted in one bucket-GC transaction.bucket_gc_batch_delay, a non-negative number that controls the delay in seconds between non-empty bucket-GC deletion transactions.bucket_gc_batch_sizeto 1000 andbucket_gc_batch_delayto 0, preserving existing behavior by default.vshard.storage.bucket_delete_garbage().Behavior
With the defaults, bucket garbage collection continues to delete at most 1000 tuples per transaction with no delay between transactions. Setting a smaller batch size creates more frequent commit and yield points. Setting a positive delay spreads deletion work over time while leaving bucket transfer controls unchanged.
For example:
This configuration deletes at most 100 tuples in each bucket-GC transaction and waits at least 50 milliseconds before starting the next non-empty deletion transaction.
Compatibility
The new options are optional and retain the previous effective batch size and no-delay behavior by default. Existing configurations do not need to change.
Tradeoffs
A positive delay increases the time required to remove transferred bucket data from the source node and may temporarily increase disk usage. A smaller batch size creates more transactions and causes the
BUCKET_EVENT.GCtrigger to run more frequently. These costs are controlled explicitly by the new settings.Testing