Skip to content

storage: Add bucket GC pacing options#660

Open
andrejbranch wants to merge 1 commit into
tarantool:masterfrom
andrejbranch:bucket-gc-pacing
Open

storage: Add bucket GC pacing options#660
andrejbranch wants to merge 1 commit into
tarantool:masterfrom
andrejbranch:bucket-gc-pacing

Conversation

@andrejbranch

Copy link
Copy Markdown

Add independent pacing controls for bucket garbage collection

Summary

This change adds independent controls for bucket garbage-collection transaction size and inter-batch delay. It allows operators to reduce sustained garbage-collection pressure on Tarantool's transaction thread without reducing bucket transfer concurrency or changing transfer transaction sizing.

Closes #659

Motivation

Bucket garbage collection already processes tuples in bounded transactions, but it immediately begins another transaction after each commit. The existing chunking therefore limits individual transaction duration without limiting sustained cleanup pressure. The chunk-size constant is also shared with bucket sending and receiving, so it cannot be tuned specifically for source cleanup.

Changes

  • Add bucket_gc_batch_size, a positive integer that controls the maximum number of tuples deleted in one bucket-GC transaction.
  • Add bucket_gc_batch_delay, a non-negative number that controls the delay in seconds between non-empty bucket-GC deletion transactions.
  • Default bucket_gc_batch_size to 1000 and bucket_gc_batch_delay to 0, preserving existing behavior by default.
  • Make both options dynamically reconfigurable.
  • Apply the controls to automatic garbage collection and vshard.storage.bucket_delete_garbage().
  • Keep one pacing state across spaces and buckets so consecutive deletion transactions cannot bypass the configured delay at an iteration boundary.
  • Sleep only after a successful commit, never while a transaction is open.
  • Recheck bucket state after pacing yields before continuing deletion.
  • Avoid a trailing delay when the preceding batch removed the final tuple.
  • Document the options and their performance and disk-usage tradeoffs.

Behavior

With the defaults, bucket garbage collection continues to delete at most 1000 tuples per transaction with no delay between transactions. Setting a smaller batch size creates more frequent commit and yield points. Setting a positive delay spreads deletion work over time while leaving bucket transfer controls unchanged.

For example:

bucket_gc_batch_size = 100
bucket_gc_batch_delay = 0.05

This configuration deletes at most 100 tuples in each bucket-GC transaction and waits at least 50 milliseconds before starting the next non-empty deletion transaction.

Compatibility

The new options are optional and retain the previous effective batch size and no-delay behavior by default. Existing configurations do not need to change.

Tradeoffs

A positive delay increases the time required to remove transferred bucket data from the source node and may temporarily increase disk usage. A smaller batch size creates more transactions and causes the BUCKET_EVENT.GC trigger to run more frequently. These costs are controlled explicitly by the new settings.

Testing

  • Validate default, valid, and invalid configuration values.
  • Verify manual garbage collection commits one configured batch before waiting.
  • Verify automatic garbage collection preserves pacing across bucket boundaries.
  • Verify pacing works for MemTX and Vinyl, with and without MVCC.
  • Verify exact batch-size multiples do not cause a trailing delay.
  • Verify a bucket-state change during the delay prevents further deletion.
  • Run the complete garbage-collector test suite, legacy unit tests, Luacheck, and Tarantool checkpatch.

VShard limits concurrent bucket transfers, but bucket garbage collection
keeps deleting data until all garbage buckets are empty. Transaction
chunking bounds a single TX-thread burst, but it does not let operators
reserve TX capacity for foreground requests.

Add bucket_gc_batch_size to control the number of tuples deleted in one
GC transaction and bucket_gc_batch_delay to pause between non-empty GC
delete transactions. Keep the defaults at 1000 tuples and zero delay to
preserve the existing behavior.

The settings are dynamic and apply to both the automatic collector and
vshard.storage.bucket_delete_garbage(). Delays happen after commit and
bucket state is checked again after each pacing yield.

@TarantoolBot document
Title: Document bucket GC pacing options

Two new root configuration options control bucket garbage collection:

* `bucket_gc_batch_size` is the maximum number of tuples deleted in one
  bucket GC transaction. It defaults to 1000. Smaller values reduce the
  longest uninterrupted TX-thread burst, but create more transactions.
* `bucket_gc_batch_delay` is the delay in seconds between non-empty bucket
  GC delete transactions. It defaults to 0, which disables the delay. A
  positive value reduces sustained GC pressure on the TX thread, but makes
  removal of transferred bucket data take longer.

Both options can be changed dynamically and affect automatic bucket GC as
well as `vshard.storage.bucket_delete_garbage()`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add independent pacing controls for bucket garbage collection

1 participant