feat(hotblocks): expose the LSM base level size

Raising write_buffer_size 8x without touching this left L0 arriving larger
than the level it merges into: four 512 MB memtables flush ~512 MB of
compressed L0 into a 256 MB L1, so every L0 compaction rewrites all of L1 and
immediately pushes it down. The canonical pairing is base >=
level0_file_num_compaction_trigger x the compressed flush size.

It is also the lever the memtable change could not reach. A byte is rewritten
once per level, and memtable size only addresses the L0->L1 hop; at ~60 GB
live the ladder runs several levels deeper and that work is untouched. The
production numbers show it: the bench predicted 2.4x fewer device writes at
~130 MB live, internal delivered 1.67x at ~60 GB.

Default stays RocksDB's 256 MB, so this is inert until a deployment sets it.
Deliberately unmeasured: the bench cannot price it at 130 MB live, where there
is barely a ladder to shorten.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Codex <[email protected]>

Author: 3 people

crates/hotblocks/src/cli.rs

@@ -91,6 +91,15 @@ pub struct CLI {
     #[arg(long, value_name = "N", default_value = "2")]
     pub rocksdb_max_write_buffers: i32,
 
+    /// Target size of the LSM base level. With ~60 GB live and the default
+    /// multiplier of 10, this decides how many levels deep the ladder runs, and
+    /// a byte is rewritten once per level it descends. The default 256 MB is
+    /// smaller than one 512 MB memtable's flush, so L0 currently arrives larger
+    /// than the level it merges into. Validate boot time and compaction behavior
+    /// before raising it.
+    #[arg(long, value_name = "MB", default_value = "256")]
+    pub rocksdb_level_base_mb: usize,
+
     /// Rewrite every table SST older than this, collecting dead data that never made a file
     /// tombstone-dense enough for the deletion collector. Lower means faster reclaim and
     /// proportionally more write amplification. 0 disables it, leaving RocksDB's 30-day
@@ -165,6 +174,7 @@ impl CLI {
             .with_periodic_compaction_secs(self.rocksdb_periodic_compaction_secs)
             .with_write_buffer_mb(self.rocksdb_write_buffer_mb)
             .with_max_write_buffers(self.rocksdb_max_write_buffers)
+            .with_level_base_mb(self.rocksdb_level_base_mb)
             .with_block_hash_index(self.block_hash_index)
             .with_transaction_hash_index(self.transaction_hash_index);
 

crates/storage/src/db/db.rs

@@ -60,6 +60,7 @@ pub struct DatabaseSettings {
     with_rocksdb_stats: bool,
     write_buffer_mb: usize,
     max_write_buffers: i32,
+    level_base_mb: usize,
     direct_io: bool,
     cache_index_and_filter_blocks: bool,
     max_log_file_size: usize,
@@ -89,6 +90,7 @@ impl Default for DatabaseSettings {
             // RocksDB's own defaults; the CLI carries the deployed values
             write_buffer_mb: 64,
             max_write_buffers: 2,
+            level_base_mb: 256,
             direct_io: false,
             cache_index_and_filter_blocks: false,
             max_log_file_size: 10,
@@ -133,6 +135,11 @@ impl DatabaseSettings {
         self
     }
 
+    pub fn with_level_base_mb(mut self, mb: usize) -> Self {
+        self.level_base_mb = mb;
+        self
+    }
+
     pub fn with_cache_index_and_filter_blocks(mut self, yes: bool) -> Self {
         self.cache_index_and_filter_blocks = yes;
         self
@@ -257,6 +264,10 @@ impl DatabaseSettings {
         // memory cost (write_buffer_mb x max_write_buffers) is paid once rather than per CF.
         options.set_write_buffer_size(self.write_buffer_mb << 20);
         options.set_max_write_buffer_number(self.max_write_buffers);
+        // Sets how many levels the ladder has, and a byte is rewritten once per level.
+        // Wants to be >= level0_file_num_compaction_trigger x the compressed flush size,
+        // or L0 arrives larger than the level it merges into.
+        options.set_max_bytes_for_level_base((self.level_base_mb as u64) << 20);
         if !self.auto_compactions {
             options.set_disable_auto_compactions(true);
         }