csdb: add RocksDB backend with runtime backend selection#73
Open
akaitrade wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add RocksDB storage backend with runtime backend selection
Why
The node has run exclusively on BerkeleyDB (
blockchain.db) since inception.BDB works, but it has real operational ceilings:
becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
migrations). On Windows in particular it wedges under sustained write pressure.
empty / low-tx blocks, which is a lot of disk for little data.
to a modern LSM engine.
RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.
What
RocksDB backend (
database_rocksdb.cpp/.hpp) implementingcsdb::Database,behaviorally equivalent to the BerkeleyDB backend:
seq+1so lexicographic ordermatches sequence order,
seq_no, andcontracts.seq_noCF, LZ4 throughout with LZ4HC at the bottommost level.put_batch()coalescing block + index writes into a single WriteBatch / WAL append.flush()(SyncWAL) for checkpoint-boundary durability; async writes by defaultwith durability anchored at checkpoints (mirrors BDB's
DB_TXN_NOSYNCmodel).set_bulk_load/compact_full) for one-shot high-rate import.Surrounding changes:
(
CSDB_BACKEND=both); no compile-time backend switch.config.ini [storage] db_backend(defaultberkeleydb,set
rocksdbto opt in), plus tuning knobsrocksdb_block_cache_mb,rocksdb_memtable_mb, and the storage write-pipeline knobsasync_write_queue_sizeandwrite_batch_size.set_tuningplumbing,Storage::flush(), and an async write-queue that batches pools throughput_batch().lz4hc.c/.h) to the vendored lz4 (RocksDB'sbottommost
kLZ4HCCompressionneeds it); definedBOOST_ALL_NO_LIBto disableBoost's MSVC auto-link pragmas (Boost is linked via CMake targets).
How to use
In
config.ini:Default config is unchanged (
berkeleydb), so existing nodes behave identicallyuntil explicitly switched.
Compatibility
and a BerkeleyDB node produce identical block hashes and interoperate normally.
db_backendon anexisting node means re-syncing (or migrating) that node's chain DB; the new
backend starts from a fresh DB directory.
db_backend.Testing
CSDB_BACKEND=both.