[Draft][MP][Maru] Diff preview: engine KV format mapping moved into MaruL1Manager#21
Closed
seohui-XCENA wants to merge 31 commits into
Closed
[Draft][MP][Maru] Diff preview: engine KV format mapping moved into MaruL1Manager#21seohui-XCENA wants to merge 31 commits into
seohui-XCENA wants to merge 31 commits into
Conversation
- l1_protocol.py: structural runtime_checkable Protocol mirroring L1Manager's 17-method surface, with per-method listener/lock contract docstrings - config.py: MaruL1Config + maru_config field, __post_init__ DRAM-clamp skip, maru CLI args (--maru-server-url/--maru-pool-size-gb/--maru-instance-id), parse_args_to_config wiring - tests: interface<->L1Manager method-set + signature conformance, maru config parsing Signed-off-by: seohui-XCENA <[email protected]>
- thin wrapper over external maru_lmcache.CxlMemoryAdapter, lazy-imported - two-phase init_layout: build MaruHandler + CxlMemoryAdapter on first layout (single-model; layout mismatch rejected) - free/batched_free no-op (page lifecycle owned by MaruServer); abort_alloc discards an allocated-but-unregistered page - MaruL1Config -> maru.MaruConfig mapping; maru-only get_by_location / create_store_handle / handler surface for MaruL1Manager - tests use mocked maru runtime (no CXL required) Signed-off-by: seohui-XCENA <[email protected]>
…ifecycle) - sibling of L1Manager over the maru shared CXL pool: membership/read protection live in the MaruServer directory (pin_count), locally only in-flight staging (_pending_read refcount, _pending_write) - reserve_read: per-key independent pins (1+extra_count via one RPC), rollback on partial pin / retrieve failure / unresolvable page - reserve_write mode=new: local staged check + batch_exists dedup (cross-instance), all-or-nothing OOM; finish_write: batch_store, dup-skip is success, definitive False reclaims the page, unknown server state never recycles - delete: staged keys and pinned keys refuse with KEY_IS_LOCKED (exists() disambiguates the handler's pinned/missing conflation) - clear(force=False) keeps locked staging (stock parity); close drains - PARITY/MARU provenance comments; RPC reply length guards - tests: stateful fake maru runtime with fault-injection knobs, failure-path coverage, and a conformance suite parametrized over both L1Manager (CUDA) and MaruL1Manager Signed-off-by: seohui-XCENA <[email protected]>
- reserve_read/reserve_write/finish_write/finish_read/delete/clear now fire the on_l1_keys_* events (feeds the eviction LRU and the store controller), reporting only the keys that actually succeeded - finish_read: temporary reads (local staging, never directory-pinned) are reclaimed via abort_alloc at refcount zero and fire deleted_by_manager; normal reads still unpin -- _drain_staging mirrors the same branch - touch_keys: no-op -> fires on_l1_keys_accessed (unsynchronized, like stock) - event_bus (mp_observability) publish deferred: observability-only, no tiering-control impact - tests: RecordingListener fake; maru firing edge cases (hit-only, store-fail, temporary reclaim, removed-only, clear) + cross-backend conformance lifecycle test parametrized over stock and maru Signed-off-by: seohui-XCENA <[email protected]>
- promote branches on the staged is_temporary flag (Decision A): temporary (default prefetch) moves the loaded page straight to read staging with no batch_store and no pin -- finish_read reclaims it at refcount zero; retained (prefetch_policy: retain) registers via batch_store then re-resolves the authoritative page with pins so a dup-skip that auto-freed our page still yields the winning shared page - fires on_l1_keys_finish_write_and_reserve_read, never on_l1_keys_write_finished (the latter would make the store controller re-store the key to L2) - extract _pin_retrieve_stage (shared by reserve_read + retained promote) and _store_staged (shared by finish_write + retained promote); reserve_read and finish_write refactored onto them, behavior unchanged - document the load-failure cleanup gap (finish_write->delete on failed keys publishes then removes; caller-side batch-abort is the future fix) - tests: temporary/retained promote, dup-skip re-resolve, extra_count pins, wrong-state/unstaged guards, store-failure, anti re-store event check; cross-backend conformance for retained promote, temporary drop, promote event Signed-off-by: seohui-XCENA <[email protected]>
- daemon sweeper (started in __init__, stopped in close) scans staging under the manager lock and reclaims entries whose TTL has elapsed: an expired write page is returned to the owner (abort_alloc); an expired read releases its pins (a temporary read reclaims its private page instead) - _PendingRead/_PendingWrite carry a monotonic deadline (default never); set at reserve/promote and refreshed on overlapping reserve (mirrors a stock re-lock extending the TTL). abandonment is a time judgement -- a refcount says how many holds exist, not whether they will ever be released - no listener fires on sweep: a late finish_read/unsafe_read then sees KEY_NOT_EXIST and recomputes (same failure path as a stock TTL expiry), and firing across the daemon thread would be a novel hazard for stock listeners - tests: sweeper lifecycle, expired write/read/temporary reclaim, live staging left intact, overlapping reserve refreshes the deadline, late finish is KEY_NOT_EXIST Signed-off-by: seohui-XCENA <[email protected]>
- store/prefetch/eviction controllers type their l1_manager param as the structural L1ManagerInterface instead of the concrete L1Manager, so either L1Manager or MaruL1Manager can drive them; runtime behavior unchanged - the controllers call only Protocol methods (reserve/finish read+write, finish_write_and_reserve_read, delete, is_key_evictable, get_memory_usage, register_listener), so L1Manager still satisfies the param structurally Signed-off-by: seohui-XCENA <[email protected]>
…_layout - StorageManager.__init__ selects the L1 backend: MaruL1Manager when memory_config.maru_config is set, else the stock L1Manager; typed as the shared L1ManagerInterface so the controllers drive either unchanged - new StorageManager.register_kv_layout: maru-gated (isinstance) forward that brings up the CXL pool once the layout is known, rejecting >1 object group (single-model maru limit); a no-op for stock. Engine call site lands later - get_l1_memory_desc is now L1MemoryDesc | None (maru has no single registerable region); the l1_memory_desc property raises if accessed while None (its only consumer, p2p, is rejected at startup for maru) - widen SerdeL2AdapterWrapper l1_manager param to L1ManagerInterface (like the controllers); collect its reserve_write buffers in one pass so the success path types as list[MemoryObj] - tests: maru vs stock selection, register_kv_layout forward / multi-group reject / stock no-op Signed-off-by: seohui-XCENA <[email protected]>
…/p2p/engine) - validate_storage_manager_config: a maru L1 config rejects the other L1 backends (gds-l1-path, l1-devdax-path), store_policy=skip_l1, and L2 adapters that require a single registerable region (nixl / mooncake-rdma); copy-type L2 and the default store policy pass - l1_exposes_single_memory_region returns False for maru (the shared CXL pool has no single registerable region), so the existing p2p startup guard fires; its message now lists maru too - run_http_server rejects maru paired with a non-lmcache_driven transfer mode (engine-driven/auto assume engine-side buffers the shared pool lacks) - tests: maru + gds/devdax/skip_l1/registered-L2 raise; maru + copy-L2 and the default policy pass; l1_exposes False for maru Signed-off-by: seohui-XCENA <[email protected]>
…ol bring-up - after the KV layout is resolved in register_kv_cache, call StorageManager.register_kv_layout with the layout desc, the storage MemoryFormat (KV_MLA_FMT / KV_2LTD via is_mla), chunk size, and object-group count; a no-op for the stock backend, idempotent for maru across instances - on failure (e.g. maru rejecting >1 object group) close the just-built cache context and re-raise so the instance is not left half-registered - runtime test deferred: importing this module needs a c_ops rebuild (built .so lacks execute_object_group_transfer); lands with the C9 G6 test post-rebuild Signed-off-by: seohui-XCENA <[email protected]>
…ister_kv_layout) - G6 (C9): run_http_server rejects maru paired with a non-lmcache_driven transfer mode - C10: register_kv_cache forwards to StorageManager.register_kv_layout with the right MemoryFormat (KV_MLA_FMT / KV_2LTD via is_mla); on a rejected layout it closes the cache context and leaves the instance unregistered; stock registers normally - both were blocked by a stale c_ops build (missing execute_object_group_transfer); unblocked after the rebuild Signed-off-by: seohui-XCENA <[email protected]>
- MaruL1Manager now publishes the same L1 events to the shared event bus as stock L1Manager, alongside the listener notifications: L1_READ_RESERVED, L1_READ_FINISHED (+ L1_KEYS_EVICTED for freed temporaries), L1_WRITE_RESERVED, L1_WRITE_FINISHED, L1_WRITE_FINISHED_AND_READ_RESERVED, and L1_KEYS_EVICTED for delete/clear; touch_keys stays listener-only (matches stock) - add get_event_bus() in __init__ and a _publish helper mirroring stock - closes the observability gap deferred in C4 -- maru L1 ops now surface in the mp_observability dashboards - tests: write/read/delete lifecycle events, promote publishes the promote event (never L1_WRITE_FINISHED), temporary finish_read publishes evicted, touch_keys publishes nothing Signed-off-by: seohui-XCENA <[email protected]>
…stack - a StorageManager harness backed by the maru fakes (fake CXL pool + MaruServer directory) + a mock L2, driving the real Store/Prefetch/Eviction controllers over MaruL1Manager - covers store -> maru directory registration, prefetch of directory-resident keys as a full L1 hit (maru reserve_read), and watermark eviction driving MaruL1Manager.delete on the shared directory - L1<->L2 byte movement (write-through, promote) is stock controller logic that needs real-memory-backed pages and is covered by the stock StorageManager tests, so it is not re-asserted here; extra_count and cross-node PINNED are covered at the manager level (C3/C5) Signed-off-by: seohui-XCENA <[email protected]>
…acts) - add cross-backend conformance cases for the shared error paths: reserve_read of a mid-write key is KEY_NOT_READABLE (not a miss), finish_read/finish_write on an unstaged key is KEY_NOT_EXIST, delete of a missing key is KEY_NOT_EXIST - clear(force=True) is intentionally not conformance-tested: stock clears all objects while maru only drops local staging (the shared directory belongs to other instances) -- a legitimate divergence, not drift Signed-off-by: seohui-XCENA <[email protected]>
- get_memory_usage total = owned pool + CXL device free (cxl_pool.free_size from get_stats), so the eviction watermark tracks whole-device fill instead of just this instance's owned pool (which evicts prematurely while the device still has room) - cache the last-known free and reuse it when a get_stats RPC omits cxl_pool (transient timeout / older server) so total never momentarily collapses to the owned pool and fires a spurious eviction Signed-off-by: seohui-XCENA <[email protected]>
- Track server pins on _PendingRead.pinned separately from refcount, so a temporary stage that absorbs an overlapping reserve_read's pins releases them; previously those pins leaked and left the page un-evictable on MaruServer. - Exclude mid-write keys from reserve_read's pin/stage step: a key mid-write on this instance stays KEY_NOT_READABLE even when a peer has registered it, instead of being double-staged in both _pending_write and _pending_read (which stranded the in-flight write and failed its promote). - Add regression tests for both paths. Signed-off-by: seohui-XCENA <[email protected]>
- Add docstrings to allocate, batched_allocate, close, and memcheck. - Note the RuntimeError raised when called before init_layout(), and the out-of-memory -> None contract that reserve_write's all-or-nothing handling depends on. Signed-off-by: seohui-XCENA <[email protected]>
- Add docs/design/v1/distributed/maru_l1.md: the sibling + Protocol seam, the MaruL1Manager API contracts, the pending-read/pending-write state-machine invariants, the TTL sweeper, and the crash-recovery premises. - Document the Maru CXL shared L1 tier in the MP configuration guide and the architecture index; point the deprecated in-process maru page to it. Signed-off-by: seohui-XCENA <[email protected]>
- Add MaruL1Config.auto_expand (default True) with a --maru-auto-expand / --no-maru-auto-expand CLI flag, threaded into MaruConfig via the allocator. - Branch get_memory_usage on it: auto_expand on keeps the device-fill watermark (owned pool + CXL device free); off anchors total to the owned pool, so a hard-capped pool evicts before it is exhausted instead of OOMing while the device still has free space the pool cannot grow into. - Extend the fake handler with a cxl_free knob and add a watermark test. Signed-off-by: seohui-XCENA <[email protected]>
- Rewrite docs/design/v1/distributed/maru_l1.md to the l2_adapters house style (Overview + components + operation flow + configuration + limits); document the auto_expand knob and the single-device region bound, and drop the internals-heavy framing. - Correct MaruMemoryAllocator docstrings: page reclamation is driven by LMCache eviction via MaruL1Manager (delete / abort_alloc), not the allocator, so free/batched_free are no-ops. Signed-off-by: seohui-XCENA <[email protected]>
- Merge the maru-specific test files into one unit file (test_maru_l1_manager.py): fold in the allocator, config/startup-guard, and control-integration tests, and drop cases already covered by the shared L1Manager conformance suite (notably the notification tests: ~14 -> 4). - Absorb test_l1_protocol.py (structural conformance + shared helpers) into test_l1_manager_conformance.py. - Delete the now-empty test_maru_memory_allocator.py, test_l1_config_maru.py, test_maru_integration.py, and test_l1_protocol.py. Signed-off-by: seohui-XCENA <[email protected]>
- Add --maru-auto-expand / --no-maru-auto-expand to the Maru CXL Shared L1 argument table. - Tighten the single-CXL-device pool-size note and align the tier wording with the design doc (cross-node). Signed-off-by: seohui-XCENA <[email protected]>
…gram Restore the word dropped when MaruL1Manager was added to the EvictionController line, for consistency with the rest of index.rst. Signed-off-by: seohui-XCENA <[email protected]>
- Add the required --l1-size-gb 0 to the quickstart command (ignored under Maru but the parser requires it). - Note that the example assumes a running MaruServer and that higher-level launchers derive --maru-server-url. Signed-off-by: seohui-XCENA <[email protected]>
- lmcache_mp.l1_memory_usage_bytes / l1_usage_ratio are registered in L1Manager.__init__, but the maru path builds MaruL1Manager instead of L1Manager (mutually exclusive), so both gauges silently vanished whenever maru is the L1 backend - mirror the stock registration in MaruL1Manager.__init__: same meter and gauge names, singleton dispatch via _gauge_registered/_gauge_target, and a self-contained ratio helper (no reach into the stock private helper) - conftest: stub maru's register_gauge for distributed tests -- OTel honors only the first registration of a gauge name, so a test-built MaruL1Manager could otherwise claim the name and starve the stock gauge assertions in test_l1_l2_state_metrics.py - test: both gauges registered at construction and their callbacks read the live manager, not the zero fallback Signed-off-by: seohui-XCENA <[email protected]>
Signed-off-by: seohui-XCENA <[email protected]>
Signed-off-by: seohui-XCENA <[email protected]>
Signed-off-by: seohui-XCENA <[email protected]> # Conflicts: # lmcache/v1/multiprocess/modules/lmcache_driven_transfer.py
- Engine hook forwards the raw engine KV format unconditionally (stock no-op) - Map EngineKVFormat to MemoryFormat inside MaruL1Manager.register_kv_layout - Drop requires_kv_layout_registration and the mock pins it forced in tests Signed-off-by: seohui-XCENA <[email protected]>
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diff-review draft — not for merge. This is
maru-mp-l1plus one commit (092f0703) moving the engine KV format mapping out of the shared engine module, opened as a separate branch so PR #20 stays untouched while the full diff againstdevis reviewed.What the extra commit changes:
register_kv_cacheforwards the raw group-0 engine KV format toStorageManager.register_kv_layoutunconditionally (silent no-op for stock L1) — no maru-conditional branch left in the shared moduleEngineKVFormat -> MemoryFormatmapping (the pybindis_mlaprobe) moves intoMaruL1Manager.register_kv_layoutrequires_kv_layout_registrationis dropped, along with the mock pins it forced intest_worker_liveness.py/test_lmcache_driven_layout_registry.py; fully-mocked upstream-style tests can no longer feed mocks into native codeIf adopted, this commit will be folded into the
maru-mp-l1branch (PR #20) and this draft closed.