FORGE: Mitigating Synchronization Amplification for Memory-Disaggregated Caching Systems

This is the implementation repository for the paper FORGE: Mitigating Synchronization Amplification for Memory-Disaggregated Caching Systems. This artifact provides the source code of FORGE and the scripts to configure the environment and run experiments.

Supported Platforms

Our test environment uses the following configurations:

• Hardware: Mellanox ConnectX-5 NICs
• Software: Ubuntu 22.04, MLNX_OFED MLNX_OFED_LINUX-24.10-2.1.8.0, CMake 3.16+, GCC 11+ (e.g., g++ 11.4.0, default on Ubuntu 22.04), libmemcached-dev, memcached, python-memcached, rsync

Repository Layout

├── src/                         FORGE source code
├── experiments/
│   ├── configs/                 Client/server JSON configs
│   ├── scripts/                 Experiment launchers and cluster settings
│   └── workloads/               Workload traces after download
├── scripts/                     Environment setup and check helpers
└── patches/                     Baseline patches for fair comparison

Cluster Model

The experiment scripts manage three types of nodes:

• Controller Node (Node-0): Runs the Memcached service to coordinate MNs/CNs and collect experimental results.
• Memory Nodes (MNs): Run FORGE MN processes to host cached data.
• Compute Nodes (CNs): Run FORGE CN processes to access and manage the cache.

Deployment Note: To save node budget, Node-0 can share a physical machine with one of the CNs or MNs. However, for accurate performance evaluation, CNs and MNs should reside on separate physical machines. Please assign the roles according to your cluster topology.

Setup Environment

0. All Nodes: Clone the Repository

On every node, clone the repository:

git clone https://github.com/cszjyang/forge.git ~/forge
cd ~/forge

1. All Nodes: Dependencies and RDMA Driver

Run the following scripts on all nodes to install required Apt packages and set up the Python environment:

./scripts/install-apt-deps.sh     # Installs build tools, Memcached, rsync, etc.
./scripts/install-python-env.sh   # Creates a repository-local Python virtual environment
./scripts/check-env.sh            # Verifies environment dependencies

If your nodes require an RDMA driver installation, use the provided helper script:

# Note: This script only downloads and unpacks the installer under `scripts/install/`.
# It prints the manual install commands; please execute them only after verifying compatibility with your kernel and OS.
./scripts/prepare-mlnx-ofed.sh

2. Node-0: Configure the Cluster

Navigate to the project root on Node-0:

cd ~/forge

Edit ./experiments/scripts/utils/settings.py to configure your cluster. Key variables include:

• FORGE_HOME: The root path of the FORGE repository.
• user: The current SSH username.
• controller: Node-0 IP.
• server_list: IPs of all involved MNs and CNs.
• mn_list: IPs of the MNs (supports multiple, defaults to 1).
• cn_list: IPs of the CNs (defaults to server_list excluding mn_list).

Ensure that user@controller has passwordless SSH access to all IPs in server_list. Run the following helper script to verify SSH accessibility:

./scripts/check-env.sh

Once verified, run the following command to rsync the project (including your settings.py modifications) to all nodes:

source .venv/bin/activate
python experiments/scripts/sync_project.py

3. All Nodes: Configure Memcached

The Controller Node-0 runs Memcached to coordinate CNs/MNs and aggregate results. Run the following script on all nodes to automatically parse settings.py and configure Memcached:

# For each node, this script sets `memcached_ip` in `./experiments/scripts/shell_settings.sh`.
# On Node-0, it additionally enables the Memcached service.
./scripts/configure-memcached.sh

4. All Nodes: Configure RDMA Parameters

On each node, update the memory-node RDMA IPs and RNIC parameters in experiments/configs/*.json according to your actual cluster setup and assigned node role. For instance, if using InfiniBand, set conn_type='"IB"' and specify the correct GID index.

5. Memory Nodes: Setup Hugepages

On every memory node, configure hugepages:

cd ~/forge
./scripts/setup-hugepages.sh 20480

Note: Hugepage allocations will be lost upon reboot. Rerun this command after restarting any memory node.

Workloads

1. Node-0: Download and Rsync Workloads

We have hosted the evaluated workloads on Google Drive. To download and extract them to experiments/workloads on Node-0:

cd ~/forge
source .venv/bin/activate
python experiments/download_workloads.py

Next, sync the project to push the workloads to all nodes. The initial workload synchronization will take some time to transfer the large files, but subsequent runs will be fast:

python experiments/scripts/sync_project.py

(Optional: Run python experiments/download_workloads.py --more to download an extended set of collected workloads. This will take additional time due to larger file sizes.)

2. Workload Settings & Fairness Note

Workload lists and specific configurations (e.g., cache_config_cmd_map, hash_bucket_num_setting, and mean_kv_size_map) are managed in experiments/scripts/utils/settings.py. These settings are derived offline via the analyzer in the helper script experiments/generate_workload_settings.py. If you introduce new workloads, extend the analyzer, run the helper script, and copy the derived entries into settings.py. Since FORGE manages cache space at the group level, the generated settings enforce cache space alignment with the group chunk size.

Fairness Comparison Note (vs. Ditto): In Ditto's original artifact, server_data_len accounts for the cached object footprint plus the hash table and some extra states (e.g., expert weights). Because FORGE utilizes a different metadata layout from Ditto (e.g., a more compact hash table without hotness metrics per slot), using Ditto's original definition to configure both systems would inadvertently lead to discrepancies in their actual object caching capacities. To ensure fairness, in this artifact, server_data_len strictly defines the cached object footprint only. We have provided patches/ditto-server-data-len-semantics.patch to align Ditto's behavior with this strict definition:

git clone https://github.com/dmemsys/Ditto.git ~/Ditto
cd ~/Ditto
git apply ~/forge/patches/ditto-server-data-len-semantics.patch

By applying the semantic patch and configuring both FORGE and Ditto with this artifact's settings.py, we evaluate all systems under the same aligned object caching capacity, ensuring an apples-to-apples comparison.

Standalone Compilation

To compile the FORGE source code without launching the full cluster scripts, run:

cd ~/forge
./build.sh

Get Started

Experiment launchers are located in experiments/scripts/:

eva1_ycsb.py                           YCSB experiment (Figure 13)
eva2_realworld.py                      Real-world workload experiment (Figure 14)
eva3_realworld_realsize.py             Real-world workload experiment with dynamic object sizes (Figure 15)
eva4_multimn.py                        Multi-memory-node experiment (Figure 16)
eva5_sensitivity_groupsize.py          Group-size sensitivity (Figure 17a)
eva6_sensitivity_penalty.py            Miss-penalty sensitivity (Figure 17b)

Node-0: Launch Experiments

source ~/forge/.venv/bin/activate
cd ~/forge/experiments/scripts
python <script>.py

This script handles the full lifecycle across all nodes: resetting old processes, syncing the project, updating configs, compiling FORGE, launching Controller/MN/CN processes, and ultimately outputting throughput, hit rate, and P50/P99 latency metrics.

To inspect the underlying compilation and execution commands without running them, append the --dry-run flag: python <script>.py --dry-run.

Acknowledgments

This repository adopts portions of the Ditto codebase. We sincerely appreciate their open-source contributions.