Skip to content

Commit 52ef359

Browse files
bobbybobby
authored andcommitted
feat: Add multi-worker support and auto CPU detection for SimuNet
- Implement multi-process architecture with automatic worker allocation - Auto-detect CPU cores and set workers to (cores - 2), minimum 1 - Support NUM_WORKERS environment variable and --workers CLI argument - Smart worker count adjustment: limit to device count if needed - Separate log files for each worker to avoid conflicts - Replace all print() statements with log.info() for better logging - Convert all code comments to English for consistency - Update documentation with multi-worker usage examples - Enhance integration test fixtures to support multi-worker mode Performance improvements: - Distribute SSH device servers across multiple worker processes - Reduce resource contention with per-worker device allocation - Maintain single-worker mode with auto-reload for development Documentation updates: - Add multi-process mode usage in quick-start guide - Update CLAUDE.md with worker configuration details - Add feature highlight in README.md Made-with: Cursor
1 parent 96161d0 commit 52ef359

7 files changed

Lines changed: 482 additions & 30 deletions

File tree

CLAUDE.md

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,19 @@ uv sync
7373
uv run agent
7474

7575
# Start simulation network service (SSH servers on configured ports)
76+
# Default: auto-detects CPU cores and uses (cores - 2) workers, minimum 1
7677
uv run simunet
78+
79+
# Specify custom number of workers
80+
NUM_WORKERS=4 uv run simunet
81+
# or
82+
uv run simunet --workers 4
83+
84+
# Force single worker mode (enables auto-reload for development)
85+
NUM_WORKERS=1 uv run simunet
86+
87+
# Auto-cleanup occupied ports before starting
88+
uv run simunet --force
7789
```
7890

7991
### Testing
@@ -97,9 +109,42 @@ uv run pytest tests/bases/netdriver/agent/test_cisco_nexus.py
97109
Configuration files in `config/`:
98110

99111
- `config/agent/agent.yml` - Agent service settings (logging, session timeouts, SSH parameters, profiles)
100-
- Logs are written to `logs/netdriver_agent.log`
112+
- Logs are written to `logs/netdriver_agent.log`
101113
- `config/simunet/simunet.yml` - Simulated device definitions and logging settings
102-
- Logs are written to `logs/netdriver_simunet.log`
114+
- Logs are written to `logs/netdriver_simunet.log`
115+
- In multi-worker mode: `logs/netdriver_simunet_worker_0.log`, `logs/netdriver_simunet_worker_1.log`, etc.
116+
117+
### Multi-Process Mode
118+
119+
Simunet **automatically** uses multi-process mode for improved performance:
120+
121+
**Default Behavior:**
122+
- Automatically detects CPU cores: `workers = max(1, cpu_cores - 2)`
123+
- Example: 8-core system → 6 workers, 4-core → 2 workers, 2-core → 1 worker
124+
- Reserves 2 cores for system and other processes
125+
- **Smart adjustment**: If worker count exceeds device count, automatically adjusts to match device count
126+
127+
**Override Default:**
128+
```bash
129+
# Specify custom number of workers
130+
NUM_WORKERS=4 uv run simunet
131+
# or
132+
uv run simunet --workers 4
133+
134+
# Force single worker (enables auto-reload)
135+
NUM_WORKERS=1 uv run simunet
136+
```
137+
138+
**Performance Guidelines:**
139+
- Small scale (< 10 devices): 1-2 workers
140+
- Medium scale (10-30 devices): 2-4 workers
141+
- Large scale (> 30 devices): 4-8 workers
142+
143+
**Notes:**
144+
- Multi-worker mode (workers > 1) does not support auto-reload
145+
- Single worker mode supports auto-reload for development
146+
- Each worker handles a portion of devices automatically
147+
- Separate log files are created for each worker: `logs/simunet_worker_N.log`
103148

104149
## Development Guidelines
105150

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ Features:
5151
- 📋 **Command Queue** : Ensures sequential command execution on devices, preventing configuration errors and failures caused by concurrent modifications
5252
-**AsyncSSH Foundation** : Superior concurrency capabilities through asynchronous SSH implementation
5353
- 🔌 **Plugin Architecture** : Simplified and accelerated development of new vendor support
54+
- 🚀 **Multi-Process Support** : Automatic worker detection based on CPU cores for optimal performance (SimuNet)
5455

5556
## Comparison
5657

docs/quick-start-simunet.md

Lines changed: 105 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,10 +118,27 @@ python -c "import netdriver.simunet; print('NetDriver Agent installed successful
118118

119119
#### Run
120120

121+
**Basic Usage**:
122+
123+
```bash
124+
# Start SimuNet service
125+
simunet
126+
127+
# Start without auto-reload (production)
128+
simunet --no-reload
129+
130+
# Force cleanup occupied ports before starting
131+
simunet --force
132+
```
133+
134+
**Custom Configuration**:
135+
121136
```bash
122-
uvicorn netdriver.agent.main:app --host 0.0.0.0 --port 8001
137+
simunet --config /path/to/simunet.yml --port 8001
123138
```
124139

140+
**Note**: When ports are already occupied, use the `--force` flag to automatically clean up and restart, avoiding manual intervention.
141+
125142
### Option 2: Install via Docker
126143

127144
#### Prerequisites
@@ -181,6 +198,93 @@ ssh admin@localhost -p 18020
181198

182199
After successful connection, you can execute commands as you would on a real device:
183200

201+
## Advanced Configuration
202+
203+
### Multi-Process Mode
204+
205+
SimuNet automatically uses multi-process mode based on your CPU cores to improve performance. Each worker process handles a portion of the devices.
206+
207+
#### Default Behavior
208+
209+
**Automatic Worker Detection:**
210+
- SimuNet automatically detects CPU cores and uses `CPU cores - 2` workers (minimum 1)
211+
- Example: 8-core CPU → 6 workers, 4-core CPU → 2 workers, 2-core CPU → 1 worker
212+
- This leaves resources for the system and other processes
213+
214+
**Override Default:**
215+
216+
```bash
217+
# Use specific number of workers via environment variable
218+
NUM_WORKERS=4 simunet
219+
220+
# Use specific number of workers via command line
221+
simunet --workers 4
222+
223+
# Force single worker mode
224+
NUM_WORKERS=1 simunet
225+
# or
226+
simunet --workers 1
227+
```
228+
229+
#### Performance Recommendations
230+
231+
- **Small Scale (< 10 devices)**: 1-2 workers
232+
- **Medium Scale (10-30 devices)**: 2-4 workers
233+
- **Large Scale (> 30 devices)**: 4-8 workers
234+
235+
Each worker should ideally manage 5-10 devices for optimal performance.
236+
237+
**Note**:
238+
- Multi-worker mode (workers > 1) does not support auto-reload
239+
- Single worker mode supports auto-reload for development
240+
241+
#### Docker Deployment with Multi-Process
242+
243+
```bash
244+
# Default (auto-detect workers)
245+
docker run -d \
246+
--name netdriver-simunet \
247+
--network host \
248+
-v $(pwd)/config:/app/config \
249+
-v $(pwd)/logs:/app/logs \
250+
ghcr.io/opensecflow/netdriver/netdriver-simunet:latest
251+
252+
# Specify worker count
253+
docker run -d \
254+
--name netdriver-simunet \
255+
--network host \
256+
-e NUM_WORKERS=4 \
257+
-v $(pwd)/config:/app/config \
258+
-v $(pwd)/logs:/app/logs \
259+
ghcr.io/opensecflow/netdriver/netdriver-simunet:latest
260+
```
261+
262+
#### How Device Distribution Works
263+
264+
When using multi-process mode, devices are automatically distributed among workers:
265+
266+
- Worker 0 handles devices 0 to N/W-1
267+
- Worker 1 handles devices N/W to 2N/W-1
268+
- ...
269+
- Last worker handles remaining devices
270+
271+
**Example 1: 21 devices, 3 workers**
272+
- Worker 0: devices 0-6 (7 devices, ports 18020-18025)
273+
- Worker 1: devices 7-13 (7 devices, ports 18026-18031)
274+
- Worker 2: devices 14-20 (7 devices, ports 18032-18040)
275+
276+
**Example 2: 4 devices, 8 workers (auto-adjusted)**
277+
- System: 10 CPU cores → 8 workers by default
278+
- SimuNet automatically adjusts to 4 workers (matches device count)
279+
- Worker 0: device 0 (1 device)
280+
- Worker 1: device 1 (1 device)
281+
- Worker 2: device 2 (1 device)
282+
- Worker 3: device 3 (1 device)
283+
284+
**Important:** If worker count exceeds device count, it's automatically adjusted to match device count to avoid empty workers.
285+
286+
Each worker creates a separate log file: `logs/simunet_worker_0.log`, `logs/simunet_worker_1.log`, etc.
287+
184288
## Next Steps
185289

186290
Now that you have SimuNet running, you can set up [NetDriver Agent](./quick-start-agent.md) to connect to SimuNet

0 commit comments

Comments
 (0)