Load Generation for Plateau#83
Open
AnIrishDuck wants to merge 14 commits into
Open
Conversation
…vers Extracts shared worker loop from run() into run_tasks(), adds run_external() that skips the embedded server, and adds a new `load` binary with CLI flags for URL, sample file, topics, partitions, rows, write interval, and duration. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
- `load`: continuous streaming load generator targeting an existing server
(--url, --sample, --topics, --partitions, --rows, --interval-ms, --duration-secs)
- `batch-load`: batch-job simulator with staggered per-partition schedules
- TOML config with per-topic Arrow sample files (each topic keeps a fixed schema)
- Deterministic per-batch RNG seed so data is reproducible
- Evenly staggered fire times across all partitions within each batch period
- --speed multiplier to compress schedule (e.g. speed=60 runs 1h batches every 1min)
- JSON state file tracks last completed batch per partition; on restart,
missed batches are caught up immediately before resuming normal schedule
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
Each topic now carries its own partitions/rows/batch_interval, with an optional [defaults] table supplying fallbacks. batch_interval is therefore per-topic, so different topics run on independent (sped-up) schedules and each topic's partitions are staggered across that topic's own period. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
…ation Replaces the hand-listed [[topics]] config with a generative approach: - [topics] count/active/rotation_interval configure a pool of N topics, with `active` topics writing at any given time - columns_min/columns_max define the random column count range per topic - Schemas are generated once (via sample_flat + FromDataType), written as Arrow IPC files to schemas_dir, and reloaded on restart — no data loss - A stable schemas_seed in state.json enables regeneration if files are lost - The active window slides forward by `active` every rotation_interval/speed; on restart, the window is recomputed from elapsed time so the sim stays consistent with the wall clock - Each topic's partitions are staggered across that topic's batch period - Catchup: missed batches fire immediately on restart before resuming schedule Example config: speed = 60.0 [topics] count = 200 active = 8 rotation_interval = "1h" columns_min = 3 columns_max = 35 batch_interval = "1h" partitions = 4 rows = 10000 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
…mpling - partitions_min/max: each topic draws its own partition count - columns_min/max: now sampled from a normal distribution (was uniform) - rows_min/max define an overall rows-per-insert distribution; each topic draws two values from it to form its own [min, max] sub-range, and every insert samples a row count from that per-topic range - All range draws use a clamped normal distribution (Box-Muller, mean at midpoint, sigma = range/4) via a shared normal_range helper - Per-topic parameters are derived deterministically from hash(schemas_seed, topic_idx), so they are stable and recomputable across restarts without a manifest; schema files are independently regenerable - Add unit tests: range bounds, param determinism, schema load-back + no spurious regeneration Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
Dockerfile: - Combined cargo build now builds both plateau and batch-load binaries - Named the existing final stage `plateau` - Added `batch-load` stage copying the bench binary CI (rust.yml): - Added BENCH_REGISTRY_IMAGE env var (ghcr.io/wallaroolabs/plateau-bench) - Added --target plateau to existing build jobs - Added build-bench-amd64, build-bench-arm64, merge-bench jobs mirroring the plateau pattern - Bench jobs run on push to main, version tags, workflow_dispatch, and PRs (guarded by same-repo check so fork PRs don't fail on missing secrets) - merge-bench tags with sha, branch, pr number, and semver Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
bench-image.yml is workflow_dispatch-only so it never runs on PRs or routine pushes. Trigger it from the Actions tab (or gh workflow run bench-image.yml --ref <branch>) whenever a new bench image is needed. rust.yml is cleaned up: BENCH_REGISTRY_IMAGE env var and the bench build jobs are removed; the plateau build jobs retain --target plateau. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
No arm build needed. Drops the digest/artifact/merge pattern in favour of a single build-and-push job that pushes tags directly. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Claude-Session: https://claude.ai/code/session_01GAMUPUAesj4bPh2y2cvaKc
append_records sends the full batch in one request, which can exceed the server's DefaultBodyLimit when rows_max is large. append_queue auto-splits at the client's DEFAULT_MAX_BATCH_BYTES (100KB), so large batches are chunked before they hit Axum's body limit. Co-Authored-By: Claude <[email protected]>
mnp
approved these changes
Jun 25, 2026
… on failure Axum 0.6's DefaultBodyLimit returns 400 (not 413) when the limit is exceeded, so append_queue's auto-shrink never fires. Fix by: - Adding Client::with_max_batch_bytes() to allow callers to set a conservative limit below the server's 10MB wall - Setting 8MB in run_batch so append_queue splits proactively before sending - Logging the row count on batch failure for easier diagnosis Co-Authored-By: Claude <[email protected]>
Running batch-load through kubectl port-forward causes spurious 400 "Failed to buffer the request body" errors — the apiserver-proxied tunnel truncates request bodies under sustained load. Running in-cluster talks straight to the Service and avoids this. Adds bench/k8s/ with: - job.yaml: ConfigMap + PVC + Job (resumable via persisted state/schemas) - batch-config.toml: sample config - README.md: buildx build/push commands (no CI needed) and deploy steps Co-Authored-By: Claude <[email protected]>
batch-load lives in the bench package, not plateau. Build both with -p plateau -p bench so both binaries are available to copy. Co-Authored-By: Claude <[email protected]>
plateau-bench is private like the other org packages, so the Job needs the same ghcr.io pull secret the plateau pods use. Reference it via imagePullSecrets (placeholder name to be set per-cluster). Co-Authored-By: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This code is very low-quality (e.g. re-implements sampling from a standard distribution instead of using
rand) and effectively throw-away. Here for reference only.