System metrics monitoring (CPU, memory, network I/O) during benchmark runs#84
Open
acere wants to merge 4 commits into
Open
System metrics monitoring (CPU, memory, network I/O) during benchmark runs#84acere wants to merge 4 commits into
acere wants to merge 4 commits into
Conversation
…I/O tracking Add an optional callback that monitors system resources during benchmark runs using psutil. Features: - Background thread samples CPU, memory (RSS/VMS), and network I/O - Aggregated stats (avg, p50, p90, p99, max) contributed to Result.stats - Live metrics displayed in the progress bar during runs - Stats persist in stats.json across save/load round-trips - Supports per-process (default) and system-wide monitoring modes - New optional dependency: psutil via `llmeter[system-metrics]` Also adds: - Callback.live_stats() hook for real-time display integration - "System" column in the live stats display for callback metrics - Best practices documentation in Callback base class - User guide documentation (metrics, installation, key concepts) - API reference page - 15 unit tests covering serialization, lifecycle, persistence, and reuse Closes awslabs#83
- Remove _system_metrics_samples monkey-patch on Result (raw samples are already accessible via monitor.samples property) - Remove redundant first-to-last net rate average that was overwritten by the per-interval stats computation
acere
force-pushed
the
feature/system-metrics-monitoring
branch
from
June 30, 2026 02:53
e92e809 to
4551389
Compare
acere
marked this pull request as ready for review
July 2, 2026 03:50
…temMetricsMonitor - Add example notebook demonstrating system metrics monitoring with time-series plots, concurrency correlation, and bottleneck detection - Add integration tests validating full lifecycle against Bedrock: basic usage, save/load persistence, reuse, system-wide mode, and higher concurrency scenarios - Add user guide page for the system metrics callback - Update mkdocs nav and callbacks index to include new docs
…integration - Add plot_samples() method to SystemMetricsMonitor for time-series visualization of collected metrics within a run - Add extra_stats parameter to LoadTestResult.plot_results() and plot_load_test_results() for plotting arbitrary stat keys vs concurrency (e.g. system metrics alongside standard charts) - Persist raw samples to system_metrics_samples.jsonl in after_run when output_path is set, with load_samples() classmethod to restore - Adopt __llmeter_class__/__llmeter_state__ envelope format for save_to_file, compatible with the serialization branch - Remove __getstate__/__setstate__ — the dataclasses.asdict() deepcopy issue in _RunConfig.save() is a library-level concern (see awslabs#94) - Rewrite example notebook to use result.stats, monitor.plot_samples(), and load_test_result.plot_results(extra_stats=...) instead of manual plotly code
acere
force-pushed
the
feature/system-metrics-monitoring
branch
from
July 3, 2026 02:27
5059f8f to
07fb541
Compare
acere
changed the title
Collaborator
|
Only had a very quick skim of this so far but one question came to mind: Since we're not passing huge amounts of data between them and want to prioritize stability of collection, might it make more sense to use a process than a thread for this - in case the main process gets so busy the metric collection thread starts getting starved? |
Contributor
Author
|
Separate process is an option, but realistically the overhead is minimal, and clients thread saturate way before that becomes relevant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
SystemMetricsMonitorcallback that tracks CPU, memory, and network I/O during benchmark runs usingpsutil. Helps detect whether the client machine is a bottleneck during high-concurrency load tests.Closes #83
Features
result.statsand persisted instats.jsonsystem_metrics_samples.jsonlfor time-series analysis after the factmonitor.plot_samples()for quick time-series visualizationload_test_result.plot_results(extra_stats=...)to plot system metrics vs concurrency alongside standard chartsCallback.live_stats()hookpsutilviallmeter[system-metrics]What's included
llmeter/callbacks/system_metrics.py— the callback implementationllmeter/plotting/plotting.py—extra_statsparameter onplot_load_test_resultsexamples/System Metrics Monitoring.ipynbdocs/user_guide/callbacks/system_metrics.mddocs/reference/callbacks/system_metrics.mdKnown limitation
_RunConfig.save()usesdataclasses.asdict()which deepcopies callbacks — this crashes if a callback holds non-serializable runtime state (threads, events). Filed as #94; will be resolved by #54 (unified serialization).