Add WebSocket round-trip metrics (Time To Last Round Trip and Avg RTT) (#850)

## Summary

Adds two round-trip metrics to the realtime WebSocket backend (`openai_websocket`): **Time To Last Round Trip** and an approximate **Avg RTT**. In streaming transcription, input packets and output tokens flow as separate streams, so the existing token metrics (TTFT, ITL, TPOT) do not capture the send-to-receive lag a user actually experiences. These two estimate that lag from the send and receive timestamps and are reported automatically whenever the WebSocket backend is used. Per the issue discussion, Time To First Round Trip is omitted because it overlaps the existing TTFT, and the names avoid `TTFT`/`TTLT` to prevent collision with the token metrics.

## Details

- `schemas/info.py`: add scalar timing fields to `RequestTimings` (`last_request_sent`, `request_sent_sum`/`request_sent_count`, `token_received_sum`/`token_received_count`). Scalars rather than lists, so they stay correct under `RequestInfo.model_copy` (which shallow-copies timings).
- `backends/openai/websocket.py`: record a timestamp after each outbound frame (`session.update`, audio appends, both commits) via a small `_record_request_sent` helper, and accumulate received content-token timestamps in `_record_content_tokens`.
- `schemas/request_stats.py`: add computed properties `time_to_last_round_trip_ms` (last received token minus last sent packet) and `avg_round_trip_time_ms` (mean received minus mean sent). Both return `None` when no sends were recorded, so non-WebSocket backends are unaffected.
- `benchmark/schemas/metrics.py` and `accumulator.py`: add the two `StatusDistributionSummary` fields, compile them in `GenerativeMetrics.compile()`, and accumulate them for live progress.
- `benchmark/outputs/console.py` and `csv.py`: show both in the request-latency table and the CSV export. JSON/YAML already include them via the schema. The HTML report is left for a follow-up.
- `docs/guides/metrics.md`: document both metrics, noting they are WebSocket-only and that Avg RTT is approximate (it assumes sent packets and received tokens line up evenly in time).

## Test Plan

- `tox -e test-unit` passes, including new tests:
  - `tests/unit/backends/openai/test_realtime_ws.py`: the backend records send/receive timestamps (counts, sums, last-sent) over an in-process WebSocket stub.
  - `tests/unit/schemas/test_request_stats.py`: the two properties compute correctly and return `None` without send timings.
  - `tests/unit/benchmark/schemas/test_metrics.py`: the metrics expose schema fields and compile into distributions.
- `tox -e lint-check` and `tox -e type-check` pass.
- Manual check on CPU (no GPU): drove `OpenAIWebSocketBackend.resolve()` against a fake realtime server with ~100 ms simulated per-token lag. Time To Last Round Trip came out around 300 ms and Avg RTT around 200 ms (matching the lag), and both were `None` for a non-WebSocket request. For a live run, point a benchmark at a vLLM realtime server (e.g. Voxtral) and confirm both metrics appear in the console latency table and `benchmarks.json`.

## Related Issues

- Resolves #832

---

- [x] "I certify that all code in this PR is my own, except as noted below."

## Use of AI

- [x] Includes code generated or substantially modified by an AI agent
- [x] Includes tests generated or substantially modified by an AI agent

> NOTE: the `Generated-by` or `Assisted-by` trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's [`DEVELOPING.md`](https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md) file.



---

# git log

commit 457d56e
Author: Suraj Singh <[email protected]>
Date:   Wed Jun 24 15:00:52 2026 -0700

    test(openai): add failing websocket round-trip metric tests (#832)
    
    Add RED unit tests for Time To Last Round Trip and Avg RTT on the openai_websocket backend (send/receive timestamp recording, computed properties, None-gating, and metric compile). They fail until the implementation lands.
    
    Assisted-by: Claude Code
    Signed-off-by: Suraj Singh <[email protected]>

commit 81a0aee
Author: Suraj Singh <[email protected]>
Date:   Wed Jun 24 15:00:52 2026 -0700

    feat(openai): record websocket send/receive timestamps (#832)
    
    Add scalar round-trip timing fields to RequestTimings and record per-frame sent timestamps and received content-token timestamps in the openai_websocket backend. These feed the Time To Last Round Trip and Avg RTT metrics; HTTP backends leave them unset.
    
    Assisted-by: Claude Code
    Signed-off-by: Suraj Singh <[email protected]>

commit 94f941c
Author: Suraj Singh <[email protected]>
Date:   Wed Jun 24 15:00:52 2026 -0700

    feat(benchmark): compute and aggregate websocket round-trip metrics (#832)
    
    Add time_to_last_round_trip_ms and avg_round_trip_time_ms as computed request stats (None unless send timings exist), aggregate them in GenerativeMetrics.compile(), and accumulate them for live progress. Avg RTT is an approximation of the mean send-to-receive lag.
    
    Assisted-by: Claude Code
    Signed-off-by: Suraj Singh <[email protected]>

commit c5efcaf
Author: Suraj Singh <[email protected]>
Date:   Wed Jun 24 15:00:52 2026 -0700

    feat(benchmark): show websocket round-trip metrics in console and CSV (#832)
    
    Surface Time To Last Round Trip and Avg RTT in the request latency console table and the CSV export. JSON/YAML already include them via the schema; the HTML report is left for a follow-up.
    
    Assisted-by: Claude Code
    Signed-off-by: Suraj Singh <[email protected]>

commit da70077
Author: Suraj Singh <[email protected]>
Date:   Wed Jun 24 15:00:53 2026 -0700

    docs(metrics): document websocket round-trip metrics (#832)
    
    Describe Time To Last Round Trip and Avg RTT in the metrics guide, noting they are websocket-only and that Avg RTT is an approximation.
    
    Assisted-by: Claude Code
    Signed-off-by: Suraj Singh <[email protected]>

commit ee76dd2
Author: Suraj Singh <[email protected]>
Date:   Tue Aug 11 09:59:50 2026 -0700

    fix(outputs): hide realtime-only metrics from console
    
    Signed-off-by: Suraj Singh <[email protected]>

---------

Assisted-by: Claude Code
Signed-off-by: Suraj Singh <[email protected]>

Author: dreamer-89

docs/guides/metrics.md

@@ -71,6 +71,16 @@ These metrics provide a breakdown of the overall request statuses, helping users
 - **Definition**: The average time taken to generate each output token, including the first token.
 - **Use Case**: Provides a detailed view of the model's token generation efficiency.
 
+### Time To Last Round Trip
+
+- **Definition**: For the realtime WebSocket backend (`openai_websocket`), the time from the last sent packet to the last received token.
+- **Use Case**: Measures tail latency of a streaming exchange (how long the final output lags the final input).
+
+### Average Round-Trip Time (Avg RTT)
+
+- **Definition**: For the WebSocket backend, the mean of received-token timestamps minus the mean of sent-packet timestamps.
+- **Use Case**: Estimates the average send-to-receive lag across a request. It is approximate, since it assumes sent packets and received tokens line up evenly in time.
+
 ## Statistical Summaries
 
 GuideLLM provides detailed statistical summaries for each of the above metrics using the `StatusDistributionSummary` and `DistributionSummary` models. These summaries include the following statistics:

src/guidellm/backends/openai/websocket.py

@@ -80,6 +80,9 @@ def _record_content_tokens(
     if content_tokens <= 0:
         return False
 
+    request_info.timings.token_received_sum += iter_time
+    request_info.timings.token_received_count += 1
+
     if request_info.timings.first_token_iteration is None:
         request_info.timings.first_token_iteration = iter_time
         request_info.timings.token_iterations = 0
@@ -92,6 +95,18 @@ def _record_content_tokens(
     return False
 
 
+def _record_request_sent(request_info: RequestInfo) -> None:
+    """
+    Record the timestamp of one outbound WebSocket frame for round-trip metrics.
+
+    :param request_info: Mutable timing state for the in-flight request.
+    """
+    sent_time = time.time()
+    request_info.timings.last_request_sent = sent_time
+    request_info.timings.request_sent_sum += sent_time
+    request_info.timings.request_sent_count += 1
+
+
 def _load_ws_event(raw: str) -> dict[str, Any]:
     """Parse a JSON WebSocket text frame; raise RuntimeError on invalid JSON."""
     try:
@@ -377,8 +392,9 @@ async def resolve(  # type: ignore[override, misc]  # noqa: C901, PLR0912, PLR09
         self,
         request: GenerationRequest,
         request_info: RequestInfo,
-        history: list[tuple[GenerationRequest, GenerationResponse | None]]
-        | None = None,
+        history: (
+            list[tuple[GenerationRequest, GenerationResponse | None]] | None
+        ) = None,
     ) -> AsyncIterator[tuple[GenerationResponse | None, RequestInfo]]:
         """
         Stream one realtime transcription over WebSocket for a single audio column.
@@ -454,18 +470,22 @@ async def resolve(  # type: ignore[override, misc]  # noqa: C901, PLR0912, PLR09
                         f"Expected session.created, got {first_event.get('type')!r}"
                     )
                 await ws.send(_json_text(session_update))
+                _record_request_sent(request_info)
                 for b64_chunk in chunks:
                     await ws.send(
                         _json_text(
                             {"type": "input_audio_buffer.append", "audio": b64_chunk}
                         )
                     )
+                    _record_request_sent(request_info)
                 await ws.send(
                     _json_text({"type": "input_audio_buffer.commit", "final": False})
                 )
+                _record_request_sent(request_info)
                 await ws.send(
                     _json_text({"type": "input_audio_buffer.commit", "final": True})
                 )
+                _record_request_sent(request_info)
 
                 ignored_events = 0
                 while True:

src/guidellm/benchmark/outputs/console.py

@@ -480,7 +480,6 @@ def print_request_latency_table(self, report: GenerativeBenchmarksReport):
                 group="TPOT",
                 name="ms",
             )
-
         headers, values = columns.get_table_data()
         self.console.print("\n")
         self.console.print_table(

src/guidellm/benchmark/outputs/csv.py

@@ -455,6 +455,20 @@ def _add_request_latency_metrics(
             "Inter Token Latency",
             "ms",
         )
+        self._add_stats_for_metric(
+            headers,
+            values,
+            benchmark.metrics.time_to_last_round_trip_ms,
+            "Time To Last Round Trip",
+            "ms",
+        )
+        self._add_stats_for_metric(
+            headers,
+            values,
+            benchmark.metrics.avg_round_trip_time_ms,
+            "Avg Round Trip Time",
+            "ms",
+        )
 
     def _add_server_throughput_metrics(
         self,

src/guidellm/benchmark/schemas/accumulator.py

@@ -495,6 +495,14 @@ class GenerativeMetricsAccumulator(StandardBaseModel):
         default_factory=RunningMetricStats,
         description="Accumulated time to first token statistics in milliseconds",
     )
+    time_to_last_round_trip_ms: RunningMetricStats = Field(
+        default_factory=RunningMetricStats,
+        description="Accumulated websocket last round-trip latency in milliseconds",
+    )
+    avg_round_trip_time_ms: RunningMetricStats = Field(
+        default_factory=RunningMetricStats,
+        description="Accumulated websocket average round-trip time in milliseconds",
+    )
     time_to_first_output_token_ms: RunningMetricStats = Field(
         default_factory=RunningMetricStats,
         description="Accumulated time to first content token stats in ms",
@@ -539,6 +547,12 @@ def update_estimate(self, stats: GenerativeRequestStats, duration: float):
         self.time_to_first_token_ms.update_estimate(
             stats.time_to_first_token_ms, duration=duration
         )
+        self.time_to_last_round_trip_ms.update_estimate(
+            stats.time_to_last_round_trip_ms, duration=duration
+        )
+        self.avg_round_trip_time_ms.update_estimate(
+            stats.avg_round_trip_time_ms, duration=duration
+        )
         self.time_to_first_output_token_ms.update_estimate(
             stats.time_to_first_output_token_ms, duration=duration
         )

src/guidellm/benchmark/schemas/metrics.py

@@ -813,6 +813,18 @@ class GenerativeMetrics(StandardBaseDict):
     inter_token_latency_ms: StatusDistributionSummary = Field(
         description="Distribution of inter-token latencies in milliseconds"
     )
+    time_to_last_round_trip_ms: StatusDistributionSummary = Field(
+        description=(
+            "Distribution of websocket last-round-trip latencies in milliseconds "
+            "(last received token minus last sent packet)"
+        )
+    )
+    avg_round_trip_time_ms: StatusDistributionSummary = Field(
+        description=(
+            "Distribution of approximate websocket average round-trip times in "
+            "milliseconds (mean received minus mean sent)"
+        )
+    )
     prompt_tokens_per_second: StatusDistributionSummary = Field(
         description="Distribution of prompt token processing rates"
     )
@@ -939,6 +951,18 @@ def compile(cls, accumulator: GenerativeBenchmarkAccumulator) -> GenerativeMetri
                 incomplete=incomplete,
                 errored=errored,
             ),
+            time_to_last_round_trip_ms=StatusDistributionSummary.from_values_function(
+                function=lambda req: req.time_to_last_round_trip_ms or 0.0,
+                successful=successful,
+                incomplete=incomplete,
+                errored=errored,
+            ),
+            avg_round_trip_time_ms=StatusDistributionSummary.from_values_function(
+                function=lambda req: req.avg_round_trip_time_ms or 0.0,
+                successful=successful,
+                incomplete=incomplete,
+                errored=errored,
+            ),
             time_to_first_output_token_ms=StatusDistributionSummary.from_values_function(
                 function=lambda req: req.time_to_first_output_token_ms or 0.0,
                 successful=successful,

src/guidellm/schemas/info.py

@@ -78,6 +78,35 @@ class RequestTimings(StandardBaseDict):
     token_iterations: int = Field(
         default=0,
     )
+    last_request_sent: float | None = Field(
+        default=None,
+        description=(
+            "Unix timestamp of the last packet sent to the server, used for "
+            "round-trip metrics (openai_websocket backend)"
+        ),
+    )
+    request_sent_sum: float = Field(
+        default=0.0,
+        description=(
+            "Sum of sent-packet timestamps for mean round-trip estimation "
+            "(openai_websocket backend)"
+        ),
+    )
+    request_sent_count: int = Field(
+        default=0,
+        description="Number of packets sent to the server (openai_websocket backend)",
+    )
+    token_received_sum: float = Field(
+        default=0.0,
+        description=(
+            "Sum of received content-token timestamps for mean round-trip "
+            "estimation (openai_websocket backend)"
+        ),
+    )
+    token_received_count: int = Field(
+        default=0,
+        description="Number of content tokens received (openai_websocket backend)",
+    )
     request_end: float | None = Field(
         default=None,
         description="Unix timestamp when the backend completed processing the request",

src/guidellm/schemas/request_stats.py

@@ -170,6 +170,45 @@ def time_to_first_token_ms(self) -> float | None:
 
         return 1000 * (first_token - start)
 
+    @computed_field  # type: ignore[misc]
+    @property
+    def time_to_last_round_trip_ms(self) -> float | None:
+        """
+        Time from the last sent packet to the last received token in milliseconds.
+
+        Only populated by the websocket backend, which records send timestamps;
+        None for backends that do not record sends.
+
+        :return: Last round-trip latency in milliseconds, or None if unavailable
+        """
+        last_received = self.info.timings.last_token_iteration
+        last_sent = self.info.timings.last_request_sent
+        if last_received is None or last_sent is None:
+            return None
+
+        return 1000 * (last_received - last_sent)
+
+    @computed_field  # type: ignore[misc]
+    @property
+    def avg_round_trip_time_ms(self) -> float | None:
+        """
+        Approximate average round-trip time in milliseconds.
+
+        Computed as the mean of received content-token timestamps minus the mean
+        of sent-packet timestamps. This is an approximation that assumes sent
+        packets and received tokens align uniformly in time. Only populated by
+        the websocket backend; None otherwise.
+
+        :return: Average round-trip time in milliseconds, or None if unavailable
+        """
+        timings = self.info.timings
+        if timings.request_sent_count <= 0 or timings.token_received_count <= 0:
+            return None
+
+        mean_sent = timings.request_sent_sum / timings.request_sent_count
+        mean_received = timings.token_received_sum / timings.token_received_count
+        return 1000 * (mean_received - mean_sent)
+
     @computed_field  # type: ignore[misc]
     @property
     def time_per_output_token_ms(self) -> float | None:

tests/unit/backends/openai/test_realtime_ws.py

@@ -656,3 +656,126 @@ def test_openai_websocket_backend_args_invalid_request_format_rejected() -> None
             target="http://localhost:8000",
             request_format="nope",
         )
+
+
+@pytest.mark.asyncio
+async def test_resolve_records_round_trip_timings() -> None:
+    """Round-trip send/receive timestamps are recorded during resolve.
+
+    ## WRITTEN BY AI ##
+    """
+
+    async def handler(ws: object) -> None:
+        await ws.send(
+            json.dumps({"type": "session.created", "id": "sess-rtt", "created": 0})
+        )
+        while True:
+            msg = await ws.recv()
+            data = json.loads(msg if isinstance(msg, str) else msg.decode())
+            if data.get("type") == "input_audio_buffer.commit" and data.get("final"):
+                break
+        await ws.send(json.dumps({"type": "transcription.delta", "delta": "hi"}))
+        await ws.send(
+            json.dumps(
+                {
+                    "type": "transcription.done",
+                    "text": "hi",
+                    "usage": {
+                        "prompt_tokens": 5,
+                        "completion_tokens": 1,
+                        "total_tokens": 6,
+                    },
+                }
+            )
+        )
+
+    async with serve(handler, "127.0.0.1", 0) as server:
+        port = server.sockets[0].getsockname()[1]
+        be = _make_ws_backend(
+            target=f"http://127.0.0.1:{port}",
+            model="m",
+            validate_backend=False,
+        )
+        await be.process_startup()
+        req = GenerationRequest(
+            request_id="rtt",
+            columns={
+                "audio_column": [
+                    {"audio": b"fake", "format": "mp3", "file_name": "f.mp3"}
+                ]
+            },
+        )
+        info = RequestInfo(timings=RequestTimings())
+        async for _ in be.resolve(req, info):
+            pass
+        await be.process_shutdown()
+
+    t = info.timings
+    # one patched chunk -> session.update + 1 append + 2 commits = 4 sends
+    assert t.request_sent_count == 4
+    assert t.last_request_sent is not None
+    assert t.request_sent_sum > 0
+    # one non-empty delta received
+    assert t.token_received_count == 1
+    assert t.token_received_sum > 0
+    # tokens arrive after the last packet is sent -> last round trip >= 0
+    assert t.last_token_iteration is not None
+    assert t.last_token_iteration >= t.last_request_sent
+
+
+@pytest.mark.asyncio
+async def test_resolve_records_round_trip_timings_done_only() -> None:
+    """Send/receive timings are recorded when text arrives only on done.
+
+    ## WRITTEN BY AI ##
+    """
+
+    async def handler(ws: object) -> None:
+        await ws.send(
+            json.dumps({"type": "session.created", "id": "sess-done", "created": 0})
+        )
+        while True:
+            msg = await ws.recv()
+            data = json.loads(msg if isinstance(msg, str) else msg.decode())
+            if data.get("type") == "input_audio_buffer.commit" and data.get("final"):
+                break
+        await ws.send(
+            json.dumps(
+                {
+                    "type": "transcription.done",
+                    "text": "hello",
+                    "usage": {
+                        "prompt_tokens": 5,
+                        "completion_tokens": 1,
+                        "total_tokens": 6,
+                    },
+                }
+            )
+        )
+
+    async with serve(handler, "127.0.0.1", 0) as server:
+        port = server.sockets[0].getsockname()[1]
+        be = _make_ws_backend(
+            target=f"http://127.0.0.1:{port}",
+            model="m",
+            validate_backend=False,
+        )
+        await be.process_startup()
+        req = GenerationRequest(
+            request_id="rtt-done",
+            columns={
+                "audio_column": [
+                    {"audio": b"fake", "format": "mp3", "file_name": "f.mp3"}
+                ]
+            },
+        )
+        info = RequestInfo(timings=RequestTimings())
+        async for _ in be.resolve(req, info):
+            pass
+        await be.process_shutdown()
+
+    t = info.timings
+    assert t.request_sent_count == 4
+    assert t.last_request_sent is not None
+    assert t.token_received_count == 1
+    assert t.token_received_sum > 0