Skip to content

Add Live Debugger runtime benchmark coverage #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
watson wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add Live Debugger runtime benchmark coverage #429

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintrc.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ module.exports = {
'packages/plugins/**/scripts/**/*',
'packages/tests/src/_jest/**/*',
'packages/tests/src/_playwright/**/*',
'packages/tests/src/bench/**/*',
'packages/tests/src/e2e/**/*',
],
rules: {
Expand Down
50 changes: 50 additions & 0 deletions .github/actions/setup-playwright-build/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Setup Playwright build
description: Install dependencies, Playwright browsers, and prebuilt plugins for Playwright jobs.
runs:
using: composite
steps:
- name: Install Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version-file: "package.json"

- name: Cache build:all
id: cache-build
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: packages/published/**/dist
key: node18-cache-build-${{ hashFiles('packages/core/**', 'packages/factory/**',
'packages/plugins/**', 'packages/published/**',
'packages/tools/src/**', 'yarn.lock') }}

- name: Cache playwright binaries
id: cache-playwright-binaries
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: |
~/.cache/ms-playwright
~/Library/Caches/ms-playwright
%USERPROFILE%\AppData\Local\ms-playwright
key: cache-playwright-binaries-${{ hashFiles('yarn.lock') }}

- run: yarn install
shell: bash

- name: Install playwright
run: yarn workspace @dd/tests playwright install --with-deps
shell: bash

- name: Build all plugins
if: steps.cache-build.outputs.cache-hit != 'true'
run: yarn build:all-no-types
shell: bash

- name: Save playwright cache
if: always() && steps.cache-playwright-binaries.outputs.cache-hit != 'true'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: |
~/.cache/ms-playwright
~/Library/Caches/ms-playwright
%USERPROFILE%\AppData\Local\ms-playwright
key: cache-playwright-binaries-${{ hashFiles('yarn.lock') }}
171 changes: 129 additions & 42 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,29 +64,7 @@ jobs:
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Install Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version-file: "package.json"

- name: Cache build:all
id: cache-build
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: packages/published/**/dist
key: node18-cache-build-${{ hashFiles('packages/core/**', 'packages/factory/**',
'packages/plugins/**', 'packages/published/**',
'packages/tools/src/**', 'yarn.lock') }}

- name: Cache playwright binaries
id: cache-playwright-binaries
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: |
~/.cache/ms-playwright
~/Library/Caches/ms-playwright
%USERPROFILE%\AppData\Local\ms-playwright
key: cache-playwright-binaries-${{ hashFiles('yarn.lock') }}
- uses: ./.github/actions/setup-playwright-build

- name: Configure Datadog Test Optimization
uses: datadog/test-visibility-github-action@f76512a963e7375dab9ad7f1abc0cacd41806c5c # v2.6.0
Expand All @@ -96,37 +74,146 @@ jobs:
api_key: ${{secrets.DATADOG_API_KEY}}
site: datadoghq.com

- run: yarn install

- name: Install playwright
run: yarn workspace @dd/tests playwright install --with-deps

- name: Build all plugins
if: steps.cache-build.outputs.cache-hit != 'true'
run: yarn build:all-no-types

- run: yarn test:e2e
env:
NODE_OPTIONS: -r ${{env.DD_TRACE_PACKAGE}}
DD_TAGS: type:e2e
DD_ENV: ci

- name: Save playwright cache
if: always() && steps.cache-playwright-binaries.outputs.cache-hit != 'true'
id: save-playwright-cache
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
with:
name: playwright
path: |
~/.cache/ms-playwright
~/Library/Caches/ms-playwright
%USERPROFILE%\AppData\Local\ms-playwright
key: cache-playwright-binaries-${{ hashFiles('yarn.lock') }}
packages/tests/playwright-report
packages/tests/test-results
retention-days: 3

runtime-bench-preflight:
timeout-minutes: 20

name: Live Debugger runtime benchmark preflight
runs-on: ubuntu-latest
if: ${{ github.event_name == 'pull_request' && !startsWith(github.ref_name, 'mq-working-branch-') }}
outputs:
should-run: ${{ steps.output-check.outputs.should-run || steps.changed-files.outputs.should-run }}
reason: ${{ steps.output-check.outputs.reason || steps.changed-files.outputs.reason }}
env:
FORCE_COLOR: true

steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0

- name: Install Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version-file: "package.json"

- run: yarn install

- name: Check benchmark and CI changes
id: changed-files
run: node packages/tests/src/bench/liveDebuggerRuntime/preflight.js --changed-files --base-ref=origin/${{ github.base_ref }}

- name: Check benchmark output changes
if: steps.changed-files.outputs.should-run != 'true'
id: output-check
run: node packages/tests/src/bench/liveDebuggerRuntime/preflight.js --compare-output --base-ref=origin/${{ github.base_ref }}

runtime-bench:
timeout-minutes: 30

name: Live Debugger runtime benchmark
runs-on: ubuntu-latest
needs: runtime-bench-preflight
if: ${{ needs.runtime-bench-preflight.outputs.should-run == 'true' }}
continue-on-error: true
permissions:
contents: read
pull-requests: write
env:
FORCE_COLOR: true

steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: ./.github/actions/setup-playwright-build

- name: Run runtime benchmark
run: yarn workspace @dd/tests bench:live-debugger:runtime

- name: Render benchmark comment
if: always()
run: |
BENCH_TMP_DIR="$(node -e "process.stdout.write(require('os').tmpdir())")"
COMMENT_PATH="$BENCH_TMP_DIR/live-debugger-runtime-bench-comment.md"
RESULTS_PATH="$(ls -t "$BENCH_TMP_DIR"/live-debugger-runtime-bench-results-*.json 2>/dev/null | head -n 1)"
if [ -f "$COMMENT_PATH" ]; then
cp "$COMMENT_PATH" runtime-bench-comment.md
else
{
echo '<!-- ld-runtime-bench -->'
echo '## Live Debugger Runtime Benchmark'
echo
echo 'Benchmark results were not produced. Check the workflow logs for details.'
} > runtime-bench-comment.md
fi
if [ -n "$RESULTS_PATH" ] && [ -f "$RESULTS_PATH" ]; then
cp "$RESULTS_PATH" runtime-bench-results.json
fi

- name: Find benchmark comment
if: always()
id: benchmark-comment
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
const marker = '<!-- ld-runtime-bench -->';
const comments = await github.paginate(github.rest.issues.listComments, {
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const comment = comments.find((candidate) => candidate.body?.includes(marker));
core.setOutput('comment-id', comment?.id || '');

- name: Create benchmark comment
if: always() && steps.benchmark-comment.outputs.comment-id == ''
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
const fs = require('fs');
const body = fs.readFileSync('runtime-bench-comment.md', 'utf8');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});

- name: Update benchmark comment
if: always() && steps.benchmark-comment.outputs.comment-id != ''
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
with:
script: |
const fs = require('fs');
const body = fs.readFileSync('runtime-bench-comment.md', 'utf8');
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: Number('${{ steps.benchmark-comment.outputs.comment-id }}'),
body,
});

- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
if: always()
with:
name: playwright
name: live-debugger-runtime-bench-results
path: |
runtime-bench-comment.md
runtime-bench-results.json
packages/tests/playwright-report
packages/tests/test-results
retention-days: 3
Expand Down
123 changes: 123 additions & 0 deletions packages/plugins/live-debugger/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Contributing to Live Debugger <!-- #omit in toc -->

Developer notes for the Live Debugger plugin.

## Table of content <!-- #omit in toc -->

<!-- #toc -->
- [Development workflow](#development-workflow)
- [Runtime benchmark](#runtime-benchmark)
- [Running it](#running-it)
- [What it measures](#what-it-measures)
- [How to interpret the results](#how-to-interpret-the-results)
- [Methodology](#methodology)
- [Caveats](#caveats)
<!-- #toc -->

## Development workflow

Use the [root contributor guide](../../../CONTRIBUTING.md) for repository setup, formatting, and release process. This page covers the Live Debugger-specific checks that are easy to miss when changing the plugin.

Run the focused unit suite while iterating:

```bash
yarn test:unit packages/plugins/live-debugger
```

Run the package typecheck when changing exported types, option handling, or transform internals:

```bash
yarn workspace @dd/live-debugger-plugin typecheck
```

When changing instrumentation output, add or update cases in [`src/transform/index.test.ts`](./src/transform/index.test.ts). If the generated before/after shape changes in a way users or reviewers should understand, update [`EXAMPLES.md`](./EXAMPLES.md) alongside the tests.

Changes that affect source positions, injected wrappers, return rewriting, or error handling should preserve source maps. Cover those cases in [`src/sourcemap.integration.test.ts`](./src/sourcemap.integration.test.ts).

The Babel packages and `magic-string` are optional peer dependencies for consumers. Keep the transform dependencies lazy-loaded and preserve the user-facing missing-dependency error path. Update [`src/transform/lazy-deps.test.ts`](./src/transform/lazy-deps.test.ts) when touching dependency loading.

Generated code should keep the dormant runtime path small: call `$dd_probes(functionId)` first, and only call `$dd_entry`, `$dd_return`, or `$dd_throw` when a probe is active. Preserve the no-SDK fallback injected from [`src/runtime-bootstrap.ts`](./src/runtime-bootstrap.ts).

When adding or changing `liveDebugger` configuration, update [`src/types.ts`](./src/types.ts), [`src/validate.ts`](./src/validate.ts), [`src/validate.test.ts`](./src/validate.test.ts), and the consumer-facing [`README.md`](./README.md).

## Runtime benchmark

The opt-in browser benchmark measures the dormant runtime overhead added by Live Debugger instrumentation. It compares instrumented code against equivalent uninstrumented code, back-to-back in the same browser session, while SDK-like dormant probe hooks are installed.

### Running it

Run it locally with:

```bash
yarn workspace @dd/tests bench:live-debugger:runtime
```

For a faster loop, pass a browser project:

```bash
yarn workspace @dd/tests bench:live-debugger:runtime --project chrome
```

The terminal output prints one row per browser and workload (`Tiny`, `Hot`). Browser projects run serially to reduce CPU contention. The benchmark uses one fixed bundler so the report focuses on runtime overhead, not on bundler-to-bundler differences.

### What it measures

Each sample measures three variants:

- **baseline**: the uninstrumented workload.
- **control**: the same baseline function measured a second time. This is an A/A diagnostic for timing noise in the benchmark apparatus.
- **instrumented**: the same workload after Live Debugger instrumentation, with dormant SDK hooks installed.

The reporter estimates overhead from `instrumented - control`. That direct paired difference avoids the old correlated-interval comparison against the shared baseline sample. The `control - baseline` result is still shown as the A/A diagnostic; it should be centered around zero if the browser session is quiet enough to trust.

There are two workloads because one number cannot describe every runtime shape:

- **Tiny** calls one very small instrumented function. It is the best row for answering: "what is the smallest cost we can measure for one dormant instrumented call?" Since the function does almost no work, its baseline time is tiny too. That means a small nanosecond cost can look like a large percentage.
- **Hot** runs an uninstrumented loop that calls a small instrumented kernel many times. It is the best row for answering: "what happens when an instrumented function sits on a hot path?" This row includes the cost of the dormant hooks and any optimizer disruption from the instrumented function shape, such as losing an inlining opportunity.

Read them together. `Tiny` shows the minimum cost and the measurement floor. `Hot` shows the repeated-call hot-path cost. If `Hot` is higher than `Tiny` in nanoseconds per call, the gap is the extra cost from the hot-path shape in this benchmark. If `Tiny` is higher in percentage, that usually means the denominator is much smaller, not that `Tiny` has a larger absolute cost.

### How to interpret the results

Start with three columns:

- **per-call overhead upper**: the headline number. It is the conservative upper bound for dormant overhead per instrumented function call, reported in nanoseconds.
- **quality**: whether the row is safe to read. `clean` means use the row, `caution` means it is usable but worth rerunning if the number matters, and `unreliable` means rerun before drawing conclusions.
- **overhead upper**: the same result as a workload-level percentage. Use this as context, not as the main comparison between `Tiny` and `Hot`.

Prefer the nanosecond number when comparing workloads. It puts `Tiny` and `Hot` on the same per-call scale. The percentage can look inverted because it divides by the workload's baseline time. `Tiny` does almost no work, so a small absolute cost can become a large percentage. `Hot` does more baseline work, so a larger absolute cost can still be a smaller percentage.

Example: if `Tiny` reports `<= 1.5 ns` and `Hot` reports `<= 5.0 ns`, the benchmark is saying the hot-path shape costs more per instrumented call. If those same rows report `Tiny <= 40%` and `Hot <= 5%`, that does not contradict the nanosecond result. It only means `Tiny` started from a much smaller baseline.

The other diagnostic columns explain why a row got its `quality` verdict:

- **95% CI**: the signed estimate range for `instrumented - control`. If it sits near zero, the overhead was too small to resolve clearly.
- **A/A diag**: `control - baseline`. This is the benchmark checking itself by timing the baseline code twice. A small value is fine; a value as large as the measured effect means the browser session was noisy.
- **Block CI** and **acf(1)**: checks for timing drift across samples. If the block interval tells the same story as the main interval, the row is usually fine even when `acf(1)` is non-zero.
- **Samples**: how many samples were recorded, plus trimming and outlier diagnostics. A few outliers are expected in browser timing. The row only becomes suspect when one side has enough outliers to survive the 20% trim.

The benchmark treats tiny "speed-ups" as measurement noise. Instrumentation only adds work, but separate baseline and instrumented bundles can land in slightly different code layouts. Below about `0.5 ns/call`, that layout noise is roughly the same size as the effect being measured, so the row is reported as clean but unresolved rather than as a real speed-up.

### Methodology

The benchmark tries to make each browser comparison fair and repeatable:

- It serves the page with cross-origin isolation headers so `performance.now()` has better precision.
- It warms up each workload before measuring, then calibrates the batch size, then warms up again with the calibrated size.
- It calibrates against the slowest variant. This keeps the slow instrumented batches from becoming much longer than the baseline batches, which would make the run more vulnerable to JIT warm-up or thermal drift.
- It records `baseline`, `control`, and `instrumented` back-to-back in rotating forward/reverse order. That keeps each variant from always running first, middle, or last.
- It rounds the sample count to a full counterbalancing period, `2 * variantCount`, so every timing position is represented evenly.

The reported point estimate is a trimmed mean of `instrumented - control`: the benchmark drops the noisiest 20% on each side and averages the middle. Confidence intervals are bootstrapped from the same paired samples. The percentage column uses the same paired data, but divides by the baseline workload time.

The code uses more specific statistical machinery than this section describes, but the practical rule is simple: trust `clean` rows, rerun `unreliable` rows, and compare `Tiny` and `Hot` primarily on `per-call overhead upper`.

### Caveats

Do not compare absolute timings across unrelated machines. Treat the report as a back-to-back comparison from one browser session.

The benchmark builds separate baseline and instrumented bundles. That is necessary for the comparison, but it also means the browser may lay out or optimize the two bundles slightly differently. The A/A diagnostic can catch noise in the baseline path, but it cannot see every instrumented-bundle-specific effect. This is why `Tiny` and `Hot` should be read as a bracket rather than as one universal overhead number.

`Tiny` is close to the measurement floor on fast engines. A tiny negative result, especially below about `0.5 ns/call`, should be read as "too small to resolve", not as instrumentation making code faster.

Browser timings can be spiky or coarsely quantized. The trimmed mean handles ordinary spikes, and the `outliers` reason appears only when the spike pattern is large enough to threaten the estimate. If a row says `unreliable (outliers)`, rerun before trusting it.
1 change: 1 addition & 0 deletions packages/tests/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"./_jest/helpers/*": "./src/_jest/helpers/*.ts"
},
"scripts": {
"bench:live-debugger:runtime": "BUILD_PLUGINS_ENV=test FORCE_COLOR=true PLAYWRIGHT_REQUESTED_BUNDLERS=rspack playwright test --config=playwright.live-debugger-runtime.config.ts",
"build": "yarn clean && tsc",
"clean": "rm -rf dist",
"test:e2e": "BUILD_PLUGINS_ENV=test FORCE_COLOR=true playwright test",
Expand Down
Loading
Loading