Skip to content

TESTING.md

Latest commit

 

History

History
197 lines (153 loc) · 8.81 KB

TESTING.md

File metadata and controls

197 lines (153 loc) · 8.81 KB
Raw

Abbenay Tests

Test Structure

This project follows the "hybrid" test layout:

  • Unit tests live co-located with the source code (src/**/*.test.ts)
  • Integration tests live in a separate tests/ folder inside each package
packages/daemon/
├── src/
│   ├── core/
│   │   ├── mock.ts
│   │   ├── mock.test.ts              <- Unit test (co-located)
│   │   ├── config.ts
│   │   ├── config.test.ts            <- Unit test (co-located)
│   │   ├── tool-registry.test.ts     <- Unit test (glob matching, registry)
│   │   ├── tool-approval.test.ts     <- Unit test (3-tier approval logic)
│   │   ├── session-store.test.ts     <- Unit test (session CRUD, index)
│   │   └── session-summarizer.test.ts <- Unit test (periodic summary logic)
│   ├── daemon/
│   │   ├── chat-prompt.test.ts       <- Unit test (parseApprovalInput)
│   │   └── web/
│   │       └── openai-compat.test.ts <- Unit test (OpenAI format mapping)
│   └── state.test.ts                 <- Unit test (DaemonState)
├── tests/
│   └── integration/
│       ├── grpc-streaming.test.ts    <- Integration (real gRPC server/client)
│       ├── grpc-tls.test.ts          <- TLS bind policy + SetSecret over TLS
│       ├── grpc-real-service.test.ts <- Real service RPCs
│       ├── web-sse.test.ts           <- Integration (real Express + HTTP)
│       ├── openai-compat.test.ts     <- Integration (OpenAI-compat API)
│       ├── sessions.test.ts          <- Integration (session CRUD + chat SSE)
│       └── helpers/
│           └── mock-daemon.ts        <- Shared mock gRPC server
├── vitest.config.ts
└── package.json

Running Tests

# Run all daemon tests (unit + integration)
cd packages/daemon
npm test

# Run with verbose output
npx vitest run --reporter verbose

# Run only unit tests
npx vitest run src/

# Run only integration tests
npx vitest run tests/

# Watch mode
npx vitest

Test Layers

Layer 1: Unit Tests (co-located)

Pure unit tests with no I/O, no network, no processes.

File What it covers
src/core/mock.test.ts Mock engine: echo, fixed, error, empty, slow modes
src/core/config.test.ts Config loading, merging, validation
src/core/config-schema.test.ts Zod ConfigFile / PolicyConfig schemas (types, field injection)
src/core/tool-registry.test.ts Glob matching, tool registration, resolution, policy filtering
src/core/tool-approval.test.ts Shared validator: disabled / auto_approve / require_approval / default ask
src/daemon/mcp-server.test.ts MCP authorizeAndExecute honors tool_policy (no bypass)
src/state.test.ts DaemonState: provider listing, model listing, chat flow
src/daemon/chat-prompt.test.ts parseApprovalInput case-sensitive routing
src/daemon/web/openai-compat.test.ts OpenAI format mapping: models, finish reasons, stream chunks, complete responses
src/daemon/web/validate-body.test.ts HTTP body parse helpers + workspace path allowlist / traversal
src/core/session-store.test.ts SessionStore: CRUD, appendMessage, updateTitle, updateSummary, index consistency
src/core/session-summarizer.test.ts generateSessionSummary, maybeSummarize interval logic, error handling

Layer 2: Integration Tests

Tests that start real servers, make real HTTP/gRPC calls.

File What it covers
tests/integration/grpc-streaming.test.ts gRPC unary RPCs + streaming + cancellation + concurrency
tests/integration/grpc-tls.test.ts gRPC TLS bind policy + SetSecret/GetSecret over TLS
tests/integration/grpc-bind-e2e.test.ts Subprocess E2E: localhost OK, 0.0.0.0 refuse, TLS/insecure + consumers/open-auth
tests/integration/consumer-auth.test.ts Consumer capability gating on sensitive RPCs
src/daemon/server/consumer-auth.test.ts Unit: timing-safe tokens, fail-closed bind, capability matrix
tests/integration/web-sse.test.ts Web API endpoints + SSE chat + config Zod validation / workspace allowlist
tests/integration/openai-compat.test.ts OpenAI-compatible API: /v1/models, streaming, non-streaming, errors, tool calls
tests/integration/sessions.test.ts Session REST API: CRUD endpoints, session chat SSE streaming + persistence
tests/integration/mcp-http-policy.test.ts /mcp auth + connection consent + tool_policy E2E
tests/integration/mcp-stdio-spawn-policy.test.ts Dynamic stdio MCP: allowlist + approval + auth (H6 / DR-043)
tests/integration/http-security.test.ts HTTP auth, CORS, bind defaults, dashboard login
tests/integration/discover-models-auth.test.ts discover-models: reject ?apiKey=; accept X-Api-Key / body
tests/integration/provider-endpoint-security.test.ts Provider base_url: unauth denied, bad endpoint rejected, allowed + audit
src/core/provider-endpoint.test.ts Unit: scheme/host policy, allowlist, audit log
src/core/secrets-audit.test.ts Unit: secret mutation audit never logs values (A1)

Mock Engine

The mock engine (mock/echo, mock/fixed, mock/error, etc.) is a real engine registered in core/engines.ts. No API keys or network needed. Use it for end-to-end testing:

# Chat with the mock engine via web API
curl -X POST http://localhost:8787/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"mock/echo","messages":[{"role":"user","content":"Hello!"}]}'

VS Code Extension Tests

The extension uses @vscode/test-cli with Mocha to run tests inside a real VS Code instance.

cd packages/vscode
npm test

On headless Linux: xvfb-run -a npm test

Test Structure

packages/vscode/
├── src/test/
│   ├── extension.test.ts               <- Smoke tests (activation, commands, config)
│   ├── helpers/
│   │   ├── mockDaemonClient.ts         <- Stub DaemonClient with test data
│   │   └── mockWebview.ts              <- Stub vscode.Webview with message recorder
│   └── webviews/
│       ├── getNonce.test.ts            <- Pure function (nonce generation)
│       ├── getWebviewContent.test.ts   <- HTML template generation
│       ├── providerHandler.test.ts     <- Provider message handler
│       └── chatHandler.test.ts         <- Chat message handler
└── .vscode-test.mjs                    <- Test runner config

Smoke Tests

Test What it covers
extension.test.ts Extension activation, command registration, chat view, config defaults

Webview Handler Tests

Handler tests mock the DaemonClient and vscode.Webview to test message routing, data transformation, and error handling without a running daemon.

Test What it covers
getNonce.test.ts Nonce format (32 hex chars), uniqueness
getWebviewContent.test.ts HTML structure, CSP headers, nonce injection, panel-specific assets
providerHandler.test.ts Ready message routing, engine listing, model discovery with credentials, config forwarding, error handling
chatHandler.test.ts Ready message routing, model listing, session create/delete, stream cancellation, error handling

Mock Helpers

  • mockDaemonClient.ts — Creates a stub DaemonClient with empty default return values. Override individual methods per test.
  • mockWebview.ts — Creates a stub vscode.Webview that records all postMessage() calls in a .messages array for assertion.

Coverage

The daemon package generates coverage via Vitest + v8:

cd packages/daemon
npm run test:coverage          # produces coverage/ with lcov + text

CI uploads the lcov report to Codecov on every push and PR. PRs get an inline coverage summary comment showing the diff impact.

Configuration

  • codecov.yml — project/patch thresholds, PR comment layout, flag definitions.
  • Project target: auto (must not drop from current baseline by more than 1%).
  • Patch target: 80% (new/changed lines in a PR).
  • The daemon flag scopes coverage to packages/daemon/src/.

Troubleshooting

  • No coverage comment on PR: check that the CODECOV_TOKEN repo secret is set.
  • Coverage report not found: ensure packages/daemon/coverage/lcov.info exists after the test step.

Adding Tests

Daemon tests

New unit tests should be co-located with their source files (e.g., src/core/foo.test.ts). New integration tests should go in packages/daemon/tests/integration/.

VS Code extension tests

New webview handler tests go in packages/vscode/src/test/webviews/. New test helpers go in packages/vscode/src/test/helpers/.