diff --git a/.changeset/verify-env-preflight.md b/.changeset/verify-env-preflight.md new file mode 100644 index 00000000..e44b7029 --- /dev/null +++ b/.changeset/verify-env-preflight.md @@ -0,0 +1,5 @@ +--- +"@dawn-ai/cli": patch +--- + +`dawn verify` now runs an environment preflight. A new `runtime` check asserts the running Node version meets Dawn's `22.13.0` floor (a stale Node fails verify) and, when `dawn.config.ts` configures a sandbox provider, runs the provider's Docker daemon preflight. The `deps` env-var check is now provider-aware: it derives the required API-key env var from the providers your routes actually use (e.g. `ANTHROPIC_API_KEY` for an Anthropic-only app) instead of always nagging about `OPENAI_API_KEY`. diff --git a/apps/web/content/docs/cli.mdx b/apps/web/content/docs/cli.mdx index b2f208fa..0662a543 100644 --- a/apps/web/content/docs/cli.mdx +++ b/apps/web/content/docs/cli.mdx @@ -42,7 +42,7 @@ When a route, tool, or config module fails to load with the opaque ESM error `do ## `dawn verify` -Runs four checks in one call (`app`, `routes`, `typegen`, `deps`) — the canonical pre-deploy integrity gate. +Runs five checks in one call (`app`, `routes`, `typegen`, `deps`, `runtime`) — the canonical preflight before `dawn dev`, `dawn start`, or a deploy. A green `dawn verify` means "this app will boot in this environment." ``` dawn verify @@ -54,7 +54,13 @@ Flags: - `--json` — emit a structured report (`{ status, appRoot, checks, counts }`) instead of human-readable text. - `--env-file ` — path to a `.env` file (overrides `dawn.config.ts` `env` and the default `./.env`). -The `deps` check covers missing packages and missing env vars (advisory) — uniquely useful before a deploy. See [Deployment](/docs/deployment) for the recommended workflow. +The `deps` check covers missing packages and missing env vars (advisory). It is **provider-aware**: it derives the API-key env var from the providers your routes actually use — an Anthropic-only app is checked for `ANTHROPIC_API_KEY`, an OpenAI app for `OPENAI_API_KEY`, a multi-provider app for the union, and a local Ollama app for none. A missing key is a warning, not a failure (the key may come from the runtime environment). + +The `runtime` check gates **environment readiness**: +- **Node** — asserts the running Node version is at least `22.13.0` (Dawn's floor; below it `node:sqlite` breaks). Below the floor **fails** `verify` with a non-zero exit. +- **Docker** — present only when `dawn.config.ts` configures a sandbox provider; it runs the provider's daemon preflight and **fails** if the daemon is unreachable. Apps with no sandbox skip this sub-check entirely. + +See [Deployment](/docs/deployment) for the recommended workflow. ## `dawn routes` diff --git a/apps/web/content/docs/configuration.mdx b/apps/web/content/docs/configuration.mdx index 1107ab55..9ef73aee 100644 --- a/apps/web/content/docs/configuration.mdx +++ b/apps/web/content/docs/configuration.mdx @@ -2,6 +2,8 @@ `dawn.config.ts` is a TypeScript file at your app root that exports a single default object conforming to the `DawnConfig` interface. Dawn loads it (via `tsx`) before every CLI command and at runtime startup. If the file is absent, all keys take their documented defaults — no file is required for a working app. +Run [`dawn verify`](/docs/cli) as your preflight before `dawn dev` or `dawn start`: it validates this config and app integrity plus environment readiness (Node version, the provider API key your routes need, and — when `sandbox` is configured — the Docker daemon). + ```ts // dawn.config.ts export default { diff --git a/apps/web/content/docs/getting-started.mdx b/apps/web/content/docs/getting-started.mdx index d0e03b10..4ece70bb 100644 --- a/apps/web/content/docs/getting-started.mdx +++ b/apps/web/content/docs/getting-started.mdx @@ -138,6 +138,8 @@ npm run eval Runs the quality eval in `src/app/research/evals/research-quality.eval.ts`. The agent run replays fixtures, then two dataset cases are scored across four scorers (`toolCalled`, `contains`, `cites-source`, `llmJudge`). The generated fixtures also cover the judge requests from `llmJudge`, so the scaffolded eval runs without an API key. +Before running the app live, run `dawn verify` as your preflight — it checks app integrity plus environment readiness (Node version, the provider API key your routes need, and the Docker daemon when a sandbox is configured) so `dawn dev` and `dawn start` boot cleanly. See [`dawn verify`](/docs/cli). + ## 4. Run it live Set your API key and start the dev server: diff --git a/docs/superpowers/plans/2026-07-12-verify-env-preflight.md b/docs/superpowers/plans/2026-07-12-verify-env-preflight.md new file mode 100644 index 00000000..59f3f81d --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-verify-env-preflight.md @@ -0,0 +1,99 @@ +# `dawn verify` environment preflight — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax. Follow TDD. + +**Goal:** Extend `dawn verify` (no new command) with a `runtime` check (Node ≥ 22.13.0 + Docker daemon when a sandbox is configured) and fix the hardcoded `OPENAI_API_KEY` deps check to be provider-derived. + +**Architecture:** New checks slot into `verify.ts`'s existing `verifyApp → checks[] → counts` pipeline, sharing its `--json`/exit semantics. + +**Spec:** `docs/superpowers/specs/2026-07-12-verify-env-preflight-design.md` + +**Conventions:** `src`→`.js` / `test`→`.ts`; `exactOptionalPropertyTypes` → conditional-spread; `pnpm --filter @dawn-ai/cli lint`; changeset **patch**. + +--- + +## Task 1: `runtime` check — Node version + Docker daemon + +**Files:** +- Create: `packages/cli/src/lib/verify/check-runtime.ts` +- Test: `packages/cli/test/check-runtime.test.ts` + +- [ ] **Step 1: Failing test** — `checkRuntime({ nodeVersion, sandboxProvider })`: + - `nodeVersion: "22.12.5"` → `node.ok === false`, `status: "failed"`, `floor: "22.13.0"`. + - `nodeVersion: "22.14.0"` → `node.ok === true`. + - `sandboxProvider` with a stub `preflight()` → `{ ok: false, detail: "..." }` → `docker.ok === false`, `status: "failed"`; `{ ok: true }` → `docker.ok === true`. + - No `sandboxProvider` → `docker` field absent, status driven by Node only. +- [ ] **Step 2: Run → fail.** +- [ ] **Step 3: Implement** `check-runtime.ts`: + +```ts +import type { SandboxProvider } from "@dawn-ai/workspace" +const NODE_FLOOR = "22.13.0" +export interface RuntimeCheckResult { + readonly name: "runtime" + readonly node: { readonly version: string; readonly ok: boolean; readonly floor: string } + readonly docker?: { readonly ok: boolean; readonly detail: string } + readonly status: "passed" | "warning" | "failed" +} +function gte(a: string, b: string): boolean { /* pure numeric semver compare of MAJOR.MINOR.PATCH */ } +export async function checkRuntime(input: { + readonly nodeVersion?: string + readonly sandboxProvider?: Pick +}): Promise { + const version = input.nodeVersion ?? process.versions.node + const nodeOk = gte(version, NODE_FLOOR) + const node = { version, ok: nodeOk, floor: NODE_FLOOR } + let docker: RuntimeCheckResult["docker"] + if (input.sandboxProvider?.preflight) { + const r = await input.sandboxProvider.preflight() + docker = { ok: r.ok, detail: r.detail ?? (r.ok ? "reachable" : "unreachable") } + } + const failed = !nodeOk || docker?.ok === false + return { name: "runtime", node, ...(docker ? { docker } : {}), status: failed ? "failed" : "passed" } +} +``` + VERIFY the `SandboxProvider.preflight` return shape (`{ ok, detail?, warnings? }`) in `packages/workspace/src/sandbox-types.ts`. Write `gte` as a pure MAJOR.MINOR.PATCH compare (no deps); test it on `22.9.0`/`22.13.0`/`22.13.1`/`23.0.0`. +- [ ] **Step 4: Run → pass**; `pnpm --filter @dawn-ai/cli typecheck && lint`. +- [ ] **Step 5: Commit** `feat(cli): checkRuntime — Node floor + Docker daemon probe for verify`. + +--- + +## Task 2: Provider-derived API-key check (fix the hardcoded OPENAI_API_KEY) + +**Files:** +- Modify: `packages/cli/src/lib/verify/check-dependencies.ts` +- Possibly modify: `packages/langchain/src/chat-model-factory.ts` (export `providerSpecs`/a `providerEnvVar` map) — or add the map in cli +- Test: `packages/cli/test/check-dependencies.test.ts` (extend) + +- [ ] **Step 1: Failing test** — `checkDependencies` on a fixture whose route model is an Anthropic id → `missingEnvVars` includes `ANTHROPIC_API_KEY` (when unset) and does NOT include `OPENAI_API_KEY`; an OpenAI app → `OPENAI_API_KEY`; a multi-provider app → the union; an ollama-only app → none. Env-file resolution still honored (set the key in a temp `.env` → not missing). +- [ ] **Step 2: Run → fail** (current code hardcodes `["OPENAI_API_KEY"]`). +- [ ] **Step 3: Implement:** + - Define `PROVIDER_ENV_VAR: Record` (`openai→OPENAI_API_KEY`, `anthropic→ANTHROPIC_API_KEY`, `google→GOOGLE_API_KEY`, `mistral→MISTRAL_API_KEY`, `groq→GROQ_API_KEY`, `xai→XAI_API_KEY`, `openrouter→OPENROUTER_API_KEY`; omit `ollama`). Source the provider list from `chat-model-factory.ts:providerSpecs` — export it if not already, else mirror with a comment pointing at the source of truth. + - Replace `RECOMMENDED_ENV_VARS` with: infer each route's provider from its `model` id (reuse the SDK's `inferProvider` — grep for it; `dawn check`/the model-id validation already infer provider), union the providers, map to env vars, dedupe. Requires the route manifest — `checkDependencies` already receives app context; thread the manifest/providers in (mirror how `verify.ts` passes data to the deps check). + - Keep it a `warning` (not failed) when a key is missing (matches current behavior); keep the env-file resolution. +- [ ] **Step 4: Run → pass**; existing `check-dependencies`/`verify` tests updated for the new behavior. +- [ ] **Step 5: Commit** `fix(cli): verify checks the API key the app actually needs (provider-derived, not hardcoded OPENAI_API_KEY)`. + +--- + +## Task 3: Wire `runtime` into `verifyApp` + output + +**Files:** +- Modify: `packages/cli/src/commands/verify.ts` +- Test: `packages/cli/test/verify-command.test.ts` (extend) + +- [ ] **Step 1: Failing test** — `dawn verify --json` on a fixture includes a `checks[]` entry with `name: "runtime"`; a mocked stale `process.versions.node` makes verify exit non-zero; the human summary (non-json) prints a `Runtime: Node ` line. +- [ ] **Step 2: Run → fail.** +- [ ] **Step 3: Implement** — in `verifyApp`, call `checkRuntime({ sandboxProvider: })`, push its result into `checks[]`, and fold its status into `counts`/overall result (a `failed` runtime check fails verify). Add `RuntimeCheckResult` to the `VerifyCheckResult` union. In `runVerifyCommand`'s human path, print a runtime summary line. Resolve the configured sandbox provider the same way `collect-sandbox-errors.ts` does (reuse, don't duplicate). +- [ ] **Step 4: Run → pass**; full `@dawn-ai/cli` suite green (update the verify tests' expected check counts). +- [ ] **Step 5: Commit** `feat(cli): dawn verify runs the runtime/env preflight`. + +--- + +## Task 4: Docs + changeset + PR + +- [ ] **Step 1:** Update `apps/web/content/docs/cli.mdx` `verify` section (now checks env/runtime: Node floor, Docker, provider-derived keys); a one-liner in `getting-started.mdx`/`configuration.mdx` that `dawn verify` is the preflight before `dawn dev`/`dawn start`. No banned phrases; gpt-5 ids only. `node scripts/check-docs.mjs` → PASS. +- [ ] **Step 2:** `.changeset/verify-env-preflight.md` — **patch** for `@dawn-ai/cli` (+ `@dawn-ai/langchain` if `providerSpecs` was exported from it). Confirm the set via `git log … --name-only`. +- [ ] **Step 3:** Full local verify (`pnpm build && typecheck && lint && test && check-docs`); rebase, push, PR, watch `validate` + review. + +**Notes:** Branch e.g. `feat/verify-env-preflight`; pin before subagent dispatch. Build after the error-code registry so runtime failures can carry `DAWN_E` codes (add them if the registry has landed; otherwise leave a follow-up note). diff --git a/docs/superpowers/specs/2026-07-12-verify-env-preflight-design.md b/docs/superpowers/specs/2026-07-12-verify-env-preflight-design.md new file mode 100644 index 00000000..44eb3076 --- /dev/null +++ b/docs/superpowers/specs/2026-07-12-verify-env-preflight-design.md @@ -0,0 +1,78 @@ +# `dawn verify` environment preflight — design + +**Date:** 2026-07-12 +**Status:** approved (brainstorm) +**Topic:** Extend the existing `dawn verify` command with environment/runtime preflight checks (Node version, Docker daemon, provider-derived API keys) so "why won't it start" is answered in one place — **without adding a new CLI command**. + +## Problem + +`dawn check` validates config/descriptors; `dawn verify` validates *app integrity* (app root, routes, typegen, deps). Neither verifies the **environment** the app will actually run in: +- No **Node-version** assertion exists anywhere at runtime, yet the real floor is 22.13.0 (below it `node:sqlite` — used by `@dawn-ai/sqlite-storage`/`@dawn-ai/memory` — needs an experimental flag and breaks). +- **Docker daemon** reachability is only probed by `dawn check`, and only when a sandbox provider is configured — not part of `verify`. +- `dawn verify`'s deps check *does* flag missing env vars, but via a **hardcoded** `RECOMMENDED_ENV_VARS = ["OPENAI_API_KEY"]` (`check-dependencies.ts`) — it always nags about `OPENAI_API_KEY` regardless of which provider the app uses, and never checks the key the app *does* need (e.g. `ANTHROPIC_API_KEY`). + +**Decision (locked):** do NOT add a `dawn doctor` command — keep the CLI surface minimal. Fold these checks into `dawn verify`. + +## Goal + +`dawn verify` gains environment readiness as part of its existing `checks[]` model: a new **`runtime`** check (Node + Docker) and a fixed, provider-aware **`deps` env-var** check. Same `--json` output, same counts, same exit semantics. A green `dawn verify` should mean "this app will boot in this environment." + +## Non-goals + +- No new command; no interactive UI; no separate colored output system (verify has its own line-based output + `--json`). +- Not re-validating config/descriptors (that's `dawn check`, which `verify` already shares its advisory model-id pass with). +- Not installing anything or mutating the environment. + +## Architecture + +Everything slots into `packages/cli/src/commands/verify.ts`'s existing check pipeline (`verifyApp` → `checks[]` → `counts` → success/failure result). New checks follow the established result-object shape (`{ name, status: "passed"|"warning"|"failed", … }`). + +### 1. New `runtime` check — Node + Docker + +New module `packages/cli/src/lib/verify/check-runtime.ts` returning: +```ts +interface RuntimeCheck { + readonly name: "runtime" + readonly node: { readonly version: string; readonly ok: boolean; readonly floor: "22.13.0" } + readonly docker?: { readonly ok: boolean; readonly detail: string } // present only if a sandbox provider is configured + readonly status: "passed" | "warning" | "failed" +} +``` +- **Node:** read `process.versions.node`, semver-compare to floor `22.13.0`. Below floor → `failed` (hard: `node:sqlite` genuinely breaks). This is new logic (no runtime version assert exists today). +- **Docker:** if `dawn.config.ts` configures a sandbox provider, call the provider's `preflight()` (the same `{ ok, detail, warnings? }` contract `dawn check` uses via `collect-sandbox-errors.ts`). Unreachable daemon → `failed` (the app can't run sandboxed). If no sandbox is configured, omit the docker sub-check entirely (don't nag about Docker an app doesn't use). Reuse the provider preflight; do not reimplement the `docker version` probe. + +Distinction from `dawn check`: check runs the sandbox preflight only as config validation; verify frames Docker as an **environment-readiness** gate and reports it in the integrity result. + +### 2. Fix the `deps` env-var check to be provider-aware + +`packages/cli/src/lib/verify/check-dependencies.ts` currently hardcodes `RECOMMENDED_ENV_VARS = ["OPENAI_API_KEY"]`. Replace with a **provider → key-env-var table** and derive the required set from the providers the app's routes actually use: +- Add a `providerEnvVar(provider)` map (`openai→OPENAI_API_KEY`, `anthropic→ANTHROPIC_API_KEY`, `google→GOOGLE_API_KEY`/`GEMINI_API_KEY`, `mistral→MISTRAL_API_KEY`, `groq→GROQ_API_KEY`, `xai→XAI_API_KEY`, `openrouter→OPENROUTER_API_KEY`; `ollama` → none). Base it on `providerSpecs` in `chat-model-factory.ts:12-22` (single source; export or mirror it). +- Determine which providers the app uses by inferring the provider from each route's `model` id (the same `inferProvider`/discovery `dawn check` already does — `verify` already builds the route manifest). Union → required key env vars. +- Missing a required key → same `warning` the deps check already emits (keep it a warning, not a hard fail — a key may legitimately come from the runtime environment; matches current behavior). Keep the existing env-file resolution (`--env-file` > config env > `./.env`). +- Result: verify stops nagging about `OPENAI_API_KEY` for an Anthropic-only app, and correctly flags the key the app actually needs. + +### 3. Wire into `verifyApp` + +Add the `runtime` check to the `checks[]` produced by `verifyApp`, and update `VerifyCheckResult` union + the human-readable summary in `runVerifyCommand` (a `Runtime: Node OK` / `Docker: reachable` line; warnings for missing keys as today). `--json` gains the new check objects automatically (they're in `checks[]`). Exit: a `failed` runtime check fails verify (non-zero), consistent with existing failed checks. + +## Interaction with the error-code registry + +Once the error-code registry (separate spec) lands, the runtime check's failure messages adopt codes (e.g. `DAWN_E5101 Node too old`, `DAWN_E2002 Docker unreachable`). This spec does not depend on the registry — if built first, add codes as a fast-follow; if the registry is built first, use codes from the start. + +## Error handling / edge cases + +- Node floor check must not itself require Node ≥22.13 to run (it runs on whatever Node invoked the CLI) — pure `process.versions.node` string compare, no `node:sqlite` import in the check. +- Docker preflight already handles "daemon not running" → `{ ok:false, detail }`; surface `detail` verbatim. +- An app with no routes / no models → no required keys (skip the provider-key derivation gracefully). +- `--json` consumers: new fields are additive; existing `checks[]` entries unchanged. + +## Testing + +- Unit (`check-runtime`): Node below/at/above floor → failed/passed; docker sub-check present only when a sandbox provider is configured (inject a fake provider with a stub `preflight()` returning ok/not-ok). +- Unit (`check-dependencies`): an Anthropic-only fixture app → requires `ANTHROPIC_API_KEY`, does NOT flag `OPENAI_API_KEY`; an OpenAI app → requires `OPENAI_API_KEY`; multi-provider → union; ollama-only → no key required. Env-file resolution still honored. +- Integration (`verify-command` test): `dawn verify --json` on a fixture includes the `runtime` check; a stale Node (mock `process.versions.node`) fails verify; the human summary prints the runtime line. +- Full `@dawn-ai/cli` suite stays green (additive checks; existing verify tests updated for the new check count). + +## Rollout + +One PR. Changeset: **patch** for `@dawn-ai/cli` (and `@dawn-ai/langchain` if `providerSpecs`/`providerEnvVar` is exported from there). Docs: update `cli.mdx`'s `verify` section + `configuration.mdx`/`getting-started` to mention `dawn verify` as the preflight. Build after the error-code registry (to emit codes) but independently shippable. diff --git a/packages/cli/src/commands/verify.ts b/packages/cli/src/commands/verify.ts index 9ab76960..311982b5 100644 --- a/packages/cli/src/commands/verify.ts +++ b/packages/cli/src/commands/verify.ts @@ -5,13 +5,17 @@ import { discoverRoutes, extractToolTypesForRoute, findDawnApp, + loadDawnConfig, renderDawnTypes, } from "@dawn-ai/core" +import type { SandboxProvider } from "@dawn-ai/workspace" import { type Command, CommanderError } from "commander" import { CliError, type CommandIo, formatErrorMessage, writeLine } from "../lib/output.js" +import { collectRouteProviders } from "../lib/runtime/collect-route-providers.js" import { collectUnknownModelIdWarnings } from "../lib/runtime/warn-unknown-model-ids.js" import { checkDependencies } from "../lib/verify/check-dependencies.js" +import { checkRuntime, type RuntimeCheckResult } from "../lib/verify/check-runtime.js" interface VerifyOptions { readonly cwd?: string @@ -62,6 +66,7 @@ interface VerifyFailedCheckResult { } type VerifyCheckResult = + | RuntimeCheckResult | VerifyAppCheckResult | VerifyDepsCheckResult | VerifyFailedCheckResult @@ -148,6 +153,19 @@ export async function runVerifyCommand(options: VerifyOptions, io: CommandIo): P ) } + const runtimeCheck = result.checks.find( + (check): check is RuntimeCheckResult => check.name === "runtime", + ) + if (runtimeCheck) { + writeLine( + io.stdout, + `Runtime: Node ${runtimeCheck.node.version} OK (floor ${runtimeCheck.node.floor}).`, + ) + if (runtimeCheck.docker) { + writeLine(io.stdout, `Sandbox: Docker ${runtimeCheck.docker.detail}.`) + } + } + if (manifest) { // Advisory model-id pass shared with `dawn check`; never affects the result. const modelIdWarnings = await collectUnknownModelIdWarnings(manifest) @@ -226,9 +244,14 @@ async function verifyApp(options: VerifyOptions): Promise { status: PASSED_STATUS, }) + // Derive the providers the app's routes actually use, so the deps check flags + // the API key the app needs (e.g. ANTHROPIC_API_KEY) rather than a hardcoded one. + const providers = await collectRouteProviders(manifest) + // Check dependencies and environment variables (advisory, not blocking) const depsResult = await checkDependencies({ appRoot: app.appRoot, + providers, ...(options.envFile !== undefined ? { envFile: options.envFile } : {}), }) const hasWarnings = depsResult.missingPackages.length > 0 || depsResult.missingEnvVars.length > 0 @@ -239,18 +262,42 @@ async function verifyApp(options: VerifyOptions): Promise { status: hasWarnings ? "warning" : PASSED_STATUS, } as VerifyDepsCheckResult) + // Environment-readiness gate: Node floor + (when a sandbox is configured) the + // provider's Docker/daemon preflight — resolved the same way collect-sandbox-errors + // does, from dawn.config.ts's sandbox.provider. A failed runtime check fails verify. + const sandboxProvider = await resolveSandboxProvider(app.appRoot) + const runtime = await checkRuntime(sandboxProvider ? { sandboxProvider } : {}) + checks.push(runtime) + + const failed = checks.filter((check) => check.status === FAILED_STATUS).length + const counts: VerifyCheckCounts = { + failed, + passed: checks.length - failed, + total: checks.length, + } + + if (failed > 0) { + return { + manifest, + result: { appRoot: app.appRoot, checks, counts, status: FAILED_STATUS }, + } + } + return { manifest, - result: { - appRoot: app.appRoot, - checks, - counts: { - failed: 0, - passed: checks.length, - total: checks.length, - }, - status: PASSED_STATUS, - }, + result: { appRoot: app.appRoot, checks, counts, status: PASSED_STATUS }, + } +} + +/** Resolve dawn.config.ts's sandbox provider (name + preflight), if configured. */ +async function resolveSandboxProvider( + appRoot: string, +): Promise | undefined> { + try { + const loaded = await loadDawnConfig({ appRoot }) + return loaded.config.sandbox?.provider + } catch { + return undefined } } @@ -288,7 +335,27 @@ function createVerifyFailureResult( function getFailureMessage(result: VerifyFailureResult): string { const failedCheck = [...result.checks].reverse().find((check) => check.status === FAILED_STATUS) - return failedCheck?.error.message ?? "Verification failed." + if (failedCheck?.name === "runtime") { + return runtimeFailureMessage(failedCheck) + } + // Only VerifyFailedCheckResult carries an `error` field. + if (failedCheck && "error" in failedCheck) { + return failedCheck.error.message + } + return "Verification failed." +} + +function runtimeFailureMessage(runtime: RuntimeCheckResult): string { + const reasons: string[] = [] + if (!runtime.node.ok) { + // TODO(error-codes): tag with DAWN_E5101 once the registry lands (#357). + reasons.push(`Node ${runtime.node.version} is below the required floor ${runtime.node.floor}.`) + } + if (runtime.docker && !runtime.docker.ok) { + // TODO(error-codes): tag with DAWN_E2002 once the registry lands (#357). + reasons.push(`Docker sandbox unavailable: ${runtime.docker.detail}.`) + } + return reasons.join(" ") || "Runtime check failed." } function inferFailureAppRoot(options: VerifyOptions, message: string): string { diff --git a/packages/cli/src/lib/runtime/collect-route-providers.ts b/packages/cli/src/lib/runtime/collect-route-providers.ts new file mode 100644 index 00000000..d4424f7b --- /dev/null +++ b/packages/cli/src/lib/runtime/collect-route-providers.ts @@ -0,0 +1,28 @@ +import type { RouteManifest } from "@dawn-ai/core" +import { inferProvider, isDawnAgent } from "@dawn-ai/sdk" + +import { type NormalizedRouteModule, normalizeRouteModule } from "./load-route-kind.js" + +/** + * The deduped set of model providers the app's agent routes actually use. + * Each route's provider is its explicit `provider`, else inferred from its + * `model` id (the same `inferProvider` the model-id validation uses). Feeds + * verify's provider-derived API-key check. Load failures are skipped — they are + * surfaced by the discovery/typegen checks, not this advisory derivation. + */ +export async function collectRouteProviders(manifest: RouteManifest): Promise { + const providers = new Set() + for (const route of manifest.routes) { + if (route.kind !== "agent") continue + let normalized: NormalizedRouteModule + try { + normalized = await normalizeRouteModule(route.entryFile, manifest.appRoot) + } catch { + continue + } + if (!isDawnAgent(normalized.entry)) continue + const provider = normalized.entry.provider ?? inferProvider(normalized.entry.model) + if (provider) providers.add(provider) + } + return [...providers] +} diff --git a/packages/cli/src/lib/verify/check-dependencies.ts b/packages/cli/src/lib/verify/check-dependencies.ts index 14e94833..93698eb2 100644 --- a/packages/cli/src/lib/verify/check-dependencies.ts +++ b/packages/cli/src/lib/verify/check-dependencies.ts @@ -1,6 +1,7 @@ import { existsSync, readFileSync } from "node:fs" import { join } from "node:path" import { loadDawnConfig } from "@dawn-ai/core" +import type { BuiltInModelProviderId } from "@dawn-ai/sdk" import { resolveEnvPath } from "../dev/resolve-env-path.js" export interface DependencyCheckResult { @@ -10,6 +11,13 @@ export interface DependencyCheckResult { export interface CheckDependenciesOptions { readonly appRoot: string + /** + * Provider ids the app's routes actually use (derived from each route's model + * id). The required API-key env vars are derived from these — an Anthropic-only + * app checks for ANTHROPIC_API_KEY, not OPENAI_API_KEY. An empty/omitted list + * means no API key is required. + */ + readonly providers?: readonly string[] /** From the --env-file CLI flag. Highest precedence. */ readonly envFile?: string | undefined } @@ -21,10 +29,31 @@ export interface CheckDependenciesOptions { const REQUIRED_PACKAGES = ["@langchain/core", "@langchain/openai", "@langchain/langgraph"] as const /** - * Environment variables that are strongly recommended for production use. - * Missing vars emit warnings, not hard failures. + * Provider → the API-key env var it authenticates with. `null` means the + * provider needs no key (e.g. a local Ollama server). Keyed exhaustively by the + * SDK's provider union so it stays in lockstep with the provider list backing + * `providerSpecs` in @dawn-ai/langchain's chat-model-factory.ts (source of truth). */ -const RECOMMENDED_ENV_VARS = ["OPENAI_API_KEY"] as const +const PROVIDER_ENV_VAR: Record = { + openai: "OPENAI_API_KEY", + anthropic: "ANTHROPIC_API_KEY", + google: "GOOGLE_API_KEY", + mistral: "MISTRAL_API_KEY", + groq: "GROQ_API_KEY", + xai: "XAI_API_KEY", + openrouter: "OPENROUTER_API_KEY", + ollama: null, +} + +/** Derive the deduped set of required API-key env vars from the app's providers. */ +function requiredEnvVars(providers: readonly string[]): readonly string[] { + const vars = new Set() + for (const provider of providers) { + const envVar = PROVIDER_ENV_VAR[provider as BuiltInModelProviderId] + if (envVar) vars.add(envVar) + } + return [...vars] +} export async function checkDependencies( options: CheckDependenciesOptions, @@ -72,8 +101,10 @@ export async function checkDependencies( const resolved = resolveEnvPath({ appRoot, flag: options.envFile, configEnv }) - // Check environment variables (from process.env or the resolved env file) - for (const envVar of RECOMMENDED_ENV_VARS) { + // Check the API-key env vars the app's providers actually need (from + // process.env or the resolved env file). A missing key is a warning, not a + // hard failure — a key may legitimately come from the runtime environment. + for (const envVar of requiredEnvVars(options.providers ?? [])) { if (!process.env[envVar]) { // Check if it's in the resolved env file if (existsSync(resolved.absPath)) { diff --git a/packages/cli/src/lib/verify/check-runtime.ts b/packages/cli/src/lib/verify/check-runtime.ts new file mode 100644 index 00000000..dd7049d8 --- /dev/null +++ b/packages/cli/src/lib/verify/check-runtime.ts @@ -0,0 +1,64 @@ +import type { SandboxProvider } from "@dawn-ai/workspace" + +/** + * Minimum Node version Dawn requires at runtime. Below this, `node:sqlite` + * (used by @dawn-ai/sqlite-storage / @dawn-ai/memory) needs an experimental + * flag and breaks. This check must not itself require Node ≥ floor to run — + * it is a pure string compare with no `node:sqlite` import. + */ +const NODE_FLOOR = "22.13.0" + +export interface RuntimeCheckResult { + readonly name: "runtime" + readonly node: { readonly version: string; readonly ok: boolean; readonly floor: string } + /** Present only when a sandbox provider is configured. */ + readonly docker?: { readonly ok: boolean; readonly detail: string } + readonly status: "passed" | "warning" | "failed" +} + +function parseVersion(version: string): readonly [number, number, number] { + const parts = version.split(".") + const major = Number.parseInt(parts[0] ?? "", 10) + const minor = Number.parseInt(parts[1] ?? "", 10) + const patch = Number.parseInt(parts[2] ?? "", 10) + return [ + Number.isNaN(major) ? 0 : major, + Number.isNaN(minor) ? 0 : minor, + Number.isNaN(patch) ? 0 : patch, + ] +} + +/** Pure numeric MAJOR.MINOR.PATCH comparison: is `a` ≥ `b`? No deps. */ +export function gte(a: string, b: string): boolean { + const [aMajor, aMinor, aPatch] = parseVersion(a) + const [bMajor, bMinor, bPatch] = parseVersion(b) + if (aMajor !== bMajor) return aMajor > bMajor + if (aMinor !== bMinor) return aMinor > bMinor + return aPatch >= bPatch +} + +export async function checkRuntime(input: { + readonly nodeVersion?: string + readonly sandboxProvider?: Pick +}): Promise { + const version = input.nodeVersion ?? process.versions.node + const nodeOk = gte(version, NODE_FLOOR) + const node = { version, ok: nodeOk, floor: NODE_FLOOR } + + let docker: RuntimeCheckResult["docker"] + if (input.sandboxProvider?.preflight) { + // Reuse the provider preflight contract ({ ok, detail?, warnings? }) — the + // same one `dawn check` runs via collect-sandbox-errors. Surface detail verbatim. + const result = await input.sandboxProvider.preflight() + docker = { ok: result.ok, detail: result.detail ?? (result.ok ? "reachable" : "unreachable") } + } + + // TODO(error-codes): tag with DAWN_E5101 (Node too old) / DAWN_E2002 (Docker unreachable) once the registry lands (#357). + const failed = !nodeOk || docker?.ok === false + return { + name: "runtime", + node, + ...(docker ? { docker } : {}), + status: failed ? "failed" : "passed", + } +} diff --git a/packages/cli/test/check-dependencies.test.ts b/packages/cli/test/check-dependencies.test.ts index bedd194a..77fd1b45 100644 --- a/packages/cli/test/check-dependencies.test.ts +++ b/packages/cli/test/check-dependencies.test.ts @@ -79,15 +79,60 @@ describe("checkDependencies", () => { expect(result.missingPackages).toEqual([]) }) - test("reports missing env var when not in process.env or .env file", async () => { + test("reports the OpenAI key for an OpenAI app when not in process.env or .env file", async () => { saveEnv("OPENAI_API_KEY") delete process.env.OPENAI_API_KEY writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) - const result = await checkDependencies({ appRoot: tempDir }) + const result = await checkDependencies({ appRoot: tempDir, providers: ["openai"] }) + + expect(result.missingEnvVars).toContain("OPENAI_API_KEY") + }) + + test("reports the Anthropic key (not OPENAI_API_KEY) for an Anthropic-only app", async () => { + saveEnv("ANTHROPIC_API_KEY", "OPENAI_API_KEY") + delete process.env.ANTHROPIC_API_KEY + delete process.env.OPENAI_API_KEY + + writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) + + const result = await checkDependencies({ appRoot: tempDir, providers: ["anthropic"] }) + + expect(result.missingEnvVars).toContain("ANTHROPIC_API_KEY") + expect(result.missingEnvVars).not.toContain("OPENAI_API_KEY") + }) + + test("reports the union of keys for a multi-provider app", async () => { + saveEnv("ANTHROPIC_API_KEY", "OPENAI_API_KEY") + delete process.env.ANTHROPIC_API_KEY + delete process.env.OPENAI_API_KEY + + writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) + + const result = await checkDependencies({ + appRoot: tempDir, + providers: ["openai", "anthropic"], + }) expect(result.missingEnvVars).toContain("OPENAI_API_KEY") + expect(result.missingEnvVars).toContain("ANTHROPIC_API_KEY") + }) + + test("requires no key for an ollama-only app", async () => { + writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) + + const result = await checkDependencies({ appRoot: tempDir, providers: ["ollama"] }) + + expect(result.missingEnvVars).toEqual([]) + }) + + test("requires no key when the app uses no providers", async () => { + writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) + + const result = await checkDependencies({ appRoot: tempDir, providers: [] }) + + expect(result.missingEnvVars).toEqual([]) }) test("passes env check when var is in process.env", async () => { @@ -96,7 +141,7 @@ describe("checkDependencies", () => { writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) - const result = await checkDependencies({ appRoot: tempDir }) + const result = await checkDependencies({ appRoot: tempDir, providers: ["openai"] }) expect(result.missingEnvVars).not.toContain("OPENAI_API_KEY") }) @@ -108,13 +153,13 @@ describe("checkDependencies", () => { writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) writeFileSync(join(tempDir, ".env"), "OPENAI_API_KEY=sk-test-key\n") - const result = await checkDependencies({ appRoot: tempDir }) + const result = await checkDependencies({ appRoot: tempDir, providers: ["openai"] }) expect(result.missingEnvVars).not.toContain("OPENAI_API_KEY") }) test("returns empty results when package.json is missing", async () => { - const result = await checkDependencies({ appRoot: tempDir }) + const result = await checkDependencies({ appRoot: tempDir, providers: ["openai"] }) expect(result.missingPackages).toEqual([]) expect(result.missingEnvVars).toEqual([]) @@ -125,11 +170,15 @@ describe("checkDependencies", () => { delete process.env.OPENAI_API_KEY writeFileSync(join(tempDir, "package.json"), JSON.stringify({ dependencies: {} })) - // Recommended var is absent from /.env but present in custom.env. + // Required var is absent from /.env but present in custom.env. writeFileSync(join(tempDir, ".env"), "SOMETHING_ELSE=1\n") writeFileSync(join(tempDir, "custom.env"), "OPENAI_API_KEY=sk-from-custom\n") - const result = await checkDependencies({ appRoot: tempDir, envFile: "custom.env" }) + const result = await checkDependencies({ + appRoot: tempDir, + providers: ["openai"], + envFile: "custom.env", + }) expect(result.missingEnvVars).not.toContain("OPENAI_API_KEY") }) diff --git a/packages/cli/test/check-runtime.test.ts b/packages/cli/test/check-runtime.test.ts new file mode 100644 index 00000000..c9947f66 --- /dev/null +++ b/packages/cli/test/check-runtime.test.ts @@ -0,0 +1,78 @@ +import { describe, expect, test } from "vitest" + +import { checkRuntime, gte } from "../src/lib/verify/check-runtime.js" + +describe("gte", () => { + test("pure MAJOR.MINOR.PATCH numeric compare", () => { + // 22.9.0 < 22.13.0 < 22.13.1 < 23.0.0 + expect(gte("22.9.0", "22.13.0")).toBe(false) + expect(gte("22.13.0", "22.13.0")).toBe(true) + expect(gte("22.13.1", "22.13.0")).toBe(true) + expect(gte("23.0.0", "22.13.0")).toBe(true) + }) + + test("compares numerically, not lexically", () => { + // "9" > "13" lexically but 9 < 13 numerically + expect(gte("22.9.0", "22.13.0")).toBe(false) + expect(gte("22.100.0", "22.13.0")).toBe(true) + }) +}) + +describe("checkRuntime", () => { + test("Node below floor → failed", async () => { + const result = await checkRuntime({ nodeVersion: "22.12.5" }) + expect(result.name).toBe("runtime") + expect(result.node.ok).toBe(false) + expect(result.node.floor).toBe("22.13.0") + expect(result.node.version).toBe("22.12.5") + expect(result.status).toBe("failed") + }) + + test("Node at/above floor → passed", async () => { + const result = await checkRuntime({ nodeVersion: "22.14.0" }) + expect(result.node.ok).toBe(true) + expect(result.status).toBe("passed") + }) + + test("no sandbox provider → docker sub-check absent", async () => { + const result = await checkRuntime({ nodeVersion: "22.14.0" }) + expect(result.docker).toBeUndefined() + }) + + test("sandbox provider with failing preflight → docker.ok false + failed", async () => { + const result = await checkRuntime({ + nodeVersion: "22.14.0", + sandboxProvider: { + name: "fake", + preflight: async () => ({ ok: false, detail: "daemon unreachable" }), + }, + }) + expect(result.docker).toEqual({ ok: false, detail: "daemon unreachable" }) + expect(result.status).toBe("failed") + }) + + test("sandbox provider with passing preflight → docker.ok true + passed", async () => { + const result = await checkRuntime({ + nodeVersion: "22.14.0", + sandboxProvider: { + name: "fake", + preflight: async () => ({ ok: true }), + }, + }) + expect(result.docker).toEqual({ ok: true, detail: "reachable" }) + expect(result.status).toBe("passed") + }) + + test("stale Node with a healthy sandbox → still failed on Node", async () => { + const result = await checkRuntime({ + nodeVersion: "20.0.0", + sandboxProvider: { + name: "fake", + preflight: async () => ({ ok: true }), + }, + }) + expect(result.node.ok).toBe(false) + expect(result.docker?.ok).toBe(true) + expect(result.status).toBe("failed") + }) +}) diff --git a/packages/cli/test/collect-route-providers.test.ts b/packages/cli/test/collect-route-providers.test.ts new file mode 100644 index 00000000..a078c9b0 --- /dev/null +++ b/packages/cli/test/collect-route-providers.test.ts @@ -0,0 +1,94 @@ +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { dirname, join } from "node:path" +import { discoverRoutes } from "@dawn-ai/core" +import { afterEach, describe, expect, test } from "vitest" + +import { collectRouteProviders } from "../src/lib/runtime/collect-route-providers.js" + +const tempDirs: string[] = [] + +afterEach(async () => { + await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true }))) +}) + +async function createFixtureApp(files: Readonly>) { + const appRoot = await mkdtemp(join(tmpdir(), "dawn-cli-providers-")) + tempDirs.push(appRoot) + + const appFiles = { + "package.json": '{"type":"module"}\n', + "dawn.config.ts": "export default {};\n", + ...files, + } + + await Promise.all( + Object.entries(appFiles).map(async ([relativePath, source]) => { + const filePath = join(appRoot, relativePath) + await mkdir(dirname(filePath), { recursive: true }) + await writeFile(filePath, source) + }), + ) + + return appRoot +} + +function agentRoute(model: string, provider?: string): string { + const providerLine = provider ? `\n provider: "${provider}",` : "" + return `import { agent } from "@dawn-ai/sdk" + +export default agent({ + model: "${model}",${providerLine} + systemPrompt: "You do things.", +}) +` +} + +describe("collectRouteProviders", () => { + test("infers openai from a gpt- model id", async () => { + const appRoot = await createFixtureApp({ + "src/app/draft/index.ts": agentRoute("gpt-5-mini"), + }) + const manifest = await discoverRoutes({ appRoot }) + + expect(await collectRouteProviders(manifest)).toEqual(["openai"]) + }) + + test("infers anthropic from a claude- model id", async () => { + const appRoot = await createFixtureApp({ + "src/app/draft/index.ts": agentRoute("claude-sonnet-4-5"), + }) + const manifest = await discoverRoutes({ appRoot }) + + expect(await collectRouteProviders(manifest)).toEqual(["anthropic"]) + }) + + test("unions providers across routes and dedupes", async () => { + const appRoot = await createFixtureApp({ + "src/app/a/index.ts": agentRoute("gpt-5-mini"), + "src/app/b/index.ts": agentRoute("claude-sonnet-4-5"), + "src/app/c/index.ts": agentRoute("gpt-5.5"), + }) + const manifest = await discoverRoutes({ appRoot }) + + expect([...(await collectRouteProviders(manifest))].sort()).toEqual(["anthropic", "openai"]) + }) + + test("honors an explicit provider over the inferred one", async () => { + const appRoot = await createFixtureApp({ + "src/app/draft/index.ts": agentRoute("some-proxy-model", "ollama"), + }) + const manifest = await discoverRoutes({ appRoot }) + + expect(await collectRouteProviders(manifest)).toEqual(["ollama"]) + }) + + test("skips non-agent routes", async () => { + const appRoot = await createFixtureApp({ + "src/app/wf/index.ts": "export async function workflow() { return {} }\n", + }) + const manifest = await discoverRoutes({ appRoot }) + + expect(await collectRouteProviders(manifest)).toEqual([]) + }) +}) diff --git a/packages/cli/test/verify-command.test.ts b/packages/cli/test/verify-command.test.ts index 0e4831d3..743b95db 100644 --- a/packages/cli/test/verify-command.test.ts +++ b/packages/cli/test/verify-command.test.ts @@ -71,8 +71,26 @@ describe("dawn verify", () => { expect(result.exitCode).toBe(0) expect(result.stderr).toBe("") expect(result.stdout).toContain("Dawn app integrity OK") - expect(result.stdout).toContain("4 checks passed") + expect(result.stdout).toContain("5 checks passed") expect(result.stdout).toContain("2 routes discovered") + expect(result.stdout).toMatch(/Runtime: Node .+ OK/) + }) + + test("fails verify with a nonzero exit when the Node runtime is below the floor", async () => { + const appRoot = await createFixtureApp({ + "src/app/hello/index.ts": "export async function workflow() { return {} }\n", + }) + + const descriptor = Object.getOwnPropertyDescriptor(process.versions, "node") + Object.defineProperty(process.versions, "node", { value: "20.11.0", configurable: true }) + try { + const result = await invoke(["verify", "--cwd", appRoot]) + expect(result.exitCode).toBe(1) + expect(result.stderr).toMatch(/^Verify failed:/) + expect(result.stderr).toContain("22.13.0") + } finally { + if (descriptor) Object.defineProperty(process.versions, "node", descriptor) + } }) test("resolves the Dawn app root from a child directory via --cwd", async () => { @@ -236,11 +254,20 @@ describe("dawn verify", () => { name: "deps", status: expect.stringMatching(/^(passed|warning)$/), }, + { + name: "runtime", + node: { + version: expect.any(String), + ok: true, + floor: "22.13.0", + }, + status: "passed", + }, ], counts: { failed: 0, - passed: 4, - total: 4, + passed: 5, + total: 5, }, status: "passed", }) diff --git a/test/generated/fixtures/basic.expected.json b/test/generated/fixtures/basic.expected.json index a9f6d81f..55480ff5 100644 --- a/test/generated/fixtures/basic.expected.json +++ b/test/generated/fixtures/basic.expected.json @@ -48,16 +48,25 @@ "status": "passed" }, { - "missingEnvVars": ["OPENAI_API_KEY"], + "missingEnvVars": [], "missingPackages": ["@langchain/core", "@langchain/openai", "@langchain/langgraph"], "name": "deps", "status": "warning" + }, + { + "name": "runtime", + "node": { + "version": "", + "ok": true, + "floor": "22.13.0" + }, + "status": "passed" } ], "counts": { "failed": 0, - "passed": 4, - "total": 4 + "passed": 5, + "total": 5 }, "status": "passed" }, diff --git a/test/generated/fixtures/custom-app-dir.expected.json b/test/generated/fixtures/custom-app-dir.expected.json index 227b3d15..23b2878c 100644 --- a/test/generated/fixtures/custom-app-dir.expected.json +++ b/test/generated/fixtures/custom-app-dir.expected.json @@ -48,16 +48,25 @@ "status": "passed" }, { - "missingEnvVars": ["OPENAI_API_KEY"], + "missingEnvVars": [], "missingPackages": ["@langchain/core", "@langchain/openai", "@langchain/langgraph"], "name": "deps", "status": "warning" + }, + { + "name": "runtime", + "node": { + "version": "", + "ok": true, + "floor": "22.13.0" + }, + "status": "passed" } ], "counts": { "failed": 0, - "passed": 4, - "total": 4 + "passed": 5, + "total": 5 }, "status": "passed" }, diff --git a/test/generated/run-generated-app.test.ts b/test/generated/run-generated-app.test.ts index 8cb0ea41..afe5be21 100644 --- a/test/generated/run-generated-app.test.ts +++ b/test/generated/run-generated-app.test.ts @@ -488,6 +488,9 @@ function normalizeForFixture( ["25.6.0", ""], ["6.0.2", ""], ["4.1.4", ""], + // Runtime check emits the live Node version (process.versions.node). Normalize + // it so the fixture stays stable across Node patch releases. + [process.versions.node, ""], ]) as GeneratedAppScenarioResult } @@ -518,6 +521,9 @@ function normalizeForInternalFixture( ["25.6.0", ""], ["6.0.2", ""], ["4.1.4", ""], + // Runtime check emits the live Node version (process.versions.node). Normalize + // it so the fixture stays stable across Node patch releases. + [process.versions.node, ""], ]) as GeneratedAppScenarioResult }