diff --git a/docs/design/drive9-agent-harness-v2-plan.md b/docs/design/drive9-agent-harness-v2-plan.md new file mode 100644 index 000000000..7f51c78fc --- /dev/null +++ b/docs/design/drive9-agent-harness-v2-plan.md @@ -0,0 +1,691 @@ +--- +title: Drive9 reusable agent harness v2 plan +updated: 2026-05-14 +watches: + - AGENTS.local.md + - claude-notes/drive9-ec2-adversarial-usage-test-plan.md + - claude-notes/drive9-ec2-adversarial-usage-test-report.md + - e2e/ + - pkg/client/client.go + - pkg/fuse/ + - pkg/metrics/ + - pkg/server/instrumentation.go +--- + +## Summary + +The EC2 adversarial run produced valuable Drive9 findings, but the harness itself was too ad hoc to reuse safely. The next step is a phase-1, in-repository, typed harness for local smoke and targeted regression workloads. It should make Drive9 agent workloads legible to future agents and maintainers through structured cases, machine-readable results, minimized repro scripts, and strict cleanup boundaries. Stress, fault injection, garbage collection, production log collection, and per-request CLI/FUSE trace propagation are explicit follow-up phases. + +## Input Sources + +1. Existing one-off plan: `claude-notes/drive9-ec2-adversarial-usage-test-plan.md` +2. Triaged test report: `claude-notes/drive9-ec2-adversarial-usage-test-report.md` +3. Harness practice reference: OpenAI, "Harness engineering: leveraging Codex in an agent-first world", https://openai.com/index/harness-engineering/ + +## Report Evaluation + +1. The report contains valuable product findings. +2. Highest-value P1 finding: path edge-case failure for `space name unicode-测试 punct_!@%.txt` in both strict and interactive modes. This likely points at path escaping, normalization, FUSE create/write handling, or server path handling. +3. Highest-value workload finding: `git clone` and local git operations are unreliable on the mount. Completed clones left `.git/index.lock` in strict runs and one interactive run, and follow-up `git status/add/commit/mv/rm` failed. +4. Useful P2 finding: strict `fio` write throughput was extremely slow at about `704984` bytes/s for 1 GiB. This needs a control baseline before being treated as a regression, but it is a strong performance target. +5. Useful UX finding: `drive9 doctor fuse` exits nonzero when `/etc/fuse.conf` lacks `user_allow_other`, even when the test does not request `--allow-other`. +6. Invalid raw finding: the `strict-tail` P0 was caused by harness misuse of `--sync-mode`. Future reports must distinguish product, harness, infrastructure, environment, and inconclusive outcomes mechanically. + +## Current Plan Assessment + +1. Good one-time campaign: + 1. Fresh remote roots and local mountpoints. + 2. Explicit avoidance of existing mounts. + 3. Strict and interactive mode coverage. + 4. Realistic agent workload using git. + 5. Stress coverage for large files, small files, concurrency, unmount, and kill-during-write. + 6. Artifact discipline under `/tmp/drive9-agent-test-$TS/`. + +2. Not yet a reusable harness: + 1. Shell orchestration allowed harness bugs such as bare `wait` and label/sync-mode confusion. + 2. Test case identity, sync mode, remote root suffix, and mountpoint suffix were not typed separately. + 3. Results were mostly markdown/text rather than trendable structured output. + 4. There was no local filesystem or prior-run baseline for performance claims. + 5. Failure classes were not mechanically separated. + 6. Resuming after partial failure required manual continuation. + 7. Observability was collected, but not in an agent-legible schema. + +## Harness Design Principles + +1. Repository-local system of record: + The harness, case definitions, schemas, reports, and known repros should live under `e2e/agent-harness/`, not in temporary scripts or chat context. + +2. Agent-legible feedback loop: + A future agent should be able to run one command, inspect JSONL events and summary JSON, identify the failing case, rerun a minimized repro, and then validate a fix. + +3. Mechanical boundaries: + The harness must enforce root and mountpoint safety in code. Case names, sync modes, mount labels, and remote roots must be separate typed values. + +4. Structured evidence first: + Markdown is for humans. `events.jsonl`, `failures.jsonl`, and `summary.json` are the primary data model. + +5. Progressive disclosure: + Default runs should be short, local, and useful. Stress, fault-injection, production observability, and destructive cleanup phases must be opt-in modes with explicit approval. + +6. Garbage collection: + Successful generated roots should be cleanable by a dedicated GC command. Failed roots should be retained with explicit retention metadata. + +7. Phase-1 implementation boundary: + The first implementation should ship only `preflight`, `run --suite smoke`, `run --suite regression --case ...`, and `report`. It should not collect production Kubernetes logs or metrics, run stress/fault cases, or delete remote roots. + +8. Contract-first implementation: + Phase 1 must define case-file structure, workload config, expected outcome semantics, and harness error classification before adding workload execution code. + +## Module And Dependency Contracts + +1. Go module path: + All harness Go imports must use the repository module root from `go.mod`: `github.com/mem9-ai/dat9`. + +2. Expected internal imports: + 1. `github.com/mem9-ai/dat9/pkg/client` for file read/write helpers and any direct filesystem API probes. + 2. `github.com/mem9-ai/dat9/pkg/traceid` for deterministic direct-HTTP trace id generation when direct HTTP probes are added. + 3. `github.com/mem9-ai/dat9/pkg/mountpath` or `github.com/mem9-ai/dat9/pkg/pathutil` for path canonicalization where local validation must match Drive9 path rules. + +3. Provision and status helper boundary: + Use a local harness `net/http` helper, not `pkg/client`, for `POST /v1/provision` and `GET /v1/status`. The SDK does not expose a public provision method or typed status response; the harness should keep these two control-plane calls explicit with small typed request/response structs. + +4. YAML parser: + Use `gopkg.in/yaml.v3` for case parsing. Promote it to a direct dependency if phase-1 code imports it. + +5. CLI execution boundary: + FUSE workloads should drive the `drive9` CLI binary specified by `--drive9-bin`. Direct API probes may use `pkg/client`; CLI/FUSE behavior must still be validated through the binary under test. + +## Proposed Layout + +```text +e2e/agent-harness/ + README.md + cases/ + smoke.yaml + regression.yaml + stress.yaml # deferred + fault.yaml # deferred + cmd/drive9-agent-harness/ + main.go + internal/ + casefile/ + runner/ + mountproc/ + oracle/ + report/ + safety/ + schemas/ + case.schema.json + manifest.schema.json + event.schema.json + failure.schema.json + metric.schema.json + summary.schema.json + gating.schema.json + repro/ + known-path-edge.sh + known-git-lock.sh +``` + +## Runner Commands + +Phase-1 commands: + +1. `drive9-agent-harness preflight` + Validate host, binary, config, server, FUSE, tool availability, mount safety, and writable artifact root. + +2. `drive9-agent-harness run --suite smoke` + Fast correctness and remount tests. Default reusable confidence check. + +3. `drive9-agent-harness run --suite regression --case path-edge,git-lock,doctor-no-allow-other` + Target known P1 failures and validate fixes quickly. + +4. `drive9-agent-harness report --run-dir /tmp/drive9-agent-test-$RUN_ID` + Rebuild `summary.md` and `summary.json` from structured artifacts. + +Common phase-1 flags: + +1. `--artifact-root /tmp` + Parent for run directories. The runner creates `drive9-agent-test-$RUN_ID` below it. +2. `--mount-root /tmp` + Parent for generated mountpoints. The runner creates `drive9-agent-$RUN_ID-$CASE_ID` below it. +3. `--remote-root-base /agent-adversarial-$RUN_ID` + Remote root prefix used for generated case roots. +4. `--drive9-bin drive9` + CLI binary or absolute path to the binary under test. +5. `--server $DRIVE9_BASE` + Server URL, defaulting to the same environment convention as existing e2e scripts. Phase 1 should default to `http://127.0.0.1:9009`; hosted or production endpoints must be selected explicitly. +6. `--api-key $DRIVE9_API_KEY` + Existing API key. Phase 1 requires this unless `--provision` is explicitly set. +7. `--provision` + Provision a fresh tenant before running cases. The harness must call `POST /v1/provision`, capture the returned `api_key`, then poll `GET /v1/status` until `status == "active"` or the provision timeout expires. +8. `--provision-timeout 120s` + Maximum wait for a provisioned tenant to become active. + +Deferred commands: + +1. `drive9-agent-harness run --suite stress` + Run bounded `fio`, small-file storm, and concurrent workloads. + +2. `drive9-agent-harness run --suite fault` + Run kill-during-write and unmount edge cases. Requires explicit human approval. + +3. `drive9-agent-harness gc --older-than 7d --successful-only` + Clean only successful generated roots and mountpoints. + +4. `drive9-agent-harness collect-server-evidence --run-dir ...` + Optional production log and metrics collection. Requires explicit external-service approval and configured Kubernetes or metrics-backend credentials. + +## Phase 1 Scope + +Phase 1 is ready to implement with this scope: + +1. Go command under `e2e/agent-harness/cmd/drive9-agent-harness`. +2. YAML case loading for smoke and regression suites only. +3. Local mount lifecycle, process tracking, command timeouts, and structured artifact writing. +4. Local-only observability: command stdout/stderr, mount logs, mount table snapshots, tool versions, process snapshots on timeout, and mount perf counters when available. +5. Report regeneration from `manifest.json`, `events.jsonl`, `failures.jsonl`, and `metrics.jsonl`. +6. Regression coverage for path-edge, git-lock, and doctor-no-allow-other cases. + +Phase 1 non-goals: + +1. Stress suite. +2. Fault-injection suite. +3. Garbage collection command. +4. Kubernetes log collection. +5. Production metrics backend queries. +6. New Drive9 client or CLI trace propagation flags. + +## Remote Root Lifecycle + +Phase 1 must create the remote source before mounting because `drive9 mount :/remote ` rejects nonexistent remote roots. + +1. For each case, derive: + 1. `remote_root_base`: canonical `--remote-root-base`, for example `/agent-adversarial-$RUN_ID`. + 2. `case_remote_root`: `path.Join(remote_root_base, case.remote_root_suffix)`, for example `/agent-adversarial-$RUN_ID/path-edge-strict`. + 3. `mountpoint`: `filepath.Join(mount_root, "drive9-agent-"+run_id+"-"+case.mountpoint_suffix)`. +2. Before starting the mount, run `drive9 fs mkdir :$case_remote_root` with the selected API key. `drive9 fs mkdir` creates parent directories, so this also creates `remote_root_base` when absent. +3. Mount exactly `:$case_remote_root` at the generated mountpoint. Workload paths are relative to the mounted case root. +4. Do not mount `/` and do not emulate scoping by prefixing every workload path under the base. The mount source is the isolation boundary. +5. Phase 1 must not delete `case_remote_root`. Successful local mountpoints may be removed after verified unmount; remote-root cleanup is deferred to `gc`. + +## Case Schema + +Each suite file contains optional defaults and a list of cases. Phase 1 uses `cases/smoke.yaml` and `cases/regression.yaml`. + +```yaml +defaults: + timeout: 2m + cleanup: retain_on_failure + mount_ready_timeout: 20s + remote_visibility_timeout: 5s + command_retry_count: 8 + command_retry_sleep: 2s +cases: + - id: smoke-strict + suite: smoke + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: smoke-strict + mountpoint_suffix: smoke-strict + workload: + type: mount_smoke + files: + - relative_path: cli-to-fuse.txt + content: smoke cli to fuse + - relative_path: fuse-to-cli.txt + content: smoke fuse to cli + oracles: + - type: cli_read_equals + - type: remount_hash_equal + severity: + failure: P1 +``` + +Each case should be declarative: + +```yaml +id: path-edge-strict +suite: regression +sync_mode: strict +expected_outcome: bug_reproduced +remote_root_suffix: path-edge-strict +mountpoint_suffix: path-edge-strict +timeout: 2m +cleanup: retain_on_failure +workload: + type: path_matrix + paths: + - "space name unicode-测试 punct_!@%.txt" +oracles: + - type: fuse_write_success + - type: cli_read_equals + - type: remount_hash_equal +severity: + failure: P1 +``` + +Required typed fields: + +1. `id`: stable case id. +2. `suite`: `smoke`, `regression`, `stress`, or `fault`. +3. `expected_outcome`: `baseline_pass`, `bug_reproduced`, or `fix_verified`. +4. `sync_mode`: `strict`, `interactive`, `auto`, or omitted for CLI-only cases. `auto` passes Drive9's automatic sync-mode selection through to `drive9 mount`. +5. `remote_root_suffix`: not reused as sync mode or label. +6. `mountpoint_suffix`: not reused as sync mode or label. +7. `timeout`: total case timeout, inherited from defaults when omitted. +8. `cleanup`: `always`, `retain_on_failure`, or `never`, inherited from defaults when omitted. +9. `workload`: typed workload config. +10. `oracles`: expected observable outcomes. +11. `severity`: default classification if an oracle fails. + +Expected outcome semantics: + +1. `baseline_pass` + All oracles must pass. Any oracle failure is a product, environment, infrastructure, harness, or inconclusive failure according to classification. +2. `bug_reproduced` + The case is expected to reproduce a known bug, but its oracles still encode correct behavior. If a correctness oracle fails, the report should count it under "known bugs reproduced", not as a new product failure or CI-gating failure. If all oracles pass, report `known_bug_fixed_candidate` so the case can be flipped to `fix_verified`. +3. `fix_verified` + The case validates a previously known bug fix. All oracles must pass. Any oracle failure is a product regression. + +Phase-1 workload types: + +1. `mount_smoke` + Mount a generated remote root, run CLI-to-FUSE and FUSE-to-CLI byte equality checks, unmount, remount, and verify durability. + Required config: + 1. `files`: list of `{relative_path, content}` objects. + 2. Optional `read_after_write_timeout`, defaulting to suite `remote_visibility_timeout`. + 3. Optional `remount: true|false`, defaulting to `true`. +2. `path_matrix` + Create each configured path through FUSE, verify exact bytes through CLI, remount, and verify exact bytes again. + Required config: + 1. `paths`: list of relative file paths. + 2. Optional `content_template`, defaulting to `case_id:path`. +3. `git_workflow` + Clone a small configured repository into the mount, check for `.git/*.lock`, run `git status`, modify a tracked file, run local `git add` and `git commit`, then verify expected file state after remount. + Required config: + 1. `clone_url`. + 2. Optional `git_binary`, defaulting to `git`. + 3. Optional `git_timeout`, defaulting to `case timeout`. + 4. Optional `git_user_name`, defaulting to `Drive9 Harness`. + 5. Optional `git_user_email`, defaulting to `drive9-harness@example.invalid`. + 6. `clone_dir`: directory name under the mountpoint. + 7. `mutation`: object with `path`, `mode` (`append` or `overwrite`), and `content`. + 8. `commit_message`: local commit message. + 9. `expected_locks`: glob list expected after the workflow. This must encode correct behavior. For phase-1 git cases it should be empty, including `expected_outcome: bug_reproduced` cases. + 10. `expected_status`: expected `git status --porcelain` state after commit, usually empty for fix verification. + 11. `expected_commit_delta`: expected commit count delta after local commit. + 12. `remount_verify_paths`: list of repository-relative paths and expected SHA-256 values to verify after remount. +4. `doctor_fuse` + Run `drive9 doctor fuse` for a generated mountpoint and classify the result against a configured expectation. + Required config: + 1. `expect_exit`: expected exit code. + 2. `allow_nonzero_when_no_allow_other`: boolean, defaulting to `false`. + +Phase-1 oracle payloads: + +1. `fuse_write_success` + Derived from workload unless explicitly overridden. For `mount_smoke`, derive paths and byte counts from `workload.files`. For `path_matrix`, derive paths from `workload.paths` and byte counts from generated content. Optional YAML override fields: `path`, `expected_bytes`, `command_id`. +2. `cli_read_equals` + Derived from workload unless explicitly overridden. For file workloads, derive remote path and expected SHA-256 from generated or configured content. Optional YAML override fields: `remote_path`, `sha256`. +3. `remount_hash_equal` + Derived from workload unless explicitly overridden. The runner records pre-unmount hashes for workload output paths, then compares them after remount. Optional YAML override fields: `paths`, `expected_hashes`. +4. `no_git_locks` + Required for `git_workflow` and derived from `workload.expected_locks`. For correctness, this list should normally be empty. Optional YAML override fields: `globs`. +5. `git_status_equals` + Required for `git_workflow` and derived from `workload.expected_status`. Optional YAML override fields: `expected_status`. +6. `git_commit_count` + Required for `git_workflow` and derived from `workload.expected_commit_delta`. Optional YAML override fields: `repo_path`, `expected_delta`. +7. `command_exit` + Required when the workload defines a command whose exit status is the primary oracle, such as `doctor_fuse`. Derived from `workload.expect_exit` when present. Optional YAML override fields: `command_id`, `expected_exit`. + +Bare oracle entries such as `{type: cli_read_equals}` are allowed only when the selected workload defines enough data for deterministic derivation. Case validation must reject a bare oracle when required derived fields are unavailable. + +All timeout, retry, and visibility waits must be explicit in the case file or inherited from suite defaults. Phase-1 defaults should match existing e2e conventions unless a case overrides them: mount readiness `20s`, remote visibility `5s`, large-file visibility `30s`, command retry count `8`, and retry sleep `2s`. + +Provisioning rules: + +1. A suite may set `requires_fresh_tenant: true` in defaults or on a case. +2. If any selected case requires a fresh tenant and `--api-key` is absent, the runner must require `--provision`. +3. If `--api-key` is present, the runner must not provision unless `--provision` is also set. +4. `preflight` must verify either an API key is available or `--provision` was requested. +5. Provisioned API keys must be written only to `manifest.json` redacted form and process environment for child `drive9` commands. Full secrets must not be emitted to logs or markdown reports. +6. `POST /v1/provision` response shape: `{ "api_key": string, "status": string }`. +7. `GET /v1/status` response shape for readiness: `{ "status": string }`. Unknown fields must be ignored. + +## Structured Artifacts + +All runs write to `$ARTIFACT_ROOT/drive9-agent-test-$RUN_ID/`; the default artifact root is `/tmp`. + +1. `manifest.json` + Run id, host, binary version, server, suites, git SHA of harness, approval mode, artifact root, mount root, remote root base, generated mountpoints, generated remote roots, and tracked process groups. + +2. `events.jsonl` + Every command, process start/stop, mount lifecycle event, timeout, artifact path, duration, exit code, and signal. + +3. `failures.jsonl` + One object per failure with fields: + `case_id`, `severity`, `class`, `oracle`, `message`, `repro_path`, `artifact_refs`. + +4. `summary.json` + Counts by severity/class/suite, durations, performance metrics, pass/fail status, cleanup status. + +5. `summary.md` + Human-readable report generated from `summary.json` and `failures.jsonl`. + +6. `repro/.sh` + Minimized reproduction for each failed product case. + +7. `mount/*.log` + Raw mount logs and perf counter output. + +8. `metrics/*.json` + Parsed metric sidecars. Phase 1 should include git duration and mount perf counters. Fio and small-file throughput sidecars are deferred until the stress suite lands. + +9. `metrics.jsonl` + One derived metric event per case and metric name. Phase 1 should include mount startup duration, command durations, git clone duration, byte counts, file counts, and parsed mount perf counters when present. + +10. `gating.json` + Machine-readable CI decision with `pass`, `fail`, `known_bug_reproduced`, `known_bug_fixed_candidate`, and `non_gating` counts. `bug_reproduced` cases must not fail the gate unless explicitly selected with a future strict mode. + +## V1 Structured Artifact Schemas + +All v1 artifacts must include `schema_version: "agent-harness.v1"` where the format is JSON object based. JSONL records must include the same field per line. + +1. `manifest.json` + Required fields: `schema_version`, `run_id`, `started_at`, `host`, `harness_git_sha`, `drive9_version`, `server`, `suites`, `selected_cases`, `artifact_root`, `mount_root`, `remote_root_base`, `generated_mountpoints`, `generated_remote_roots`, `process_groups`, `api_key_redacted`, `approval_mode`. +2. `events.jsonl` + Required fields: `schema_version`, `run_id`, `case_id`, `ts`, `type`, `message`, `duration_ms`, `command_id`, `exit_code`, `signal`, `artifact_refs`. + Allowed `type` values: `run_start`, `run_end`, `case_start`, `case_end`, `command_start`, `command_end`, `mount_start`, `mount_ready`, `mount_end`, `unmount_start`, `unmount_end`, `oracle_start`, `oracle_end`, `timeout`, `artifact_written`, `cleanup_start`, `cleanup_end`. +3. `failures.jsonl` + Required fields: `schema_version`, `run_id`, `case_id`, `ts`, `severity`, `class`, `oracle`, `expected_outcome`, `message`, `observed`, `expected`, `repro_path`, `artifact_refs`. + Allowed `severity` values: `P0`, `P1`, `P2`, `P3`. + Allowed `class` values: `product`, `harness`, `environment`, `infrastructure`, `inconclusive`. +4. `metrics.jsonl` + Required fields: `schema_version`, `run_id`, `case_id`, `ts`, `name`, `value`, `unit`, `source`, `artifact_refs`. + Allowed `unit` values: `ms`, `bytes`, `files`, `count`, `bool`. + Phase-1 metric names: `mount_startup_ms`, `command_duration_ms`, `git_clone_duration_ms`, `bytes_written`, `bytes_read`, `file_count`, `mount_perf_counter`. +5. `summary.json` + Required fields: `schema_version`, `run_id`, `status`, `started_at`, `ended_at`, `duration_ms`, `cases`, `counts`, `artifacts`, `cleanup`. + Allowed `status` values: `passed`, `failed`, `known_bugs_reproduced`, `inconclusive`, `harness_failed`. + `counts` must include totals by `suite`, `expected_outcome`, `severity`, and `class`. +6. `gating.json` + Required fields: `schema_version`, `run_id`, `gate_status`, `pass`, `fail`, `known_bug_reproduced`, `known_bug_fixed_candidate`, `non_gating`, `blocking_failures`. + Allowed `gate_status` values: `pass`, `fail`, `non_gating`, `harness_failed`. + `blocking_failures` must include only failures from `baseline_pass` and `fix_verified` cases unless a future strict-known-bug mode is explicitly enabled. + +## Failure Classification + +1. `product` + Drive9 behavior violates the oracle. + +2. `harness` + Runner bug, invalid case definition, bad shell snippet, or invalid argument generation. + +3. `environment` + Host setup issue such as missing tool or unavailable FUSE capability. + +4. `infrastructure` + Network, EC2, GitHub, or external service availability issue. + +5. `inconclusive` + Evidence is insufficient; data is preserved but not counted as product failure. + +The report should show both raw and triaged counts, but product release decisions should use triaged counts. + +Internal error contracts: + +1. `internal/casefile` + Define sentinel errors for parse and validation failures, for example `ErrParse` and `ErrValidation`. Both classify as `harness`. +2. `internal/safety` + Define sentinel errors for unsafe generated paths, for example `ErrInvalidRoot`, `ErrInvalidGeneratedPath`, and `ErrExistingMountpoint`. These classify as `harness` unless they reveal host state outside the harness contract, in which case they classify as `environment`. +3. `internal/mountproc` + Define sentinel errors for mount startup timeout, unmount timeout, process-state mismatch, and unexpected process exit. Startup timeout is usually `environment`; process-state mismatch and unsafe cleanup are `harness`; remote Drive9 errors observed after successful mount are `product` or `infrastructure` according to oracle evidence. +4. `internal/oracle` + Oracle failures must carry `case_id`, `oracle`, observed value, expected value, and enough artifact references for report regeneration. + +## Core Oracles + +1. CLI write to FUSE read: bytes match within timeout. +2. FUSE write to CLI read: bytes match within timeout. +3. Rename: target visible, old path absent. +4. Overwrite/truncate: final remote bytes match exact expected bytes. +5. Delete: deleted path absent via CLI and FUSE after TTL budget. +6. Path matrix: FUSE write succeeds and CLI read returns exact bytes. +7. Remount durability: normalized file hash set matches before/after. +8. Dual mount: mount B observes mount A write and mount A observes mount B write within TTL budget. +9. Git clone: no `.git/*.lock` after successful clone. +10. Git local workflow: expected commit count and expected renamed/deleted state. +11. Small-file storm: expected file count, read count, delete completion within threshold. +12. Open-fd unmount: first unmount may return busy, follow-up after fd close must succeed. +13. Kill-during-write: classify partial data as inconclusive unless writer acknowledged completion. +14. Performance: compare against configured local baseline and previous Drive9 median before calling a regression. + +## Workload Suites + +1. Smoke: + 1. preflight + 2. mount strict + 3. CLI write to FUSE read + 4. FUSE write to CLI read + 5. remount hash check + 6. clean unmount + +2. Regression: + 1. Phase 1: path-edge strict and interactive + 2. Phase 1: git-lock strict and interactive + 3. Phase 1: doctor-no-allow-other UX check + 4. Deferred: dual-mount stale-cache check + +3. Stress (deferred): + 1. `fio` 1 GiB sequential write with fsync + 2. cold sequential read + 3. random read + 4. 1000 small files + 5. 8 parallel 64 MiB writers + +4. Fault (deferred): + 1. unmount with open fd + 2. kill fresh mount process during active write + 3. remount and classify recovered, partial, missing, or corrupt state + +## Safety Model + +1. `--artifact-root`, `--mount-root`, and `--remote-root-base` must be absolute, canonicalized paths with no `..`, no empty generated suffixes, and no shell expansion. +2. Generated mountpoints must be direct children of `--mount-root` and must match `drive9-agent-$RUN_ID-$CASE_ID`. +3. Generated remote roots must be direct children of `--remote-root-base`: `path.Join(remote_root_base, remote_root_suffix)`. The suffix must be a single clean path segment equal to the case's `remote_root_suffix`; it must not start with `/` and must not contain `/`, `.`, `..`, or backslashes. +4. The runner must reject a mountpoint that already exists as a mount, symlink, non-empty directory, regular file, or path outside `--mount-root`. +5. The runner must reject remote roots outside the generated remote-root base and must reject a computed `case_remote_root` equal to `remote_root_base` itself. +6. Existing EC2 mounts `/home/ubuntu/w3`, `/home/ubuntu/w4`, `/home/ubuntu/w5` are only part of the EC2 preset and remain read-only context. They are not phase-1 defaults. +7. Each mount process runs in its own process group. +8. Cleanup may kill only process groups recorded in `manifest.json`. +9. No bare `wait`; wait only explicitly tracked workload pids. +10. Every external command has a timeout. +11. Failed cases retain data by default. +12. Successful case cleanup may remove only the generated local mountpoint after a verified unmount. Remote-root GC is deferred to the future `gc` command. +13. Fault suite requires explicit human approval and is out of phase-1 scope. + +## Observability + +1. Client-side harness observability: + 1. Capture `drive9 --version`, mount table, config server, tool versions, `/dev/fuse`, and `doctor fuse`. + 2. Capture mount perf counters and parse them into `metrics/mount-*.json`. + 3. On timeout, capture `ps`, `/proc/$pid/status`, `/proc/$pid/stack` if readable, `/proc/$pid/fd`, and mount table. + 4. Capture git timings, file counts, lock paths, and exact git stderr. + 5. Deferred stress suite: capture fio JSON and derive summary metrics. + 6. Capture cleanup status for each mountpoint and remote root. + +2. Agent-readable debug interface: + 1. `events.jsonl` is the unified run timeline. + 2. `metrics.jsonl` records derived metrics per case. + 3. `failures.jsonl` records oracle failures with artifact references. + 4. `debug//trace-ids.jsonl` records request step, trace id, route/path, start time, end time, and response status when known. + 5. `debug//` contains sliced logs, process snapshots, mount table, proc data, stderr/stdout, metric snapshots, and a repro script. + +3. Phase 1 trace correlation: + 1. Drive9 server supports caller-provided `X-Trace-ID`; if absent, it generates one and echoes it in the response header. + 2. Direct HTTP harness probes may generate deterministic trace IDs, for example `drive9-harness-$RUN_ID-$CASE_ID-$STEP-$N`. + 3. The harness must record both the requested trace ID and the response `X-Trace-ID` for direct HTTP probes. If they differ, the case should be marked `trace_status: mismatch`. + 4. Current CLI and FUSE flows do not expose a trace ID option. Phase-1 CLI/FUSE cases must record `trace_status: server_generated_unlinked` and rely on local artifacts, generated remote paths, bounded time windows, actor IDs, and tenant/auth fields. + +## Deferred: Server-Side Evidence + +These sections are phase-2+ design notes. They are not phase-1 implementation tasks. + +1. Future CLI/FUSE trace propagation: + 1. Add one of: + 1. a CLI flag such as `--trace-id-prefix` or `--trace-id`; + 2. an environment variable such as `DRIVE9_TRACE_ID_PREFIX`; + 3. a client option that injects `X-Trace-ID` before each request. + 2. Do not put per-case `trace_id`, `run_id`, or `case_id` into Prometheus labels. Those belong in logs and structured artifacts, not metrics labels. + +2. Server-side log access and local LogQL analysis workflow: + This is a deferred, opt-in mode and not part of phase 1. It requires the user to approve the external service action and provide or select the target environment. The production preset currently uses kube context `prod-dat9-eks-ap-southeast-1`, namespace `dat9`, and deployment selector `app=dat9-server`. + + 1. Current verified Kubernetes commands for the production preset: + + ```bash + kubectl config get-contexts + kubectl --context prod-dat9-eks-ap-southeast-1 -n dat9 get pods -l app=dat9-server -o wide + kubectl --context prod-dat9-eks-ap-southeast-1 -n dat9 get deploy dat9-server -o wide + kubectl --context prod-dat9-eks-ap-southeast-1 -n dat9 logs -l app=dat9-server --since=10m --tail=500 > "$RUN_DIR/debug//server-logs.jsonl" + ``` + + 2. Query the downloaded slice with `logcli --stdin`. The verified stdin selector is `{source="logcli"}`: + + ```bash + SERVER_LOG="$RUN_DIR/debug//server-logs.jsonl" + cat "$SERVER_LOG" | logcli --stdin labels + cat "$SERVER_LOG" | logcli --stdin --quiet --output raw query '{source="logcli"} | json | msg="http_request"' + cat "$SERVER_LOG" | logcli --stdin --quiet --output raw query '{source="logcli"} | json | msg="http_request" | duration_ms > 300' + cat "$SERVER_LOG" | logcli --stdin --quiet --output raw query '{source="logcli"} | json | msg="http_request" | status >= 400' + cat "$SERVER_LOG" | logcli --stdin --quiet --output raw query '{source="logcli"} | json | trace_id=""' + cat "$SERVER_LOG" | logcli --stdin --quiet --output raw query '{source="logcli"} | json | msg="datastore_op_timing" | duration_ms > 100 | line_format "{{.operation}} result={{.result}} dur={{.duration_ms}} trace={{.trace_id}}"' + ``` + + 3. In the future opt-in server-evidence mode, every failing or slow case should attach logcli extracts: + 1. `debug//server-http-errors.txt` for `status >= 400`. + 2. `debug//server-http-slow.txt` for slow `http_request` entries. + 3. `debug//server-datastore-slow.txt` for slow datastore operations. + 4. `debug//server-trace-.jsonl` for full trace expansion when a relevant `trace_id` is known. + + 4. Current log format is JSON and includes fields such as `trace_id`, `msg`, `path`, `method`, `route`, `status`, `duration_ms`, `tenant_id`, `api_key_id`, backend timing fields, and datastore operation timings. + 5. `logcli --stdin` is useful for log filtering and JSON field extraction, but not metric queries. For aggregate summaries from downloaded slices, use `jq` or a small harness parser unless the logs are queried from Loki directly. + 6. `--limit` may not cap stdin output reliably, so generated extracts should use deterministic post-processing such as `head`, `sed -n`, or harness-side line limits. + 7. Stdin mode assigns local ingestion timestamps, so case analysis should rely on embedded JSON fields such as `ts`, `trace_id`, `duration_ms`, `path`, and `msg`. + +3. Server-side metrics workflow: + This is a deferred, opt-in mode and not part of phase 1. + + 1. The server exposes Prometheus text at `/metrics`. + 2. In Kubernetes, the verified service proxy path uses the named service port `http`: + + ```bash + kubectl --context prod-dat9-eks-ap-southeast-1 get --raw '/api/v1/namespaces/dat9/services/dat9-server:http/proxy/metrics' + ``` + + 3. Current exposed metric families: + 1. `dat9_http_requests_total` + 2. `dat9_http_request_duration_seconds` + 3. `dat9_http_inflight_requests` + 4. `dat9_db_operations_total` + 5. `dat9_db_operation_duration_seconds` + 6. `dat9_db_pool_registered` + 7. `dat9_db_pool_connections` + 8. `dat9_db_pool_wait_count_total` + 9. `dat9_db_pool_wait_duration_seconds_total` + 10. `dat9_db_pool_closes_total` + 11. `dat9_service_operations_total` + 12. `dat9_service_operation_duration_seconds` + 13. `dat9_service_gauge` + 14. `dat9_tenant_events_total` + 15. `dat9_module_up` + 16. `dat9_module_uptime_seconds` + 17. `dat9_fuse_operations_total` + 18. `dat9_fuse_operation_duration_seconds` + 19. `dat9_fuse_operation_bytes_total` + 20. `dat9_fuse_remote_operations_total` + 21. `dat9_fuse_remote_operation_duration_seconds` + 22. `dat9_fuse_remote_operation_bytes_total` + + 4. These metrics are useful for service-level triage: + 1. route-level HTTP error and latency changes; + 2. DB operation latency and errors; + 3. backend, datastore, S3, tenant pool, upload, quota, media extraction, and worker operation failures; + 4. DB pool pressure; + 5. tenant lifecycle/auth event counts. + 5. These metrics are not sufficient for per-case root cause by themselves because they do not include `trace_id`, `case_id`, `run_id`, path, tenant, or API key labels. + 6. In the future opt-in server-evidence mode, the harness should collect `metrics/server-before.prom` and `metrics/server-after.prom` around each case, then write `metrics/server-delta.json` with changed counters, histogram counts/sums, and gauges. + 7. When the production metrics backend query interface is known, the harness should prefer backend range queries around `[case_start - 30s, case_end + 30s]` and store results under `metrics/backend//`. + 8. Metrics findings should be attached as supporting evidence, not primary case identity. The primary per-case join key remains `trace_id` for direct HTTP probes and log time/path filters for CLI/FUSE until trace propagation exists. + +4. Server-side case correlation contract: + 1. Direct HTTP case correlation is solved: inject `X-Trace-ID`, record the echoed header, and query logs by `trace_id`. + 2. CLI/FUSE phase-1 cases must record `server_trace_status: server_generated_unlinked` and rely on local artifacts, generated remote paths, bounded time windows, tenant/auth fields, and the FUSE `X-Dat9-Actor` value. + 3. CLI/FUSE trace propagation is a future Drive9 client/CLI change, not a blocker for phase-1 harness implementation. + 4. Need to document the production metrics backend query interface, credentials, and recommended query syntax before implementing server-evidence mode. + 5. Need to decide whether per-case server metric evidence should come from live `/metrics` before/after snapshots, metrics backend range queries, or both. + 6. Until CLI/FUSE trace propagation is implemented, harness reports should include `server_trace_status: direct_http_trace_verified_cli_fuse_unlinked` when a run mixes direct HTTP and CLI/FUSE cases. + +## Report Format + +`summary.md` should include: + +1. Run identity. +2. Product findings first. +3. Harness/environment/infrastructure failures second. +4. Performance table with baselines. +5. Passed oracles. +6. Inconclusive evidence. +7. Cleanup status. +8. Repro script paths. + +`summary.json` should be treated as authoritative for automation. + +## Harness Testing + +1. `internal/casefile` + Table-driven tests for suite defaults, multi-case YAML loading, expected-outcome validation, missing required workload fields, and invalid enum values. +2. `internal/safety` + Unit tests for rejecting empty roots, relative roots, `..`, symlinks, existing non-empty directories, existing mounts, and generated paths outside configured roots. +3. `internal/oracle` + Unit tests for pass/fail evaluation, `bug_reproduced` handling, `known_bug_fixed_candidate`, and artifact references. +4. `internal/report` + Golden tests that regenerate `summary.json` and `summary.md` from fixture `manifest.json`, `events.jsonl`, `failures.jsonl`, and `metrics.jsonl`. +5. `internal/mountproc` + Unit tests for command construction, timeout classification, process state recording, and cleanup decisions using fake process runners where possible. + +## Phase 1 Delivery Split + +1. Phase 1a: + Preflight, suite YAML parsing, safety validation, process/artifact writers, `mount_smoke`, and report regeneration. +2. Phase 1b: + Regression workloads, oracle semantics for `bug_reproduced` and `fix_verified`, `gating.json`, and harness unit/golden tests. + +## Implementation Plan + +1. Build a minimal Go runner for `preflight`, `run --suite smoke`, `run --suite regression --case ...`, and `report`. +2. Add phase-1 case schema parsing and validation for `mount_smoke`, `path_matrix`, `git_workflow`, and `doctor_fuse`. +3. Add safety validation for artifact root, mount root, remote root base, generated mountpoints, and generated remote roots. +4. Add mount process manager with process groups, explicit PID tracking, timeouts, and verified unmount. +5. Add event, failure, metric, and manifest JSONL/JSON writers. +6. Add local artifact collectors for versions, mount table, mount logs, command stdout/stderr, timeout process snapshots, and mount perf counters. +7. Port the path-edge, git-lock, and doctor-no-allow-other cases as regression tests. +8. Add report regeneration from structured artifacts. +9. Defer stress suite, fault suite, remote-root GC, production logs/metrics, and CLI/FUSE trace propagation. + +## Estimated Scope + +1. Phase-1 smoke and regression runner: `800-1200 LoC`. +2. JSON schemas, docs, and harness tests: `250-450 LoC` equivalent documentation/test code. +3. Deferred stress and fault suites: additional `250-500 LoC`. +4. Deferred production server-evidence mode: additional `150-300 LoC`. + +## Success Criteria + +1. A future agent can run a single smoke command and get a structured pass/fail report. +2. Known path-edge, git-lock, and doctor-no-allow-other findings are reproducible by named regression cases. +3. Harness failures cannot be confused with product P0/P1 findings. +4. All fresh mounts are unmounted after a completed run. +5. All failed data is retained with explicit metadata. +6. Reports can be regenerated from raw structured artifacts without rerunning workloads. diff --git a/e2e/agent-harness/README.md b/e2e/agent-harness/README.md new file mode 100644 index 000000000..bb292666a --- /dev/null +++ b/e2e/agent-harness/README.md @@ -0,0 +1,74 @@ +--- +title: Drive9 Agent Harness +--- + +## Commands + +1. Build: + + ```bash + go build ./e2e/agent-harness/cmd/drive9-agent-harness + ``` + +2. Preflight: + + ```bash + drive9-agent-harness preflight --api-key "$DRIVE9_API_KEY" + ``` + +3. Smoke: + + ```bash + drive9-agent-harness run --suite smoke --api-key "$DRIVE9_API_KEY" + ``` + +4. Targeted regression: + + ```bash + drive9-agent-harness run --suite regression --case path-edge-strict,git-lock-strict,doctor-no-allow-other --api-key "$DRIVE9_API_KEY" + ``` + +5. Stress: + + ```bash + drive9-agent-harness run --suite stress --api-key "$DRIVE9_API_KEY" + ``` + +6. Fault injection: + + ```bash + drive9-agent-harness run --suite fault --allow-fault --api-key "$DRIVE9_API_KEY" + ``` + +7. Regenerate report: + + ```bash + drive9-agent-harness report --run-dir /tmp/drive9-agent-test-YYYYMMDDTHHMMSSZ + ``` + +8. Garbage collect generated local mountpoints and remote roots: + + ```bash + drive9-agent-harness gc --run-dir /tmp/drive9-agent-test-YYYYMMDDTHHMMSSZ --confirm-delete + ``` + +9. Collect server evidence: + + ```bash + drive9-agent-harness collect-server-evidence --run-dir /tmp/drive9-agent-test-YYYYMMDDTHHMMSSZ --approve-external --kube-context prod-dat9-eks-ap-southeast-1 + ``` + +## Phase 1 Contract + +1. Each case gets a generated remote root under `--remote-root-base`. +2. The harness creates the remote root with `drive9 fs mkdir` before mounting. +3. The mount source is exactly the generated case root. +4. Known-bug cases keep correct-behavior oracles. Failing product oracles are non-gating and passing product oracles become fixed candidates. +5. Remote-root deletion is deferred. Local mountpoints are removed only for `cleanup: always` after verified unmount. + +## Extended Harness + +1. `stress.yaml` covers sequential `fio`, small-file storm, and parallel writer workloads. +2. `fault.yaml` covers open-file unmount and kill-during-write recovery classification. +3. `gc` refuses to delete without `--confirm-delete` and checks run gating when `--successful-only` is set. +4. `collect-server-evidence` refuses external reads without `--approve-external`. diff --git a/e2e/agent-harness/cases/fault.yaml b/e2e/agent-harness/cases/fault.yaml new file mode 100644 index 000000000..bb918003e --- /dev/null +++ b/e2e/agent-harness/cases/fault.yaml @@ -0,0 +1,38 @@ +defaults: + timeout: 5m + cleanup: retain_on_failure + mount_ready_timeout: 20s + remote_visibility_timeout: 30s + command_retry_count: 8 + command_retry_sleep: 2s + requires_fresh_tenant: false +cases: + - id: open-fd-unmount-strict + suite: fault + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: open-fd-unmount-strict + mountpoint_suffix: open-fd-unmount-strict + workload: + type: open_fd_unmount + hold_open_duration: 1s + remount: false + oracles: + - type: unmount_busy_then_clean + severity: + failure: P2 + - id: kill-during-write-strict + suite: fault + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: kill-during-write-strict + mountpoint_suffix: kill-during-write-strict + workload: + type: kill_during_write + writer_bytes: 67108864 + kill_after: 500ms + remount: false + oracles: + - type: recovery_classified + severity: + failure: P2 diff --git a/e2e/agent-harness/cases/regression.yaml b/e2e/agent-harness/cases/regression.yaml new file mode 100644 index 000000000..a83a8d4c5 --- /dev/null +++ b/e2e/agent-harness/cases/regression.yaml @@ -0,0 +1,133 @@ +defaults: + timeout: 2m + cleanup: retain_on_failure + mount_ready_timeout: 20s + remote_visibility_timeout: 5s + command_retry_count: 8 + command_retry_sleep: 2s + requires_fresh_tenant: false +cases: + - id: path-edge-strict + suite: regression + sync_mode: strict + expected_outcome: bug_reproduced + remote_root_suffix: path-edge-strict + mountpoint_suffix: path-edge-strict + workload: + type: path_matrix + paths: + - "space name unicode-测试 punct_!@%.txt" + oracles: + - type: fuse_write_success + - type: cli_read_equals + - type: remount_hash_equal + severity: + failure: P1 + - id: path-edge-interactive + suite: regression + sync_mode: interactive + expected_outcome: bug_reproduced + remote_root_suffix: path-edge-interactive + mountpoint_suffix: path-edge-interactive + workload: + type: path_matrix + paths: + - "space name unicode-测试 punct_!@%.txt" + oracles: + - type: fuse_write_success + - type: cli_read_equals + - type: remount_hash_equal + severity: + failure: P1 + - id: git-lock-strict + suite: regression + sync_mode: strict + expected_outcome: bug_reproduced + remote_root_suffix: git-lock-strict + mountpoint_suffix: git-lock-strict + timeout: 3m + workload: + type: git_workflow + clone_url: fixture://basic + clone_dir: repo + mutation: + path: README.md + mode: append + content: | + + strict git workflow mutation + commit_message: drive9 harness mutation + expected_locks: [] + expected_status: "" + expected_commit_delta: 1 + remount_verify_paths: + - path: README.md + oracles: + - type: no_git_locks + - type: git_status_equals + - type: git_commit_count + - type: remount_hash_equal + severity: + failure: P1 + - id: git-lock-interactive + suite: regression + sync_mode: interactive + expected_outcome: bug_reproduced + remote_root_suffix: git-lock-interactive + mountpoint_suffix: git-lock-interactive + timeout: 3m + workload: + type: git_workflow + clone_url: fixture://basic + clone_dir: repo + mutation: + path: README.md + mode: append + content: | + + interactive git workflow mutation + commit_message: drive9 harness mutation + expected_locks: [] + expected_status: "" + expected_commit_delta: 1 + remount_verify_paths: + - path: README.md + oracles: + - type: no_git_locks + - type: git_status_equals + - type: git_commit_count + - type: remount_hash_equal + severity: + failure: P1 + - id: doctor-no-allow-other + suite: regression + expected_outcome: bug_reproduced + remote_root_suffix: doctor-no-allow-other + mountpoint_suffix: doctor-no-allow-other + workload: + type: doctor_fuse + expect_exit: 0 + allow_nonzero_when_no_allow_other: false + oracles: + - type: command_exit + severity: + failure: P2 + - id: dual-mount-visibility + suite: regression + sync_mode: interactive + expected_outcome: baseline_pass + remote_root_suffix: dual-mount-visibility + mountpoint_suffix: dual-mount-visibility + workload: + type: dual_mount_visibility + remount: true + files: + - relative_path: a-to-b.txt + content: dual mount a to b + - relative_path: b-to-a.txt + content: dual mount b to a + oracles: + - type: dual_mount_visibility + - type: remount_hash_equal + severity: + failure: P1 diff --git a/e2e/agent-harness/cases/smoke.yaml b/e2e/agent-harness/cases/smoke.yaml new file mode 100644 index 000000000..95c202d9a --- /dev/null +++ b/e2e/agent-harness/cases/smoke.yaml @@ -0,0 +1,49 @@ +defaults: + timeout: 2m + cleanup: always + mount_ready_timeout: 20s + remote_visibility_timeout: 5s + command_retry_count: 8 + command_retry_sleep: 2s + requires_fresh_tenant: false +cases: + - id: smoke-strict + suite: smoke + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: smoke-strict + mountpoint_suffix: smoke-strict + workload: + type: mount_smoke + remount: true + files: + - relative_path: cli-to-fuse.txt + content: smoke cli to fuse + - relative_path: fuse-to-cli.txt + content: smoke fuse to cli + oracles: + - type: cli_read_equals + - type: fuse_write_success + - type: remount_hash_equal + severity: + failure: P1 + - id: smoke-interactive + suite: smoke + sync_mode: interactive + expected_outcome: baseline_pass + remote_root_suffix: smoke-interactive + mountpoint_suffix: smoke-interactive + workload: + type: mount_smoke + remount: true + files: + - relative_path: cli-to-fuse.txt + content: smoke interactive cli to fuse + - relative_path: fuse-to-cli.txt + content: smoke interactive fuse to cli + oracles: + - type: cli_read_equals + - type: fuse_write_success + - type: remount_hash_equal + severity: + failure: P1 diff --git a/e2e/agent-harness/cases/stress.yaml b/e2e/agent-harness/cases/stress.yaml new file mode 100644 index 000000000..e1f8d57da --- /dev/null +++ b/e2e/agent-harness/cases/stress.yaml @@ -0,0 +1,60 @@ +defaults: + timeout: 20m + cleanup: retain_on_failure + mount_ready_timeout: 20s + remote_visibility_timeout: 30s + command_retry_count: 8 + command_retry_sleep: 2s + requires_fresh_tenant: false +cases: + - id: fio-seq-1g-strict + suite: stress + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: fio-seq-1g-strict + mountpoint_suffix: fio-seq-1g-strict + workload: + type: fio + size_bytes: 1073741824 + block_bytes: 1048576 + min_bytes_per_second: 1 + remount: false + oracles: + - type: throughput_min + severity: + failure: P2 + - id: small-file-storm-interactive + suite: stress + sync_mode: interactive + expected_outcome: baseline_pass + remote_root_suffix: small-file-storm-interactive + mountpoint_suffix: small-file-storm-interactive + workload: + type: small_file_storm + file_count: 1000 + read_sample_count: 20 + remount: true + oracles: + - type: file_count_equals + - type: cli_read_equals + - type: remount_hash_equal + severity: + failure: P2 + - id: parallel-writers-strict + suite: stress + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: parallel-writers-strict + mountpoint_suffix: parallel-writers-strict + workload: + type: parallel_writes + parallel_writers: 8 + writer_bytes: 67108864 + min_bytes_per_second: 1 + remount: true + oracles: + - type: fuse_write_success + - type: throughput_min + - type: remount_hash_equal + severity: + failure: P2 diff --git a/e2e/agent-harness/cmd/drive9-agent-harness/main.go b/e2e/agent-harness/cmd/drive9-agent-harness/main.go new file mode 100644 index 000000000..4842288f5 --- /dev/null +++ b/e2e/agent-harness/cmd/drive9-agent-harness/main.go @@ -0,0 +1,144 @@ +// Command drive9-agent-harness runs reusable Drive9 agent workloads. +package main + +import ( + "context" + "flag" + "fmt" + "os" + "strings" + + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/runner" +) + +func main() { + if len(os.Args) < 2 { + usage() + os.Exit(2) + } + ctx := context.Background() + switch os.Args[1] { + case "preflight": + cfg, err := parseConfig(os.Args[2:], "preflight") + exitOnErr(err) + exitOnErr(runner.Preflight(ctx, cfg)) + fmt.Println("preflight ok") + case "run": + cfg, err := parseConfig(os.Args[2:], "run") + exitOnErr(err) + runDir, err := runner.Run(ctx, cfg) + exitOnErr(err) + fmt.Println(runDir) + case "report": + fs := flag.NewFlagSet("report", flag.ExitOnError) + runDir := fs.String("run-dir", "", "existing run directory") + exitOnErr(fs.Parse(os.Args[2:])) + if *runDir == "" { + exitOnErr(fmt.Errorf("report requires --run-dir")) + } + exitOnErr(runner.Regenerate(*runDir)) + fmt.Println(*runDir) + case "gc": + cfg, err := parseGC(os.Args[2:]) + exitOnErr(err) + exitOnErr(runner.GC(ctx, cfg)) + fmt.Println("gc ok") + case "collect-server-evidence": + cfg, err := parseEvidence(os.Args[2:]) + exitOnErr(err) + exitOnErr(runner.CollectServerEvidence(ctx, cfg)) + fmt.Println("server evidence ok") + default: + usage() + os.Exit(2) + } +} + +func parseConfig(args []string, name string) (runner.Config, error) { + cfg := runner.DefaultConfig() + fs := flag.NewFlagSet(name, flag.ExitOnError) + suites := strings.Join(cfg.Suites, ",") + provisionTimeout := cfg.ProvisionTimeout + fs.StringVar(&cfg.ArtifactRoot, "artifact-root", cfg.ArtifactRoot, "parent for run artifacts") + fs.StringVar(&cfg.MountRoot, "mount-root", cfg.MountRoot, "parent for generated mountpoints") + fs.StringVar(&cfg.RemoteRootBase, "remote-root-base", cfg.RemoteRootBase, "remote generated root base") + fs.StringVar(&cfg.Drive9Bin, "drive9-bin", cfg.Drive9Bin, "drive9 binary under test") + fs.StringVar(&cfg.Server, "server", cfg.Server, "Drive9 server URL") + fs.StringVar(&cfg.APIKey, "api-key", cfg.APIKey, "Drive9 API key") + fs.BoolVar(&cfg.Provision, "provision", cfg.Provision, "provision a fresh tenant") + fs.BoolVar(&cfg.AllowFault, "allow-fault", cfg.AllowFault, "allow fault-injection suite cases") + fs.DurationVar(&provisionTimeout, "provision-timeout", provisionTimeout, "fresh tenant provision timeout") + fs.StringVar(&cfg.SuiteDir, "suite-dir", cfg.SuiteDir, "suite YAML directory") + fs.StringVar(&suites, "suite", suites, "comma-separated suites") + fs.StringVar(&cfg.CaseFilter, "case", cfg.CaseFilter, "comma-separated case ids") + if err := fs.Parse(args); err != nil { + return cfg, err + } + cfg.ProvisionTimeout = provisionTimeout + cfg.Suites = splitCSV(suites) + if len(cfg.Suites) == 0 { + return cfg, fmt.Errorf("--suite must select at least one suite") + } + return cfg, nil +} + +func parseGC(args []string) (runner.GCConfig, error) { + fs := flag.NewFlagSet("gc", flag.ExitOnError) + cfg := runner.GCConfig{} + fs.StringVar(&cfg.RunDir, "run-dir", "", "run directory") + fs.StringVar(&cfg.Drive9Bin, "drive9-bin", "drive9", "drive9 binary") + fs.StringVar(&cfg.Server, "server", os.Getenv("DRIVE9_BASE"), "Drive9 server URL") + fs.StringVar(&cfg.APIKey, "api-key", os.Getenv("DRIVE9_API_KEY"), "Drive9 API key") + fs.BoolVar(&cfg.SuccessfulOnly, "successful-only", true, "refuse GC for failed runs") + fs.BoolVar(&cfg.ConfirmDelete, "confirm-delete", false, "delete generated local mountpoints and remote roots") + if err := fs.Parse(args); err != nil { + return cfg, err + } + if cfg.RunDir == "" { + return cfg, fmt.Errorf("gc requires --run-dir") + } + return cfg, nil +} + +func parseEvidence(args []string) (runner.EvidenceConfig, error) { + fs := flag.NewFlagSet("collect-server-evidence", flag.ExitOnError) + cfg := runner.EvidenceConfig{} + fs.StringVar(&cfg.RunDir, "run-dir", "", "run directory") + fs.StringVar(&cfg.KubeContext, "kube-context", "", "Kubernetes context") + fs.StringVar(&cfg.Namespace, "namespace", "dat9", "Kubernetes namespace") + fs.StringVar(&cfg.Selector, "selector", "app=dat9-server", "pod label selector") + fs.StringVar(&cfg.Since, "since", "10m", "kubectl logs since duration") + fs.IntVar(&cfg.Tail, "tail", 500, "kubectl logs tail lines") + fs.StringVar(&cfg.MetricsRawPath, "metrics-raw", "", "existing Prometheus text file to attach") + fs.BoolVar(&cfg.ApproveExternal, "approve-external", false, "allow external Kubernetes/API reads") + if err := fs.Parse(args); err != nil { + return cfg, err + } + if cfg.RunDir == "" { + return cfg, fmt.Errorf("collect-server-evidence requires --run-dir") + } + return cfg, nil +} + +func splitCSV(v string) []string { + var out []string + for _, s := range strings.Split(v, ",") { + s = strings.TrimSpace(s) + if s != "" { + out = append(out, s) + } + } + return out +} + +func exitOnErr(err error) { + if err == nil { + return + } + fmt.Fprintln(os.Stderr, err) + os.Exit(1) +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: drive9-agent-harness [flags]") +} diff --git a/e2e/agent-harness/internal/casefile/casefile.go b/e2e/agent-harness/internal/casefile/casefile.go new file mode 100644 index 000000000..8a8c9fdef --- /dev/null +++ b/e2e/agent-harness/internal/casefile/casefile.go @@ -0,0 +1,469 @@ +// Package casefile loads and validates drive9 agent harness suite files. +package casefile + +import ( + "errors" + "fmt" + "os" + "path" + "path/filepath" + "strings" + "time" + + "gopkg.in/yaml.v3" +) + +var ( + ErrParse = errors.New("casefile parse") + ErrValidation = errors.New("casefile validation") +) + +type Duration struct { + time.Duration +} + +func (d *Duration) UnmarshalYAML(value *yaml.Node) error { + var s string + if err := value.Decode(&s); err != nil { + return err + } + parsed, err := time.ParseDuration(s) + if err != nil { + return err + } + d.Duration = parsed + return nil +} + +type SuiteFile struct { + Defaults Defaults `yaml:"defaults"` + Cases []Case `yaml:"cases"` +} + +type Defaults struct { + Timeout Duration `yaml:"timeout"` + Cleanup string `yaml:"cleanup"` + MountReadyTimeout Duration `yaml:"mount_ready_timeout"` + RemoteVisibilityTimeout Duration `yaml:"remote_visibility_timeout"` + CommandRetryCount int `yaml:"command_retry_count"` + CommandRetrySleep Duration `yaml:"command_retry_sleep"` + RequiresFreshTenant *bool `yaml:"requires_fresh_tenant"` +} + +type Case struct { + ID string `yaml:"id" json:"id"` + Suite string `yaml:"suite" json:"suite"` + SyncMode string `yaml:"sync_mode" json:"sync_mode,omitempty"` + ExpectedOutcome string `yaml:"expected_outcome" json:"expected_outcome"` + RemoteRootSuffix string `yaml:"remote_root_suffix" json:"remote_root_suffix"` + MountpointSuffix string `yaml:"mountpoint_suffix" json:"mountpoint_suffix"` + Timeout Duration `yaml:"timeout" json:"-"` + Cleanup string `yaml:"cleanup" json:"cleanup"` + RequiresFresh *bool `yaml:"requires_fresh_tenant" json:"requires_fresh_tenant,omitempty"` + Workload Workload `yaml:"workload" json:"workload"` + Oracles []OracleSpec `yaml:"oracles" json:"oracles"` + Severity SeveritySpec `yaml:"severity" json:"severity"` + defaults Defaults + SourceFile string `yaml:"-" json:"source_file,omitempty"` +} + +type SeveritySpec struct { + Failure string `yaml:"failure" json:"failure"` +} + +type Workload struct { + Type string `yaml:"type" json:"type"` + Files []FileSpec `yaml:"files" json:"files,omitempty"` + Paths []string `yaml:"paths" json:"paths,omitempty"` + ContentTemplate string `yaml:"content_template" json:"content_template,omitempty"` + ReadAfterWriteTimeout Duration `yaml:"read_after_write_timeout" json:"-"` + Remount *bool `yaml:"remount" json:"remount,omitempty"` + SizeBytes int64 `yaml:"size_bytes" json:"size_bytes,omitempty"` + BlockBytes int64 `yaml:"block_bytes" json:"block_bytes,omitempty"` + FileCount int `yaml:"file_count" json:"file_count,omitempty"` + ParallelWriters int `yaml:"parallel_writers" json:"parallel_writers,omitempty"` + WriterBytes int64 `yaml:"writer_bytes" json:"writer_bytes,omitempty"` + ReadSampleCount int `yaml:"read_sample_count" json:"read_sample_count,omitempty"` + MinBytesPerSecond int64 `yaml:"min_bytes_per_second" json:"min_bytes_per_second,omitempty"` + FioBinary string `yaml:"fio_binary" json:"fio_binary,omitempty"` + HoldOpenDuration Duration `yaml:"hold_open_duration" json:"-"` + KillAfter Duration `yaml:"kill_after" json:"-"` + CloneURL string `yaml:"clone_url" json:"clone_url,omitempty"` + GitBinary string `yaml:"git_binary" json:"git_binary,omitempty"` + GitTimeout Duration `yaml:"git_timeout" json:"-"` + GitUserName string `yaml:"git_user_name" json:"git_user_name,omitempty"` + GitUserEmail string `yaml:"git_user_email" json:"git_user_email,omitempty"` + CloneDir string `yaml:"clone_dir" json:"clone_dir,omitempty"` + Mutation MutationSpec `yaml:"mutation" json:"mutation,omitempty"` + CommitMessage string `yaml:"commit_message" json:"commit_message,omitempty"` + ExpectedLocks []string `yaml:"expected_locks" json:"expected_locks,omitempty"` + ExpectedStatus string `yaml:"expected_status" json:"expected_status,omitempty"` + ExpectedCommitDelta int `yaml:"expected_commit_delta" json:"expected_commit_delta,omitempty"` + RemountVerifyPaths []VerifySpec `yaml:"remount_verify_paths" json:"remount_verify_paths,omitempty"` + ExpectExit *int `yaml:"expect_exit" json:"expect_exit,omitempty"` + AllowNonzeroWhenNoAllowOther bool `yaml:"allow_nonzero_when_no_allow_other" json:"allow_nonzero_when_no_allow_other,omitempty"` +} + +type FileSpec struct { + RelativePath string `yaml:"relative_path" json:"relative_path"` + Content string `yaml:"content" json:"content"` +} + +type MutationSpec struct { + Path string `yaml:"path" json:"path"` + Mode string `yaml:"mode" json:"mode"` + Content string `yaml:"content" json:"content"` +} + +type VerifySpec struct { + Path string `yaml:"path" json:"path"` + SHA256 string `yaml:"sha256" json:"sha256"` +} + +type OracleSpec struct { + Type string `yaml:"type" json:"type"` + Path string `yaml:"path" json:"path,omitempty"` + ExpectedBytes int64 `yaml:"expected_bytes" json:"expected_bytes,omitempty"` + CommandID string `yaml:"command_id" json:"command_id,omitempty"` + RemotePath string `yaml:"remote_path" json:"remote_path,omitempty"` + SHA256 string `yaml:"sha256" json:"sha256,omitempty"` + Paths []string `yaml:"paths" json:"paths,omitempty"` + ExpectedHashes map[string]string `yaml:"expected_hashes" json:"expected_hashes,omitempty"` + Globs []string `yaml:"globs" json:"globs,omitempty"` + ExpectedStatus string `yaml:"expected_status" json:"expected_status,omitempty"` + RepoPath string `yaml:"repo_path" json:"repo_path,omitempty"` + ExpectedDelta int `yaml:"expected_delta" json:"expected_delta,omitempty"` + ExpectedExit *int `yaml:"expected_exit" json:"expected_exit,omitempty"` +} + +func LoadSuites(dir string, names []string) ([]Case, error) { + var all []Case + for _, name := range names { + path := filepath.Join(dir, name+".yaml") + cases, err := LoadFile(path) + if err != nil { + return nil, err + } + all = append(all, cases...) + } + return all, nil +} + +func LoadFile(path string) ([]Case, error) { + b, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("%w: read %s: %v", ErrParse, path, err) + } + var sf SuiteFile + if err := yaml.Unmarshal(b, &sf); err != nil { + return nil, fmt.Errorf("%w: unmarshal %s: %v", ErrParse, path, err) + } + if len(sf.Cases) == 0 { + return nil, fmt.Errorf("%w: %s has no cases", ErrValidation, path) + } + for i := range sf.Cases { + applyDefaults(&sf.Cases[i], sf.Defaults) + sf.Cases[i].SourceFile = path + if err := Validate(sf.Cases[i]); err != nil { + return nil, fmt.Errorf("%w: %s case %q: %v", ErrValidation, path, sf.Cases[i].ID, err) + } + } + return sf.Cases, nil +} + +func applyDefaults(c *Case, d Defaults) { + c.defaults = d + if c.Timeout.Duration == 0 { + c.Timeout = d.Timeout + } + if c.Cleanup == "" { + c.Cleanup = d.Cleanup + } + if c.RequiresFresh == nil { + c.RequiresFresh = d.RequiresFreshTenant + } +} + +func Validate(c Case) error { + if c.ID == "" { + return errors.New("id is required") + } + if !oneOf(c.Suite, "smoke", "regression", "stress", "fault") { + return fmt.Errorf("invalid suite %q", c.Suite) + } + if !oneOf(c.ExpectedOutcome, "baseline_pass", "bug_reproduced", "fix_verified") { + return fmt.Errorf("invalid expected_outcome %q", c.ExpectedOutcome) + } + if c.SyncMode != "" && !oneOf(c.SyncMode, "strict", "interactive", "auto") { + return fmt.Errorf("invalid sync_mode %q", c.SyncMode) + } + if c.RemoteRootSuffix == "" || c.MountpointSuffix == "" { + return errors.New("remote_root_suffix and mountpoint_suffix are required") + } + if c.Timeout.Duration <= 0 { + return errors.New("timeout must be > 0") + } + if !oneOf(c.Cleanup, "always", "retain_on_failure", "never") { + return fmt.Errorf("invalid cleanup %q", c.Cleanup) + } + if !oneOf(c.Severity.Failure, "P0", "P1", "P2", "P3") { + return fmt.Errorf("invalid failure severity %q", c.Severity.Failure) + } + if err := validateWorkload(c.Workload); err != nil { + return err + } + if len(c.Oracles) == 0 { + return errors.New("at least one oracle is required") + } + for _, o := range c.Oracles { + if err := validateOracle(c.Workload, o); err != nil { + return err + } + } + return nil +} + +func ValidateRelativeSlashPath(field, p string) error { + if p == "" { + return fmt.Errorf("%s is required", field) + } + if strings.Contains(p, "\\") { + return fmt.Errorf("%s %q must use slash separators", field, p) + } + if path.IsAbs(p) { + return fmt.Errorf("%s %q must be relative", field, p) + } + parts := strings.Split(p, "/") + for _, part := range parts { + if part == "" || part == "." || part == ".." { + return fmt.Errorf("%s %q contains unsafe path segment", field, p) + } + } + if clean := path.Clean(p); clean != p { + return fmt.Errorf("%s %q is not clean", field, p) + } + return nil +} + +func validateWorkload(w Workload) error { + switch w.Type { + case "mount_smoke": + if len(w.Files) == 0 { + return errors.New("mount_smoke requires files") + } + for _, f := range w.Files { + if err := ValidateRelativeSlashPath("mount_smoke file relative_path", f.RelativePath); err != nil { + return err + } + } + case "path_matrix": + if len(w.Paths) == 0 { + return errors.New("path_matrix requires paths") + } + for _, p := range w.Paths { + if err := ValidateRelativeSlashPath("path_matrix path", p); err != nil { + return err + } + } + case "dual_mount_visibility": + if len(w.Files) == 0 { + return errors.New("dual_mount_visibility requires files") + } + for _, f := range w.Files { + if err := ValidateRelativeSlashPath("dual_mount_visibility file relative_path", f.RelativePath); err != nil { + return err + } + } + case "fio": + if w.SizeBytes <= 0 { + return errors.New("fio requires size_bytes") + } + case "small_file_storm": + if w.FileCount <= 0 { + return errors.New("small_file_storm requires file_count") + } + case "parallel_writes": + if w.ParallelWriters <= 0 || w.WriterBytes <= 0 { + return errors.New("parallel_writes requires parallel_writers and writer_bytes") + } + case "git_workflow": + if w.CloneURL == "" || w.CloneDir == "" || w.Mutation.Path == "" || w.CommitMessage == "" { + return errors.New("git_workflow requires clone_url, clone_dir, mutation.path, and commit_message") + } + if err := ValidateRelativeSlashPath("git_workflow clone_dir", w.CloneDir); err != nil { + return err + } + if err := ValidateRelativeSlashPath("git_workflow mutation.path", w.Mutation.Path); err != nil { + return err + } + if !oneOf(w.Mutation.Mode, "append", "overwrite") { + return fmt.Errorf("invalid git mutation mode %q", w.Mutation.Mode) + } + if len(w.ExpectedLocks) != 0 { + return errors.New("git_workflow expected_locks must encode correct behavior and be empty in phase 1") + } + for _, p := range w.RemountVerifyPaths { + if err := ValidateRelativeSlashPath("git_workflow remount_verify_paths.path", p.Path); err != nil { + return err + } + } + case "doctor_fuse": + if w.ExpectExit == nil { + return errors.New("doctor_fuse requires expect_exit") + } + case "open_fd_unmount": + if w.HoldOpenDuration.Duration < 0 { + return errors.New("open_fd_unmount hold_open_duration must be >= 0") + } + case "kill_during_write": + if w.WriterBytes <= 0 || w.KillAfter.Duration <= 0 { + return errors.New("kill_during_write requires writer_bytes and kill_after") + } + default: + return fmt.Errorf("invalid workload type %q", w.Type) + } + return nil +} + +func validateOracle(w Workload, o OracleSpec) error { + if o.Path != "" { + if err := ValidateRelativeSlashPath("oracle path", o.Path); err != nil { + return err + } + } + if o.RemotePath != "" { + if err := ValidateRelativeSlashPath("oracle remote_path", o.RemotePath); err != nil { + return err + } + } + for _, p := range o.Paths { + if err := ValidateRelativeSlashPath("oracle paths", p); err != nil { + return err + } + } + for p := range o.ExpectedHashes { + if err := ValidateRelativeSlashPath("oracle expected_hashes path", p); err != nil { + return err + } + } + for _, p := range o.Globs { + if err := ValidateRelativeSlashPath("oracle globs", p); err != nil { + return err + } + } + if o.RepoPath != "" { + if err := ValidateRelativeSlashPath("oracle repo_path", o.RepoPath); err != nil { + return err + } + } + switch o.Type { + case "fuse_write_success": + if w.Type != "mount_smoke" && w.Type != "path_matrix" && w.Type != "parallel_writes" && o.Path == "" { + return errors.New("fuse_write_success needs path unless derived from workload") + } + case "cli_read_equals": + if w.Type != "mount_smoke" && w.Type != "path_matrix" && w.Type != "small_file_storm" && o.RemotePath == "" { + return errors.New("cli_read_equals needs remote_path unless derived from workload") + } + case "remount_hash_equal": + if w.Type == "git_workflow" && len(w.RemountVerifyPaths) == 0 && len(o.Paths) == 0 { + return errors.New("remount_hash_equal needs remount_verify_paths or explicit paths") + } + if w.Type != "mount_smoke" && w.Type != "path_matrix" && w.Type != "git_workflow" && + w.Type != "dual_mount_visibility" && w.Type != "small_file_storm" && w.Type != "parallel_writes" && len(o.Paths) == 0 { + return errors.New("remount_hash_equal needs paths unless derived from workload") + } + case "no_git_locks", "git_status_equals", "git_commit_count": + if w.Type != "git_workflow" { + return fmt.Errorf("%s requires git_workflow", o.Type) + } + case "dual_mount_visibility": + if w.Type != "dual_mount_visibility" { + return errors.New("dual_mount_visibility oracle requires dual_mount_visibility workload") + } + case "throughput_min": + if w.Type != "fio" && w.Type != "parallel_writes" { + return errors.New("throughput_min oracle requires fio or parallel_writes workload") + } + case "file_count_equals": + if w.Type != "small_file_storm" { + return errors.New("file_count_equals oracle requires small_file_storm workload") + } + case "unmount_busy_then_clean": + if w.Type != "open_fd_unmount" { + return errors.New("unmount_busy_then_clean oracle requires open_fd_unmount workload") + } + case "recovery_classified": + if w.Type != "kill_during_write" { + return errors.New("recovery_classified oracle requires kill_during_write workload") + } + case "command_exit": + if w.Type != "doctor_fuse" && o.CommandID == "" { + return errors.New("command_exit needs command_id unless derived from workload") + } + default: + return fmt.Errorf("invalid oracle type %q", o.Type) + } + return nil +} + +func oneOf(v string, allowed ...string) bool { + for _, a := range allowed { + if v == a { + return true + } + } + return false +} + +func FilterCases(cases []Case, selected string) []Case { + if strings.TrimSpace(selected) == "" { + return cases + } + want := map[string]bool{} + for _, s := range strings.Split(selected, ",") { + want[strings.TrimSpace(s)] = true + } + var out []Case + for _, c := range cases { + if want[c.ID] { + out = append(out, c) + } + } + return out +} + +func (c Case) RequiresFreshTenant() bool { + return c.RequiresFresh != nil && *c.RequiresFresh +} + +func (c Case) MountReadyTimeout() time.Duration { + if c.defaults.MountReadyTimeout.Duration > 0 { + return c.defaults.MountReadyTimeout.Duration + } + return 20 * time.Second +} + +func (c Case) RemoteVisibilityTimeout() time.Duration { + if c.Workload.ReadAfterWriteTimeout.Duration > 0 { + return c.Workload.ReadAfterWriteTimeout.Duration + } + if c.defaults.RemoteVisibilityTimeout.Duration > 0 { + return c.defaults.RemoteVisibilityTimeout.Duration + } + return 5 * time.Second +} + +func (c Case) CommandRetryCount() int { + if c.defaults.CommandRetryCount > 0 { + return c.defaults.CommandRetryCount + } + return 1 +} + +func (c Case) CommandRetrySleep() time.Duration { + if c.defaults.CommandRetrySleep.Duration > 0 { + return c.defaults.CommandRetrySleep.Duration + } + return 250 * time.Millisecond +} diff --git a/e2e/agent-harness/internal/casefile/casefile_test.go b/e2e/agent-harness/internal/casefile/casefile_test.go new file mode 100644 index 000000000..b3fc99d99 --- /dev/null +++ b/e2e/agent-harness/internal/casefile/casefile_test.go @@ -0,0 +1,212 @@ +package casefile + +import ( + "errors" + "os" + "path/filepath" + "testing" + "time" +) + +func TestLoadFileAppliesDefaults(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "smoke.yaml") + if err := os.WriteFile(path, []byte(` +defaults: + timeout: 2m + cleanup: retain_on_failure + mount_ready_timeout: 11s + remote_visibility_timeout: 12s + command_retry_count: 7 + command_retry_sleep: 2s +cases: + - id: smoke-strict + suite: smoke + sync_mode: strict + expected_outcome: baseline_pass + remote_root_suffix: smoke-strict + mountpoint_suffix: smoke-strict + workload: + type: mount_smoke + files: + - relative_path: a.txt + content: hello + oracles: + - type: cli_read_equals + severity: + failure: P1 +`), 0o644); err != nil { + t.Fatal(err) + } + cases, err := LoadFile(path) + if err != nil { + t.Fatal(err) + } + if len(cases) != 1 { + t.Fatalf("case count = %d, want 1", len(cases)) + } + if got := cases[0].Timeout.String(); got != "2m0s" { + t.Fatalf("timeout = %s", got) + } + if got := cases[0].Cleanup; got != "retain_on_failure" { + t.Fatalf("cleanup = %q", got) + } + if got := cases[0].MountReadyTimeout(); got != 11*time.Second { + t.Fatalf("mount ready timeout = %s", got) + } + if got := cases[0].RemoteVisibilityTimeout(); got != 12*time.Second { + t.Fatalf("remote visibility timeout = %s", got) + } + if got := cases[0].CommandRetryCount(); got != 7 { + t.Fatalf("retry count = %d", got) + } + if got := cases[0].CommandRetrySleep(); got != 2*time.Second { + t.Fatalf("retry sleep = %s", got) + } +} + +func TestLoadFileRejectsGitExpectedLocks(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "regression.yaml") + if err := os.WriteFile(path, []byte(` +defaults: + timeout: 2m + cleanup: retain_on_failure +cases: + - id: git-lock + suite: regression + sync_mode: strict + expected_outcome: bug_reproduced + remote_root_suffix: git-lock + mountpoint_suffix: git-lock + workload: + type: git_workflow + clone_url: https://example.invalid/repo.git + clone_dir: repo + mutation: + path: README.md + mode: append + content: hello + commit_message: test + expected_locks: + - .git/index.lock + expected_commit_delta: 1 + oracles: + - type: no_git_locks + severity: + failure: P1 +`), 0o644); err != nil { + t.Fatal(err) + } + _, err := LoadFile(path) + if !errors.Is(err, ErrValidation) { + t.Fatalf("err = %v, want ErrValidation", err) + } +} + +func TestValidateRejectsUnsafeRelativePaths(t *testing.T) { + expectExit := 0 + tests := []struct { + name string + c Case + }{ + { + name: "mount smoke file dotdot", + c: baseCase("mount_smoke", Workload{Type: "mount_smoke", Files: []FileSpec{{RelativePath: "../x", Content: "x"}}}, []OracleSpec{{Type: "cli_read_equals"}}), + }, + { + name: "path matrix absolute", + c: baseCase("path_matrix", Workload{Type: "path_matrix", Paths: []string{"/abs"}}, []OracleSpec{{Type: "cli_read_equals"}}), + }, + { + name: "git clone dir dotdot", + c: baseCase("git_workflow", Workload{ + Type: "git_workflow", + CloneURL: "fixture://basic", + CloneDir: "../repo", + Mutation: MutationSpec{Path: "README.md", Mode: "append", Content: "x"}, + CommitMessage: "test", + }, []OracleSpec{{Type: "no_git_locks"}}), + }, + { + name: "git mutation backslash", + c: baseCase("git_workflow", Workload{ + Type: "git_workflow", + CloneURL: "fixture://basic", + CloneDir: "repo", + Mutation: MutationSpec{Path: `dir\README.md`, Mode: "append", Content: "x"}, + CommitMessage: "test", + }, []OracleSpec{{Type: "no_git_locks"}}), + }, + { + name: "git remount verify dotdot", + c: baseCase("git_workflow", Workload{ + Type: "git_workflow", + CloneURL: "fixture://basic", + CloneDir: "repo", + Mutation: MutationSpec{Path: "README.md", Mode: "append", Content: "x"}, + CommitMessage: "test", + RemountVerifyPaths: []VerifySpec{{Path: "../README.md"}}, + }, []OracleSpec{{Type: "remount_hash_equal"}}), + }, + { + name: "oracle path dotdot", + c: baseCase("doctor_fuse", Workload{Type: "doctor_fuse", ExpectExit: &expectExit}, []OracleSpec{{Type: "command_exit", CommandID: "doctor", Path: "../x"}}), + }, + { + name: "oracle remote absolute", + c: baseCase("mount_smoke", Workload{Type: "mount_smoke", Files: []FileSpec{{RelativePath: "x", Content: "x"}}}, []OracleSpec{{Type: "cli_read_equals", RemotePath: "/x"}}), + }, + { + name: "oracle expected hash dotdot", + c: baseCase("mount_smoke", Workload{Type: "mount_smoke", Files: []FileSpec{{RelativePath: "x", Content: "x"}}}, []OracleSpec{{Type: "remount_hash_equal", ExpectedHashes: map[string]string{"../x": "abc"}}}), + }, + { + name: "oracle glob dotdot", + c: baseCase("git_workflow", Workload{ + Type: "git_workflow", + CloneURL: "fixture://basic", + CloneDir: "repo", + Mutation: MutationSpec{Path: "README.md", Mode: "append", Content: "x"}, + CommitMessage: "test", + }, []OracleSpec{{Type: "no_git_locks", Globs: []string{"../*.lock"}}}), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if err := Validate(tt.c); err == nil { + t.Fatal("expected validation error") + } + }) + } +} + +func baseCase(workloadType string, workload Workload, oracles []OracleSpec) Case { + return Case{ + ID: "case-" + workloadType, + Suite: "regression", + ExpectedOutcome: "baseline_pass", + RemoteRootSuffix: "case-" + workloadType, + MountpointSuffix: "case-" + workloadType, + Timeout: Duration{Duration: 2 * time.Minute}, + Cleanup: "always", + Workload: workload, + Oracles: oracles, + Severity: SeveritySpec{Failure: "P1"}, + } +} + +func TestLoadRepositorySuites(t *testing.T) { + cases, err := LoadSuites(filepath.Join("..", "..", "cases"), []string{"smoke", "regression", "stress", "fault"}) + if err != nil { + t.Fatal(err) + } + if len(cases) != 13 { + t.Fatalf("case count = %d, want 13", len(cases)) + } + for _, c := range cases { + if c.Workload.Type == "git_workflow" && len(c.Workload.ExpectedLocks) != 0 { + t.Fatalf("%s expected_locks must be empty", c.ID) + } + } +} diff --git a/e2e/agent-harness/internal/mountproc/mountproc.go b/e2e/agent-harness/internal/mountproc/mountproc.go new file mode 100644 index 000000000..72d04d458 --- /dev/null +++ b/e2e/agent-harness/internal/mountproc/mountproc.go @@ -0,0 +1,140 @@ +// Package mountproc runs harness commands and tracks mount processes. +package mountproc + +import ( + "bytes" + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "time" +) + +var ( + ErrMountTimeout = errors.New("mount startup timeout") + ErrUnmountTimeout = errors.New("unmount timeout") + ErrUnexpectedProcessExit = errors.New("unexpected process exit") +) + +type Result struct { + ID string + Args []string + Stdout string + Stderr string + ExitCode int + Duration time.Duration + Err error +} + +type Env struct { + Server string + APIKey string +} + +type Mount struct { + ID string + Cmd *exec.Cmd + Mountpoint string + LogPath string + ProcessGroup int +} + +func Run(ctx context.Context, id string, env Env, dir string, name string, args ...string) Result { + start := time.Now() + cmd := exec.CommandContext(ctx, name, args...) + cmd.Dir = dir + cmd.Env = withEnv(os.Environ(), env) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + err := cmd.Run() + exit := 0 + if err != nil { + exit = 1 + var ee *exec.ExitError + if errors.As(err, &ee) { + exit = ee.ExitCode() + } + } + return Result{ID: id, Args: append([]string{name}, args...), Stdout: stdout.String(), Stderr: stderr.String(), ExitCode: exit, Duration: time.Since(start), Err: err} +} + +func StartMount(ctx context.Context, id string, env Env, drive9Bin, remoteRoot, mountpoint, syncMode, logPath string) (*Mount, error) { + if err := os.MkdirAll(mountpoint, 0o755); err != nil { + return nil, err + } + if err := os.MkdirAll(filepath.Dir(logPath), 0o755); err != nil { + return nil, err + } + logFile, err := os.OpenFile(logPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return nil, err + } + args := []string{"mount", "--mode", "fuse", "--perf-counters"} + if syncMode != "" { + args = append(args, "--sync-mode", syncMode) + } + args = append(args, ":"+remoteRoot, mountpoint) + cmd := exec.CommandContext(ctx, drive9Bin, args...) + cmd.Env = withEnv(os.Environ(), env) + cmd.Stdout = logFile + cmd.Stderr = logFile + setProcessGroup(cmd) + if err := cmd.Start(); err != nil { + _ = logFile.Close() + return nil, err + } + pgid := processGroupID(cmd) + go func() { + _ = cmd.Wait() + _ = logFile.Close() + }() + return &Mount{ID: id, Cmd: cmd, Mountpoint: mountpoint, LogPath: logPath, ProcessGroup: pgid}, nil +} + +func WaitMounted(ctx context.Context, mountpoint string, check func(string) (bool, error)) error { + ticker := time.NewTicker(250 * time.Millisecond) + defer ticker.Stop() + var lastErr error + for { + mounted, err := check(mountpoint) + if err == nil && mounted { + return nil + } + if err != nil { + lastErr = err + } + select { + case <-ctx.Done(): + if lastErr != nil { + return fmt.Errorf("%w: %s: %v", ErrMountTimeout, mountpoint, lastErr) + } + return fmt.Errorf("%w: %s", ErrMountTimeout, mountpoint) + case <-ticker.C: + } + } +} + +func Stop(ctx context.Context, env Env, drive9Bin, mountpoint string) Result { + return Run(ctx, "umount", env, "", drive9Bin, "umount", "--timeout", "30s", mountpoint) +} + +func KillMount(m *Mount) error { + if m == nil || m.Cmd == nil || m.Cmd.Process == nil { + return nil + } + return killProcessGroup(m) +} + +func withEnv(base []string, env Env) []string { + out := append([]string{}, base...) + if env.Server != "" { + out = append(out, "DRIVE9_SERVER="+env.Server) + } + if env.APIKey != "" { + out = append(out, "DRIVE9_API_KEY="+env.APIKey) + } + return out +} diff --git a/e2e/agent-harness/internal/mountproc/mountproc_test.go b/e2e/agent-harness/internal/mountproc/mountproc_test.go new file mode 100644 index 000000000..a6d77ec39 --- /dev/null +++ b/e2e/agent-harness/internal/mountproc/mountproc_test.go @@ -0,0 +1,23 @@ +package mountproc + +import ( + "context" + "errors" + "strings" + "testing" + "time" +) + +func TestWaitMountedReturnsLastCheckError(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), time.Millisecond) + defer cancel() + err := WaitMounted(ctx, "/mnt/test", func(string) (bool, error) { + return false, errors.New("mount table unavailable") + }) + if !errors.Is(err, ErrMountTimeout) { + t.Fatalf("err = %v, want ErrMountTimeout", err) + } + if !strings.Contains(err.Error(), "mount table unavailable") { + t.Fatalf("err = %v, want last check error", err) + } +} diff --git a/e2e/agent-harness/internal/mountproc/proc_unix.go b/e2e/agent-harness/internal/mountproc/proc_unix.go new file mode 100644 index 000000000..8b40982b4 --- /dev/null +++ b/e2e/agent-harness/internal/mountproc/proc_unix.go @@ -0,0 +1,31 @@ +//go:build !windows + +package mountproc + +import ( + "os/exec" + "syscall" +) + +func setProcessGroup(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} +} + +func processGroupID(cmd *exec.Cmd) int { + if cmd == nil || cmd.Process == nil { + return 0 + } + pgid, err := syscall.Getpgid(cmd.Process.Pid) + if err != nil { + return cmd.Process.Pid + } + return pgid +} + +func killProcessGroup(m *Mount) error { + if m.ProcessGroup > 0 { + return syscall.Kill(-m.ProcessGroup, syscall.SIGTERM) + } + return m.Cmd.Process.Kill() +} + diff --git a/e2e/agent-harness/internal/mountproc/proc_windows.go b/e2e/agent-harness/internal/mountproc/proc_windows.go new file mode 100644 index 000000000..953ad08a4 --- /dev/null +++ b/e2e/agent-harness/internal/mountproc/proc_windows.go @@ -0,0 +1,19 @@ +//go:build windows + +package mountproc + +import "os/exec" + +func setProcessGroup(_ *exec.Cmd) {} + +func processGroupID(cmd *exec.Cmd) int { + if cmd == nil || cmd.Process == nil { + return 0 + } + return cmd.Process.Pid +} + +func killProcessGroup(m *Mount) error { + return m.Cmd.Process.Kill() +} + diff --git a/e2e/agent-harness/internal/report/report.go b/e2e/agent-harness/internal/report/report.go new file mode 100644 index 000000000..0d3145de6 --- /dev/null +++ b/e2e/agent-harness/internal/report/report.go @@ -0,0 +1,386 @@ +// Package report writes and regenerates drive9 agent harness artifacts. +package report + +import ( + "bufio" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "time" +) + +const ( + SchemaVersion = "agent-harness.v1" + jsonlInitialBuffer = 64 * 1024 + jsonlMaxTokenSize = 4 * 1024 * 1024 +) + +type Manifest struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + StartedAt string `json:"started_at"` + Host string `json:"host"` + HarnessGitSHA string `json:"harness_git_sha"` + Drive9Version string `json:"drive9_version"` + Server string `json:"server"` + Suites []string `json:"suites"` + SelectedCases []string `json:"selected_cases"` + Cases []CaseSummary `json:"cases"` + ArtifactRoot string `json:"artifact_root"` + MountRoot string `json:"mount_root"` + RemoteRootBase string `json:"remote_root_base"` + GeneratedMountpoints map[string]string `json:"generated_mountpoints"` + GeneratedRemoteRoots map[string]string `json:"generated_remote_roots"` + ProcessGroups map[string]int `json:"process_groups"` + APIKeyRedacted string `json:"api_key_redacted"` + ApprovalMode string `json:"approval_mode"` +} + +type Event struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + CaseID string `json:"case_id"` + TS string `json:"ts"` + Type string `json:"type"` + Message string `json:"message"` + DurationMS int64 `json:"duration_ms"` + CommandID string `json:"command_id"` + ExitCode *int `json:"exit_code"` + Signal string `json:"signal"` + ArtifactRefs []string `json:"artifact_refs"` +} + +type Failure struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + CaseID string `json:"case_id"` + TS string `json:"ts"` + Severity string `json:"severity"` + Class string `json:"class"` + Oracle string `json:"oracle"` + ExpectedOutcome string `json:"expected_outcome"` + Message string `json:"message"` + Observed any `json:"observed"` + Expected any `json:"expected"` + ReproPath string `json:"repro_path"` + ArtifactRefs []string `json:"artifact_refs"` +} + +type Metric struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + CaseID string `json:"case_id"` + TS string `json:"ts"` + Name string `json:"name"` + Value float64 `json:"value"` + Unit string `json:"unit"` + Source string `json:"source"` + ArtifactRefs []string `json:"artifact_refs"` +} + +type Summary struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + Status string `json:"status"` + StartedAt string `json:"started_at"` + EndedAt string `json:"ended_at"` + DurationMS int64 `json:"duration_ms"` + Cases []CaseSummary `json:"cases"` + Counts SummaryCounts `json:"counts"` + Artifacts map[string]string `json:"artifacts"` + Cleanup map[string]string `json:"cleanup"` +} + +type CaseSummary struct { + ID string `json:"id"` + Suite string `json:"suite"` + ExpectedOutcome string `json:"expected_outcome"` + Status string `json:"status"` +} + +type SummaryCounts struct { + BySuite map[string]int `json:"by_suite"` + ByExpectedOutcome map[string]int `json:"by_expected_outcome"` + BySeverity map[string]int `json:"by_severity"` + ByClass map[string]int `json:"by_class"` +} + +type Gating struct { + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + GateStatus string `json:"gate_status"` + Pass int `json:"pass"` + Fail int `json:"fail"` + KnownBugReproduced int `json:"known_bug_reproduced"` + KnownBugFixedCandidate int `json:"known_bug_fixed_candidate"` + NonGating int `json:"non_gating"` + BlockingFailures []Failure `json:"blocking_failures"` +} + +type Recorder struct { + RunDir string + RunID string +} + +func NewRecorder(runDir, runID string) (*Recorder, error) { + for _, dir := range []string{runDir, filepath.Join(runDir, "mount"), filepath.Join(runDir, "debug"), filepath.Join(runDir, "metrics"), filepath.Join(runDir, "repro")} { + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, err + } + } + return &Recorder{RunDir: runDir, RunID: runID}, nil +} + +func (r *Recorder) WriteManifest(m Manifest) error { + m.SchemaVersion = SchemaVersion + return writeJSON(filepath.Join(r.RunDir, "manifest.json"), m) +} + +func (r *Recorder) Event(e Event) error { + e.SchemaVersion = SchemaVersion + e.RunID = r.RunID + if e.TS == "" { + e.TS = now() + } + return appendJSONL(filepath.Join(r.RunDir, "events.jsonl"), e) +} + +func (r *Recorder) Failure(f Failure) error { + f.SchemaVersion = SchemaVersion + f.RunID = r.RunID + if f.TS == "" { + f.TS = now() + } + return appendJSONL(filepath.Join(r.RunDir, "failures.jsonl"), f) +} + +func (r *Recorder) Metric(m Metric) error { + m.SchemaVersion = SchemaVersion + m.RunID = r.RunID + if m.TS == "" { + m.TS = now() + } + return appendJSONL(filepath.Join(r.RunDir, "metrics.jsonl"), m) +} + +func Generate(runDir string) (Summary, Gating, error) { + manifest := Manifest{} + if err := readJSON(filepath.Join(runDir, "manifest.json"), &manifest); err != nil { + return Summary{}, Gating{}, err + } + failures, err := readFailures(filepath.Join(runDir, "failures.jsonl")) + if err != nil { + return Summary{}, Gating{}, err + } + ended := now() + if runEnd, err := readRunEnd(filepath.Join(runDir, "events.jsonl")); err != nil { + return Summary{}, Gating{}, err + } else if runEnd != "" { + ended = runEnd + } + var durationMS int64 + if manifest.StartedAt != "" { + if startedAt, err := time.Parse(time.RFC3339Nano, manifest.StartedAt); err == nil { + if endedAt, err := time.Parse(time.RFC3339Nano, ended); err == nil { + durationMS = endedAt.Sub(startedAt).Milliseconds() + } + } + } + summary := Summary{ + SchemaVersion: SchemaVersion, + RunID: manifest.RunID, + Status: "passed", + StartedAt: manifest.StartedAt, + EndedAt: ended, + DurationMS: durationMS, + Counts: SummaryCounts{ + BySuite: map[string]int{}, + ByExpectedOutcome: map[string]int{}, + BySeverity: map[string]int{}, + ByClass: map[string]int{}, + }, + Artifacts: map[string]string{ + "manifest": "manifest.json", + "events": "events.jsonl", + "failures": "failures.jsonl", + "metrics": "metrics.jsonl", + "summary": "summary.json", + "gating": "gating.json", + }, + Cleanup: map[string]string{}, + } + gating := Gating{SchemaVersion: SchemaVersion, RunID: manifest.RunID, GateStatus: "pass"} + selected := map[string]CaseSummary{} + for _, cs := range manifest.Cases { + cs.Status = "passed" + selected[cs.ID] = cs + } + for _, id := range manifest.SelectedCases { + if _, ok := selected[id]; !ok { + selected[id] = CaseSummary{ID: id, Status: "passed"} + } + } + knownBugCases := map[string]bool{} + failedCases := map[string]bool{} + inconclusiveCases := map[string]bool{} + for _, f := range failures { + summary.Counts.BySeverity[f.Severity]++ + summary.Counts.ByClass[f.Class]++ + cs := selected[f.CaseID] + cs.ID = f.CaseID + cs.ExpectedOutcome = f.ExpectedOutcome + if f.Class == "inconclusive" { + cs.Status = "inconclusive" + if !inconclusiveCases[f.CaseID] { + gating.NonGating++ + inconclusiveCases[f.CaseID] = true + } + selected[f.CaseID] = cs + continue + } + if f.ExpectedOutcome == "bug_reproduced" && f.Class == "product" { + cs.Status = "known_bug_reproduced" + if !knownBugCases[f.CaseID] { + gating.KnownBugReproduced++ + gating.NonGating++ + knownBugCases[f.CaseID] = true + } + selected[f.CaseID] = cs + continue + } + cs.Status = "failed" + if !failedCases[f.CaseID] { + gating.Fail++ + failedCases[f.CaseID] = true + } + gating.BlockingFailures = append(gating.BlockingFailures, f) + selected[f.CaseID] = cs + } + for _, cs := range selected { + if cs.Status == "" { + cs.Status = "passed" + } + if cs.Status == "passed" && cs.ExpectedOutcome == "bug_reproduced" { + cs.Status = "known_bug_fixed_candidate" + gating.KnownBugFixedCandidate++ + } + if cs.Status == "passed" { + gating.Pass++ + } + summary.Cases = append(summary.Cases, cs) + summary.Counts.BySuite[cs.Suite]++ + summary.Counts.ByExpectedOutcome[cs.ExpectedOutcome]++ + } + sort.Slice(summary.Cases, func(i, j int) bool { + return summary.Cases[i].ID < summary.Cases[j].ID + }) + if gating.Fail > 0 { + gating.GateStatus = "fail" + summary.Status = "failed" + } else if len(inconclusiveCases) > 0 { + gating.GateStatus = "non_gating" + summary.Status = "inconclusive" + } else if gating.KnownBugReproduced > 0 { + gating.GateStatus = "non_gating" + summary.Status = "known_bugs_reproduced" + } + if err := writeJSON(filepath.Join(runDir, "summary.json"), summary); err != nil { + return Summary{}, Gating{}, err + } + if err := writeJSON(filepath.Join(runDir, "gating.json"), gating); err != nil { + return Summary{}, Gating{}, err + } + if err := writeMarkdown(filepath.Join(runDir, "summary.md"), summary, gating); err != nil { + return Summary{}, Gating{}, err + } + return summary, gating, nil +} + +func readFailures(path string) ([]Failure, error) { + f, err := os.Open(path) + if os.IsNotExist(err) { + return nil, nil + } + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + var out []Failure + scanner := bufio.NewScanner(f) + scanner.Buffer(make([]byte, 0, jsonlInitialBuffer), jsonlMaxTokenSize) + for scanner.Scan() { + var failure Failure + if err := json.Unmarshal(scanner.Bytes(), &failure); err != nil { + return nil, err + } + out = append(out, failure) + } + return out, scanner.Err() +} + +func readRunEnd(path string) (string, error) { + f, err := os.Open(path) + if os.IsNotExist(err) { + return "", nil + } + if err != nil { + return "", err + } + defer func() { _ = f.Close() }() + var ended string + scanner := bufio.NewScanner(f) + scanner.Buffer(make([]byte, 0, jsonlInitialBuffer), jsonlMaxTokenSize) + for scanner.Scan() { + var event Event + if err := json.Unmarshal(scanner.Bytes(), &event); err != nil { + return "", err + } + if event.Type == "run_end" && event.TS != "" { + ended = event.TS + } + } + return ended, scanner.Err() +} + +func appendJSONL(path string, v any) error { + f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer func() { _ = f.Close() }() + b, err := json.Marshal(v) + if err != nil { + return err + } + if _, err := f.Write(append(b, '\n')); err != nil { + return err + } + return nil +} + +func writeJSON(path string, v any) error { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, append(b, '\n'), 0o644) +} + +func readJSON(path string, v any) error { + b, err := os.ReadFile(path) + if err != nil { + return err + } + return json.Unmarshal(b, v) +} + +func writeMarkdown(path string, s Summary, g Gating) error { + body := fmt.Sprintf("---\ntitle: Drive9 Agent Harness Summary\n---\n\n## Run\n\n1. Run ID: `%s`\n2. Status: `%s`\n3. Gate: `%s`\n\n## Counts\n\n1. Pass: %d\n2. Fail: %d\n3. Known bug reproduced: %d\n4. Known bug fixed candidate: %d\n", s.RunID, s.Status, g.GateStatus, g.Pass, g.Fail, g.KnownBugReproduced, g.KnownBugFixedCandidate) + return os.WriteFile(path, []byte(body), 0o644) +} + +func now() string { + return time.Now().UTC().Format(time.RFC3339Nano) +} diff --git a/e2e/agent-harness/internal/report/report_test.go b/e2e/agent-harness/internal/report/report_test.go new file mode 100644 index 000000000..d9866b470 --- /dev/null +++ b/e2e/agent-harness/internal/report/report_test.go @@ -0,0 +1,112 @@ +package report + +import ( + "strings" + "testing" +) + +func TestGenerateKnownBugNonGating(t *testing.T) { + dir := t.TempDir() + rec, err := NewRecorder(dir, "run1") + if err != nil { + t.Fatal(err) + } + if err := rec.WriteManifest(Manifest{RunID: "run1", SelectedCases: []string{"case1"}}); err != nil { + t.Fatal(err) + } + if err := rec.Failure(Failure{CaseID: "case1", Class: "product", Severity: "P1", ExpectedOutcome: "bug_reproduced", Oracle: "cli_read_equals"}); err != nil { + t.Fatal(err) + } + _, gating, err := Generate(dir) + if err != nil { + t.Fatal(err) + } + if gating.GateStatus != "non_gating" { + t.Fatalf("gate = %q", gating.GateStatus) + } + if gating.KnownBugReproduced != 1 || gating.Fail != 0 { + t.Fatalf("gating = %+v", gating) + } +} + +func TestGenerateInconclusiveNonGating(t *testing.T) { + dir := t.TempDir() + rec, err := NewRecorder(dir, "run1") + if err != nil { + t.Fatal(err) + } + if err := rec.WriteManifest(Manifest{RunID: "run1", SelectedCases: []string{"case1"}, Cases: []CaseSummary{{ID: "case1", Suite: "fault", ExpectedOutcome: "baseline_pass"}}}); err != nil { + t.Fatal(err) + } + if err := rec.Failure(Failure{CaseID: "case1", Class: "inconclusive", Severity: "P2", ExpectedOutcome: "baseline_pass", Oracle: "recovery_classified"}); err != nil { + t.Fatal(err) + } + summary, gating, err := Generate(dir) + if err != nil { + t.Fatal(err) + } + if gating.GateStatus != "non_gating" || summary.Status != "inconclusive" { + t.Fatalf("summary = %+v gating = %+v", summary, gating) + } + if gating.Fail != 0 || gating.NonGating != 1 { + t.Fatalf("gating = %+v", gating) + } +} + +func TestGenerateUsesPersistedRunEnd(t *testing.T) { + dir := t.TempDir() + rec, err := NewRecorder(dir, "run1") + if err != nil { + t.Fatal(err) + } + if err := rec.WriteManifest(Manifest{ + RunID: "run1", + StartedAt: "2026-01-01T00:00:00Z", + SelectedCases: []string{"case1"}, + Cases: []CaseSummary{{ID: "case1", Suite: "smoke", ExpectedOutcome: "baseline_pass"}}, + }); err != nil { + t.Fatal(err) + } + if err := rec.Event(Event{Type: "run_end", TS: "2026-01-01T00:00:05Z"}); err != nil { + t.Fatal(err) + } + summary, _, err := Generate(dir) + if err != nil { + t.Fatal(err) + } + if summary.EndedAt != "2026-01-01T00:00:05Z" || summary.DurationMS != 5000 { + t.Fatalf("summary ended_at=%q duration=%d, want persisted run end", summary.EndedAt, summary.DurationMS) + } +} + +func TestGenerateReadsLargeFailureEntries(t *testing.T) { + dir := t.TempDir() + rec, err := NewRecorder(dir, "run1") + if err != nil { + t.Fatal(err) + } + if err := rec.WriteManifest(Manifest{ + RunID: "run1", + SelectedCases: []string{"case1"}, + Cases: []CaseSummary{{ID: "case1", Suite: "smoke", ExpectedOutcome: "baseline_pass"}}, + }); err != nil { + t.Fatal(err) + } + if err := rec.Failure(Failure{ + CaseID: "case1", + Class: "product", + Severity: "P1", + ExpectedOutcome: "baseline_pass", + Oracle: "cli_read_equals", + Message: strings.Repeat("x", 128*1024), + }); err != nil { + t.Fatal(err) + } + _, gating, err := Generate(dir) + if err != nil { + t.Fatal(err) + } + if gating.Fail != 1 { + t.Fatalf("gating fail = %d, want 1", gating.Fail) + } +} diff --git a/e2e/agent-harness/internal/runner/runner.go b/e2e/agent-harness/internal/runner/runner.go new file mode 100644 index 000000000..ef0527adf --- /dev/null +++ b/e2e/agent-harness/internal/runner/runner.go @@ -0,0 +1,1627 @@ +// Package runner executes drive9 agent harness cases. +package runner + +import ( + "bufio" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path" + "path/filepath" + "runtime" + "strings" + "sync" + "time" + + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/casefile" + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/mountproc" + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/report" + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/safety" +) + +type Config struct { + ArtifactRoot string + MountRoot string + RemoteRootBase string + Drive9Bin string + Server string + APIKey string + Provision bool + ProvisionTimeout time.Duration + SuiteDir string + Suites []string + CaseFilter string + AllowFault bool +} + +type GCConfig struct { + RunDir string + Drive9Bin string + Server string + APIKey string + SuccessfulOnly bool + ConfirmDelete bool +} + +type EvidenceConfig struct { + RunDir string + KubeContext string + Namespace string + Selector string + Since string + Tail int + MetricsRawPath string + ApproveExternal bool +} + +var ( + ErrConfirmDeleteRequired = errors.New("gc requires --confirm-delete") + ErrGateFailed = errors.New("harness gate failed") + ErrUnsafeMountpoint = errors.New("unsafe mountpoint") + + harnessHTTPClient = &http.Client{Timeout: 30 * time.Second} +) + +func DefaultConfig() Config { + return Config{ + ArtifactRoot: "/tmp", + MountRoot: "/tmp", + RemoteRootBase: "/agent-adversarial-$RUN_ID", + Drive9Bin: "drive9", + Server: getenv("DRIVE9_BASE", "http://127.0.0.1:9009"), + APIKey: os.Getenv("DRIVE9_API_KEY"), + ProvisionTimeout: 120 * time.Second, + SuiteDir: filepath.Join("e2e", "agent-harness", "cases"), + Suites: []string{"smoke"}, + } +} + +func Preflight(ctx context.Context, cfg Config) error { + if _, err := resolveBin(cfg.Drive9Bin); err != nil { + return err + } + if _, err := safety.ValidateRoot("artifact-root", cfg.ArtifactRoot); err != nil { + return err + } + if _, err := safety.ValidateRoot("mount-root", cfg.MountRoot); err != nil { + return err + } + if _, err := safety.ValidateRoot("remote-root-base", cfg.RemoteRootBase); err != nil { + return err + } + if cfg.APIKey == "" && !cfg.Provision { + return errors.New("preflight: --api-key or --provision is required") + } + if cfg.APIKey != "" && !cfg.Provision { + if err := checkStatus(ctx, cfg.Server, cfg.APIKey); err != nil { + return err + } + } + return nil +} + +func Run(ctx context.Context, cfg Config) (string, error) { + id := runID() + if cfg.RemoteRootBase == "" { + cfg.RemoteRootBase = "/agent-adversarial-$RUN_ID" + } + if strings.Contains(cfg.RemoteRootBase, "$RUN_ID") { + cfg.RemoteRootBase = strings.ReplaceAll(cfg.RemoteRootBase, "$RUN_ID", id) + } + artifactRoot, err := safety.ValidateRoot("artifact-root", cfg.ArtifactRoot) + if err != nil { + return "", err + } + mountRoot, err := safety.ValidateRoot("mount-root", cfg.MountRoot) + if err != nil { + return "", err + } + remoteRootBase, err := safety.ValidateRoot("remote-root-base", cfg.RemoteRootBase) + if err != nil { + return "", err + } + drive9Bin, err := resolveBin(cfg.Drive9Bin) + if err != nil { + return "", err + } + cases, err := casefile.LoadSuites(cfg.SuiteDir, cfg.Suites) + if err != nil { + return "", err + } + cases = casefile.FilterCases(cases, cfg.CaseFilter) + if len(cases) == 0 { + return "", errors.New("no selected cases") + } + if containsSuite(cases, "fault") && !cfg.AllowFault { + return "", errors.New("fault suite requires --allow-fault") + } + if requiresFreshTenant(cases) && cfg.APIKey == "" && !cfg.Provision { + return "", errors.New("selected cases require a fresh tenant; pass --provision or provide --api-key") + } + apiKey := cfg.APIKey + if apiKey == "" || cfg.Provision { + key, err := provision(ctx, cfg.Server, cfg.ProvisionTimeout) + if err != nil { + return "", err + } + apiKey = key + } + runDir := filepath.Join(artifactRoot, "drive9-agent-test-"+id) + rec, err := report.NewRecorder(runDir, id) + if err != nil { + return "", err + } + _ = captureDebug(ctx, rec, "run-start") + manifest := report.Manifest{ + RunID: id, + StartedAt: now(), + Host: host(), + HarnessGitSHA: gitSHA(), + Drive9Version: drive9Version(ctx, drive9Bin), + Server: cfg.Server, + Suites: cfg.Suites, + SelectedCases: caseIDs(cases), + Cases: caseSummaries(cases), + ArtifactRoot: artifactRoot, + MountRoot: mountRoot, + RemoteRootBase: remoteRootBase, + GeneratedMountpoints: map[string]string{}, + GeneratedRemoteRoots: map[string]string{}, + ProcessGroups: map[string]int{}, + APIKeyRedacted: redact(apiKey), + ApprovalMode: "phase1-local", + } + _ = rec.Event(report.Event{Type: "run_start"}) + env := mountproc.Env{Server: cfg.Server, APIKey: apiKey} + for _, c := range cases { + if err := runCase(ctx, rec, &manifest, env, drive9Bin, remoteRootBase, mountRoot, id, c); err != nil { + _ = rec.Failure(report.Failure{CaseID: c.ID, Severity: c.Severity.Failure, Class: "harness", Oracle: "runner", ExpectedOutcome: c.ExpectedOutcome, Message: err.Error()}) + } + } + _ = captureDebug(ctx, rec, "run-end") + _ = rec.Event(report.Event{Type: "run_end"}) + if err := rec.WriteManifest(manifest); err != nil { + return "", err + } + _, gating, err := report.Generate(runDir) + if err != nil { + return runDir, err + } + return runDir, gateError(runDir, gating) +} + +func Regenerate(runDir string) error { + _, _, err := report.Generate(runDir) + return err +} + +func gateError(runDir string, gating report.Gating) error { + if gating.GateStatus != "fail" && gating.GateStatus != "harness_failed" { + return nil + } + return fmt.Errorf("%w: status=%s fail=%d run_dir=%s", ErrGateFailed, gating.GateStatus, gating.Fail, runDir) +} + +func runCase(ctx context.Context, rec *report.Recorder, manifest *report.Manifest, env mountproc.Env, drive9Bin, remoteRootBase, mountRoot, runID string, c casefile.Case) (retErr error) { + caseCtx, cancelCase := context.WithTimeout(ctx, c.Timeout.Duration) + defer cancelCase() + _ = rec.Event(report.Event{CaseID: c.ID, Type: "case_start"}) + caseRemote, err := safety.CaseRemoteRoot(remoteRootBase, c.RemoteRootSuffix) + if err != nil { + return err + } + mountpoint, err := safety.Mountpoint(mountRoot, runID, c.MountpointSuffix) + if err != nil { + return err + } + if err := safety.ValidateMountpointAvailable(mountpoint); err != nil { + return err + } + manifest.GeneratedRemoteRoots[c.ID] = caseRemote + manifest.GeneratedMountpoints[c.ID] = mountpoint + + if c.Workload.Type == "doctor_fuse" { + if err := os.MkdirAll(mountpoint, 0o755); err != nil { + return err + } + if err := runDoctor(caseCtx, rec, env, drive9Bin, mountpoint, c); err != nil { + return err + } + _ = rec.Event(report.Event{CaseID: c.ID, Type: "case_end"}) + if shouldRemoveMountpoint(c, rec, retErr) { + _ = os.Remove(mountpoint) + } + return nil + } + + if result := runCmd(caseCtx, rec, c.ID, "mkdir-remote", env, "", drive9Bin, "fs", "mkdir", ":"+caseRemote); result.ExitCode != 0 { + return fmt.Errorf("create remote root: %s", result.Stderr) + } + mount, err := startCaseMount(caseCtx, rec, manifest, env, drive9Bin, caseRemote, mountpoint, c) + if err != nil { + return err + } + defer func() { + stopCtx, stopCancel := context.WithTimeout(context.Background(), 35*time.Second) + defer stopCancel() + if err := stopCaseMount(stopCtx, rec, env, drive9Bin, c.ID, &mount); err != nil && mount != nil { + _ = mountproc.KillMount(mount) + } + if shouldRemoveMountpoint(c, rec, retErr) { + _ = os.Remove(mountpoint) + } + }() + switch c.Workload.Type { + case "mount_smoke": + err = runMountSmoke(caseCtx, rec, env, drive9Bin, caseRemote, mountpoint, c) + case "path_matrix": + err = runPathMatrix(caseCtx, rec, env, drive9Bin, caseRemote, mountpoint, c) + case "dual_mount_visibility": + err = runDualMountVisibility(caseCtx, rec, manifest, env, drive9Bin, caseRemote, mountRoot, runID, mountpoint, c) + case "fio": + err = runFIO(caseCtx, rec, mountpoint, c) + case "small_file_storm": + err = runSmallFileStorm(caseCtx, rec, mountpoint, c) + case "parallel_writes": + err = runParallelWrites(caseCtx, rec, mountpoint, c) + case "git_workflow": + err = runGitWorkflow(caseCtx, rec, mountpoint, c) + case "open_fd_unmount": + err = runOpenFDUnmount(caseCtx, rec, env, drive9Bin, mountpoint, c, &mount) + case "kill_during_write": + err = runKillDuringWrite(caseCtx, rec, manifest, env, drive9Bin, caseRemote, mountpoint, c, &mount) + default: + err = fmt.Errorf("unsupported workload %q", c.Workload.Type) + } + if err != nil { + return err + } + if err := verifyRemount(caseCtx, rec, manifest, env, drive9Bin, caseRemote, mountpoint, c, &mount); err != nil { + return err + } + _ = rec.Event(report.Event{CaseID: c.ID, Type: "case_end"}) + return nil +} + +func startCaseMount(ctx context.Context, rec *report.Recorder, manifest *report.Manifest, env mountproc.Env, drive9Bin, caseRemote, mountpoint string, c casefile.Case) (*mountproc.Mount, error) { + mountCtx, cancel := context.WithTimeout(ctx, mountReadyTimeout(c)) + defer cancel() + logPath := filepath.Join(rec.RunDir, "mount", c.ID+".log") + started := time.Now() + _ = rec.Event(report.Event{CaseID: c.ID, Type: "mount_start", ArtifactRefs: []string{filepath.ToSlash(logPath)}}) + mount, err := mountproc.StartMount(context.Background(), c.ID, env, drive9Bin, caseRemote, mountpoint, c.SyncMode, logPath) + if err != nil { + return nil, err + } + manifest.ProcessGroups[c.ID] = mount.ProcessGroup + if err := mountproc.WaitMounted(mountCtx, mountpoint, safety.IsMounted); err != nil { + _ = mountproc.KillMount(mount) + return nil, err + } + elapsed := time.Since(started) + _ = rec.Event(report.Event{CaseID: c.ID, Type: "mount_ready", DurationMS: elapsed.Milliseconds(), ArtifactRefs: []string{filepath.ToSlash(logPath)}}) + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "mount_startup_ms", Value: float64(elapsed.Milliseconds()), Unit: "ms", Source: "drive9 mount"}) + return mount, nil +} + +func stopCaseMount(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin, caseID string, mount **mountproc.Mount) error { + if mount == nil || *mount == nil { + return nil + } + mountpoint := (*mount).Mountpoint + _ = rec.Event(report.Event{CaseID: caseID, Type: "unmount_start"}) + result := mountproc.Stop(ctx, env, drive9Bin, mountpoint) + recordResult(rec, caseID, result) + if result.ExitCode != 0 { + return fmt.Errorf("unmount %s: %s", mountpoint, result.Stderr) + } + if err := waitUnmounted(ctx, mountpoint); err != nil { + return err + } + _ = rec.Event(report.Event{CaseID: caseID, Type: "unmount_end"}) + *mount = nil + return nil +} + +func waitUnmounted(ctx context.Context, mountpoint string) error { + return poll(ctx, func() error { + mounted, err := safety.IsMounted(mountpoint) + if err != nil { + return err + } + if mounted { + return fmt.Errorf("still mounted: %s", mountpoint) + } + return nil + }) +} + +func verifyRemount(ctx context.Context, rec *report.Recorder, manifest *report.Manifest, env mountproc.Env, drive9Bin, caseRemote, mountpoint string, c casefile.Case, mount **mountproc.Mount) error { + paths := remountPaths(c) + if len(paths) == 0 { + return nil + } + before := map[string]string{} + for _, rel := range paths { + local, err := safeJoinLocal(mountpoint, rel) + if err != nil { + return err + } + sum, err := hashFile(local) + if err != nil { + recordOracleFailure(rec, c, "remount_hash_equal", err.Error(), "pre-remount hash") + continue + } + before[rel] = sum + } + if len(before) == 0 { + return nil + } + if err := stopCaseMount(ctx, rec, env, drive9Bin, c.ID, mount); err != nil { + return err + } + nextMount, err := startCaseMount(ctx, rec, manifest, env, drive9Bin, caseRemote, mountpoint, c) + if err != nil { + return err + } + *mount = nextMount + expected := expectedRemountHashes(c) + for rel, want := range before { + local, err := safeJoinLocal(mountpoint, rel) + if err != nil { + return err + } + got, err := hashFile(local) + if err != nil { + recordOracleFailure(rec, c, "remount_hash_equal", err.Error(), want) + continue + } + if got != want { + recordOracleFailure(rec, c, "remount_hash_equal", got, want) + } + if expectedHash := expected[rel]; expectedHash != "" && got != expectedHash { + recordOracleFailure(rec, c, "remount_hash_equal", got, expectedHash) + } + } + return nil +} + +func runMountSmoke(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin, remoteRoot, mountpoint string, c casefile.Case) error { + totalBytes := 0 + for i, f := range c.Workload.Files { + totalBytes += len(f.Content) + if i%2 == 0 { + tmp := filepath.Join(rec.RunDir, "debug", c.ID+"-"+safeName(f.RelativePath)) + if err := os.MkdirAll(filepath.Dir(tmp), 0o755); err != nil { + return err + } + if err := os.WriteFile(tmp, []byte(f.Content), 0o644); err != nil { + return err + } + remote := path.Join(remoteRoot, f.RelativePath) + result := runCmd(ctx, rec, c.ID, "cli-write-"+f.RelativePath, env, "", drive9Bin, "fs", "cp", tmp, ":"+remote) + if result.ExitCode != 0 { + recordOracleFailure(rec, c, "fuse_write_success", result.Stderr, "exit 0") + continue + } + local, err := safeJoinLocal(mountpoint, f.RelativePath) + if err != nil { + return err + } + if err := waitLocalContent(ctx, c, local, f.Content); err != nil { + recordOracleFailure(rec, c, "cli_read_equals", err.Error(), f.Content) + } + continue + } + local, err := safeJoinLocal(mountpoint, f.RelativePath) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(local), 0o755); err != nil { + return err + } + if err := os.WriteFile(local, []byte(f.Content), 0o644); err != nil { + recordOracleFailure(rec, c, "fuse_write_success", err.Error(), "write success") + continue + } + remote := path.Join(remoteRoot, f.RelativePath) + if err := waitRemoteContent(ctx, rec, env, drive9Bin, c, remote, f.Content); err != nil { + recordOracleFailure(rec, c, "cli_read_equals", err.Error(), f.Content) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "bytes_written", Value: float64(totalBytes), Unit: "bytes", Source: c.Workload.Type}) + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "file_count", Value: float64(len(c.Workload.Files)), Unit: "files", Source: c.Workload.Type}) + return nil +} + +func runPathMatrix(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin, remoteRoot, mountpoint string, c casefile.Case) error { + totalBytes := 0 + for _, p := range c.Workload.Paths { + content := c.Workload.ContentTemplate + if content == "" { + content = c.ID + ":" + p + } + totalBytes += len(content) + local, err := safeJoinLocal(mountpoint, p) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(local), 0o755); err != nil { + return err + } + if err := os.WriteFile(local, []byte(content), 0o644); err != nil { + recordOracleFailure(rec, c, "fuse_write_success", err.Error(), "write success") + continue + } + if err := waitRemoteContent(ctx, rec, env, drive9Bin, c, path.Join(remoteRoot, p), content); err != nil { + recordOracleFailure(rec, c, "cli_read_equals", err.Error(), content) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "bytes_written", Value: float64(totalBytes), Unit: "bytes", Source: c.Workload.Type}) + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "file_count", Value: float64(len(c.Workload.Paths)), Unit: "files", Source: c.Workload.Type}) + return nil +} + +func runDualMountVisibility(ctx context.Context, rec *report.Recorder, manifest *report.Manifest, env mountproc.Env, drive9Bin, caseRemote, mountRoot, runID, primaryMountpoint string, c casefile.Case) error { + secondMountpoint, err := safety.Mountpoint(mountRoot, runID, c.MountpointSuffix+"-b") + if err != nil { + return err + } + if err := safety.ValidateMountpointAvailable(secondMountpoint); err != nil { + return err + } + manifest.GeneratedMountpoints[c.ID+"-b"] = secondMountpoint + secondMount, err := startCaseMount(ctx, rec, manifest, env, drive9Bin, caseRemote, secondMountpoint, c) + if err != nil { + return err + } + defer func() { + stopCtx, cancel := context.WithTimeout(context.Background(), 35*time.Second) + defer cancel() + if err := stopCaseMount(stopCtx, rec, env, drive9Bin, c.ID, &secondMount); err != nil && secondMount != nil { + _ = mountproc.KillMount(secondMount) + } + if shouldRemoveMountpoint(c, rec, nil) { + _ = os.Remove(secondMountpoint) + } + }() + for i, f := range c.Workload.Files { + fromA := i%2 == 0 + src := primaryMountpoint + dst := secondMountpoint + if !fromA { + src, dst = secondMountpoint, primaryMountpoint + } + rel := f.RelativePath + if rel == "" { + rel = fmt.Sprintf("dual-%d.txt", i) + } + content := f.Content + if content == "" { + content = fmt.Sprintf("%s:%d", c.ID, i) + } + local, err := safeJoinLocal(src, rel) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(local), 0o755); err != nil { + return err + } + if err := os.WriteFile(local, []byte(content), 0o644); err != nil { + recordOracleFailure(rec, c, "dual_mount_visibility", err.Error(), "write success") + continue + } + dstLocal, err := safeJoinLocal(dst, rel) + if err != nil { + return err + } + if err := waitLocalContent(ctx, c, dstLocal, content); err != nil { + recordOracleFailure(rec, c, "dual_mount_visibility", err.Error(), content) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "file_count", Value: float64(len(c.Workload.Files)), Unit: "files", Source: c.Workload.Type}) + return nil +} + +func runFIO(ctx context.Context, rec *report.Recorder, mountpoint string, c casefile.Case) error { + fio := c.Workload.FioBinary + if fio == "" { + fio = "fio" + } + if _, err := resolveBin(fio); err != nil { + return fmt.Errorf("fio workload requires %q in PATH or workload.fio_binary: %w", fio, err) + } + blockBytes := c.Workload.BlockBytes + if blockBytes <= 0 { + blockBytes = 1 << 20 + } + output := filepath.Join(rec.RunDir, "metrics", c.ID+"-fio.json") + target := filepath.Join(mountpoint, "fio-seq.dat") + args := []string{ + "--name=" + c.ID, + "--filename=" + target, + "--size=" + fmt.Sprint(c.Workload.SizeBytes), + "--bs=" + fmt.Sprint(blockBytes), + "--rw=write", + "--ioengine=sync", + "--fsync=1", + "--output-format=json", + } + result := mountproc.Run(ctx, "fio-write", mountproc.Env{}, "", fio, args...) + recordResult(rec, c.ID, result) + _ = os.WriteFile(output, []byte(result.Stdout), 0o644) + if result.ExitCode != 0 { + recordOracleFailure(rec, c, "throughput_min", result.Stderr, "fio exit 0") + return nil + } + throughput := fioWriteBytesPerSecond(result.Stdout) + if throughput > 0 { + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "mount_perf_counter", Value: throughput, Unit: "bytes", Source: "fio.write_bw_bytes_per_second", ArtifactRefs: []string{filepath.ToSlash(output)}}) + } + if c.Workload.MinBytesPerSecond > 0 && int64(throughput) < c.Workload.MinBytesPerSecond { + recordOracleFailure(rec, c, "throughput_min", int64(throughput), c.Workload.MinBytesPerSecond) + } + start := time.Now() + if _, err := hashFile(target); err != nil { + recordOracleFailure(rec, c, "cli_read_equals", err.Error(), "readable fio output") + } else { + elapsed := time.Since(start) + if elapsed > 0 { + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "bytes_read", Value: float64(c.Workload.SizeBytes), Unit: "bytes", Source: "fio-readback"}) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "bytes_written", Value: float64(c.Workload.SizeBytes), Unit: "bytes", Source: c.Workload.Type}) + return nil +} + +func runSmallFileStorm(ctx context.Context, rec *report.Recorder, mountpoint string, c casefile.Case) error { + count := c.Workload.FileCount + for i := 0; i < count; i++ { + select { + case <-ctx.Done(): + return ctx.Err() + default: + } + rel := filepath.Join("small-files", fmt.Sprintf("%05d.txt", i)) + content := fmt.Sprintf("%s:%05d\n", c.ID, i) + local, err := safeJoinLocal(mountpoint, filepath.ToSlash(rel)) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(local), 0o755); err != nil { + return err + } + if err := os.WriteFile(local, []byte(content), 0o644); err != nil { + recordOracleFailure(rec, c, "fuse_write_success", err.Error(), "write success") + continue + } + } + entries, err := os.ReadDir(filepath.Join(mountpoint, "small-files")) + if err != nil { + recordOracleFailure(rec, c, "file_count_equals", err.Error(), count) + return nil + } + if len(entries) != count { + recordOracleFailure(rec, c, "file_count_equals", len(entries), count) + } + samples := c.Workload.ReadSampleCount + if samples <= 0 || samples > count { + samples = min(count, 10) + } + for i := 0; i < samples; i++ { + idx := 0 + if samples > 1 { + idx = i * (count - 1) / (samples - 1) + } + want := fmt.Sprintf("%s:%05d\n", c.ID, idx) + local, err := safeJoinLocal(mountpoint, path.Join("small-files", fmt.Sprintf("%05d.txt", idx))) + if err != nil { + return err + } + if err := waitLocalContent(ctx, c, local, want); err != nil { + recordOracleFailure(rec, c, "cli_read_equals", err.Error(), want) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "file_count", Value: float64(count), Unit: "files", Source: c.Workload.Type}) + return nil +} + +func runParallelWrites(ctx context.Context, rec *report.Recorder, mountpoint string, c casefile.Case) error { + start := time.Now() + var wg sync.WaitGroup + var mu sync.Mutex + var errs []string + for i := 0; i < c.Workload.ParallelWriters; i++ { + i := i + wg.Add(1) + go func() { + defer wg.Done() + file := filepath.Join(mountpoint, "parallel", fmt.Sprintf("writer-%02d.bin", i)) + if err := writeSizedFile(ctx, file, c.Workload.WriterBytes, byte('A'+i%26)); err != nil { + mu.Lock() + errs = append(errs, err.Error()) + mu.Unlock() + } + }() + } + wg.Wait() + if len(errs) > 0 { + recordOracleFailure(rec, c, "fuse_write_success", errs, "all writers succeed") + return nil + } + total := c.Workload.WriterBytes * int64(c.Workload.ParallelWriters) + elapsed := time.Since(start) + if elapsed > 0 { + bps := float64(total) / elapsed.Seconds() + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "mount_perf_counter", Value: bps, Unit: "bytes", Source: "parallel_write_bytes_per_second"}) + if c.Workload.MinBytesPerSecond > 0 && int64(bps) < c.Workload.MinBytesPerSecond { + recordOracleFailure(rec, c, "throughput_min", int64(bps), c.Workload.MinBytesPerSecond) + } + } + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "bytes_written", Value: float64(total), Unit: "bytes", Source: c.Workload.Type}) + return nil +} + +func runGitWorkflow(ctx context.Context, rec *report.Recorder, mountpoint string, c casefile.Case) error { + git := c.Workload.GitBinary + if git == "" { + git = "git" + } + repoPath, err := safeJoinLocal(mountpoint, c.Workload.CloneDir) + if err != nil { + return err + } + timeout := c.Workload.GitTimeout.Duration + if timeout == 0 { + timeout = c.Timeout.Duration + } + gitCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + cloneURL, err := resolveCloneURL(gitCtx, rec, c, git) + if err != nil { + return err + } + result := mountproc.Run(gitCtx, "git-clone", mountproc.Env{}, mountpoint, git, "clone", "--depth", "1", cloneURL, c.Workload.CloneDir) + recordResult(rec, c.ID, result) + _ = rec.Metric(report.Metric{CaseID: c.ID, Name: "git_clone_duration_ms", Value: float64(result.Duration.Milliseconds()), Unit: "ms", Source: "git clone"}) + if result.ExitCode != 0 { + recordOracleFailure(rec, c, "git_clone", result.Stderr, "exit 0") + return nil + } + checkNoGitLocks(rec, c, repoPath, "after clone") + before := gitCommitCount(gitCtx, git, repoPath) + userName := c.Workload.GitUserName + if userName == "" { + userName = "Drive9 Harness" + } + userEmail := c.Workload.GitUserEmail + if userEmail == "" { + userEmail = "drive9-harness@example.invalid" + } + if result := mountproc.Run(gitCtx, "git-config-name", mountproc.Env{}, repoPath, git, "config", "user.name", userName); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + recordOracleFailure(rec, c, "git_config", result.Stderr, "exit 0") + return nil + } else { + recordResult(rec, c.ID, result) + } + if result := mountproc.Run(gitCtx, "git-config-email", mountproc.Env{}, repoPath, git, "config", "user.email", userEmail); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + recordOracleFailure(rec, c, "git_config", result.Stderr, "exit 0") + return nil + } else { + recordResult(rec, c.ID, result) + } + mutationPath, err := safeJoinLocal(repoPath, c.Workload.Mutation.Path) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(mutationPath), 0o755); err != nil { + return err + } + flag := os.O_CREATE | os.O_WRONLY + if c.Workload.Mutation.Mode == "append" { + flag |= os.O_APPEND + } else { + flag |= os.O_TRUNC + } + f, err := os.OpenFile(mutationPath, flag, 0o644) + if err != nil { + recordOracleFailure(rec, c, "git_mutation", err.Error(), "write success") + return nil + } + if _, err := f.WriteString(c.Workload.Mutation.Content); err != nil { + _ = f.Close() + recordOracleFailure(rec, c, "git_mutation", err.Error(), "write success") + return nil + } + if err := f.Close(); err != nil { + recordOracleFailure(rec, c, "git_mutation", err.Error(), "close success") + return nil + } + if result := mountproc.Run(gitCtx, "git-add", mountproc.Env{}, repoPath, git, "add", c.Workload.Mutation.Path); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + recordOracleFailure(rec, c, "git_add", result.Stderr, "exit 0") + return nil + } else { + recordResult(rec, c.ID, result) + } + if result := mountproc.Run(gitCtx, "git-commit", mountproc.Env{}, repoPath, git, "commit", "-m", c.Workload.CommitMessage); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + recordOracleFailure(rec, c, "git_commit", result.Stderr, "exit 0") + return nil + } else { + recordResult(rec, c.ID, result) + } + after := gitCommitCount(gitCtx, git, repoPath) + if after-before != c.Workload.ExpectedCommitDelta { + recordOracleFailure(rec, c, "git_commit_count", after-before, c.Workload.ExpectedCommitDelta) + } + status := mountproc.Run(gitCtx, "git-status", mountproc.Env{}, repoPath, git, "status", "--porcelain") + recordResult(rec, c.ID, status) + if strings.TrimSpace(status.Stdout) != strings.TrimSpace(c.Workload.ExpectedStatus) { + recordOracleFailure(rec, c, "git_status_equals", status.Stdout, c.Workload.ExpectedStatus) + } + checkNoGitLocks(rec, c, repoPath, "after workflow") + return nil +} + +func runDoctor(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin, mountpoint string, c casefile.Case) error { + result := runCmd(ctx, rec, c.ID, "doctor-fuse", env, "", drive9Bin, "doctor", "fuse", "--mountpoint", mountpoint, "--server", env.Server) + want := 0 + if c.Workload.ExpectExit != nil { + want = *c.Workload.ExpectExit + } + if result.ExitCode != want { + if c.Workload.AllowNonzeroWhenNoAllowOther && doctorOnlyNoAllowOtherFailed(result.Stdout+result.Stderr) { + return nil + } + recordOracleFailure(rec, c, "command_exit", result.ExitCode, want) + } + return nil +} + +func doctorOnlyNoAllowOtherFailed(output string) bool { + failures := 0 + noAllowOther := false + for _, line := range strings.Split(output, "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "FAIL ") { + continue + } + failures++ + if strings.Contains(line, "/etc/fuse.conf user_allow_other:") { + noAllowOther = true + } + } + return failures == 1 && noAllowOther +} + +func runOpenFDUnmount(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin, mountpoint string, c casefile.Case, mount **mountproc.Mount) error { + local := filepath.Join(mountpoint, "held-open.txt") + if err := os.MkdirAll(filepath.Dir(local), 0o755); err != nil { + return err + } + f, err := os.OpenFile(local, os.O_CREATE|os.O_RDWR|os.O_TRUNC, 0o644) + if err != nil { + recordOracleFailure(rec, c, "unmount_busy_then_clean", err.Error(), "open file") + return nil + } + _, _ = f.WriteString("held open by harness\n") + if c.Workload.HoldOpenDuration.Duration > 0 { + timer := time.NewTimer(c.Workload.HoldOpenDuration.Duration) + select { + case <-ctx.Done(): + timer.Stop() + _ = f.Close() + return ctx.Err() + case <-timer.C: + } + } + firstCtx, cancel := context.WithTimeout(ctx, 10*time.Second) + first := mountproc.Stop(firstCtx, env, drive9Bin, mountpoint) + cancel() + recordResult(rec, c.ID, first) + _ = f.Close() + if first.ExitCode == 0 { + if err := waitUnmounted(ctx, mountpoint); err != nil { + recordOracleFailure(rec, c, "unmount_busy_then_clean", err.Error(), "unmounted after first umount") + return nil + } + *mount = nil + return nil + } + if err := stopCaseMount(ctx, rec, env, drive9Bin, c.ID, mount); err != nil { + recordOracleFailure(rec, c, "unmount_busy_then_clean", err.Error(), "clean unmount after fd close") + } + return nil +} + +func runKillDuringWrite(ctx context.Context, rec *report.Recorder, manifest *report.Manifest, env mountproc.Env, drive9Bin, caseRemote, mountpoint string, c casefile.Case, mount **mountproc.Mount) error { + target := filepath.Join(mountpoint, "kill-during-write.bin") + writerDone := make(chan error, 1) + go func() { + writerDone <- writeSizedFile(ctx, target, c.Workload.WriterBytes, 'K') + }() + timer := time.NewTimer(c.Workload.KillAfter.Duration) + var writerErr error + writerFinished := false + killed := false + select { + case <-ctx.Done(): + timer.Stop() + return ctx.Err() + case writerErr = <-writerDone: + writerFinished = true + if !timer.Stop() { + select { + case <-timer.C: + default: + } + } + case <-timer.C: + killed = true + if mount != nil && *mount != nil { + _ = mountproc.KillMount(*mount) + *mount = nil + _ = rec.Event(report.Event{CaseID: c.ID, Type: "mount_end", Message: "mount process group killed during active write"}) + } + } + if killed && !writerFinished { + select { + case writerErr = <-writerDone: + default: + writerErr = errors.New("writer interrupted before acknowledged completion") + } + } + if killed { + nextMount, err := startCaseMount(ctx, rec, manifest, env, drive9Bin, caseRemote, mountpoint, c) + if err != nil { + return err + } + *mount = nextMount + } + info, statErr := os.Stat(target) + if writerErr != nil { + observed := map[string]any{"writer_error": writerErr.Error()} + if statErr == nil { + observed["recovered_bytes"] = info.Size() + } else { + observed["stat_error"] = statErr.Error() + } + recordFailure(rec, c, "recovery_classified", "inconclusive", observed, "partial/missing data classified") + return nil + } + if statErr != nil { + recordOracleFailure(rec, c, "recovery_classified", statErr.Error(), "completed writer output exists") + return nil + } + if info.Size() != c.Workload.WriterBytes { + recordOracleFailure(rec, c, "recovery_classified", info.Size(), c.Workload.WriterBytes) + } + return nil +} + +func runCmd(ctx context.Context, rec *report.Recorder, caseID, id string, env mountproc.Env, dir, name string, args ...string) mountproc.Result { + _ = rec.Event(report.Event{CaseID: caseID, Type: "command_start", CommandID: id}) + result := mountproc.Run(ctx, id, env, dir, name, args...) + recordResult(rec, caseID, result) + return result +} + +func recordResult(rec *report.Recorder, caseID string, result mountproc.Result) { + exit := result.ExitCode + _ = rec.Event(report.Event{CaseID: caseID, Type: "command_end", CommandID: result.ID, ExitCode: &exit, DurationMS: result.Duration.Milliseconds()}) + _ = rec.Metric(report.Metric{CaseID: caseID, Name: "command_duration_ms", Value: float64(result.Duration.Milliseconds()), Unit: "ms", Source: result.ID}) +} + +func recordOracleFailure(rec *report.Recorder, c casefile.Case, oracle string, observed, expected any) { + recordFailure(rec, c, oracle, "product", observed, expected) +} + +func recordFailure(rec *report.Recorder, c casefile.Case, oracle, class string, observed, expected any) { + reproPath := filepath.ToSlash(filepath.Join("repro", c.ID+".sh")) + _ = writeRepro(filepath.Join(rec.RunDir, reproPath), c) + _ = rec.Failure(report.Failure{CaseID: c.ID, Severity: c.Severity.Failure, Class: class, Oracle: oracle, ExpectedOutcome: c.ExpectedOutcome, Message: "oracle failed", Observed: observed, Expected: expected, ReproPath: reproPath}) +} + +func waitLocalContent(ctx context.Context, c casefile.Case, local, want string) error { + waitCtx, cancel := context.WithTimeout(ctx, c.RemoteVisibilityTimeout()) + defer cancel() + return pollWith(waitCtx, c.CommandRetryCount(), c.CommandRetrySleep(), func() error { + b, err := os.ReadFile(local) + if err != nil { + return err + } + if string(b) != want { + return fmt.Errorf("content %q", string(b)) + } + return nil + }) +} + +func waitRemoteContent(ctx context.Context, rec *report.Recorder, env mountproc.Env, drive9Bin string, c casefile.Case, remote, want string) error { + waitCtx, cancel := context.WithTimeout(ctx, c.RemoteVisibilityTimeout()) + defer cancel() + return pollWith(waitCtx, c.CommandRetryCount(), c.CommandRetrySleep(), func() error { + result := runCmd(waitCtx, rec, c.ID, "cli-cat-"+safeName(remote), env, "", drive9Bin, "fs", "cat", ":"+remote) + if result.ExitCode != 0 { + return fmt.Errorf("%s", result.Stderr) + } + if result.Stdout != want { + return fmt.Errorf("content %q", result.Stdout) + } + return nil + }) +} + +func pollWith(ctx context.Context, attempts int, sleep time.Duration, fn func() error) error { + if attempts <= 0 { + attempts = 1 + } + if sleep <= 0 { + sleep = 250 * time.Millisecond + } + var last error + for i := 0; i < attempts; i++ { + if err := fn(); err == nil { + return nil + } else { + last = err + } + if i == attempts-1 { + break + } + timer := time.NewTimer(sleep) + select { + case <-ctx.Done(): + timer.Stop() + if last != nil { + return last + } + return ctx.Err() + case <-timer.C: + } + } + return last +} + +func poll(ctx context.Context, fn func() error) error { + ticker := time.NewTicker(250 * time.Millisecond) + defer ticker.Stop() + var last error + for { + if err := fn(); err == nil { + return nil + } else { + last = err + } + select { + case <-ctx.Done(): + return last + case <-ticker.C: + } + } +} + +func safeJoinLocal(root, rel string) (string, error) { + if err := casefile.ValidateRelativeSlashPath("local relative path", rel); err != nil { + return "", err + } + cleanRoot := filepath.Clean(root) + joined := filepath.Clean(filepath.Join(cleanRoot, filepath.FromSlash(rel))) + if joined == cleanRoot || !strings.HasPrefix(joined, cleanRoot+string(os.PathSeparator)) { + return "", fmt.Errorf("path %q escapes root %q", rel, cleanRoot) + } + return joined, nil +} + +func gitCommitCount(ctx context.Context, git, repo string) int { + result := mountproc.Run(ctx, "git-count", mountproc.Env{}, repo, git, "rev-list", "--count", "HEAD") + if result.ExitCode != 0 { + return 0 + } + var n int + _, _ = fmt.Sscanf(strings.TrimSpace(result.Stdout), "%d", &n) + return n +} + +func resolveCloneURL(ctx context.Context, rec *report.Recorder, c casefile.Case, git string) (string, error) { + if !strings.HasPrefix(c.Workload.CloneURL, "fixture://") { + return c.Workload.CloneURL, nil + } + name := strings.TrimPrefix(c.Workload.CloneURL, "fixture://") + switch name { + case "basic": + repoPath := filepath.Join(rec.RunDir, "debug", "git-fixtures", c.ID, "basic") + if err := os.MkdirAll(repoPath, 0o755); err != nil { + return "", err + } + if _, err := os.Stat(filepath.Join(repoPath, ".git")); os.IsNotExist(err) { + if result := mountproc.Run(ctx, "fixture-git-init", mountproc.Env{}, repoPath, git, "init", "."); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + return "", fmt.Errorf("create git fixture: %s", result.Stderr) + } else { + recordResult(rec, c.ID, result) + } + if err := os.WriteFile(filepath.Join(repoPath, "README.md"), []byte("drive9 harness fixture\n"), 0o644); err != nil { + return "", err + } + recordResult(rec, c.ID, mountproc.Run(ctx, "fixture-git-config-name", mountproc.Env{}, repoPath, git, "config", "user.name", "Drive9 Harness")) + recordResult(rec, c.ID, mountproc.Run(ctx, "fixture-git-config-email", mountproc.Env{}, repoPath, git, "config", "user.email", "drive9-harness@example.invalid")) + if result := mountproc.Run(ctx, "fixture-git-add", mountproc.Env{}, repoPath, git, "add", "README.md"); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + return "", fmt.Errorf("stage git fixture: %s", result.Stderr) + } else { + recordResult(rec, c.ID, result) + } + if result := mountproc.Run(ctx, "fixture-git-commit", mountproc.Env{}, repoPath, git, "commit", "-m", "initial fixture"); result.ExitCode != 0 { + recordResult(rec, c.ID, result) + return "", fmt.Errorf("commit git fixture: %s", result.Stderr) + } else { + recordResult(rec, c.ID, result) + } + } + return "file://" + filepath.ToSlash(repoPath), nil + default: + return "", fmt.Errorf("unknown git fixture %q", name) + } +} + +func checkNoGitLocks(rec *report.Recorder, c casefile.Case, repoPath, stage string) { + globs := []string{".git/*.lock"} + for _, o := range c.Oracles { + if o.Type == "no_git_locks" && len(o.Globs) > 0 { + globs = o.Globs + break + } + } + var matches []string + for _, glob := range globs { + pattern, err := safeJoinLocal(repoPath, glob) + if err != nil { + recordOracleFailure(rec, c, "no_git_locks", err.Error(), "safe lock glob") + continue + } + found, _ := filepath.Glob(pattern) + matches = append(matches, found...) + } + if len(matches) > 0 { + recordOracleFailure(rec, c, "no_git_locks", map[string]any{"stage": stage, "matches": matches}, "no lock files") + } +} + +func mountReadyTimeout(c casefile.Case) time.Duration { + timeout := c.MountReadyTimeout() + if c.Timeout.Duration > 0 && c.Timeout.Duration < timeout { + return c.Timeout.Duration + } + return timeout +} + +func remountPaths(c casefile.Case) []string { + if c.Workload.Remount != nil && !*c.Workload.Remount { + return nil + } + for _, o := range c.Oracles { + if o.Type == "remount_hash_equal" && len(o.Paths) > 0 { + return cleanRelativePaths(o.Paths) + } + } + switch c.Workload.Type { + case "mount_smoke": + out := make([]string, 0, len(c.Workload.Files)) + for _, f := range c.Workload.Files { + out = append(out, f.RelativePath) + } + return cleanRelativePaths(out) + case "path_matrix": + return cleanRelativePaths(c.Workload.Paths) + case "dual_mount_visibility": + out := make([]string, 0, len(c.Workload.Files)) + for _, f := range c.Workload.Files { + out = append(out, f.RelativePath) + } + return cleanRelativePaths(out) + case "small_file_storm": + out := make([]string, 0, c.Workload.FileCount) + for i := 0; i < c.Workload.FileCount; i++ { + out = append(out, path.Join("small-files", fmt.Sprintf("%05d.txt", i))) + } + return cleanRelativePaths(out) + case "parallel_writes": + out := make([]string, 0, c.Workload.ParallelWriters) + for i := 0; i < c.Workload.ParallelWriters; i++ { + out = append(out, path.Join("parallel", fmt.Sprintf("writer-%02d.bin", i))) + } + return cleanRelativePaths(out) + case "git_workflow": + out := make([]string, 0, len(c.Workload.RemountVerifyPaths)) + for _, v := range c.Workload.RemountVerifyPaths { + out = append(out, path.Join(c.Workload.CloneDir, v.Path)) + } + return cleanRelativePaths(out) + default: + return nil + } +} + +func expectedRemountHashes(c casefile.Case) map[string]string { + out := map[string]string{} + for _, o := range c.Oracles { + if o.Type == "remount_hash_equal" { + for p, h := range o.ExpectedHashes { + out[path.Clean(p)] = h + } + } + } + if c.Workload.Type == "git_workflow" { + for _, v := range c.Workload.RemountVerifyPaths { + if v.SHA256 != "" { + out[path.Join(c.Workload.CloneDir, v.Path)] = v.SHA256 + } + } + } + return out +} + +func cleanRelativePaths(paths []string) []string { + out := make([]string, 0, len(paths)) + seen := map[string]bool{} + for _, p := range paths { + clean := path.Clean(strings.TrimPrefix(p, "/")) + if clean == "." || clean == ".." || strings.HasPrefix(clean, "../") || seen[clean] { + continue + } + seen[clean] = true + out = append(out, clean) + } + return out +} + +func hashFile(file string) (string, error) { + f, err := os.Open(file) + if err != nil { + return "", err + } + defer func() { _ = f.Close() }() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + +func writeSizedFile(ctx context.Context, file string, size int64, fill byte) error { + if err := os.MkdirAll(filepath.Dir(file), 0o755); err != nil { + return err + } + f, err := os.OpenFile(file, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644) + if err != nil { + return err + } + defer func() { _ = f.Close() }() + block := make([]byte, 1<<20) + for i := range block { + block[i] = fill + } + var written int64 + for written < size { + select { + case <-ctx.Done(): + return ctx.Err() + default: + } + n := int64(len(block)) + if remaining := size - written; remaining < n { + n = remaining + } + if _, err := f.Write(block[:n]); err != nil { + return err + } + written += n + } + return f.Sync() +} + +func fioWriteBytesPerSecond(stdout string) float64 { + var body struct { + Jobs []struct { + Write struct { + BWBytes float64 `json:"bw_bytes"` + } `json:"write"` + } `json:"jobs"` + } + if err := json.Unmarshal([]byte(stdout), &body); err != nil { + return 0 + } + var total float64 + for _, job := range body.Jobs { + total += job.Write.BWBytes + } + return total +} + +func writeRepro(path string, c casefile.Case) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + body := fmt.Sprintf("#!/usr/bin/env bash\nset -euo pipefail\n\ndrive9-agent-harness run --suite %q --case %q \"$@\"\n", c.Suite, c.ID) + if err := os.WriteFile(path, []byte(body), 0o755); err != nil { + return err + } + return nil +} + +func shouldRemoveMountpoint(c casefile.Case, rec *report.Recorder, caseErr error) bool { + switch c.Cleanup { + case "always": + return true + case "retain_on_failure": + return caseErr == nil && !caseHadFailures(rec, c.ID) + default: + return false + } +} + +func caseHadFailures(rec *report.Recorder, caseID string) bool { + f, err := os.Open(filepath.Join(rec.RunDir, "failures.jsonl")) + if os.IsNotExist(err) { + return false + } + if err != nil { + return true + } + defer func() { _ = f.Close() }() + scanner := bufio.NewScanner(f) + for scanner.Scan() { + var failure report.Failure + if err := json.Unmarshal(scanner.Bytes(), &failure); err != nil { + return true + } + if failure.CaseID == caseID { + return true + } + } + return scanner.Err() != nil +} + +func requiresFreshTenant(cases []casefile.Case) bool { + for _, c := range cases { + if c.RequiresFreshTenant() { + return true + } + } + return false +} + +func containsSuite(cases []casefile.Case, suite string) bool { + for _, c := range cases { + if c.Suite == suite { + return true + } + } + return false +} + +func captureDebug(ctx context.Context, rec *report.Recorder, label string) error { + debugDir := filepath.Join(rec.RunDir, "debug") + if err := os.MkdirAll(debugDir, 0o755); err != nil { + return err + } + for _, cmd := range []struct { + id string + name string + args []string + }{ + {id: "mount-table-" + label, name: "mount"}, + {id: "ps-" + label, name: "ps", args: []string{"-ax"}}, + } { + result := mountproc.Run(ctx, cmd.id, mountproc.Env{}, "", cmd.name, cmd.args...) + path := filepath.Join(debugDir, cmd.id+".txt") + body := result.Stdout + if result.Stderr != "" { + body += "\nSTDERR:\n" + result.Stderr + } + _ = os.WriteFile(path, []byte(body), 0o644) + } + return nil +} + +func GC(ctx context.Context, cfg GCConfig) error { + if !cfg.ConfirmDelete { + return ErrConfirmDeleteRequired + } + manifest := report.Manifest{} + if err := readJSON(filepath.Join(cfg.RunDir, "manifest.json"), &manifest); err != nil { + return err + } + if cfg.SuccessfulOnly { + gating := report.Gating{} + if err := readJSON(filepath.Join(cfg.RunDir, "gating.json"), &gating); err != nil { + return err + } + if gating.GateStatus == "fail" || gating.GateStatus == "harness_failed" { + return fmt.Errorf("refusing successful-only GC for gate status %q", gating.GateStatus) + } + } + drive9Bin := cfg.Drive9Bin + if drive9Bin == "" { + drive9Bin = "drive9" + } + env := mountproc.Env{Server: cfg.Server, APIKey: cfg.APIKey} + if cfg.Server == "" { + env.Server = manifest.Server + } + for id, mp := range manifest.GeneratedMountpoints { + cleanMP := filepath.Clean(mp) + cleanRoot := filepath.Clean(manifest.MountRoot) + if cleanMP == cleanRoot || !strings.HasPrefix(cleanMP, cleanRoot+string(os.PathSeparator)) { + return fmt.Errorf("%w for %s: %q", ErrUnsafeMountpoint, id, mp) + } + mounted, err := safety.IsMounted(mp) + if err != nil { + return fmt.Errorf("check mountpoint %s: %w", mp, err) + } + if mounted { + result := mountproc.Stop(ctx, env, drive9Bin, mp) + if result.ExitCode != 0 { + return fmt.Errorf("unmount %s (%s): %s", id, mp, result.Stderr) + } + } + if err := os.RemoveAll(mp); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove mountpoint %s: %w", mp, err) + } + } + for id, remote := range manifest.GeneratedRemoteRoots { + if remote == "" || remote == manifest.RemoteRootBase || !strings.HasPrefix(remote, strings.TrimRight(manifest.RemoteRootBase, "/")+"/") { + return fmt.Errorf("refusing to delete unsafe remote root for %s: %q", id, remote) + } + result := mountproc.Run(ctx, "gc-remote-"+id, env, "", drive9Bin, "fs", "rm", "-r", ":"+remote) + if result.ExitCode != 0 { + return fmt.Errorf("delete remote root %s: %s", remote, result.Stderr) + } + } + return nil +} + +func CollectServerEvidence(ctx context.Context, cfg EvidenceConfig) error { + if !cfg.ApproveExternal { + return errors.New("collect-server-evidence requires --approve-external") + } + if cfg.RunDir == "" { + return errors.New("collect-server-evidence requires run dir") + } + debugDir := filepath.Join(cfg.RunDir, "debug", "server") + metricsDir := filepath.Join(cfg.RunDir, "metrics") + if err := os.MkdirAll(debugDir, 0o755); err != nil { + return err + } + if err := os.MkdirAll(metricsDir, 0o755); err != nil { + return err + } + if cfg.KubeContext != "" { + if cfg.Namespace == "" { + cfg.Namespace = "dat9" + } + if cfg.Selector == "" { + cfg.Selector = "app=dat9-server" + } + if cfg.Since == "" { + cfg.Since = "10m" + } + if cfg.Tail == 0 { + cfg.Tail = 500 + } + logs := mountproc.Run(ctx, "kubectl-logs", mountproc.Env{}, "", "kubectl", "--context", cfg.KubeContext, "-n", cfg.Namespace, "logs", "-l", cfg.Selector, "--since="+cfg.Since, fmt.Sprintf("--tail=%d", cfg.Tail)) + if err := os.WriteFile(filepath.Join(debugDir, "server-logs.jsonl"), []byte(logs.Stdout), 0o644); err != nil { + return err + } + if logs.ExitCode != 0 { + return fmt.Errorf("kubectl logs: %s", logs.Stderr) + } + metrics := mountproc.Run(ctx, "kubectl-metrics", mountproc.Env{}, "", "kubectl", "--context", cfg.KubeContext, "get", "--raw", "/api/v1/namespaces/"+cfg.Namespace+"/services/dat9-server:http/proxy/metrics") + if err := os.WriteFile(filepath.Join(metricsDir, "server-after.prom"), []byte(metrics.Stdout), 0o644); err != nil { + return err + } + if metrics.ExitCode != 0 { + return fmt.Errorf("kubectl metrics: %s", metrics.Stderr) + } + } + if cfg.MetricsRawPath != "" { + b, err := os.ReadFile(cfg.MetricsRawPath) + if err != nil { + return err + } + if err := os.WriteFile(filepath.Join(metricsDir, "server-after.prom"), b, 0o644); err != nil { + return err + } + } + summary := map[string]any{ + "schema_version": report.SchemaVersion, + "run_dir": cfg.RunDir, + "kube_context": cfg.KubeContext, + "namespace": cfg.Namespace, + "selector": cfg.Selector, + } + return writeJSON(filepath.Join(debugDir, "server-evidence.json"), summary) +} + +func provision(ctx context.Context, server string, timeout time.Duration) (string, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(server, "/")+"/v1/provision", nil) + if err != nil { + return "", err + } + resp, err := harnessHTTPClient.Do(req) + if err != nil { + return "", err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode >= 400 { + return "", fmt.Errorf("provision returned HTTP %d", resp.StatusCode) + } + var body struct { + APIKey string `json:"api_key"` + Status string `json:"status"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return "", err + } + if body.APIKey == "" { + return "", errors.New("provision response missing api_key") + } + statusCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + if err := pollStatus(statusCtx, server, body.APIKey); err != nil { + return "", err + } + return body.APIKey, nil +} + +func checkStatus(ctx context.Context, server, apiKey string) error { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(server, "/")+"/v1/status", nil) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+apiKey) + resp, err := harnessHTTPClient.Do(req) + if err != nil { + return err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode >= 400 { + return fmt.Errorf("status returned HTTP %d", resp.StatusCode) + } + return nil +} + +func pollStatus(ctx context.Context, server, apiKey string) error { + return poll(ctx, func() error { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(server, "/")+"/v1/status", nil) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+apiKey) + resp, err := harnessHTTPClient.Do(req) + if err != nil { + return err + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode >= 400 { + return fmt.Errorf("status returned HTTP %d", resp.StatusCode) + } + var body struct { + Status string `json:"status"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return err + } + if body.Status != "active" { + return fmt.Errorf("tenant status %q", body.Status) + } + return nil + }) +} + +func resolveBin(bin string) (string, error) { + if strings.Contains(bin, string(os.PathSeparator)) { + if _, err := os.Stat(bin); err != nil { + return "", err + } + return bin, nil + } + return exec.LookPath(bin) +} + +func drive9Version(ctx context.Context, bin string) string { + result := mountproc.Run(ctx, "drive9-version", mountproc.Env{}, "", bin, "--version") + if result.ExitCode != 0 { + return "unknown" + } + return strings.TrimSpace(result.Stdout + result.Stderr) +} + +func gitSHA() string { + result := mountproc.Run(context.Background(), "git-sha", mountproc.Env{}, "", "git", "rev-parse", "HEAD") + if result.ExitCode != 0 { + return "unknown" + } + return strings.TrimSpace(result.Stdout) +} + +func host() string { + name, err := os.Hostname() + if err != nil { + name = "unknown" + } + return name + " " + runtime.GOOS + "/" + runtime.GOARCH +} + +func runID() string { + return time.Now().UTC().Format("20060102T150405Z") +} + +func now() string { + return time.Now().UTC().Format(time.RFC3339Nano) +} + +func getenv(k, fallback string) string { + if v := os.Getenv(k); v != "" { + return v + } + return fallback +} + +func caseIDs(cases []casefile.Case) []string { + out := make([]string, 0, len(cases)) + for _, c := range cases { + out = append(out, c.ID) + } + return out +} + +func caseSummaries(cases []casefile.Case) []report.CaseSummary { + out := make([]report.CaseSummary, 0, len(cases)) + for _, c := range cases { + out = append(out, report.CaseSummary{ID: c.ID, Suite: c.Suite, ExpectedOutcome: c.ExpectedOutcome, Status: "passed"}) + } + return out +} + +func redact(key string) string { + if key == "" { + return "" + } + if len(key) <= 8 { + return "redacted" + } + return key[:4] + "..." + key[len(key)-4:] +} + +func safeName(s string) string { + sum := sha256.Sum256([]byte(s)) + return hex.EncodeToString(sum[:8]) +} + +func readJSON(path string, v any) error { + b, err := os.ReadFile(path) + if err != nil { + return err + } + return json.Unmarshal(b, v) +} + +func writeJSON(path string, v any) error { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, append(b, '\n'), 0o644) +} diff --git a/e2e/agent-harness/internal/runner/runner_test.go b/e2e/agent-harness/internal/runner/runner_test.go new file mode 100644 index 000000000..727d02a4b --- /dev/null +++ b/e2e/agent-harness/internal/runner/runner_test.go @@ -0,0 +1,168 @@ +package runner + +import ( + "context" + "errors" + "os" + "path/filepath" + "runtime" + "testing" + + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/casefile" + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/mountproc" + "github.com/mem9-ai/dat9/e2e/agent-harness/internal/report" +) + +func TestGCRequiresConfirmDelete(t *testing.T) { + err := GC(context.Background(), GCConfig{RunDir: t.TempDir()}) + if !errors.Is(err, ErrConfirmDeleteRequired) { + t.Fatalf("err = %v, want ErrConfirmDeleteRequired", err) + } +} + +func TestPreflightSkipsStatusWhenProvisioning(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("uses a POSIX fake drive9 script") + } + dir := t.TempDir() + bin := filepath.Join(dir, "drive9") + if err := os.WriteFile(bin, []byte("#!/usr/bin/env sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + err := Preflight(context.Background(), Config{ + ArtifactRoot: dir, + MountRoot: dir, + RemoteRootBase: "/agent-test", + Drive9Bin: bin, + Server: "http://127.0.0.1:1", + APIKey: "stale-key", + Provision: true, + }) + if err != nil { + t.Fatalf("Preflight returned %v, want nil", err) + } +} + +func TestFioWriteBytesPerSecond(t *testing.T) { + got := fioWriteBytesPerSecond(`{"jobs":[{"write":{"bw_bytes":123.5}},{"write":{"bw_bytes":10}}]}`) + if got != 133.5 { + t.Fatalf("bw = %v, want 133.5", got) + } +} + +func TestRunReturnsGateFailed(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("uses a POSIX fake drive9 script") + } + dir := t.TempDir() + bin := filepath.Join(dir, "drive9") + if err := os.WriteFile(bin, []byte("#!/usr/bin/env sh\nif [ \"$1\" = \"--version\" ]; then echo fake-drive9; exit 0; fi\nif [ \"$1\" = \"doctor\" ]; then exit 1; fi\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + suiteDir := filepath.Join(dir, "cases") + if err := os.MkdirAll(suiteDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(suiteDir, "regression.yaml"), []byte(` +defaults: + timeout: 2m + cleanup: always +cases: + - id: doctor-fails + suite: regression + expected_outcome: baseline_pass + remote_root_suffix: doctor-fails + mountpoint_suffix: doctor-fails + workload: + type: doctor_fuse + expect_exit: 0 + oracles: + - type: command_exit + severity: + failure: P1 +`), 0o644); err != nil { + t.Fatal(err) + } + runDir, err := Run(context.Background(), Config{ + ArtifactRoot: dir, + MountRoot: dir, + RemoteRootBase: "/agent-test-$RUN_ID", + Drive9Bin: bin, + Server: "http://127.0.0.1:1", + APIKey: "test-key", + SuiteDir: suiteDir, + Suites: []string{"regression"}, + }) + if !errors.Is(err, ErrGateFailed) { + t.Fatalf("err = %v, want ErrGateFailed", err) + } + if runDir == "" { + t.Fatal("runDir is empty") + } + var gating struct { + GateStatus string `json:"gate_status"` + Fail int `json:"fail"` + } + if err := readJSON(filepath.Join(runDir, "gating.json"), &gating); err != nil { + t.Fatal(err) + } + if gating.GateStatus != "fail" || gating.Fail != 1 { + t.Fatalf("gating = %+v, want fail with 1 failure", gating) + } +} + +func TestGCRejectsUnsafeMountpoint(t *testing.T) { + dir := t.TempDir() + runDir := filepath.Join(dir, "run") + if err := os.MkdirAll(runDir, 0o755); err != nil { + t.Fatal(err) + } + if err := writeJSON(filepath.Join(runDir, "manifest.json"), map[string]any{ + "mount_root": filepath.Join(dir, "mounts"), + "remote_root_base": "/agent-test", + "generated_mountpoints": map[string]string{"case": filepath.Join(dir, "outside")}, + "generated_remote_roots": map[string]string{}, + "generated_process_group": map[string]int{}, + }); err != nil { + t.Fatal(err) + } + if err := writeJSON(filepath.Join(runDir, "gating.json"), map[string]any{"gate_status": "pass"}); err != nil { + t.Fatal(err) + } + err := GC(context.Background(), GCConfig{RunDir: runDir, ConfirmDelete: true, SuccessfulOnly: true}) + if !errors.Is(err, ErrUnsafeMountpoint) { + t.Fatalf("err = %v, want ErrUnsafeMountpoint", err) + } +} + +func TestRunDoctorAllowsNoAllowOtherOnlyFailure(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("uses a POSIX fake drive9 script") + } + dir := t.TempDir() + bin := filepath.Join(dir, "drive9") + script := "#!/usr/bin/env sh\nprintf '%s\\n' 'drive9 doctor fuse' 'FAIL /etc/fuse.conf user_allow_other: user_allow_other is not enabled'\nexit 1\n" + if err := os.WriteFile(bin, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + rec, err := report.NewRecorder(dir, "run1") + if err != nil { + t.Fatal(err) + } + zero := 0 + c := casefile.Case{ + ID: "doctor", + ExpectedOutcome: "bug_reproduced", + Severity: casefile.SeveritySpec{Failure: "P2"}, + Workload: casefile.Workload{ + ExpectExit: &zero, + AllowNonzeroWhenNoAllowOther: true, + }, + } + if err := runDoctor(context.Background(), rec, mountproc.Env{}, bin, dir, c); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dir, "failures.jsonl")); !os.IsNotExist(err) { + t.Fatalf("failures.jsonl err = %v, want not exist", err) + } +} diff --git a/e2e/agent-harness/internal/safety/safety.go b/e2e/agent-harness/internal/safety/safety.go new file mode 100644 index 000000000..f3511baa1 --- /dev/null +++ b/e2e/agent-harness/internal/safety/safety.go @@ -0,0 +1,123 @@ +// Package safety validates generated paths before the harness touches them. +package safety + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path" + "path/filepath" + "runtime" + "strings" + + "github.com/mem9-ai/dat9/pkg/pathutil" +) + +var ( + ErrInvalidRoot = errors.New("invalid root") + ErrInvalidGeneratedPath = errors.New("invalid generated path") + ErrExistingMountpoint = errors.New("existing mountpoint") +) + +func ValidateRoot(name, p string) (string, error) { + if p == "" || !strings.HasPrefix(p, "/") { + return "", fmt.Errorf("%w: %s must be absolute", ErrInvalidRoot, name) + } + clean := filepath.Clean(p) + if strings.Contains(clean, "..") { + return "", fmt.Errorf("%w: %s contains traversal", ErrInvalidRoot, name) + } + return clean, nil +} + +func CaseRemoteRoot(base, suffix string) (string, error) { + if err := validateSuffix("remote_root_suffix", suffix); err != nil { + return "", err + } + root, err := pathutil.Canonicalize(path.Join(base, suffix)) + if err != nil { + return "", fmt.Errorf("%w: remote root: %v", ErrInvalidGeneratedPath, err) + } + baseClean, err := pathutil.Canonicalize(base) + if err != nil { + return "", fmt.Errorf("%w: remote base: %v", ErrInvalidRoot, err) + } + if root == baseClean || !strings.HasPrefix(root, strings.TrimRight(baseClean, "/")+"/") { + return "", fmt.Errorf("%w: remote root %q escapes %q", ErrInvalidGeneratedPath, root, baseClean) + } + return root, nil +} + +func Mountpoint(root, runID, suffix string) (string, error) { + if err := validateSuffix("mountpoint_suffix", suffix); err != nil { + return "", err + } + name := "drive9-agent-" + runID + "-" + suffix + mp := filepath.Join(root, name) + cleanRoot := filepath.Clean(root) + cleanMP := filepath.Clean(mp) + if cleanMP == cleanRoot || !strings.HasPrefix(cleanMP, cleanRoot+string(os.PathSeparator)) { + return "", fmt.Errorf("%w: mountpoint %q escapes %q", ErrInvalidGeneratedPath, cleanMP, cleanRoot) + } + return cleanMP, nil +} + +func ValidateMountpointAvailable(mp string) error { + if mounted, err := IsMounted(mp); err != nil { + return fmt.Errorf("%w: mount status check failed for %s: %v", ErrExistingMountpoint, mp, err) + } else if mounted { + return fmt.Errorf("%w: %s is already mounted", ErrExistingMountpoint, mp) + } + info, err := os.Lstat(mp) + if os.IsNotExist(err) { + return nil + } + if err != nil { + return fmt.Errorf("%w: inspect %s: %v", ErrExistingMountpoint, mp, err) + } + if info.Mode()&os.ModeSymlink != 0 { + return fmt.Errorf("%w: %s is a symlink", ErrExistingMountpoint, mp) + } + if !info.IsDir() { + return fmt.Errorf("%w: %s is not a directory", ErrExistingMountpoint, mp) + } + entries, err := os.ReadDir(mp) + if err != nil { + return fmt.Errorf("%w: read %s: %v", ErrExistingMountpoint, mp, err) + } + if len(entries) != 0 { + return fmt.Errorf("%w: %s is non-empty", ErrExistingMountpoint, mp) + } + return nil +} + +func IsMounted(mp string) (bool, error) { + clean := filepath.Clean(mp) + if runtime.GOOS == "linux" { + b, err := os.ReadFile("/proc/mounts") + if err != nil { + return false, err + } + for _, line := range strings.Split(string(b), "\n") { + fields := strings.Fields(line) + if len(fields) >= 2 && fields[1] == clean { + return true, nil + } + } + return false, nil + } + out, err := exec.Command("mount").Output() + if err != nil { + return false, err + } + return strings.Contains(string(out), " on "+clean+" "), nil +} + +func validateSuffix(field, suffix string) error { + if suffix == "" || strings.HasPrefix(suffix, "/") || strings.Contains(suffix, "/") || + strings.Contains(suffix, "\\") || suffix == "." || suffix == ".." || strings.Contains(suffix, "..") { + return fmt.Errorf("%w: invalid %s %q", ErrInvalidGeneratedPath, field, suffix) + } + return nil +} diff --git a/e2e/agent-harness/internal/safety/safety_test.go b/e2e/agent-harness/internal/safety/safety_test.go new file mode 100644 index 000000000..be530fd66 --- /dev/null +++ b/e2e/agent-harness/internal/safety/safety_test.go @@ -0,0 +1,34 @@ +package safety + +import ( + "errors" + "os" + "path/filepath" + "testing" +) + +func TestCaseRemoteRoot(t *testing.T) { + got, err := CaseRemoteRoot("/agent-run", "case-a") + if err != nil { + t.Fatal(err) + } + if got != "/agent-run/case-a" { + t.Fatalf("root = %q", got) + } + if _, err := CaseRemoteRoot("/agent-run", "../bad"); !errors.Is(err, ErrInvalidGeneratedPath) { + t.Fatalf("err = %v, want ErrInvalidGeneratedPath", err) + } +} + +func TestValidateMountpointAvailableRejectsNonEmpty(t *testing.T) { + dir := filepath.Join(t.TempDir(), "mnt") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "x"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + if err := ValidateMountpointAvailable(dir); !errors.Is(err, ErrExistingMountpoint) { + t.Fatalf("err = %v, want ErrExistingMountpoint", err) + } +} diff --git a/e2e/agent-harness/repro/known-git-lock.sh b/e2e/agent-harness/repro/known-git-lock.sh new file mode 100755 index 000000000..06eab88e6 --- /dev/null +++ b/e2e/agent-harness/repro/known-git-lock.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail + +drive9-agent-harness run --suite regression --case git-lock-strict,git-lock-interactive "$@" diff --git a/e2e/agent-harness/repro/known-path-edge.sh b/e2e/agent-harness/repro/known-path-edge.sh new file mode 100755 index 000000000..cede6f1e6 --- /dev/null +++ b/e2e/agent-harness/repro/known-path-edge.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -euo pipefail + +drive9-agent-harness run --suite regression --case path-edge-strict,path-edge-interactive "$@" diff --git a/e2e/agent-harness/schemas/case.schema.json b/e2e/agent-harness/schemas/case.schema.json new file mode 100644 index 000000000..058c6e168 --- /dev/null +++ b/e2e/agent-harness/schemas/case.schema.json @@ -0,0 +1,347 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Case Suite", + "type": "object", + "additionalProperties": false, + "properties": { + "defaults": { + "type": "object" + }, + "cases": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "suite", + "expected_outcome", + "remote_root_suffix", + "mountpoint_suffix", + "workload", + "oracles", + "severity" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "suite": { + "enum": [ + "smoke", + "regression", + "stress", + "fault" + ] + }, + "sync_mode": { + "enum": [ + "strict", + "interactive", + "auto" + ] + }, + "expected_outcome": { + "enum": [ + "baseline_pass", + "bug_reproduced", + "fix_verified" + ] + }, + "remote_root_suffix": { + "type": "string", + "pattern": "^[^/\\\\.][^/\\\\]*$" + }, + "mountpoint_suffix": { + "type": "string", + "pattern": "^[^/\\\\.][^/\\\\]*$" + }, + "timeout": { + "type": "string" + }, + "cleanup": { + "enum": [ + "always", + "retain_on_failure", + "never" + ] + }, + "requires_fresh_tenant": { + "type": "boolean" + }, + "workload": { + "type": "object", + "additionalProperties": false, + "required": [ + "type" + ], + "properties": { + "type": { + "enum": [ + "mount_smoke", + "path_matrix", + "dual_mount_visibility", + "fio", + "small_file_storm", + "parallel_writes", + "git_workflow", + "doctor_fuse", + "open_fd_unmount", + "kill_during_write" + ] + }, + "files": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "relative_path", + "content" + ], + "properties": { + "relative_path": { + "type": "string" + }, + "content": { + "type": "string" + } + } + } + }, + "paths": { + "type": "array", + "items": { + "type": "string" + } + }, + "content_template": { + "type": "string" + }, + "read_after_write_timeout": { + "type": "string" + }, + "remount": { + "type": "boolean" + }, + "size_bytes": { + "type": "integer", + "minimum": 0 + }, + "block_bytes": { + "type": "integer", + "minimum": 0 + }, + "file_count": { + "type": "integer", + "minimum": 0 + }, + "parallel_writers": { + "type": "integer", + "minimum": 0 + }, + "writer_bytes": { + "type": "integer", + "minimum": 0 + }, + "read_sample_count": { + "type": "integer", + "minimum": 0 + }, + "min_bytes_per_second": { + "type": "integer", + "minimum": 0 + }, + "fio_binary": { + "type": "string" + }, + "hold_open_duration": { + "type": "string" + }, + "kill_after": { + "type": "string" + }, + "clone_url": { + "type": "string" + }, + "git_binary": { + "type": "string" + }, + "git_timeout": { + "type": "string" + }, + "git_user_name": { + "type": "string" + }, + "git_user_email": { + "type": "string" + }, + "clone_dir": { + "type": "string" + }, + "mutation": { + "type": "object", + "additionalProperties": false, + "required": [ + "path", + "mode", + "content" + ], + "properties": { + "path": { + "type": "string" + }, + "mode": { + "enum": [ + "append", + "overwrite" + ] + }, + "content": { + "type": "string" + } + } + }, + "commit_message": { + "type": "string" + }, + "expected_locks": { + "type": "array", + "items": { + "type": "string" + } + }, + "expected_status": { + "type": "string" + }, + "expected_commit_delta": { + "type": "integer" + }, + "remount_verify_paths": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "path" + ], + "properties": { + "path": { + "type": "string" + }, + "sha256": { + "type": "string" + } + } + } + }, + "expect_exit": { + "type": "integer" + }, + "allow_nonzero_when_no_allow_other": { + "type": "boolean" + } + } + }, + "oracles": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "type" + ], + "properties": { + "type": { + "enum": [ + "fuse_write_success", + "cli_read_equals", + "remount_hash_equal", + "no_git_locks", + "git_status_equals", + "git_commit_count", + "command_exit", + "dual_mount_visibility", + "throughput_min", + "file_count_equals", + "unmount_busy_then_clean", + "recovery_classified" + ] + }, + "path": { + "type": "string" + }, + "expected_bytes": { + "type": "integer", + "minimum": 0 + }, + "command_id": { + "type": "string" + }, + "remote_path": { + "type": "string" + }, + "sha256": { + "type": "string" + }, + "paths": { + "type": "array", + "items": { + "type": "string" + } + }, + "expected_hashes": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "globs": { + "type": "array", + "items": { + "type": "string" + } + }, + "expected_status": { + "type": "string" + }, + "repo_path": { + "type": "string" + }, + "expected_delta": { + "type": "integer" + }, + "expected_exit": { + "type": "integer" + } + } + } + }, + "severity": { + "type": "object", + "additionalProperties": false, + "required": [ + "failure" + ], + "properties": { + "failure": { + "enum": [ + "P0", + "P1", + "P2", + "P3" + ] + } + } + } + } + } + } + }, + "required": [ + "cases" + ] +} diff --git a/e2e/agent-harness/schemas/event.schema.json b/e2e/agent-harness/schemas/event.schema.json new file mode 100644 index 000000000..eb0d63c60 --- /dev/null +++ b/e2e/agent-harness/schemas/event.schema.json @@ -0,0 +1,80 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Event", + "type": "object", + "required": [ + "schema_version", + "run_id", + "case_id", + "ts", + "type", + "message", + "duration_ms", + "command_id", + "exit_code", + "signal", + "artifact_refs" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "case_id": { + "type": "string" + }, + "ts": { + "type": "string" + }, + "type": { + "enum": [ + "run_start", + "run_end", + "case_start", + "case_end", + "command_start", + "command_end", + "mount_start", + "mount_ready", + "mount_end", + "unmount_start", + "unmount_end", + "oracle_start", + "oracle_end", + "timeout", + "artifact_written", + "cleanup_start", + "cleanup_end" + ] + }, + "message": { + "type": "string" + }, + "duration_ms": { + "type": "integer" + }, + "command_id": { + "type": "string" + }, + "exit_code": { + "type": [ + "integer", + "null" + ] + }, + "signal": { + "type": "string" + }, + "artifact_refs": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + } + } + } +} diff --git a/e2e/agent-harness/schemas/failure.schema.json b/e2e/agent-harness/schemas/failure.schema.json new file mode 100644 index 000000000..64a7c9a9e --- /dev/null +++ b/e2e/agent-harness/schemas/failure.schema.json @@ -0,0 +1,78 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Failure", + "type": "object", + "required": [ + "schema_version", + "run_id", + "case_id", + "ts", + "severity", + "class", + "oracle", + "expected_outcome", + "message", + "observed", + "expected", + "repro_path", + "artifact_refs" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "case_id": { + "type": "string" + }, + "ts": { + "type": "string" + }, + "severity": { + "enum": [ + "P0", + "P1", + "P2", + "P3" + ] + }, + "class": { + "enum": [ + "product", + "harness", + "environment", + "infrastructure", + "inconclusive" + ] + }, + "oracle": { + "type": "string" + }, + "expected_outcome": { + "enum": [ + "baseline_pass", + "bug_reproduced", + "fix_verified" + ] + }, + "message": { + "type": "string" + }, + "observed": {}, + "expected": {}, + "repro_path": { + "type": "string" + }, + "artifact_refs": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + } + } + } +} diff --git a/e2e/agent-harness/schemas/gating.schema.json b/e2e/agent-harness/schemas/gating.schema.json new file mode 100644 index 000000000..a0544c8f2 --- /dev/null +++ b/e2e/agent-harness/schemas/gating.schema.json @@ -0,0 +1,55 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Gating", + "type": "object", + "required": [ + "schema_version", + "run_id", + "gate_status", + "pass", + "fail", + "known_bug_reproduced", + "known_bug_fixed_candidate", + "non_gating", + "blocking_failures" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "gate_status": { + "enum": [ + "pass", + "fail", + "non_gating", + "harness_failed" + ] + }, + "pass": { + "type": "integer", + "minimum": 0 + }, + "fail": { + "type": "integer", + "minimum": 0 + }, + "known_bug_reproduced": { + "type": "integer", + "minimum": 0 + }, + "known_bug_fixed_candidate": { + "type": "integer", + "minimum": 0 + }, + "non_gating": { + "type": "integer", + "minimum": 0 + }, + "blocking_failures": { + "type": "array" + } + } +} diff --git a/e2e/agent-harness/schemas/manifest.schema.json b/e2e/agent-harness/schemas/manifest.schema.json new file mode 100644 index 000000000..6d9e91860 --- /dev/null +++ b/e2e/agent-harness/schemas/manifest.schema.json @@ -0,0 +1,135 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Manifest", + "type": "object", + "required": [ + "schema_version", + "run_id", + "started_at", + "host", + "harness_git_sha", + "drive9_version", + "server", + "suites", + "selected_cases", + "cases", + "artifact_root", + "mount_root", + "remote_root_base", + "generated_mountpoints", + "generated_remote_roots", + "process_groups", + "api_key_redacted", + "approval_mode" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "started_at": { + "type": "string" + }, + "host": { + "type": "string" + }, + "harness_git_sha": { + "type": "string" + }, + "drive9_version": { + "type": "string" + }, + "server": { + "type": "string" + }, + "suites": { + "type": "array", + "items": { + "type": "string" + } + }, + "selected_cases": { + "type": "array", + "items": { + "type": "string" + } + }, + "cases": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "suite", + "expected_outcome" + ], + "properties": { + "id": { + + "minLength": 1 + }, + "suite": { + "enum": [ + "smoke", + "regression", + "stress", + "fault" + ] + }, + "expected_outcome": { + "enum": [ + "baseline_pass", + "bug_reproduced", + "fix_verified" + ] + }, + "status": { + "enum": [ + "passed", + "failed", + "inconclusive", + "known_bug_reproduced", + "known_bug_fixed_candidate" + ] + } + } + } + }, + "artifact_root": { + "type": "string" + }, + "mount_root": { + "type": "string" + }, + "remote_root_base": { + "type": "string" + }, + "generated_mountpoints": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "generated_remote_roots": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "process_groups": { + "type": "object", + "additionalProperties": { + "type": "integer" + } + }, + "api_key_redacted": { + "type": "string" + }, + "approval_mode": { + "type": "string" + } + } +} diff --git a/e2e/agent-harness/schemas/metric.schema.json b/e2e/agent-harness/schemas/metric.schema.json new file mode 100644 index 000000000..4cdedf8b6 --- /dev/null +++ b/e2e/agent-harness/schemas/metric.schema.json @@ -0,0 +1,65 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Metric", + "type": "object", + "required": [ + "schema_version", + "run_id", + "case_id", + "ts", + "name", + "value", + "unit", + "source", + "artifact_refs" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "case_id": { + "type": "string" + }, + "ts": { + "type": "string" + }, + "name": { + "enum": [ + "mount_startup_ms", + "command_duration_ms", + "git_clone_duration_ms", + "bytes_written", + "bytes_read", + "file_count", + "mount_perf_counter" + ] + }, + "value": { + "type": "number" + }, + "unit": { + "enum": [ + "ms", + "bytes", + "files", + "count", + "bool" + ] + }, + "source": { + "type": "string" + }, + "artifact_refs": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + } + } + } +} diff --git a/e2e/agent-harness/schemas/summary.schema.json b/e2e/agent-harness/schemas/summary.schema.json new file mode 100644 index 000000000..031ce018f --- /dev/null +++ b/e2e/agent-harness/schemas/summary.schema.json @@ -0,0 +1,95 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Drive9 Agent Harness Summary", + "type": "object", + "required": [ + "schema_version", + "run_id", + "status", + "started_at", + "ended_at", + "duration_ms", + "cases", + "counts", + "artifacts", + "cleanup" + ], + "properties": { + "schema_version": { + "const": "agent-harness.v1" + }, + "run_id": { + "type": "string" + }, + "status": { + "enum": [ + "passed", + "failed", + "known_bugs_reproduced", + "inconclusive", + "harness_failed" + ] + }, + "started_at": { + "type": "string" + }, + "ended_at": { + "type": "string" + }, + "duration_ms": { + "type": "integer" + }, + "cases": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "suite", + "expected_outcome", + "status" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "suite": { + "enum": [ + "smoke", + "regression", + "stress", + "fault" + ] + }, + "expected_outcome": { + "enum": [ + "baseline_pass", + "bug_reproduced", + "fix_verified" + ] + }, + "status": { + "enum": [ + "passed", + "failed", + "inconclusive", + "known_bug_reproduced", + "known_bug_fixed_candidate" + ] + } + } + } + }, + "counts": { + "type": "object" + }, + "artifacts": { + "type": "object" + }, + "cleanup": { + "type": "object" + } + } +} diff --git a/go.mod b/go.mod index 5cc18f6ab..ce26ab4fd 100644 --- a/go.mod +++ b/go.mod @@ -22,6 +22,7 @@ require ( golang.org/x/term v0.42.0 golang.org/x/text v0.36.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 + gopkg.in/yaml.v3 v3.0.1 ) require ( @@ -98,5 +99,4 @@ require ( golang.org/x/sys v0.43.0 // indirect google.golang.org/grpc v1.79.1 // indirect google.golang.org/protobuf v1.36.11 // indirect - gopkg.in/yaml.v3 v3.0.1 // indirect )