Introducing lading MVP scenario

AGENTS.md

@@ -13,6 +13,7 @@ behind these rules, see the [Architecture Decision Records](docs/adr/README.md).
 | Shape rule | 3+ repetitions = create abstraction |
 | Dependencies | When in doubt, implement rather than import |
 | HashMap allowed? | No - use `IndexMap`/`IndexSet` for determinism |
+| Recording a design decision | Write an ADR in `docs/adr/` (see [README](docs/adr/README.md)) |
 
 ## Task Routing
 
@@ -27,6 +28,26 @@ behind these rules, see the [Architecture Decision Records](docs/adr/README.md).
 | Adding a dependency | Evaluate: performance, determinism, control | [008](docs/adr/008-dependency-philosophy.md) |
 | Code style questions | Shape rule, no mod.rs, imports at top | [007](docs/adr/007-code-style-and-abstraction.md) |
 
+## Architecture Decisions
+
+Significant architectural or design decisions are recorded as Architecture
+Decision Records (ADRs) in [`docs/adr/`](docs/adr/), never as ad-hoc design or
+spec documents elsewhere.
+
+When a task involves a design decision worth capturing -- a new subsystem, a
+cross-cutting policy, a non-obvious trade-off:
+
+1. Copy [`docs/adr/000-template.md`](docs/adr/000-template.md) to
+   `docs/adr/NNN-title-with-dashes.md` using the next sequential number.
+2. Fill in every section: Context, Decision, Consequences (positive, negative,
+   neutral), Alternatives Considered, References.
+3. Update the [ADR README](docs/adr/README.md) index and knowledge map.
+
+Do not write standalone design docs, brainstorming specs, or plan files in other
+locations. The ADR is the durable record; a decision that is not in `docs/adr/`
+is not recorded. See the [ADR README](docs/adr/README.md) for the full workflow,
+including how to supersede an existing decision.
+
 ## Architecture Overview
 
 ```
@@ -182,3 +203,4 @@ Override locally: `PROPTEST_CASES=512 ci/test`
 8. Document invariants with formal language in tests/proofs
 9. Consider measurement interference
 10. No mod.rs files, imports at file top only
+11. Record design decisions as ADRs in `docs/adr/`, not ad-hoc spec docs

Cargo.toml

@@ -10,6 +10,7 @@ members = [
   "lading_fuzz",
   "lading_payload",
   "lading_throttle",
+  "test/antithesis/sink",
 ]
 
 [workspace.dependencies]
@@ -71,6 +72,8 @@ datadog-protos = { git = "https://github.com/DataDog/saluki", rev = "f47a7ef588c
 protobuf = { version = "3.7" }
 enum_dispatch = { version = "0.3" }
 antithesis_sdk = { git = "https://github.com/antithesishq/antithesis-sdk-rust", rev = "78c9db56771f0f6b01cb5404765c5a96852c159c", default-features = false } # 0.2.8 is pinned to rand 0.8, rev version allows us to select workspace rand
+antithesis-instrumentation = { version = "0.1" }
+lading-antithesis = { path = "lading_antithesis" }
 
 [workspace.lints.clippy]
 all = "deny"

deny.toml

@@ -18,7 +18,10 @@ unused-allowed-license = "allow"
 
 [sources]
 unknown-git = "deny"
-allow-git = ["https://github.com/DataDog/saluki"]
+allow-git = [
+  "https://github.com/DataDog/saluki",
+  "https://github.com/antithesishq/antithesis-sdk-rust",
+]
 
 [advisories]
 version = 2

docs/adr/009-antithesis-test-harness.md

@@ -0,0 +1,123 @@
+# ADR-009: Antithesis Test Harness Architecture
+
+## Status
+
+**Draft**
+
+## Date
+
+2026-07-23
+
+## Context
+
+lading's integration correctness is currently checked by sheepdog/ducks, under
+`integration/`. sheepdog orchestrates: it builds lading and ducks, spawns ducks,
+tells it over a gRPC-on-unix-socket control plane to listen on a random port,
+templates that port into a lading config, spawns lading, waits for lading to
+exit, pulls counters back from ducks, and asserts "lading pushed load" via
+`request_count > 10` or `total_bytes > 100_000`.
+
+We want to test lading under Antithesis, a deterministic hypervisor that controls
+scheduling and randomness, injects faults, and observes properties through
+in-process SDK assertions. This requires deciding how lading is driven and where
+the "load arrived" claim lives.
+
+Three constraints shape the decision:
+
+1. **Antithesis is the orchestrator.** docker-compose brings containers up. This
+   replaces 'sheepdog'.
+2. **lading is its own load driver.** Unlike a typical system under test that
+   needs a separate workload container to exercise it, driving load is lading's
+   function.
+3. **The observer must be independent of the config under test.** In Antithesis
+   the lading config is the search space. A `sometimes!` assertion that never
+   executes is reported as unsatisfied, so if the claim lived in lading's own
+   blackhole, a healthy config that omits a blackhole would fault the harness.
+
+## Decision
+
+Introduce an Antithesis harness under `test/antithesis/`, mirroring
+Datadog/saluki's layout, with a "general" MVP scenario of three containers:
+
+- **sink** (oracle): the standalone `test/antithesis/sink/` crate that binds a
+  fixed TCP port, counts received bytes, and owns the claim
+  `sometimes!(total_bytes > 0, "sink received bytes")` through the
+  `lading_antithesis` SDK facade. Not faulted. SDK-linked but built without
+  sancov coverage instrumentation.
+- **lading** (system under test): the real lading binary, built
+  `--features antithesis` with sancov coverage instrumentation and
+  `panic="abort"`, run with `--no-target --experiment-duration-infinite` against
+  a hard-wired `lading.yaml`, a tcp generator pointed at the sink, with telemetry
+  exposed via `--prometheus-addr` (lading requires telemetry to be configured).
+  Faulted.
+- **workload** (driver): emits the Antithesis `setup_complete` signal, then
+  idles. lading drives the load, so this container is the seam where
+  config-variation test commands will land later.
+
+In a like manner to saluki we introduce a generic
+`test/antithesis/bin/launch.sh`, driven by per-scenario `launch.env`, tags
+images by git SHA, builds, renders the compose with concrete tags, and submits
+through `snouty launch` with a pinned fault profile. `test/antithesis/CLAUDE.md`
+documents the launcher and the Antithesis skills: launch, triage, and
+setup/research/workload.
+
+The SDK facade `lading_antithesis`, feature-gated and a no-op when the feature
+is off, is the single path both the sink and lading's bootstrap use to reach
+`antithesis_sdk`. lading's instrumentation wiring lives in a feature-gated
+`lading/src/antithesis_hooks.rs`, referenced from `lading/src/bin/lading.rs`. It
+does SDK init plus a panic-reporting hook.
+
+Key sub-decisions:
+
+- **Standalone sink, not lading's blackhole, as the oracle, per constraint 3.**
+  The observer is separate and always-on, as ducks was.
+- **The sink is SDK-linked but uninstrumented.** sancov coverage and the SDK are
+  independent mechanisms. sancov steers exploration by coverage, and that budget
+  must target lading, not the oracle. Instrumenting the sink would add its
+  branches to the coverage surface and pull the search off the system under test.
+  The oracle must be correct by construction: kept minimal and covered by
+  ordinary unit and property tests, so it stays a trusted, fixed instrument
+  rather than something the fuzzer probes. This matches saluki's `tools-builder`
+  stage.
+- **Three containers.** `setup_complete` and future config-variation
+  live in a dedicated workload container rather than being owned by the faulted
+  system under test.
+- **Config hard-wired for the MVP.** Config variation and test commands are
+  deferred to the workload seam.
+
+## Alternatives Considered
+
+### Assert in lading's own blackhole (self-loop, or lading-generator to lading-blackhole)
+
+Rejected: the claim's reachability would depend on the config under test, and a
+config without a blackhole would fault the harness with an unsatisfied
+`sometimes!`. The observer must be independent of the search space.
+
+### Reuse ducks as the Antithesis target
+
+Rejected: ducks carries the sheepdog gRPC control plane it no longer needs, and
+the goal is to replace ducks, not extend it. The sink fills ducks' structural
+role as an independent, always-on receiver, without the orchestration baggage.
+
+### Instrument the sink with sancov
+
+Rejected: coverage should guide exploration into the system under test, not the
+oracle. Instrumenting the sink dilutes the coverage signal and spends exploration
+budget on harness code. The oracle must be bug-free, but that is achieved by
+keeping the sink minimal and testing it conventionally, not by fuzzing it. The
+sink still links the SDK to emit its assertion.
+
+### Two containers (fold `setup_complete` into the system under test)
+
+Rejected: the faulted system under test would own the setup signal, and
+config-variation test commands would have no home. A dedicated workload
+container keeps those concerns separate.
+
+## References
+
+- `lading_antithesis/` - SDK facade over `antithesis_sdk`
+- `test/antithesis/` - harness (to be created)
+- `integration/sheepdog/`, `integration/ducks/` - the mechanism this replaces
+- saluki `test/antithesis/` - pattern source
+- ADR-001: Generator-Target-Blackhole Architecture (the sink is an
+  out-of-process, blackhole-like oracle)

docs/adr/README.md

@@ -22,6 +22,7 @@ helping future contributors understand the reasoning behind the current design.
 | [006](006-testing-strategy.md) | Testing Strategy (Property Tests + Kani) | Accepted |
 | [007](007-code-style-and-abstraction.md) | Code Style and Abstraction Rules | Accepted |
 | [008](008-dependency-philosophy.md) | Dependency Philosophy | Accepted |
+| [009](009-antithesis-test-harness.md) | Antithesis Test Harness Architecture | Draft |
 
 ## Knowledge Map
 
@@ -50,6 +51,13 @@ How to write code for lading:
 - **ADR-007**: Code style and abstraction - The "shape rule" and naive style
 - **ADR-008**: Dependency philosophy - When to implement vs. import
 
+### Testing Infrastructure
+
+How lading is tested end-to-end:
+
+- **ADR-006**: Testing strategy - Property tests and Kani proofs
+- **ADR-009**: Antithesis test harness - Fault-injected exploration and the load-arrived oracle
+
 ### Decision Dependencies
 
 ```
@@ -78,6 +86,7 @@ When working on lading, consult these ADRs based on the task:
 | Modifying throttle | 003, 005, 006 |
 | Adding a dependency | 008 |
 | Writing tests | 006 |
+| Antithesis harness work | 009, 001, 006 |
 | Error handling | 004 |
 | Code style questions | 007 |
 | Performance work | 002, 005 |

lading/Cargo.toml

@@ -21,6 +21,8 @@ lading-capture-schema = { path = "../lading_capture_schema" }
 lading-payload = { version = "0.1", path = "../lading_payload" }
 lading-throttle = { version = "0.1", path = "../lading_throttle" }
 lading-signal = { version = "0.1", path = "../lading_signal" }
+lading-antithesis = { workspace = true }
+antithesis-instrumentation = { workspace = true, optional = true }
 
 anyhow = { workspace = true }
 arrow-array = { workspace = true }
@@ -115,6 +117,7 @@ tonic-prost-build = { workspace = true }
 [features]
 default = []
 logrotate_fs = ["fuser"]
+antithesis = ["lading-antithesis/antithesis", "dep:antithesis-instrumentation"]
 
 [lib]
 doctest = false

lading/src/antithesis_hooks.rs

@@ -0,0 +1,33 @@
+//! Antithesis system-under-test bootstrap for the lading binary.
+//!
+//! Compiled only under the `antithesis` feature. Initializes the Antithesis SDK
+//! and installs a panic hook that reports any panic as an Antithesis
+//! `unreachable!` before the process aborts. All SDK access goes through the
+//! `lading_antithesis` facade, never `antithesis_sdk` directly.
+
+/// Initialize the Antithesis SDK and install a panic-reporting hook.
+///
+/// Call once, as early in startup as possible, before any panic can occur. The
+/// installed hook forwards to the previous hook after reporting, so normal panic
+/// output is preserved.
+pub fn init() {
+    lading_antithesis::init();
+
+    let default_hook = std::panic::take_hook();
+    std::panic::set_hook(Box::new(move |info| {
+        let location = info
+            .location()
+            .map_or_else(String::new, ToString::to_string);
+        let payload = info.payload();
+        let message = payload
+            .downcast_ref::<&str>()
+            .map(|s| (*s).to_string())
+            .or_else(|| payload.downcast_ref::<String>().cloned())
+            .unwrap_or_else(|| "<non-string panic payload>".to_string());
+        lading_antithesis::unreachable!(
+            "lading panicked",
+            { "message": message, "location": location }
+        );
+        default_hook(info);
+    }));
+}

lading/src/bin/lading.rs

@@ -33,6 +33,13 @@ use tokio::{
 use tracing::{Instrument, debug, error, info, info_span, warn};
 use tracing_subscriber::{EnvFilter, util::SubscriberInitExt};
 
+// Pull in the Antithesis coverage-instrumentation runtime shim only when
+// building for antithesis. Load-bearing: the `use ... as _` keeps the shim from
+// being dropped as unused, so its sancov symbols stay linked in the binary.
+#[cfg(feature = "antithesis")]
+// ast-grep-ignore: no-as-imports
+use antithesis_instrumentation as _;
+
 #[derive(thiserror::Error, Debug)]
 enum Error {
     #[error("Target related error: {0}")]
@@ -730,6 +737,10 @@ fn init_tracing(json_output: bool) {
 }
 
 fn main() -> Result<(), Error> {
+    // Initialize the Antithesis SDK and panic hook before anything can panic.
+    #[cfg(feature = "antithesis")]
+    lading::antithesis_hooks::init();
+
     // Two-parser fallback logic until CliFlatLegacy is removed
     let (json_output, args) = match CliWithSubcommands::try_parse() {
         Ok(cli) => match cli.command {
@@ -777,6 +788,10 @@ fn main() -> Result<(), Error> {
     let max_shutdown_delay = Duration::from_secs(args.max_shutdown_delay.into());
     let disable_inspector = args.disable_inspector;
 
+    // Bootstrap probe: proves the SDK is linked and the instrumentation path is
+    // wired. No-op when the `antithesis` feature is off.
+    lading_antithesis::reachable!("lading completed bootstrap");
+
     let runtime = Builder::new_multi_thread()
         .enable_io()
         .enable_time()

lading/src/lib.rs

@@ -11,6 +11,8 @@
 
 use http_body_util::BodyExt;
 
+#[cfg(feature = "antithesis")]
+pub mod antithesis_hooks;
 pub mod blackhole;
 pub(crate) mod codec;
 mod common;

test/antithesis/.gitignore

@@ -0,0 +1,2 @@
+# Rendered compose files produced by bin/launch.sh at submit time.
+scenarios/*/.launch/