V2 phase 2#14
Merged
Merged
Conversation
Replaced the legacy v1 single-event ingest handler with a schema-2.0 validation-only handler mounted at /v1/events. The new path accepts single JSON events, NDJSON batches, and optional gzip encoding, enforces the 1 MB body limit and 256-event batch limit, validates raw decoded event shapes against the embedded v2.0 schema, and returns the structured ingest envelope with accepted, duplicate, rejected, and deprecations fields. Embed the v2 schema into pkg/event/v2 so the ingest binary no longer depends on the docs tree at runtime. Export reusable CompileSchema and ValidateAny helpers and keep the existing Validate API as a compatibility wrapper. Extend ingest metrics with new rejection reasons and the waylog_ingest_batch_size histogram while preserving existing validation and sampling labels. Do not increment waylog_events_accepted_total from this path because Slice 1 only validates and discards; durable WAL persistence and graph merge remain deferred to Slice 2/3. Teach Go and TypeScript SDK transports to count Deprecation/Sunset response headers independently from envelope-body parsing, so empty 2xx responses still surface deprecation signals. Expose the count in SDK stats. Update tests for the new v2 handler, raw-shape schema validation regressions, gzip/body/batch limits, auth/method handling, metrics, SDK header counting, and integration helpers after removing Server.Events. Document the temporary validate-and-discard semantics in README and CHANGELOG.
- Add schema-2.0 raw JSONL WAL writer under internal/eventlog/v2 - Write validated raw event JSON as one JSONL line - Reuse EVENT_LOG_SYNC and EVENT_LOG_MAX_FILE_MB - Rotate with events-YYYYMMDD-HHMMSS.jsonl naming plus suffixes - Expose ActivePath and idempotent Close - Add v2 WAL replay for event_id dedupe warmup - Read events-*.jsonl oldest-to-newest by mtime - Seed the recent event_id LRU before serving HTTP - Skip malformed, missing-ID, and oversized corrupt lines with warnings - Increment waylog_event_dedup_replay_loaded_total once at startup - Add ingestv2 event_id dedupe - Exact recent-ID LRU with default capacity 65536 - Configurable via WAYLOG_V2_DEDUP_CAPACITY - No read-promotion, bounded memory, mutex-safe - Track waylog_event_dedup_cache_size - Make dedupe atomic with WAL write to prevent concurrent duplicate writes - Make /v1/events durable for schema-2.0 ingest - Validate raw event shape against embedded v2 schema - Return duplicate hits without rewriting WAL - Write every accepted event to v2 WAL before responding - Add event_id to dedupe cache only after WAL success - Return plain 503 durability unavailable on WAL failure with no envelope - Increment waylog_events_accepted_total only after WAL success - Increment waylog_events_duplicate_total on dedupe hits - Rewrite /v1/events/validate onto the v2 parser and envelope - Reuse JSON, NDJSON, gzip, body-limit, and schema validation paths - Skip WAL and dedupe mutation - Return the same v2 envelope shape with duplicate always zero - Remove the legacy v1 validate handler and tests - Wire v2 durability into cmd/ingest - Compute EVENT_LOG_V2_DIR from explicit env, EVENT_LOG_DIR/v2, or ./data/eventlog-v2 - Initialize v2 WAL and dedupe before mounting routes - Warm dedupe synchronously before ListenAndServe - Mount /v1/events and /v1/events/validate behind existing write auth - Extend retention pruning to cover both v1 and v2 WAL dirs - Update metrics and docs - Update waylog_events_accepted_total help to durable WAL semantics - Add duplicate, dedupe cache size, and replay-loaded metrics - Add durability_unavailable rejection label for metrics only - Document accepted-as-durable semantics, EVENT_LOG_V2_DIR, and WAYLOG_V2_DEDUP_CAPACITY - Add CHANGELOG entry for Slice 2 behavior and validate response change - Add regression coverage - Dedup capacity, no read-promotion, gauge, and race tests - WAL append, rotation, sync, active path, and close tests - Replay ordering, malformed lines, 1 MB lines, oversized corrupt lines, and empty dir tests - Handler tests for first accept then duplicate, concurrent duplicate single-write, WAL 503, mid-batch failure, validate dry-run, and accepted metric semantics Verification: - go test ./... - go test ./pkg/... - go test -race ./internal/ingest ./internal/ingest/v2 ./internal/eventlog/v2 - git diff --check
- added schema-2.0 RecentIndex for event, trace, service, error-family, and call aggregates - added v2 Projector that derives only from structured event fields - index suppressed events by id and trace without error or call aggregates - derived error families from service, anchor.step, and anchor.error_code triplets - derive calls only from steps[].downstream.service, never from step names - update /v1/events accepted semantics to require WAL write plus projection - typed-decode events after raw schema validation and before WAL/projection - return schema_validation_failed with decode_typed detail on typed decode drift - recover projection panics, increment panic metric, roll back dedupe, and return plain 503 - warn if projector sees a duplicate event_id after dedupe - keep WAL lines intact on projection failure so replay can repair after restart - refactor eventlog/v2 replay into a raw JSONL iterator without ingest imports - add ingestv2 ReplayWAL to validate, typed-decode, warm dedupe, and project from WAL - bound startup replay by GRAPH_HOT_WINDOW with file-level and event-level filtering - add replay skip metrics by reason for malformed, schema-invalid, typed-decode, and stale lines - wire v2 recent index replay before readiness and prune it on the existing retention tick - add metrics for v2 projection, index size, replay projection, replay skips, typed decode failures, projection panics, and per-stage latency - add tests for projection rules, dedupe rollback, replay rebuild, replay skips, pruning, and trace-order preservation
Added the dark-launched schema-2.0 read surface behind WAYLOG_V2_READS. When the
flag is enabled, the ingest server serves v2 reads from the in-memory
RecentIndex instead of the legacy v1 graph for index-direct and derived read
endpoints.
Add index-direct endpoints for /v1/events/{event_id}, /v1/events/search,
/v1/traces/{trace_id}, and /v1/traces/recent with strict query parsing,
opaque cursor pagination, v2 JSON error envelopes, suppressed-event semantics,
and hot-window bounds. RecentIndex now exposes snapshot helpers that copy
pointer slices under RLock so readers can sort/filter outside the index lock.
Add trace ordering and the §5.5 failure-anchor resolver as pure helpers. Trace
ordering uses only structured parent_span_id ↔ steps[].span_id linkage and
never parses step names. The resolver supports suppressed-event exclusion for
story resolution while preserving Slice 4 recent-trace summary behavior.
Add derived v2 read endpoints for /v1/traces/story, /v1/errors, and
/v1/blast_radius. Story responses use explicit snake_case DTOs and derive
anchor metadata, contributing steps, warn/error logs, downstream edges, route,
status, and linkage from the selected event. Error rollups group by
{service, step, error_code}, paginate by count/triplet, and compute affected
trace/user counts plus deterministic sample traces. Blast radius supports
structured triplets, escaped display-form error families, and bare cross-family
error_code queries.
Route all v2 read handlers only when WAYLOG_V2_READS=true. With the flag off,
existing v1 story/search/recent/blast routes remain unchanged and /v1/errors
stays unavailable. Add v2 read latency, empty-result, and not-found metrics
with preinitialized handler labels. Document the feature flag and v2 read
status in README, CHANGELOG, and docs/env.md.
Tests cover cursor handling, query validation, trace ordering, anchor
resolution, index-direct reads, story derivation, suppressed-event behavior,
error-family aggregation, blast-radius keying/counts, route-prefix safety, CORS
preflight, JSON error envelopes, and read metrics.
Replaced cmd/waylog's snapshot/LLM-driven CLI with the Phase 2 schema-v2 operator CLI. The new CLI talks to the running ingest server's v2 read APIs and implements the §6.2 operator verbs: - waylog errors - waylog trace - waylog explain - waylog blast - waylog search Add internal/cli/v2 with command dispatch, an HTTP read client, capability preflight, typed API/transport errors, deterministic exit codes, text renderers, JSON output, and unit tests. The CLI reads INGEST_ADDR, WAYLOG_READ_KEY, and WAYLOG_CLI_TIMEOUT, supports global --addr, --api-key, --timeout, and --json, and normalizes bare host:port addresses. Add /v1/capabilities.v2_reads.enabled, wired from WAYLOG_V2_READS, so the CLI can fail early with a clear message when the server has not enabled v2 reads. Keep WAYLOG_V2_READS default false. Add pkg/api/v2 as the shared schema-v2 read API contract package. Move shared response DTOs, linkage/view-mode constants, and error-family display parse/format helpers there so ingest and CLI compile against the same JSON shapes instead of duplicating structs. Land the Slice 6 cleanup prologue: - add eventv2.Status.IsFailed() - add StepStatus and LogLevel constants - replace local failed-status predicates in ingest/v2 - centralize error-family round-trip behavior - restore include_suppressed=true behavior for suppressed-only recent traces Update README, CHANGELOG, and docs/env.md to document the v2 CLI and required server capability. Verification: - go vet ./... - go build ./... - go test ./... - go test -race ./internal/cli/v2 ./internal/ingest/v2 - make build - make test - git diff --check
Rewrite the local reference demo as an event-driven schema-2.0 mini-commerce flow using the Go v2 SDK. The demo services now emit via pkg/waylog/v2 and pkg/waylog/http during real HTTP request flows across api-gateway, checkout, db, and payment. The demo supports three deterministic scenarios: happy, payment_502, and suppressed_payment_502. Key behavior: - api-gateway exposes GET /demo and POST /purchase. - checkout is the primary narrative event for payment_502. - checkout records db.load_cart and payment.charge steps. - payment.charge carries downstream linkage to payment. - checkout records retry/failure logs and anchor payment.charge:PMT_502. - suppressed_payment_502 emits suppressed events for the known issue path. - payment returns HTTP 502 without stealing the trace-story failure anchor, keeping checkout as the first observable failing step. Remove Kafka and cmd/bridge from the local demo path. make demo and make micro-demo now start ingest plus the four demo services with WAYLOG_V2_READS=true. Kafka targets and bridge code remain available for legacy paths and are not deleted in this slice. Update the demo UI with scenario buttons and accessible interaction basics. Update the smoke script, README, docs/env.md, and CHANGELOG to describe the schema-2.0 demo flow and v2 CLI investigation path. Add v2-focused microdemo tests using SDK local Output buffers. The tests pin checkout event shape, payment_502 narrative behavior, suppressed header-only events, downstream linkage, logs, route/user/demo fields, and the gateway UI. Verification: - make build - make test - go test -race ./examples/microdemo ./pkg/waylog/v2/... - go vet ./... - git diff --check - manual live smoke with waylog errors/explain/blast
- replaced the default demo path with a local no-Docker v2 showcase - started ingest plus api-gateway, checkout, payment, and db via make demo - isolated demo state under data/demo-state and add demo-stop cleanup - rewrite the microdemo UI as the primary scenario trigger surface - add payment outage, cart not found, checkout 500, happy, and suppressed scenarios - add bounded traffic burst to generate realistic mixed trace volume - enrich checkout traces with validation, cart, inventory, payment, and order steps - emit schema-2.0 events through the Go v2 SDK with real downstream propagation - rebuild the dashboard as an errors → explain → blast v2 triage surface - add dark/light theme, KPI strip, recent requests, and inline mini-graphs - remove v1 dashboard ask/explain proxies and vendored chart/topology bundles - update README, env docs, Makefile, and demo scripts for the new flow - add dashboard, demo UI, burst, scenario, and recent-trace coverage Verification: - make build - make test - go test -race ./examples/microdemo ./internal/dashboard ./internal/ingest/v2 - go vet ./... - git diff --check
Made OTLP/HTTP traces a first-class schema-2.0 emitter path. - added IngestRaw to the v2 ingest handler so non-HTTP emitters can reuse the same validation, dedupe, WAL, projection, metrics, and envelope logic as POST /v1/events - rewrote OTLP span conversion to emit schema-2.0 events instead of v1 WideEvents, including deterministic event IDs, HTTP metadata, anchors, failed steps, downstream service hints, and error logs - route /v1/otlp/v1/traces through the v2 ingest handler and remove the old v1 OTLP pipeline wiring from cmd/ingest - add OTLP-to-v2 read API integration coverage for errors, trace story, blast radius, and capabilities - update OTLP handler tests for v2 partial-success/rejection behavior - update the standalone e2e OTLP example to emit HTTP span attributes accepted by the v2 converter - clarify README/internal comments so OTLP is documented as feeding the schema-2.0 read APIs, not the old graph path Verification: - GOCACHE=/tmp/go-build go test ./internal/ingest/v2 - GOCACHE=/tmp/go-build go test ./internal/otel/... - GOCACHE=/tmp/go-build go test ./tests/integration - GOCACHE=/tmp/go-build go test ./pkg/event/v2 ./pkg/waylog/v2 ./pkg/waylog/v2/parity - cd packages/waylog-ts && npm test -- --run parity - GOCACHE=/tmp/go-build go test ./... - GOCACHE=/tmp/go-build go vet ./... - git diff --check
Completed the operator CLI as the terminal entrypoint for the v2 read APIs. - added `waylog capabilities` so operators can inspect server support without requiring v2-read gating - added `waylog recent` over `/v1/traces/recent` with window, service, status, limit, cursor, and include-suppressed filters - add `waylog event <event_id>` over `/v1/events/<event_id>` - add client methods and response/param types for recent traces, event lookup, and expanded capability output - add human renderers for capabilities, recent trace summaries, and single event summaries - keep JSON output behavior consistent with existing verbs - update CLI help with the recommended triage loop: recent -> errors -> blast -> explain - update README CLI examples to include capabilities, recent, event, and display-family blast usage - add command and renderer tests for new verbs, query serialization, v2-read gating, ungated capabilities, readable output, and cursor rendering Verification: - GOCACHE=/tmp/go-build go test ./internal/cli/v2 - GOCACHE=/tmp/go-build go test ./cmd/waylog ./internal/cli/v2 - GOCACHE=/tmp/go-build go test ./... - GOCACHE=/tmp/go-build go vet ./... - git diff --check
Fixed `waylog event <event_id>` to match the actual v2 read API response shape.
- decoded `/v1/events/{id}` as `{"event": {...}}` instead of a raw event body
- added a small internal response wrapper for event lookup
- update the CLI event command test to assert the real server envelope shape
- Added schema-2.0 WAL replay equivalence coverage across recent, errors, story, blast, event, and trace reads. - Assert replay warms event_id dedupe so replayed events are not accepted again. - Add mixed NDJSON durability coverage proving invalid entries are rejected while accepted entries are durable and readable. - Strengthen suppressed-event invariants for search, errors, and blast radius. - Add prune/read coverage to ensure old traces disappear while fresh cursor ordering remains valid. - Added scripts/demo-acceptance.sh and make demo-acceptance to verify the live demo path: burst traffic, capabilities, recent, errors, blast, explain, and event. - Fixed CLI parsing for positional commands with trailing flags in blast and search. - Aligned README, env docs, internals docs, and SDK contract docs with the current schema-2.0 demo/operator flow. - Removed obsolete e2e-mark2 and v1 e2e emitter support path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the schema-2.0 ingest/runtime flow, recent-index read APIs, the v2 operator CLI, OTLP-to-v2 ingestion, and the new product demo/dashboard path. It also hardens the Go and TypeScript SDK surfaces, updates the docs/OpenAPI contract, and adds acceptance coverage for the demo flow.
What changed
/v1/eventsingest for v2 with dedupe, WAL, replay, projection, and read modelserrorstraceexplainblastsearchWhy
Before this branch, v2 existed in pieces. After this branch, there is a coherent operator path:
Testing
Notes
make demo