A web application that manages logs and shows analytical metrics about them. Senior/lead full-stack engineer assignment — designed and built end-to-end across FastAPI, TimescaleDB, and Next.js with a one-command Docker setup.
Companion docs worth reading first:
Plan.md— the architectural plan, signed off before any code was written. Requirements broken into core / good-to-have / out-of-scope, every item tagged[F]functional or[NF]non-functional, plus a decision log (§3) explaining each choice.SESSION_FLOW.md— chronological audit trail of how AI tools were used while building this. Each session lists the user's prompt, actions taken, decisions logged, gaps surfaced, and files produced. Satisfies the brief's "demonstrate AI-tool usage" and "identify gaps proactively" rubric items.
cp .env.example .env
docker-compose up --build| Service | URL | Notes |
|---|---|---|
| Frontend | http://localhost:3000 | HTTP Basic from .env (default admin / admin) |
| API | http://localhost:8000 | OpenAPI at /docs; click Authorize to use the secret |
| Grafana | http://localhost:3001 | admin / admin; "Logs Overview" auto-provisioned |
| Database | localhost:5432 | TimescaleDB on Postgres 16 |
| Generator | no port | Continuously POSTs synthetic logs to the API |
On first boot:
- The DB starts; the API runs
alembic upgrade head(creates the severity enum, thelogshypertable + indexes, and thelogs_by_hourcontinuous aggregate). - The frontend builds with
next build --output standaloneand serves on port 3000. - The generator waits for
/health, then starts POSTing logs. - Grafana provisions the TimescaleDB datasource and the "Logs Overview" dashboard.
To populate historical data with a visible incident spike:
# Defaults: 7 days, 0.5 logs/sec, ~10% in distributed traces, no spike
docker-compose exec api python scripts/seed.py
# A more dramatic dataset with an incident spike at the 2-day mark
docker-compose exec api python scripts/seed.py \
--rate 5 --duration 7d --error-rate 0.05 \
--spike-at 2d --spike-multiplier 8 --spike-duration 5m \
--trace-fraction 0.1See scripts/seed.py --help for the full option list.
/— landing card with the live API health check./logs— list view. Substring search, severity chips, source filter, exact-matchtrace_idfilter, datetime range, sort, server-driven pagination, CSV export, auto-refresh (Off / 10s / 30s / 1m / 5m)./logs/[id]— detail view. Read-only metadata (id,timestamp,message,created_at); curation form forseverity/source/trace_id; two-step delete confirm./logs/new— manual create form (react-hook-form + zod validation)./dashboard— total / error / critical / top-source cards; stacked trend chart (bucket1m/1h/1d× group-by severity / source / total); severity histogram; top-sources list. All three queries share the filter panel and the auto-refresh interval.
| Layer | Choice | Why |
|---|---|---|
| Backend | FastAPI · SQLAlchemy 2 async · asyncpg · Alembic · structlog | Brief preference; async-native; free OpenAPI; clean DDL diffs. |
| Database | TimescaleDB (Postgres 16 + extension) | Logs are time-series. Hypertables + continuous aggregates change the perf shape. |
| IDs | cuid2 |
K-sortable, opaque, URL-safe, distributed-safe; no leak of row count. |
| Frontend | Next.js 16 (App Router) · React 19 · TypeScript | Brief preference; server components fetch direct, client components via proxies. |
| UI | Tailwind v4 + custom business-named components | Names like <CtaButton>/<DeleteLogButton> survive design-system changes. |
| Charts | Recharts | Declarative; small surface area; theme-friendly. |
| Data fetching | TanStack Query | Powers list pagination, dashboard auto-refresh, and cache invalidation on edit. |
| Forms | react-hook-form + zod | Mirrors Pydantic on the backend. |
| Ops surface | Grafana with provisioned dashboard | Additive to the React dashboard; matches how an ops team would actually look. |
| Tests | pytest + httpx (backend) · Vitest (frontend) | Integration on the API; smoke on the lib utilities. |
| Orchestration | docker-compose | Required deliverable; 5 services (db, api, web, generator, grafana). |
Full reasoning in Plan.md §3. Highlights:
- TimescaleDB rather than vanilla Postgres. Logs are textbook time-series.
The
logstable is a hypertable with a configurable chunk interval (CHUNK_TIME_INTERVALenv var — default1 month, supports2 weeks,1 week, …). The dashboard reads from thelogs_by_hourcontinuous aggregate, not the raw hypertable. - CUID primary keys. Composite PK
(id, timestamp)to satisfy TimescaleDB's "partitioning column must be in any UNIQUE constraint" rule; single-row lookups still hit the PK index via the leading column. PATCH /api/logs/{id}is restricted toseverity/source/trace_id. A log is a fact about something that happened at a time. Only its classification can be curated. Sending any other field returns 400 with a structureddisallowed_fieldspayload.- Two-layer auth.
- Browser ↔ Next.js — HTTP Basic from
BASIC_AUTH_USERNAME/BASIC_AUTH_PASSWORDenv vars, enforced infrontend/middleware.tswith a timing-safe comparison. - Next.js ↔ FastAPI — every server-side fetch from Next.js to FastAPI
carries
X-Service-Secret: ${SERVICE_SECRET}. The secret never reaches the browser because all client fetches go through Next.js route handlers underapp/api/**that inject it server-side.
- Browser ↔ Next.js — HTTP Basic from
- Auto-refresh interval buttons instead of websockets. Values
Off / 10s / 30s / 1m / 5mdriven by TanStack Query'srefetchInterval. trace_idas an optional indexed column for distributed-trace lookups. The generator weaves multi-source traces (2–7 logs across services sharing one CUID) so the dashboard demos are non-empty.- Business-named React components (
<CtaButton>,<ExitButton>,<DeleteLogButton>,<SeverityBadge>,<SourceBadge>,<FilterPanel>, …). Visual names like<PrimaryButton>become lies the moment the design system shifts. - Filter input safety. All SQL is parameterized via SQLAlchemy ORM ops;
user-provided
qis_escape_like'd before ILIKE wildcard matching; unknownsortkeys 422 via aLiteraltype at the route boundary.
scripts/seed.py— direct DB INSERT via psycopg2, for fast historical backfill (hundreds of thousands of rows). Refresheslogs_by_hourat the end so the dashboard reflects new data immediately.generatorservice — asynchttpxclient that POSTs to/api/logs/bulkcontinuously, going through the real ingest path including the shared-secret auth middleware. Drives the dashboard's auto-refresh control. Configurable viaGEN_*env vars (see.env.example).
Disable the live generator with docker-compose stop generator; tweak its
parameters without rebuilding via env vars
(e.g. GEN_RATE=5 GEN_SPIKE_EVERY=5m docker-compose up -d generator).
Order-of-magnitude estimates for the default docker-compose setup
(single 1-vCPU API container + single Postgres-16 / TimescaleDB container).
Measure under real load to confirm — the numbers below are budgeting, not a
benchmark.
| Stage | Approx cost | Notes |
|---|---|---|
| HTTP + JSON parse | ~5 ms | FastAPI/uvicorn on 1 vCPU |
Pydantic validation (1 000 × LogCreate) |
~5–15 ms | One pass per item; AwareDatetime + bounds are cheap |
1 000 × CUID.generate() |
~4 ms | cuid2 ≈ 250 k/s on one core |
1 000 × SQLAlchemy ORM add() |
~10–30 ms | ORM overhead dominates the Python side |
| Batched INSERT (asyncpg, one statement) | ~30–80 ms | One round-trip; updates PK + 4 secondary indexes |
| Commit (WAL fsync) | ~5–15 ms | Local SSD; the dominant cost on slower disks |
| Total | ~60–150 ms | → ~7–15 /bulk requests/sec on one API worker |
| Setup | Path | Sustainable | Soft ceiling |
|---|---|---|---|
| Default compose (1 API container, 1 DB container, ~1 vCPU each) | POST /bulk(1000) |
~7–12 k logs/s | DB CPU saturates around ~15–20 k/s; INSERT latency climbs, asyncpg pool queues |
| Same setup | POST /logs (single) |
~200–400 logs/s | Per-request overhead dominates; fsync-per-commit walls at ~500–800 commits/s |
| API scaled to 4 uvicorn workers (4 vCPU), same DB | POST /bulk(1000) |
~15–25 k logs/s | DB is the bottleneck |
4 API workers + DB on db.m5.large (2 vCPU / 8 GB) + gp3 SSD |
POST /bulk(1000) |
~30–50 k logs/s | Hot indexes live in shared_buffers; fewer disk reads |
The live generator at default settings emits ~0.5–10 logs/s — three orders of magnitude under any of these ceilings, so it is not the thing to worry about.
- DB write CPU — the realistic ceiling at default compose scale. Each
INSERT touches 4 secondary indexes (
timestamp DESC,(severity, ts DESC),(source, ts DESC),trace_id); index maintenance is CPU-bound. Watchpg_stat_user_tables.n_tup_insand host%cpuondb. - DB fsync rate — only relevant if you switch to one commit per single
POST /logs. SATA SSD: ~500 fsyncs/s; NVMe: ~5 k/s. Mitigation: always batch via/bulk, or setsynchronous_commit=off(durability trade-off documented). - Connection pool — asyncpg's default pool is 5. Once they're all busy,
requests queue. At >2 k req/s of single POSTs you'd widen the pool and put
pgbouncer in transaction mode in front of TimescaleDB. Irrelevant at
/bulkvolumes. - API CPU — Pydantic + SQLAlchemy ORM each cost ~20 µs per item. The linear fix is horizontal scaling of uvicorn workers / pods.
- Continuous aggregate refresh —
logs_by_hourrefreshes every 30 min over rows older than 1 h. At demo volumes (≤ a few million rows/hour) the refresh is a sub-secondINSERT … SELECT. Real-time aggregation handles the unmaterialised tail at query time, so the dashboard never blocks on it.
| Component | At rest | Per 1 000-log bulk |
|---|---|---|
api (uvicorn + FastAPI + SQLAlchemy + asyncpg) |
~120–180 MB | +1–2 MB ORM rows, released after commit |
db (TimescaleDB / Postgres 16, default shared_buffers) |
~150–250 MB | grows with WAL + index cache until flushed |
web (Next.js standalone, mostly a proxy) |
~80–150 MB | negligible |
generator |
~40–60 MB | negligible |
The 1 GB API / 2 GB DB limits in docker-compose.yml are comfortable for
demo. For production give the DB ≥ 4 GB and raise shared_buffers to ~25 %
of RAM; the API scales horizontally — keep individual workers small (~500 MB)
and add replicas instead of growing one.
Default
docker-composesetup sustains ~5–10 k logs/sec via/bulk(1000)and starts queueing around 15–20 k/sec where DB CPU saturates updating four secondary indexes. Single-logPOSTs are limited by fsync-per-commit to ~200–400/sec.To push past ~20 k/sec: scale uvicorn horizontally first (linear), give the DB more CPU + RAM second, switch to gp3 / NVMe storage third, and only then tune Postgres internals (shared_buffers, parallel workers) or batch the CAGG refresh.
# Backend integration tests. They TRUNCATE the logs table before each test, so
# stop the live generator first to avoid a race.
docker-compose stop generator
docker-compose exec api pytest -q
# Frontend smoke tests. The production Docker image strips dev deps, so run
# locally against frontend/.
cd frontend && npm install && npm testBackend tests cover: health/ready, 401 paths, CRUD happy paths, the PATCH
restriction (400 with disallowed_fields), trace_id filtering, severity
filtering, bulk insert + size cap, CSV export streaming, pagination
metadata, dashboard endpoint shape, and pure-unit tests on the seed-script
parsers.
Frontend smoke tests cover the lib utilities: formatTimestamp,
formatRelative, buildLogsQuery, exportCsvUrl.
The api and web containers build their source into the image at build
time — there are no host bind mounts (they were generating heavy WSL
vmmem / fsync CPU on Windows). To pick up a change:
docker-compose up --build api # backend change
docker-compose up --build web # frontend change
docker-compose up --build # everythingFirst build is the slow one (pip install / npm install / next build);
subsequent rebuilds reuse cached layers and finish in seconds because
pyproject.toml / package.json rarely change.
backend/ FastAPI app, Alembic migrations, scripts, tests
app/api/ route modules: logs, dashboard, health
app/core/ config, structlog setup, error handlers, auth dep
app/db/ base, session, models
app/schemas/ Pydantic models
app/services/ query/aggregation logic
alembic/versions/ 0001 init + 0002 continuous aggregate
scripts/ seed.py (historical), live_generator.py (continuous)
tests/ pytest integration + seed-helper unit tests
frontend/
app/ Next.js App Router pages + route-handler proxies
components/ business-named React components
lib/ types, api client, zod schemas, format helpers
middleware.ts HTTP Basic gate
grafana/provisioning/ datasource YAML + dashboard JSON
docker-compose.yml db, api, web, generator, grafana
kustomize/ Kubernetes manifests (base + local overlay)
terraform/ AWS deployment (VPC + RDS + ECS Fargate + ALB + EFS)
Plan.md architecture and decision log
SESSION_FLOW.md AI-usage audit trail
CLAUDE.md guidance for future AI-assisted work in this repo
- Kubernetes via Kustomize — see
kustomize/README.md.kubectl apply -k kustomize/overlays/localbrings the same five services up on any cluster (with alocaloverlay that drops the Ingress for port-forward-style access on kind/minikube). - AWS via Terraform — see
terraform/README.md. VPC + RDS (Postgres with TimescaleDB) + ECS Fargate + ALB + EFS + Secrets Manager.terraform applyfrom a clean account stands the whole thing up.
Decided in Plan.md §1.4. Most notably:
- Real-time streaming / websockets — replaced by the auto-refresh interval control.
- Microservices / message brokers, Kubernetes, multi-tenancy, custom chart library — out of scope for a 1–2 day exercise.
- GitHub Actions CI and API-key management for ingestion — explicitly
deferred to keep the scope honest. Both are sketched in
Plan.md§1.3 and §3.9 in case a future iteration wants them.
Public Git repo. Treat the commit history, Plan.md, SESSION_FLOW.md, and
this README as part of the deliverable — not afterthoughts.