diff --git a/README.md b/README.md index c729f58..eb97393 100644 --- a/README.md +++ b/README.md @@ -1,243 +1,146 @@ -# FieldTrack 2.0 +# FieldTrack API -> Production-grade multi-tenant backend for real-time field workforce tracking — attendance, GPS, expense management, and admin analytics. +Production backend for FieldTrack workforce operations. -[![CI](https://github.com/fieldtrack-tech/api/actions/workflows/deploy.yml/badge.svg)](https://github.com/fieldtrack-tech/api/actions/workflows/deploy.yml) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -[![Node.js](https://img.shields.io/badge/node-%3E%3D24-brightgreen)](https://nodejs.org) -[![TypeScript](https://img.shields.io/badge/TypeScript-5.9-blue)](https://www.typescriptlang.org) - ---- +This service is implemented with Fastify + TypeScript, Supabase (Postgres), Redis, and BullMQ workers. ## Overview -FieldTrack 2.0 is a production-ready REST API for managing field workforce operations. It provides secure, multi-tenant APIs for tracking employee attendance, real-time GPS location, expense workflows, and aggregate analytics. - -**Boundaries:** This repository is the API only. Infrastructure (nginx, monitoring stack, VPS provisioning) lives in the infra repository. - ---- - -## Features - -- **Multi-tenant isolation** — every query is scoped to the authenticated organization; cross-tenant access is architecturally impossible -- **Attendance sessions** — check-in / check-out lifecycle with state machine enforcement -- **Real-time GPS ingestion** — single and batch endpoints (up to 100 points), idempotent upsert, per-user rate limiting -- **Async distance calculation** — BullMQ background worker computes Haversine distance after check-out; never blocks the HTTP response -- **Expense workflow** — PENDING → APPROVED / REJECTED lifecycle, with re-review guard -- **Admin analytics** — org-wide summaries, per-user breakdowns, configurable leaderboard -- **Redis-backed rate limiting** — per-JWT-sub limits survive corporate NAT and horizontal scaling -- **Security** — Helmet, CORS, Redis rate limiter, brute-force detection -- **Distributed tracing** — OpenTelemetry → OTLP; trace IDs injected into every Pino log line -- **Blue-green zero-downtime deployments** — nginx upstream swap, health-check gate, 5-SHA rollback history -- **Full test suite** — Vitest unit + integration coverage; CI blocks deploy on failure - ---- - -## Tech Stack - -| Layer | Technology | -|-------|------------| -| **Runtime** | Node.js 24 (Debian slim / distroless) | -| **Language** | TypeScript 5.9 (strict, ESM) | -| **Framework** | Fastify 5 | -| **Database** | PostgreSQL via [Supabase](https://supabase.com) | -| **Auth** | JWT (`@fastify/jwt`) — Supabase-issued tokens | -| **Job Queue** | [BullMQ](https://docs.bullmq.io/) + Redis | -| **Validation** | [Zod 4](https://zod.dev/) | -| **Tracing** | OpenTelemetry (OTLP export) | -| **Security** | `@fastify/helmet` · `@fastify/cors` · `@fastify/rate-limit` · `@fastify/compress` | -| **Testing** | [Vitest](https://vitest.dev/) | -| **CI/CD** | GitHub Actions → GHCR → Blue-Green VPS Deploy | - ---- - -## Local Development - -**Prerequisites:** Node.js ≥ 24, npm, a running Redis instance, a Supabase project - -```bash -# Install dependencies -npm install - -# Configure environment -cp .env.example .env -# Edit .env — fill in SUPABASE_URL, keys, REDIS_URL, and CORS_ORIGIN - -# Start in development mode (hot reload) -npm run dev -``` - -The API will start on `http://localhost:3000`. - ---- +The API provides multi-tenant endpoints for: -## Environment Variables +- authentication and identity (`/auth/*`) +- attendance and location tracking +- expenses and approvals +- admin dashboards, analytics, and operational tooling +- webhook and API key management -All variables are validated at startup by `src/config/env.ts` (Zod schema, fail-fast). +All tenant data access is scoped by `organization_id` derived from authenticated identity. -### URLs +## Architecture -| Variable | Required | Purpose | -|----------|:---:|---------| -| `API_BASE_URL` | ✅ | Canonical public URL of this API (`https://…`, no trailing slash) | -| `APP_BASE_URL` | ✅ | Root URL of the application — used in email footers and redirects | -| `FRONTEND_BASE_URL` | ✅ prod | URL of the web frontend — used to build email links | +Core stack: -### Runtime +- Fastify 5 + TypeScript (ESM) +- Supabase Postgres clients (anon + service role) +- Redis + BullMQ +- OpenTelemetry + Prometheus metrics +- Zod validation -| Variable | Required | Default | Purpose | -|----------|:---:|---------|---------| -| `CONFIG_VERSION` | ✅ | `"1"` | Schema version guard — must be `"1"` | -| `APP_ENV` | ✅ | `development` | Application environment — drives all app-level logic | -| `PORT` | ✅ | `3000` | Container listen port | +Runtime layout: -### Auth & Data +- `src/routes`: route registration and system routes +- `src/modules`: domain modules (auth, attendance, expenses, analytics, admin, etc.) +- `src/middleware`: auth and role guards +- `src/db`: tenant-scoped query helpers +- `src/workers`: queue workers and scheduled jobs -| Variable | Required | Purpose | -|----------|:---:|---------| -| `SUPABASE_URL` | ✅ | Supabase project URL | -| `SUPABASE_ANON_KEY` | ✅ | Supabase public/anon key | -| `SUPABASE_SERVICE_ROLE_KEY` | ✅ | Service role key — bypasses RLS, never expose to clients | -| `SUPABASE_JWT_SECRET` | ✅ | JWT signing secret (≥ 32 chars, HS256) | -| `REDIS_URL` | ✅ | Redis connection URL (`redis://` or `rediss://`) | +Workers started by `src/workers/startup.ts`: -### Security +- distance worker +- analytics worker +- webhook worker +- snapshot worker -| Variable | Required in Prod | Default | Purpose | -|----------|:---:|---------|---------| -| `CORS_ORIGIN` | ✅ | `""` | Comma-separated allowed CORS origins. Empty activates localhost fallback in dev | -| `METRICS_SCRAPE_TOKEN` | ✅ | — | Token required to scrape `/metrics`. Unset = open in dev/test | -| `TEMPO_ENDPOINT` | — | `http://tempo:4318` | OTLP HTTP endpoint for trace export | +Scheduled jobs: -> **Observability variables (`METRICS_SCRAPE_TOKEN`, `TEMPO_ENDPOINT`) are optional for standalone operation.** The API starts and handles requests without them. `METRICS_SCRAPE_TOKEN` gates the `/metrics` endpoint (unset = endpoint is open, safe in dev/test). `TEMPO_ENDPOINT` controls where traces are exported; if the Tempo collector is unreachable, traces are silently dropped with no impact to request handling. The monitoring stack that scrapes these endpoints is managed in the [infra repository]. +- snapshot reconciliation job (`reconcile_snapshot_tables()` every 5 minutes) +- retry-intent cleanup job ---- +## Key Features -## Scripts +- JWT and API key auth (`Authorization: Bearer ...` or `X-API-Key`) +- strict role-based authorization (`ADMIN`, `EMPLOYEE`) +- attendance lifecycle (`check-in` and `check-out`) with async post-processing +- location ingest (single + batch) +- expense lifecycle with admin review and CSV export +- admin SSE stream at `/admin/events` +- queue/system/internal admin observability endpoints -| Command | Purpose | -|---------|---------| -| `npm run dev` | Start development server with hot reload | -| `npm run typecheck` | TypeScript type check (no emit) | -| `npm test` | Run full test suite (Vitest) | -| `npm run build` | Compile TypeScript to `dist/` | -| `npm start` | Start compiled production server | -| `./scripts/deploy.sh ` | Blue-green deploy a specific image SHA | -| `./scripts/deploy.sh --rollback` | Interactive rollback to previous SHA | -| `./scripts/deploy.sh --rollback --auto` | Non-interactive rollback (CI) | +## Environment Setup ---- +Environment is validated by `src/config/env.ts` at startup. -## Health Endpoints +Required baseline variables: -| Endpoint | Purpose | Deploy Gate | -|----------|---------|-------------| -| `GET /health` | Liveness check — returns `{"status":"ok"}` once the server bootstraps | **YES** — used by deploy.sh and CI | -| `GET /ready` | Dependency check — verifies Redis and Supabase connectivity | NO — informational only, not a deploy gate | +- `APP_ENV` +- `PORT` +- `SUPABASE_URL` +- `SUPABASE_ANON_KEY` +- `SUPABASE_SERVICE_ROLE_KEY` +- `SUPABASE_JWT_SECRET` +- `REDIS_URL` +- `API_BASE_URL` +- `APP_BASE_URL` +- `CORS_ORIGIN` -`/health` returns 200 after server bootstrap regardless of dependency status. `/ready` failing does not block a deployment; a degraded-but-running API is preferred over a stuck deploy. +Common optional variables: ---- +- `FRONTEND_BASE_URL` +- `METRICS_SCRAPE_TOKEN` +- `TEMPO_ENDPOINT` +- `WORKERS_ENABLED` -## Deployment Overview +## Local Run -> **First-deployment requirement:** The API container joins `api_network`. On a fresh VPS, **nginx** (reverse-proxy) and **Redis** must already be running and attached to that network via the infra repository before the first `deploy.sh` run. Subsequent deploys are fully self-contained. +Prerequisites: -## Infra Requirement +- Node.js 24+ +- npm +- Redis +- Supabase project -This API requires an external infra repository. +Commands: -Expected on server (all under **`INFRA_ROOT=/opt/infra`**): -- `$INFRA_ROOT/docker-compose.nginx.yml` — operator runs nginx from here -- `$INFRA_ROOT/docker-compose.redis.yml` — operator runs Redis from here -- `$INFRA_ROOT/nginx/live`, `nginx/backup`, `nginx/api.conf` — layout enforced by `deploy.sh` and readiness check -- nginx container on `api_network`; Redis at `redis:6379` on `api_network` - -Deployments run automatically via GitHub Actions on every push to `master` (after CodeQL scan passes). - -``` -CodeQL deep scan (master) - → validate (typecheck + audit) ──┐ - → test-api ─────────────────────┼──► build-scan-push ──► vps-readiness-check ──► deploy - ┘ │ - api-health-gate ◄────────────────┘ - │ - health-and-smoke ──► rollback (on failure) -``` - -**Blue-green strategy:** The VPS always runs two containers (`api-blue`, `api-green`). On each deploy, the inactive slot is updated and nginx is reloaded to point at it. The previous slot is stopped only after the health gate passes. - -**nginx is managed by the infra repository.** The API container joins `api_network`; nginx is expected to already be running and configured. - -**Manual deploy:** -```bash -./scripts/deploy.sh -``` - -**Rollback:** ```bash -./scripts/deploy.sh --rollback +npm install +npm run dev ``` -See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for full deployment details. +Useful scripts: ---- +- `npm run dev` +- `npm run build` +- `npm start` +- `npm run typecheck` +- `npm run lint` +- `npm test` -## Project Structure +## API Surface -``` -api/ -├── src/ # Application source -│ ├── modules/ # Domain modules (attendance · locations · expenses · analytics) -│ ├── plugins/ # Fastify plugins (JWT · metrics · security) -│ ├── workers/ # BullMQ distance calculation worker -│ ├── middleware/ # Auth + role guard -│ └── utils/ # Shared utilities (errors · response · tenant) -├── tests/ # Vitest unit and integration tests -├── scripts/ # Deploy, rollback, and utility scripts -├── docs/ # Project documentation -└── .github/workflows/ # GitHub Actions CI/CD -``` +System endpoints: -> The web frontend is in a separate repository: [fieldtrack-tech/web](https://github.com/fieldtrack-tech/web) -> Infrastructure (nginx, monitoring, VPS setup) is in a separate infra repository. +- `GET /` +- `GET /health` +- `GET /ready` +- `GET /metrics` +- `GET /openapi.json` +- `GET /docs` ---- +Full endpoint contract and examples: `docs/API_REFERENCE.md`. -## Documentation +## Deployment Summary -| Document | Description | -|----------|-------------| -| [Architecture](docs/ARCHITECTURE.md) | System design, component diagrams, data flows | -| [API Reference](docs/API_REFERENCE.md) | All endpoints, auth requirements, request/response schemas, error codes | -| [Deployment Guide](docs/DEPLOYMENT.md) | VPS provisioning, CI/CD setup, blue-green deploy, troubleshooting | -| [Rollback System](docs/ROLLBACK_SYSTEM.md) | Rollback architecture, deployment history, safety features | -| [Rollback Quick Reference](docs/ROLLBACK_QUICKREF.md) | Fast operator reference card | -| [Environment Contract](docs/env-contract.md) | All environment variables, naming rules | -| [Infra Contract](docs/infra-contract.md) | External infra responsibilities and path contract (`INFRA_ROOT`) | -| [Changelog](CHANGELOG.md) | Full history of every phase | -| [Contributing](CONTRIBUTING.md) | Contribution workflow, branching, code conventions | -| [Security Policy](SECURITY.md) | How to report vulnerabilities | +Production deploy is driven by GitHub Actions and `scripts/deploy.sh`. ---- +High-level flow: -## Contributing +1. CodeQL deep scan gate +2. validate + tests +3. docker build, vulnerability scan, push to GHCR +4. VPS readiness check +5. blue-green deploy +6. health and smoke checks +7. rollback on failure -See [CONTRIBUTING.md](CONTRIBUTING.md) for setup instructions, branch naming conventions, and commit format. +Liveness gate is `/health` (not `/ready`). -**Branch naming:** -``` -feature/ # new functionality -fix/ # bug fixes -docs/ # documentation -test/ # test additions -chore/ # maintenance / deps -``` +See `docs/DEPLOYMENT.md` for detailed CI/CD and rollback behavior. -**Commit format:** -``` -type(scope): short imperative description -``` -Allowed types: `feat` `fix` `refactor` `ci` `docs` `test` `chore` +## Documentation -All PRs require review from CODEOWNERS and must pass CI before merge. +- `docs/API_REFERENCE.md` +- `docs/ARCHITECTURE.md` +- `docs/DEPLOYMENT.md` +- `docs/ROLLBACK_SYSTEM.md` +- `docs/env-contract.md` +- `docs/infra-contract.md` diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md index f2f812e..52e37a9 100644 --- a/docs/API_REFERENCE.md +++ b/docs/API_REFERENCE.md @@ -1,676 +1,295 @@ -# FieldTrack 2.0 — API Reference +# FieldTrack API Reference -Complete reference for all HTTP endpoints exposed by the backend. +Implementation-aligned contract reference for the current API. ---- +For generated OpenAPI, use: -## Interactive Documentation +- `GET /openapi.json` +- `GET /docs` -**Phase 19: OpenAPI Integration** +## Authentication Model -The API now provides interactive documentation via **Swagger UI** and a machine-readable **OpenAPI 3.0 specification**. +Supported auth methods: -### Accessing Documentation +- JWT bearer token: + - `Authorization: Bearer ` +- API key (for allowed scopes only): + - `X-API-Key: ` -| Resource | URL | Description | -|----------|-----|-------------| -| **Swagger UI** | `/docs` | Interactive API explorer with request/response examples | -| **OpenAPI Spec** | `/openapi.json` | JSON schema for API contract (OpenAPI 3.0) | +Identity claims used by the API: -### Using Swagger UI +- `sub` +- `organization_id` +- `role` (`ADMIN` or `EMPLOYEE`) -1. Navigate to `http://localhost:4000/docs` (development) or `https://api.fieldtrack.app/docs` (production) -2. Click **Authorize** button in the top right -3. Enter your JWT token in the format: `Bearer ` -4. Click **Authorize** to save -5. All subsequent requests will include the authentication header +## Standard Response Envelope -### Example cURL Command - -```bash -curl -X POST http://localhost:4000/attendance/check-in \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" -``` - -### API Tags - -Endpoints are organized by the following tags: - -- **health** — Health check and system status endpoints -- **attendance** — Attendance tracking and session management -- **locations** — Location tracking and route calculation -- **expenses** — Expense reporting and management -- **analytics** — Business analytics and reporting (ADMIN only) -- **admin** — Administrative operations (ADMIN role required) - ---- - -## Authentication - -All endpoints except the public health/metrics routes require a Supabase-issued JWT passed as a Bearer token. - -``` -Authorization: Bearer -``` - -The JWT payload must include these claims (validated by Zod on every request): - -| Claim | Type | Description | -|-------|------|-------------| -| `sub` | `string` (UUID) | User identity — used as the primary actor identifier | -| `organization_id` | `string` (UUID) | Tenant identifier — enforced on every data query | -| `role` | `"EMPLOYEE"` \| `"ADMIN"` | Determines access to protected endpoints | - ---- - -## Standard Error Response - -All error responses share this structure: +Success: ```json { - "success": false, - "error": "Human-readable error message", - "requestId": "uuid-of-this-request" + "success": true, + "data": {} } ``` -Validation errors (`400`) include `details`: +Error: ```json { "success": false, - "error": "Validation failed", - "details": [{ "path": ["field"], "message": "must be a valid UUID" }], - "requestId": "..." + "error": "Human readable message", + "requestId": "uuid" } ``` -### Common Status Codes - -| Code | Meaning | -|------|---------| -| `200` | Success | -| `201` | Resource created | -| `400` | Validation failure or business rule violation | -| `401` | Missing or invalid JWT | -| `403` | JWT valid but role insufficient | -| `404` | Resource not found | -| `429` | Rate limit exceeded (`{ success: false, error: "Too many requests", retryAfter: "Ns" }`) | -| `500` | Unexpected server error | - ---- - -## Response Headers - -Every response includes: - -| Header | Value | -|--------|-------| -| `x-request-id` | UUID generated per-request (matches `requestId` in error bodies) | +Common HTTP status codes: ---- +- `200` success +- `201` created +- `202` accepted / queued +- `204` no content +- `400` validation/business rule error +- `401` unauthenticated +- `403` forbidden +- `404` not found +- `409` conflict +- `410` removed/deprecated endpoint +- `429` rate limited +- `500` internal error -## System Routes +Example typed error codes used in responses include values such as `VALIDATION_ERROR`, `INVALID_CREDENTIALS`, and route-specific domain codes. -### `GET /health` +## System Endpoints -Public health check. No authentication required. +- `GET /` service identity +- `GET /health` liveness +- `GET /ready` dependency readiness +- `GET /metrics` Prometheus/OpenMetrics +- `GET /openapi.json` OpenAPI spec +- `GET /docs` Swagger UI -**Response `200`:** -```json -{ "status": "ok", "timestamp": "2026-03-10T12:00:00.000Z" } -``` - ---- - -### `GET /metrics` - -Prometheus scrape endpoint. Returns metrics in OpenMetrics text format. - -- No authentication required -- Scraped automatically by Prometheus every 15 s -- Required response format: `Content-Type: application/openmetrics-text` (for exemplar support) +## Auth Endpoints ---- +### POST /auth/login -### `GET /internal/metrics` +Body: -Internal operational snapshot. Requires JWT + **ADMIN** role. - -**Response `200`:** ```json { - "uptimeSeconds": 3600, - "queueDepth": 2, - "totalRecalculations": 1540, - "totalLocationsInserted": 287430, - "avgRecalculationMs": 42.7 + "email": "user@example.com", + "password": "password" } ``` -| Field | Description | -|-------|-------------| -| `uptimeSeconds` | Seconds since process start | -| `queueDepth` | Sessions currently waiting in the BullMQ worker queue | -| `totalRecalculations` | Cumulative completed distance recalculations since last restart | -| `totalLocationsInserted` | Cumulative GPS points written (after deduplication) since last restart | -| `avgRecalculationMs` | Rolling average recalculation latency (last 100 jobs) | - ---- - -### `GET /debug/redis` - -**Development / staging only.** Disabled in production (`NODE_ENV=production` returns 404). - -Pings Redis via the BullMQ connection and produces an OTel span (visible in Tempo service graph). - -**Response `200`:** -```json -{ "status": "ok", "redis": "PONG" } -``` - -**Response `503`:** -```json -{ "status": "error", "redis": "unreachable" } -``` - ---- - -## Attendance Module - -All attendance endpoints require JWT authentication. ADMIN routes additionally require `role: "ADMIN"`. - ---- - -### `POST /attendance/check-in` - -Start a new attendance session. Creates a new record with `checked_in_at = now()` and `status = "ACTIVE"`. +Response: -**Auth:** Any authenticated user - -**Request body:** None - -**Response `201`:** ```json { "success": true, "data": { - "id": "uuid", - "employee_id": "uuid", - "organization_id": "uuid", - "checked_in_at": "2026-03-10T08:00:00.000Z", - "checked_out_at": null, - "status": "ACTIVE" + "access_token": "...", + "refresh_token": "...", + "token_type": "bearer", + "expires_in": 3600, + "expires_at": 1741686400 } } ``` -**Error `400`:** `"Cannot check in: you already have an active session. Check out first."` — thrown by `EmployeeAlreadyCheckedIn` - ---- +### GET /auth/me -### `POST /attendance/check-out` +Auth required. -Close the caller's active session. Sets `checked_out_at = now()`, `status = "CLOSED"`, and enqueues a BullMQ job to recalculate distance and duration. +Response: -**Auth:** Any authenticated user - -**Request body:** None - -**Response `200`:** ```json { "success": true, "data": { - "id": "uuid", - "employee_id": "uuid", - "organization_id": "uuid", - "checked_in_at": "2026-03-10T08:00:00.000Z", - "checked_out_at": "2026-03-10T17:00:00.000Z", - "status": "CLOSED" + "id": "uuid-or-api-key-sub", + "email": "user@example.com", + "role": "ADMIN", + "orgId": "organization-uuid" } } ``` -**Error `400`:** `"Cannot check out: no active session found. Check in first."` — thrown by `SessionAlreadyClosed` - ---- +## Employee Endpoints -### `POST /attendance/:sessionId/recalculate` +### Attendance -Manually trigger an async distance/duration recalculation for a specific session. Useful after data corrections or for debugging. +- `POST /attendance/check-in` (EMPLOYEE) +- `POST /attendance/check-out` (EMPLOYEE) +- `POST /attendance/:sessionId/recalculate` (authenticated, rate-limited) +- `GET /attendance/my-sessions?page&limit&status` +- `GET /attendance/org-sessions` is removed and returns `410`; use admin sessions endpoint. -**Auth:** Any authenticated user -**Rate limit:** 5 requests per 60 seconds per JWT `sub` +Session status filter values: -**Path params:** +- `all` +- `active` +- `recent` +- `inactive` -| Param | Type | Required | -|-------|------|----------| -| `sessionId` | UUID | Yes | +Example response item: -**Request body:** None - -**Response `202`:** -```json -{ "success": true, "queued": true } -``` - -**Error `404`:** Session not found or does not belong to the caller's organization. - ---- - -### `GET /attendance/my-sessions` - -List the caller's own attendance sessions, newest first. - -**Auth:** Any authenticated user -**Query params:** - -| Param | Type | Default | Constraints | -|-------|------|---------|-------------| -| `page` | integer | `1` | min 1 | -| `limit` | integer | `20` | min 1, max 100 | - -**Response `200`:** ```json { - "success": true, - "data": [ - { - "id": "uuid", - "employee_id": "uuid", - "organization_id": "uuid", - "checked_in_at": "...", - "checked_out_at": "...", - "status": "CLOSED" - } - ] + "id": "session-uuid", + "employee_id": "employee-uuid", + "organization_id": "org-uuid", + "checkin_at": "2026-04-01T09:00:00.000Z", + "checkout_at": null, + "total_distance_km": null, + "total_duration_seconds": null, + "distance_recalculation_status": null, + "created_at": "...", + "updated_at": "..." } ``` ---- - -### `GET /attendance/org-sessions` - -List all attendance sessions for the caller's organization, newest first. ADMIN only. - -**Auth:** JWT + `role: "ADMIN"` -**Query params:** Same as `/attendance/my-sessions` - -**Response `200`:** Same shape as `my-sessions` but includes sessions from all employees in the organization. - ---- - -## Locations Module +### Locations -High-frequency GPS ingestion. Both write endpoints are rate-limited per JWT `sub` to prevent individual employees from flooding the ingestion pipeline, even when sharing an IP (e.g. corporate NAT). +- `POST /locations` (EMPLOYEE) +- `POST /locations/batch` (EMPLOYEE) +- `GET /locations/my-route?sessionId=` ---- +`/locations/my-route` accepts `sessionId` and legacy-compatible `session_id` query params. -### `POST /locations` +Single-point request example: -Ingest a single GPS point for an active session. - -**Auth:** JWT + `role: "EMPLOYEE"` -**Rate limit:** 10 requests per 10 seconds per JWT `sub` - -**Request body:** ```json { - "session_id": "uuid", + "session_id": "session-uuid", "latitude": 28.6139, - "longitude": 77.2090, + "longitude": 77.209, "accuracy": 12.5, - "recorded_at": "2026-03-10T09:30:00.000Z" + "recorded_at": "2026-04-01T09:30:00.000Z" } ``` -| Field | Type | Constraints | -|-------|------|-------------| -| `session_id` | UUID | Must be a valid UUID | -| `latitude` | number | -90 to 90 | -| `longitude` | number | -180 to 180 | -| `accuracy` | number | ≥ 0 (metres) | -| `recorded_at` | ISO-8601 datetime | Must not be more than 2 minutes in the future | +### Employee Expenses -**Response `201`:** -```json -{ - "success": true, - "data": { - "id": "uuid", - "session_id": "uuid", - "organization_id": "uuid", - "latitude": 28.6139, - "longitude": 77.2090, - "accuracy": 12.5, - "recorded_at": "2026-03-10T09:30:00.000Z", - "sequence_number": null - } -} -``` +- `POST /expenses/receipt-upload-url` (EMPLOYEE) +- `POST /expenses` (EMPLOYEE) +- `GET /expenses/my?page&limit&status` -> **Note:** `sequence_number` is nullable during mobile app stabilization. Distance calculations use `ORDER BY recorded_at` as the primary ordering. +Employee expense status values: ---- +- `all` +- `PENDING` +- `APPROVED` +- `REJECTED` +- `processed` -### `POST /locations/batch` +### Employee Dashboard/Profile -Ingest up to 100 GPS points in a single request. Duplicate points (same `session_id` + `recorded_at`) are silently ignored (upsert with `ignoreDuplicates: true`). +- `GET /dashboard/my-summary` +- `GET /profile/me` +- `GET /leaderboard` -**Auth:** JWT + `role: "EMPLOYEE"` -**Rate limit:** 10 requests per 10 seconds per JWT `sub` +## Expenses (Admin + Employee) -**Request body:** -```json -{ - "session_id": "uuid", - "points": [ - { "latitude": 28.6139, "longitude": 77.2090, "accuracy": 12.5, "recorded_at": "2026-03-10T09:30:00.000Z" }, - { "latitude": 28.6142, "longitude": 77.2094, "accuracy": 11.0, "recorded_at": "2026-03-10T09:30:30.000Z" } - ] -} -``` +### Employee write/list -| Field | Constraints | -|-------|-------------| -| `points` | Array, min 1, max 100 items | -| Each point | Same field constraints as single-insert, *without* `session_id` | +- `POST /expenses` +- `GET /expenses/my` -**Response `201`:** -```json -{ - "success": true, - "inserted": 2 -} -``` +### Admin review/reporting -> `inserted` may be less than `points.length` when duplicates are suppressed. The difference is logged as `duplicatesSuppressed`. +- `GET /admin/expenses` +- `PATCH /admin/expenses/:id` +- `GET /admin/expenses/summary` +- `GET /admin/expenses/export` ---- +Review body example: -### `GET /locations/my-route` - -Retrieve all GPS points for a specific session belonging to the caller's organization, ordered by `recorded_at` ascending. - -**Auth:** JWT + `role: "EMPLOYEE"` -**Query params:** - -| Param | Type | Required | -|-------|------|----------| -| `sessionId` | UUID | Yes | - -**Response `200`:** ```json { - "success": true, - "data": [ - { - "id": "uuid", - "session_id": "uuid", - "organization_id": "uuid", - "latitude": 28.6139, - "longitude": 77.2090, - "accuracy": 12.5, - "recorded_at": "...", - "sequence_number": null - } - ] + "status": "APPROVED" } ``` ---- +## Admin Endpoints -## Expenses Module +All endpoints below require `ADMIN` role unless noted. ---- +### Sessions and Employees -### `POST /expenses` +- `GET /admin/sessions` +- `GET /admin/sessions/:id/locations` +- `POST /admin/force-checkout` +- `GET /admin/employees` +- `POST /admin/employees` +- `GET /admin/employees/:id` +- `PATCH /admin/employees/:id` +- `PATCH /admin/employees/:id/status` +- `GET /admin/employees/:employeeId/profile` +- `GET /admin/search` -Submit a new expense claim. Created with `status: "PENDING"` pending admin review. +### Analytics and Dashboard -**Auth:** JWT + `role: "EMPLOYEE"` -**Rate limit:** 10 requests per 60 seconds per JWT `sub` +- `GET /admin/dashboard` +- `GET /admin/org-summary` +- `GET /admin/user-summary` +- `GET /admin/top-performers` +- `GET /admin/session-trend` +- `GET /admin/leaderboard` +- `GET /admin/monitoring/map` -**Request body:** -```json -{ - "amount": 250.50, - "description": "Fuel for client visit", - "receipt_url": "https://storage.example.com/receipts/abc123.jpg" -} -``` - -| Field | Type | Constraints | -|-------|------|-------------| -| `amount` | number | Positive | -| `description` | string | 3–500 characters | -| `receipt_url` | string (URL) | Optional; must be a valid URL if provided | - -**Response `201`:** -```json -{ - "success": true, - "data": { - "id": "uuid", - "employee_id": "uuid", - "organization_id": "uuid", - "amount": 250.50, - "description": "Fuel for client visit", - "receipt_url": "https://...", - "status": "PENDING", - "created_at": "2026-03-10T10:00:00.000Z" - } -} -``` - ---- - -### `GET /expenses/my` - -List the caller's own expense submissions, newest first. - -**Auth:** JWT + `role: "EMPLOYEE"` -**Query params:** - -| Param | Type | Default | Constraints | -|-------|------|---------|-------------| -| `page` | integer | `1` | min 1 | -| `limit` | integer | `20` | min 1, max 100 | - -**Response `200`:** -```json -{ - "success": true, - "data": [ /* array of expense records */ ] -} -``` - ---- - -### `GET /admin/expenses` +Date range query params use `from` and `to` ISO datetime values. -List all expense submissions for the caller's organization, newest first. ADMIN only. +### Monitoring, Queues, and System Ops -**Auth:** JWT + `role: "ADMIN"` -**Query params:** Same as `GET /expenses/my` +- `POST /admin/start-monitoring` +- `POST /admin/stop-monitoring` +- `GET /admin/monitoring-history` +- `GET /admin/events` (SSE stream) +- `GET /admin/queues` +- `POST /admin/queues/replay-distance-dlq` +- `GET /admin/retry-intents` +- `GET /admin/system-health` +- `GET /admin/audit-log` ---- +### API Keys -### `PATCH /admin/expenses/:id` +- `POST /admin/api-keys` +- `GET /admin/api-keys` +- `PATCH /admin/api-keys/:id` +- `DELETE /admin/api-keys/:id` -Approve or reject a pending expense. Only `PENDING` expenses can be acted on — attempting to update an already-reviewed expense returns a `400`. +API key routes require JWT auth and cannot be managed with API key auth itself. -**Auth:** JWT + `role: "ADMIN"` +### Webhooks -**Path params:** - -| Param | Type | Required | -|-------|------|----------| -| `id` | UUID | Yes | - -**Request body:** -```json -{ "status": "APPROVED" } -``` +- `GET /admin/webhooks` +- `POST /admin/webhooks` +- `PATCH /admin/webhooks/:id` +- `DELETE /admin/webhooks/:id` +- `GET /admin/webhook-deliveries` +- `GET /admin/webhooks/logs` +- `POST /admin/webhook-deliveries/:id/retry` +- `POST /admin/webhooks/logs/:id/retry` +- `POST /admin/webhooks/:id/test` +- `GET /admin/webhook-dlq` +- `POST /admin/webhook-dlq/:id/retry` -| `status` | Meaning | -|----------|---------| -| `"APPROVED"` | Marks expense as approved | -| `"REJECTED"` | Marks expense as rejected | +Legacy aliases under `/webhooks*` exist for backward compatibility and are hidden from OpenAPI. -**Response `200`:** -```json -{ - "success": true, - "data": { /* updated expense record */ } -} -``` +### Internal Admin Probes -**Error `400`:** `"Expense has already been reviewed (current status: APPROVED)"` — thrown by `ExpenseAlreadyReviewed` -**Error `404`:** Expense not found or does not belong to the caller's organization. +- `GET /internal/metrics` +- `GET /internal/queues/status` +- `GET /internal/snapshot-health` +- `GET /internal/ready-deep` ---- +## Rate Limit Notes -## Analytics Module +Route-level limits include: -All analytics endpoints require JWT + **ADMIN** role. EMPLOYEE tokens receive `403`. - ---- - -### `GET /admin/org-summary` - -Organisation-wide aggregated totals for a given date range. - -**Auth:** JWT + `role: "ADMIN"` -**Query params:** - -| Param | Type | Required | Description | -|-------|------|----------|-------------| -| `from` | ISO-8601 datetime | No | Range start (inclusive) | -| `to` | ISO-8601 datetime | No | Range end (inclusive) | - -**Response `200`:** -```json -{ - "success": true, - "data": { - "totalSessions": 142, - "totalDistanceKm": 4820.5, - "totalDurationSeconds": 1843200, - "totalExpenses": 38, - "approvedExpenseAmount": 9450.00, - "rejectedExpenseAmount": 550.00, - "activeEmployeesCount": 12 - } -} -``` - ---- - -### `GET /admin/user-summary` - -Per-user totals and averages for a given date range. - -**Auth:** JWT + `role: "ADMIN"` -**Query params:** - -| Param | Type | Required | -|-------|------|----------| -| `userId` | UUID | **Yes** | -| `from` | ISO-8601 datetime | No | -| `to` | ISO-8601 datetime | No | - -**Response `200`:** -```json -{ - "success": true, - "data": { - "userId": "uuid", - "totalSessions": 22, - "totalDistanceKm": 741.3, - "totalDurationSeconds": 284400, - "avgDistanceKmPerSession": 33.7, - "avgDurationSecondsPerSession": 12927, - "totalExpenses": 6, - "approvedExpenseAmount": 1650.00 - } -} -``` - ---- - -### `GET /admin/top-performers` - -Ranked leaderboard of employees sorted by a chosen metric. - -**Auth:** JWT + `role: "ADMIN"` -**Query params:** - -| Param | Type | Required | Notes | -|-------|------|----------|-------| -| `metric` | `"distance"` \| `"duration"` \| `"sessions"` | **Yes** | Ranking criterion | -| `from` | ISO-8601 datetime | No | | -| `to` | ISO-8601 datetime | No | | -| `limit` | integer | No | 1–50, default 10 | - -**Response `200`:** -```json -{ - "success": true, - "data": [ - { - "employeeId": "uuid", - "totalDistanceKm": 741.3, - "totalDurationSeconds": 284400, - "totalSessions": 22 - } - ] -} -``` - -Results are ordered descending by the chosen `metric`. - ---- - -## Rate Limit Summary - -| Endpoint | Limit | Window | Key | -|----------|-------|--------|-----| -| All routes (global) | 100 req | 1 minute | IP | -| `POST /locations` | 10 req | 10 seconds | JWT `sub` | -| `POST /locations/batch` | 10 req | 10 seconds | JWT `sub` | -| `POST /expenses` | 10 req | 60 seconds | JWT `sub` | -| `POST /attendance/:id/recalculate` | 5 req | 60 seconds | JWT `sub` | - -`localhost` / `::1` are exempt from all rate limits (health checks, monitoring scrapes). - -When a rate limit is exceeded: -```json -{ - "success": false, - "error": "Too many requests", - "retryAfter": "42s" -} -``` - ---- - -## JWT Payload Reference - -FieldTrack uses **Supabase-issued JWTs**. The backend validates the following claims with Zod: - -```json -{ - "sub": "550e8400-e29b-41d4-a716-446655440000", - "organization_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7", - "role": "EMPLOYEE", - "iat": 1741600000, - "exp": 1741686400 -} -``` +- `POST /locations` and `POST /locations/batch`: 10 requests / 10 seconds per user key +- `POST /expenses`: 10 requests / 60 seconds per user key +- `POST /attendance/:sessionId/recalculate`: 5 requests / 60 seconds per user key -The `organization_id` claim is attached to `request.organizationId` by the `authenticate` middleware and used by every repository method for tenant isolation. No cross-organization data access is possible via the API. +Additional global and security middleware limits are enforced by Fastify plugins. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d425872..ae8b6cd 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,380 +1,113 @@ -# FieldTrack 2.0 System Architecture - -## High-Level Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ CLIENT LAYER │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ -│ │ Mobile │ │ Web │ │ Desktop │ │ -│ │ App │ │ Dashboard │ │ Client │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ -│ │ │ │ │ -│ └───────────────────┼────────────────────┘ │ -│ │ │ -│ │ HTTPS / REST API │ -│ │ │ -└─────────────────────────────┼────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ APPLICATION LAYER │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────────────────────────────────────────────────────────────┐ │ -│ │ Fastify API Server │ │ -│ │ (Node.js + TypeScript) │ │ -│ ├────────────────────────────────────────────────────────────────┤ │ -│ │ │ │ -│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ -│ │ │ Auth │ │ Business │ │ Validation │ │ │ -│ │ │ Middleware │ │ Logic │ │ (Zod) │ │ │ -│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ -│ │ │ │ -│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ -│ │ │ Rate Limit │ │ CORS │ │ Helmet │ │ │ -│ │ │ Security │ │ Security │ │ Security │ │ │ -│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ -│ │ │ │ -│ └─────────────────────────┬─────────────────────────────────────── │ -│ │ │ -└────────────────────────────┼─────────────────────────────────────────────┘ - │ - ┌────────────┼────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────────────────────────────────────────────────────────────┐ -│ DATA LAYER │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Supabase Platform │ │ -│ ├──────────────────────────────────────────────────────────────┤ │ -│ │ │ │ -│ │ ┌────────────────────────────────────────────────────┐ │ │ -│ │ │ PostgreSQL Database │ │ │ -│ │ ├────────────────────────────────────────────────────┤ │ │ -│ │ │ │ │ │ -│ │ │ • organizations │ │ │ -│ │ │ • users │ │ │ -│ │ │ • employees │ │ │ -│ │ │ • attendance_sessions │ │ │ -│ │ │ • gps_locations │ │ │ -│ │ │ • expenses │ │ │ -│ │ │ │ │ │ -│ │ │ Multi-tenant: Row Level Security (RLS) │ │ │ -│ │ │ │ │ │ -│ │ └────────────────────────────────────────────────────┘ │ │ -│ │ │ │ -│ │ ┌────────────────────────────────────────────────────┐ │ │ -│ │ │ Authentication (JWT) │ │ │ -│ │ └────────────────────────────────────────────────────┘ │ │ -│ │ │ │ -│ └─────────────────────────────────────────────────────────────── │ -│ │ -└───────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────┐ -│ BACKGROUND JOBS LAYER │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Redis (BullMQ) │ │ -│ │ Job Queue Manager │ │ -│ └────────────────────────┬─────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Distance Worker │ │ -│ ├──────────────────────────────────────────────────────────────┤ │ -│ │ │ │ -│ │ • Processes GPS location updates │ │ -│ │ • Calculates distances between locations │ │ -│ │ • Updates session travel metrics │ │ -│ │ • Handles concurrent job processing │ │ -│ │ │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -└───────────────────────────────────────────────────────────────────────────┘ - -``` - -> Monitoring stack (Prometheus, Grafana, Loki, Tempo) is managed by the **infra repository**. -> The API exposes `/metrics` and OTLP traces, which the infra repo consumes. -``` - -## Component Details - -### Client Layer -- **Mobile App**: Field employee mobile application for attendance and location tracking -- **Web Dashboard**: Admin dashboard for management and analytics -- **Desktop Client**: Desktop application for supervisors and managers - -### Application Layer -- **Fastify API Server**: High-performance Node.js REST API - - JWT authentication via Supabase - - Multi-tenant isolation with organization context - - Rate limiting and security middleware - - Zod schema validation via `fastify-type-provider-zod` (`zod.plugin.ts` is the single registration point) - - `preValidation` hook for auth — ensures 401/403 always fires before body/querystring schema validation - - OpenTelemetry instrumentation - -### Data Layer -- **Supabase PostgreSQL**: Primary database with Row Level Security - - Multi-tenant data isolation - - Real-time subscriptions support - - Built-in authentication - -### Background Jobs Layer -- **Redis + BullMQ**: Distributed job queue -- **Distance Worker**: Asynchronous GPS processing - - Haversine distance calculations - - Session travel metrics - - Configurable concurrency (`WORKER_CONCURRENCY` env var) - - Job retention limits: 1 000 completed, 5 000 failed (prevents Redis memory growth) - -### Observability -- The API emits metrics (Prometheus format on `/metrics`), structured logs (Pino/JSON), and traces (OpenTelemetry OTLP) -- Collection, dashboards, and alerting are handled by the **infra repository** - -## Data Flow - -### Attendance Check-In Flow -``` -Mobile App - │ - │ POST /attendance/check-in - │ { latitude, longitude } - │ - ▼ -Fastify API - │ - ├─▶ preValidation: Auth Middleware (verify JWT) ← runs first - │ - ├─▶ Validate Request Body (Zod) ← runs after auth - │ - ├─▶ Create Session (Supabase) - │ - └─▶ Queue Distance Job (BullMQ) - │ - ▼ - Distance Worker - │ - ├─▶ Calculate Distance - │ - └─▶ Update Session (Supabase) -``` - -### Location Update Flow -``` -Mobile App - │ - │ POST /locations - │ { session_id, latitude, longitude, accuracy, recorded_at } - │ - ▼ -Fastify API - │ - ├─▶ preValidation: Auth Middleware ← runs first - │ - ├─▶ Validate Body (Zod createLocationSchema) - │ - ├─▶ Validate Active Session - │ - ├─▶ Store Location (Supabase) - │ - └─▶ Queue Distance Job (BullMQ) - │ - ▼ - Distance Worker - │ - ├─▶ Get Previous Location - │ - ├─▶ Calculate Distance - │ - └─▶ Update Total Distance -``` - -### Analytics Query Flow -``` -Web Dashboard - │ - │ GET /analytics/summary - │ - ▼ -Fastify API - │ - ├─▶ Auth Middleware (ADMIN role) - │ - ├─▶ Validate Date Range - │ - ├─▶ Query Aggregated Data (Supabase) - │ - └─▶ Return Metrics -``` - -## Deployment Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ VPS DEPLOYMENT │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Nginx │ │ -│ │ (Reverse Proxy) │ │ -│ │ │ │ -│ │ • SSL/TLS Termination │ │ -│ │ • Load Balancing │ │ -│ │ • Blue-Green Routing │ │ -│ │ │ │ -│ └────────────────────┬─────────────────────────────────────────┘ │ -│ │ │ -│ ┌─────────────┴─────────────┐ │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌─────────────┐ ┌─────────────┐ │ -│ │ Blue │ │ Green │ │ -│ │ Container │ │ Container │ │ -│ │ (Active) │ │ (Standby) │ │ -│ │ │ │ │ │ -│ │ Port: 3001 │ │ Port: 3002 │ │ -│ └─────────────┘ └─────────────┘ │ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Docker Network │ │ -│ │ (api_network) │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ Monitoring Stack │ │ -│ │ │ │ -│ │ Prometheus | Grafana | Loki | Tempo | Promtail │ │ -│ │ │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -└───────────────────────────────────────────────────────────────────────────┘ - - ▲ - │ - │ GitHub Actions CI/CD - │ -┌─────────────────────────────┴─────────────────────────────────────────────┐ -│ CI/CD PIPELINE │ -├───────────────────────────────────────────────────────────────────────────┤ -│ │ -│ GitHub Push → Test → Build Docker → Push GHCR → Deploy Blue-Green │ -│ │ -│ • Automated testing (125 tests) │ -│ • TypeScript compilation check │ -│ • Docker image build with caching │ -│ • Push to GitHub Container Registry │ -│ • Blue-green deployment with health checks │ -│ • Rollback capability (last 5 deployments) │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -## Security Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ SECURITY LAYERS │ -├─────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Layer 1: Network Security │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ • HTTPS/TLS encryption │ │ -│ │ • Nginx reverse proxy │ │ -│ │ • CORS policy enforcement │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -│ Layer 2: Application Security │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ • Helmet.js security headers │ │ -│ │ • Rate limiting (per IP/user) │ │ -│ │ • Request validation (Zod schemas) │ │ -│ │ • JWT authentication │ │ -│ │ • Role-based access control (RBAC) │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -│ Layer 3: Data Security │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ • Row Level Security (RLS) │ │ -│ │ • Multi-tenant isolation │ │ -│ │ • Encrypted connections │ │ -│ │ • Audit logging │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -│ Layer 4: Monitoring & Response │ -│ ┌──────────────────────────────────────────────────────────────┐ │ -│ │ • Abuse detection logging │ │ - │ • Alerting (handled by infra repository) │ │ - │ • Distributed tracing (OpenTelemetry OTLP) │ │ -│ │ • Error tracking │ │ -│ └──────────────────────────────────────────────────────────────┘ │ -│ │ -└───────────────────────────────────────────────────────────────────────────┘ -``` - -## Technology Stack - -### Backend -- **Runtime**: Node.js 24+ -- **Language**: TypeScript 5.9 (strict mode, ESM) -- **Framework**: Fastify 5 -- **Validation**: Zod 4 (`fastify-type-provider-zod`) -- **Authentication**: @fastify/jwt -- **Job Queue**: BullMQ + Redis - -### Database -- **Primary**: PostgreSQL (via Supabase) -- **Cache/Queue**: Redis -- **ORM**: Supabase Client - -### Security -- **Headers**: @fastify/helmet -- **CORS**: @fastify/cors -- **Rate Limiting**: @fastify/rate-limit -- **Compression**: @fastify/compress - -### Observability -- **Metrics**: prom-client (exposed on `/metrics`, scraped by infra repo) -- **Logs**: Pino (structured JSON, collected by infra repo) -- **Traces**: OpenTelemetry 2.x (exported via OTLP to `TEMPO_ENDPOINT`) - -### DevOps -- **Containerization**: Docker (node:24-alpine) -- **Registry**: GitHub Container Registry (GHCR) -- **CI/CD**: GitHub Actions -- **Deployment**: Blue-Green with rollback -- **Reverse Proxy**: Nginx -- **Testing**: Vitest (125 tests) - -## Scalability Considerations - -### Horizontal Scaling -- Stateless API design allows multiple instances -- Redis-backed job queue for distributed workers -- Database connection pooling - -### Vertical Scaling -- Configurable worker concurrency -- Adjustable rate limits -- Database query optimization - -### Performance Optimizations -- Docker layer caching in CI/CD -- npm dependency caching -- Fastify's high-performance routing -- Async/await for non-blocking I/O -- Background job processing for heavy operations - -## Related Documentation - -- [Deployment Guide](../docs/DEPLOYMENT.md) -- [Rollback System](../docs/ROLLBACK_SYSTEM.md) -- [API Documentation](../README.md) -- [CI/CD Pipeline](../.github/workflows/deploy.yml) +# FieldTrack API Architecture + +This document describes the implemented backend architecture in `api`. + +## Service Structure + +Entry flow: + +1. `src/server.ts` initializes telemetry and validates env +2. `src/app.ts` builds Fastify app, registers plugins and routes +3. `src/routes/index.ts` registers all module routes +4. workers are started only after the HTTP server is listening + +Core layers: + +- `src/modules/*`: business domains and route handlers +- `src/middleware/*`: authentication and role enforcement +- `src/db/*`: tenant-scoped query helper (`orgTable`) and repository patterns +- `src/plugins/*`: security, docs, metrics, and validation integrations +- `src/workers/*`: asynchronous processing and scheduled maintenance + +## Request Lifecycle + +1. security plugins run (`helmet`, `cors`, rate limiting, abuse logging) +2. auth middleware resolves identity (JWT/API key) +3. role guard enforces endpoint access +4. Zod validation enforces request schemas +5. handler/service executes tenant-scoped data operations +6. standardized response envelope is returned + +## Repository and Data Access Pattern + +Tenant isolation in application code is enforced through organization scoping: + +- `request.organizationId` is injected by auth middleware +- repositories use org-scoped filters via `orgTable(...)` or explicit `.eq("organization_id", orgId)` +- service-role client is used carefully with explicit tenant constraints + +This protects against cross-tenant data reads/writes at API layer even when using privileged DB credentials. + +## Worker System (BullMQ + Scheduled Jobs) + +Workers defined and started from `src/workers/startup.ts`: + +- distance worker + - recalculates session distance/duration +- analytics worker + - updates daily/org aggregates and leaderboard data +- webhook worker + - processes outbound webhook deliveries +- snapshot worker + - maintains denormalized snapshot tables from event jobs + +Support queues and reliability components: + +- retry-intent persistence and replay +- dead-letter queue replay endpoint for distance jobs +- webhook DLQ listing/retry endpoints + +Scheduled jobs: + +- `reconciliation.job.ts`: calls `reconcile_snapshot_tables()` every 5 minutes +- `retry-cleanup.job.ts`: cleans stale retry intents + +## Snapshot Table Logic + +Snapshot/event model keeps admin reads fast and deterministic. + +Primary snapshot surfaces maintained by worker + reconciliation: + +- `employee_last_state` +- `active_users` +- `employee_latest_sessions` +- `employee_metrics_snapshot` +- `org_dashboard_snapshot` +- `pending_expenses` + +Operational model: + +- event-driven updates on check-in/check-out/location/expense actions +- idempotent UPSERT/delete semantics +- periodic reconciliation self-heals drift when transient failures occur + +## Auth and Tenant Security + +Supported auth: + +- JWT bearer +- scoped API keys + +Authorization: + +- role checks via middleware (`ADMIN`, `EMPLOYEE`) +- endpoint-level `preValidation` guards + +Tenant boundaries: + +- all org data operations apply `organization_id` scoping +- row-level security exists in DB layer and is complemented by API-layer tenant scoping + +## Observability and Operations + +Implemented observability: + +- Prometheus/OpenMetrics endpoint (`/metrics`) +- OpenTelemetry traces +- structured request logging with request id and trace correlation +- internal and admin operational endpoints (`/internal/*`, `/admin/system-health`, `/admin/queues`) + +Real-time admin updates: + +- SSE stream at `/admin/events` +- org-scoped event bus for session and expense updates diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 76249b2..6e4138e 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,136 +1,89 @@ -# Deployment Guide +# API Deployment Guide -This document covers deploying FieldTrack API to a Linux VPS using the included blue-green deployment system. +This guide reflects the implemented production deployment flow for the API repository. -> **Scope:** This document covers the API only. Nginx configuration, TLS, and the monitoring stack are managed by the **infra repository**. +## CI/CD Flow (GitHub Actions) ---- +Primary workflow: `.github/workflows/deploy.yml` -## Prerequisites +Pipeline behavior: -- A Linux VPS (Ubuntu 22.04 recommended) accessible via SSH -- A GitHub Container Registry (GHCR) account with push access to the repository -- GitHub Actions secrets configured (see [CI/CD Setup](#cicd-setup)) -- Docker installed on the VPS -- Nginx already running and configured via the **infra repository** +1. `codeql-gate` + - deploy triggers from CodeQL deep-scan completion on `master` + - blocked if scan conclusion is not `success` +2. `validate` + `test-api` + `infra-leakage-guard` +3. `build-scan-push` + - Docker build + - image scanning + - push to GHCR +4. readiness/deploy jobs execute VPS deployment using `scripts/deploy.sh` +5. post-deploy health and smoke checks +6. rollback path on deploy/smoke failure ---- +## Docker Build and Runtime -## API Deployment +Image is built in CI and tagged by commit SHA. -1. SSH into VPS -2. Ensure nginx is running (managed via infra repository) -3. Copy `.env.example` to `.env` and fill in all values -4. Deploy: `./scripts/deploy.sh ` -5. Confirm health: `curl https:///health` +Deployment script pulls the target image and starts it in inactive blue/green slot. -## Rollback +Important deploy invariant from `scripts/deploy.sh`: -```bash -./scripts/deploy.sh --rollback # interactive -./scripts/deploy.sh --rollback --auto # non-interactive (CI) -``` - -## Monitoring - -The observability stack (Prometheus, Grafana, Loki, Tempo) is **handled by the infra repository**. The API exposes: -- `GET /metrics` — Prometheus-format metrics (protected by `METRICS_SCRAPE_TOKEN`) -- Traces exported via OTLP to `TEMPO_ENDPOINT` +- deployment success is based on container startup and `/health` routing checks +- script intentionally does not gate success on `/ready` ---- +## Blue-Green Deploy Model -## Blue-Green Deployment +Slots: -The deployment uses a blue-green strategy for zero-downtime releases. +- `api-blue` +- `api-green` -### How It Works +Flow: -The VPS keeps **two named slots** (`api-blue`, `api-green`). Only the active slot receives traffic through nginx over `api_network`. -The API containers do **not** bind host ports. +1. resolve currently active slot +2. start/update inactive slot with new image +3. validate container health +4. switch nginx routing to new slot +5. validate routed `/health` +6. remove old slot after success -On each deploy: +State files and lock handling are maintained by deploy script for deterministic slot switching and recovery behavior. -1. The new image is pulled from GHCR -2. The **inactive** container is replaced with the new image -3. The new container is health-checked via `GET /health` -4. Nginx upstream is switched to the new container (`nginx -s reload`) -5. The previously active container is stopped and removed -6. The deployed SHA is prepended to `.deploy_history` (keeps last 5) +## Health Checks -### Manual Deploy +API endpoints: -```bash -# SSH into the VPS -cd $HOME/api - -# Deploy a specific image SHA (e.g. from CI output) -./scripts/deploy.sh a4f91c2 -``` +- `GET /health`: liveness/deploy gate +- `GET /ready`: deep dependency check (informational for ops, not deploy gate) ---- - -## Rollback - -To instantly revert to the previous deployment: - -```bash -cd $HOME/api -./scripts/deploy.sh --rollback -``` +Nginx and infra health are managed by infra repo and include `/infra/health` for proxy liveness. -The script: -1. Reads `.deploy_history` (requires at least 2 recorded deployments) -2. Displays the full history with current/target markers -3. Prompts for confirmation before proceeding -4. Redeploys the previous SHA — no rebuild, image already in GHCR +## Rollback Logic -**Typical rollback time: under 10 seconds.** +`scripts/deploy.sh` supports: -For full rollback system documentation, see [ROLLBACK_SYSTEM.md](ROLLBACK_SYSTEM.md). +- interactive rollback: `--rollback` +- non-interactive rollback: `--rollback --auto` ---- +Deploy outcomes are phase-aware and emit explicit result states. -## Environment Variables +Rollback is attempted automatically on failed post-switch validation paths. -Copy `.env.example` to `.env` on the VPS and fill in all values before the first deploy. +## Manual Deploy Commands -See [README.md](../README.md) and [env-contract.md](env-contract.md) for the full variable reference. - ---- - -## Health Endpoints - -| Endpoint | Purpose | Deploy gate | -|----------|---------|-------------| -| `GET /health` | Liveness — returns `{"status":"ok"}` after bootstrap | **YES** | -| `GET /ready` | Dependency check (Redis + Supabase) | NO — informational only | - -The deploy script uses `/health` exclusively. `/ready` failing does not block a deployment. - ---- - -## Troubleshooting - -**Deployment hangs on health check** -The new container failed to start. Check Docker logs: ```bash -docker logs api-green # or api-blue +./scripts/deploy.sh +./scripts/deploy.sh --rollback +./scripts/deploy.sh --rollback --auto ``` -**Rollback fails: "insufficient deployment history"** -Only one deployment has been recorded. Deploy manually with a known-good SHA: -```bash -./scripts/deploy.sh -``` +## Prerequisites on VPS -**Container image not found in GHCR** -The SHA must match a tag pushed to GHCR. Verify with: -```bash -docker pull ghcr.io/fieldtrack-tech/api: -``` +Managed primarily by infra repository: -**Nginx fails to reload** -Nginx is managed by the infra repository. Check its configuration and reload there. +- nginx deployed and healthy +- redis deployed and reachable +- `api_network` Docker network present +- infra path contracts available under configured infra root -**API starts but /ready fails** -Acceptable — Redis or Supabase may be temporarily unavailable. The deploy is still considered successful if `/health` returns 200. +Without these, API deployment will fail preflight validation. diff --git a/src/modules/admin/sessions.routes.ts b/src/modules/admin/sessions.routes.ts index 5430765..861065f 100644 --- a/src/modules/admin/sessions.routes.ts +++ b/src/modules/admin/sessions.routes.ts @@ -9,8 +9,8 @@ import { orgSessionsQuerySchema } from "../attendance/attendance.schema.js"; // ─── Query schema ───────────────────────────────────────────────────────────── -// TODO (future phase): replace offset pagination with cursor-based pagination -// to support large datasets without heavy DB scans. +// Future scalability: replace offset pagination with cursor-based pagination +// when admin session history moves beyond current snapshot-sized reads. const adminSessionsQuerySchema = orgSessionsQuerySchema.extend({ limit: z.coerce.number().int().min(1).max(100).default(50), }); diff --git a/src/modules/crashes/crashes.routes.ts b/src/modules/crashes/crashes.routes.ts new file mode 100644 index 0000000..ee741c8 --- /dev/null +++ b/src/modules/crashes/crashes.routes.ts @@ -0,0 +1,56 @@ +import type { FastifyInstance } from "fastify"; +import { z } from "zod"; +import { supabaseServiceClient as supabase } from "../../config/supabase.js"; +import { authenticate } from "../../middleware/auth.js"; +import { handleError, ok } from "../../utils/response.js"; + +const crashReportBodySchema = z.object({ + file_name: z.string().min(1).max(255), + platform: z.literal("android"), + raw_report: z.string().min(1).max(200_000), +}); + +export async function crashRoutes(app: FastifyInstance): Promise { + app.post( + "/crashes", + { + schema: { + tags: ["crashes"], + summary: "Ingest a mobile crash report", + body: crashReportBodySchema, + response: { + 201: z.object({ + success: z.literal(true), + data: z.object({ id: z.string().uuid() }), + }), + }, + }, + preValidation: [authenticate], + }, + async (request, reply) => { + try { + const body = crashReportBodySchema.parse(request.body); + const { data, error } = await supabase + .from("mobile_crash_reports") + .insert({ + organization_id: request.organizationId, + user_id: request.user.sub, + role: request.user.role, + platform: body.platform, + file_name: body.file_name, + raw_report: body.raw_report, + }) + .select("id") + .single(); + + if (error) { + throw new Error(`Crash report insert failed: ${error.message}`); + } + + reply.status(201).send(ok({ id: String((data as { id: string }).id) })); + } catch (error) { + handleError(error, request, reply, "Unexpected error ingesting crash report"); + } + }, + ); +} diff --git a/src/modules/expenses/expenses.schema.ts b/src/modules/expenses/expenses.schema.ts index 21071ab..265a4d9 100644 --- a/src/modules/expenses/expenses.schema.ts +++ b/src/modules/expenses/expenses.schema.ts @@ -47,8 +47,8 @@ export type UpdateExpenseStatusBody = z.infer< typeof updateExpenseStatusBodySchema >; -// TODO (future phase): replace offset pagination with cursor-based pagination -// to support large datasets without heavy DB scans. +// Future scalability: replace offset pagination with cursor-based pagination +// for very large expense histories. export const expensePaginationSchema = z.object({ page: z.coerce.number().int().min(1).default(1), limit: z.coerce.number().int().min(1).max(100).default(50), diff --git a/src/modules/locations/locations.schema.ts b/src/modules/locations/locations.schema.ts index e670a73..5f965ab 100644 --- a/src/modules/locations/locations.schema.ts +++ b/src/modules/locations/locations.schema.ts @@ -36,7 +36,8 @@ export const createLocationSchema = z.object({ * Must be a non-negative integer. Monotonic increase per session is expected but * not enforced here — the service layer validates against the session start time. * - * TODO (post-mobile stabilisation): add DB NOT NULL + CHECK (sequence_number >= 0) + * Future hardening after mobile stabilization: add DB NOT NULL + + * CHECK (sequence_number >= 0). */ sequence_number: z.number().int().min(0, "sequence_number must be >= 0").optional(), }); diff --git a/src/routes/index.ts b/src/routes/index.ts index b73d8c0..c81b3c0 100644 --- a/src/routes/index.ts +++ b/src/routes/index.ts @@ -17,6 +17,7 @@ import { webhookDlqRoutes } from "../modules/admin/webhook-dlq.routes.js"; import { eventsRoutes } from "./events.routes.js"; import { webhooksRoutes } from "../modules/webhooks/webhooks.routes.js"; import { authRoutes } from "../modules/auth/auth.routes.js"; +import { crashRoutes } from "../modules/crashes/crashes.routes.js"; import { auditLogRoutes } from "../modules/admin/audit-log.routes.js"; import { adminQueuesRoutes } from "../modules/admin/queues.routes.js"; import { adminRetryIntentsRoutes } from "../modules/admin/retry-intents.routes.js"; @@ -30,6 +31,7 @@ export async function registerRoutes(app: FastifyInstance): Promise { await app.register(internalRoutes); await app.register(debugRoutes); await app.register(authRoutes); + await app.register(crashRoutes); await app.register(attendanceRoutes); await app.register(locationsRoutes); await app.register(expensesRoutes); diff --git a/src/utils/url-validator.ts b/src/utils/url-validator.ts index 69f92fe..2b81840 100644 --- a/src/utils/url-validator.ts +++ b/src/utils/url-validator.ts @@ -22,7 +22,7 @@ * reject private IPs there (defense-in-depth). This validator is the first * gate, not the only one. * - * TODO(Phase 25 — delivery worker HTTP client): + * Future delivery-worker HTTP client hardening: * Resolve the webhook hostname immediately before opening the outbound * TCP connection and validate the resolved IP against the same private * ranges checked here. This closes the DNS rebinding window: diff --git a/supabase/migrations/20260521000100_mobile_crash_reports.sql b/supabase/migrations/20260521000100_mobile_crash_reports.sql new file mode 100644 index 0000000..1d82630 --- /dev/null +++ b/supabase/migrations/20260521000100_mobile_crash_reports.sql @@ -0,0 +1,25 @@ +CREATE TABLE IF NOT EXISTS public.mobile_crash_reports ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + organization_id uuid NOT NULL REFERENCES public.organizations(id) ON DELETE CASCADE, + user_id uuid NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, + role public.user_role NOT NULL, + platform text NOT NULL CHECK (platform = 'android'), + file_name text NOT NULL CHECK (length(file_name) BETWEEN 1 AND 255), + raw_report text NOT NULL CHECK (length(raw_report) BETWEEN 1 AND 200000), + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX IF NOT EXISTS idx_mobile_crash_reports_org_created + ON public.mobile_crash_reports (organization_id, created_at DESC); + +CREATE INDEX IF NOT EXISTS idx_mobile_crash_reports_user_created + ON public.mobile_crash_reports (user_id, created_at DESC); + +ALTER TABLE public.mobile_crash_reports ENABLE ROW LEVEL SECURITY; + +DROP POLICY IF EXISTS "org_isolation_mobile_crash_reports" + ON public.mobile_crash_reports; + +CREATE POLICY "org_isolation_mobile_crash_reports" + ON public.mobile_crash_reports FOR ALL + USING (organization_id::text = coalesce(auth.jwt() ->> 'org_id', '')); diff --git a/tests/integration/crashes.test.ts b/tests/integration/crashes.test.ts new file mode 100644 index 0000000..a07ebfd --- /dev/null +++ b/tests/integration/crashes.test.ts @@ -0,0 +1,113 @@ +import { describe, it, expect, vi, beforeAll, afterAll, beforeEach } from "vitest"; +import type { FastifyInstance } from "fastify"; + +vi.mock("../../src/config/redis.js", () => ({ + redisClient: { on: vi.fn(), quit: vi.fn(), disconnect: vi.fn() }, +})); + +vi.mock("../../src/workers/distance.queue.js", () => ({ + enqueueDistanceJob: vi.fn().mockResolvedValue(undefined), +})); + +vi.mock("../../src/workers/analytics.queue.js", () => ({ + enqueueAnalyticsJob: vi.fn().mockResolvedValue(undefined), +})); + +const supabaseMock = vi.hoisted(() => { + const single = vi.fn(); + const select = vi.fn(() => ({ single })); + const insert = vi.fn(() => ({ select })); + const from = vi.fn(() => ({ insert })); + return { from, insert, select, single }; +}); + +vi.mock("../../src/config/supabase.js", () => ({ + supabaseServiceClient: { from: supabaseMock.from }, + supabaseAnonClient: { auth: { signInWithPassword: vi.fn() } }, +})); + +vi.mock("../../src/auth/jwtVerifier.js", () => ({ + verifySupabaseToken: vi.fn().mockImplementation(async (token: string) => { + const parts = token.split("."); + if (parts.length !== 3) throw new Error("Invalid JWT structure"); + return JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")); + }), +})); + +import { buildTestApp, signEmployeeToken } from "../setup/test-server.js"; + +describe("POST /crashes", () => { + let app: FastifyInstance; + let employeeToken: string; + + beforeAll(async () => { + app = await buildTestApp(); + employeeToken = signEmployeeToken(app); + }); + + afterAll(async () => { + await app.close(); + }); + + beforeEach(() => { + vi.clearAllMocks(); + supabaseMock.single.mockResolvedValue({ + data: { id: "11111111-1111-4111-8111-111111111111" }, + error: null, + }); + }); + + it("requires authentication", async () => { + const res = await app.inject({ + method: "POST", + url: "/crashes", + payload: { file_name: "crash.txt", platform: "android", raw_report: "stack" }, + }); + + expect(res.statusCode).toBe(401); + }); + + it("stores an authenticated Android crash report", async () => { + const res = await app.inject({ + method: "POST", + url: "/crashes", + headers: { authorization: `Bearer ${employeeToken}` }, + payload: { file_name: "crash_1.txt", platform: "android", raw_report: "stacktrace" }, + }); + + expect(res.statusCode).toBe(201); + expect(supabaseMock.from).toHaveBeenCalledWith("mobile_crash_reports"); + expect(supabaseMock.insert).toHaveBeenCalledWith(expect.objectContaining({ + platform: "android", + file_name: "crash_1.txt", + raw_report: "stacktrace", + })); + }); + + it("rejects invalid payloads", async () => { + const res = await app.inject({ + method: "POST", + url: "/crashes", + headers: { authorization: `Bearer ${employeeToken}` }, + payload: { file_name: "", platform: "ios", raw_report: "" }, + }); + + expect(res.statusCode).toBe(400); + }); + + it("returns 500 when persistence fails", async () => { + supabaseMock.single.mockResolvedValue({ + data: null, + error: { message: "insert failed" }, + }); + + const res = await app.inject({ + method: "POST", + url: "/crashes", + headers: { authorization: `Bearer ${employeeToken}` }, + payload: { file_name: "crash_2.txt", platform: "android", raw_report: "stacktrace" }, + }); + + expect(res.statusCode).toBe(500); + }); +});