diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..90ecdb0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,43 @@ +# Changelog + +Notable changes to Beckham Share, newest first. The project follows a simple +pre-1.0 scheme; dates are when the change reached `main`. + +## Unreleased + +### Added +- Full reference documentation under `docs/`: architecture, configuration, + operations runbook, security model, and API reference, with a docs index and + a `CONTRIBUTING` guide. + +## 2026-06-23 + +### Added +- Troubleshooting write-up for the Android upload "microphone" prompt, plus a + standalone file-picker diagnostic page kept for reference (#10). + +### Fixed +- CI: use a writable `DATA_DIR` for the end-to-end job and stop duplicate + workflow runs. + +## 2026-06-22 + +### Added +- **Initial release.** Upload a file and share it by an opaque UUID link with a + configurable expiry; copy-to-clipboard and email actions. +- Public landing page with a constrained anonymous upload (size cap, rate limit, + short auto-expiry, IP + fingerprint audit trail). +- Identity-gated workspace: OIDC sign-in against Authentik, restricted to the + `dropbox` group at the IdP and re-checked in-app. +- Server-side "email this link" over SMTP, restricted to signed-in members; + anonymous visitors fall back to a `mailto:` link. +- Per-link management: change expiry, revoke, delete; download counting. +- Brand logo and favicon. +- Three-layer automated test suite (backend integration, browser E2E, live + smoke + configuration check) with continuous integration on every push and + pull request, plus `docs/TESTING.md`. + +### Fixed +- Sign-in failure caused by an empty `grant_types` on the OIDC provider; the + Authentik setup script now sets them explicitly and the smoke test guards it. +- Share modal that could open stuck and uncloseable. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..2f32c51 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,77 @@ +# Contributing + +Thanks for working on Beckham Share. This guide covers local setup, the workflow, +and the conventions a change is expected to follow. + +## Local setup + +```sh +python -m venv .venv && . .venv/bin/activate +pip install -r requirements.txt +pip install -r requirements-dev.txt # test + tooling deps + +# Point DATABASE_URL at a local Postgres (or run the compose db service), then: +uvicorn app.main:app --reload +``` + +The landing page, anonymous uploads, share pages, and downloads all work without +OIDC configured. Only the `/app` workspace requires Authentik — see +[Configuration](docs/CONFIGURATION.md) for the `OIDC_*` variables. + +## Workflow + +- **Never commit or push to `main`.** Branch, push, open a pull request, merge, + then delete the branch. +- **One focused change per pull request.** Keep diffs reviewable. +- **Use issues for tracking.** File bugs with the `bug` label and planned work + with `enhancement`. Reference the issue from the PR. +- Branch names describe the change, e.g. `add-download-analytics`, + `fix-share-modal`. + +## Tests are part of the change + +A feature is not done until its real behavior is covered by an automated test — +"the endpoint returns 200" is not "the feature works". Pick the layer that fits +(full detail in [docs/TESTING.md](docs/TESTING.md)): + +- New route or business rule → a **backend** test (`tests/backend`, pytest). +- New button / component / page interaction → a **browser E2E** test + (`tests/e2e`, Playwright) so the rendered behavior is covered. +- New deploy-time invariant (a hostname, a provider setting) → a check in + `scripts/smoke.sh`. + +Run the relevant suite before opening a PR: + +```sh +sh scripts/run-tests.sh # backend integration (throwaway Postgres) +sh scripts/run-e2e.sh # browser end-to-end (throwaway local instance) +``` + +CI runs the backend and E2E layers on every push and pull request; the PR must be +green before merge. + +## Conventions + +- **Configuration goes through `app/config.py`.** Don't read `os.environ` + elsewhere; add a typed setting and document it in + [docs/CONFIGURATION.md](docs/CONFIGURATION.md). +- **Keep the access tiers explicit.** Public, member, and infra routes are + distinct; member routes must check `in_required_group`. +- **Never expose the original filename in a URL or path.** Use the UUID; restore + the name only in `Content-Disposition`. +- **Secrets never enter the repo.** `.env`, `secrets/`, and host-specific files + are gitignored; document new settings with safe placeholders in `.env.example`. +- **Update the docs with the code.** A change that adds a route, a setting, or an + operational step updates the matching document under `docs/`. +- Match the surrounding code's style; keep functions small and the modules' + single responsibilities intact (see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)). + +## Project layout + +``` +app/ FastAPI application (routes, models, auth, storage, templates, static) +caddy/ reverse-proxy route snippet for the shared Caddy front +scripts/ deploy, secret generation, Authentik setup, test runners +tests/ backend (pytest) + e2e (Playwright) suites +docs/ architecture, configuration, operations, security, API, testing +``` diff --git a/README.md b/README.md index dd6d1c1..88861c5 100644 --- a/README.md +++ b/README.md @@ -109,11 +109,24 @@ sh scripts/run-e2e.sh # browser end-to-end tests (throwaway local instance) sh scripts/smoke.sh # post-deploy checks against the live deployment ``` +## Documentation + +Full reference documentation lives in [`docs/`](docs/README.md): + +- [Architecture](docs/ARCHITECTURE.md) — components, request flows, data model. +- [Configuration](docs/CONFIGURATION.md) — every environment variable. +- [Operations](docs/OPERATIONS.md) — deploy and the day-two runbook. +- [Security](docs/SECURITY.md) — auth, abuse controls, threat model. +- [API reference](docs/API.md) — every HTTP route. +- [Testing](docs/TESTING.md) — the three test layers. +- [Contributing](CONTRIBUTING.md) · [Changelog](CHANGELOG.md) + ## Project layout ``` app/ FastAPI application (routes, models, auth, storage, templates, static) caddy/ reverse-proxy route snippet for the shared Caddy front -scripts/ deploy, secret generation, Authentik setup -docs/ project notes +scripts/ deploy, secret generation, Authentik setup, test runners +tests/ backend (pytest) + e2e (Playwright) suites +docs/ architecture, configuration, operations, security, API, testing ``` diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..3150aa6 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,126 @@ +# API reference + +Every HTTP route exposed by the application, grouped by access tier. Routes are +defined in [`app/main.py`](../app/main.py). The interactive OpenAPI docs +(`/docs`, `/redoc`) are intentionally disabled. + +Auth tiers: + +- **Public** — no authentication. +- **Member** — requires a valid OIDC session **and** membership in the + `REQUIRED_GROUP` (`dropbox`). Otherwise `403`. +- **Infra** — health/operational endpoints. + +## Public — pages + +### `GET /` +The landing page. If the request already has a valid member session, redirects to +`/app` (302). Otherwise renders the public upload page. + +### `GET /s/{token}` +Human-facing share page for a link: filename, size, expiry, and copy / email +actions. Returns `404` if the link is unknown or its file was deleted, `410` if +the link has expired. + +### `GET /d/{token}` +Downloads the file bytes for a link. Increments the link's `download_count`. +Restores the original filename via `Content-Disposition`. Returns `404` if +missing/deleted, `410` if expired. + +## Public — anonymous upload + +### `POST /api/anon-upload` +Constrained, account-free upload from the landing page. + +**Body** (`multipart/form-data`): + +| Field | Type | Notes | +|---|---|---| +| `file` | file | **Required.** The upload. Capped at `ANON_MAX_UPLOAD_BYTES`. | +| `fp` | string | Optional client-computed fingerprint hash. | +| `fp_data` | string | Optional JSON bundle of raw fingerprint signals. | + +**Responses:** + +- `200` — `{ ok: true, token, share_url, filename, size, expires_at }`. The + anonymous link expires after `ANON_SHARE_EXPIRY_HOURS`. +- `413` — `{ ok: false, error }` when the file exceeds the anonymous size cap. +- `429` — `{ ok: false, error }` with a `Retry-After` header when the client is + over its hourly or daily budget (keyed by IP **or** fingerprint). + +Every call writes an `upload_events` audit row (IP, UA, fingerprint). See +[Security](SECURITY.md). + +## Authentication + +### `GET /login` +Starts the OIDC Authorization Code flow (redirects to Authentik). Redirects to +the canonical host first if reached on a non-canonical hostname. Returns `503` if +OIDC is not configured. + +### `GET /auth/callback` +OIDC redirect target. Exchanges the code for tokens, reads `userinfo` (including +the `groups` claim), stores the identity in the session cookie, and redirects to +`/app`. On failure, renders a friendly error page with status `400`. + +### `GET /logout` +Clears the session and redirects to `/`. + +## Member — workspace & file management + +### `GET /app` +The authenticated workspace: the signed-in member's files, each with its share +link, expiry, download count, and management actions. Redirects to `/login` if +unauthenticated; renders a `403` "not authorized" page for a signed-in +non-member. + +### `POST /api/files` +Upload a file as a member. + +**Body** (`multipart/form-data`): + +| Field | Type | Notes | +|---|---|---| +| `file` | file | **Required.** Capped at `MAX_UPLOAD_BYTES`. | +| `expiry_hours` | int | One of `SHARE_EXPIRY_OPTIONS_HOURS`; invalid values fall back to the default. | +| `fp`, `fp_data` | string | Optional fingerprint signals (recorded on the audit row). | + +**Responses:** `200` `{ ok: true, ...file_row }`; `403` if not a member; `413` if +over the size cap. + +### `POST /api/shares/{token}/expiry` +Change a link's expiry. Body: `expiry_hours` (must be an allowed option; `400` +otherwise). Clears any prior revocation. Owner-only (`404` for a token the caller +doesn't own). Returns `{ ok: true, token, expires_at }`. + +### `POST /api/shares/{token}/revoke` +Revoke a link immediately (it then reads as expired). Owner-only. Returns +`{ ok: true }`. + +### `DELETE /api/files/{file_id}` +Soft-delete a file and remove its blob from disk. Owner-only (`404` otherwise), +member-gated (`403` otherwise). Returns `{ ok: true }`. + +### `POST /api/shares/{token}/email` +Send the share link by email via the server-side SMTP relay. Body: `to` (the +recipient). **Members only** — anonymous visitors use a `mailto:` link instead, +so the public share page can't be turned into a spam relay. + +**Responses:** `200` `{ ok: true }`; `403` for non-members; `404` if the link is +missing/expired; `503` `{ ok:false, reason:"email_not_configured" }` if SMTP is +unset; `502` `{ ok:false, reason:"send_failed" }` on a send error. + +## Infra + +### `GET /healthz` +Returns `{ status: "ok" }`. Used by the container health check and the smoke +test. Hidden from the schema. + +## Conventions + +- JSON API endpoints return `{ ok: true, ... }` on success and `{ ok: false, + error | reason }` on handled failures, with an appropriate HTTP status. +- The only identifier in any public URL is the share-link `token` (a UUID). File + IDs appear only in member-authenticated management calls. +- All cookies are `Secure`, `HttpOnly` is managed by Starlette's session + middleware, and `SameSite=Lax`. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..385cee4 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,184 @@ +# Architecture + +Beckham Share is a small FastAPI service backed by PostgreSQL, fronted by the +host's shared reverse-proxy and identity infrastructure. This document describes +the moving parts, the request flows, and the data model. + +## 1. Components + +``` + https://share.beckham.ai + https://dropbox.beckham.ai (redirect → share) + https://buckets.beckham.ai (redirect → share) + │ + ▼ + ┌─────────────────────────────────────────────────────┐ + │ HAProxy (public :443, SNI routing, PROXY protocol) │ + └─────────────────────────────────────────────────────┘ + │ real client IP preserved + ▼ + ┌─────────────────────────────────────────────────────┐ + │ idp-caddy (shared Caddy front; issues TLS certs) │ + └─────────────────────────────────────────────────────┘ + │ reverse_proxy + ▼ + ┌───────────────────────────────────────────────────────────────┐ + │ beckham-share-app : 8000 (FastAPI / Uvicorn, this repo) │ + │ │ + │ routes ── auth (OIDC) ── storage ── fingerprint ── ratelimit │ + └───────────────────────────────────────────────────────────────┘ + │ │ + │ SQLAlchemy │ file blobs + ▼ ▼ + ┌──────────────────┐ ┌────────────────────────┐ + │ PostgreSQL (db) │ │ share-data volume │ + │ metadata + audit │ │ {data_dir}/blobs/ │ + └──────────────────┘ └────────────────────────┘ + + Authentication: app ──OIDC (Authorization Code)──▶ Authentik (auth.beckham.ai) + Email (members): app ──SMTP STARTTLS──▶ relay (mail.eigbox.net) +``` + +Two containers are defined in [`docker-compose.yml`](../docker-compose.yml) under +the project name `beckham-share`: + +- **app** — the FastAPI application. Renders the UI, handles uploads, share + links, downloads, the OIDC login flow, and the audit trail. +- **db** — PostgreSQL 16, holding file/link metadata and the upload-event audit + log. It stays on a private Docker network with no host port exposure. + +The app attaches to two networks: the **external `idp_proxy`** network (created +by the `idp` project) so the shared `idp-caddy` front can route to it, and an +**internal** network shared only with the database. + +### Application modules + +| Module | Responsibility | +|---|---| +| [`app/main.py`](../app/main.py) | FastAPI app, HTTP routes, request orchestration. | +| [`app/config.py`](../app/config.py) | Typed settings loaded from the environment (`pydantic-settings`). | +| [`app/db.py`](../app/db.py) | Engine, session factory, `Base`, table creation. | +| [`app/models.py`](../app/models.py) | ORM models: `File`, `ShareLink`, `UploadEvent`. | +| [`app/auth.py`](../app/auth.py) | OIDC client, session user, group-membership check. | +| [`app/storage.py`](../app/storage.py) | Streaming blob storage on the local volume. | +| [`app/fingerprint.py`](../app/fingerprint.py) | Client IP, UA parsing, fingerprint hashing. | +| [`app/ratelimit.py`](../app/ratelimit.py) | Rolling-window anonymous-upload limits. | +| [`app/emailer.py`](../app/emailer.py) | Server-side "email this link" over SMTP. | +| `app/templates/`, `app/static/` | Jinja2 templates and the front-end assets. | + +## 2. Access tiers + +Every route falls into one of three tiers (see [API reference](API.md) for the +full list): + +1. **Public, anonymous** — the landing page (`/`), the anonymous upload endpoint + (`/api/anon-upload`), and the public share/download routes (`/s/{token}`, + `/d/{token}`). No account required; constrained by the abuse controls. +2. **Authenticated member** — the workspace (`/app`) and the file-management API + (`/api/files`, expiry/revoke/delete, server-side email). Requires a valid + OIDC session **and** membership in the `dropbox` group. +3. **Infrastructure** — `/healthz`, used by the container health check and smoke + tests. + +## 3. Request flows + +### Anonymous upload (the first-order feature) + +``` +Browser app db / disk + │ POST /api/anon-upload │ │ + │ (multipart: file, fp) │ │ + │ ─────────────────────────▶ │ │ + │ │ compute fingerprint │ + │ │ check_anonymous() ─────────▶ count recent events + │ │ (429 if over budget) │ + │ │ stream to {data}/blobs/id ─▶ blob written, sha256 + │ │ insert File + ShareLink ──▶ rows committed + │ │ record UploadEvent ───────▶ audit row + │ ◀───────────────────────── │ │ + │ { token, share_url, ... } │ │ +``` + +The response carries a `share_url` of the form `BASE_URL/s/`. The original +filename is **not** in the URL — it is stored in the database and restored only +in the download's `Content-Disposition` header. + +### Member sign-in (OIDC Authorization Code) + +``` +/login ─▶ redirect to Authentik authorize endpoint + ◀─ user authenticates, Authentik enforces the `dropbox` application binding +/auth/callback ─▶ exchange code for tokens, read userinfo (incl. `groups` claim) + ─▶ store {sub, email, name, groups} in the signed session cookie + ─▶ redirect to /app +``` + +`/app` re-checks the `groups` claim as an in-app backstop; non-members get a +403 "not authorized" page. Because the session cookie is bound to one host, the +auth routes always run on the canonical host (`BASE_URL`); requests arriving on +`dropbox.` or `buckets.` are redirected there first (`ensure_canonical_host`). + +### Download + +``` +GET /d/{token} ─▶ look up ShareLink ─▶ 404 if missing/deleted, 410 if expired + ─▶ increment download_count + ─▶ FileResponse(blob, filename=original_name) +``` + +`/s/{token}` is the human-facing share page (file name, size, expiry, copy / +email actions); `/d/{token}` is the raw byte stream. + +## 4. Data model + +Three tables, defined in [`app/models.py`](../app/models.py): + +### `files` +The uploaded blob and its metadata. The primary key is a UUID that **doubles as +the on-disk storage key** and is never exposed in a URL. A `NULL` `owner_sub` +marks an anonymous (landing-page) upload. `expires_at` is a hard expiry for the +blob itself (anonymous uploads always get one); `deleted` is a soft-delete flag. + +### `share_links` +A UUID-addressed link to a file — the `token` is the only identifier that ever +appears in a URL. A link is considered expired (`is_expired`) when it is +`revoked`, past its `expires_at`, or has reached an optional `max_downloads`. +`download_count` is incremented on every successful download. + +### `upload_events` +One row per upload attempt — the abuse-review audit trail. Captures `ip`, +`user_agent`, `accept_language`, a stable `fingerprint` hash plus the raw +`fingerprint_data` bundle, and a parsed UA breakdown (`ua_browser`, `ua_os`, +`ua_device`). The rate limiter counts these rows; see [Security](SECURITY.md). + +``` +files (1) ───< (N) share_links + │ + └──────────< (N) upload_events +``` + +## 5. Hostnames and the canonical host + +The service answers on three hostnames, but share links and the OIDC session +cookie are tied to a single canonical origin (`BASE_URL`, +`https://share.beckham.ai`). `dropbox.beckham.ai` and `buckets.beckham.ai` +resolve to the same app but redirect the authenticated flow (`/login`, `/app`, +`/auth/callback`) to the canonical host so the session cookie and redirect URIs +stay consistent. The Caddy snippet that performs the routing is +[`caddy/dropbox.beckham.ai.caddy`](../caddy/dropbox.beckham.ai.caddy). + +## 6. Why these choices + +- **App-side OIDC (not proxy `forward_auth`).** The app needs the `groups` claim + and its own session to distinguish members from anonymous visitors on the same + origin; doing auth in-app keeps the public landing page and the gated workspace + under one service. +- **UUID storage keys.** Using the file UUID as both the URL token's backing + record and the on-disk name means the real filename never touches a path or a + URL — only the database and the download header. +- **Database-backed rate limiting.** Counting the audit rows that already exist + means the limit holds across worker processes and restarts with no extra + store. See [Security](SECURITY.md). +- **Streaming uploads with an early size cap.** Blobs are written in 1 MiB chunks + and aborted (with cleanup) the moment they exceed the limit, so an oversized + upload never has to be fully buffered. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md new file mode 100644 index 0000000..b34c943 --- /dev/null +++ b/docs/CONFIGURATION.md @@ -0,0 +1,93 @@ +# Configuration + +All configuration is read from the environment at startup by +[`app/config.py`](../app/config.py); the rest of the code never touches +`os.environ` directly. In production the values are supplied through the +container's `.env` file (see [`.env.example`](../.env.example)), which is **never +committed**. `scripts/generate-secrets.sh` creates a `.env` with a random +`SECRET_KEY` and `DB_PASSWORD`; the OIDC credentials are filled in after the +Authentik setup step (see [Operations](OPERATIONS.md)). + +Settings are typed and validated by `pydantic-settings`; unknown variables are +ignored. Booleans accept `true`/`false`/`1`/`0`. + +## Identity & branding + +| Variable | Default | Purpose | +|---|---|---| +| `APP_NAME` | `Beckham Share` | Display name used in the UI and outgoing email. | +| `BRAND_DOMAIN` | `beckham.ai` | Brand domain shown in email footers. | +| `BASE_URL` | `https://share.beckham.ai` | **Canonical public origin.** Used to build absolute share links and the OIDC redirect URI, and to decide the canonical host for redirects. | + +## Security & sessions + +| Variable | Default | Purpose | +|---|---|---| +| `SECRET_KEY` | `dev-insecure-change-me` | Signs the session cookie. **Must** be a long random value in production — rotating it logs everyone out. | +| `TRUST_FORWARDED_FOR` | `true` | Trust `X-Forwarded-For` / `X-Real-IP` for the client IP. Correct when behind the Caddy/HAProxy front; set `false` only if the app is exposed directly. | + +## Storage & database + +| Variable | Default | Purpose | +|---|---|---| +| `DATA_DIR` | `/data` | Blob root; files live under `{DATA_DIR}/blobs/`. Backed by the `share-data` volume. | +| `DATABASE_URL` | `postgresql+psycopg2://share:share@db:5432/share` | SQLAlchemy database URL. Compose injects the real password via `DB_PASSWORD`. | + +## OIDC (Authentik) + +| Variable | Default | Purpose | +|---|---|---| +| `OIDC_DISCOVERY_URL` | *(empty)* | The provider's OpenID discovery document, e.g. `https://auth.beckham.ai/application/o/beckham-share/.well-known/openid-configuration`. | +| `OIDC_CLIENT_ID` | *(empty)* | OAuth2 client ID from the Authentik provider. | +| `OIDC_CLIENT_SECRET` | *(empty)* | OAuth2 client secret. | +| `OIDC_SCOPES` | `openid profile email groups` | Requested scopes. The `groups` scope is required for the in-app group check. | +| `REQUIRED_GROUP` | `dropbox` | Group required for the authenticated workspace. Enforced at Authentik **and** in-app. | + +> Sign-in stays disabled until all three of discovery URL, client ID, and client +> secret are set (`settings.oidc_configured`). The public landing page, uploads, +> share pages, and downloads all work without OIDC configured. + +## Upload limits & abuse controls + +| Variable | Default | Purpose | +|---|---|---| +| `MAX_UPLOAD_BYTES` | `5368709120` (5 GiB) | Per-file cap for authenticated members. | +| `ANON_MAX_UPLOAD_BYTES` | `104857600` (100 MiB) | Per-file cap for anonymous landing-page uploads. | +| `ANON_UPLOADS_PER_HOUR` | `5` | Rolling 1-hour anonymous upload budget per client (IP or fingerprint). | +| `ANON_UPLOADS_PER_DAY` | `20` | Rolling 24-hour anonymous upload budget per client. | + +See [Security](SECURITY.md) for how the limits are keyed and enforced. + +## Share-link expiry + +| Variable | Default | Purpose | +|---|---|---| +| `SHARE_EXPIRY_OPTIONS_HOURS` | `1,24,168,720,0` | Expiry choices offered in the UI, in hours (`0` = never). Default set is 1 h, 1 d, 7 d, 30 d, never. | +| `DEFAULT_SHARE_EXPIRY_HOURS` | `168` (7 days) | Default expiry for member uploads. | +| `ANON_SHARE_EXPIRY_HOURS` | `24` | Fixed expiry for anonymous uploads (they never get a "never" option). | + +## Email (share-by-email) + +Optional. Until SMTP is configured, the front end falls back to a `mailto:` link, +and the server-side relay (members only) returns a "not configured" response. + +| Variable | Default | Purpose | +|---|---|---| +| `SMTP_HOST` | *(empty)* | SMTP server hostname. Setting host + user + password enables server-side email (`settings.email_enabled`). | +| `SMTP_PORT` | `587` | SMTP port (587 for STARTTLS). | +| `SMTP_USER` | *(empty)* | SMTP username. | +| `SMTP_PASSWORD` | *(empty)* | SMTP password. | +| `SMTP_FROM` | `Beckham Share ` | `From` header on outgoing mail. | +| `SMTP_USE_TLS` | `true` | Use STARTTLS before authenticating. | + +> The relay used in production presents a shared `*.eigbox.net` certificate, so +> `docker-compose.yml` maps `mail.eigbox.net` to its IP via `extra_hosts` to keep +> STARTTLS certificate verification valid. Set `SMTP_HOST=mail.eigbox.net`. + +## Derived settings (not environment variables) + +These are computed properties on the settings object: + +- `email_enabled` — true when `SMTP_HOST`, `SMTP_USER`, and `SMTP_PASSWORD` are all set. +- `expiry_options` — `SHARE_EXPIRY_OPTIONS_HOURS` parsed into a list of ints. +- `oidc_configured` — true when discovery URL, client ID, and client secret are all set. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md new file mode 100644 index 0000000..04eb78b --- /dev/null +++ b/docs/OPERATIONS.md @@ -0,0 +1,127 @@ +# Operations + +How to deploy Beckham Share and run it day to day. The service runs as two +containers (`app` + `db`) behind the host's shared Caddy/HAProxy front and signs +in against an existing Authentik identity provider. + +Host-specific connection details (SSH target, destination path) live in +`scripts/deploy.env`, which is **gitignored**. `deploy.sh` reads it, or falls +back to the `SHARE_HOST` / `SHARE_SSH_KEY` / `SHARE_DEST` environment variables. + +## Prerequisites + +- A host running Docker with the shared `idp` stack already up: Authentik, the + `idp-caddy` front, and the external `idp_proxy` Docker network. +- DNS for `share` / `dropbox` / `buckets.beckham.ai` pointing at the public IP + (one-time, manual). +- An SMTP mailbox if server-side email is wanted (optional; the UI falls back to + `mailto:` until it exists). + +## First deploy + +```sh +# 1. Ship the code, build the image, and start the stack on the host. +# (tar | ssh; secrets and data are never synced. Generates .env on first run.) +sh scripts/deploy.sh + +# 2. Configure Authentik: create the OIDC provider/application, the `dropbox` +# group, and its members. Prints OIDC_CLIENT_ID / OIDC_CLIENT_SECRET. +docker compose -p idp exec -T authentik-server ak shell < scripts/setup-authentik.py + +# 3. Put the printed OIDC_CLIENT_ID / OIDC_CLIENT_SECRET (and OIDC_DISCOVERY_URL) +# into the host's .env, then restart the app. +docker compose up -d app + +# 4. Wire the reverse proxy: add caddy/dropbox.beckham.ai.caddy to the shared +# Caddyfile and reload Caddy. +docker compose -p idp exec caddy caddy reload --config /etc/caddy/Caddyfile + +# 5. Verify. +sh scripts/smoke.sh +``` + +`scripts/generate-secrets.sh` writes a `.env` with a random `SECRET_KEY` and +`DB_PASSWORD` on first deploy; everything else has a sane default (see +[Configuration](CONFIGURATION.md)). + +## Routine deploy (subsequent changes) + +```sh +sh scripts/deploy.sh # re-ships, rebuilds, restarts +sh scripts/smoke.sh # confirm health, a real round-trip, TLS, provider config +``` + +`deploy.sh` ships the working tree with `tar | ssh` (no rsync dependency), +excluding `.git`, `.env`, `secrets/`, and `data/`, then runs +`docker compose up -d --build` on the host. Secrets and uploaded blobs stay on +the host across deploys (the `share-data` and `share-db` volumes persist). + +## Day-two runbook + +### Logs +```sh +# On the host, in the deploy directory: +docker compose logs -f app # application log (uploads, auth, email) +docker compose logs db # database +docker compose ps # container status / health +``` +The app logs each anonymous upload with its id, IP, fingerprint prefix, and size, +and warns on OIDC callback failures or email send failures. + +### Health +`GET /healthz` returns `{ "status": "ok" }` and backs the container health check. +`scripts/smoke.sh` exercises health on all three hostnames plus a full +upload → share → download round-trip, the canonical redirect, the TLS +certificate, and the Authentik provider configuration. + +### Backups +Two pieces of state to back up: +- **Database** — `docker compose exec db pg_dump -U share share > backup.sql`. +- **Blobs** — the `share-data` volume (`{DATA_DIR}/blobs/`). Back up the volume + directory or `docker run --rm -v beckham-share_share-data:/data ...` to tar it. + +A blob is only meaningful alongside its database row, so snapshot both together. + +### Restart / recover +```sh +docker compose restart app # restart just the app +docker compose up -d --force-recreate # recreate both containers +``` +On startup the app ensures the blob directory exists and creates any missing +tables (`init_db`), so a fresh container against the existing volumes comes back +clean. + +### Rotating secrets +- `SECRET_KEY` — changing it invalidates all sessions (everyone re-logs in). +- OIDC client secret — rotate in Authentik, update `.env`, `docker compose up -d app`. +- SMTP password — update `.env`, restart the app. + +## Authentik configuration + +`scripts/setup-authentik.py` is idempotent — re-running it reconciles the +provider, application, group, members, the `groups` scope mapping, and the policy +binding. It configures three redirect URIs (the `share`, `dropbox`, and `buckets` +callbacks) and ensures the provider's grant types include `authorization_code` +and `refresh_token`. + +> A historical sign-in outage was caused by an **empty `grant_types`** on the +> provider. The setup script sets them explicitly, and `smoke.sh` asserts that +> `authorization_code` is present and the `share` callback is registered, so the +> same failure can't ship silently again. + +## Reverse proxy + +`caddy/dropbox.beckham.ai.caddy` is a plain `reverse_proxy` to +`beckham-share-app:8000` — **authentication is in-app (OIDC), not Caddy +`forward_auth`.** Add the snippet to the shared Caddyfile and reload Caddy. TLS +certificates are issued and renewed automatically by Caddy. + +## Common issues + +| Symptom | Likely cause | Check | +|---|---|---| +| Sign-in returns `400` immediately | Provider misconfiguration (e.g. empty grant types) | `smoke.sh` provider check; re-run `setup-authentik.py`. | +| `503` on `/login` | OIDC not configured | `OIDC_*` set in `.env`? | +| Anonymous upload `429` | Rate limit hit | Expected; tune `ANON_UPLOADS_PER_*`. | +| Email returns `email_not_configured` | SMTP unset | Set `SMTP_*`; STARTTLS host mapping present. | +| Member sees "not authorized" | Not in the `dropbox` group | Group membership in Authentik; `groups` scope mapped. | diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..d8c1128 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,23 @@ +# Documentation + +Reference documentation for Beckham Share. Start with the [project README](../README.md) +for a high-level overview; the documents below go deeper. + +| Document | What it covers | +|---|---| +| [Architecture](ARCHITECTURE.md) | Components, request flows, the data model, and how the service plugs into the host's reverse-proxy / identity stack. | +| [Configuration](CONFIGURATION.md) | Every environment variable — purpose, default, and how it is used. | +| [Operations](OPERATIONS.md) | First deploy, the Authentik / Caddy wiring, and the day-two runbook (logs, backups, upgrades, recovery). | +| [Security](SECURITY.md) | Authentication, the group gate, the anonymous-upload abuse controls, fingerprinting, and the threat model. | +| [API reference](API.md) | Every HTTP route — method, auth tier, parameters, and responses. | +| [Testing](TESTING.md) | The three test layers (backend, browser E2E, live smoke) and how to run and extend them. | +| [Contributing](../CONTRIBUTING.md) | Local setup, the branch/PR workflow, and the conventions a change must follow. | +| [Changelog](../CHANGELOG.md) | Notable changes, newest first. | + +## Troubleshooting notes + +Longer-form write-ups of specific problems and how they were diagnosed live in +[`troubleshooting/`](troubleshooting/): + +- [Why is the upload page asking for my microphone?](troubleshooting/android-file-picker.md) + — an Android file-input / `accept`-attribute investigation. diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 0000000..75e5940 --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,118 @@ +# Security + +Beckham Share exposes a **public** upload form to the internet, so its security +posture is deliberately split: a tightly constrained anonymous tier and an +identity-gated member tier. This document covers the authentication model, the +abuse controls, the data handled, and the threat model. + +## 1. Authentication & authorization + +- **Protocol.** OIDC Authorization Code flow against Authentik + (`auth.beckham.ai`), via Authlib. See [`app/auth.py`](../app/auth.py). +- **Session.** On success the app stores `{sub, email, name, groups}` in a + signed, `Secure`, `SameSite=Lax` session cookie (Starlette's + `SessionMiddleware`, signed with `SECRET_KEY`). The app owns its own session; + it does not rely on a proxy to inject identity. +- **Two-layer group gate.** The workspace is restricted to members of the + `dropbox` group in two independent places: + 1. **At Authentik** — the application is bound to the group, so non-members + can't complete the flow at the IdP. + 2. **In-app backstop** — `/app` and every management route re-check the + `groups` claim (`CurrentUser.in_required_group`). +- **No-groups fallback.** If a token arrives with no `groups` claim (scope + mapping missing), the app trusts Authentik's application-level binding rather + than locking everyone out, and logs a warning so the gap is visible. Keep the + `groups` scope configured so the in-app check is authoritative. + +## 2. Anonymous-upload abuse controls + +The landing page is a public form. Four layers keep it from becoming a dump: + +### Size cap +Anonymous uploads are capped at `ANON_MAX_UPLOAD_BYTES` (default 100 MiB), +enforced *while streaming* — the upload is aborted and the partial file deleted +the moment it exceeds the limit (`app/storage.py`), so an oversized file is never +fully buffered. Members have a separate, larger cap (`MAX_UPLOAD_BYTES`). + +### Rate limiting +`app/ratelimit.py` enforces a rolling-window budget: + +- `ANON_UPLOADS_PER_HOUR` (default 5) in the last hour, and +- `ANON_UPLOADS_PER_DAY` (default 20) in the last 24 hours. + +The count is taken from the `upload_events` audit rows, keyed by **IP _or_ +fingerprint** (`OR`), so rotating just one signal does not reset the budget. +Because it counts rows that already exist in the database, the limit holds across +worker processes and restarts with no separate store. Over-budget requests get +`429` with a `Retry-After` header. + +### Short auto-expiry +Anonymous links always expire after `ANON_SHARE_EXPIRY_HOURS` (default 24 h); +there is no "never" option for them. The blob carries the same hard expiry. + +### Audit trail + fingerprinting +Every upload writes an `upload_events` row for review — see below. + +## 3. Fingerprinting & the audit trail + +`app/fingerprint.py` records, per upload: + +- **Client IP** — from `X-Forwarded-For` / `X-Real-IP` when `TRUST_FORWARDED_FOR` + is set (the real client IP is preserved through HAProxy's PROXY protocol), + otherwise the socket peer. +- **User agent** — raw, plus a parsed `browser` / `os` / `device` breakdown for + human-readable review. +- **Accept-Language**. +- **Fingerprint hash** — a SHA-256 over either the client-supplied browser + fingerprint (canvas / WebGL / screen / timezone signals, richer) or, for + JS-less clients, a header/network-derived basis so they are still grouped. +- **Raw fingerprint bundle** (`fingerprint_data`) for later inspection. + +> Fingerprinting here is a **deterrent and an audit aid, not a security +> boundary.** It can be spoofed; it exists to make casual abuse traceable and +> rate-limitable, not to authenticate anyone. + +## 4. Data handling + +- **Filename privacy.** The original filename never appears in a URL or an + on-disk path. Blobs are stored under `{DATA_DIR}/blobs/`; the real name + lives only in the database and is restored in the download's + `Content-Disposition`. +- **Opaque links.** Share URLs contain only a random UUID token; they are + unguessable and reveal nothing about the file. +- **Integrity.** Each blob's SHA-256 is computed on upload and stored. +- **Deletion.** Deleting a file soft-deletes the row and removes the blob from + disk; expired/revoked links stop serving immediately. +- **Secrets.** `SECRET_KEY`, `DB_PASSWORD`, the OIDC client secret, and SMTP + credentials live only in the host's `.env` (gitignored) — never in the repo. + +## 5. Transport & network + +- **HTTPS everywhere.** TLS is terminated by the shared Caddy front (Let's + Encrypt); the smoke test asserts a valid certificate after each deploy. +- **Database isolation.** PostgreSQL is on an internal Docker network with no + host port published. +- **Canonical host.** The session cookie and OIDC redirect URIs are bound to + `BASE_URL`; other hostnames redirect the authenticated flow there, so a cookie + can't be set on an unexpected origin. + +## 6. Threat model (summary) + +| Threat | Mitigation | +|---|---| +| Anonymous form used to host abusive material | Size cap, rate limit, short auto-expiry, IP + fingerprint audit trail. | +| Link guessing / enumeration | UUID tokens; no filename or sequential ID in URLs. | +| Unauthorized workspace access | OIDC + group binding at the IdP, re-checked in-app. | +| Public share page abused as a spam relay | Server-side email is members-only; anonymous uses `mailto:`. | +| Oversized-upload resource exhaustion | Streaming write with an early abort + partial-file cleanup. | +| Session forgery | Signed session cookie (`SECRET_KEY`); `Secure` + `SameSite=Lax`. | +| Spoofed client IP | Real IP preserved via PROXY protocol; `X-Forwarded-For` trusted only behind the front. | + +## 7. Operational guidance + +- Set a strong, unique `SECRET_KEY` in production; rotating it invalidates all + sessions. +- Keep the `groups` scope mapping configured so the in-app group check is + authoritative rather than relying on the fallback. +- Review the `upload_events` table periodically for anomalous IPs/fingerprints. +- Tune `ANON_*` limits to the host's tolerance; they are all environment-driven. diff --git a/docs/reports/Beckham-Share-Build-Report.pdf b/docs/reports/Beckham-Share-Build-Report.pdf new file mode 100644 index 0000000..dbf41d7 Binary files /dev/null and b/docs/reports/Beckham-Share-Build-Report.pdf differ