-
Notifications
You must be signed in to change notification settings - Fork 0
docs: rewrite READMEs and remove outdated MediaConvert docs (#455) #473
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
9b08834
docs: rewrite READMEs and remove outdated MediaConvert docs
cursoragent e904c3c
fix(api-node): repair Deno Deploy bundle boot (AWS SDK require)
cursoragent faae6da
fix(api-node): make Deno Deploy install/build self-contained
cursoragent 0435ba2
fix(api-node): drop invalid Deno deploy org/app keys; cover root app dir
cursoragent 78fa0dc
fix(api-node): mirror Deno Deploy config at repo root
cursoragent 3453963
fix(api-node): stop overriding Deno Deploy dashboard build config
cursoragent 69d80eb
fix(api-node): source Deno Deploy install/build without stale exports
cursoragent 75247e7
fix(api-node): Deno Deploy source config at root and package
cursoragent 7bf863d
fix(api-node): remove root deno workspace to include package in Deploy
cursoragent a6cd891
fix(api-node): restore dashboard-driven Deno Deploy config
cursoragent a64ac2c
docs: tighten Wrangler/Deno deploy notes; harden smoke-bundle
cursoragent File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,147 +1,139 @@ | ||
| # VMP (Video Monetization Platform) | ||
|
|
||
| VMP is a Cloudflare-based video subscription platform with a Worker API, Nuxt web app, and shared TypeScript types. | ||
| Subscription-gated HLS video streaming: Cloudflare Worker API + Nuxt 4 web app, with optional Deno backup API and a media VM for transcoding. | ||
|
|
||
| ## Contents | ||
|
|
||
| - [Architecture](#architecture) | ||
| - [Monorepo packages](#monorepo-packages) | ||
| - [Documentation map](#documentation-map) | ||
| - [Deployment model (high level)](#deployment-model-high-level) | ||
| - [Documentation](#documentation) | ||
| - [Prerequisites](#prerequisites) | ||
| - [Local setup](#local-setup) | ||
| - [Manual deploy commands](#manual-deploy-commands) | ||
| - [Runtime secrets/vars](#runtime-secretsvars) | ||
| - [Optional MediaConvert pipeline](#optional-mediaconvert-pipeline-additive-to-local-rclone-flow) | ||
| - [API docs (`docs/`)](#api-docs-docs) | ||
| - [Notes](#notes) | ||
| - [Local development](#local-development) | ||
| - [Deploy](#deploy) | ||
| - [Secrets and configuration](#secrets-and-configuration) | ||
| - [Media encoding](#media-encoding) | ||
|
|
||
| ## Architecture | ||
|
|
||
| | Layer | Technology | | ||
| | --- | --- | | ||
| | API | Cloudflare Worker (`@vmp/api`) — REST, auth, Stripe, push, RSS | | ||
| | Web | Nuxt 4 / Vue 3 (`@vmp/web`) — Cloudflare **Workers** SSR (`vmp-web-worker-dev` / `vmp-web-worker-prod`) | | ||
| | Database | Cloudflare D1 (SQLite); Postgres shim on Deno backup | | ||
| | Object storage | Cloudflare R2 (pluggable via `@vmp/storage`) | | ||
| | Payments | Stripe (+ optional legacy Qerko via `@vmp/payments`) | | ||
| | Email / push | Brevo transactional, Web Push (VAPID) | | ||
| | Transcoding | `@vmp/media-pipeline` on a media VM (SVT Encore + Shaka → R2) | | ||
| | Backup API | `@vmp/api-node` on Deno Deploy (same handlers, Postgres + S3) | | ||
|
|
||
| Cloudflare **Pages** (`vmp-fe`) is deprecated. Do not attach production hostnames to Pages. | ||
|
|
||
| ```text | ||
| Browser ──► @vmp/web (Worker SSR) | ||
| │ | ||
| ▼ | ||
| @vmp/api (Worker) ──► D1 / R2 / KV | ||
| │ | ||
| ├── Stripe webhooks, Brevo, Web Push | ||
| └── pipeline-status ◄── @vmp/media-pipeline (VM) | ||
| ``` | ||
|
|
||
| ## Monorepo packages | ||
|
|
||
| | Package | Path | Role | | ||
| | --- | --- | --- | | ||
| | `@vmp/api` | [`packages/api`](packages/api) | Cloudflare Worker API + D1/R2/KV integrations | | ||
| | `@vmp/web` | [`packages/web`](packages/web) | Nuxt 4 frontend deployed to Cloudflare Pages | | ||
| | `@vmp/shared` | [`packages/shared`](packages/shared) | Shared TypeScript contracts | | ||
| | `@vmp/api-node` | [`packages/api-node`](packages/api-node) | Deno Deploy backup API (Postgres + S3 adapters) — see [README](packages/api-node/README.md) | | ||
| | `@vmp/media-pipeline` | [`packages/media-pipeline`](packages/media-pipeline) | Media VM: SVT Encore transcoding + Shaka HLS + R2 — see [README](packages/media-pipeline/README.md) | | ||
| | `@vmp/offloading` | [`packages/offloading`](packages/offloading) | R2↔Garage hot/cold tier orchestration — see [README](packages/offloading/README.md) | | ||
| | `@vmp/moq-probe` | [`packages/moq-probe`](packages/moq-probe) | MoQ broadcast diagnostic probe — see [README](packages/moq-probe/README.md) | | ||
| | `@vmp/api` | [`packages/api`](packages/api) | Primary Cloudflare Worker API | | ||
| | `@vmp/web` | [`packages/web`](packages/web) | Nuxt 4 frontend (Workers SSR) | | ||
| | `@vmp/shared` | [`packages/shared`](packages/shared) | Shared TypeScript types | | ||
| | `@vmp/storage` | [`packages/storage`](packages/storage) | Pluggable object storage (R2 / S3-compatible) — [README](packages/storage/README.md) | | ||
| | `@vmp/payments` | [`packages/payments`](packages/payments) | Payment provider registry (Stripe, legacy Qerko) — [README](packages/payments/README.md) | | ||
| | `@vmp/api-node` | [`packages/api-node`](packages/api-node) | Deno Deploy backup API — [README](packages/api-node/README.md) | | ||
| | `@vmp/media-pipeline` | [`packages/media-pipeline`](packages/media-pipeline) | Media VM: Encore + Shaka HLS + R2 — [README](packages/media-pipeline/README.md) | | ||
| | `@vmp/offloading` | [`packages/offloading`](packages/offloading) | R2 ↔ Garage hot/cold tiering — [README](packages/offloading/README.md) | | ||
| | `@vmp/moq-probe` | [`packages/moq-probe`](packages/moq-probe) | MoQ live broadcast diagnostic probe — [README](packages/moq-probe/README.md) | | ||
|
|
||
| Core API and web packages do not ship separate READMEs; see [AGENTS.md](AGENTS.md) for architecture, auth, schema, and local dev. | ||
| Core API/web packages do not ship separate READMEs; see [AGENTS.md](AGENTS.md) for auth, D1 schema, roles, and agent workflow. | ||
|
|
||
| ## Documentation map | ||
| ## Documentation | ||
|
|
||
| | Document | Description | | ||
| | Document | Audience | | ||
| | --- | --- | | ||
| | [AGENTS.md](AGENTS.md) | Canonical agent/dev guide: git workflow, D1 schema, auth, secrets, roadmap | | ||
| | [DEPLOYMENT.md](DEPLOYMENT.md) | Environment variables, CI/CD, multi-domain deploy | | ||
| | [packages/api-node/README.md](packages/api-node/README.md) | Deno Deploy API backup, Postgres replication ingest | | ||
| | [packages/media-pipeline/README.md](packages/media-pipeline/README.md) | Encore transcoding orchestration, supervisor, webhooks, TTP logging | | ||
| | [packages/media-pipeline/MIGRATION.md](packages/media-pipeline/MIGRATION.md) | Cutover guide from legacy `podcast-host` | | ||
| | [packages/media-pipeline/systemd/README.md](packages/media-pipeline/systemd/README.md) | `vmp-supervisor` systemd unit install and ops | | ||
| | [packages/offloading/README.md](packages/offloading/README.md) | Garage compose, demote/promote scripts | | ||
| | [packages/offloading/DEPLOYMENT.md](packages/offloading/DEPLOYMENT.md) | Docker/Compose deployment for offloading | | ||
| | [packages/moq-probe/README.md](packages/moq-probe/README.md) | Live MoQ probe usage and recorder recommendations | | ||
| | [.cursor/README.md](.cursor/README.md) | Cursor workspace git/deploy policy for agents | | ||
| | [docs/pills-external-update-api.md](docs/pills-external-update-api.md) | Pills external update API | | ||
|
|
||
| ## Deployment model (high level) | ||
|
|
||
| - Pushes to `main` run staging deploy in [`.github/workflows/deploy.yml`](.github/workflows/deploy.yml): API Worker + web Worker (`vmp-web-worker-dev`). | ||
| - Version tags (`v*.*.*`) run production deploy: API Worker + web Worker (`vmp-web-worker-prod`). | ||
| - Frontend is **Cloudflare Workers only** (Nuxt SSR). Pages is deprecated. | ||
| - Deploy pipeline fails fast on type-checking before build/deploy: | ||
| - `@vmp/shared` `tsc --noEmit` | ||
| - `@vmp/api` `tsc --noEmit` | ||
| - `@vmp/web` `nuxi prepare && nuxi typecheck` | ||
|
|
||
| See [DEPLOYMENT.md](DEPLOYMENT.md) for env templates, secrets, and domain overrides. | ||
| | [AGENTS.md](AGENTS.md) | **Canonical** architecture, git workflow, secrets, local Cloud setup, roadmap | | ||
| | [DEPLOYMENT.md](DEPLOYMENT.md) | CI/CD, env vars, smoke checks, bootstrap / rollback | | ||
| | [docs/README.md](docs/README.md) | Index of API notes and historical design docs | | ||
| | [packages/web/docs/workers-deploy-env.md](packages/web/docs/workers-deploy-env.md) | Web Worker build-time env vars | | ||
| | Package READMEs under `packages/*` | Package-specific ops (pipeline, api-node, storage, …) | | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. Node.js 20+ and npm 10+. | ||
| 2. Cloudflare account with: | ||
| - Two Workers per environment (API + Nuxt web frontend) | ||
| - D1 database | ||
| - R2 bucket | ||
| - KV namespaces | ||
| 3. Repository secrets configured for CI deploy: | ||
| - `CLOUDFLARE_API_TOKEN_STAGING` | ||
| - `CLOUDFLARE_ACCOUNT_ID_STAGING` | ||
| - `CLOUDFLARE_API_TOKEN_PROD` | ||
| - `CLOUDFLARE_ACCOUNT_ID_PROD` | ||
| 1. Node.js 20+ and npm 10+ (`packageManager` in root `package.json`). | ||
| 2. Cloudflare account with Workers, D1, R2, and KV (per environment). | ||
| 3. CI deploy secrets: `CLOUDFLARE_API_TOKEN_{STAGING,PROD}` and `CLOUDFLARE_ACCOUNT_ID_{STAGING,PROD}`. | ||
|
|
||
| ## Local setup | ||
| ## Local development | ||
|
|
||
| 1. Install dependencies: | ||
| - `npm ci` | ||
| 2. Build service worker helper from TS source: | ||
| - `npm run build:sw-push --workspace=@vmp/web` | ||
| 3. Type-check everything: | ||
| - `npm run typecheck` | ||
| ```bash | ||
| npm ci | ||
| npm run typecheck | ||
|
|
||
| Full local dev (API on `:8787`, web on `:3000`, D1 migrations): [AGENTS.md → Cursor Cloud-specific instructions](AGENTS.md#cursor-cloud-specific-instructions). | ||
| # API (Wrangler, port 8787) — needs packages/api/.dev.vars | ||
| npm run dev --workspace=@vmp/api | ||
|
|
||
| ## Manual deploy commands | ||
| # Web (Nuxt, port 3000) | ||
| API_URL=http://localhost:8787 npm run dev --workspace=@vmp/web | ||
| ``` | ||
|
|
||
| These are useful for controlled/manual rollouts outside GitHub Actions. | ||
| Apply D1 migrations locally before serving data: | ||
|
|
||
| 1. API deploy: | ||
| - `npm run deploy:api` | ||
| 2. Web deploy (from `packages/web` after `npm run build`): | ||
| - Staging Worker: `npm run deploy --workspace=@vmp/web` | ||
| - Production Worker: `npm run deploy:prod --workspace=@vmp/web` | ||
| ```bash | ||
| cd packages/api | ||
| for f in $(ls -1 migrations/*.sql | sort); do | ||
| npx wrangler d1 execute video-subscription-db --local --file="$f" | ||
| done | ||
| ``` | ||
|
|
||
| ## Runtime secrets/vars | ||
| Seed videos start as drafts. To publish them for the homepage: | ||
|
|
||
| Use Wrangler secrets for sensitive values (never commit secrets). Required values are documented in [AGENTS.md](AGENTS.md) and [DEPLOYMENT.md](DEPLOYMENT.md) (JWT, Stripe, Brevo, VAPID, RSS, TOTP encryption, etc.). | ||
| ```bash | ||
| npx wrangler d1 execute video-subscription-db --local \ | ||
| --command="UPDATE videos SET publish_status = 'published', published_at = CURRENT_TIMESTAMP WHERE publish_status = 'draft';" | ||
| ``` | ||
|
|
||
| ### Optional MediaConvert pipeline (additive to local rclone flow) | ||
| Full Cloud-agent local notes (secrets, gotchas): [AGENTS.md → Cursor Cloud-specific instructions](AGENTS.md#cursor-cloud-specific-instructions). | ||
|
|
||
| The existing local/watchfolder/rclone pipeline remains the default and is unchanged. | ||
| An optional cloud path can be enabled for direct source uploads + AWS Elemental MediaConvert. | ||
| ## Deploy | ||
|
|
||
| | Trigger | What deploys | | ||
| | --- | --- | | ||
| | Push to `main` | Staging: API Worker + web Worker (`vmp-web-worker-dev`) via [`.github/workflows/deploy.yml`](.github/workflows/deploy.yml) | | ||
| | Tag `v*.*.*` | Production: API Worker + web Worker (`vmp-web-worker-prod`) | | ||
| | Push / PR (Deno git integration) | `@vmp/api-node` preview/production build on Deno Deploy — **not** from `deploy.yml`. PR check status is **pending until** `deploy/tjm/vmp` clears; maintainer log review on [console.deno.com](https://console.deno.com) is required when that check stays red. | | ||
|
|
||
| Required API env vars (Worker): | ||
| Manual commands: | ||
|
|
||
| - `MEDIA_CONVERT_ENABLED=1` | ||
| - `AWS_REGION` (for S3 + MediaConvert, e.g. `eu-central-1`) | ||
| - `AWS_ACCESS_KEY_ID` | ||
| - `AWS_SECRET_ACCESS_KEY` | ||
| - `AWS_SESSION_TOKEN` (optional, only when using temporary credentials) | ||
| - `MEDIA_CONVERT_ENDPOINT` (account-specific endpoint from MediaConvert console) | ||
| - `MEDIA_CONVERT_ROLE_ARN` (IAM role MediaConvert assumes) | ||
| - `MEDIA_CONVERT_INPUT_BUCKET` (S3 source uploads) | ||
| - `MEDIA_CONVERT_OUTPUT_BUCKET` (S3 transcode outputs) | ||
| - `MEDIA_CONVERT_INPUT_PREFIX` (optional, default `mediaconvert-input`) | ||
| - `MEDIA_CONVERT_OUTPUT_PREFIX` (optional, default `mediaconvert-output`) | ||
| - `MEDIA_CONVERT_MAX_UPLOAD_MB` (optional, default `4096`) | ||
| - `MEDIA_CONVERT_PRICE_HD_PER_MIN` (optional, default `0.015`; rough estimator) | ||
| ```bash | ||
| npm run deploy:api | ||
| npm run deploy --workspace=@vmp/web # staging Worker | ||
| npm run deploy:prod --workspace=@vmp/web # production Worker | ||
| ``` | ||
|
|
||
| AWS setup checklist: | ||
| Deploy gates typecheck `@vmp/shared`, `@vmp/storage`, `@vmp/api`, and `@vmp/web` before build. Details: [DEPLOYMENT.md](DEPLOYMENT.md). | ||
|
|
||
| 1. Create/choose two S3 prefixes/buckets for input and output. | ||
| 2. Create a MediaConvert IAM service role (`MEDIA_CONVERT_ROLE_ARN`) with read access to input and write access to output. | ||
| 3. Create an IAM principal for the Worker credentials with: | ||
| - `mediaconvert:CreateJob`, `mediaconvert:GetJob` | ||
| - `iam:PassRole` scoped to `MEDIA_CONVERT_ROLE_ARN` | ||
| - `s3:PutObject` on input prefix | ||
| - `s3:GetObject` on output object prefix | ||
| - `s3:ListBucket` on output bucket (bucket-level action), scoped by `s3:prefix` condition to the output prefix | ||
| 4. In MediaConvert console, copy the account endpoint into `MEDIA_CONVERT_ENDPOINT`. | ||
| 5. Run migration `packages/api/migrations/0017_media_convert_jobs.sql`. | ||
| **Never push feature work directly to `main`** — use a branch + pull request (autodeploy + CodeRabbit). See [AGENTS.md → Git workflow](AGENTS.md#git-workflow-mandatory--read-first). | ||
|
|
||
| Notes: | ||
| ## Secrets and configuration | ||
|
|
||
| - Current cloud profile is H.264/HLS with 720p only, fps capped at 30. | ||
| - Architecture is rendition-based and supports future expansion (480p/1080p/4K, alternate codecs). | ||
| - Usage/cost values are approximate normalized-minute estimates, not billing-grade accounting. | ||
| - Template: root [`.env.example`](.env.example). Copy to `.env.staging` / `.env.production` as needed. | ||
| - Local API secrets: `packages/api/.dev.vars` (never commit). | ||
| - Required Worker secrets (JWT, Stripe, Brevo, VAPID, RSS, TOTP, …): listed in [AGENTS.md](AGENTS.md) and [DEPLOYMENT.md](DEPLOYMENT.md). | ||
| - Prices, plan names, and limits live in D1 `admin_settings` — not hardcoded. | ||
|
|
||
| ## API docs (`docs/`) | ||
| ## Media encoding | ||
|
|
||
| - [Pills external update API](docs/pills-external-update-api.md) | ||
| Transcoding runs on a **media VM** via [`@vmp/media-pipeline`](packages/media-pipeline/README.md): | ||
|
|
||
| ## Notes | ||
| 1. Watchfolder intake → SVT Encore encode | ||
| 2. encore-packager (Shaka) → fMP4 HLS ladder uploaded to R2 | ||
| 3. HMAC callback to `POST /api/admin/videos/:id/pipeline-status` | ||
|
|
||
| - API entrypoint is TypeScript (`packages/api/src/index.ts`) referenced by `packages/api/wrangler.json`. | ||
| - Worker/service scripts that must remain JavaScript at runtime (for browser/service-worker execution) are generated from TypeScript sources during build. | ||
| - Media encoding on a VM is handled by [`@vmp/media-pipeline`](packages/media-pipeline/README.md) (SVT Encore + orchestrator); optional Deno backup API by [`@vmp/api-node`](packages/api-node/README.md). | ||
| There is **no** AWS Elemental MediaConvert admin upload/transcode UI in this repo anymore. The historical `media_convert_jobs` D1 table remains: playback and offline-download code may still read completed **Bunny Stream** rows (`provider = 'bunnystream'`, `bunny_playback_url`) as an alternate HLS entrypoint. Do not drop that table without a migration that replaces those reads. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # Documentation index | ||
|
|
||
| Start with the [repository README](../README.md) and [AGENTS.md](../AGENTS.md). This folder holds API notes and historical design write-ups. | ||
|
|
||
| ## Current | ||
|
|
||
| | Document | Description | | ||
| | --- | --- | | ||
| | [pills-external-update-api.md](pills-external-update-api.md) | `POST /api/pills/update` external API contract | | ||
| | [console-errors.md](console-errors.md) | Common browser console messages and how to interpret them | | ||
| | [i18n-prep.md](i18n-prep.md) | Per-instance UI locale (`NUXT_PUBLIC_UI_LOCALE`) and translation workflow | | ||
|
|
||
| Related (outside `docs/`): | ||
|
|
||
| | Document | Description | | ||
| | --- | --- | | ||
| | [../DEPLOYMENT.md](../DEPLOYMENT.md) | CI/CD, env vars, smoke checks | | ||
| | [../packages/web/docs/workers-deploy-env.md](../packages/web/docs/workers-deploy-env.md) | Web Worker build-time environment | | ||
| | [../packages/web/docs/workers-pages-compatibility.md](../packages/web/docs/workers-pages-compatibility.md) | Workers vs deprecated Pages routing notes | | ||
|
|
||
| ## Archive (historical / planning) | ||
|
|
||
| These are kept for context. They are **not** the source of truth for current behavior. | ||
|
|
||
| | Document | Notes | | ||
| | --- | --- | | ||
| | [archive/admin-homescreen-layout-redesign.md](archive/admin-homescreen-layout-redesign.md) | Homescreen editor redesign plan | | ||
| | [archive/offline-downloads-roadmap.md](archive/offline-downloads-roadmap.md) | Offline downloads milestones (M1–M6 shipped) | | ||
| | [archive/stripe-express-checkout-investigation.md](archive/stripe-express-checkout-investigation.md) | Stripe Express Checkout investigation notes | |
4 changes: 3 additions & 1 deletion
4
docs/admin-homescreen-layout-redesign.md → ...chive/admin-homescreen-layout-redesign.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
docs/offline-downloads-roadmap.md → docs/archive/offline-downloads-roadmap.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Warning
The new Documentation table links
[docs/README.md](docs/README.md)as "Index of API notes and historical design docs", but that file does not exist in the repository and is not being created in this PR. Any reader following the link will hit a 404.