Skip to content

docs(web): expand storage module page #139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kkopanidis merged 1 commit into from
Jun 22, 2026
Merged

docs(web): expand storage module page #139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 39 additions & 32 deletions apps/web/content/docs/modules/storage.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
---
title: Storage
description: File uploads, storageFileId linking, and preview proxy pattern.
description: File uploads, cloud providers (S3, Azure, GCS), signed URLs, public access, folder markers, and Prometheus metrics.
agent_summary: "Presigned upload server-side; store storageFileId on documents; serve binaries via server preview proxy — never expose presigned URLs to browsers."
---

The **storage** module handles blobs across local, S3, Azure, GCS, and Aliyun providers. All binary assets flow through Conduit Storage — never base64 in database documents.
import StorageDeepDive from "@/mdx/deep-dives/storage-deep-dive.mdx";

The **storage** module handles blobs across **local**, **AWS S3** (and S3-compatible endpoints), **Azure Blob Storage**, **Google Cloud Storage**, and **Aliyun OSS**. All binary assets flow through Conduit Storage — never base64 in database documents.

## Use cases

Expand All @@ -22,6 +24,10 @@ The **storage** module handles blobs across local, S3, Azure, GCS, and Aliyun pr
title: "Web app file display",
outcome: "Preview proxy route serves bytes — browsers never see presigned URLs",
},
{
title: "Public assets",
outcome: "isPublic: true with CDN host mapping for cacheable delivery",
},
{
title: "Chat file messages",
outcome: "Upload to storage, reference file IDs in chat message payload",
Expand All @@ -33,13 +39,18 @@ The **storage** module handles blobs across local, S3, Azure, GCS, and Aliyun pr

<ModuleCapabilities
items={[
"Presigned upload/download",
"S3, Azure, GCS, local",
"Presigned upload/download (60 min expiry)",
"AWS S3 & S3-compatible providers",
"Azure Blob Storage",
"Google Cloud Storage",
"Local filesystem",
"storageFileId linking",
"Preview proxy pattern",
"Public & private files",
"Public & private files and containers",
"ReBAC scope on reads",
"Container/folder paths",
"Container/folder hierarchy with marker files",
"CDN host mapping per container",
"Prometheus metrics (files, folders, size)",
]}
/>

Expand All @@ -65,52 +76,48 @@ The **storage** module handles blobs across local, S3, Azure, GCS, and Aliyun pr
## How it works

<ModuleDeepDive>
### Presigned two-step upload

1. **Metadata on Conduit** — `POST /storage/upload` returns `{ file: { _id }, url }`
2. **Bytes to object store** — `PUT {url}` with `Content-Type` and raw bytes (no Authorization header)
3. **Link domain entity** — store `file._id` as `storageFileId` / `avatarFileId` on your document

Never store presigned URLs in the database — they expire.

### Preview proxy (required for browsers)

Presigned SAS/S3 URLs must not reach browsers or mobile clients. Your app server calls `GET /storage/getFileUrl/:id` with the user's token, fetches the presigned URL **server-side**, and streams bytes to the browser — the presigned URL never reaches the client. See the [Next.js guide](/docs/guides/first-app-nextjs).

`GET /storage/file/data/:id` returns base64 and is for **small server-side transforms only** — not general image or video delivery to browsers.

### Public vs private

`isPublic: true` skips auth on `GET /storage/getFileUrl/:id`. Private files enforce authorization when the authorization module is enabled; pass `scope` for ReBAC checks.

### Cleanup

On entity delete, call `DELETE /storage/file/{storageFileId}` to remove the blob.
<StorageDeepDive />
</ModuleDeepDive>

## Configure

Provider credentials and `defaultContainer` via MCP `?modules=storage`:
Provider credentials and module settings via MCP `?modules=storage`:

| Tool | Purpose |
|------|---------|
| `get_config_storage` | Read provider settings |
| `patch_config_storage` | S3/Azure/GCS/local config |
| `get_config_storage` | Read provider, container, and CDN settings |
| `patch_config_storage` | Set provider (`aws` / `azure` / `google` / `aliyun` / `local`), credentials, `defaultContainer`, `allowContainerCreation`, `suffixOnNameConflict`, `cdnConfiguration` |

Key config fields:

| Field | Default | Description |
|-------|---------|-------------|
| `provider` | `local` | Active storage backend |
| `defaultContainer` | `conduit` | Container used when none is specified on upload |
| `allowContainerCreation` | `true` | Allow Client API to create containers on first use |
| `suffixOnNameConflict` | `false` | Append ` (n)` to filename when a duplicate exists |
| `authorization.enabled` | `false` | Enable ReBAC checks and `scope` parameter |

Requires the [database](/docs/modules/database) module (file metadata is stored as documents).

## Client API

| Method | Path | Auth |
|--------|------|------|
| POST | `/storage/upload` | Required |
| PATCH | `/storage/upload/:id` | Required |
| GET | `/storage/getFileUrl/:id` | Optional (returns presigned URL in `{ result }` — use server-side only) |
| POST | `/storage/file` | Required (base64 inline upload) |
| PATCH | `/storage/file/:id` | Required (base64 inline update) |
| GET | `/storage/getFileUrl/:id` | Optional (`redirect`, `download` query params) |
| GET | `/storage/file/:id` | Optional (metadata document) |
| GET | `/storage/file/data/:id` | Required (base64 — small server-side transforms only) |
| DELETE | `/storage/file/:id` | Required |

When authorization is enabled, add `scope` as a query parameter on mutating routes and reads of private files.

## MCP

Enable `?modules=storage` for provider and container configuration.
Enable `?modules=storage` for provider and container configuration. Admin API routes under `/storage/` expose container, folder, and file management for operators.

<NextSteps steps={[
{ title: "Next.js integration", href: "/docs/guides/first-app-nextjs" },
Expand Down
123 changes: 123 additions & 0 deletions apps/web/src/mdx/deep-dives/storage-deep-dive.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
### Presigned two-step upload

1. **Metadata on Conduit** — `POST /storage/upload` returns `{ file: { _id }, url }`
2. **Bytes to object store** — `PUT {url}` with `Content-Type` and raw bytes (no Authorization header)
3. **Link domain entity** — store `file._id` as `storageFileId` / `avatarFileId` on your document

Never store presigned URLs in the database — they expire after **60 minutes**.

For inline base64 uploads (small files, server-side only), use `POST /storage/file` with a base64 `data` field instead.

### Signed URLs

Conduit generates time-limited signed URLs for upload and download:

| Operation | How | Expiry |
|-----------|-----|--------|
| Upload | `POST /storage/upload` → presigned PUT/SAS/write URL | 60 minutes |
| Replace bytes | `PATCH /storage/upload/:id` → new presigned upload URL | 60 minutes |
| Private download | `GET /storage/getFileUrl/:id` → signed read URL in `{ result }` | 60 minutes |
| Public download | `GET /storage/getFileUrl/:id` → stored `url` (no signing) | N/A |

Query parameters on `GET /storage/getFileUrl/:id`:

| Param | Effect |
|-------|--------|
| `redirect=true` | HTTP redirect to the URL instead of returning JSON |
| `download=true` | Sets `Content-Disposition: attachment` on the signed URL |

Signed URLs (S3 presigned URLs, Azure SAS tokens, GCS signed URLs) **must not reach browsers or mobile clients**. Your app server calls `GET /storage/getFileUrl/:id` with the user's token, fetches the presigned URL **server-side**, and streams bytes to the browser. See the [Next.js guide](/docs/guides/first-app-nextjs).

`GET /storage/file/data/:id` returns base64 and is for **small server-side transforms only** — not general image or video delivery to browsers.

### Public vs private

Access is controlled at two levels:

**File level** — `isPublic` on upload/create:

- `isPublic: true` — `GET /storage/getFileUrl/:id` returns the stored `url` without auth or signing
- `isPublic: false` — requires authentication (and ReBAC when authorization is enabled); download uses a signed URL

**Container level** — `isPublic` on the container document:

- Public containers apply provider-level read policies (S3 bucket policy, Azure `blob` access, GCS `makePublic`)
- **Files in public containers cannot be marked private** — Conduit rejects `isPublic: false` when the container is public

When authorization is enabled, pass `scope` on create/upload to bind ownership: `File:{id}` is linked to the scope resource (or the uploading user) via ReBAC.

### Folder markers

Object stores have no real directories. Conduit models folders in the database (`_StorageFolder`) and creates **marker objects** in the provider:

| Provider | Marker object |
|----------|---------------|
| S3 / Azure | `{folderPath}.keep.txt` (body: `DO NOT DELETE`) |
| GCS | `{folderPath}/keep.txt` |

Folder paths are normalized to end with `/` (e.g. `users/abc/avatars/`). On upload, `findOrCreateFolders` walks the path hierarchy and creates missing folders and markers. Nested paths like `a/b/c/` create markers for `a/`, `a/b/`, and `a/b/c/`.

Specify `folder` on upload; omit or pass `/` for the container root. Containers are created automatically when `allowContainerCreation` is true (default).

### Cloud providers

Set `provider` via MCP `patch_config_storage`. Each provider has its own credential block:

#### AWS S3 (`provider: "aws"`)

| Config key | Purpose |
|------------|---------|
| `aws.region` | AWS region |
| `aws.accessKeyId` / `aws.secretAccessKey` | IAM credentials |
| `aws.accountId` | Used to prefix bucket names as `conduit-{accountId}-{container}` |
| `aws.endpoint` | Custom endpoint for S3-compatible providers (MinIO, DigitalOcean Spaces, etc.) |
| `aws.usePathStyle` | Path-style addressing for non-AWS endpoints (default: `true`) |

When `endpoint` is set (non-AWS S3), `accountId` is auto-generated. Public containers disable the public access block and attach a bucket policy allowing `s3:GetObject` for all principals.

#### Azure Blob Storage (`provider: "azure"`)

| Config key | Purpose |
|------------|---------|
| `azure.connectionString` | Storage account connection string |

Containers map to Azure blob containers. Public containers are created with `blob`-level public read access. Upload and download use SAS tokens.

#### Google Cloud Storage (`provider: "google"`)

| Config key | Purpose |
|------------|---------|
| `google.serviceAccountKeyPath` | Path to service account JSON key file |

Buckets map to GCS buckets. Public containers call `makePublic()` on the bucket; public files call `makePublic()` on the object. Folder markers use `{path}/keep.txt`. Upload and download use GCS signed URLs.

#### CDN mapping

Map container names to CDN hosts via `cdnConfiguration`:

```json
{
"cdnConfiguration": {
"myapp-uploads": "cdn.example.com"
}
}
```

Public files store both `sourceUrl` (provider URL) and `url` (CDN-applied). Private downloads apply CDN host replacement on signed URLs when configured.

### Metrics

The storage module exports Prometheus gauges (scraped via Conduit Core):

| Metric | Type | Description |
|--------|------|-------------|
| `containers_total` | Gauge | Number of containers |
| `folders_total` | Gauge | Number of folders (marker objects created) |
| `files_total` | Gauge | Number of file documents |
| `storage_size_bytes_total` | Gauge | Cumulative size of all files in bytes |

Gauges are updated on create and delete. File size changes adjust `storage_size_bytes_total` by the delta.

### Cleanup

On entity delete, call `DELETE /storage/file/{storageFileId}` to remove the blob and decrement metrics.
Loading