Skip to content

feat(storage): add URL upload/download support for local storage provider#1557

Open
ChrisPdgn wants to merge 7 commits into
mainfrom
feat/local-storage-url-support
Open

feat(storage): add URL upload/download support for local storage provider#1557
ChrisPdgn wants to merge 7 commits into
mainfrom
feat/local-storage-url-support

Conversation

@ChrisPdgn

Copy link
Copy Markdown
Contributor

Summary

  • Add an embedded HTTP file server to the local storage provider so getUploadUrl, getSignedUrl, and getPublicUrl work the same way as cloud providers (presigned-URL pattern on port 3100 by default).
  • Implement provider lifecycle cleanup via dispose() so the HTTP server is torn down and recreated on config changes (activate, deactivate, provider switch).
  • Fix local provider directory creation: store() now creates parent dirs before writing, and createFolder() uses synchronous recursive mkdirSync instead of racy async mkdir callbacks.

Test plan

  • POST /storage/upload with a new folder path succeeds without manually creating directories
  • PUT to the returned upload URL writes file bytes to disk
  • GET on the local storage URL serves the file with correct MIME type (text + image)
  • GET /storage/getFileUrl/:id returns the local storage URL for private files
  • Verify HTTP server restarts cleanly when storage config changes (port/path/provider switch)
  • Verify module deactivation calls dispose() and releases port 3100

ChrisPdgn and others added 2 commits July 20, 2026 15:04
Adds an embedded HTTP file server so the local storage provider can
serve presigned-style upload/download URLs, matching the behavior of
cloud providers (AWS/Azure/GCS/Aliyun) for local development.

- getUploadUrl(), getSignedUrl(), and getPublicUrl() now return working
  URLs instead of throwing "Method not implemented"
- Public containers/files are now supported on local storage
- Added CORS, path-traversal protection, and Content-Disposition
  handling to the embedded file server
- Added dispose() to IStorageProvider so the server is cleanly torn
  down and recreated on config changes without port conflicts
- Added local.httpPort and local.httpBaseUrl config options

Co-authored-by: Cursor <[email protected]>
…rite

- store(): create parent directories with mkdirSync before writeFile
- createFolder(): replace async mkdir race with synchronous recursive mkdir
- dispose(): call closeAllConnections() to avoid hanging on teardown
- Remove unused mkdir import

Co-authored-by: Cursor <[email protected]>
Comment thread modules/storage/src/config/config.ts
Comment thread modules/storage/src/providers/local/index.ts Outdated
Comment thread modules/storage/src/Storage.ts
Auto-derive httpBaseUrl from httpPort when unset.
Handle PUT mkdir failures with a 500 response.
Reject URL generation after the local HTTP server is disposed.

Co-authored-by: Cursor <[email protected]>

@kkopanidis kkopanidis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to support some sort of authentication mechanism in the non-public URLs of the provider. For example the "upload" URL part needs something too. We can somehow probably "re-use" authentication's JWT

Protect non-public local storage URLs with HMAC-SHA256 signatures and
expiry, mirroring cloud presigned URL behavior instead of reusing auth JWTs.

- getUploadUrl() and getSignedUrl() now return signed URLs with sig/expires
- PUT requests without a valid signature are rejected with 403
- Public URLs from getPublicUrl() remain unsigned for open GET access

Co-authored-by: Cursor <[email protected]>
@ChrisPdgn

Copy link
Copy Markdown
Contributor Author

Good point on needing auth on the non-public local storage URLs (especially upload). We looked at reusing the authentication module's JWT, but went with HMAC-SHA256 signed URLs instead — the same pattern S3/Azure/GCS use for presigned URLs.

What we implemented:

  • getUploadUrl() and getSignedUrl() now return URLs with sig and expires query params
  • The embedded HTTP server (port 3100) validates the signature and expiry before serving
  • PUT without a valid signature → 403 (upload URLs are protected)
  • GET with a valid signed URL → serves the file (private download URLs are protected)
  • getPublicUrl() stays unsigned — public files are meant to be openly accessible, and the URL is stored permanently in the DB

Why HMAC instead of auth JWT:

  1. Matches production semantics — cloud providers don't use user JWTs for presigned URLs; they use HMAC signatures scoped to a specific path + HTTP method + expiry. Local dev should behave the same way.
  2. No cross-module dependency — the storage provider doesn't need access to authentication's jwtSecret or grpcSdk. It's self-contained.
  3. Operation-scoped — a GET signature can't be used for PUT and vice versa. JWTs don't carry that constraint.
  4. Time-limited by design — uses the same 1-hour SIGNED_URL_EXPIRY constant as AWS/Azure/Aliyun. JWTs have their own longer-lived expiry tied to user sessions.
  5. No JWT in URLs — putting bearer tokens in query params would leak them in server logs, browser history, and referrer headers. HMAC signatures are opaque and purpose-built for URL-based auth.

The signing secret is ephemeral per provider instance (generated on startup). If the server restarts, outstanding signed URLs expire — same as when cloud signing keys rotate. Upload and download URLs are always generated on-demand through Conduit's authenticated endpoints, so this is fine in practice.

@ChrisPdgn
ChrisPdgn marked this pull request as draft July 23, 2026 11:44
Comment thread modules/storage/src/providers/local/index.ts
Comment thread modules/storage/src/providers/local/index.ts
ChrisPdgn and others added 2 commits July 23, 2026 15:32
Close the unsigned GET security gap and return long-lived signed public URLs
for cloud parity. Add optional signingSecret config so URLs survive restarts.

Co-authored-by: Cursor <[email protected]>
…ning

Check server.listening before returning upload, signed, or public URLs so bind
failures like EADDRINUSE fail fast instead of returning unreachable URLs.

Co-authored-by: Cursor <[email protected]>
Comment thread modules/storage/src/providers/local/index.ts
Validate signingSecret in preConfig, document URL persistence requirements,
normalize httpBaseUrl trailing slashes, trim secret on decode, and handle
read-stream errors on GET.

Co-authored-by: Cursor <[email protected]>
@ChrisPdgn
ChrisPdgn marked this pull request as ready for review July 23, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants