feat(storage): add URL upload/download support for local storage provider#1557
feat(storage): add URL upload/download support for local storage provider#1557ChrisPdgn wants to merge 7 commits into
Conversation
Adds an embedded HTTP file server so the local storage provider can serve presigned-style upload/download URLs, matching the behavior of cloud providers (AWS/Azure/GCS/Aliyun) for local development. - getUploadUrl(), getSignedUrl(), and getPublicUrl() now return working URLs instead of throwing "Method not implemented" - Public containers/files are now supported on local storage - Added CORS, path-traversal protection, and Content-Disposition handling to the embedded file server - Added dispose() to IStorageProvider so the server is cleanly torn down and recreated on config changes without port conflicts - Added local.httpPort and local.httpBaseUrl config options Co-authored-by: Cursor <[email protected]>
…rite - store(): create parent directories with mkdirSync before writeFile - createFolder(): replace async mkdir race with synchronous recursive mkdir - dispose(): call closeAllConnections() to avoid hanging on teardown - Remove unused mkdir import Co-authored-by: Cursor <[email protected]>
Auto-derive httpBaseUrl from httpPort when unset. Handle PUT mkdir failures with a 500 response. Reject URL generation after the local HTTP server is disposed. Co-authored-by: Cursor <[email protected]>
kkopanidis
left a comment
There was a problem hiding this comment.
We need to support some sort of authentication mechanism in the non-public URLs of the provider. For example the "upload" URL part needs something too. We can somehow probably "re-use" authentication's JWT
Protect non-public local storage URLs with HMAC-SHA256 signatures and expiry, mirroring cloud presigned URL behavior instead of reusing auth JWTs. - getUploadUrl() and getSignedUrl() now return signed URLs with sig/expires - PUT requests without a valid signature are rejected with 403 - Public URLs from getPublicUrl() remain unsigned for open GET access Co-authored-by: Cursor <[email protected]>
|
Good point on needing auth on the non-public local storage URLs (especially upload). We looked at reusing the authentication module's JWT, but went with HMAC-SHA256 signed URLs instead — the same pattern S3/Azure/GCS use for presigned URLs. What we implemented:
Why HMAC instead of auth JWT:
The signing secret is ephemeral per provider instance (generated on startup). If the server restarts, outstanding signed URLs expire — same as when cloud signing keys rotate. Upload and download URLs are always generated on-demand through Conduit's authenticated endpoints, so this is fine in practice. |
Close the unsigned GET security gap and return long-lived signed public URLs for cloud parity. Add optional signingSecret config so URLs survive restarts. Co-authored-by: Cursor <[email protected]>
…ning Check server.listening before returning upload, signed, or public URLs so bind failures like EADDRINUSE fail fast instead of returning unreachable URLs. Co-authored-by: Cursor <[email protected]>
Validate signingSecret in preConfig, document URL persistence requirements, normalize httpBaseUrl trailing slashes, trim secret on decode, and handle read-stream errors on GET. Co-authored-by: Cursor <[email protected]>
Summary
getUploadUrl,getSignedUrl, andgetPublicUrlwork the same way as cloud providers (presigned-URL pattern on port 3100 by default).dispose()so the HTTP server is torn down and recreated on config changes (activate, deactivate, provider switch).store()now creates parent dirs before writing, andcreateFolder()uses synchronous recursivemkdirSyncinstead of racy asyncmkdircallbacks.Test plan
POST /storage/uploadwith a new folder path succeeds without manually creating directoriesPUTto the returned upload URL writes file bytes to diskGETon the local storage URL serves the file with correct MIME type (text + image)GET /storage/getFileUrl/:idreturns the local storage URL for private filesdispose()and releases port 3100