Skip to content

[miniflare] Share local storage across processes via a single owner (experimental) - #14449

Draft
penalosa wants to merge 39 commits into
mainfrom
penalosa/global-resources
Draft

[miniflare] Share local storage across processes via a single owner (experimental)#14449
penalosa wants to merge 39 commits into
mainfrom
penalosa/global-resources

Conversation

@penalosa

@penalosa penalosa commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this does

Adds an experimental option to Miniflare that makes a single process own local storage for a given persist root, so multiple Miniflare instances (e.g. several wrangler dev / vite dev sessions) no longer each open the same SQLite/blob files — the root cause of cross-process SQLITE_BUSY errors under concurrent access.

When enabled:

  • The first instance to find no published owner elects itself (via a per-persist-root spawn lock) and launches a detached "owner" process — a headless Miniflare that opens the files and hosts the storage services.
  • Every instance then routes its storage operations to the owner over the remote-bindings boundary and skips standing up its own local storage. Only the owner performs storage I/O.
  • The owner publishes its address to the persist root, heartbeats it, and self-terminates once no instances remain (startup grace + idle debounce). Crashed owners are reclaimed on the next election.

Transport

Routing reuses the existing remote-bindings boundary rather than a bespoke channel:

  • fetch for KV, R2, D1 and Images.
  • capnweb JSRPC for Streams and Secrets Store (these have no fetch entrypoint).

Bindings are repointed by id (type:id) against the owner server, so the owner avoids binding-name collisions across instances and resolves dynamic ids via an MF-Storage-Owner-Namespace request header. This replaces an earlier workerd-debug-port prototype and drops the previous unsafeDevRegistryPath requirement.

The owner-side server logic is shared with Wrangler's remote-bindings proxy server — both are thin adapters over one parameterised remote-bindings-proxy-server module (JSRPC-vs-fetch dispatch and the MF-Binding / MF-URL / MF-Header-* wire protocol), each supplying its own binding-resolution strategy.

What is and isn't shared

  • Routed to the owner: KV, R2, D1, Images, Streams, Secrets Store.
  • Kept per-instance (via isolateLocalStorage): Cache, Durable Objects, Workflows — when the feature is on and no explicit *Persist path is set, these use a per-instance temp path.
  • An explicit *Persist path for a given type always takes precedence and excludes that type from routing.

This also lands a supporting refactor: storage and remote (mixed-mode) bindings now carry their per-resource config (namespace/bucket/database id, remote connection string) via ctx.props against a single shared object-entry / proxy service, instead of one workerd service per resource.

Validation

A high-concurrency oracle runs multiple instances against one persist root and asserts cross-instance sharing with exactly-correct results under contention:

  • KV / R2 / D1 round-trips through the owner, including a D1 SQL round-trip and concurrent-write stress (60 concurrent D1 writes across 3 instances, exact final count).
  • Streams (upload on A, listed on B), Secrets Store (created via A's admin API, read by a worker in B), and Images (binding usable in every instance).
  • The full oracle (70 tests) is green under both direct Vitest and Turbo, with MINIFLARE_TEST_SHARED_OWNER added to turbo.json globalPassThroughEnv so it gates CI. The same tests also pass in baseline (feature-off) mode, where sharing is provided by the persist root.

Status — draft

Known follow-ups before this is review-ready:

  • Liveness should be pid-authoritative; a busy-but-alive owner whose heartbeat goes stale could currently be treated as absent and double-spawned. Intend to convert the spawn lock into a lifetime-held exclusive lease.
  • Register client presence before resolving routing (close a narrow teardown race).
  • Exclude the Workers Sites KV binding from rewriting.
  • Wrangler opt-in wiring; Windows validation.
  • Tests
    • Tests included/updated
    • Automated tests not possible - manual testing has been completed as follows:
    • Additional testing not necessary because:
  • Public documentation
    • Cloudflare docs PR(s):
    • Documentation not necessary because: the option is an internal/experimental unsafe* flag not intended for general use yet; user-facing docs will follow once the design stabilises and it is opted into by Wrangler.

@changeset-bot

changeset-bot Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e8fe614

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
miniflare Minor
@cloudflare/deploy-helpers Patch
@cloudflare/pages-shared Patch
@cloudflare/remote-bindings Patch
@cloudflare/runtime-types Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-pool-workers Patch
wrangler Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-project-automation github-project-automation Bot moved this to Untriaged in workers-sdk Jun 26, 2026
@ask-bonk

ask-bonk Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

UnknownError: ProviderInitError

github run

@ask-bonk

ask-bonk Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

@penalosa Bonk workflow failed. Check the logs for details.

View workflow run · To retry, trigger Bonk again.

@github-actions

github-actions Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ Issues found

.changeset/shared-storage-owner.md

  • This changeset describes an experimental feature, but it does not include a note on how users can opt in. Please add a brief note indicating how to configure or enable the unsafeSharedStorageOwner option (for example, where in the configuration it should be set).

@pkg-pr-new

pkg-pr-new Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@14449

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@14449

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@14449

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@14449

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@14449

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@14449

miniflare

npm i https://pkg.pr.new/miniflare@14449

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@14449

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@14449

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@14449

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@14449

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@14449

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@14449

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@14449

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@14449

wrangler

npm i https://pkg.pr.new/wrangler@14449

commit: e8fe614

@penalosa
penalosa force-pushed the penalosa/global-resources branch 2 times, most recently from cb05917 to 0e7a7ed Compare June 30, 2026 13:52
@penalosa
penalosa force-pushed the penalosa/global-resources branch from ae287f4 to 9a1418f Compare July 17, 2026 12:13
@penalosa
penalosa changed the base branch from main to penalosa/manual-remote-bindings July 17, 2026 12:14
penalosa added 4 commits July 17, 2026 13:18
@penalosa
[miniflare] Route storage/remote bindings through shared services via…
25c6866 
… props

Move the per-binding configuration for storage and remote (mixed-mode)
bindings out of per-resource workerd services and into runtime `ctx.props`,
so a single shared service can serve any number of bindings.

- Local KV namespaces now share one entry service; the namespace id is
  passed via props and resolved in object-entry.worker.ts (idFromName).
- remoteProxyClientWorker() is now script-only; the connection string,
  binding name and trace id travel via props (buildRemoteProxyProps),
  read in remote-proxy-client.worker.ts and the dispatch-namespace proxy.
- All remote-binding plugins emit one shared remote-proxy service instead
  of one per resource.
- explorer.ts reads the KV namespace id from binding props rather than
  parsing it out of the service name.
@penalosa
[miniflare] Add shared storage owner routed over the remote-bindings …
a4d0fef 
…boundary

Adds an experimental option that makes a single detached process own local
storage for a given persist root, so multiple Miniflare instances (e.g. several
wrangler dev / vite dev sessions) no longer each open the same SQLite/blob
files — the root cause of cross-process SQLITE_BUSY errors.

The first instance to find no published owner elects itself (via a per-persist
-root spawn lock) and launches a detached owner process. Every instance then
routes its storage operations to the owner over the remote-bindings boundary —
fetch for KV/R2/D1/Images and capnweb JSRPC for Streams/Secrets Store — and
skips standing up its own local storage. Bindings are repointed by id (type:id)
so the owner avoids binding-name collisions and resolves dynamic ids via the
MF-Storage-Owner-Namespace header. The owner publishes its address, heartbeats
it, and self-terminates once no instances remain.

Cache, Durable Objects and Workflows stay per-instance via isolateLocalStorage.
@penalosa
[miniflare] Add storage-owner tests and high-concurrency oracle
9705724 
Unit tests for owner election, presence, routing and teardown, plus a high
-concurrency oracle that runs multiple instances against one persist root and
asserts KV/R2/D1/Streams/Secrets/Images sharing with exactly-correct results
under contention. The oracle runs under MINIFLARE_TEST_SHARED_OWNER (added to
turbo.json globalPassThroughEnv) so it gates both feature-on and baseline modes.
@penalosa
[miniflare] Deduplicate the remote-bindings proxy server
9e35a0f 
Extract the shared server logic for the remote-bindings boundary (JSRPC vs
fetch dispatch, MF-Binding / MF-URL / MF-Header-* handling) into a single
parameterised module, `remote-bindings-proxy-server.ts`, and have both consumers
adapt to it:

- Miniflare's storage-owner server passes a "<type>:<id>" resolver that forwards
  the id via the MF-Storage-Owner-Namespace header.
- Wrangler's ProxyServerWorker passes the existing env[binding] resolver with its
  SendEmail / Dispatch Namespace special-cases.

The two previously near-identical implementations now share one source of truth;
behaviour is unchanged (wrangler remote-bindings and miniflare storage-owner
suites both pass).
@penalosa
penalosa force-pushed the penalosa/global-resources branch from 9a1418f to 9e35a0f Compare July 17, 2026 12:32
penalosa added 2 commits July 17, 2026 16:19
@penalosa
[miniflare] Move storage-owner per-plugin logic into the plugins
d46ce09 
Push the storage/remote-owner routing knowledge out of `Miniflare` and
into each storage plugin, so `index.ts` no longer type-switches on plugin
key. Two optional `PluginBase` hooks own the per-plugin shapes:

- `routeBindingToStorageOwner` — rewrite a local storage binding to route
  through the storage-owner proxy (KV/R2/D1/Stream/Secrets); Images stays
  self-handled in its `getServices`.
- `getStorageOwnerHosting` — describe how a spawned owner hosts the
  plugin's local storage (owner-process options + owner-server bindings).

`extractObjectEntryId` and `storageOwnerProxyDesignator` move to
`plugins/shared` (the client-proxy service name goes with them), and
`Miniflare` now iterates plugins generically for the client rewrite, the
owner bindings, and the owner spawn config. Net `index.ts` change is
-263 lines; no behaviour change.
@penalosa
[miniflare] Drop redundant startup lock, log the storage owner, param…
340065c 
…eterise persist-sharing

- Remove `withPersistRootStartupLock` (and its `wait` helper/constants) — with
  the shared storage owner it guarded nothing: routed storage goes to the owner
  and everything else is per-instance, so no two instances race the same files.
  `#startRuntime` now calls `runtime.updateConfig` directly.
- Give the detached owner a voice — spawn it with stdio redirected to
  `.miniflare-owner.log` in the persist root (was `stdio: "ignore"`), and have
  `runStorageOwnerProcess` log start/ready/shutdown and surface uncaught
  errors, so a broken owner is diagnosable instead of a silent client fallback.
- Parameterise `persist-sharing.spec.ts` — add `startPair`/`startMany` helpers,
  collapse the KV/R2 "same key" and the 4/6/10-instance "distinct keys" tests
  into `describe.each`, and drop the duplicate `makeMany` (-116 lines).
@penalosa
penalosa force-pushed the penalosa/manual-remote-bindings branch 2 times, most recently from 79808aa to 2b3ea8b Compare July 21, 2026 14:21
Base automatically changed from penalosa/manual-remote-bindings to main July 21, 2026 14:46
penalosa added 2 commits August 3, 2026 18:25
@penalosa
Merge remote-tracking branch 'origin/main' into opencode/pr-14449-mai…
00739a9 
…n-merge

# Conflicts:
#	fixtures/get-platform-proxy-remote-bindings/tests/index.test.ts
#	packages/miniflare/src/index.ts
#	packages/miniflare/src/plugins/cache/index.ts
#	packages/miniflare/src/plugins/core/explorer.ts
#	packages/miniflare/src/plugins/core/index.ts
#	packages/miniflare/src/plugins/d1/index.ts
#	packages/miniflare/src/plugins/do/index.ts
#	packages/miniflare/src/plugins/images/index.ts
#	packages/miniflare/src/plugins/kv/index.ts
#	packages/miniflare/src/plugins/r2/index.ts
#	packages/miniflare/src/plugins/secret-store/index.ts
#	packages/miniflare/src/plugins/shared/constants.ts
#	packages/miniflare/src/plugins/stream/index.ts
#	packages/miniflare/src/plugins/vpc-networks/index.ts
#	packages/miniflare/src/plugins/vpc-services/index.ts
#	packages/miniflare/src/plugins/workflows/index.ts
#	packages/miniflare/src/workers/shared/object-entry.worker.ts
#	packages/miniflare/src/workers/shared/remote-proxy-client.worker.ts
#	packages/remote-bindings/package.json
#	packages/remote-bindings/src/start-remote-proxy-session.ts
#	packages/remote-bindings/src/startDevWorker/ProxyController.ts
#	packages/remote-bindings/src/startDevWorker/RemoteRuntimeController.ts
#	packages/remote-bindings/src/utils/miniflare.ts
#	packages/remote-bindings/src/utils/remote.ts
#	packages/remote-bindings/templates/remoteBindings/ProxyServerWorker.ts
#	packages/vitest-pool-workers/src/pool/config.ts
#	packages/workers-auth/src/access.ts
#	packages/wrangler/templates/startDevWorker/ProxyWorker.ts
#	pnpm-lock.yaml
@penalosa
Merge remote-tracking branch 'origin/main' into opencode/pr-14449-mai…
e8fe614 
…n-merge-2

# Conflicts:
#	packages/miniflare/src/plugins/d1/index.ts
#	packages/miniflare/src/plugins/r2/index.ts
#	packages/miniflare/src/plugins/ratelimit/index.ts
#	packages/miniflare/src/plugins/secret-store/index.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@workers-devprod workers-devprod Awaiting requested review from workers-devprod workers-devprod will be requested when the pull request is marked ready for review workers-devprod is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

workers-sdk
Status: Untriaged

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@penalosa @workers-devprod