Skip to content

Scope stores ​

A scope store is one platform-run object for one scope. The vertical names a binding and a driver. The platform mints the object and is the only thing that knows which object that binding reaches. The vertical never holds the store itself: its calls go through the platform.

This sits below the per-tenant stores. A tenant store is one database or bucket for the whole tenant. A scope store is one object for one scope, and two scopes keep their own.

What you can use today ​

Nothing here is live for a vertical yet. The graph driver is registered, but a store script runs only a release a person has approved, and none is approved yet. A hosted push that declares a graph store passes the push check and is then refused before anything is uploaded. Everything below is what the platform does once a release is approved and rolled out.

Today a vertical author can:

  • read the declaration and the call shape below, and write code against getScopeStore and StoreStub;
  • run a store locally, on the pure adapter, with a ScopeStoreRunner a driver supplies (see the end of this page).

A vertical author cannot yet:

  • complete a hosted push of a vertical that declares scopeStores;
  • reach a store from a hosted deploy.

Declaring it ​

runtimeNeeds.scopeStores is a list of { binding, driver }. binding is a screaming-snake name. driver has no default. The only driver named so far is graph.

ts
scopeStores: [{ binding: 'GRAPH', driver: 'graph' }]

A hosted vertical's upload carries no store binding. The vertical is given neither the store's namespace nor anything that opens it. A binding already used by another store is refused, and so is the name ATTACHMENTS. The vertical's own Durable Object bindings count as stores for that check. A name in the SUBSTRAT_ namespace is the platform's and is refused. The push and the control plane apply the same checks, so a manifest that did not come through substrat push is refused the same way.

A driver is registered before a push that names it can succeed. There is one store script per driver and vertical; its objects keep their identities when the script changes. The graph driver is registered, and a push that names it succeeds once a release of it is approved and rolled out. It is written outside this repo, so its registration rests on a decision record that accepts that the tenants of one vertical share its script, kept apart by the driver's own design and the review of each release.

Store releases and vertical deploys are separate ​

A vertical push does not upgrade its store driver. A new store script starts at the driver's current good release; an existing script keeps its release, even when the platform rebuilds its entry or rotates verification keys. Release bytes are identified by a digest, and a script is uploaded only at a release a person has approved and nobody has revoked since.

An explicit rollout first runs the reserved operations against a fresh canary object, then moves the verticals' scripts in bounded batches. The canary object takes the release's own fixture, driver_<package>/substrat_canary.json: a small store dump, { tables }, holding at most 300 rows. The canary stages it a row a page and must read it back unchanged, a page at a time, before substrat.selfcheck builds the runtime over it. A release without the fixture is not rolled out. The fixture is part of the approved bytes and is not uploaded into the store script. A paused rollout resumes when posted again, from the verticals it has not reached yet, however many verticals declare the driver. A rollback also needs an approved release; it cannot lower the stored schema version. A failure can leave the fleet diverged when returning a script would lower its schema or the return itself fails. The rollout records that state so an operator can finish recovery rather than assuming every script moved together.

Calling it ​

getScopeStore is the door. It does not check a permission. The operation that calls does that first.

ts
const store = await host.getScopeStore(principal, tenantId, scopeId, 'GRAPH');
const count = await store.invoke('increment', { by: 1 });

On the platform path the call leaves the vertical as a fetch of store.substrat.internal. The egress worker hands it to the router, together with who the caller is. The vertical cannot change that part: it comes from the dispatch, not from the request. The router refuses a substrat.* operation, a scope other than the one the request was routed to, and a binding the running version does not declare. Then it signs the call and addresses the store script. The signature is Ed25519, over the audience, the key, the store script, the object, the operation, the exact body and the time, and the store script holds only the public keys. A Cloudflare host takes this path unless it was given a runner or scopeStores: { transport: 'binding' }. A control plane does not change that. On this path the host does not hold the store objects. The control plane sizes, exports, erases and wipes them through the store script itself, alongside the host's part for the scope (an erasure reaches the stores before the key is destroyed), so the host's admin answers for no store. A local size, export or erase asked of the host directly, with no control plane in front, refuses (unavailable) rather than answer that there are none.

invoke takes an operation and its input, and nothing else. Idempotency belongs to a scope operation, and a store call has none. An operation name is a fixed label: no @, no whitespace and no :, at most 64 characters. Any other name is refused before the call leaves the vertical. So is a name that starts with substrat.. substrat.export-page, substrat.stage-page, substrat.commit-stage, substrat.discard-stage, substrat.size, substrat.wipe, substrat.erase and substrat.selfcheck are the platform's operations. The control plane calls the store script for those itself. The vertical is not on that path.

The call on the wire is { operation, input }. The answer is { result }, or { error: { code, status, body } }, where status is an HTTP error status from 400 to 599. An answer with any other status is not the envelope, and it arrives as substrat.bad_envelope. A store error arrives as StoreError. The platform code comes from the HTTP status. The store's own code travels in storeCode. Its body travels in storeBody only for a 4xx whose JSON is at most 1 KiB (STORE_BODY_MAX_BYTES). A client can read that body, so a 5xx body, a larger one, or one that is not JSON stays in the store. The vertical's own code gets the same filtered storeBody. On the platform path the router cuts the store's error to that rule before it answers, so a withheld body never reaches the vertical's isolate, and the code is capped at 64 characters. An { error } envelope is the store's answer whatever HTTP status carried it, and a { result } only on a 2xx. A store that answers anything else, such as the store script's own refusal of a signature, arrives as substrat.bad_envelope too, and its body is neither passed on nor logged. A storeCode starting with substrat. is the platform's, never a store's. A store that answers a substrat. code itself arrives as substrat.bad_envelope, so a store's answer is never mistaken for the platform's.

The platform's codes on the platform path (PLATFORM_STORE_CODE):

storeCodeStatusPlatform codeWhenWhat the vertical can do
substrat.scope_mismatch403forbiddenThe call names a scope other than the one the request was routed to.Nothing at run time: call the store of the scope you are in.
substrat.undeclared_binding404not_foundThe running version does not declare the binding.Nothing at run time: declare the store and push.
substrat.reserved_operation403forbiddenThe operation starts with substrat.. The vertical's own host refuses this before the call leaves, so it arrives only from a caller that skips it.Nothing at run time.
substrat.too_large413unavailableThe call body is over STORE_CALL_MAX_BYTES. Nothing was sent.Send less in one call. Not retryable: the same body is refused again, so branch on storeCode here, not on the platform code.
substrat.refused4xxfrom the statusAny other platform refusal.Treat it as a refusal.
substrat.unavailable503unavailableA store the platform cannot reach, or a platform not set up for stores.Retry later.
substrat.bad_envelope502unavailableThe store answered outside the envelope, or with a substrat. code.Treat it as a store fault.

A vertical may branch on the platform code, on whether storeCode starts with substrat., and on the codes in this table. The storeBody of a platform refusal is a sentence for a person, not a value to match. More codes may be added, so read a substrat. code you do not know as substrat.refused on a 4xx and as substrat.unavailable on a 5xx. What the vertical's own host refuses is not a StoreError and has no storeCode: a reserved operation on either path (forbidden), and on the in-process path an unbound binding (not_found).

A direct binding exists for local development and for tests, and only when the host is created with transport: 'binding'. An ambient SUBSTRAT_STORE_* namespace is not used otherwise. The binding is not an isolation boundary.

On the platform path the store script's main module is the platform's store entry, not the driver. It checks every call before the driver's class is constructed: the signature, the key, the audience against the operation, the time, the store script, and that the object being asked is the one the signed name addresses. It hands the driver the exact body it checked, and its own bindings only. A call body is at most 4 MiB (STORE_CALL_MAX_BYTES): the router and the control plane refuse a larger one before they sign it, and the entry refuses it too, with a plain 413, without reading a body whose Content-Length already says it is over. A page of a store's dump is at most 3 MiB (STORE_DUMP_PAGE_MAX_BYTES), so a page and the call around it stay under the limit. The entry's SUBSTRAT_STORE* bindings are not in the environment the driver is constructed with, not in an env it imports from cloudflare:workers, and not reachable through ctx.exports (the upload sets disallow_importable_env and disable_ctx_exports). The driver does keep its object's ctx, storage included. The entry is not a boundary against the driver itself: the driver's code runs in the same isolate, so a driver that set out to interfere with the checks could. What stands between a driver and every tenant it serves is the review of each release the platform pins.

The invocation line ​

On a pushed vertical, each store call is recorded on the invocation line of the request that made it. The vertical passes nothing for that: the platform entry in front of a pushed vertical tracks the current request. A vertical run without that entry, locally for example, records nothing unless it passes onStoreCall (below). An entry has four fields: the binding, the operation, ok or error, and the duration. The input is not on the line, and neither is the store's error code or body.

The line names the first 16 calls in storeCalls and counts the rest in storeCallsDropped. A line for a request that made no store call has neither key. A call made outside a request is not recorded and still runs. A call made after the response has gone, inside waitUntil, is not recorded either: the line was written when the response left.

getScopeStore takes an optional last argument, { onStoreCall }. When it is passed, that observer hears each call instead of the request's line. It carries no authority, and a throw from it does not change the store's answer.

Erasure ​

Subject erasure calls substrat.erase on every store any version of the scope's vertical has declared, before the subject's key is destroyed. A binding a later version dropped still names an object that may hold the subject, so it is erased too. The erasure's answer lists each binding and how many rows that store said it erased. When the control plane erases a scope its own host holds, the host's audited receipt does not list those store erasures yet; only the answer does. A store that answers with an error stops the erasure before the key is touched, and the next run finishes it.

Erasure fails closed. When the platform cannot tell which stores a scope's version declares, or cannot reach them, the erasure is refused with 503 and the key stays until a retry reaches the store. A scope whose vertical has never declared a store, in any version, holds none, so its erasure proceeds even when its version cannot be told.

A deploy that lands while an erasure runs does not slip past it. Before the key goes, the erasure checks that the scope still runs the version it read, on the same pin, with the vertical's serving pointer where it was. A bind, a promote, a rollback or a serving re-upload in between makes the erasure answer 412, and the key stays for the retry, which reads the new version's stores. A bind that comes after that check is refused instead.

Erasure reaches the live store. It does not reach a copy already in a backup. See below.

Snapshot, restore, export and backup ​

A scope dump carries stores: one entry per binding, with the driver and the tables its pages (substrat.export-page) returned, merged. An old dump has no list, and loads as a scope that had none. On the hosted plane the control plane reads those tables from the store script. A store it cannot name or reach fails the backup, the pull or the export, instead of leaving the store out.

A store's tables in a backup are not sealed per subject. The platform cannot see inside them, so it cannot seal a subject's data in them the way it seals the scope's own classified fields. Erasing a subject reaches the live store and never the backup file at rest. The restore replay below is what removes that subject again when the backup comes back.

A binding a later version dropped. Export, wipe, erasure and the meter reach every binding any version of the vertical has declared, so an object under a dropped binding is still wiped, erased and measured, whatever moved the scope off the version that declared it: a bind, a promote, a rollback or a serving re-upload. A restore imports only the bindings the scope's version declares, so a dump leaves a dropped binding out when its export is empty. One that still holds data refuses the backup, the pull or the export (409), naming the binding: declare it again, with its driver, in a version the scope runs, or reap the scope without a backup, which wipes it. A reap without a backup, erasure and wipe never export, so they reach the store either way.

A binding keeps its driver for the life of its vertical. A push that declares an existing binding as another driver is refused (409); give the new store a new binding name.

A masked export carries no stores. The masker cannot see inside them, so they are left out rather than shipped unmasked. The full tenant export carries them.

A backup holds each store as pages of its own, so no step of a backup, a restore or a snapshot holds a whole store. A restore stages every store first (substrat.stage-page), then loads the scope, then calls substrat.commit-stage on each store, which replaces the object, then replays erasures. A refused page, stage or load leaves every store as it was. A commit that fails part-way leaves some stores restored and some not: the answer names which, the erasure replay still runs, and running the same restore again finishes it. A backup, a restore or a snapshot moves at most 300 pages of stores, or runs at most 20 seconds, in one request, and refuses past that with nothing changed. A backup whose stores hold more than one page each is not served whole by GET …/backups/:capturedAt: restore it instead.

An upload restore stages the dump's stores the same way. A stage that fails before its commit is discarded and the object stays as it was. A commit whose answer is lost can leave the object replaced while the restore reports failure: the commit is safe to retry, and running the restore again finishes it. A dump store whose binding the destination's version does not declare, or declares as another driver, refuses the restore before anything is written.

After a restore, and after a snapshot copy, the platform runs substrat.erase again on every store of the destination, for every subject erased in the destination or in the dump's source. Each subject is erased once. The restore's answer reports it, and only when there is something to report:

  • storeErasuresReplayed: how many erase calls ran.
  • incomplete: some erasure that applies was not replayed. A failed call is one cause, and running the same restore again finishes it.
  • sourceUnknown: the dump's source is not a scope of this tenant on this platform, for example another environment's backup or an uploaded file. Its erasures cannot be read, and running the restore again does not change that. An unknown source never refuses the restore.

The host's own restoreScope result carries the same facts as storeErasuresReplayed, storeErasureIncomplete and storeErasureSourceUnknown. With none of them present, nothing had to be replayed and nothing is missing.

Reap, deleting a snapshot, reaping a preview and deleting the scope fence and wipe each store before the scope's own storage goes. If that cannot run, the delete stops, and the scope's storage is left for the retry.

Every path that deletes a scope stops routing it first, by archiving it or by unbinding its hostnames. It takes every refusal it can decide up front (a copy move in flight, a copy the platform cannot reach, stores it cannot name), then fences and wipes its stores, then deletes it. A refused delete leaves its stores untouched, and a snapshot or a preview still routed.

Neither of those stops a request already running or a call on its way. The fence does. The platform's store entry answers substrat.fence, in front of any driver: it marks the object, waits for the calls inside it, and hands the driver its substrat.wipe. From then on the object answers 410 with the code store_fenced to a vertical's call, whenever that call was signed, and refuses the platform's own stage and commit. A vertical reads this as not_found with storeCode: store_fenced. The object never comes back: a deleted scope's data comes back only in a new scope, with new objects, and only from a backup small enough to download whole (see below). A plain substrat.wipe fences nothing.

Once a reap has started fencing a scope whose row stays, the directory records it. Unarchiving that scope, restoring into it, backing it up, forking it, pulling it whole and un-deleting its tenant are all refused from then on, even if the fence never landed, and the refusal names the way out: finish the reap. Its data is in the backup the reap took. A backup stored in one entry, with each store in one page, downloads whole and restores into a new scope. A larger one restores only into its own scope, which is now closed, so it cannot be restored anywhere yet. A reap asked again on such a scope takes no new backup, since its stores may already be wiped: a scope reap finishes with backup=false, and a tenant reap skips that scope's backup and backs up the rest. A full tenant export is refused while one of its scopes is in this state; the masked export is not.

A driver does all its writing before its fetch returns. The fence waits for a call until then, and work left running after it (ctx.waitUntil, an unawaited task, a streaming body) could write after the wipe. The review of a release holds this.

The storage meter ​

The storage gauge samples each scope's stores in the same pass as the scope database, at most once a day. The hosted reading asks the store script for substrat.size. It keeps one row per binding per day in _substrat_scope_store_storage. Retention drops those rows on the same horizon as the database samples.

  • A store read has its own deadline. A store that hangs fails that one scope's store reading. The database size just read stands, and the rest of the batch is recorded.
  • A store that cannot report its size is not recorded as size 0. The database sample stands, and the store keeps its last reading.
  • A version that drops its last binding stops being billed for stores: the day is recorded as measured with no stores in it.

/meters adds the store bytes up as stores when a sample exists, and leaves the field off when it does not.

Known limits ​

These are open, and are tracked in the scope-stores RFC (§10.8). The first group is to be closed before a release of a driver is approved in production, since only an approved release can hold data. A test environment holds synthetic data, so a release may be approved there first. The second group can follow.

Before the first approved release in production:

  • Unroute is not a drain, and the fence that closes it is new. Deleting a scope unbinds its hostnames, and a call already past the router is then refused by the store object itself, which a deletion has fenced (see Snapshot, restore, export and backup, #2285). The fence ran on local workerd on both entries, the Python one under both SDK shapes, and on a test account. The store entry's probe under pywrangler dev has to pass in CI.

Later:

  • A deleted snapshot's id. A deleted snapshot or preview loses its row, so creating a scope with that exact id gives one whose stores refuse for good.

  • The graph driver does not erase a subject yet. Its substrat.erase answers { erased: 0 }, so the platform refuses a subject erasure that would reach a graph store (501) and keeps the key, rather than record an erasure that removed nothing. Until it does, a graph store is meant for synthetic data only, and the driver serves only the verticals its registration names, modeling-studio alone so far.

  • Retiring a binding. A binding no version still declares stays in the vertical's record, so export, wipe, erasure and the meter keep asking its store, which answers empty once wiped, and its name cannot be declared again as another driver. The explicit retire step is still to come.

  • A store past one request. A store of more than 300 pages cannot be backed up, restored or snapshotted yet: nothing moves the rest outside the request. A full pull or a tenant export still reads a store whole into its one answer.

  • Replay budget. The restore's erasure replay runs inside one request and starts again from the first subject on a retry. It converges only while it fits one request.

  • Audited receipt. On the path where the control plane erases a scope its own host holds, the audited receipt lacks the platform's store erasures. The answer carries them.

  • Versions without a kept manifest. Replay and wipe skip a destination whose version predates kept manifests. Such a version holds no store, and a test that says so is still to come.

  • Delete without a tenant id. The Cloudflare host's scope delete skips the store wipe when it is not given a tenant id. Hosted, the control plane's own wipe covers it.

  • Peer calls. A vertical serving a peer call has its store calls refused.

  • Only during a request. A call from a Durable Object, including a scheduled job, does not pass egress and is refused. The platform's own jobs are not affected.

Writing a driver for the pure adapter ​

A driver that runs on the pure adapter ships a ScopeStoreRunner: driver, start, stop, and invoke(address, request). The address is { tenantId, vertical, scopeId, binding }. A host starts each runner once, on the first store call. If one runner's start throws, the next call starts only the runners that have not started. close waits for a start in flight, starts no runner after that, and then calls stop on every runner. A stop that throws does not keep the others from being called. One failure is rethrown as it is, and several as an AggregateError.

What a runner answers:

  • One object per whole address. Two addresses that differ in any one field never share an object.
  • stop without start resolves. A stop and then a start serve the same objects: they outlive the process, not the runner.
  • Every answer is the envelope, { result } or { error: { code, status, body? } }. An operation the driver does not know is an error answer, not a throw.
  • substrat.size answers { bytes }.
  • substrat.selfcheck takes no input, builds the driver's whole runtime, and answers { operations }: the catalog that runtime serves. A rollout's canary compares it with the catalog the release was approved with.
  • substrat.export-page takes { cursor, maxBytes } and answers { tables, next, revision }: one page of the dump, as table fragments in the scope-dump table shape. A table's rows may spread over pages, each fragment names its table again, and every table appears at least once across the chain; a driver may answer an empty last page. cursor is opaque (null starts) and next is null after the last page. maxBytes bounds the serialized page, except that a page always holds at least one table heading and one row: maxBytes: 1 is valid and means the smallest page, and one row larger than maxBytes rides alone. A row the driver could not take back in one substrat.stage-page is refused here, naming the table, so that a backup fails when it is taken and not when it is restored.
  • A chain of pages is one moment of the object. revision is opaque: while it stays the same along a chain the pages are one read, and when it changes the platform reads again from the start. The driver chooses how it keeps that promise: a pinned revision, refused writes during the read, or a snapshot. Any change to what the dump holds changes the revision, the reserved writes included: a commit, a wipe, and an erase that erased something. The same state reads the same revision across chains, and a revision that is a hash of the content is allowed, so a commit of identical content need not move it. A driver never refuses a maxBytes; it may cut a smaller page. Under steady writes a read of several pages can fail with a conflict after three attempts.
  • substrat.stage-page takes { stageId, seq, tables }, holds the page aside and changes no data a call reads (only substrat.size, which counts held stages), and answers { staged }. Page seq 0 starts the stage and drops any other, so the old id can then neither stage nor commit. A seq already held answers { staged: false } and is not held twice, so a lost answer can be retried. A gap is refused. A stage id is not used again: naming the stage last committed is refused. A stage the object does not hold is refused with status 409, code stage_refused. A stage unused for longer than STORE_STAGE_MAX_AGE_MS (seven and a half hours) is dropped no later than the next call to the object, so a driver needs no timer. substrat.size counts the stages the object holds.
  • substrat.commit-stage takes { stageId }, replaces the whole object with the stage atomically, and answers { tables } as the count of distinct tables it wrote. It is the only step that changes the object. A stage the object does not hold is refused, except the one last committed: asked again, it answers the same count and changes nothing, which makes a commit retry safe. Only that stage answers: an older one is refused, and a wipe ends the memory (a new page 0 or an erase does not).
  • substrat.discard-stage takes { stageId } and answers { discarded }: whether the object held the stage. The driver also drops a stage that was abandoned.
  • substrat.wipe answers {} and leaves an empty object, also for an address nothing was written to. It drops every stage. It may deleteAll(): on a deletion, the entry holds its own mark through it.
  • substrat.erase takes { subjectId } and answers { erased }, counting what it removed. A subject with nothing on the object, a repeat included, answers { erased: 0 }. It drops every stage, since a stage may hold the subject: a restore racing it fails at its commit.

A driver's registration declares the largest page of its dump it takes in one call (dumpPageMaxBytes, counted on the serialized page), and the platform and every host ask for pages within it: the graph driver declares 900 000 bytes, under its 1 MB limit on a request. A registration declares at most 3 MiB (STORE_DUMP_PAGE_MAX_BYTES), and a driver the registry does not hold is asked for pages of that size.

scopeStoreRunnerContractSuite in @substrat-run/contract-tests holds a runner to each of these, with four exceptions. It does not hold the age of an abandoned stage (that needs a clock the suite does not have), and it does not hold the refusal of a row the driver could not take back in one substrat.stage-page (that limit is the driver's own). It holds that a refused stage leaves the object as it was, and that an erase which erased something moves the revision, only when the fixture gives it refusedStage and writeSubject; without them those two checks show as skipped, by name. A driver runs it in its own repository, with a write, a way to write the n-th distinct row, a read of its own catalog, and the whole catalog:

ts
import { scopeStoreRunnerContractSuite } from '@substrat-run/contract-tests';

scopeStoreRunnerContractSuite('graph', {
  makeRunner: () => graphRunner({ dir }),
  write: { operation: 'apply', input: change },
  writeRow: (n) => ({ operation: 'apply', input: rowChange(n) }),
  read: { operation: 'inspect' },
  // Every operation the driver serves: what its substrat.selfcheck must answer.
  catalog: ['apply', 'inspect'],
});

A dev host reads its bindings from the vertical's declaration with scopeStoreRunnersFor. How a blob cell is spelled in an export that crosses JSON is not settled yet, and the suite does not assert one.