The Substrat book, in one file. Published at https://substrat.net/book/ — this edition is generated from those chapters at build time, so it cannot fall behind them.
This is the book. The rest of these docs are a reference — you arrive with a question, find the page that answers it, and leave. That works well once you know the shape of the system, and badly before. Nothing in a reference tells you how the pieces join: which thing calls which, what happens between the moment a request lands and the moment a row is written, who retries what when it fails.
So this section is the other thing. Thirteen chapters, meant to be read in order, front to back, in a long afternoon. It assumes nothing except that you have written server software before. It repeats very little — where a reference page already says a thing well, this book links to it and moves on.
await blocks.ctx.emit to a consumer in another module, what happens when
the consumer throws, and which failures retry and which do not. Also how a platform intent
gets run in seconds instead of at the next sweep.substrat push actually does, and what happens to data across a version change.Straight through. Each chapter ends where the next begins, and later chapters lean on earlier ones — chapter 5 assumes chapter 4's picture of a handler, chapter 9 assumes chapter 5's outbox, and chapters 10 to 12 assume both. If a word stops you, the glossary is at the end of chapter 1. The Next link at the foot of every page is the intended path.
Two chapters are worth reading even out of order, because they are the ones the reference genuinely does not cover anywhere: The life of one event and The two clocks.
The whole book is also published as a single document, three ways:
pandoc, or for handing to a model in one shot.All three are generated from these chapters at build time, so they cannot fall behind.
Tip — Reading it on a phone
There is no URL that opens Books directly — iOS decides that from the file itself. Tapping book.epub in Safari downloads it; opening it from Files then offers Books, which syncs it to your other devices. On a Mac, double-clicking the download does the same thing in one step.
If you want the reference in one file instead — every page on this site, not just the book — that is llms-full.txt, and the index agents read first is llms.txt.
The book first shipped with ten chapters. This edition adds three: Seeing what happened, The audit trail and the lake and Metering and billing. Operating it is now chapter 13.
It also answers questions readers of the first edition asked. Chapter 1 now has a glossary.
Chapter 4 explains the absence of a query builder, the text timestamps, and what an await
holds. Chapter 5 explains the kick that makes platform intents fast. And the chapters now
reflect what shipped since: invocation ids and the event walks, the MCP surface, the lake drain,
the dashboard's four tabs, and the new reference verticals.
Substrat is 0.x. Several things in this book are described as built because they are, and a few are described as not built because they are not. Where a mechanism has a known gap, the chapter says so in the place you would otherwise assume it was covered, rather than in a footnote. What Substrat doesn't have (yet) is the concentrated version of that.
Every piece of business software for a specific trade — a field-service app, a workshop system, a clinic's booking tool — is mostly the same software. Not in the part anyone pays for, which is the vocabulary and the screens and the pricing. In the part underneath: tenancy, roles, permission checks, an audit trail, migrations, deploys, backups, an invite flow, a way to run a copy of production without touching production.
That underneath is perhaps 80% of the code and nearly all of the risk. It is also the part that is wrong in almost every such system, in the same few ways, and the ways are not subtle:
WHERE tenant_id = ?. One forgotten clause, and a query reads another
customer's rows. The type checker cannot see it, the tests do not have two tenants in
them, and the failure is silent until it is a disclosure.log.info call. It is complete exactly as long as
everyone remembers to write one, which is to say it is not complete.draft to invoiced
because every code path happens to move it that way — until one does not.Each of these is an instance of the same thing: an invariant the system depends on, enforced by a person remembering. The remedy is not better discipline. It is moving the invariant somewhere discipline is not required.
Substrat's bet is that this substrate is worth building once, properly, and that "properly" means the runtime refuses, not that the docs advise.
So: a kernel owns tenancy, permissions, events, migrations and the transaction boundary,
and it owns them in a way module code cannot opt out of. There is no WHERE tenant_id
to forget, because a handler's ctx.sql is already inside one tenant's database and has
no syntax for reaching another. There is no un-audited mutation, because the event is
written in the same transaction as the row and the envelope is stamped by the kernel, not
supplied by the caller. There is no unchecked operation, because a check that was never
called leaves a trail of its own.
That is the claim Why runtime enforcement? argues at length. This book takes it as given and shows the machinery.
Diagram — the three-layer stack. Six bands, top to bottom: verticals, the line, engines, the kernel, the adapters under it, and connectors at the edge.
Callout — field serviceHandlebar — bike workshopKallkälla — coffee shopMeridian — HRManyfold — headless CMSworkorder — one state machine · append-only time + materialbooking — resource × interval × capacity · one allocation, no locksinvoicing — consumes billable events · immutable once exportedprotocol — checklists + signed docs · freeze → immutable, hashedinvites — hashed identifier · accept-required · non-enumerablectx.sql, ctx.check, ctx.emit, ctx.link.adapter-sqlite — dev · CI · self-host / escrowadapter-cloudflare — production · Durable-Object per scopeadapter-email — notification transport · CF Email + mockScrive eSign — signatures-requested → BankID signing → recorded back…more — one port per capabilityThe four rules that hold it together
Read it bottom-up.
The kernel owns what must never be wrong. It provides an OperationContext — sql,
emit, check, link, grant, now — and nothing else. It knows no domain entities
at all: it has never heard of a work order or an invoice, and could not be made to
special-case one.
Engines own invariants inside a domain. The work-order engine knows that a completed order cannot go back to scheduled, that every state change emits an event, that an exported billing basis is immutable afterwards. There are seven of them, each headless: no screens, no vocabulary, no opinion about what you call things.
Verticals own everything a user touches: the words, the roles, the prices, the screens, the workflow. A vertical composes engines the way an application composes libraries, except that the library's invariants are enforced from below rather than trusted.
The line between the bottom two bands and the top one is the interesting one, and it is drawn where it is on purpose. Below it, change is slow, reviewed, and versioned, because being wrong there is expensive. Above it, change is fast — a vertical is exactly the kind of code an agent or a small team can write quickly, because the expensive mistakes are not available to make. You cannot skip a state. You cannot forget a check and have nothing notice. You cannot read another tenant.
That is the actual product: not "AI writes your app", but a place where AI writing your app is not reckless. Where AI mistakes stop is that argument with the specific failure modes named.
It is worth being blunt about the shape of the trade, because it is a real one.
You give up arbitrary data access. A handler reads and writes through ctx.sql,
inside one scope, and nothing else. No connection pool, no second database, no fetch to
a sibling service. If your design needs a handler to read across tenants, the design has
to change; the runtime will not bend.
You give up cross-module joins. Another module's tables are private — not "please don't", but a linted, reviewable boundary. A vertical that wants extra fields on an engine's entity keeps its own side table keyed by the engine's id, and never a column upstream.
You give up synchronous everything. Effects that are not scope-local — sending mail, calling a provider, writing tenant-wide directory state — do not happen inline inside your transaction. They are asked for, and effected out of band. Chapter 5 is largely about this.
In exchange, a class of bug becomes structurally unavailable rather than merely rare, and you get the operational half — audit, snapshots, previews, migrations, backout — as properties of the substrate rather than as projects.
Substrat is not a BaaS, not an ORM, and not a low-code builder. It does not generate your UI, though it will generate your API client. It has opinions about tenancy and none about CSS.
It is also not finished. What Substrat doesn't have (yet) is the current list, and this book flags the relevant gaps where they arise rather than saving them up.
A handful of terms carry most of the weight from here on. Each one gets its full chapter later, but none of them should be used before it is defined, so here they are in one place.
Tenant. The business that pays. It is the billing and identity boundary.
Scope. One isolation domain inside a tenant, such as a branch, a client or a property. Each scope is its own database (chapter 2). An installed app is a scope running a vertical.
Module. The unit of code the kernel loads into a scope. A module is one registration: a manifest (its id, the permission keys it declares, the events it emits and consumes, its schedules), plus its SQL migrations, its operation handlers, their input schemas, and its event consumers. Every engine is a module, and so is every vertical. A deployed app registers its own vertical module next to the engine modules it composes, in a fixed order, because that order is also the order migrations run in.
Operation. A named, permission-checked, transactional entry point into a module, invoked as
module/name, such as workorder/create. The only way anything changes a scope's data.
Engine. A module that owns invariants in one domain, such as work orders, invoicing or booking, and has no screens and no vocabulary.
Vertical. A module that owns everything a user touches in one trade: the words, roles, prices and workflow. It composes engines. It also ships the worker around the module: HTTP routes, authentication and the app.
Harness. Code in the vertical's worker that is not module code: the server, the seed, the routes, the tests. The rules in chapter 4 bind module code, and harness code is where the exempt work, such as calling a model, happens.
The spine. The kernel's own tables, all named _substrat_*, inside every scope and in the
directory: the event outbox, the delivery journal, the denial log, the permission tuples, the
platform-intent queue, the admin log. Module code may read the spine and may never write it.
It is the record the rest of the system trusts.
Event. A kernel-stamped fact that a mutation happened, written to the outbox in the same transaction as the mutation (chapter 5).
Consumer. A module's handler for another module's event. It runs inside the same scope, in its own transaction, as a system actor.
Executor. A handler the host registers for an event, which runs outside the scope with platform authority, for effects one scope cannot make on its own. Connector is the executor you will meet most: one that is also given a credential and permission to call exactly one outside provider.
Platform intent. A request, written by an operation into the spine, for the platform to do something privileged on the scope's behalf, such as provisioning another scope. The platform executes it later and writes the outcome back.
Control plane, directory. The platform's own service, and the database it keeps: which tenants and scopes exist, where they route, who holds which role, and the admin log.
There are two ways to look at what follows, and you need both.
The layer stack above is what the code is — which package owns which concern, which direction dependencies point. It is the picture you need when deciding where a change belongs.
The other picture is how it runs: a request arriving at a hostname, resolving to a tenant and a scope, reaching a database that holds that scope's data and no one else's. That is the next chapter, and it is the one that makes the rest of the book legible.
Everything in Substrat hangs off one structural decision, and it is the one to understand first, because every later mechanism is shaped by it.
A tenant is the business that pays you. A scope is one isolation domain inside that business.
Diagram — a tenant and its scopes.
stockholm (branch, active · eu), göteborg (branch, active · eu), malmö (branch, provisioning).The tenant is who you bill and who can log in. The scope is where data lives and where consistency is decided — which is why isolation is a property of the substrate here, not a WHERE clause somebody has to remember.
What a scope means is your vocabulary, not the kernel's. It is a kind string the
kernel never branches on: a housing association for a property manager, a branch for a
chain, a client company for an agency, a brand for a publisher. Users belong to the
tenant, to a scope, or to several scopes with different roles in each.
This shape is in nearly every vertical B2B product, and it is close to impossible to retrofit — which is why it is kernel-owned rather than a convention you follow. Two details are worth knowing early:
(tenantId, scopeId) and cross-checks it — so a confused-deputy bug in calling
code fails closed rather than resolving to somebody else's scope.parentScopeId exists and is always null today. The column is there so deeper
trees are an additive change later rather than a data migration.The full entity definitions are in Tenants & scopes.
Here is the part that matters. A scope is not a partition key. It is its own database.
On the local adapter, that is one SQLite file. On the hosted runtime, it is one
SQLite-backed Durable Object. There is no shared cluster and no shared table with a
tenant_id column in it. A query issued inside a scope is issued against storage that
contains that scope's rows and nothing else.
Diagram — the topology, adapter-neutral. Vertical code reaches data one way, and every scope feeds one spine.
getScope(principal, tenant, scope) returns a capability stub. holding a stub is the authorization.The same shape on every adapter — one SQLite file per scope locally, one Durable Object per scope in production. Nothing above the stub can widen its own reach.
Follow the consequences:
Cross-tenant reads are not prevented, they are unavailable. There is no WHERE tenant_id = ? to forget, because the other tenant's rows are not in the database you are
querying. The most common multi-tenancy bug in the industry has no syntax here.
Blast radius is one scope. A corrupted row, a runaway query, a migration that throws — each is contained to one customer's one domain. There is no lock contention between tenants because there is no shared lock.
"Give me a copy of production" is a file copy. Snapshots, forks and preview environments (chapter 8) are cheap because a scope's entire state is one self-contained thing. On a shared cluster this is a export-filter-import project; here it is a copy.
Deleting a customer is deleting databases. Reaping one scope destroys one file; reaping a tenant walks every scope beneath it and then clears the tenant's own PII and configuration rows. Which is a real property when somebody invokes a right to erasure, and a real hazard too — chapter 13 covers what actually frees those bytes, because the answer is less automatic than you would expect, and because the tombstones and the admin log are built to survive it.
The cost is equally real: you cannot join across scopes. A question like "how many
work orders did this tenant complete this quarter, across all forty branches" is forty
reads and a fold, not one GROUP BY. Reads & scaling is the page on
how that is handled — projections, per-tenant D1, and what is honestly still open.
Inside one scope, one operation runs at a time, to completion, before the next one starts.
This is a stronger guarantee than most databases give you and it eliminates a whole
category of code. No interleaved read-modify-write. No lost update. No row locking in
module code, no SELECT … FOR UPDATE, no optimistic-retry loop. A handler that reads a
balance, decides, and writes it back is correct as written, because nothing ran in
between.
It is enforced explicitly, on both adapters, by a per-scope task queue — about fifteen lines of code:
export class OperationQueue {
private tail: Promise<unknown> = Promise.resolve();
enqueue<T>(op: () => Promise<T> | T): Promise<T> {
const result = this.tail.then(op);
// The chain must survive failures; callers still see the rejection.
this.tail = result.catch(() => undefined);
return result;
}
}
On Cloudflare this queue sits in front of the Durable Object's own input gate rather than relying on it. The gate is real, but it over-delivers: it permits subtler interleavings around non-storage awaits. Kernel and module code are allowed to depend on strict serialization and nothing weaker, so the adapter enforces strict serialization itself instead of inheriting whatever the platform happens to provide. The two adapters then mean the same thing by "serialized", which is what lets one conformance suite hold both.
Serialization is per scope. Two operations on the same scope queue. Two operations on different scopes, whether in the same tenant or not, run at the same moment on different databases, and neither can slow the other. Chapter 4 covers what a slow operation holds, and how a vertical keeps genuinely slow work out of the queue.
The thing to notice is the pattern, because it repeats: where the platform's guarantee is close to the contract, Substrat implements the contract anyway rather than documenting the gap.
Inputs and results cross the scope boundary as structured clones — even when the caller and the scope are in the same process.
So module code can never hold a reference to something outside its scope, and outside code can never hold a reference into it. There is no object shared across the boundary to mutate by accident. Locally this is deliberate extra work; on Cloudflare it is free, because the boundary is a real RPC hop. Doing it locally too is what stops "works on my machine, fails in production" from being a category of bug — the local runtime is the strict one.
Once you hold a scope stub, you never pass a tenant or scope id again.
const stub = await host.getScope(principal, tenantId, scopeId);
await stub.invoke('workorder/create', { title: 'Leaking radiator' });
The handler behind workorder/create receives a context already bound to that tenant and
that scope. It has no parameter for them, so there is no parameter to get wrong, and no
call site where the right ids could be passed to the wrong operation.
Holding the stub is the authorization to talk to that scope — it is a capability, and minting one is gated (chapter 3). The scope still re-validates every call against its own access rules, because a capability that is never re-checked is a capability that cannot be revoked.
Two fields are fixed at provisioning and never change afterwards:
jurisdiction — eu, us, or global. Data residency is a property of the scope
from birth. global is the honest name for unconstrained, which is what every scope is
today; eu and us name guarantees whose enforcement (Durable Object jurisdiction
subnamespaces, Regional Services) is not built yet, so provisioning currently gates them.
The vocabulary ships ahead of the enforcement deliberately, precisely because the field
is immutable: a scope that took a value nobody could name later would be a data migration
rather than a config change.
storageShape — A (the DO's embedded SQLite is primary) for everything today; B
(a control-plane DO fronting per-tenant D1) is designed and not built.
A Durable Object cannot relocate. That is not a Substrat limitation but a platform one, and it is why these are provisioning-time decisions rather than settings.
provisioning → active ⇄ suspended
↓
archiving → archived → reaped
active serves traffic. suspended fails closed — every getScope refuses — but the
data is intact and the transition is reversible. archived is inert and reversible; the
bytes are still there. reaped is terminal: the storage has been wiped, the
directory row survives as a tombstone for audit history and to burn the slug, and there
is no restore.
Tenants have a parallel ladder — active, suspended, deleting, reaped — where
deleting is a reversible grace state that makes every scope under it inert without
reclaiming anything.
What is worth flagging here, and what chapter 13 returns to: nothing moves a scope from
archived to reaped on its own unless you have configured it to. Cloudflare never
garbage-collects a Durable Object. An archived scope's bytes persist indefinitely until
something explicitly wipes them, and the sweep that can do that is opt-in because the
operation is irreversible.
Someone opens a browser, types a hostname, and clicks a button that creates a work order. This chapter is every hop between that click and the committed row, in order.
It is the chapter the reference does not have. Router stops at the
service binding. @substrat-run/vertical-host starts at
/internal/*. Operations & the scope host starts at invoke().
Each describes its own hop well and none of them describes the joins, so here they are.
browser
│ https://acme-north.example.com/api/invoke
▼
[1] router worker hostname → (tenant, scope, vertical, surface)
│ x-substrat-tenant / -scope / -surface / -vertical / -router
▼
[2] dispatch env.DISPATCH.get(deploymentRef)
│
▼
[3] the vertical's worker readRoutedNode → authenticate → principal
│
▼
[4] CloudflareScopeHost lifecycle gates → lazy migrate → mint stub
│ stub.invoke('workorder/create', input)
▼
[5] ScopeDO queue → transaction → parse → guards → handler
│
▼
[6] post-commit outbox → consumers
│
▼
response
One worker sits in front of every vertical. Its entire job is: turn a hostname into a
(tenant, scope, vertical, surface) target, and forward.
It resolves that against the control-plane directory, uncached, once per request. Not caching a route is a deliberate choice rather than an omission: a cached route that keeps serving a suspended tenant blunts suspension, and suspension is a live weapon — it is what you reach for when a customer must stop being served now. The cost is a directory read per request; the alternative is a suspension that takes effect eventually.
Only active hostname bindings resolve. A domain still validating its DNS, or one whose
certificate failed, is simply unknown and gets "No application is configured for this
hostname."
Notice what the router deliberately cannot do. It binds what forwarding needs — the
control-plane directory it reads, the dispatch namespace it forwards into — and no
SCOPE namespace. It resolves names; it has no way to open a scope's database.
That boundary is a deployment fact written into its wrangler config rather than a rule
somebody follows. Handing the router the full scope host would have saved a file and
given the name-resolution worker authority over every tenant's data.
It also does not re-check tenant suspension, even though it reads the directory and
could. getScope owns that, inside the vertical. A second enforcement point is a second
thing that can disagree with the first; the router's job is to find the door, not to
decide who may open it.
The router forwards with the resolution in headers:
x-substrat-tenant: 01J8F...
x-substrat-scope: 01J8G...
x-substrat-surface: app
x-substrat-vertical: fsm
x-substrat-router: <shared secret>
The vertical trusts these absolutely — they name whose data it is about to serve. Two things make that safe, and both are required:
workers_dev: false, no route; reachable
only by service binding or dispatch from the router.ROUTER_SECRET — the same value on the router and every vertical, presented as
x-substrat-router and verified in the kernel's readRoutedNode.The second exists because the first is a deployment fact and workers.dev is on by
default. One forgotten toggle makes (1) false with nothing in the code noticing, and the
consequence is a cross-tenant read.
Both halves fail closed. A router with no ROUTER_SECRET answers 500 to everything.
A vertical deployed without one refuses any asserted node with a RouterAssertionError
rather than trusting unsigned headers — this used to be a trust-by-default, which meant a
worker missing its secret would accept a forged tenant from anyone who could reach the
script directly. The only opt-out is ALLOW_DEV_NODE, which names an un-routed local
instance and authenticates nobody.
And the router strips every inbound x-substrat-* header by prefix before setting its
own, so a client cannot forge a node by simply sending one. Stripping by prefix rather
than by name is why adding a sixth asserted header later cannot reopen the hole.
With a target in hand, the router picks the worker.
If the resolved route carries a deploymentRef — the Workers-for-Platforms handle for the
scope's bound version — it dispatches into the namespace: env.DISPATCH.get(deploymentRef).
This is how customer-pushed verticals are reached, and it is the normal path. If there is
no bound version, it falls back to a static VERTICAL_<SLUG> service binding, the
original shape, kept for routes that predate the registry.
Every dispatch also carries the outbound policy for that version — the declared egress allowlist from its manifest — handed to the egress worker through the dispatch binding's outbound parameters. The router never inspects that list; resolution produced it and the egress worker enforces it. Chapter 5 picks this up where connectors do.
Two facts fall out of dispatching on deploymentRef rather than on the vertical's name.
Each scope is bound to a version, so two scopes of the same vertical can be running
different code at the same time — which is what makes previews and staged rollouts
possible at all (chapter 8). And the router does not re-check jurisdiction: residency is
pinned by configuration ahead of this worker and by the scope's DO placement below it,
and a third enforcement point here could only ever disagree with those two.
Now we are inside the vertical's own worker, an ordinary Hono app. The first thing registered on it is the invocation log, which writes one tenant-stamped line for this request and mints the invocation id that every event the request emits will carry (chapter 10). It is first because Hono runs handlers in registration order, and a log mounted below a route cannot see that route.
Then the worker does three things before any Substrat API is involved.
It reads the asserted node. readRoutedNode(request.headers, { expectedSecret })
returns (tenantId, scopeId, surface, verticalSlug) or throws. This is the vertical's
side of the router contract, and it lives in the kernel because every vertical needs it
and none of them should re-derive how to trust it.
It authenticates the caller. This is where OIDC happens — a session cookie, or a
sign-in redirect to the issuer and back. The vertical is a relying party; it runs no
credential store of its own. What it gets back is a sub.
It maps that sub to a principal. The authenticated subject is not yet an identity in
this scope. The per-tenant identity directory resolves it to a PrincipalId, or to
nothing — in which case this person has authenticated successfully and is still allowed to
do nothing at all. Chapter 6 is about that line.
The important thing about this step is that it is the vertical's code, not the kernel's. Substrat has an opinion about authorization and only a seam for authentication. Authentication & identity is that seam.
The same declared operations are served twice over, with no extra declaration: as REST routes
described by an OpenAPI document, and as an MCP server at /api/mcp, with one tool per
operation that faces HTTP. An agent holding a user's token can do exactly what that user could
do with curl, behind the same authentication and the same ctx.check inside the operation.
Exposing an operation is not authorizing it. A refused tool call comes back as a tool error the
agent can read, not as a failed session.
Alongside this, the platform's own management routes are mounted in one call —
mountPlatformSurface(app, deps) — which owns every /internal/* route the control plane
calls to provision, reconcile, snapshot, export, restore and configure this install,
behind a platform-secret gate that fails closed when the secret is unset. Those are not on
the request path a user takes; they are how chapters 8 and 13 reach in.
const stub = await host.getScope(principal, tenantId, scopeId);
CloudflareScopeHost is the coordinator. It is stateless, rebuilt per request, and it
does two things before it will hand back a stub.
Lifecycle gates. validateScopeAccess(tenantId, scopeId) is evaluated durably in the
control-plane DO. It refuses a scope that is not servable and a tenant that is not
servable — suspended, deleting, archived, reaped, or simply not in this tenant. This is
the K-3 fail-closed path, and it is where suspension actually bites. A throw here
propagates out; there is no degraded mode.
Lazy migration. migrateAndRecord(scopeId) asks the scope to bring itself to the
current migration frontier. Migrations are not run at deploy time across the fleet; each
scope migrates on wake, inside its own serialization domain, the first time anyone
touches it after a version change. If it throws, the coordinator makes a best-effort
attempt to record why into the directory — and then rethrows the original error, on
purpose, because a broken recorder must not replace a diagnosable migration failure with a
confusing one. A scope whose migration failed fails closed and serves nothing, which is
what stops it from rendering as healthy in the console.
Only then is the stub minted. Holding the stub is the authorization to talk to that scope — but it is not the authorization to do anything in particular, and the scope re-validates every call regardless.
stub.invoke('workorder/create', input) is an RPC into the scope's own Durable Object.
The structured-clone boundary is that RPC hop. What happens on the other side, in order:
OperationQueue. It waits until
the operation ahead of it has run to completion.ctx.storage.transaction(async …) — the DO analogue of BEGIN IMMEDIATE, which rolls
back on a throw across awaits.workorder/create is looked up in the module
registry. A module whose entitlement the tenant does not hold did not register at all,
so its operations simply do not resolve — there is no half-loaded engine and no
operation that exists but refuses.If-Match against an entity's version). Idempotency keys are resolved here too: a
replayed key returns the original result without re-running anything.There is one deliberate exception to "on a throw, all of it is gone". A permission denial is recorded after the rollback, as its own write, outside the transaction that just disappeared. That is the whole point of it: the denial is precisely the write the refused operation could not make, so keeping it inside the rolled-back transaction would erase the only evidence the refusal happened. Chapter 6 returns to it.
The moment the transaction commits, the DO drains its outbox to consumers, each delivery in its own transaction and still inside this operation's turn in the queue. That is chapter 5, and it is where most of the system's interesting behaviour lives.
Back on the coordinator, outside the scope, any executors and connectors due for the new events run next. If the operation raised platform intents, the vertical flags its response so the router can ask the control plane to act on them straight away (chapter 5, "the kick").
Then the result travels back out: clone boundary, coordinator, vertical worker, JSON, router, browser.
Nothing that matters, which is the point.
Locally, a scope is a SQLite file instead of a Durable Object, the coordinator is in-process
instead of across an RPC, and there is no router — you address the instance directly with
ALLOW_DEV_NODE. The queue, the clone boundary, the transaction semantics, the lazy
migration, the fail-closed addressing and the stamped envelopes are the same, because the
same conformance suite runs green on both, unchanged, in Node
and in real workerd against real Durable Objects. Neither side is a mock.
That equivalence is what makes local development deterministic and CI cloud-free, and it is the reason a vertical moves from a laptop to Cloudflare with no code change.
We left the last chapter one step from the handler. This chapter is that step: what an operation is handed, what it is denied, and why each denial is worth its inconvenience.
An operation handler receives exactly one thing.
async function createWorkOrder(ctx: OperationContext, input: CreateInput) {
assertAllowed(await ctx.check('workorder:create'));
const id = ulid();
ctx.sql.exec(
'INSERT INTO wo_orders (id, title, state, created_at) VALUES (?, ?, ?, ?)',
[id, input.title, 'scheduled', ctx.now()],
);
ctx.emit({
type: 'workorder.created',
entity: { entityType: 'workorder', entityId: id },
piiClass: 'none',
payload: { id, title: input.title },
});
return { id };
}
That is representative, not simplified. ctx is the entire capability surface:
ctx.tenantId / scopeId / principal |
where this runs and on whose behalf, read-only |
ctx.sql |
query / exec against this scope's database |
ctx.now() |
the operation's instant, and the only clock |
ctx.check(perm, entity?) |
a permission decision |
ctx.emit(event) |
a domain event, into the outbox |
ctx.page(entityType, params) |
one page of a declared list: filters, sort, keyset cursor, total |
ctx.search(entityType, term) |
full-text lookup over declared searchable fields |
ctx.versionOf(entity) |
an entity's current version, for If-Match preconditions |
ctx.link(child, parent) |
a permanent entity relationship |
ctx.grant / ctx.revoke |
user-initiated per-entity sharing |
ctx.canAssign(roleKey) |
whether the caller holds everything a role would confer |
ctx.atomic(fn) |
a sub-transaction |
ctx.entitlement(key) / entitlements() |
the tenant's plan, quota and expiry for a module it holds |
ctx.requestPlatform(req) |
ask the platform for a privileged effect |
ctx.platformRequests(filter) |
read back what the platform did with those |
ctx.sealToConnection(provider, value) |
encrypt a value so only that connector can open it |
Nothing on ctx checks a permission except ctx.check. page, search and versionOf
read data, and the operation's own assertAllowed comes first, as always.
A connector's handler gets a different context with a scoped connection(provider) handle.
That handle is host code's, not an operation's, and it appears later in this chapter.
There is no env. No fetch. No database handle. No process. No import of anything
that could provide one. Capabilities come from ctx and from nowhere else, and that is
the single sentence the rest of this chapter unpacks.
ctx.sql — and why it is the only way inctx.sql is already inside one scope's database. It has no parameter for tenant or scope
because there is nothing else it could address.
Module code may not import better-sqlite3, an adapter, node:*, or cloudflare:workers.
Most of those bans are obvious portability hygiene. cloudflare:workers is not, and it
is the sharpest one. That module exports an ambient env. One import hands module code
every binding and secret the script declares — including its own SCOPE Durable Object
namespace, which reaches another scope's data, precisely where ctx.sql cannot. It is a
single import that dissolves the isolation the whole architecture is built on, and it
looks completely innocuous in a diff.
The linter refuses it (node tools/boundary-lint.mjs, rule R2), and harness code —
seed.ts, server.ts, tests — is exempt because it is not module code.
One rule inside ctx.sql is enforced by the runtime, not only the linter: a write
whose target is a _substrat_* table is refused, on both adapters.
Those tables are the spine: the kernel's own tables, created in every scope database next
to your module's. The outbox and the delivery journal (chapter 5), the denial log, the
platform-intent queue, the permission tuples and their projections (chapter 6), the migration
journal, schedule state, attachments and idempotency records all live there. The directory has a
spine of its own, holding the admin log, the access log, identities and entitlements. Only the
kernel writes the spine. Forging a row in it is forging the audit trail or the permission model. This is a mechanism rather than a lint
rule because the linter does not run on the hosted push path, and a guarantee that holds
only where CI happens to look is not a guarantee. Only the write's target is judged, so
INSERT INTO my_timeline SELECT … FROM _substrat_outbox is fine.
Reading the spine is sanctioned, and there are helpers for it — use them:
readTimeline and readHistory from @substrat-run/kernel take an EntityRef, page
like an ordinary list read, and decode the envelope. readHistory additionally returns
the payload, the authorization chain, the impersonation stamp, the PII class, the emitting
operation and the version the emitting code was deployed as. On most of those a null is
a fact — the payload was erased, nobody was impersonating, a consumer emitted it — which
a hand-rolled SELECT would read as missing data instead. Neither helper checks a
permission; the caller does that first, as always.
ctx.sql takes a SQL string and a parameter array. There is no ORM, no query builder, and no
typed query layer generated from the model. The model emits tables, migrations, an API
description and a browser client, but not queries.
That can look like an omission. It is a choice, and the reasoning is worth checking against your own threat model, because the protections are real and so is one gap.
What a query builder would buy, and where you already have it. Builders earn their keep in three places: preventing injection, composing dynamic filters, and hiding dialect differences. Here:
ctx.sql.query('SELECT … WHERE id = ?', [id]) sends the value as a
bound parameter, never as SQL text. That is the same protection a builder gives, provided
values go in the array.ctx.page, which builds the WHERE, ORDER BY, keyset comparison and LIMIT from what the
operation declared filterable and sortable, and refuses anything else by name. The query that
most tempts people to concatenate strings is the one you do not write.Why injection is survivable when it happens. Everything else here is defence in depth, and it is deep:
id declared as a ULID cannot arrive
as 1 OR 1=1.; INSERT INTO _substrat_tuples … is refused like a hand-written one.The gap, stated. Nothing mechanical stops a handler from writing
`… WHERE name = '${input.name}'`. No lint rule inspects SQL strings for interpolation.
Engines interpolate constant column lists on purpose, which is why a crude rule would drown in
false positives. So "values go in the array" is a review convention, not a mechanism. And on the
local SQLite adapter, statements that reach past the scope's own file, such as ATTACH, are not
refused by the kernel. The spine check deliberately leaves them to the runtime, and on Durable
Objects the runtime restricts them. On self-hosted SQLite, an injected ATTACH is the one path
from a scope's SQL to another file on the same disk.
Why plain SQL at all: a migration is a human checkpoint, and SQL is what a reviewer can read. A query in a diff is a query, not a builder chain whose SQL you have to imagine.
ctx.now() — the only clocknew Date() and Date.now() are boundary-lint violations (R6), the same class of ban as
node:*.
Two reasons, and the second is the one that bites. First, stability: ctx.now()
returns the same instant for the whole invocation, stamped when the context was built. Two
rows written in one transaction cannot disagree about when they were written, and an event
cannot disagree with the row it describes. Second, testability: anything genuinely
time-dependent — an absence window, a metering period, a booking hold, a grant expiry — is
untestable against the wall clock except by sleeping or by shrinking the window to zero,
both of which produce tests that assert less than they appear to.
The pure host takes a clock, so a scenario uses manualClock or frozenClock and
asserts an elapsed-time transition directly. The Durable-Object host declares
clock?: never and means it: the reads that matter happen inside the ScopeDO that workerd
constructs, which a host option cannot reach. So grant expiry transitions are held to
the contract on SQLite only, and so is facet recency — that a bucket's lastSeen is its
latest event, which the wall clock cannot tell from its first. Those are the two suites the
two adapters do not share, for that one reason, and it is written down rather than quietly
tolerated.
Code that must read the real clock — a JWT whose exp a remote server judges — opts out
with a reviewable boundary-lint-allow R6 block.
Timestamps are stored as ISO 8601 text, never epoch integers. ctx.now() returns an
Instant, and every Instant is normalized to UTC in exactly one shape:
2026-09-17T08:41:05.123Z. Offsets are converted and precision is fixed at milliseconds, so every
value is exactly 24 characters.
The reasons, in order of how much they matter:
expires_at > ?, a metering
close horizon is a string comparison, and ORDER BY occurred_at sorts. Text sorts
chronologically only when every value shares a zone and a shape, and the normalization
guarantees that. An offset timestamp compared as text against a Z timestamp is simply wrong,
which is why the parser rewrites rather than accepts..sqlite file someone opens by hand, in an exported dump, and in an
event payload. The row and the event announcing it hold the identical string.A fair worry about text is that values change size on update, which in some storage engines
means rows move and pages fragment. It does not apply here, for two reasons. First, the
normalization makes every instant the same length, so updating 2026-09-17T08:41:05.123Z to a
later instant rewrites 24 bytes with 24 bytes. The row does not grow. (Moving a column from
NULL to a value grows a row whatever its type.) Second, SQLite has no append-only heap and no
MVCC row versions. An update rewrites the record inside its B-tree page, and space freed inside
a page is reused by that page.
The real cost is width. SQLite stores an epoch integer in 4 bytes as seconds or 6 as milliseconds, so 24 bytes is four to six times as much per timestamp column. It is a real cost, it is small next to a row's other text, and it buys comparisons that cannot silently go wrong.
For dates without a time, such as a birthday or a leave day, the contract is calendarDate,
YYYY-MM-DD, which sorts the same way.
ctx.check — first line, every timeEvery operation's first line is a permission check:
assertAllowed(await ctx.check('workorder:create'));
Not "usually". The reason to make it unconditional is that the alternative is a judgement call at every handler, and judgement calls are where the missing check comes from six months later.
Portal-style walks — "which of these can this person see?" — use the entity-scoped form,
ctx.check(perm, entityRef), per row. Chapter 6 is how those are answered without a
network call.
A refused check throws PermissionDenied, the transaction rolls back, and the denial is
then written outside that transaction so it survives. You get a queryable record of
what was refused, to whom, on which operation, under which impersonation session — which
is the data you want when a customer says "it says I can't" and you need to know which
permission key to look at.
ctx.emit — and the envelope you do not controlA mutation emits an event. The kernel stamps the envelope: id, occurred-at, tenant, scope, actor, authorization chain, impersonation, the emitting operation, the deployed version. The module supplies the type, the entity, the PII class and the payload — and the payload is validated.
You cannot suppress the stamp, forge an actor, or emit on another scope's behalf, because
emit is below the API surface rather than beside it. That is what makes the event stream
an audit log by construction instead of by discipline.
Two rules about payloads matter enough to state here, and chapter 5 explains them:
ctx.atomic — the only way to catch an engine errorThis one is a real trap, and it has a mechanism behind it.
An engine call composed inside your transaction has no boundary of its own. So a bare
catch around it leaves you holding its partial writes — the rows its invariants were
protecting — and then commits them:
// WRONG — boundary-lint R7 rejects this
try {
await completeWorkOrder(ctx, { orderId, billable });
} catch {
// the engine's half-written rows are still here, and will commit
}
The correct shape gives the engine call a boundary:
try {
await ctx.atomic(() => completeWorkOrder(ctx, { orderId, billable }));
} catch {
// the engine's rows, events, links, grants and platform intents are all gone;
// your own writes survive, and it still commits once
}
A succeeded atomic is still provisional — if the operation later throws, its writes
go with everything else. Sub-transactions nest but must not interleave; starting two
concurrently throws.
Outside ctx.atomic, catching an engine error is forbidden and R7 rejects it with no
escape hatch — unlike R5, R6 and R8, which all have reviewable boundary-lint-allow
blocks. There is no legitimate reason to swallow an engine error unprotected.
try/finally with no catch is fine, and so is a catch that always rethrows: the
operation still fails and the whole transaction still rolls back.
There is no fetch in module code. None. Boundary-lint rule R3 refuses it, along with any HTTP
client import.
Effects on the outside world are somebody else's job. There are three mechanisms, and all three share one shape: the module writes a durable row in its own transaction, and something outside the transaction acts on it afterwards.
HostAdmin) instead of a ctx. It is the
shape for effects that are not scope-local, such as writing tenant-wide directory state,
because a module cannot write to another database inside its own transaction without leaving
an orphan on rollback.ctx.requestPlatform) is how a sandbox-clean vertical asks for a
privileged action it has no authority to perform itself. The intent is a durable row written
atomically with your operation. The platform pulls and executes it later, and
ctx.platformRequests() reads back what happened, including a failure, so a screen can say
"this did not go out" instead of showing something that only appears to have.All three are chapter 5.
await holdsHandlers are async, so a fair question is what happens while one waits. Does a slow
operation block other requests on its scope, or on every scope?
Its own scope only. Chapter 2's queue gives each scope one turn at a time, and the turn lasts from the moment the operation starts until its post-commit work is done. On Durable Objects, post-commit work includes draining in-scope consumers (chapter 5). So an operation that takes two seconds, or a consumer that takes two seconds, delays the next operation on that scope by two seconds.
Everything else runs on:
In practice a handler has almost nothing slow to wait on, and that is the ban on the network
working as designed. ctx.check, ctx.entitlement and ctx.canAssign read scope-local
projections (chapter 6). sealToConnection is local Web Crypto. Nothing on ctx crosses a
network on the normal path, so the queue drains quickly because module code cannot reach
anything slow. Nothing in the kernel times an operation out, either. The platform's own limits
are the only ceiling.
Some work is slow by nature: calling a language model, rendering a document, waiting on a provider. The pattern is to keep the slow part out of the operation and bracket it with operations:
// in the vertical's worker (harness code, not module code)
const context = await stub.invoke('desk/prepare-answer', { conversationId }); // fast: read + record intent
const answer = await models.answer(context); // slow: no scope turn held
await stub.invoke('desk/record-answer', { conversationId, answer }); // fast: write + meter + emit
The model call holds no transaction and no queue turn. Each operation around it is short,
checks its own permission, and commits on its own. The reference support-desk vertical answers
questions exactly this way, with the whole sequence handed to waitUntil so the user's
request returns first. If the work must survive the isolate going away, make the first
operation record that it is pending, so a sweep or the next request can find unfinished work.
The other option is to make the slow part an effect: emit an event, and let a connector do it with retries (chapter 5). Pick by one question: does the vertical need the result inside its own logic (bracket it), or is the result a side effect in someone else's system (emit it)?
Operation inputs go through Zod schemas at the boundary, and the host applies them.
A module passes operationInputs: operationInputsOf(ops) beside its operations, and
every invocation is parsed before the guards and before the handler, on every path in —
HTTP, test, seed, schedule. Handlers do not hand-parse.
The distinction is not stylistic. If parsing were the handler's job, a declared input that nobody validated would be an ordinary oversight. Because it is the host's job, it is not possible.
Each rule in this chapter closes a specific failure that is otherwise silent:
| Rule | The failure it closes |
|---|---|
ctx.sql only |
reading another tenant |
bound parameters, ctx.page for dynamic reads |
injection, and filters nobody declared |
no cloudflare:workers |
one import, every secret and every scope |
| no spine writes | a forged audit trail |
ctx.now() only |
rows disagreeing about when; untestable time logic |
| normalized ISO text | timestamps that compare wrongly |
| no network in handlers | a slow provider holding a scope's queue |
| check first, always | the guard that was never written |
| kernel-stamped envelopes | an event that lies about who did it |
ctx.atomic to catch |
committing an engine's half-finished state |
| host-side input parse | a declared schema nobody applied |
None of these is enforced by asking nicely. Most are linted, several are runtime mechanisms, and the ones that are only conventions are named as such in Agent rules rather than allowed to look enforced.
This is the chapter the reference genuinely does not have. Every fact in it is published
somewhere — Events & audit has the semantics, the adapter pages have
the machinery, Modules & the manifest has the declarations — but no
page walks the loop, so here it is, from ctx.emit to the consumer's commit, including
every way it can fail.
ctx.emit({
type: 'workorder.completed',
entity: { entityType: 'workorder', entityId: id },
piiClass: 'none',
payload: { id, lines, total },
});
Nothing is sent. emit writes a row into _substrat_outbox, inside the operation's
transaction, and returns.
That is the whole foundation of the design. The event and the domain write commit together or not at all. There is no window in which the row exists and the event does not, and none in which an event announces a row that rolled back. No two-phase commit, no coordinator, no reconciliation job to find the mismatches — because the mismatch is not representable.
The envelope is stamped kernel-side. The module supplies four things (type, entity, PII
class, payload); the kernel supplies the id (a ULID, which is monotonic, so ORDER BY id
is creation order), the instant — the operation's instant, so every event one operation
emits carries the identical one — the tenant, the scope, the actor, the authorization
chain, the impersonation session if there was one, the operation name, and the version the
emitting code was deployed as.
A consumer must never need a cross-module read to interpret an event. So the payload carries everything: the billable lines, the prices, the totals — not an id to go look up.
The consumer therefore snapshots rather than joins, and prices are frozen at the moment the event happened. Which is exactly what an invoice wants, and it stops being a stylistic preference the moment you notice the alternative: a consumer that reads back into the producer's tables is a consumer that breaks when the producer's schema moves, and that reads today's price for last month's work.
And the consumer validates with its own Zod parse, never the producer's types. Importing a producer's type would make the contract a compile-time coupling; parsing the payload makes it a runtime contract, which is the only kind that survives two modules being on different versions.
piiClass is declared, not defaulted. It is what makes erasure a query rather than an
audit: when a subject invokes a right to erasure, the payloads that must be cleared are
the ones the class names, and the envelope — who did what, when — survives, because the
envelope is the compliance record.
The transaction commits. Immediately after, still inside the same invocation, the scope drains its outbox:
// scope-do.ts, the tail of invoke()
if (!replayed) await this.dispatch(tenantId, scopeId);
dispatch is small enough to describe exactly. It runs up to 50 rounds. Each round
walks every registered module, and for each of that module's consumers, selects the outbox
rows of the matching type that have no row in _substrat_deliveries for that
(event, module) pair, oldest first.
For each such event:
await this.ctx.storage.transaction(async () => {
const ctx = this.operationContext(systemPrincipal, tenantId, scopeId, { system: mod.id });
await consumer.handler(ctx, event);
this.sql.exec(
`INSERT INTO _substrat_deliveries (event_id, consumer_module, delivered_at)
VALUES (?, ?, ?)`,
event.id, mod.id, new Date().toISOString(),
);
});
Three things are load-bearing here.
Each delivery is its own transaction. The consumer's writes and the journal row marking it delivered commit together. So a crash between the handler finishing and the journal being written cannot exist — either both happened or neither did. Delivery is exactly-once with respect to the journal, and at-least-once with respect to the world.
The consumer runs as a system actor. Its context carries { system: '@substrat-run/engine-invoicing' },
and that is what the audit trail shows for anything it writes. It is an ordinary in-scope
operation in every other respect — same ctx, same rules, same transaction semantics, same
inability to reach another scope.
The loop rounds because consumers emit. A consumer's own ctx.emit writes another
outbox row, which the next round picks up. Fifty rounds is the cascade limit, and the loop
exits as soon as a round delivers nothing. An event emitted this way records causedBy, the
id of the event whose delivery emitted it. It also inherits the originating request's
invocation id, because the drain is part of the same call. Chapter 10 walks both links.
All of this happens inside the scope's turn. The drain runs after the commit, but before the queue lets the next operation in. A slow consumer therefore delays the next operation on its own scope (chapter 4), and no other scope.
This is the part worth reading twice, because it is not what most queue-shaped systems do.
} catch (err) {
// Dead-letter (v0): journal the failure so one poison event
// can't wedge the loop. Written outside the rolled-back txn.
this.sql.exec(
`INSERT INTO _substrat_deliveries (event_id, consumer_module, delivered_at, error)
VALUES (?, ?, ?, ?)`,
event.id, mod.id, new Date().toISOString(), String(err),
);
}
The consumer's transaction rolls back — none of its writes survive — and a journal row is written with the error, outside that rolled-back transaction, so it persists.
In-scope consumers do not retry. One failure is terminal for that (event, consumer)
pair. The next round's query excludes the event because a delivery row now exists, so the
loop cannot wedge on a poison event, and the failure is evidence rather than a silent drop.
That is a deliberate v0 choice and the code says so. It is the right default for a consumer that is pure in-scope logic — such a consumer fails because of a bug or bad data, and retrying a bug five times produces five identical failures and a delay. It is the wrong default if you were expecting queue semantics, so: a failed in-scope consumer needs a human or a replay, not a wait.
Effects that genuinely need retrying are not in-scope consumers. They are executors, and they work differently.
Guaranteed within one (scope, module) pair, and not across them. Within a pair, the
ORDER BY o.id on a monotonic ULID mint means creation order.
Some effects are not scope-local. Adding someone to an organization writes tenant-wide directory state, outside any one scope's transaction. Sending mail leaves the building entirely. A consumer cannot do either, because a consumer runs inside the scope and can only touch that scope's data.
So a module asks, and an executor effects:
module (in-scope) executor (out-of-band)
ctx.emit('member.add-requested') ──▶ admin.addMember(...)
commits WITH the domain write writes the audit row
host.registerExecutor('member-adder', 'member.add-requested', async (admin, event) => {
// receives HostAdmin, not ctx — it acts with platform authority,
// which is exactly what module code must never hold
});
Why not write directly from the module? Because that would be a write to another database inside a scope transaction: two independent commits, no coordinator, and an orphaned membership if the scope rolls back after the directory write lands.
The event has no such hazard, because it enters the outbox in the same transaction as the domain write. A rollback leaves no event and therefore nothing to effect.
The membership example above is the textbook one, and it lives in the contract suite. In the running platform, the executor you will actually meet is a connector: an executor that is also handed a credential and one provider's worth of egress. An e-signature connector is the reference, registered against the event a protocol emits when it requests signatures.
Where it runs depends on the host, and the difference matters:
connector:<provider> platform intent (below). The control plane, which holds the
credential, runs the connector when it drains that intent.A third kind of connector subscribes to no event. An accounting or planning integration that pulls data runs as a sweeper inside the platform sweep (chapter 9), on a schedule.
An executor runs on the coordinator, not in the DO — it acts through HostAdmin, which
is outside the scope entirely. So the drain is a read inside the DO, the effect happens
outside, and the journal is written afterwards:
SELECT o.* FROM _substrat_outbox o
LEFT JOIN _substrat_deliveries d
ON d.event_id = o.id AND d.consumer_module = ?
WHERE o.type = ?
AND (d.event_id IS NULL
OR (d.next_attempt_at IS NOT NULL AND d.next_attempt_at <= ?))
ORDER BY o.id
"Due" means never attempted, or retrying and now past its next attempt time. Terminal rows
— delivered, or dead-lettered with next_attempt_at IS NULL — are excluded by the join.
The journal is written after the effect, on purpose. Claiming a delivery before running it would make delivery at-most-once and lose the effect on any crash in between. Writing afterwards means a crash mid-effect retries — which is why executors, like consumers, must be idempotent, with the event id as the idempotency key.
Retry policy is per executor, not a host-wide constant, because the right answer differs:
| default | |
|---|---|
maxAttempts |
5, including the first; reaching it dead-letters |
baseDelayMs |
1000ms, doubling per attempt |
maxDelayMs |
300000ms (5 minutes) — the ceiling on the doubling |
with jitter of ±20% on each computed delay. Those defaults suit a directory write. A connector making an outbound HTTP call wants a longer tail, and sets one.
A failed attempt with no next attempt time is a dead letter — the row keeps the last
error, and the drain report counts it. ExecutorDrainReport returns attempted,
delivered, retrying and deadLettered, and those last two are the numbers a health
surface reports. A caller that ignores them learns nothing, which is exactly the failure
mode the older silent path had.
Dispatch is prompt: the drain runs inline, in the request that emitted the event, so the common case completes before the response goes out. The outbox is the backstop, not the mechanism.
The backstop needs something to run it, and this is where the reference splits across pages
in a way that genuinely confuses people. It is the scope sweeper — an alarm-driven
singleton Durable Object in the vertical's own deployment, calling drainDue over its
roster of scopes. Not the platform sweeper, which is a different clock doing different
work in a different place. Chapter 9 is both of them and the distinction.
Splitting a change across two halves would otherwise split its audit trail. So admin rows
an executor writes carry causedBy — the id of the event that caused them.
That is the event id rather than a separate correlation field, because the envelope already carries a unique kernel-stamped id, and reusing it avoids widening a frozen contract to say something it already says.
There is a third shape, for when a sandbox-clean vertical needs a privileged action — provisioning a sibling scope, say — and holds no privilege at all.
ctx.requestPlatform(request) writes a durable row into this scope's
_substrat_platform_requests spine, atomic with the operation, exactly as emit is. The
platform pulls and executes it later, knowing the tenant inherently because it read this
scope's DO. Origin fields are stamped kernel-side; the call returns the new id.
Call it after your own permission check. Authorization is the vertical's decision; isolation is the platform's. And it applies backpressure: a scope holding the maximum pending intents throws rather than queueing without limit.
Intent kinds are a closed set: provisioning a sibling scope or a whole tenant, archiving a
scope, reconciling a managed tenant's entitlements, recording a model-usage line (chapter 12),
and the connector:<provider> family. The platform uses one more kind for its own sweep
records, and module code is refused it, so a vertical cannot forge a verdict about its own
schedules.
An intent is pulled, never pushed, and by default the pull is the platform sweep: a fifteen-minute cron on the hosted control plane (chapter 9). An intent raised at 10:01 can therefore sit until 10:15. For an intent a user is waiting on, such as "provision my new branch", that is too slow.
So there is a fast path, and it has three parts, each owned by a different party:
operation commits with ≥1 intent
│ host fires onPlatformRequests(count) ← only after COMMIT, never on rollback
▼
vertical sets response header x-substrat-platform-request: 1
│
▼
router sees the header → waitUntil(kickDrain) ← after the response has gone
│ POST /internal/drain-scope {tenantId, scopeId} (service binding + platform secret)
▼
control plane drains that one scope now
The header carries no payload and no privilege. The control plane re-derives the tenant and vertical from the directory and simply pulls. A forged or spurious header costs one empty pull. The user never waits for the kick either, because the router runs it after returning the response. With it, an intent settles in seconds.
The part people miss is the vertical's. The kernel exposes the signal and does not set the header for you. A vertical opts in when it mints its stub:
const stub = await host.getScope(principal, tenantId, scopeId, {
onPlatformRequests: () => c.header(PLATFORM_REQUEST_HEADER, '1'),
});
A vertical that skips this still works, and its intents wait for the sweep. That is the designed failure direction: the kick is advisory, and a missed kick costs one sweep interval, never an intent. It is also why "provisioning works but takes a quarter of an hour" is almost always a vertical that never wired the header, and not a slow platform.
A drain that throws leaves the intent pending for the next pull. After 100 attempts (about a
day at sweep cadence) it settles as failed and records an operational failure, so a stuck
intent eventually becomes a visible one.
The outcome comes back. ctx.platformRequests(filter) reads this scope's own intent
journal — newest first, filterable by kind and status — with the result or lastError
the platform settled. That read exists for one concrete reason: a contract whose signature
request settled failed can say so on its own screen, instead of showing a document that
appears to be out for signature and is not. The kernel owns every write to that table, so a
status is only ever the platform's answer.
Nothing leaves the scope inside the transaction. The intent is a row; something outside has to notice it. Two things do, and they are not alternatives.
Diagram — how committed work leaves a scope.
The kick and the sweep are not alternatives. The kick makes it fast; the sweep makes it certain — a lost kick costs a wait, never a missed intent or event. Neither carries data: the control plane always reads the scope itself.
The kick is the fast path. The vertical flags its response, and the router — the one hop that already knows which tenant and scope it just served — returns the response and then asks the control plane to drain that scope. The sweep is the guarantee: every 15 minutes the platform walks every active scope and drains what is pending, so a kick that was lost costs a wait and never an intent.
The kick is safe for a reason worth stating. It carries no data. It names a scope, and the control plane reads that scope itself and re-derives the tenant from its own directory, so the most a kick can do is make a scope's own work happen sooner.
The invoicing engine consumes workorder.completed and commerce.order-placed, events from
two different domains, without importing a single type from either producer. From them it
builds a customer's billing basis, the lines an invoice will be issued from (chapter 12). It has
deliberately no in-scope exports: the vertical emits, the engine consumes, and the
vertical reads results back through the engine's own operations or by consuming its events.
That is not an omission. The engine being the only writer of its rows is what keeps invariants like immutable-after-export safe from a half-finished caller. Chapter 7 is when to reach for that shape versus a direct call.
Event payload fields are frozen once shipped. Rename, remove or retype means a
schemaVersion bump, and a bump is a replace — you cannot emit v1 and v2 side by side
during a deprecation window.
The reason is mechanical rather than philosophical. Consumer dispatch selects on event
type alone (WHERE o.type = ?), and the schemaVersion a manifest's consumes entry
carries is discarded at registration. So emitting both versions delivers both to the
same consumer. The invoicing engine's export event is meant to be consumed by an accounting
connector. None consumes it yet (chapter 12), but once one does, a dual-emit is a double
invoice in production, silently. The rule is not hypothetical either: the export event has
been through exactly one version bump, and it was shipped as a replace.
A replace fails loudly instead: a v1 consumer's strict parse rejects v2, and the event dead-letters where somebody sees it. Loud and stopped beats quiet and doubled.
So no plan should promise a deprecation window that nothing can deliver. Routable
dual-emit needs (type, schemaVersion) in the dispatch predicate plus a version dimension
on the consumer registry — designed, filed, and postponed rather than abandoned.
Because every event carries tenant, scope, actor, entity and time — stamped, not supplied — the stream is the audit log. Complete by construction. "Who did what, when, to which entity" is a query, and the answer is the same data reporting runs on.
Which is why reading one entity's history is a sanctioned projection over
_substrat_outbox — rule 3 bans writes to the spine, not reads — and why readTimeline
and readHistory exist rather than everyone writing their own SELECT.
The same events also leave for a longer-lived store. The sweep drawn above ships each scope's
unshipped events to the platform's Tier-2 event lake, where history across scopes and over
long periods can be queried, and marks each one shipped only after the lake has confirmed it.
A failure in between sends the same events again, which the event id makes harmless; marking
first would risk a gap nothing could detect. Shipping removes nothing from the scope — the
outbox still serves consumers, entity history, and each entity's version.
Two questions that look like one and are not.
Authentication asks who someone is. It happens at the edge, it involves an issuer, and Substrat has an opinion about none of it beyond the shape of the seam.
Authorization asks what they may do. It is kernel-owned, it is enforcement input, and it is never delegated to an auth provider — because an authorization model that lives in your identity vendor is a model you cannot reason about, version, diff, or take with you.
A vertical is an OIDC relying party. Sign-in is a redirect to an issuer and back; the vertical runs no credential store, holds no passwords, and implements no reset flow.
What comes back is a sub — a subject identifier from the issuer. That is not yet
anything. The per-tenant identity directory maps sub → PrincipalId, and a sub with
no mapping is a person who has authenticated perfectly and may do nothing. The two
halves are genuinely independent: proving who you are gets you no authority at all.
Every demo vertical in the repo is OIDC-only, and the local development story is a real
OIDC provider (@substrat-run/dev-issuer) whose only shortcut is that /authorize lists
names instead of asking for a password. So the local login is the production round-trip,
and changing issuer is a change of OIDC_ISSUER rather than a change of code. There is no
dev-auth branch to diverge.
Authentication & identity is the detail — issuers, adapters, first-login sync, and the owner-claim flow for a freshly provisioned instance.
Humans, and agents under review, write exactly three kinds of thing.
Permission keys — module-namespaced strings, declared in the manifest with human-readable descriptions:
workorder:create Create work orders
workorder:report Start work, report time and material
invoicing:export Export an invoice basis (makes it immutable)
Roles at nodes — a role bundles permissions; an assignment binds a principal to a role at a node of the tenancy tree (the tenant root, or one scope), inheriting downward.
host.admin.assignRole({
principalId: tech,
roleKey: 'technician',
node: { tenantId, scopeId: stockholmBranch }, // scopeId: null = whole tenant
});
Capability grants — one permission, one node, optionally narrowed to one entity and its declared descendants, optionally time-boxed:
host.admin.grant({
principalId: portalCustomer,
permission: 'workorder:read',
node: { tenantId, scopeId: branch },
entity: { entityType: 'facility', entityId: theirBuilding },
expiresAt: nextMonth,
grantedBy: adminPrincipal,
});
Entity-narrowed grants are how a customer, a board member or a subcontractor sees only their things inside a shared scope. A grant can also target an organization, and members reach it through membership.
That is the entire administrative surface, and there is no fourth kind of thing to author. What arrives later in this chapter is not a fourth kind but a fourth author: a user sharing one record from inside an operation, which mints a capability grant of exactly the shape above.
Diagram — a permission, from declaration to check.
The branch off the top ends in a person: PERMISSIONS.md exists to be read in a diff, and CI going red is what makes the reading unskippable — it is not itself the approval.
Internally the checker compiles those three things into relationship tuples
(subject → relation → object) and evaluates with a fixed four-rule algebra:
workorder → facility) and written at
runtime by ctx.link; entity-narrowed grants flow along them, depth-capped.No negation. No configurable rewrite rules. Verticals never see or author tuples.
The fixed algebra is the point. A policy language rich enough to express anything is rich enough to express a policy nobody can predict the behaviour of, and "why can this person see this?" stops having an answer. Four rules have an answer, always.
This is the design decision with the widest consequences, so it is worth stating plainly.
Scope and entity tuples (rules 2 and 3) live in the scope's own database and evaluate inside its serialization domain. Tenant-level facts — role definitions, role assignments, tenant grants, org membership (rules 1 and 4) — are tenant-wide, so the control-plane directory is their authority.
But the checker never reads the directory on a request. The control plane projects
those facts into scope-local tables (_substrat_roles, _substrat_tenant_tuples) at
write time, and the checker reads the projection exactly as it reads scope tuples.
The control plane is a write-time authority that projects into scopes. It is never a read-time dependency on the request path.
Cost moves from the read path (every request) to the write path (rare role and grant changes), which is the correct direction for a read-heavy multi-tenant system. And it is what makes a scope genuinely isolated: a vertical can run as its own Worker with no control-plane binding at all, which is the shape a customer-pushed vertical ships in.
The consistency consequence is small and one-directional. Scope-level grants and roles stay synchronous and immediately consistent — the checker runs in the scope's own serialization domain, so there is no "did my write land" question. Only tenant-level changes are eventually consistent across a tenant's scopes, bounded by the write-time fan-out and a reconciliation sweep.
And the failure direction is safe: an absent or empty projection is a deny, byte for byte the normal deny path. Missing data can only ever remove authority, never grant it.
Access is withdrawn by tombstoning a tuple: it keeps its row, gains a revokedAt, and
the checker's walk skips it. Nothing deletes a tuple, ever.
A tuple that once granted access is the evidence of why an access was allowed. Deleting it destroys the audit trail exactly where it is most needed — a deleted row can show neither that access was revoked nor that it was ever granted.
"Ever" means for as long as the scope does. Reaping is the one thing that takes a tuple away, because reaping takes the whole database the projection lives in; what outlives it is the admin log, which is never swept (chapter 11).
Liveness is therefore one predicate applied identically everywhere: a tuple grants only while it is unexpired and unrevoked. Expiry and revocation are siblings, not separate mechanisms.
type Decision =
| { allowed: true; proof: RelationTuple[] } // the chain that granted access
| { allowed: false; checked: PermissionKey; node: Node };
An allow always carries the chain that produced it. An unexplained allow is unrepresentable — which is a strong statement, and it is what makes three features possible rather than aspirational: explain ("why does this user see this?"), view-as-user (render any screen as any principal, with real decisions), and the human-readable permission diff that the merge checkpoint reads.
The proof also travels. When an operation emits, the kernel stamps onto the envelope which
permissions it checked and passed, and — when the allow came through a capability grant
rather than a role — a reference to the granting tuple. explain can re-derive the chain
later, but it cannot recover which grant was consulted at write time once the tuples have
moved since. That pointer is what the envelope keeps, so a mutation carries its own proof
of authority.
An allow leaves its trace on the event it authorized. A denial has no event to ride, so it
is captured on its own: a row in the scope-local _substrat_denials table with the actor,
the permission, the node, the operation, the impersonation session, and the time.
This is the one moment where an actor's intent and the permission model visibly disagree, and no other log witnesses it — the admin log records changes, the outbox records allowed mutations.
The mechanical detail from chapter 3 matters here: because a denial rolls its operation
back, the row is written after the rollback, as its own write. Only an enforced denial
is recorded — one that went through assertAllowed and therefore carries the checked key
and node. A module's own hand-thrown PermissionDenied is its policy, carries no key, and
is left alone.
Everything above is administrative. The same capability grant has a second author, and it
is the user rather than the admin: ctx.grant and ctx.revoke, called from inside an
operation.
await ctx.grant(theirPrincipal, 'todo:read', { entityType: 'list', entityId: listId });
await ctx.revoke(theirPrincipal, 'todo:read', { entityType: 'list', entityId: listId });
Both are entity-required and delegating — the caller's own decision on that entity is re-checked, so you cannot share what you cannot see. Both are transactional with the operation: if it rolls back, so does the share.
This is the primitive for "share this record with a person", and neither alternative is it.
A ctx.link edge is permanent and not revocable at all. Org membership is revocable but
coarse — a whole org, not one record — and minting an org per domain row to obtain a revoke
is a known anti-pattern with an issue number attached to it.
demos/todo/src/module.ts shows both in one line each, and is worth reading before you
design any sharing feature.
New permission keys and role definitions are one of the two things an agent may never
self-approve. The mechanism: each vertical exports MODULES and ROLES from its seed,
pnpm lint:permissions renders demos/*/PERMISSIONS.md from those same objects, the file
is checked in, and CI re-emits with --check.
So a widened role cannot merge without appearing in the PR diff as a table of key → description → which roles hold it. CI going red is what makes the reading unskippable; it is not itself the approval. A human reads the diff.
Chapter 1 named the three layers. This chapter is the middle and top ones in working detail: what belongs in an engine, what belongs in a vertical, and the rules that keep seven engines from becoming forty-two pairwise integrations.
There are seven: work orders, invoicing, booking, protocols, invites, metering, absence. Each one is headless. No screens, no words a user reads, no opinion about what you call things.
What an engine owns is the set of statements that must never be false:
The test for whether something belongs in an engine is not "is it reusable". It is: would it be a bug in any vertical for this to be violated? Pricing is not that — one customer rounds differently. State transitions are.
Vocabulary, roles, pricing, screens, workflow, the shape of the API its own app calls. A vertical composes engines the way an application composes libraries, except the library's invariants are enforced from below rather than trusted.
The repo carries eight demo verticals. Two of them are the reference implementations. They answer different questions, and they are references because they are used. A demo nobody runs rots, however complete it looks.
demos/todo: the shape. A record app built forward from its declared model, with
per-list sharing by email, revoke, and a React app that tells a 403 from an empty list. It
has no engine at all. Not every vertical needs one, and pretending otherwise leads to
engines that exist to be composed rather than to hold an invariant. It is what a scaffolded
project's docs point at.demos/ticket0: a deployed vertical. A support desk that runs in production. It composes
the metering engine by call, runs sandbox-clean on the hosted runtime, sends and receives mail
through a relay, answers with a model around its operations (chapter 4), and ships through a
CI push. It is the one to read for how a real vertical meets the platform.Diagram — the line. Two bands either side of one boundary.
The layer where models are weakest is the layer where mistakes are fatal. So the split is structural rather than instructional: prompting a model to "be careful with tenancy" is a suggestion, and a getScope call that fails closed on a mismatched pair is a fact.
No engine imports a sibling. None. Composition happens through three kernel-mediated channels:
(entityType, entityId) without
knowing what the entity is.workorder.completed and commerce.order-placed — two different domains —
without importing a single type from either producer.The arithmetic is the argument: compatibility stays at N kernel contracts instead of N² engine pairs, and each engine versions independently.
There is a corollary test worth remembering: if two engines need chatty synchronous communication, they are one engine drawn wrong. That is why "work orders + time reporting" is one engine rather than two that call each other constantly.
And if a vertical finds it must fork an engine, the engine drew its line wrong. That is a bug report about the engine, not a fork to maintain.
An engine is composed in one of two modes, and which one it is determines its whole shape.
Operations are thin: the permission check, plus one exported in-scope function. All the logic lives in composable exports, so a vertical wraps them inside its own transaction:
// in the vertical's operation, same transaction, own permission check
assertAllowed(await ctx.check('fsm:complete-job'));
await ctx.atomic(() => completeWorkOrder(ctx, { orderId, billable }));
await ctx.sql.exec('UPDATE fsm_jobs SET closed_at = ? WHERE order_id = ?', [ctx.now(), orderId]);
One transaction, both writes, atomic. The vertical extends by composition and never by
forking. Note the ctx.atomic from chapter 4 — it is what makes catching that engine call
safe.
The vertical emits, the engine consumes, and the vertical reads results back through the engine's own operations or by consuming its events into a side table keyed by the engine's id.
There are deliberately no in-scope exports. That is not an omission — the engine being the only writer of its rows is what keeps immutable-after-export safe from a half-finished caller. If a vertical could call in mid-flow, the invariant would depend on the caller finishing.
Which mode an engine is, is a fact about its exports. State it in the engine's header, so an absence reads as intent rather than as something nobody got round to.
Not "please don't" — a linted boundary with a reviewable escape hatch.
Engine data is reached through the engine's exported in-scope functions. The stable surface
is entity ids, EntityRefs, and event payloads. A vertical needing extra data on an engine
entity adds its own side table keyed by the engine's id, and never a column upstream.
One-time extraction handoffs — a genuine migration where the data must move once — use an
explicit boundary-lint-allow R5 … boundary-lint-end R5 block. That is a reviewable
escape hatch rather than an absent rule, which is the right shape: rules with no hatch grow
workarounds that are worse than the hatch.
Three rules, and one of them has teeth that surprise people.
New operation inputs are optional, with behavior-preserving defaults. Adding a required input is a breaking change to every existing caller.
Permission keys are never renamed. A rename silently removes authority from everyone who held the old key.
Emitted event payload fields are frozen once shipped. Rename, remove or retype means a
schemaVersion bump — and chapter 5 explains why a bump is a replace with no dual-emit
window available.
There is one call site of every engine that used to be invisible to the compiler: the
scaffold template. It imports the engines, so it is a real call site — and nothing compiled
it, so a non-additive change could merge, release, and reach npm create substrat broken.
That is now closed. tools/template-sync.mjs materializes the template into
packages/template-check, a workspace member holding workspace:* links, so typecheck,
test and boundary-lint all reach it with no new command to remember. A non-additive
engine surface is now red in its own PR, against the workspace.
There is a second, separate gate — pnpm lint:scaffold — that runs the same scaffold
against the published registry, post-release and weekly, never on a PR. The distinction
is the whole point: between a merge that adds a surface and the release that publishes it,
the template legitimately runs ahead of npm. Being ahead of the registry is a pass on the
PR gate and a legitimate red on the release one.
A type-checked seam is not enough, because a vertical compiled against engine 0.3 and running against 0.4 has no compiler in the room.
So a value crossing the engine seam is .parsed by the schema the engine publishes — on
the way out as well as in. And a read names its columns rather than using SELECT *,
which would pin the published shape to whatever the physical table currently holds.
Without this, the failure is not a throw. It is a field that moved, read as a different field, rendering wrong data on a screen. Silently. That is why the parse is unconditional including on bulk reads: dev-only validation is absent exactly where version skew lives.
Two helpers in @substrat-run/contracts do the work — returns(schema, surface, value) and
columnsOf(schema) — and an engine binds them to its own name in one line
(engineSeam('engine-workorder')). Every engine is converted, each with a test/seam.test.ts
that moves its tables underneath it and asserts a throw rather than wrong data.
The SELECT * half is mechanical: boundary-lint R8 fires on a star read anywhere in an
engine's module code, with the same reviewable allow-block hatch for a maintenance read whose
row never leaves the engine. The returns() half is still convention — proving a
row-returning export is parsed needs a type checker the linter deliberately does not carry —
so a new engine is expected to do that half itself. Said plainly rather than implied.
A vertical increasingly does not hand-write its manifest. It declares entities, operations and permissions once in a typed module, and the manifest's entity fragments, permission list, event list and DDL are derived from that declaration.
The compiler checks the joins: a parent naming no entity, an entityIdFrom naming no output
field, a payload carrying an erasable field are compile errors rather than runtime
surprises. pnpm lint:model --check gates the emitted model.json the way
lint:permissions gates the permission surface.
The browser client is generated from the same declaration — entity interfaces, paths,
methods, request bodies, the paged link walk. What stays hand-written beside it is only what
the model does not declare: which principal a request is made as, and the error envelope the
vertical chose in its own app.onError.
The model is that story in full, and it is the page to read before building anything.
A change is committed. This chapter follows it from a laptop to a scope's database.
The decision that shapes everything here is that uploading code and serving it are two acts, deliberately not fused.
substrat push
The CLI builds your worker locally, then POSTs the bundle plus a manifest to the control
plane, which records an immutable version: a deploymentRef, a permission digest, a
migration digest.
That is all. Nothing is serving the new code. A push is cheap and reversible — it adds a row to a registry. The reason for the split is that "here is a new build" and "run this build against real data" are different decisions, and a mistake in the second costs far more than a mistake in the first, and fusing them means you can only ever make both at once.
Before the upload, the CLI runs the layer rules on the source tree — the same
boundary-lint that guards this repo, now shipped as a package — plus a permission
preflight that derives your declared surface and refuses a drifted one. Both run before the
wrangler build, because a refusal is worth more in a second than at the end of a build that
was going to ship broken.
The push carries more than code. It flattens what each module declares, including the events
it emits and consumes, its schedules and its freshness expectations, onto the version's manifest.
That is what lets the dashboard later draw declared behaviour against observed behaviour
(chapter 10). It also refuses a manifest that declares a binding in the SUBSTRAT_* namespace,
because the platform injects names there, such as the version id every event is stamped with,
and a vertical must not be able to shadow them.
That gate is newer than it sounds, and the gap it closed is worth knowing. Until it existed,
the mechanical rules this book has been describing ran only in this repo's CI. A vertical
developed anywhere else was built, uploaded and admitted having been checked by nothing,
which made the rules advisory for every real customer — the opposite of the claim made for
them. There is an honest bound on the fix, too: this runs on the builder's own machine, not
platform-side over the uploaded bundle, and --skip-lint exists. The flag says out loud that
the push was ungated, on the reasoning that a flag which silently weakens a gate becomes the
default in somebody's CI.
Before a version can be promoted, it must be admitted, and admission is answered mechanically by the sandbox contract.
The bundle may declare only its own durable stores — a positive binding allowlist. No
CONTROL_PLANE. No platform secret. It runs inside a Workers-for-Platforms isolate, held
to quotas. Satisfy the contract and nothing else is in question.
The allowlist deliberately excludes egress-shaped bindings (send_email, ai, browser).
Reaching the outside world is not a binding — it is a granted capability. A vertical
that needs one declares a request in its manifest (substrat.sendsEmail,
substrat.usesModels, substrat.provisions), and something outside the bundle turns it on.
The request is refreshed on every push and grants nothing by itself; no push can set the
flag it needs.
What comes back differs by capability, and the difference is worth knowing:
substrat.usesModels is answered with an ai binding
appended after the allowlist check has run on the declared set, so a vertical still
cannot declare it for itself — only be handed one. It takes both halves: the platform
willing to bind at all, and this version having asked.Who must vouch for a version depends on who will be exposed to it:
The human gate did not disappear; it moved to the boundary where it means something. Publishing is the checkpoint, not every push.
prodA vertical has exactly one channel. There is no dev and no staging.
Those existed once and were write-only — nothing ever read or served them. The reasoning is worth keeping: a channel names a pointer at code, and an immutable version id already does that. What a real non-production environment needs is not a second pointer but a second scope with data, which is a preview.
A promotion re-points prod at a version. It is a rebind, not a rename: the same scope,
the same data, serving new code.
Two things a promotion always respects:
The surface checkpoint. A promotion whose permission or migration digest differs from what is live is refused until the diff is acknowledged. This is the same two-checkpoint discipline the kernel applies to modules, applied at the deploy boundary, with the owner as the human at their own checkpoint.
Channel history. Every promotion appends a row: what went live, what it replaced, who did it, exactly when. That history is the dashboard's rollback picker, and each timestamp is an instant a point-in-time rewind could restore data to.
A version update is not a rebind to fresh, empty storage. A promote re-uploads the bundle onto the vertical's one stable serving script; the scope's Durable Object and its SQLite stay put.
Which means:
And migrations do not run in a fleet-wide deploy step. Each scope migrates on wake —
the migrateAndRecord call from chapter 3 — inside its own serialization domain, the first
time anyone touches it after the version moved. A thousand scopes do not migrate at once;
each migrates when it is next used, and a scope that fails to migrate fails closed and
serves nothing, which is what stops it from rendering as healthy.
Stragglers are not left to chance: the platform sweep reconciles migrations as its first phase, walks scopes that are behind the frontier, and flags the ones that keep failing. Chapter 9 is that pass.
Each scope records the version its provision hook last ran against. Once a scope is bound to
a new version — by a promote or a per-scope bind, never by an upload alone — the platform
sweep re-runs /internal/reconcile on every active install whose receipt is behind.
So whatever a new release's onProvision mints reaches existing installs too. Which is
exactly why that hook must be idempotent: it runs more than once over a scope's life,
and a hook that assumes it runs at provision time only will do the wrong thing the first
time a release changes what it mints.
A version is badged code-only or schema-change at publish, so you know before you promote whether the schema is involved.
Before a promotion whose migrations changed, you want to see the new version run against real-shaped data without touching production.
Substrat can do this because the scope-host contract already runs identical module code on two adapters. A preview is a fork of a scope's data bound to the new version, reachable at its own URL.
The governing law is that migrations are forward-only. A snapshot taken today, at production's frontier, bound to today's or a later version, rolls its migrations forward on the copy. Rehearse, throw the fork away if it breaks, and production never saw it.
The inverse is invalid: pointing old code at today's data across a schema change cannot work, because you cannot un-run a migration. So the rollback strategy is fork-before-you-promote — snapshot right before binding the new version, and a bad version leaves the prior one still runnable at its correct frontier.
A fork is a governed dead end. The export copies the kernel spine too, so no connector, no schedule and no billing consumes from a preview — a test copy cannot send a real invoice. Preview URLs are non-public: gated code running against real-shaped data.
Because every app is one scope with its own database, recovery is a per-tenant primitive rather than an environment-wide runbook. One tenant's rewind has a clean, self-contained blast radius.
PITR rewind — the first-hours backout. Right before an upgrade migrates, the scope bookmarks the instant (Durable Object point-in-time recovery, roughly 30 days of history). If a promotion goes wrong, an audited rewind rolls that scope's data back to the bookmark, time-boxed to about 24 hours unless forced. It is a destructive, in-place rollback of one DO — the opposite lever to a fork, which is a non-destructive copy. It rewinds the whole database, so a stale bookmark is refused rather than silently skipped.
One sharp edge, stated because it will bite otherwise: data and version binding live in different DOs. Rewinding past a bad migration means also rebinding the version — two coordinated rewinds, with no atomicity between them.
Backup / restore — the considered path. substrat scope restore loads a backup into an
existing hosted scope, replacing its data: a pulled .sqlite, a local scope file, or a
dump. This is the deliberate recovery when a time-boxed rewind is not the right tool.
Push uploads a version. Admission asks whether it may run here, mechanically when you are the only tenant exposed and with a human when you list it. Promotion points the one channel and serves in place, so data carries forward and migrations run forward against it, each scope on wake. Previews rehearse a risky version against a fork. PITR rewind and backup/restore are the backout.
The staff gate lives at publish — the moment other tenants can run your code against their data — and nowhere else in the loop.
Everything so far has been driven by a request. This chapter is the work nothing asks for: retries, recurring jobs, reconciliation, reaping.
There are two background clocks, they run in different places, they do different work, and the reference describes them in two pages that never sit next to each other. Getting them straight is most of understanding how Substrat behaves when nobody is looking.
Both clocks ship as Durable Object alarms. Three reasons, and the first one is decisive:
A dispatch-namespace script gets no crons. A hosted vertical pushed into
Workers-for-Platforms does not honour triggers.crons. A DO alarm is the only timer such a
deployment can own. Everything else follows from needing a mechanism that works there.
No overlap. A cron fires on a schedule regardless of whether the last run finished, so two passes overlap the moment one runs long. Here the next alarm is set only after the current pass settles — so the interval is a gap between passes, not a rate. A slow pass delays the next one rather than racing it.
No configuration. An alarm self-arms from code. There is no wrangler.toml entry to
forget in one environment and remember in another.
The first reason is also the limit of the argument, and it is worth being exact about which clock it binds. Clock one runs inside a dispatched vertical, so the alarm is the only timer available to it and there is no choice to make. Clock two runs on the platform's own deployment, which is an ordinary worker and can hold a cron — and the hosted control plane uses one, firing the same pass every fifteen minutes. So the no-overlap property above is the alarm's, not the pass's: on the cron path a long pass can still meet the next tick, and it is the pass itself that has to tolerate that.
And the loop is built never to die. A pass that throws whole is still caught, still reported
through onPass, and the alarm is still re-armed — the alternative being workerd's own
alarm retry, which would tighten the interval rather than keep it.
Arming is idempotent. ensureArmed() does not move an alarm that is already set, so it is
safe and cheap to call on every request, which is exactly how it is kept alive. Where a cron
is available, pointing scheduled() at ensureArmed() makes the cron a safety net that
recovers a lost alarm, never a second thing that runs passes.
Where: inside the vertical's own deployment, where its modules and its scopes' data are.
Singleton name: scope-sweeper.
What it holds: a roster in its own storage — scope:<scopeId> → tenantId.
This is the clock a vertical cannot do without, and the one most people mean when they ask "what retries my failed delivery?".
One pass walks the roster, eight scopes at a time by default, and does two things per scope:
drainDue(tenantId, scopeId) — the executor retry driver from chapter 5. This is the
outbox's backstop: connector deliveries and executor effects that failed and are now past
their next_attempt_at. The pass returns attempted / delivered / retrying /
deadLettered summed across the roster.
runDueSchedules(...) — the recurring work a module declared in its manifest. Two
properties make that safe to run unattended:
{ system: '@your/module' }, not
as whichever admin last touched the data. Modelling a nightly job as a signed-in user is
precisely the audit-trail laundering this exists to avoid.ctx.check is still the only gate. A schedule's declared permissions are granted to
the module's system principal at provision, so the operation's own assertAllowed resolves
the same way it does for any caller. A schedule can do exactly what it declared and no
more — and revoking that grant for one tenant turns the schedule off for them, with no
special "disabled" flag, because the operation simply fails its own check closed.cadence is a floor, not a promise: a schedule fires no more often than everyMinutes,
and the sweep is what actually runs it, so sub-sweep cadences round up.
The sweeper does not enumerate anything. /internal/provision and /internal/reconcile call
noteScope, /internal/delete-scope calls forgetScope.
That is what lets a CP-less vertical — one with no control-plane binding at all — have recurring work despite having no directory to read. It knows its scopes because it was told.
And the alarm follows the roster: it lapses on an empty roster, and noteScope restarts
the loop when the first scope arrives. A vertical with no scopes burns no alarms.
Where: the platform's own deployment.
What fires it: either trigger, over the same runPlatformSweep. The adapter ships
definePlatformSweeperDO — singleton platform-sweeper, the alarm loop clock one uses —
for a deployment that wants one. The hosted control plane instead points its scheduled()
handler straight at the pass, on a fifteen-minute cron.
What it holds: nothing — it reads the directory.
This is the fleet-wide maintenance pass. One pass, in this order, each phase recording per-unit outcomes and stepping over failures rather than letting one sink the pass. Every unit of work also leaves a durable sweep run record, which is how "this schedule never ran" becomes a visible verdict rather than an absence (chapter 10):
1. Migration reconciliation. First, deliberately. The drain phase below wakes scopes, and waking migrates lazily — so running this after it would count every failed scope's attempt twice per pass. It walks scopes behind the deployed frontier, retries them with backoff, and flags the ones that keep failing. Scopes it leaves in a failed state are skipped by every later phase: they fail closed, so touching them would only re-throw the same migration error as noise.
2. Executor drain. drainDue over every active scope — the same call the scope sweeper
makes, here for deployments the platform drives.
3. Platform-intent drain. Pull and execute each active scope's pending intents from
chapter 5. Injected, because the kernel cannot reach a vertical's DO — the control plane
supplies a function that goes over the vertical's /internal surface.
4. Provision reconcile. The phase chapter 8 described: re-run /internal/reconcile on
every active install whose provision receipt is behind the version it now serves.
5. Recurring schedules. For each vertical that declares them, enumerate live scopes and invoke each due operation. A fork or a snapshot is skipped, so a test copy never runs real recurring side effects.
6. Freshness expectations. Record a verdict on event types that were expected to have been seen recently — the "nothing ran" signal, which is otherwise invisible because absence produces no row anywhere.
7. Snapshot GC. Delete expired forks. A preview with a TTL disappears here.
8. Reap long-archived scopes. See below.
9. Reap tenants past their grace window. After the scope reap, so a tenant's scopes are gone before its row becomes a tombstone.
10. Event drain to the lake. Ship each active scope's undrained outbox rows to the event sink (Tier 2, chapter 11), up to 200 per scope per pass, then stamp them drained. With no sink bound, the phase does not run and nothing is marked. Draining copies events and never prunes the outbox.
11. Access-log drain and prune. Ship staff access rows to durable storage, stamp them drained, then prune what was shipped. The sink must be durable before it resolves — everything downstream treats a resolved ship as proof the evidence survives outside the directory, so a sink that buffers and returns early turns a retention policy into data loss.
This is the mechanism the reference mentions in a single clause, and it deserves the space.
Cloudflare never garbage-collects a Durable Object. An archived scope's bytes persist forever unless something explicitly wipes them. There is no platform-level cleanup to inherit, and "we archived it" is not the same sentence as "it is gone".
So the platform sweep can do it — and it is opt-in:
runPlatformSweep(host, {
reapArchivedAfterDays: 90, // UNSET ⇒ the phase is skipped entirely
reapDeletingAfterDays: 30, // same, for tenants
});
Unset, the phase does not run. The reap is irreversible, so a deployment must name a retention window before the sweep will ever destroy anything. The default is to keep the bytes and let a human decide.
Three details you will meet in practice:
archivedAt — archived before that column shipped — has no
knowable age and is never auto-reaped. It must be reaped by hand.reapScope re-checks the archived status below the seam, so a row that changed
underneath fails closed there rather than being reaped on stale information.The tenant reap composes the same seam: for each of the tenant's non-reaped scopes, archive if needed, then reap, then clear the tenant's PII and config directory rows. The tenant row survives as a tombstone — audit history, and a burned slug — and the admin log is kept whole, because it is the compliance witness and is never swept.
What ends up gone: the storage. What remains: the fact that it existed, who ended it, and when.
Self-hosting on Node, neither DO exists. A server calls startPlatformSweeper at boot and
gets the same pass on an interval. The unit of work is identical — runPlatformSweep is
kernel code that holds no timer of its own. Only the thing that fires it differs.
That separation is why the sweep is testable at all: a test calls runPlatformSweep directly
and asserts the report, with no clock involved.
Both sweepers return a report rather than logging and forgetting. retrying and
deadLettered are the numbers a health surface should show; errors is per-unit, so one
failed connector reconciliation does not hide behind a green pass.
A caller that ignores those learns nothing, which is exactly the failure mode the older silent drain had, and it is why the reports exist in the shape they do. The next chapter is about the reading.
The last chapter ended on reports that background work returns so that something can read them. This chapter is about the reading. It covers what the platform records about a running vertical, where each record lives, and which question each one can answer.
There are four instruments, at four different grains, and the most common mistake is to ask one of them a question only another can answer:
| Instrument | Grain | Exact or sampled | Answers |
|---|---|---|---|
| Traffic | one datapoint per request, at the router | sampled | how much, how fast, how many errors |
| Invocation logs | one line per request, written by the vertical | exact while retained | what this request did, and what it printed |
| Traces | spans the runtime produces | sampled, opt-in | where outbound time went |
| The spine | one row per mutation, per delivery, per denial | exact, forever | what happened to the data, who did it, and what it set off |
The first three are about requests. The fourth is about meaning, and it is the one that cannot be rebuilt from anything else.
Every request passes the router (chapter 3), so that is where it is counted. Once dispatch
returns, in a finally so errors count too, the router writes one Analytics Engine datapoint:
index: [tenantId]
blobs: [verticalSlug, scopeId, surface, statusClass, rayId]
doubles: [durationMs, status]
The index is the tenant, so every read of this dataset is a read of one tenant. The dashboard's
chart is a sampling-weighted sum(_sample_interval) with weighted quantiles for latency, run
through the Analytics Engine SQL API.
The data is sampled. It is the right instrument for "is this app getting slower", and the wrong one for anything that has to be exact, such as an invoice (chapter 12) or proof that a particular request happened. A sampled count is an estimate, and it presents itself as one.
This is the instrument with the least obvious design, and the reason for it is a platform fact.
Workers observability is keyed on the worker script. A vertical's script serves every tenant that installed it. And a trace does not cross the router's dispatch hop into the vertical. So the router knows the tenant and cannot label the vertical's log lines with it, and the vertical's log lines, unlabelled, belong to nobody in particular. A tenant's Observability page would be either empty or someone else's.
So the vertical writes the line itself. One middleware, mounted first:
app.use('*', invocationLog({ routerSecret: (env) => env.ROUTER_SECRET }));
It writes one JSON line per request to Workers Logs, in a finally, so logging can never fail
the request:
substrat: 'invocation'
tenantId, scopeId, vertical, surface
method, path // the path only; the query string is dropped, so no OIDC code is logged
invocationId // a ULID minted for this request
status, threw, durationMs
Workers Logs indexes JSON fields, so tenantId becomes a filter. Reading a tenant's logs takes
two phases. The first finds that tenant's invocation lines. The second uses Cloudflare's own
per-request $metadata.requestId to fetch every other line the same request printed, including
a stack trace from code that has never heard of tenants.
Three details come from getting this wrong once:
readRoutedNode, never from a header read directly. A forged stamp
would put text the forger chose onto another tenant's dashboard. So routerSecret is
required: without it every assertion fails verification and nothing is written. From the
outside that looks the same as no traffic, which is why a bare invocationLog() is refused.lint:invocation-log checks the order, not just that the
mount exists. The scaffold template ships with the mount in place.Automatic tracing is set per push, from a sampling rate the control plane holds. Today that rate
is on in the test environment and off in production. Nothing in the platform emits a span per
operation, permission check or engine call. What reaches the trace data is what the runtime
generates on its own, mainly outbound fetch and the Durable Object entry hop. The platform
reads it for one purpose: comparing a version's observed egress with its declared
allowlist (chapter 8).
A per-operation waterfall is designed and not built. That is part of why the next section matters.
Chapter 5 described the outbox as the audit log. It is also the best observability data the platform has. Every row is exact, typed, stamped by the kernel, and kept. The kernel ships sanctioned reads over it, and the dashboard puts each one behind a button, so nobody has to write SQL against the spine to use it.
History. readHistory returns one entity's events, newest first, with the fields a hand-rolled
SELECT would misread: the authorization chain, the impersonation stamp, the operation, the
deployed version. In the dashboard, opening a record under an app's Data tab shows this
timeline.
Why? Every event emitted while a delivery is in flight records causedBy, the id of the
event whose consumer emitted it. walkEventCause follows that link backwards. It says how the
walk ended (complete, cut, depth limit, or a missing link), so a short chain is never mistaken
for a complete one.
What did it do? walkEventEffects goes forward. It builds a tree from the delivery journal:
which consumers an event reached, whether each one handled it, is retrying, or gave up, and how
many attempts it took. This is where _substrat_deliveries shows up in the dashboard. It is not
a raw table of deliveries but the delivery rows hanging off the event that caused them.
Same call. This joins a data change to a request. The vertical host mints the
invocationId that the log line carries, passes it into invoke, and the kernel stamps it
on every event the call emits, including events that consumers emit in that call's
post-commit tail. readInvocation returns everything one call did. The same id is on the log
line, so a record's call can be matched to the request that made it. That match is by hand
today: the Logs view does not yet filter by invocation id.
That join has limits, and they are exact:
null.rayId, not the invocationId, so traffic does
not join to events.Facets. facetEvents is a small group-and-count over one scope's outbox. It filters by type
and time window, groups by type, actor, operation, version, entity type, PII class, invocation,
or one payload field, and returns counts per bucket. Each bucket carries lastSeen, the time of
its latest event. That tells a path that stopped (it has old events) from a path that
never ran (it has none). Erased payloads are counted separately rather than folded into a
"null" bucket. This is the Events explorer.
Flow. substrat push records what each module declares it emits and consumes. The dashboard
compares that with what the outbox shows happening. Triggers lead to modules, modules to event
types, event types to the outside world, and anything declared but never observed is drawn
dashed. Next to it, per-operation health joins emitted events with the denial log, grouped by
operation. That is where a role missing a permission key shows up before a user reports it.
Everything above records things that happened. The hardest failure to see is a job that never ran, because that failure leaves no row anywhere.
Two mechanisms produce the row:
freshness: [{ eventType, within: { hours } }]. The platform sweep (chapter 9) checks each one
and records a verdict. "No invoice.sent in 26 hours" becomes a fact instead of a hunch.ctx.requestPlatform refuses the kind the platform uses
to write them.For a customer's admin, in the Dashboard:
For Substrat operators, in the Console: the fleet views, plus Operations → Failures, Issues and Sweeps. Failures are grouped into issues by fingerprint, so a thousand identical errors appear as one row with a count.
One property holds across all of them. Every read that reaches into a scope on a tenant's behalf leaves an access-log row that names what was read. Looking at data is itself recorded, which is where the next chapter starts.
"Audit log" is one phrase for several records, kept in different places, by different code, under different retention. Asking which one is the audit log is how a compliance question gets a confident wrong answer. So this chapter names each witness, then follows the one that leaves the scope: the event stream, drained into a table that SQL can query across the whole fleet.
| Record | Lives in | Written by | Witnesses | Kept |
|---|---|---|---|---|
| The outbox | each scope | the kernel, inside the operation's transaction | every allowed mutation, with actor, authorization chain, impersonation, operation, version, and the call it belonged to | for the life of the scope, and copied to the lake |
| The denial log | each scope | the kernel, after the rollback | every enforced refusal: actor, permission, node, operation, impersonation | for the life of the scope |
| The admin log | the control-plane directory | every audited HostAdmin verb |
every privileged change: who provisioned, suspended, granted, admitted, rewound, reaped, with before and after | never swept, and it outlives a reaped tenant |
| The access log | the directory, then R2 | every staff or delegated read | every look: who read what in which tenant, with how many rows | drained to R2, pruned per retention |
The delivery journal (chapter 5) is a fifth record, but it is an operational one. It says what happened to an event, not who did something.
Read the table by columns rather than rows, and each record covers a gap the others leave:
Each row holds actor, action, tenant_id, scope_id, vertical, before, after,
caused_by and at. When an executor wrote the row, caused_by is the event that set it off
(chapter 5), so an automated change links back to the domain event behind it.
Two properties are worth knowing exactly.
It is never swept. Grant provenance and bookkeeping-law retention both depend on it, so it has no TTL and no retention phase. Its growth is handled by bounding reads: the HTTP surface pages, and a cursor walks the whole thing. When a tenant is reaped, its scope data and personal directory rows are destroyed and its admin-log rows are kept. The record of what was done to a tenant has to outlive the tenant.
The row follows the mutation, in a separate write. An audited verb commits its change and then writes its row. A failed log write fails the verb, but the change already stands, and a crash between the two leaves a change with no row. That window is accepted and documented rather than hidden. Two verbs deliberately reverse the order, because for them the missing row is the harmful half-state: a point-in-time rewind records its intent before it destroys anything, and an impersonation session is logged before the session is handed back.
In the dashboard this is the Audit page. It is a team page rather than an app tab, because some entries, such as role changes and entitlements, name no app at all.
Every read the control plane makes into a scope on someone else's behalf writes a row. That covers the history, facet, cause, effect and same-call reads from chapter 10 and the denial summary. This includes the dashboard's own delegated reads. A tenant's admin opening a record in their own app is recorded, not just staff doing it.
The access log does not stay in the directory. The platform sweep's last phase ships unshipped rows to R2 as newline-delimited JSON, stamps them as drained, and only then prunes them from the directory. The order is the whole design. The sink must be durable before it returns, because everything downstream treats a returned ship as proof the evidence exists outside the directory. A sink that buffered and returned early would turn a retention policy into data loss. The R2 copies are pruned only if a retention period has been configured.
Chapter 2 made a trade: every scope is its own database, so no query crosses scopes. For operational reads that is the right trade. For history it is a real cost. "Every invoice basis exported across all tenants last quarter" should not be ten thousand reads and a fold.
So the platform keeps three tiers of data:
Tier 2 is the lake.
_substrat_outbox (each scope)
│ platform sweep, event-drain phase: read undrained → ship → stamp drained_at
▼
control-plane worker ── EventSink ──▶ Cloudflare Pipelines stream
│ INSERT INTO sink SELECT * FROM stream
▼
Iceberg table kernel.events, in R2 (Data Catalog)
Tracing each hop:
The drain is a sweep phase. It runs after the reaps, before the access-log drain, over every
active scope, up to 200 events per scope per pass. A scope with more reports itself incomplete
rather than looping, and the next pass continues. The control plane reads undrained rows over
the vertical's /internal surface, ships them, and stamps them drained.
Read, ship, stamp is at-least-once, on purpose. Stamping before shipping would lose events
on any crash in between. Stamping after means a crash re-ships, so the lake can hold the same
event twice. Every row carries its event id, which makes the duplicate recognisable: dedupe on
id before counting anything.
Draining copies, it does not move. The outbox is never pruned after a drain. The scope still
holds its full history for readHistory and every chapter 10 view. The lake is a second copy,
built for cross-scope questions.
The schema is generated. The lake's columns are derived from the outbox DDL by a tool, and
lint:lake-schema fails CI when the two disagree. So an envelope field added to the kernel
cannot silently fail to reach the lake. The pipeline's SQL is SELECT * for the same reason: a
record whose job is completeness should pass a new field through or fail loudly, not drop it
without a word.
id, type, schema_version, occurred_at,
tenant_id, scope_id, actor, entity_type, entity_id, pii_class,
subject_id?, payload?, authorization?, impersonation?,
operation?, version?, caused_by?, invocation_id?,
bytes
drained_at is not shipped, since it is a fact about the scope, not the event. bytes is added
by the shipper and is the row's serialized size. Chapter 12 explains why that column exists.
Files roll on size or time: parquet with zstd, at most 100 MB or 300 seconds, whichever comes first. At real outbox volumes the timer always wins, so history reaches the lake about five minutes after the drain ships it. The sweep itself runs every fifteen minutes. Freshness is Tier 1's job.
A lake table can need recreating, for example after a schema change. Because the outbox is never pruned, nothing is lost. Redrain clears the drained stamp on events stamped before a given instant, so the next sweeps ship them again. It is a staff-only, audited admin verb, and the script that drives it works one scope at a time in bounded batches.
Provisioning the lake is deliberately not automated in CI. The script that declares the
bucket, table, stream, sink and pipeline runs as the operator's own wrangler login, checks the
live configuration for drift, and refuses to recreate a table that holds snapshots unless the
operator names that exact table as the one whose history they are discarding.
The table is Iceberg in R2, so R2 SQL reads it directly. Here is the query the platform uses to measure per-tenant event volume, with duplicates removed first:
SELECT tenant_id, SUM(bytes) FROM (
SELECT DISTINCT tenant_id, id, bytes FROM kernel.events
) GROUP BY tenant_id
The same shape answers audit questions that no single scope can: every event a given actor produced across every tenant, every change made under impersonation in a month, or every call from a version later found to be faulty.
Here is what is not built, stated where you would otherwise assume it was:
subject_id
alongside. That column exists so "delete every row for this subject" can be expressed as a
query, but nothing runs it yet. Erasure is handled in Tier 1. Until lake erasure is built,
an erased subject's payloads survive in Tier 2, and a deployment that drains to a lake has
to account for that.A tenant's data can be exported in full: every scope's dump, plus directory rows and, in unmasked mode, the tenant's admin-log entries. That is the portability export. It is a staff route today, not a self-service one.
The denial log has a drained_at column and no drain. Denials stay in their scope. They are
readable per scope, summarized by operation in the dashboard's Flow view (chapter 10), and
listed in the console's per-scope panel, but they do not reach the lake. A fleet-wide
"what was refused" question is still a fold over scopes.
Two different subjects share these words, and mixing them up leads to wrong designs in both directions:
They share one principle, and it is the thread through this chapter: count exactly, price separately, and never let a price change a count.
The metering engine records how much. It has no currency, no rate, no plan, and no idea what anything costs. That line is drawn deliberately. One customer rounds differently, another offers a volume discount, a third changes price mid-month. None of those decisions can be wrong for every vertical, so by chapter 7's test none of them belongs in an engine. What would be wrong for every vertical is losing a unit or counting one twice, so that is what the engine owns.
A meter has a key, a unit, and a kind:
A meter's kind and unit are frozen once set. A counter that later became a gauge would reinterpret every entry already written.
// inside the vertical's own operation, in the same transaction as the work
recordUsage(ctx, {
meter: 'ai.tokens.output',
qty: String(outputTokens),
subject: { entityType: 'conversation', entityId: conversationId },
dedupeKey: `${turnId}:out`,
});
The engine is composed by call (chapter 7), and that choice matters here more than anywhere. The usage entry commits in the same transaction as the work it measures. There is no gap in which an answer was delivered and its tokens were not recorded, or the tokens were recorded for an answer that rolled back.
What the engine refuses:
closePeriod totals every meter over a window and freezes the result as period lines.
Closes are monotonic: a new period may not overlap one already closed. It emits
metering.period-closed with the lines in the payload, fat as chapter 5 requires, so a
consumer never has to read back. Nothing schedules a close. When a period ends is a business
decision (calendar month, contract anniversary, on demand), so it belongs to the vertical.
The vertical holds the rate card, in its own table, with its own permission. The support-desk reference vertical stores rates per meter with an effective date, and prices usage for display by joining each entry to the rate in force when it was recorded. A rate change next month does not reprice last month.
Invoicing is the other half, and it is composed by event. A vertical or engine emits a billable event, and the invoicing engine consumes it into the customer's single open billing basis: the set of lines a real invoice will be issued from. Today it consumes three events: a completed work order, a placed commerce order paid by invoice, and a closed timesheet period.
Each line copies its price from the event payload at write time, so last month's work is never billed at today's price. A basis holds one currency, checked on every write. Once exported it is immutable, which is why invoicing has no in-scope exports (chapter 5): nothing may call in halfway through and change a basis the accounts already hold.
Two gaps are worth stating plainly:
metering.period-closed.
The intended shape is for the vertical to price the closed lines with its own rate card and
emit them as a billable event. On the invoicing side that is an additive new consumed event,
designed and not built.The platform's rule is count everything exactly, store no bill. Usage is recorded as facts. Prices and margins are applied when someone reads, and nothing stores a computed charge. So a pricing change never requires a data migration, and a count is never tangled with a number that was a commercial decision.
The commercial design names four meters. Here is the state of each:
| Meter | Counts | State |
|---|---|---|
| 1. Base fee | active tenants and active scopes | counted, live |
| 2. Engine licensing | entitlements, grouped by key and plan, with expired ones counted apart | counted, live |
| 3. Usage | platform-provided model calls, with tokens and list price | counted, and priced when read |
| event history retained | measurable in the lake, and nothing reads it yet | |
| storage, API calls | not counted | |
| 4. Network transactions | orders flowing between tenants | not countable, because that flow does not exist |
Meters 1 and 2 are folds over the directory, computed on read and never stored. Operators see them, and model usage, in the console's Meters view, with a per-tenant card on each tenant.
An entitlement is a row naming a tenant and a key, optionally with a plan, a quota and an
expiry. The kernel enforces presence and expiry: a module whose key the tenant does not
hold is not registered, so its operations do not resolve (chapter 3). plan and quota are
expression only. A module can read them through ctx.entitlement(key) and act on them,
but nothing in the platform enforces a quota.
Platform-provided models are the one part of usage that is complete from call to price:
Two numbers exist, and both are correct. The platform's list price plus margin is what the platform charges the tenant. The vertical's own rate card is what the tenant charges its customers. Reconciling them is the tenant's business.
One known hole: a vertical that holds the models binding can call it directly, outside the model host, and that call is not metered.
Nothing measures storage used by a scope or a tenant today.
Here is what exists nearby, and why none of it is the answer:
bytes column (chapter 11) records every event's serialized size as shipped.
Summed per tenant after dedupe, it is exact, and it is the only per-tenant volume measure the
platform has. But it measures event history volume, not database size. It excludes a
scope's current rows, its indexes, its attachments, and every row an update overwrote. It is a
defensible basis for billing history retention, and it is not a storage meter.So a storage line on an invoice needs one of two things built: a sweep phase that collects each scope's database size into the directory next to the other meters, or a decision that event-history bytes are the unit being sold.
The router's per-request datapoints (chapter 10) are sampled. They are good enough to draw a chart and not good enough to put on an invoice. And a read emits no event, so the spine cannot count API calls either. That is why "API calls" sits uncounted next to storage.
The dashboard has a Billing page, and today it is a placeholder, marked as not yet enabled. No payment provider is integrated, no invoice is generated, and nothing takes payment. The platform counts, and the operator bills from those counts outside the product.
In both halves the count is exact, deduplicated, and written in the same transaction or intent as the thing it counts. The price is applied later, by whoever owns the commercial decision: the vertical for its customers, the platform for its tenants. What is missing is also the same in both halves: the hand-off from a closed count to a document someone pays.
The last twelve chapters were how the system works and what it records. This one is how it behaves when something is wrong, which is a different subject.
The pieces that turn "a vertical" into "a vertical serving a customer at a hostname" are four separate deployments.
| Surface | Audience | Answers |
|---|---|---|
| Control plane | the platform | the shared directory every vertical registers against — tenants, scopes, roles, entitlements, the admin log |
| Console | Substrat operators | run the platform — the whole fleet, every tenant, provisioning, the audit log |
| Router | inbound traffic | hostname → (tenant, scope, surface), then dispatch |
| Dashboard | a customer's admin | run my org — their tenant only |
The split that matters most is Console versus Dashboard: the same platform, opposite audience and opposite blast radius. The Console is the operator's back office — all tenants, staff SSO. The Dashboard is the customer's home — one tenant, customer sign-up. Neither is called a "portal", which is reserved for a vertical's own end-user surface.
The Dashboard is itself a Substrat vertical, which is the useful kind of dogfooding: the tenancy model either works for the platform's own product or it does not.
Its shape is worth knowing before you go looking for something in it. Each app has four tabs: Overview, Deployments, Data and Settings. Anything about more than one app is a team page in the left menu, narrowed with an app filter: Observability (chapter 10) and Audit (chapter 11) sit under Operate, next to Domains, Integrations, Team and Billing under Configure. The rule is deliberate: a fifth tab on the app is almost always a filter on a team page instead.
Three more deployments sit beside those four:
Chapter 2 listed them. Here is what each one means when you are the person deciding.
| State | Serves traffic | Data | Reversible |
|---|---|---|---|
provisioning |
no | being created | — |
active |
yes | live | — |
suspended |
no, fails closed | intact | yes |
archiving → archived |
no | intact | yes |
reaped |
no | gone | no |
Tenants have the parallel ladder, where deleting is the reversible grace state that makes
every scope beneath it inert without reclaiming anything.
Two things to internalise:
Suspension is the live weapon. It takes effect on the next request, because the router
does not cache routes and getScope re-checks the directory. When a customer must stop being
served now, this is the lever, and its immediacy is paid for with a directory read per
request.
archived is not deleted. The bytes are still there and cost money. Moving to reaped
is what frees them, it is irreversible, and — per chapter 9 — nothing does it automatically
unless a retention window has been configured. If your mental model is "archiving cleans up",
it does not.
Backup is where people's mental models are usually wrong, because "backup" is four different things here.
| What is lost | What covers it |
|---|---|
| A scope's data, wrongly changed | Durable Object PITR — ~30 days, continuous, per scope. A destructive rewind of the live app. |
| A scope, reaped | The copy the reap takes first. A reap is irreversible, so the control plane stores a full-fidelity dump before any byte goes and records its address on the admin-log entry. |
| A scope, wanted elsewhere | scope pull / snapshots — a non-destructive copy that leaves the live app alone. |
| The directory itself | The scheduled directory backup. |
That last row is the one the others cannot cover, and it is worth understanding why.
The directory is a single Durable Object. Lose it and no scope can be found, even though every scope's data is sitting there completely intact. PITR does not help: PITR rewinds a database that still exists, and having nothing to point it at is a different failure. And no scope can rebuild the map from below, because a scope does not know its own tenancy, hostname, or bound version.
So the control plane backs itself up: a scheduled dump of the whole directory — tenants, scopes, hostnames, verticals, entitlements, identities, and the audit log — to an object store outside the DO, once a day, keeping 30 copies.
Two details in that design are worth borrowing for anything similar. The cadence is derived by reading the newest stored copy rather than from a separate timer, so a missed run is caught up on the next pass rather than skipped. And retention is pruned only after a new copy lands, so a failed backup can never be the thing that deletes the last good one.
RPO ≤ 24h, RTO ≤ 1h for a directory loss, and the restore is rehearsed rather than merely designed — the round trip runs in the contract suite against both adapters: capture, diverge, restore, keep serving.
Restoring replaces. The dump's contents become the directory; anything created since is gone. A merge would silently interleave two histories of the same tenant, so replace is the honest semantic, and the restore refuses a directory that still holds tenants unless the request explicitly says to overwrite — a guard for the realistic hazard of a restore replayed against a control plane that has already recovered.
What it does not cover, stated plainly: the backup bucket lives in the platform's own account. This survives losing the directory; it does not survive losing the account. And a restore does not bring back what was never in the directory — the staff roster, worker secrets, or the key any sealed credential was sealed with.
Chapters 10 and 11 describe the instruments. What matters when something is wrong is which one witnessed the thing you are asking about, because reaching for the wrong one is the usual reason an investigation stalls:
| Question | Instrument | In the dashboard |
|---|---|---|
| Is it slow, or failing, for everyone? | router traffic (sampled) | Observability → Traffic, Health |
| What did this request do, and what did it print? | invocation log | Observability → Logs |
| What happened to this record, and who did it? | the outbox, via readHistory |
the app's Data tab → a record |
| Why does this event exist? | causedBy |
a record's event → Why? |
| Did the consumer ever handle it? | the delivery journal | a record's event → What did it do? |
| Everything this request changed | invocation_id |
a record's event → Same call |
| Has this job stopped running? | sweep runs, freshness | Observability → Schedules |
| Why was this person refused? | the denial log | Observability → Flow (per-operation health) |
| Who changed this app's configuration? | the admin log | Audit |
| Who looked at this tenant's data? | the access log | no view yet: it drains to R2 (chapter 11) |
| Did the platform act on what the app asked for? | the platform-request journal | Settings → Integrations |
Add to those the sweep reports from chapter 9 (retrying, deadLettered, and per-unit
errors), which are the signal that background work is degrading rather than failing loudly.
Symptoms whose cause is rarely where you first look:
"The deploy went green and production is unchanged." A push is not a deploy. Check whether the version was promoted, and whether the scope is bound to it.
"One tenant is broken, everyone else is fine." Very likely a failed migration on that
scope. It fails closed and serves nothing by design. The migration-reconciliation phase records
why; the scope's migrationFailure carries the version and the error.
"The event fired but nothing happened." Open the event's What did it do? tree, which is
the delivery journal. If the consumer is
in-scope, remember from chapter 5 that it does not retry — one failure is terminal for that
pair, and waiting will not fix it. If it is an executor, check next_attempt_at: it may be
backing off, or already dead-lettered at maxAttempts.
"The app asked for something and it took a quarter of an hour." The platform intent waited
for the sweep. The vertical is not flagging its responses, so the router never kicked a drain.
Wire onPlatformRequests when minting the stub (chapter 5). If the intent never settles at
all, find it in the platform-request journal (Settings → Integrations for a connector's
intents). A handler that keeps throwing leaves the intent pending for about a day and then fails
it, with the last error kept.
"Observability shows no traffic for an app that is clearly in use." Either the version
predates the invocation log, or the log is mounted below some routes, or it has no
routerSecret and verifies nothing. All three produce silence rather than an error (chapter 10).
"Recurring work stopped." Check the scope sweeper's roster. The alarm lapses on an empty
roster, and the roster is maintained by /internal/provision and /internal/delete-scope
rather than read from a directory. A scope that never called noteScope is not swept.
"Permission denied and I do not know why." The denial log gives the key and the node.
explain gives the chain. If the permission is tenant-level, remember it reaches the scope by
projection — and an absent projection is a deny, which is the safe direction but also a
real cause.
"It works locally and fails deployed." The adapters are contract-equivalent, so suspect the things that are not module code: the router secret, a binding, an unset platform secret, a grant that was never made. Both trust boundaries fail closed, which means the symptom of a missing secret is a blanket refusal rather than a subtle misbehaviour.
Ending on the gaps, because a book that implies completeness is worse than one that names the edges.
eu and us name guarantees whose enforcement is
not there yet, which is why provisioning gates them. global is what every scope is today.What Substrat doesn't have (yet) is the maintained version of this list.
You now have the shape. The reference is the right tool from here:
And if you are an agent rather than a person: Agent rules is the page to read before writing any code, and llms.txt is the index.