# Substrat, end to end

The Substrat book, in one file. Published at https://substrat.net/book/ — this edition is
generated from those chapters at build time, so it cannot fall behind them.

This is the book. The rest of these docs are a reference — you arrive with a question,
find the page that answers it, and leave. That works well once you know the shape of the
system, and badly before. Nothing in a reference tells you how the pieces *join*: which
thing calls which, what happens between the moment a request lands and the moment a row
is written, who retries what when it fails.

So this section is the other thing. Thirteen chapters, meant to be read in order, front to
back, in a long afternoon. It assumes nothing except that you have written server software
before. It repeats very little — where a reference page already says a thing well, this
book links to it and moves on.

## What you will know at the end

- What the words mean: tenant, scope, module, engine, vertical, the spine, consumer, executor,
  connector, platform intent.
- What a scope is, why every one of them is its own database, and what that buys.
- The complete path of one HTTP request, from hostname to SQL and back, with every hop
  named.
- What a handler can reach, what it cannot, and why each ban exists. Also why plain SQL with
  no query builder is safe enough, why timestamps are text, and what a slow `await` blocks.
- How an event gets from `ctx.emit` to a consumer in another module, what happens when
  the consumer throws, and which failures retry and which do not. Also how a platform intent
  gets run in seconds instead of at the next sweep.
- How permission checks are answered without a network call.
- Why engines never call each other, and what they do instead.
- What `substrat push` actually does, and what happens to data across a version change.
- What the two background clocks are, what each sweeps, and why both are alarms rather
  than crons.
- What the platform records about a running app, from sampled traffic to exact event history,
  and how a record joins to the request that changed it.
- Which log is the audit log for which question, and how events leave their scope for a lake
  you can query with SQL.
- How a vertical meters and bills its own customers, how the platform meters a tenant, and
  what is not counted yet.
- What breaks in production, where you look, and what the lifecycle states mean.

## How to read it

Straight through. Each chapter ends where the next begins, and later chapters lean on
earlier ones — chapter 5 assumes chapter 4's picture of a handler, chapter 9 assumes
chapter 5's outbox, and chapters 10 to 12 assume both. If a word stops you, the glossary is
at the end of chapter 1. The **Next** link at the foot of every page is the intended path.

Two chapters are worth reading even out of order, because they are the ones the
reference genuinely does not cover anywhere: [The life of one
event](https://substrat.net/book/05-one-event.md) and [The two clocks](https://substrat.net/book/09-the-two-clocks.md).

## Read it in one file

The whole book is also published as a single document, three ways:

- **[book.epub](https://substrat.net/book.epub)** — a real EPUB 3, with a cover and a table of contents. The
  link downloads the file; open it from there in Apple Books, Kobo, or anything else that
  reads EPUB, and it remembers where you got to and lets you set the type the way you like
  it. On a phone that is one step more than tapping a link — see below.
- **[book.txt](https://substrat.net/book.txt)** — every chapter concatenated, plain markdown. Good for
  printing, for `pandoc`, or for handing to a model in one shot.
- **[book/read.html](https://substrat.net/book/read.html)** — the same thing as one
  scrolling, printable web page.

All three are generated from these chapters at build time, so they cannot fall behind.

> **Tip — Reading it on a phone**
>
> There is no URL that opens Books directly — iOS decides that from the file itself. Tapping
> **book.epub** in Safari downloads it; opening it from Files then offers Books, which syncs
> it to your other devices. On a Mac, double-clicking the download does the same thing in one
> step.

If you want the *reference* in one file instead — every page on this site, not just the
book — that is [llms-full.txt](https://substrat.net/llms-full.txt), and the index agents read first is
[llms.txt](https://substrat.net/llms.txt).

## What changed in this edition

The book first shipped with ten chapters. This edition adds three:
[Seeing what happened](https://substrat.net/book/10-seeing-what-happened.md),
[The audit trail and the lake](https://substrat.net/book/11-audit-and-the-lake.md) and
[Metering and billing](https://substrat.net/book/12-metering-and-billing.md). *Operating it* is now chapter 13.

It also answers questions readers of the first edition asked. Chapter 1 now has a glossary.
Chapter 4 explains the absence of a query builder, the text timestamps, and what an `await`
holds. Chapter 5 explains the kick that makes platform intents fast. And the chapters now
reflect what shipped since: invocation ids and the event walks, the MCP surface, the lake drain,
the dashboard's four tabs, and the new reference verticals.

## A note on honesty

Substrat is 0.x. Several things in this book are described as built because they are,
and a few are described as not built because they are not. Where a mechanism has a known
gap, the chapter says so in the place you would otherwise assume it was covered, rather
than in a footnote. [What Substrat doesn't have (yet)](https://substrat.net/guide/what-substrat-lacks.md) is the
concentrated version of that.

## 1. Why a substrate at all

Every piece of business software for a specific trade — a field-service app, a workshop
system, a clinic's booking tool — is mostly the same software. Not in the part anyone
pays for, which is the vocabulary and the screens and the pricing. In the part underneath:
tenancy, roles, permission checks, an audit trail, migrations, deploys, backups, an
invite flow, a way to run a copy of production without touching production.

That underneath is perhaps 80% of the code and nearly all of the risk. It is also the
part that is wrong in almost every such system, in the same few ways, and the ways are
not subtle:

- **A missing `WHERE tenant_id = ?`.** One forgotten clause, and a query reads another
  customer's rows. The type checker cannot see it, the tests do not have two tenants in
  them, and the failure is silent until it is a disclosure.
- **A permission check that was never written.** The handler that lists is guarded; the
  handler that exports is not, because it was added in a hurry six months later.
- **An audit trail that is a `log.info` call.** It is complete exactly as long as
  everyone remembers to write one, which is to say it is not complete.
- **State machines enforced by convention.** A row goes from `draft` to `invoiced`
  because every code path happens to move it that way — until one does not.

Each of these is an instance of the same thing: an invariant the system *depends on*,
enforced by a person remembering. The remedy is not better discipline. It is moving the
invariant somewhere discipline is not required.

### The bet

Substrat's bet is that this substrate is worth building once, properly, and that
"properly" means **the runtime refuses**, not that the docs advise.

So: a kernel owns tenancy, permissions, events, migrations and the transaction boundary,
and it owns them in a way module code cannot opt out of. There is no `WHERE tenant_id`
to forget, because a handler's `ctx.sql` is already inside one tenant's database and has
no syntax for reaching another. There is no un-audited mutation, because the event is
written in the same transaction as the row and the envelope is stamped by the kernel, not
supplied by the caller. There is no unchecked operation, because a check that was never
called leaves a trail of its own.

That is the claim [Why runtime enforcement?](https://substrat.net/guide/why-substrat.md) argues at length. This
book takes it as given and shows the machinery.

### The three layers

**Diagram — the three-layer stack.** Six bands, top to bottom: verticals, the line, engines, the kernel, the adapters under it, and connectors at the edge.

- **Verticals** — Everything a user touches — the businesses themselves. *(own vocabulary · screens · pricing · roles)*
  - `Callout` — field service
  - `Handlebar` — bike workshop
  - `Kallkälla` — coffee shop
  - `Meridian` — HR
  - `Manyfold` — headless CMS
- **The line.** Above: AI velocity — mistakes are cosmetic (a wrong screen). Below: humans + runtime guarantees — mistakes are catastrophic (a tenant leak).
  - Verticals meet engines by: composes engines in-scope, same transaction.
- **Engines** — Headless domain machinery that owns invariants. Star topology — they talk to the kernel, never to each other. *(own invariants · versioned · never forked)*
  - `workorder` — one state machine · append-only time + material
  - `booking` — resource × interval × capacity · one allocation, no locks
  - `invoicing` — consumes billable events · immutable once exported
  - `protocol` — checklists + signed docs · freeze → immutable, hashed
  - `invites` — hashed identifier · accept-required · non-enumerable
  - Engines meet the kernel by: ↓ ctx (sql · check · emit · link) · events + audit ↑.
- **Kernel** — The substrate. Everything true of every B2B SaaS — and nothing true of any one. *(owns no domain entities)*
  - Provides: Identity, Nested tenancy, Permissions + grants, Events / audit spine, Migrations, GDPR machinery, Notifications, Jobs, Billing entitlements, Module system, Attachment contracts, App shell.
  - Every operation runs inside: `ctx.sql`, `ctx.check`, `ctx.emit`, `ctx.link`.
  - No customer table, no work-order table. It offers attachment contracts that bind to opaque (entityType, entityId) refs the vertical defines.
  - Kernel meets the adapters by: same kernel semantics on any ground — one contract-test suite gates them all.
- **Adapters** — Scope hosts — the interchangeable ground the kernel is seated on. *(swappable · escrowable · self-hostable)*
  - `adapter-sqlite` — dev · CI · self-host / escrow
  - `adapter-cloudflare` — production · Durable-Object per scope
  - `adapter-email` — notification transport · CF Email + mock
- **Connectors** — The outside world, at the edges. React to events on the spine — host code, never module code. *(no fetch inside a module — ever)*
  - `Scrive eSign` — signatures-requested → BankID signing → recorded back
  - `…more` — one port per capability

**The four rules that hold it together**

1. **Kernel owns no domain entities.** It provides the spine; verticals define what the entities mean.
2. **Star topology.** Engines cooperate through fat events and opaque refs — never by importing each other. N contracts, not N².
3. **Enforced at runtime.** Guarantees are defaults of the substrate, not config a builder — human or AI — can get wrong.
4. **No forking.** If a vertical ever needs to fork an engine, the engine drew its line wrong.

Read it bottom-up.

**The kernel** owns what must never be wrong. It provides an `OperationContext` — `sql`,
`emit`, `check`, `link`, `grant`, `now` — and nothing else. It knows no domain entities
at all: it has never heard of a work order or an invoice, and could not be made to
special-case one.

**Engines** own invariants inside a domain. The work-order engine knows that a completed
order cannot go back to scheduled, that every state change emits an event, that an
exported billing basis is immutable afterwards. There are seven of them, each headless:
no screens, no vocabulary, no opinion about what you call things.

**Verticals** own everything a user touches: the words, the roles, the prices, the
screens, the workflow. A vertical composes engines the way an application composes
libraries, except that the library's invariants are enforced from below rather than
trusted.

The line between the bottom two bands and the top one is the interesting one, and it is
drawn where it is on purpose. Below it, change is slow, reviewed, and versioned, because
being wrong there is expensive. Above it, change is fast — a vertical is exactly the kind
of code an agent or a small team can write quickly, because the expensive mistakes are
not available to make. You cannot skip a state. You cannot forget a check and have
nothing notice. You cannot read another tenant.

That is the actual product: not "AI writes your app", but *a place where AI writing your
app is not reckless*. [Where AI mistakes stop](https://substrat.net/guide/ai-guardrails.md) is that argument
with the specific failure modes named.

### What this costs you

It is worth being blunt about the shape of the trade, because it is a real one.

**You give up arbitrary data access.** A handler reads and writes through `ctx.sql`,
inside one scope, and nothing else. No connection pool, no second database, no `fetch` to
a sibling service. If your design needs a handler to read across tenants, the design has
to change; the runtime will not bend.

**You give up cross-module joins.** Another module's tables are private — not "please
don't", but a linted, reviewable boundary. A vertical that wants extra fields on an
engine's entity keeps its own side table keyed by the engine's id, and never a column
upstream.

**You give up synchronous everything.** Effects that are not scope-local — sending mail,
calling a provider, writing tenant-wide directory state — do not happen inline inside
your transaction. They are asked for, and effected out of band. Chapter 5 is largely
about this.

In exchange, a class of bug becomes structurally unavailable rather than merely rare, and
you get the operational half — audit, snapshots, previews, migrations, backout — as
properties of the substrate rather than as projects.

### What is not in the box

Substrat is not a BaaS, not an ORM, and not a low-code builder. It does not generate your
UI, though it will generate your API client. It has opinions about tenancy and none about
CSS.

It is also not finished. [What Substrat doesn't have (yet)](https://substrat.net/guide/what-substrat-lacks.md)
is the current list, and this book flags the relevant gaps where they arise rather than
saving them up.

### The words this book uses

A handful of terms carry most of the weight from here on. Each one gets its full chapter later,
but none of them should be used before it is defined, so here they are in one place.

**Tenant.** The business that pays. It is the billing and identity boundary.

**Scope.** One isolation domain inside a tenant, such as a branch, a client or a property. Each
scope is its own database (chapter 2). An installed app is a scope running a vertical.

**Module.** The unit of code the kernel loads into a scope. A module is one registration: a
**manifest** (its id, the permission keys it declares, the events it emits and consumes, its
schedules), plus its SQL migrations, its operation handlers, their input schemas, and its event
consumers. **Every engine is a module, and so is every vertical.** A deployed app registers its
own vertical module next to the engine modules it composes, in a fixed order, because that order
is also the order migrations run in.

**Operation.** A named, permission-checked, transactional entry point into a module, invoked as
`module/name`, such as `workorder/create`. The only way anything changes a scope's data.

**Engine.** A module that owns invariants in one domain, such as work orders, invoicing or
booking, and has no screens and no vocabulary.

**Vertical.** A module that owns everything a user touches in one trade: the words, roles,
prices and workflow. It composes engines. It also ships the worker around the module: HTTP
routes, authentication and the app.

**Harness.** Code in the vertical's worker that is *not* module code: the server, the seed, the
routes, the tests. The rules in chapter 4 bind module code, and harness code is where the
exempt work, such as calling a model, happens.

**The spine.** The kernel's own tables, all named `_substrat_*`, inside every scope and in the
directory: the event outbox, the delivery journal, the denial log, the permission tuples, the
platform-intent queue, the admin log. Module code may read the spine and may never write it.
It is the record the rest of the system trusts.

**Event.** A kernel-stamped fact that a mutation happened, written to the **outbox** in the same
transaction as the mutation (chapter 5).

**Consumer.** A module's handler for another module's event. It runs inside the same scope, in
its own transaction, as a system actor.

**Executor.** A handler the *host* registers for an event, which runs outside the scope with
platform authority, for effects one scope cannot make on its own. **Connector** is the executor
you will meet most: one that is also given a credential and permission to call exactly one
outside provider.

**Platform intent.** A request, written by an operation into the spine, for the platform to do
something privileged on the scope's behalf, such as provisioning another scope. The platform
executes it later and writes the outcome back.

**Control plane, directory.** The platform's own service, and the database it keeps: which
tenants and scopes exist, where they route, who holds which role, and the admin log.

### Two pictures, one system

There are two ways to look at what follows, and you need both.

The layer stack above is *what the code is* — which package owns which concern, which
direction dependencies point. It is the picture you need when deciding where a change
belongs.

The other picture is *how it runs*: a request arriving at a hostname, resolving to a
tenant and a scope, reaching a database that holds that scope's data and no one else's.
That is the next chapter, and it is the one that makes the rest of the book legible.

---

## 2. Tenants, scopes, and one database each

Everything in Substrat hangs off one structural decision, and it is the one to understand
first, because every later mechanism is shaped by it.

### Two levels, tree-shaped

A **tenant** is the business that pays you. A **scope** is one isolation domain inside
that business.

**Diagram — a tenant and its scopes.**

- **Tenant** (billing · identity · the contract) — slug · status · one Identity DO for all its people. Holds no operational rows of its own.
- It contains scopes: `stockholm` (branch, active · eu), `göteborg` (branch, active · eu), `malmö` (branch, provisioning).
- Every scope is: its own database — one SQLite file, or one Durable Object; one operation at a time, run to completion; kind is your vocabulary: brf, branch, brand, clinic…; jurisdiction and bound version are fixed per scope.
- **No query crosses these lines. There is no join between two scopes.**
- parentScopeId is null on every scope today. The column exists so a deeper tree is an addition rather than a migration.

The tenant is who you bill and who can log in. The scope is where data lives and where consistency is decided — which is why isolation is a property of the substrate here, not a WHERE clause somebody has to remember.

What a scope *means* is your vocabulary, not the kernel's. It is a `kind` string the
kernel never branches on: a housing association for a property manager, a branch for a
chain, a client company for an agency, a brand for a publisher. Users belong to the
tenant, to a scope, or to several scopes with different roles in each.

This shape is in nearly every vertical B2B product, and it is close to impossible to
retrofit — which is why it is kernel-owned rather than a convention you follow. Two
details are worth knowing early:

- **Scope IDs are globally unique ULIDs**, not per-tenant counters. An event or an opaque
  reference never needs the tenant to disambiguate. But every kernel API still takes the
  pair `(tenantId, scopeId)` and cross-checks it — so a confused-deputy bug in calling
  code **fails closed** rather than resolving to somebody else's scope.
- **`parentScopeId` exists and is always `null` today.** The column is there so deeper
  trees are an additive change later rather than a data migration.

The full entity definitions are in [Tenants & scopes](https://substrat.net/concepts/tenancy.md).

### One scope, one database

Here is the part that matters. A scope is not a partition key. It is **its own database**.

On the local adapter, that is one SQLite file. On the hosted runtime, it is one
SQLite-backed Durable Object. There is no shared cluster and no shared table with a
`tenant_id` column in it. A query issued inside a scope is issued against storage that
contains that scope's rows and nothing else.

**Diagram — the topology, adapter-neutral.** Vertical code reaches data one way, and every scope feeds one spine.

1. **Vertical** (your code, TypeScript · often AI-built) reaches the host by **@substrat-run/kernel API only**.
2. **Scope host** (kernel) — `getScope(principal, tenant, scope)` returns a capability stub. holding a stub is the authorization.
3. Each scope is its own database + ACL, serialized — one op at a time — drawn here as Scope · branch #1 and Scope · branch #240, but there are as many as the tenant has.
4. Every scope emits **events, kernel-stamped** into the **Event spine** (kernel-owned): audit · reporting · integrations.

The same shape on every adapter — one SQLite file per scope locally, one Durable Object per scope in production. Nothing above the stub can widen its own reach.

Follow the consequences:

**Cross-tenant reads are not prevented, they are unavailable.** There is no `WHERE
tenant_id = ?` to forget, because the other tenant's rows are not in the database you are
querying. The most common multi-tenancy bug in the industry has no syntax here.

**Blast radius is one scope.** A corrupted row, a runaway query, a migration that throws
— each is contained to one customer's one domain. There is no lock contention between
tenants because there is no shared lock.

**"Give me a copy of production" is a file copy.** Snapshots, forks and preview
environments (chapter 8) are cheap because a scope's entire state is one self-contained
thing. On a shared cluster this is a export-filter-import project; here it is a copy.

**Deleting a customer is deleting databases.** Reaping one scope destroys one file;
reaping a tenant walks every scope beneath it and then clears the tenant's own PII and
configuration rows. Which is a real property when somebody invokes a right to erasure, and
a real hazard too — chapter 13 covers what actually frees those bytes, because the answer
is less automatic than you would expect, and because the tombstones and the admin log are
built to survive it.

The cost is equally real: **you cannot join across scopes**. A question like "how many
work orders did this tenant complete this quarter, across all forty branches" is forty
reads and a fold, not one `GROUP BY`. [Reads & scaling](https://substrat.net/concepts/reads.md) is the page on
how that is handled — projections, per-tenant D1, and what is honestly still open.

### Strict serialization

Inside one scope, **one operation runs at a time, to completion**, before the next one
starts.

This is a stronger guarantee than most databases give you and it eliminates a whole
category of code. No interleaved read-modify-write. No lost update. No row locking in
module code, no `SELECT … FOR UPDATE`, no optimistic-retry loop. A handler that reads a
balance, decides, and writes it back is correct as written, because nothing ran in
between.

It is enforced explicitly, on both adapters, by a per-scope task queue — about fifteen
lines of code:

```ts
export class OperationQueue {
  private tail: Promise<unknown> = Promise.resolve();

  enqueue<T>(op: () => Promise<T> | T): Promise<T> {
    const result = this.tail.then(op);
    // The chain must survive failures; callers still see the rejection.
    this.tail = result.catch(() => undefined);
    return result;
  }
}
```

On Cloudflare this queue sits *in front of* the Durable Object's own input gate rather
than relying on it. The gate is real, but it over-delivers: it permits subtler
interleavings around non-storage awaits. Kernel and module code are allowed to depend on
strict serialization and nothing weaker, so the adapter enforces strict serialization
itself instead of inheriting whatever the platform happens to provide. The two adapters
then mean the same thing by "serialized", which is what lets one conformance suite hold
both.

Serialization is **per scope**. Two operations on the same scope queue. Two operations on
different scopes, whether in the same tenant or not, run at the same moment on different
databases, and neither can slow the other. Chapter 4 covers what a slow operation holds, and how
a vertical keeps genuinely slow work out of the queue.

The thing to notice is the pattern, because it repeats: where the platform's guarantee is
*close* to the contract, Substrat implements the contract anyway rather than documenting
the gap.

### The clone boundary

Inputs and results cross the scope boundary as **structured clones** — even when the
caller and the scope are in the same process.

So module code can never hold a reference to something outside its scope, and outside
code can never hold a reference into it. There is no object shared across the boundary to
mutate by accident. Locally this is deliberate extra work; on Cloudflare it is free,
because the boundary is a real RPC hop. Doing it locally too is what stops "works on my
machine, fails in production" from being a category of bug — the local runtime is the
strict one.

### Ambient tenancy

Once you hold a scope stub, you never pass a tenant or scope id again.

```ts
const stub = await host.getScope(principal, tenantId, scopeId);
await stub.invoke('workorder/create', { title: 'Leaking radiator' });
```

The handler behind `workorder/create` receives a context already bound to that tenant and
that scope. It has no parameter for them, so there is no parameter to get wrong, and no
call site where the right ids could be passed to the wrong operation.

Holding the stub *is* the authorization to talk to that scope — it is a capability, and
minting one is gated (chapter 3). The scope still re-validates every call against its own
access rules, because a capability that is never re-checked is a capability that cannot
be revoked.

### Where a scope lives is decided once

Two fields are fixed at provisioning and never change afterwards:

**`jurisdiction`** — `eu`, `us`, or `global`. Data residency is a property of the scope
from birth. `global` is the honest name for unconstrained, which is what every scope is
today; `eu` and `us` name guarantees whose enforcement (Durable Object jurisdiction
subnamespaces, Regional Services) is not built yet, so provisioning currently gates them.
The vocabulary ships ahead of the enforcement deliberately, precisely *because* the field
is immutable: a scope that took a value nobody could name later would be a data migration
rather than a config change.

**`storageShape`** — `A` (the DO's embedded SQLite is primary) for everything today; `B`
(a control-plane DO fronting per-tenant D1) is designed and not built.

A Durable Object cannot relocate. That is not a Substrat limitation but a platform one,
and it is why these are provisioning-time decisions rather than settings.

### The states a scope moves through

```
provisioning → active ⇄ suspended
                  ↓
              archiving → archived → reaped
```

`active` serves traffic. `suspended` fails closed — every `getScope` refuses — but the
data is intact and the transition is reversible. `archived` is inert and reversible; the
bytes are still there. **`reaped` is terminal**: the storage has been wiped, the
directory row survives as a tombstone for audit history and to burn the slug, and there
is no restore.

Tenants have a parallel ladder — `active`, `suspended`, `deleting`, `reaped` — where
`deleting` is a reversible grace state that makes every scope under it inert without
reclaiming anything.

What is worth flagging here, and what chapter 13 returns to: **nothing moves a scope from
`archived` to `reaped` on its own unless you have configured it to**. Cloudflare never
garbage-collects a Durable Object. An archived scope's bytes persist indefinitely until
something explicitly wipes them, and the sweep that can do that is opt-in because the
operation is irreversible.

---

## 3. The path of one request

Someone opens a browser, types a hostname, and clicks a button that creates a work order.
This chapter is every hop between that click and the committed row, in order.

It is the chapter the reference does not have. [Router](https://substrat.net/platform/router.md) stops at the
service binding. [`@substrat-run/vertical-host`](https://substrat.net/reference/vertical-host.md) starts at
`/internal/*`. [Operations & the scope host](https://substrat.net/concepts/scope-host.md) starts at `invoke()`.
Each describes its own hop well and none of them describes the joins, so here they are.

```
browser
  │  https://acme-north.example.com/api/invoke
  ▼
[1] router worker            hostname → (tenant, scope, vertical, surface)
  │  x-substrat-tenant / -scope / -surface / -vertical / -router
  ▼
[2] dispatch                 env.DISPATCH.get(deploymentRef)
  │
  ▼
[3] the vertical's worker    readRoutedNode → authenticate → principal
  │
  ▼
[4] CloudflareScopeHost      lifecycle gates → lazy migrate → mint stub
  │  stub.invoke('workorder/create', input)
  ▼
[5] ScopeDO                  queue → transaction → parse → guards → handler
  │
  ▼
[6] post-commit              outbox → consumers
  │
  ▼
response
```

### [1] The router finds the door

One worker sits in front of every vertical. Its entire job is: turn a hostname into a
`(tenant, scope, vertical, surface)` target, and forward.

It resolves that against the control-plane directory, uncached, once per request. Not
caching a route is a deliberate choice rather than an omission: a cached route that keeps
serving a suspended tenant blunts suspension, and suspension is a live weapon — it is
what you reach for when a customer must stop being served *now*. The cost is a directory
read per request; the alternative is a suspension that takes effect eventually.

Only `active` hostname bindings resolve. A domain still validating its DNS, or one whose
certificate failed, is simply unknown and gets "No application is configured for this
hostname."

Notice what the router deliberately cannot do. It binds what forwarding needs — the
control-plane directory it reads, the dispatch namespace it forwards into — and **no
`SCOPE` namespace**. It resolves names; it has no way to open a scope's database.
That boundary is a deployment fact written into its wrangler config rather than a rule
somebody follows. Handing the router the full scope host would have saved a file and
given the name-resolution worker authority over every tenant's data.

It also does not re-check tenant suspension, even though it reads the directory and
could. `getScope` owns that, inside the vertical. A second enforcement point is a second
thing that can disagree with the first; the router's job is to find the door, not to
decide who may open it.

#### The trust boundary

The router forwards with the resolution in headers:

```
x-substrat-tenant:   01J8F...
x-substrat-scope:    01J8G...
x-substrat-surface:  app
x-substrat-vertical: fsm
x-substrat-router:   <shared secret>
```

The vertical trusts these **absolutely** — they name whose data it is about to serve. Two
things make that safe, and both are required:

1. **Vertical workers have no public route.** `workers_dev: false`, no route; reachable
   only by service binding or dispatch from the router.
2. **`ROUTER_SECRET`** — the same value on the router and every vertical, presented as
   `x-substrat-router` and verified in the kernel's `readRoutedNode`.

The second exists because the first is a deployment fact and `workers.dev` is on by
default. One forgotten toggle makes (1) false with nothing in the code noticing, and the
consequence is a cross-tenant read.

Both halves **fail closed**. A router with no `ROUTER_SECRET` answers 500 to everything.
A vertical deployed without one refuses any asserted node with a `RouterAssertionError`
rather than trusting unsigned headers — this used to be a trust-by-default, which meant a
worker missing its secret would accept a forged tenant from anyone who could reach the
script directly. The only opt-out is `ALLOW_DEV_NODE`, which names an un-routed local
instance and authenticates nobody.

And the router **strips every inbound `x-substrat-*` header by prefix** before setting its
own, so a client cannot forge a node by simply sending one. Stripping by prefix rather
than by name is why adding a sixth asserted header later cannot reopen the hole.

### [2] Dispatch

With a target in hand, the router picks the worker.

If the resolved route carries a `deploymentRef` — the Workers-for-Platforms handle for the
scope's bound version — it dispatches into the namespace: `env.DISPATCH.get(deploymentRef)`.
This is how customer-pushed verticals are reached, and it is the normal path. If there is
no bound version, it falls back to a static `VERTICAL_<SLUG>` service binding, the
original shape, kept for routes that predate the registry.

Every dispatch also carries the **outbound policy** for that version — the declared egress
allowlist from its manifest — handed to the egress worker through the dispatch binding's
outbound parameters. The router never inspects that list; resolution produced it and the
egress worker enforces it. Chapter 5 picks this up where connectors do.

Two facts fall out of dispatching on `deploymentRef` rather than on the vertical's name.
Each scope is bound to a *version*, so two scopes of the same vertical can be running
different code at the same time — which is what makes previews and staged rollouts
possible at all (chapter 8). And the router does not re-check jurisdiction: residency is
pinned by configuration ahead of this worker and by the scope's DO placement below it,
and a third enforcement point here could only ever disagree with those two.

### [3] The vertical authenticates a person

Now we are inside the vertical's own worker, an ordinary Hono app. The first thing registered
on it is the invocation log, which writes one tenant-stamped line for this request and mints the
**invocation id** that every event the request emits will carry (chapter 10). It is first
because Hono runs handlers in registration order, and a log mounted below a route cannot see
that route.

Then the worker does three things before any Substrat API is involved.

**It reads the asserted node.** `readRoutedNode(request.headers, { expectedSecret })`
returns `(tenantId, scopeId, surface, verticalSlug)` or throws. This is the vertical's
side of the router contract, and it lives in the kernel because every vertical needs it
and none of them should re-derive how to trust it.

**It authenticates the caller.** This is where OIDC happens — a session cookie, or a
sign-in redirect to the issuer and back. The vertical is a relying party; it runs no
credential store of its own. What it gets back is a `sub`.

**It maps that `sub` to a principal.** The authenticated subject is not yet an identity in
this scope. The per-tenant identity directory resolves it to a `PrincipalId`, or to
nothing — in which case this person has authenticated successfully and is still allowed to
do nothing at all. Chapter 6 is about that line.

The important thing about this step is that it is the *vertical's* code, not the kernel's.
Substrat has an opinion about authorization and only a seam for authentication.
[Authentication & identity](https://substrat.net/concepts/identity.md) is that seam.

The same declared operations are served twice over, with no extra declaration: as REST routes
described by an OpenAPI document, and as an **MCP server at `/api/mcp`**, with one tool per
operation that faces HTTP. An agent holding a user's token can do exactly what that user could
do with `curl`, behind the same authentication and the same `ctx.check` inside the operation.
Exposing an operation is not authorizing it. A refused tool call comes back as a tool error the
agent can read, not as a failed session.

Alongside this, the platform's own management routes are mounted in one call —
`mountPlatformSurface(app, deps)` — which owns every `/internal/*` route the control plane
calls to provision, reconcile, snapshot, export, restore and configure this install,
behind a platform-secret gate that fails closed when the secret is unset. Those are not on
the request path a user takes; they are how chapters 8 and 13 reach in.

### [4] Getting a scope stub

```ts
const stub = await host.getScope(principal, tenantId, scopeId);
```

`CloudflareScopeHost` is the coordinator. It is **stateless**, rebuilt per request, and it
does two things before it will hand back a stub.

**Lifecycle gates.** `validateScopeAccess(tenantId, scopeId)` is evaluated durably in the
control-plane DO. It refuses a scope that is not servable and a tenant that is not
servable — suspended, deleting, archived, reaped, or simply not in this tenant. This is
the K-3 fail-closed path, and it is where suspension actually bites. A throw here
propagates out; there is no degraded mode.

**Lazy migration.** `migrateAndRecord(scopeId)` asks the scope to bring itself to the
current migration frontier. Migrations are not run at deploy time across the fleet; each
scope migrates **on wake**, inside its own serialization domain, the first time anyone
touches it after a version change. If it throws, the coordinator makes a best-effort
attempt to record *why* into the directory — and then rethrows the original error, on
purpose, because a broken recorder must not replace a diagnosable migration failure with a
confusing one. A scope whose migration failed fails closed and serves nothing, which is
what stops it from rendering as healthy in the console.

Only then is the stub minted. **Holding the stub is the authorization to talk to that
scope** — but it is not the authorization to do anything in particular, and the scope
re-validates every call regardless.

### [5] Inside the Durable Object

`stub.invoke('workorder/create', input)` is an RPC into the scope's own Durable Object.
The structured-clone boundary is that RPC hop. What happens on the other side, in order:

1. **The queue.** The call is enqueued on the per-scope `OperationQueue`. It waits until
   the operation ahead of it has run to completion.
2. **The transaction opens.** Everything below happens inside
   `ctx.storage.transaction(async …)` — the DO analogue of `BEGIN IMMEDIATE`, which rolls
   back on a throw *across awaits*.
3. **The operation resolves.** A name like `workorder/create` is looked up in the module
   registry. A module whose entitlement the tenant does not hold did not register at all,
   so its operations simply do not resolve — there is no half-loaded engine and no
   operation that exists but refuses.
4. **The input is parsed.** The host applies the module's declared Zod schema before any
   guard and before the handler. Handlers do not hand-parse. This happens on *every* path
   in — HTTP, test, seed, schedule — so a declared input that nobody validates is not
   possible rather than merely discouraged.
5. **Guards run.** Manifest guards, and any concurrency precondition the caller sent
   (`If-Match` against an entity's version). Idempotency keys are resolved here too: a
   replayed key returns the original result without re-running anything.
6. **The handler runs.** Its first line is a permission check. Chapter 4 is this step.
7. **Commit, or roll back.** On success the transaction commits — the domain rows, the
   events, the links, the grants, all of it, atomically. On a throw, all of it is gone.

There is one deliberate exception to "on a throw, all of it is gone". A **permission
denial is recorded after the rollback**, as its own write, outside the transaction that
just disappeared. That is the whole point of it: the denial is precisely the write the
refused operation could not make, so keeping it inside the rolled-back transaction would
erase the only evidence the refusal happened. Chapter 6 returns to it.

### [6] After the commit

The moment the transaction commits, the DO drains its outbox to consumers, each delivery in
its own transaction and still inside this operation's turn in the queue. That is chapter 5, and
it is where most of the system's interesting behaviour lives.

Back on the coordinator, outside the scope, any executors and connectors due for the new events
run next. If the operation raised platform intents, the vertical flags its response so the
router can ask the control plane to act on them straight away (chapter 5, "the kick").

Then the result travels back out: clone boundary, coordinator, vertical worker, JSON,
router, browser.

### What the local runtime does differently

Nothing that matters, which is the point.

Locally, a scope is a SQLite file instead of a Durable Object, the coordinator is in-process
instead of across an RPC, and there is no router — you address the instance directly with
`ALLOW_DEV_NODE`. The queue, the clone boundary, the transaction semantics, the lazy
migration, the fail-closed addressing and the stamped envelopes are the same, because the
same [conformance suite](https://substrat.net/reference/contract-tests.md) runs green on both, unchanged, in Node
and in real `workerd` against real Durable Objects. Neither side is a mock.

That equivalence is what makes local development deterministic and CI cloud-free, and it
is the reason a vertical moves from a laptop to Cloudflare with no code change.

---

## 4. What a handler can and cannot do

We left the last chapter one step from the handler. This chapter is that step: what an
operation is handed, what it is denied, and why each denial is worth its inconvenience.

### The whole surface

An operation handler receives exactly one thing.

```ts
async function createWorkOrder(ctx: OperationContext, input: CreateInput) {
  assertAllowed(await ctx.check('workorder:create'));

  const id = ulid();
  ctx.sql.exec(
    'INSERT INTO wo_orders (id, title, state, created_at) VALUES (?, ?, ?, ?)',
    [id, input.title, 'scheduled', ctx.now()],
  );
  ctx.emit({
    type: 'workorder.created',
    entity: { entityType: 'workorder', entityId: id },
    piiClass: 'none',
    payload: { id, title: input.title },
  });
  return { id };
}
```

That is representative, not simplified. `ctx` is the entire capability surface:

| | |
|---|---|
| `ctx.tenantId` / `scopeId` / `principal` | where this runs and on whose behalf, read-only |
| `ctx.sql` | `query` / `exec` against **this scope's** database |
| `ctx.now()` | the operation's instant, and the only clock |
| `ctx.check(perm, entity?)` | a permission decision |
| `ctx.emit(event)` | a domain event, into the outbox |
| `ctx.page(entityType, params)` | one page of a declared list: filters, sort, keyset cursor, total |
| `ctx.search(entityType, term)` | full-text lookup over declared searchable fields |
| `ctx.versionOf(entity)` | an entity's current version, for `If-Match` preconditions |
| `ctx.link(child, parent)` | a permanent entity relationship |
| `ctx.grant` / `ctx.revoke` | user-initiated per-entity sharing |
| `ctx.canAssign(roleKey)` | whether the caller holds everything a role would confer |
| `ctx.atomic(fn)` | a sub-transaction |
| `ctx.entitlement(key)` / `entitlements()` | the tenant's plan, quota and expiry for a module it holds |
| `ctx.requestPlatform(req)` | ask the platform for a privileged effect |
| `ctx.platformRequests(filter)` | read back what the platform did with those |
| `ctx.sealToConnection(provider, value)` | encrypt a value so only that connector can open it |

**Nothing on `ctx` checks a permission except `ctx.check`.** `page`, `search` and `versionOf`
read data, and the operation's own `assertAllowed` comes first, as always.

A connector's handler gets a *different* context with a scoped `connection(provider)` handle.
That handle is host code's, not an operation's, and it appears later in this chapter.

There is no `env`. No `fetch`. No database handle. No `process`. No import of anything
that could provide one. Capabilities come from `ctx` and from nowhere else, and that is
the single sentence the rest of this chapter unpacks.

### `ctx.sql` — and why it is the only way in

`ctx.sql` is already inside one scope's database. It has no parameter for tenant or scope
because there is nothing else it could address.

Module code may not import `better-sqlite3`, an adapter, `node:*`, or `cloudflare:workers`.
Most of those bans are obvious portability hygiene. **`cloudflare:workers` is not, and it
is the sharpest one.** That module exports an ambient `env`. One import hands module code
every binding and secret the script declares — including its own `SCOPE` Durable Object
namespace, which reaches *another scope's data*, precisely where `ctx.sql` cannot. It is a
single import that dissolves the isolation the whole architecture is built on, and it
looks completely innocuous in a diff.

The linter refuses it (`node tools/boundary-lint.mjs`, rule R2), and harness code —
`seed.ts`, `server.ts`, tests — is exempt because it is not module code.

One rule inside `ctx.sql` is enforced by the **runtime**, not only the linter: a write
whose target is a `_substrat_*` table is refused, on both adapters.

Those tables are **the spine**: the kernel's own tables, created in every scope database next
to your module's. The outbox and the delivery journal (chapter 5), the denial log, the
platform-intent queue, the permission tuples and their projections (chapter 6), the migration
journal, schedule state, attachments and idempotency records all live there. The directory has a
spine of its own, holding the admin log, the access log, identities and entitlements. Only the
kernel writes the spine. Forging a row in it is forging the audit trail or the permission model. This is a mechanism rather than a lint
rule because the linter does not run on the hosted push path, and a guarantee that holds
only where CI happens to look is not a guarantee. Only the write's *target* is judged, so
`INSERT INTO my_timeline SELECT … FROM _substrat_outbox` is fine.

Reading the spine **is** sanctioned, and there are helpers for it — use them:
`readTimeline` and `readHistory` from `@substrat-run/kernel` take an `EntityRef`, page
like an ordinary list read, and decode the envelope. `readHistory` additionally returns
the payload, the authorization chain, the impersonation stamp, the PII class, the emitting
operation and the version the emitting code was deployed as. On most of those a `null` is
a *fact* — the payload was erased, nobody was impersonating, a consumer emitted it — which
a hand-rolled `SELECT` would read as missing data instead. Neither helper checks a
permission; the caller does that first, as always.

### SQL strings, and why there is no query builder

`ctx.sql` takes a SQL string and a parameter array. There is no ORM, no query builder, and no
typed query layer generated from the model. The model emits tables, migrations, an API
description and a browser client, but not queries.

That can look like an omission. It is a choice, and the reasoning is worth checking against your
own threat model, because the protections are real and so is one gap.

**What a query builder would buy, and where you already have it.** Builders earn their keep in
three places: preventing injection, composing dynamic filters, and hiding dialect differences.
Here:

- **Binding is positional.** `ctx.sql.query('SELECT … WHERE id = ?', [id])` sends the value as a
  bound parameter, never as SQL text. That is the same protection a builder gives, provided
  values go in the array.
- **Dynamic reads are kernel-composed.** A list with user-chosen filters, sort and cursor is
  `ctx.page`, which builds the `WHERE`, `ORDER BY`, keyset comparison and `LIMIT` from what the
  operation *declared* filterable and sortable, and refuses anything else by name. The query that
  most tempts people to concatenate strings is the one you do not write.
- **There is one dialect.** Both adapters are SQLite, so there is no dialect to hide.

**Why injection is survivable when it happens.** Everything else here is defence in depth, and
it is deep:

- **Inputs are parsed before the handler runs** (below). An `id` declared as a ULID cannot arrive
  as `1 OR 1=1`.
- **The database holds one scope.** An injected statement runs against a database containing that
  scope's rows and nothing else. It cannot read another tenant, because there is no other tenant
  in the file (chapter 2).
- **The spine refuses writes regardless of where the SQL came from.** The spine-write check
  parses every statement in the string, including stacked ones, so an injected
  `; INSERT INTO _substrat_tuples …` is refused like a hand-written one.

**The gap, stated.** Nothing mechanical stops a handler from writing
`` `… WHERE name = '${input.name}'` ``. No lint rule inspects SQL strings for interpolation.
Engines interpolate constant column lists on purpose, which is why a crude rule would drown in
false positives. So "values go in the array" is a review convention, not a mechanism. And on the
local SQLite adapter, statements that reach past the scope's own file, such as `ATTACH`, are not
refused by the kernel. The spine check deliberately leaves them to the runtime, and on Durable
Objects the runtime restricts them. On self-hosted SQLite, an injected `ATTACH` is the one path
from a scope's SQL to another file on the same disk.

Why plain SQL at all: a migration is a human checkpoint, and SQL is what a reviewer can read.
A query in a diff is a query, not a builder chain whose SQL you have to imagine.

### `ctx.now()` — the only clock

`new Date()` and `Date.now()` are boundary-lint violations (R6), the same class of ban as
`node:*`.

Two reasons, and the second is the one that bites. First, **stability**: `ctx.now()`
returns the same instant for the whole invocation, stamped when the context was built. Two
rows written in one transaction cannot disagree about when they were written, and an event
cannot disagree with the row it describes. Second, **testability**: anything genuinely
time-dependent — an absence window, a metering period, a booking hold, a grant expiry — is
untestable against the wall clock except by sleeping or by shrinking the window to zero,
both of which produce tests that assert less than they appear to.

The pure host takes a `clock`, so a scenario uses `manualClock` or `frozenClock` and
asserts an elapsed-time transition directly. The Durable-Object host declares
`clock?: never` and means it: the reads that matter happen inside the ScopeDO that workerd
constructs, which a host option cannot reach. So grant *expiry transitions* are held to
the contract on SQLite only, and so is facet *recency* — that a bucket's `lastSeen` is its
latest event, which the wall clock cannot tell from its first. Those are the two suites the
two adapters do not share, for that one reason, and it is written down rather than quietly
tolerated.

Code that must read the real clock — a JWT whose `exp` a remote server judges — opts out
with a reviewable `boundary-lint-allow R6` block.

#### Timestamps are text, and a fixed width of it

Timestamps are stored as **ISO 8601 text**, never epoch integers. `ctx.now()` returns an
`Instant`, and every `Instant` is normalized to UTC in exactly one shape:
`2026-09-17T08:41:05.123Z`. Offsets are converted and precision is fixed at milliseconds, so every
value is exactly 24 characters.

The reasons, in order of how much they matter:

- **Comparison is lexicographic, and correct.** Grant liveness is `expires_at > ?`, a metering
  close horizon is a string comparison, and `ORDER BY occurred_at` sorts. Text sorts
  chronologically only when every value shares a zone and a shape, and the normalization
  guarantees that. An offset timestamp compared as text against a `Z` timestamp is simply wrong,
  which is why the parser rewrites rather than accepts.
- **No unit convention.** An integer column cannot say whether it holds seconds or milliseconds,
  and a system with many modules eventually has both.
- **It reads correctly in a `.sqlite` file someone opens by hand**, in an exported dump, and in an
  event payload. The row and the event announcing it hold the identical string.

A fair worry about text is that values change size on update, which in some storage engines
means rows move and pages fragment. It does not apply here, for two reasons. First, the
normalization makes every instant the same length, so updating `2026-09-17T08:41:05.123Z` to a
later instant rewrites 24 bytes with 24 bytes. The row does not grow. (Moving a column from
`NULL` to a value grows a row whatever its type.) Second, SQLite has no append-only heap and no
MVCC row versions. An update rewrites the record inside its B-tree page, and space freed inside
a page is reused by that page.

The real cost is width. SQLite stores an epoch integer in 4 bytes as seconds or 6 as
milliseconds, so 24 bytes is four to six times as much per timestamp column. It is a real cost, it is small next to a row's other text,
and it buys comparisons that cannot silently go wrong.

For dates without a time, such as a birthday or a leave day, the contract is `calendarDate`,
`YYYY-MM-DD`, which sorts the same way.

### `ctx.check` — first line, every time

Every operation's first line is a permission check:

```ts
assertAllowed(await ctx.check('workorder:create'));
```

Not "usually". The reason to make it unconditional is that the alternative is a judgement
call at every handler, and judgement calls are where the missing check comes from six
months later.

Portal-style walks — "which of these can this person see?" — use the entity-scoped form,
`ctx.check(perm, entityRef)`, per row. Chapter 6 is how those are answered without a
network call.

A refused check throws `PermissionDenied`, the transaction rolls back, and the denial is
then written **outside** that transaction so it survives. You get a queryable record of
what was refused, to whom, on which operation, under which impersonation session — which
is the data you want when a customer says "it says I can't" and you need to know which
permission key to look at.

### `ctx.emit` — and the envelope you do not control

A mutation emits an event. The kernel stamps the envelope: id, occurred-at, tenant, scope,
actor, authorization chain, impersonation, the emitting operation, the deployed version.
The module supplies the type, the entity, the PII class and the payload — and the payload
is validated.

You cannot suppress the stamp, forge an actor, or emit on another scope's behalf, because
`emit` is below the API surface rather than beside it. That is what makes the event stream
an audit log *by construction* instead of by discipline.

Two rules about payloads matter enough to state here, and chapter 5 explains them:

- **Events are fat.** The payload carries everything a consumer needs, so the consumer
  snapshots rather than joins. A consumer that has to read back across a module boundary
  to interpret an event is an event that was emitted too thin.
- **PII classification is mandatory.** Not defaulted — declared. It is what makes erasure
  a query rather than an audit.

### `ctx.atomic` — the only way to catch an engine error

This one is a real trap, and it has a mechanism behind it.

An engine call composed inside your transaction has **no boundary of its own**. So a bare
`catch` around it leaves you holding its partial writes — the rows its invariants were
protecting — and then commits them:

```ts
// WRONG — boundary-lint R7 rejects this
try {
  await completeWorkOrder(ctx, { orderId, billable });
} catch {
  // the engine's half-written rows are still here, and will commit
}
```

The correct shape gives the engine call a boundary:

```ts
try {
  await ctx.atomic(() => completeWorkOrder(ctx, { orderId, billable }));
} catch {
  // the engine's rows, events, links, grants and platform intents are all gone;
  // your own writes survive, and it still commits once
}
```

A succeeded `atomic` is still **provisional** — if the operation later throws, its writes
go with everything else. Sub-transactions nest but must not interleave; starting two
concurrently throws.

Outside `ctx.atomic`, catching an engine error is forbidden and **R7 rejects it with no
escape hatch** — unlike R5, R6 and R8, which all have reviewable `boundary-lint-allow`
blocks. There is no legitimate reason to swallow an engine error unprotected.
`try`/`finally` with no `catch` is fine, and so is a catch that always rethrows: the
operation still fails and the whole transaction still rolls back.

### What is not here: the network

There is no `fetch` in module code. None. Boundary-lint rule R3 refuses it, along with any HTTP
client import.

Effects on the outside world are somebody else's job. There are three mechanisms, and all three
share one shape: **the module writes a durable row in its own transaction, and something outside
the transaction acts on it afterwards.**

- **An executor** is a handler the *host* registers against an event type. It runs after the
  commit, outside the scope, with platform authority (`HostAdmin`) instead of a `ctx`. It is the
  shape for effects that are not scope-local, such as writing tenant-wide directory state,
  because a module cannot write to another database inside its own transaction without leaving
  an orphan on rollback.
- **A connector** is an executor that is also handed a per-tenant credential and permission to
  call one provider over the network, and nothing more. Signing, accounting and planning
  integrations are connectors. A connector is host code, never module code.
- **A platform intent** (`ctx.requestPlatform`) is how a sandbox-clean vertical asks for a
  privileged action it has no authority to perform itself. The intent is a durable row written
  atomically with your operation. The platform pulls and executes it later, and
  `ctx.platformRequests()` reads back what happened, including a failure, so a screen can say
  "this did not go out" instead of showing something that only appears to have.

All three are chapter 5.

### What an `await` holds

Handlers are `async`, so a fair question is what happens while one waits. Does a slow
operation block other requests on its scope, or on every scope?

**Its own scope only.** Chapter 2's queue gives each scope one turn at a time, and the turn
lasts from the moment the operation starts until its post-commit work is done. On Durable
Objects, post-commit work includes draining in-scope consumers (chapter 5). So an operation that
takes two seconds, or a consumer that takes two seconds, delays the *next operation on that
scope* by two seconds.

Everything else runs on:

- **Other scopes** are other Durable Objects, or other actors on the local adapter, each with
  its own queue. A slow scope costs nobody else anything.
- **The vertical's worker is not serialized.** Requests to it run concurrently. Only the hop
  into a scope waits its turn.
- **Executors and connectors on Durable Objects run on the coordinator**, after the scope's turn
  has ended. A slow provider delays that HTTP response and holds no scope's queue. (On the local
  SQLite adapter they currently run inside the scope's turn. That is a real difference between
  the two hosts, and it only shows up under a slow connector.)

In practice a handler has almost nothing slow to wait on, and that is the ban on the network
working as designed. `ctx.check`, `ctx.entitlement` and `ctx.canAssign` read scope-local
projections (chapter 6). `sealToConnection` is local Web Crypto. Nothing on `ctx` crosses a
network on the normal path, so the queue drains quickly *because* module code cannot reach
anything slow. Nothing in the kernel times an operation out, either. The platform's own limits
are the only ceiling.

#### When the work genuinely takes time

Some work is slow by nature: calling a language model, rendering a document, waiting on a
provider. The pattern is to **keep the slow part out of the operation and bracket it with
operations**:

```ts
// in the vertical's worker (harness code, not module code)
const context = await stub.invoke('desk/prepare-answer', { conversationId });  // fast: read + record intent
const answer  = await models.answer(context);                                  // slow: no scope turn held
await stub.invoke('desk/record-answer', { conversationId, answer });           // fast: write + meter + emit
```

The model call holds no transaction and no queue turn. Each operation around it is short,
checks its own permission, and commits on its own. The reference support-desk vertical answers
questions exactly this way, with the whole sequence handed to `waitUntil` so the user's
request returns first. If the work must survive the isolate going away, make the first
operation record that it is pending, so a sweep or the next request can find unfinished work.

The other option is to make the slow part an effect: emit an event, and let a connector do it
with retries (chapter 5). Pick by one question: does the vertical need the result *inside*
its own logic (bracket it), or is the result a side effect in someone else's system (emit
it)?

### Parse, don't trust — and the host is what parses

Operation inputs go through Zod schemas at the boundary, and **the host applies them**.

A module passes `operationInputs: operationInputsOf(ops)` beside its `operations`, and
every invocation is parsed before the guards and before the handler, on every path in —
HTTP, test, seed, schedule. Handlers do not hand-parse.

The distinction is not stylistic. If parsing were the handler's job, a declared input that
nobody validated would be an ordinary oversight. Because it is the host's job, it is not
possible.

### Why all of it is worth the inconvenience

Each rule in this chapter closes a specific failure that is otherwise silent:

| Rule | The failure it closes |
|---|---|
| `ctx.sql` only | reading another tenant |
| bound parameters, `ctx.page` for dynamic reads | injection, and filters nobody declared |
| no `cloudflare:workers` | one import, every secret and every scope |
| no spine writes | a forged audit trail |
| `ctx.now()` only | rows disagreeing about when; untestable time logic |
| normalized ISO text | timestamps that compare wrongly |
| no network in handlers | a slow provider holding a scope's queue |
| check first, always | the guard that was never written |
| kernel-stamped envelopes | an event that lies about who did it |
| `ctx.atomic` to catch | committing an engine's half-finished state |
| host-side input parse | a declared schema nobody applied |

None of these is enforced by asking nicely. Most are linted, several are runtime
mechanisms, and the ones that are only conventions are named as such in
[Agent rules](https://substrat.net/guide/agent-rules.md) rather than allowed to look enforced.

---

## 5. The life of one event

This is the chapter the reference genuinely does not have. Every fact in it is published
somewhere — [Events & audit](https://substrat.net/concepts/events.md) has the semantics, the adapter pages have
the machinery, [Modules & the manifest](https://substrat.net/concepts/modules.md) has the declarations — but no
page walks the loop, so here it is, from `ctx.emit` to the consumer's commit, including
every way it can fail.

### Emit is a write, not a send

```ts
ctx.emit({
  type: 'workorder.completed',
  entity: { entityType: 'workorder', entityId: id },
  piiClass: 'none',
  payload: { id, lines, total },
});
```

Nothing is sent. `emit` writes a row into `_substrat_outbox`, **inside the operation's
transaction**, and returns.

That is the whole foundation of the design. The event and the domain write commit together
or not at all. There is no window in which the row exists and the event does not, and none
in which an event announces a row that rolled back. No two-phase commit, no coordinator,
no reconciliation job to find the mismatches — because the mismatch is not representable.

The envelope is stamped kernel-side. The module supplies four things (type, entity, PII
class, payload); the kernel supplies the id (a ULID, which is monotonic, so `ORDER BY id`
is creation order), the instant — the *operation's* instant, so every event one operation
emits carries the identical one — the tenant, the scope, the actor, the authorization
chain, the impersonation session if there was one, the operation name, and the version the
emitting code was deployed as.

#### Fat payloads, and why

A consumer must never need a cross-module read to interpret an event. So the payload
carries everything: the billable lines, the prices, the totals — not an id to go look up.

The consumer therefore **snapshots** rather than joins, and prices are frozen at the moment
the event happened. Which is exactly what an invoice wants, and it stops being a stylistic
preference the moment you notice the alternative: a consumer that reads back into the
producer's tables is a consumer that breaks when the producer's schema moves, and that
reads *today's* price for *last month's* work.

And the consumer validates with **its own Zod parse**, never the producer's types. Importing
a producer's type would make the contract a compile-time coupling; parsing the payload
makes it a runtime contract, which is the only kind that survives two modules being on
different versions.

#### PII classification is mandatory

`piiClass` is declared, not defaulted. It is what makes erasure a query rather than an
audit: when a subject invokes a right to erasure, the payloads that must be cleared are
the ones the class names, and the envelope — who did what, when — survives, because the
envelope is the compliance record.

### Post-commit: the consumer loop

The transaction commits. Immediately after, still inside the same invocation, the scope
drains its outbox:

```ts
// scope-do.ts, the tail of invoke()
if (!replayed) await this.dispatch(tenantId, scopeId);
```

`dispatch` is small enough to describe exactly. It runs **up to 50 rounds**. Each round
walks every registered module, and for each of that module's consumers, selects the outbox
rows of the matching type that have **no row in `_substrat_deliveries`** for that
`(event, module)` pair, oldest first.

For each such event:

```ts
await this.ctx.storage.transaction(async () => {
  const ctx = this.operationContext(systemPrincipal, tenantId, scopeId, { system: mod.id });
  await consumer.handler(ctx, event);
  this.sql.exec(
    `INSERT INTO _substrat_deliveries (event_id, consumer_module, delivered_at)
     VALUES (?, ?, ?)`,
    event.id, mod.id, new Date().toISOString(),
  );
});
```

Three things are load-bearing here.

**Each delivery is its own transaction.** The consumer's writes and the journal row marking
it delivered commit together. So a crash between the handler finishing and the journal being
written cannot exist — either both happened or neither did. Delivery is exactly-once with
respect to the journal, and at-least-once with respect to the world.

**The consumer runs as a system actor.** Its context carries `{ system: '@substrat-run/engine-invoicing' }`,
and that is what the audit trail shows for anything it writes. It is an ordinary in-scope
operation in every other respect — same `ctx`, same rules, same transaction semantics, same
inability to reach another scope.

**The loop rounds because consumers emit.** A consumer's own `ctx.emit` writes another
outbox row, which the next round picks up. Fifty rounds is the cascade limit, and the loop
exits as soon as a round delivers nothing. An event emitted this way records **`causedBy`**, the
id of the event whose delivery emitted it. It also inherits the originating request's
**invocation id**, because the drain is part of the same call. Chapter 10 walks both links.

**All of this happens inside the scope's turn.** The drain runs after the commit, but before the
queue lets the next operation in. A slow consumer therefore delays the next operation on its
own scope (chapter 4), and no other scope.

#### When a consumer throws

This is the part worth reading twice, because it is not what most queue-shaped systems do.

```ts
} catch (err) {
  // Dead-letter (v0): journal the failure so one poison event
  // can't wedge the loop. Written outside the rolled-back txn.
  this.sql.exec(
    `INSERT INTO _substrat_deliveries (event_id, consumer_module, delivered_at, error)
     VALUES (?, ?, ?, ?)`,
    event.id, mod.id, new Date().toISOString(), String(err),
  );
}
```

The consumer's transaction rolls back — none of its writes survive — and a journal row is
written **with the error**, outside that rolled-back transaction, so it persists.

**In-scope consumers do not retry.** One failure is terminal for that `(event, consumer)`
pair. The next round's query excludes the event because a delivery row now exists, so the
loop cannot wedge on a poison event, and the failure is evidence rather than a silent drop.

That is a deliberate v0 choice and the code says so. It is the right default for a consumer
that is pure in-scope logic — such a consumer fails because of a bug or bad data, and
retrying a bug five times produces five identical failures and a delay. It is the wrong
default if you were expecting queue semantics, so: **a failed in-scope consumer needs a
human or a replay, not a wait.**

Effects that genuinely need retrying are not in-scope consumers. They are executors, and
they work differently.

#### Ordering

Guaranteed within one `(scope, module)` pair, and not across them. Within a pair, the
`ORDER BY o.id` on a monotonic ULID mint means creation order.

### The connector seam: executors

Some effects are not scope-local. Adding someone to an organization writes tenant-wide
directory state, outside any one scope's transaction. Sending mail leaves the building
entirely. A consumer cannot do either, because a consumer runs *inside* the scope and can
only touch that scope's data.

So a module **asks**, and an **executor** effects:

```
module (in-scope)                        executor (out-of-band)
  ctx.emit('member.add-requested')  ──▶    admin.addMember(...)
  commits WITH the domain write            writes the audit row
```

```ts
host.registerExecutor('member-adder', 'member.add-requested', async (admin, event) => {
  // receives HostAdmin, not ctx — it acts with platform authority,
  // which is exactly what module code must never hold
});
```

Why not write directly from the module? Because that would be a write to another database
inside a scope transaction: two independent commits, no coordinator, and an orphaned
membership if the scope rolls back after the directory write lands.

The event has no such hazard, because it enters the outbox in the **same transaction** as
the domain write. A rollback leaves no event and therefore nothing to effect.

#### Who registers executors in practice

The membership example above is the textbook one, and it lives in the contract suite. In the
running platform, the executor you will actually meet is a **connector**: an executor that is
also handed a credential and one provider's worth of egress. An e-signature connector is the
reference, registered against the event a protocol emits when it requests signatures.

Where it runs depends on the host, and the difference matters:

- **Self-hosted, or with a control-plane binding**, the host runs the connector on the
  coordinator, with the retry policy below.
- **A hosted, sandbox-clean vertical has no credentials to hand anyone.** It declares the
  connector as a routing entry only, and the host turns each due delivery into a
  `connector:<provider>` **platform intent** (below). The control plane, which holds the
  credential, runs the connector when it drains that intent.

A third kind of connector subscribes to no event. An accounting or planning integration that
*pulls* data runs as a sweeper inside the platform sweep (chapter 9), on a schedule.

#### Executors retry, and the ordering is the point

An executor runs on the **coordinator**, not in the DO — it acts through `HostAdmin`, which
is outside the scope entirely. So the drain is a read inside the DO, the effect happens
outside, and the journal is written afterwards:

```sql
SELECT o.* FROM _substrat_outbox o
LEFT JOIN _substrat_deliveries d
  ON d.event_id = o.id AND d.consumer_module = ?
WHERE o.type = ?
  AND (d.event_id IS NULL
       OR (d.next_attempt_at IS NOT NULL AND d.next_attempt_at <= ?))
ORDER BY o.id
```

"Due" means never attempted, or retrying and now past its next attempt time. Terminal rows
— delivered, or dead-lettered with `next_attempt_at IS NULL` — are excluded by the join.

The journal is written **after** the effect, on purpose. Claiming a delivery before running
it would make delivery at-most-once and lose the effect on any crash in between. Writing
afterwards means a crash mid-effect retries — which is why executors, like consumers, must
be idempotent, with the event id as the idempotency key.

Retry policy is **per executor**, not a host-wide constant, because the right answer differs:

| | default |
|---|---|
| `maxAttempts` | 5, including the first; reaching it dead-letters |
| `baseDelayMs` | 1000ms, doubling per attempt |
| `maxDelayMs` | 300000ms (5 minutes) — the ceiling on the doubling |

with jitter of ±20% on each computed delay. Those defaults suit a directory write. A
connector making an outbound HTTP call wants a longer tail, and sets one.

A failed attempt with **no next attempt time** is a dead letter — the row keeps the last
error, and the drain report counts it. `ExecutorDrainReport` returns `attempted`,
`delivered`, `retrying` and `deadLettered`, and those last two are the numbers a health
surface reports. A caller that ignores them learns nothing, which is exactly the failure
mode the older silent path had.

#### Who actually retries

Dispatch is **prompt**: the drain runs inline, in the request that emitted the event, so
the common case completes before the response goes out. The outbox is the *backstop*, not
the mechanism.

The backstop needs something to run it, and this is where the reference splits across pages
in a way that genuinely confuses people. It is the **scope sweeper** — an alarm-driven
singleton Durable Object in the vertical's own deployment, calling `drainDue` over its
roster of scopes. Not the platform sweeper, which is a different clock doing different
work in a different place. Chapter 9 is both of them and the distinction.

#### The trail joins

Splitting a change across two halves would otherwise split its audit trail. So admin rows
an executor writes carry `causedBy` — the id of the event that caused them.

That is the event id rather than a separate correlation field, because the envelope already
carries a unique kernel-stamped id, and reusing it avoids widening a frozen contract to say
something it already says.

### Platform intents: asking rather than calling

There is a third shape, for when a sandbox-clean vertical needs a privileged action —
provisioning a sibling scope, say — and holds no privilege at all.

`ctx.requestPlatform(request)` writes a durable row into this scope's
`_substrat_platform_requests` spine, atomic with the operation, exactly as `emit` is. The
platform pulls and executes it later, knowing the tenant inherently because it read this
scope's DO. Origin fields are stamped kernel-side; the call returns the new id.

Call it **after** your own permission check. Authorization is the vertical's decision;
isolation is the platform's. And it applies backpressure: a scope holding the maximum
pending intents throws rather than queueing without limit.

Intent kinds are a closed set: provisioning a sibling scope or a whole tenant, archiving a
scope, reconciling a managed tenant's entitlements, recording a model-usage line (chapter 12),
and the `connector:<provider>` family. The platform uses one more kind for its own sweep
records, and module code is refused it, so a vertical cannot forge a verdict about its own
schedules.

#### How fast an intent runs, and the kick

An intent is **pulled**, never pushed, and by default the pull is the platform sweep: a
fifteen-minute cron on the hosted control plane (chapter 9). An intent raised at 10:01 can
therefore sit until 10:15. For an intent a user is waiting on, such as "provision my new
branch", that is too slow.

So there is a fast path, and it has three parts, each owned by a different party:

```
operation commits with ≥1 intent
  │  host fires onPlatformRequests(count)      ← only after COMMIT, never on rollback
  ▼
vertical sets response header  x-substrat-platform-request: 1
  │
  ▼
router sees the header → waitUntil(kickDrain)  ← after the response has gone
  │  POST /internal/drain-scope {tenantId, scopeId}   (service binding + platform secret)
  ▼
control plane drains that one scope now
```

The header carries no payload and no privilege. The control plane re-derives the tenant and
vertical from the directory and simply pulls. A forged or spurious header costs one empty pull.
The user never waits for the kick either, because the router runs it after returning the
response. With it, an intent settles in seconds.

**The part people miss is the vertical's.** The kernel exposes the signal and does not set the
header for you. A vertical opts in when it mints its stub:

```ts
const stub = await host.getScope(principal, tenantId, scopeId, {
  onPlatformRequests: () => c.header(PLATFORM_REQUEST_HEADER, '1'),
});
```

A vertical that skips this still works, and its intents wait for the sweep. That is the
designed failure direction: the kick is advisory, and **a missed kick costs one sweep interval,
never an intent**. It is also why "provisioning works but takes a quarter of an hour" is almost
always a vertical that never wired the header, and not a slow platform.

A drain that throws leaves the intent `pending` for the next pull. After 100 attempts (about a
day at sweep cadence) it settles as `failed` and records an operational failure, so a stuck
intent eventually becomes a visible one.

#### Reading the outcome

The outcome comes back. `ctx.platformRequests(filter)` reads this scope's own intent
journal — newest first, filterable by kind and status — with the `result` or `lastError`
the platform settled. That read exists for one concrete reason: a contract whose signature
request settled `failed` can say so on its own screen, instead of showing a document that
appears to be out for signature and is not. The kernel owns every write to that table, so a
status is only ever the platform's answer.

#### How the platform finds out

Nothing leaves the scope inside the transaction. The intent is a row; something outside has
to notice it. Two things do, and they are not alternatives.

**Diagram — how committed work leaves a scope.**

1. **An operation commits** (in the scope · one transaction) — your rows · its events in _substrat_outbox; a platform intent, if it asked for one.
2. Two paths reach the control plane:
   - response flagged: **The router kicks** (latency · seconds) — returns the response to the user first; then asks for a drain · intents only, today. It names a scope, carries no data.
   - **The platform sweep** (backstop · every 15 minutes) — walks every active scope; catches anything a kick missed. It runs on a timer rather than a signal, and drains each active scope.
3. **The control plane drains the scope** (platform · the only thing that acts) — reads the rows itself, through the vertical's /internal routes; so a kick can speed up a scope's own work, never forge it.
4. It executes intents — **Run with platform authority** (intents) — executed outside the scope; outcome written back to its intent journal.
5. It ships events — **Shipped to the lake** (events · tier 2) — marked shipped only once the lake confirms; through the sweep only — no kick yet.

The kick and the sweep are not alternatives. The kick makes it fast; the sweep makes it certain — a lost kick costs a wait, never a missed intent or event. Neither carries data: the control plane always reads the scope itself.

The **kick** is the fast path. The vertical flags its response, and the router — the one hop
that already knows which tenant and scope it just served — returns the response and then asks
the control plane to drain that scope. The **sweep** is the guarantee: every 15 minutes the
platform walks every active scope and drains what is pending, so a kick that was lost costs a
wait and never an intent.

The kick is safe for a reason worth stating. It carries no data. It names a scope, and the
control plane reads that scope itself and re-derives the tenant from its own directory, so the
most a kick can do is make a scope's own work happen sooner.

### Why an engine composed by event has no exports

The invoicing engine consumes `workorder.completed` *and* `commerce.order-placed`, events from
two different domains, without importing a single type from either producer. From them it
builds a customer's **billing basis**, the lines an invoice will be issued from (chapter 12). It has
deliberately **no in-scope exports**: the vertical emits, the engine consumes, and the
vertical reads results back through the engine's own operations or by consuming its events.

That is not an omission. The engine being the only writer of its rows is what keeps
invariants like immutable-after-export safe from a half-finished caller. Chapter 7 is when
to reach for that shape versus a direct call.

### Schema versions, and why dual-emit is not available

Event payload fields are **frozen once shipped**. Rename, remove or retype means a
`schemaVersion` bump, and a bump is a **replace** — you cannot emit v1 and v2 side by side
during a deprecation window.

The reason is mechanical rather than philosophical. Consumer dispatch selects on event
*type* alone (`WHERE o.type = ?`), and the `schemaVersion` a manifest's `consumes` entry
carries is discarded at registration. So emitting both versions delivers **both** to the
same consumer. The invoicing engine's export event is meant to be consumed by an accounting
connector. None consumes it yet (chapter 12), but once one does, a dual-emit is a double
invoice in production, silently. The rule is not hypothetical either: the export event has
been through exactly one version bump, and it was shipped as a replace.

A replace fails loudly instead: a v1 consumer's strict parse rejects v2, and the event
dead-letters where somebody sees it. Loud and stopped beats quiet and doubled.

So no plan should promise a deprecation window that nothing can deliver. Routable
dual-emit needs `(type, schemaVersion)` in the dispatch predicate plus a version dimension
on the consumer registry — designed, filed, and postponed rather than abandoned.

### The event stream is the audit log

Because every event carries tenant, scope, actor, entity and time — stamped, not supplied —
the stream *is* the audit log. Complete by construction. "Who did what, when, to which
entity" is a query, and the answer is the same data reporting runs on.

Which is why reading one entity's history is a sanctioned projection over
`_substrat_outbox` — rule 3 bans *writes* to the spine, not reads — and why `readTimeline`
and `readHistory` exist rather than everyone writing their own `SELECT`.

The same events also leave for a longer-lived store. The sweep drawn above ships each scope's
unshipped events to the platform's Tier-2 event lake, where history across scopes and over
long periods can be queried, and marks each one shipped only after the lake has confirmed it.
A failure in between sends the same events again, which the event `id` makes harmless; marking
first would risk a gap nothing could detect. Shipping removes nothing from the scope — the
outbox still serves consumers, entity history, and each entity's version.

---


## 6. Permissions and identity

Two questions that look like one and are not.

**Authentication** asks who someone is. It happens at the edge, it involves an issuer, and
Substrat has an opinion about none of it beyond the shape of the seam.

**Authorization** asks what they may do. It is kernel-owned, it is enforcement input, and
it is never delegated to an auth provider — because an authorization model that lives in
your identity vendor is a model you cannot reason about, version, diff, or take with you.

### Authenticate: the neutral seam

A vertical is an OIDC relying party. Sign-in is a redirect to an issuer and back; the
vertical runs no credential store, holds no passwords, and implements no reset flow.

What comes back is a `sub` — a subject identifier from the issuer. That is not yet
anything. The per-tenant identity directory maps `sub` → `PrincipalId`, and a `sub` with
no mapping is a person who has authenticated perfectly and may do **nothing**. The two
halves are genuinely independent: proving who you are gets you no authority at all.

Every demo vertical in the repo is OIDC-only, and the local development story is a real
OIDC provider (`@substrat-run/dev-issuer`) whose only shortcut is that `/authorize` lists
names instead of asking for a password. So the local login *is* the production round-trip,
and changing issuer is a change of `OIDC_ISSUER` rather than a change of code. There is no
dev-auth branch to diverge.

[Authentication & identity](https://substrat.net/concepts/identity.md) is the detail — issuers, adapters, first-login
sync, and the owner-claim flow for a freshly provisioned instance.

### Authorize: three authored things

Humans, and agents under review, write exactly three kinds of thing.

**Permission keys** — module-namespaced strings, declared in the manifest with
human-readable descriptions:

```
workorder:create   Create work orders
workorder:report   Start work, report time and material
invoicing:export   Export an invoice basis (makes it immutable)
```

**Roles at nodes** — a role bundles permissions; an assignment binds a principal to a role
**at a node of the tenancy tree** (the tenant root, or one scope), inheriting downward.

```ts
host.admin.assignRole({
  principalId: tech,
  roleKey: 'technician',
  node: { tenantId, scopeId: stockholmBranch },  // scopeId: null = whole tenant
});
```

**Capability grants** — one permission, one node, optionally narrowed to **one entity** and
its declared descendants, optionally time-boxed:

```ts
host.admin.grant({
  principalId: portalCustomer,
  permission: 'workorder:read',
  node: { tenantId, scopeId: branch },
  entity: { entityType: 'facility', entityId: theirBuilding },
  expiresAt: nextMonth,
  grantedBy: adminPrincipal,
});
```

Entity-narrowed grants are how a customer, a board member or a subcontractor sees only
*their* things inside a shared scope. A grant can also target an organization, and members
reach it through membership.

That is the entire *administrative* surface, and there is no fourth kind of thing to
author. What arrives later in this chapter is not a fourth kind but a fourth *author*: a
user sharing one record from inside an operation, which mints a capability grant of exactly
the shape above.

### How a check is answered

**Diagram — a permission, from declaration to check.**

1. **Declared in TypeScript** (authored) — module manifests → keys + descriptions; roles → templates · entity grants → shapes.
   - Branches off to **pnpm lint:permissions** (checkpoint · a human reads this), which renders emits PERMISSIONS.md, the review artifact; CI --check fails the build on drift. This branch ends in a human review, not at runtime.
2. *substrat push* → **The deploy manifest** (ships) — the surface rides along as a registry; content-hashed → digests.permission.
3. *promote* → **Admission** (gate) — a real diff between two versions; a widened surface is visible, not implicit.
4. *at write time* → **Provisioning** (per tenant) — role templates projected into each; tenant's own _substrat_roles.
5. *every operation* → **ctx.check** (runtime) — reads scope-local tables only; absent or empty projection = deny.

The branch off the top ends in a person: PERMISSIONS.md exists to be read in a diff, and CI going red is what makes the reading unskippable — it is not itself the approval.

Internally the checker compiles those three things into relationship tuples
(`subject → relation → object`) and evaluates with a **fixed four-rule algebra**:

1. **Role expansion** — principal has role, role carries permission.
2. **Tenancy-tree inheritance** — a permission at a node flows down to child scopes.
3. **Entity parent edges** — declared in manifests (`workorder → facility`) and written at
   runtime by `ctx.link`; entity-narrowed grants flow along them, depth-capped.
4. **Org membership** — grants to an organization reach its members.

No negation. No configurable rewrite rules. Verticals never see or author tuples.

The fixed algebra is the point. A policy language rich enough to express anything is rich
enough to express a policy nobody can predict the behaviour of, and "why can this person
see this?" stops having an answer. Four rules have an answer, always.

### The check does not leave the scope

This is the design decision with the widest consequences, so it is worth stating plainly.

Scope and entity tuples (rules 2 and 3) live in the scope's own database and evaluate
inside its serialization domain. Tenant-level facts — role definitions, role assignments,
tenant grants, org membership (rules 1 and 4) — are tenant-wide, so the control-plane
directory is their authority.

But **the checker never reads the directory on a request.** The control plane *projects*
those facts into scope-local tables (`_substrat_roles`, `_substrat_tenant_tuples`) at
**write** time, and the checker reads the projection exactly as it reads scope tuples.

> The control plane is a **write-time authority that projects into scopes**. It is never a
> read-time dependency on the request path.

Cost moves from the read path (every request) to the write path (rare role and grant
changes), which is the correct direction for a read-heavy multi-tenant system. And it is
what makes a scope genuinely isolated: a vertical can run as its own Worker with **no
control-plane binding at all**, which is the shape a customer-pushed vertical ships in.

The consistency consequence is small and one-directional. Scope-level grants and roles stay
synchronous and immediately consistent — the checker runs in the scope's own serialization
domain, so there is no "did my write land" question. Only *tenant-level* changes are
eventually consistent across a tenant's scopes, bounded by the write-time fan-out and a
reconciliation sweep.

And the failure direction is safe: **an absent or empty projection is a deny**, byte for
byte the normal deny path. Missing data can only ever remove authority, never grant it.

### Revocation tombstones

Access is withdrawn by **tombstoning** a tuple: it keeps its row, gains a `revokedAt`, and
the checker's walk skips it. Nothing deletes a tuple, ever.

A tuple that once granted access is the evidence of *why* an access was allowed. Deleting
it destroys the audit trail exactly where it is most needed — a deleted row can show
neither that access was revoked nor that it was ever granted.

"Ever" means for as long as the scope does. Reaping is the one thing that takes a tuple
away, because reaping takes the whole database the projection lives in; what outlives it is
the admin log, which is never swept (chapter 11).

Liveness is therefore one predicate applied identically everywhere: a tuple grants only
while it is **unexpired and unrevoked**. Expiry and revocation are siblings, not separate
mechanisms.

### Decisions carry proof

```ts
type Decision =
  | { allowed: true; proof: RelationTuple[] }   // the chain that granted access
  | { allowed: false; checked: PermissionKey; node: Node };
```

An allow **always** carries the chain that produced it. An unexplained allow is
unrepresentable — which is a strong statement, and it is what makes three features possible
rather than aspirational: **explain** ("why does this user see this?"), **view-as-user**
(render any screen as any principal, with real decisions), and the human-readable
**permission diff** that the merge checkpoint reads.

The proof also travels. When an operation emits, the kernel stamps onto the envelope which
permissions it checked and passed, and — when the allow came through a capability grant
rather than a role — a reference to the granting tuple. `explain` can re-derive the chain
later, but it cannot recover *which* grant was consulted at write time once the tuples have
moved since. That pointer is what the envelope keeps, so a mutation carries its own proof
of authority.

### Denials are recorded

An allow leaves its trace on the event it authorized. A denial has no event to ride, so it
is captured on its own: a row in the scope-local `_substrat_denials` table with the actor,
the permission, the node, the operation, the impersonation session, and the time.

This is the one moment where an actor's intent and the permission model visibly disagree,
and no other log witnesses it — the admin log records changes, the outbox records *allowed*
mutations.

The mechanical detail from chapter 3 matters here: because a denial rolls its operation
back, the row is written **after** the rollback, as its own write. Only an *enforced* denial
is recorded — one that went through `assertAllowed` and therefore carries the checked key
and node. A module's own hand-thrown `PermissionDenied` is its policy, carries no key, and
is left alone.

### Sharing at runtime

Everything above is administrative. The same capability grant has a second author, and it
is the user rather than the admin: **`ctx.grant` and `ctx.revoke`**, called from inside an
operation.

```ts
await ctx.grant(theirPrincipal, 'todo:read', { entityType: 'list', entityId: listId });
await ctx.revoke(theirPrincipal, 'todo:read', { entityType: 'list', entityId: listId });
```

Both are entity-required and **delegating** — the caller's own decision on that entity is
re-checked, so you cannot share what you cannot see. Both are transactional with the
operation: if it rolls back, so does the share.

This is the primitive for "share this record with a person", and neither alternative is it.
A `ctx.link` edge is permanent and not revocable at all. Org membership is revocable but
coarse — a whole org, not one record — and minting an org per domain row to obtain a revoke
is a known anti-pattern with an issue number attached to it.

`demos/todo/src/module.ts` shows both in one line each, and is worth reading before you
design any sharing feature.

### The permission checkpoint

New permission keys and role definitions are one of the two things an agent may never
self-approve. The mechanism: each vertical exports `MODULES` and `ROLES` from its seed,
`pnpm lint:permissions` renders `demos/*/PERMISSIONS.md` from those same objects, the file
is checked in, and CI re-emits with `--check`.

So a widened role cannot merge without appearing in the PR diff as a table of key →
description → which roles hold it. CI going red is what makes the reading unskippable; it
is not itself the approval. A human reads the diff.

---

## 7. Engines, verticals, composition

Chapter 1 named the three layers. This chapter is the middle and top ones in working
detail: what belongs in an engine, what belongs in a vertical, and the rules that keep
seven engines from becoming forty-two pairwise integrations.

### An engine is an invariant with no vocabulary

There are seven: work orders, invoicing, booking, protocols, invites, metering, absence.
Each one is headless. No screens, no words a user reads, no opinion about what you call
things.

What an engine owns is the set of statements that must never be false:

- a completed work order cannot return to scheduled;
- an exported invoice basis is immutable afterwards;
- a protocol entry is append-only;
- every mutation emits an event;
- every operation checks a permission.

The test for whether something belongs in an engine is not "is it reusable". It is: **would
it be a bug in any vertical for this to be violated?** Pricing is not that — one customer
rounds differently. State transitions are.

### A vertical is everything a person touches

Vocabulary, roles, pricing, screens, workflow, the shape of the API its own app calls. A
vertical composes engines the way an application composes libraries, except the library's
invariants are enforced from below rather than trusted.

The repo carries eight demo verticals. Two of them are the reference implementations. They
answer different questions, and they are references *because they are used*. A demo nobody
runs rots, however complete it looks.

- **`demos/todo`: the shape.** A record app built forward from its declared model, with
  per-list sharing by email, revoke, and a React app that tells a 403 from an empty list. It
  has **no engine at all**. Not every vertical needs one, and pretending otherwise leads to
  engines that exist to be composed rather than to hold an invariant. It is what a scaffolded
  project's docs point at.
- **`demos/ticket0`: a deployed vertical.** A support desk that runs in production. It composes
  the metering engine by call, runs sandbox-clean on the hosted runtime, sends and receives mail
  through a relay, answers with a model around its operations (chapter 4), and ships through a
  CI push. It is the one to read for how a real vertical meets the platform.

### Star topology: engines never call each other

**Diagram — the line.** Two bands either side of one boundary.

- **What an agent writes** — mistakes are cosmetic: screens, forms, workflows, reports, pricing, vocabulary.
- **the line.** Above: AI velocity — mistakes are cosmetic (a wrong screen). Below: humans + runtime guarantees — mistakes are catastrophic (a tenant leak).
- **What the substrate owns** — mistakes are catastrophic: tenancy, auth, migrations, integrations, audit, compliance.

The layer where models are weakest is the layer where mistakes are fatal. So the split is structural rather than instructional: prompting a model to "be careful with tenancy" is a suggestion, and a `getScope` call that fails closed on a mismatched pair is a fact.

No engine imports a sibling. None. Composition happens through three kernel-mediated
channels:

1. **Opaque refs** — an attachment contract binds to `(entityType, entityId)` without
   knowing what the entity is.
2. **Events** — an engine reacts to another's schema-versioned events. The invoicing engine
   consumes `workorder.completed` *and* `commerce.order-placed` — two different domains —
   without importing a single type from either producer.
3. **Vertical-owned orchestration** — a synchronous flow needing two engines is wired in the
   vertical, where the glue is visible and editable.

The arithmetic is the argument: compatibility stays at *N* kernel contracts instead of *N²*
engine pairs, and each engine versions independently.

There is a corollary test worth remembering: **if two engines need chatty synchronous
communication, they are one engine drawn wrong.** That is why "work orders + time
reporting" is one engine rather than two that call each other constantly.

And if a vertical finds it must *fork* an engine, the engine drew its line wrong. That is a
bug report about the engine, not a fork to maintain.

### By call, or by event — and it is a fact about the exports

An engine is composed in one of two modes, and which one it is determines its whole shape.

#### By call (work orders, protocols, booking, metering)

Operations are thin: the permission check, plus one exported in-scope function. All the
logic lives in composable exports, so a vertical wraps them inside its **own** transaction:

```ts
// in the vertical's operation, same transaction, own permission check
assertAllowed(await ctx.check('fsm:complete-job'));
await ctx.atomic(() => completeWorkOrder(ctx, { orderId, billable }));
await ctx.sql.exec('UPDATE fsm_jobs SET closed_at = ? WHERE order_id = ?', [ctx.now(), orderId]);
```

One transaction, both writes, atomic. The vertical extends by composition and never by
forking. Note the `ctx.atomic` from chapter 4 — it is what makes catching that engine call
safe.

#### By event (invoicing)

The vertical **emits**, the engine consumes, and the vertical reads results back through
the engine's own operations or by consuming its events into a side table keyed by the
engine's id.

There are deliberately **no in-scope exports**. That is not an omission — the engine being
the only writer of its rows is what keeps immutable-after-export safe from a half-finished
caller. If a vertical could call in mid-flow, the invariant would depend on the caller
finishing.

Which mode an engine is, is a fact about its exports. State it in the engine's header, so
an absence reads as intent rather than as something nobody got round to.

### Another module's tables are private

Not "please don't" — a linted boundary with a reviewable escape hatch.

Engine data is reached through the engine's exported in-scope functions. The stable surface
is entity ids, `EntityRef`s, and event payloads. A vertical needing extra data on an engine
entity adds its **own side table keyed by the engine's id**, and never a column upstream.

One-time extraction handoffs — a genuine migration where the data must move once — use an
explicit `boundary-lint-allow R5` … `boundary-lint-end R5` block. That is a reviewable
escape hatch rather than an absent rule, which is the right shape: rules with no hatch grow
workarounds that are worse than the hatch.

### Engine surfaces evolve additively

Three rules, and one of them has teeth that surprise people.

**New operation inputs are optional, with behavior-preserving defaults.** Adding a required
input is a breaking change to every existing caller.

**Permission keys are never renamed.** A rename silently removes authority from everyone
who held the old key.

**Emitted event payload fields are frozen once shipped.** Rename, remove or retype means a
`schemaVersion` bump — and chapter 5 explains why a bump is a *replace* with no dual-emit
window available.

#### The gate that makes a break red

There is one call site of every engine that used to be invisible to the compiler: the
scaffold template. It imports the engines, so it is a real call site — and nothing compiled
it, so a non-additive change could merge, release, and reach `npm create substrat` broken.

That is now closed. `tools/template-sync.mjs` materializes the template into
`packages/template-check`, a workspace member holding `workspace:*` links, so `typecheck`,
`test` and `boundary-lint` all reach it with no new command to remember. A non-additive
engine surface is now red **in its own PR**, against the workspace.

There is a second, separate gate — `pnpm lint:scaffold` — that runs the same scaffold
against the **published registry**, post-release and weekly, never on a PR. The distinction
is the whole point: between a merge that adds a surface and the release that publishes it,
the template legitimately runs ahead of npm. Being ahead of the registry is a pass on the
PR gate and a legitimate red on the release one.

### The runtime half: parse on the way out

A type-checked seam is not enough, because a vertical compiled against engine 0.3 and
*running* against 0.4 has no compiler in the room.

So a value crossing the engine seam is `.parse`d by the schema the engine publishes — on
the way **out** as well as in. And a read names its columns rather than using `SELECT *`,
which would pin the published shape to whatever the physical table currently holds.

Without this, the failure is not a throw. It is a field that moved, read as a different
field, rendering **wrong data on a screen**. Silently. That is why the parse is unconditional
including on bulk reads: dev-only validation is absent exactly where version skew lives.

Two helpers in `@substrat-run/contracts` do the work — `returns(schema, surface, value)` and
`columnsOf(schema)` — and an engine binds them to its own name in one line
(`engineSeam('engine-workorder')`). Every engine is converted, each with a `test/seam.test.ts`
that moves its tables underneath it and asserts a throw rather than wrong data.

The `SELECT *` half is mechanical: boundary-lint **R8** fires on a star read anywhere in an
engine's module code, with the same reviewable allow-block hatch for a maintenance read whose
row never leaves the engine. The `returns()` half is still convention — proving a
row-returning export is parsed needs a type checker the linter deliberately does not carry —
so a new engine is expected to do that half itself. Said plainly rather than implied.

### Declaring, not writing

A vertical increasingly does not hand-write its manifest. It declares entities, operations
and permissions once in a typed module, and the manifest's entity fragments, permission
list, event list and DDL are **derived** from that declaration.

The compiler checks the joins: a parent naming no entity, an `entityIdFrom` naming no output
field, a payload carrying an `erasable` field are compile errors rather than runtime
surprises. `pnpm lint:model --check` gates the emitted `model.json` the way
`lint:permissions` gates the permission surface.

The browser client is generated from the same declaration — entity interfaces, paths,
methods, request bodies, the paged link walk. What stays hand-written beside it is only what
the model does not declare: which principal a request is made as, and the error envelope the
vertical chose in its own `app.onError`.

[The model](https://substrat.net/concepts/model.md) is that story in full, and it is the page to read before
building anything.

---

## 8. The life of one deploy

A change is committed. This chapter follows it from a laptop to a scope's database.

The decision that shapes everything here is that **uploading code and serving it are two
acts, deliberately not fused.**

### Push uploads a version. It serves nothing.

```sh
substrat push
```

The CLI builds your worker locally, then POSTs the bundle plus a manifest to the control
plane, which records an **immutable version**: a `deploymentRef`, a permission digest, a
migration digest.

That is all. Nothing is serving the new code. A push is cheap and reversible — it adds a
row to a registry. The reason for the split is that "here is a new build" and "run this
build against real data" are different decisions, and a mistake in the second costs far more than a mistake in the first, and
fusing them means you can only ever make both at once.

Before the upload, the CLI runs the layer rules on the **source tree** — the same
`boundary-lint` that guards this repo, now shipped as a package — plus a permission
preflight that derives your declared surface and refuses a drifted one. Both run before the
wrangler build, because a refusal is worth more in a second than at the end of a build that
was going to ship broken.

The push carries more than code. It flattens what each module **declares**, including the events
it emits and consumes, its schedules and its freshness expectations, onto the version's manifest.
That is what lets the dashboard later draw declared behaviour against observed behaviour
(chapter 10). It also refuses a manifest that declares a binding in the `SUBSTRAT_*` namespace,
because the platform injects names there, such as the version id every event is stamped with,
and a vertical must not be able to shadow them.

That gate is newer than it sounds, and the gap it closed is worth knowing. Until it existed,
the mechanical rules this book has been describing ran **only in this repo's CI**. A vertical
developed anywhere else was built, uploaded and admitted having been checked by nothing,
which made the rules advisory for every real customer — the opposite of the claim made for
them. There is an honest bound on the fix, too: this runs on the builder's own machine, not
platform-side over the uploaded bundle, and `--skip-lint` exists. The flag says out loud that
the push was ungated, on the reasoning that a flag which silently weakens a gate becomes the
default in somebody's CI.

### Admission: may this code run here at all?

Before a version can be promoted, it must be **admitted**, and admission is answered
**mechanically** by the sandbox contract.

The bundle may declare only its own durable stores — a positive binding allowlist. No
`CONTROL_PLANE`. No platform secret. It runs inside a Workers-for-Platforms isolate, held
to quotas. Satisfy the contract and nothing else is in question.

The allowlist deliberately excludes egress-shaped bindings (`send_email`, `ai`, `browser`).
**Reaching the outside world is not a binding — it is a granted capability.** A vertical
that needs one *declares a request* in its manifest (`substrat.sendsEmail`,
`substrat.usesModels`, `substrat.provisions`), and something outside the bundle turns it on.
The request is refreshed on every push and grants nothing by itself; no push can set the
flag it needs.

What comes back differs by capability, and the difference is worth knowing:

- **A relay**, for a capability that rides a credential. Email is the reference: the
  vertical POSTs to the control plane — the one worker holding an outbound-mail credential —
  which sends on its behalf and re-checks the grant on every call. The vertical holds no
  credential and reaches no third party directly.
- **A real binding**, for models. `substrat.usesModels` is answered with an `ai` binding
  appended *after* the allowlist check has run on the declared set, so a vertical still
  cannot declare it for itself — only be handed one. It takes both halves: the platform
  willing to bind at all, and this version having asked.

Who must **vouch** for a version depends on who will be exposed to it:

- **A private vertical** — you own it, nobody else can install it — lands **admitted
  automatically**. There is no third party to protect, so the sandbox contract is the whole
  gate. An auto-admission note records that no human vouched, so the platform can still tell
  the two apart.
- **A listed vertical** — published, so *other* tenants run it against *their* data — raises
  a second question that no contract can answer, and its pushes land **pending** for staff
  admission.

The human gate did not disappear; it moved to the boundary where it means something.
**Publishing is the checkpoint, not every push.**

### One channel, called `prod`

A vertical has exactly one channel. There is no `dev` and no `staging`.

Those existed once and were write-only — nothing ever read or served them. The reasoning
is worth keeping: a channel names a *pointer at code*, and an immutable version id already
does that. What a real non-production environment needs is not a second pointer but a second
**scope with data**, which is a preview.

A promotion re-points `prod` at a version. It is a **rebind, not a rename**: the same scope,
the same data, serving new code.

Two things a promotion always respects:

**The surface checkpoint.** A promotion whose permission or migration digest differs from
what is live is **refused** until the diff is acknowledged. This is the same two-checkpoint
discipline the kernel applies to modules, applied at the deploy boundary, with the owner as
the human at their own checkpoint.

**Channel history.** Every promotion appends a row: what went live, what it replaced, who
did it, exactly when. That history is the dashboard's rollback picker, and each timestamp is
an instant a point-in-time rewind could restore data to.

### In place: the property everything else depends on

A version update is **not** a rebind to fresh, empty storage. A promote re-uploads the
bundle onto the vertical's one stable serving script; the scope's Durable Object and its
SQLite **stay put**.

Which means:

- **Data carries forward.** Nothing moves.
- **Migrations run forward against production data**, exactly as the migration model always
  assumed. Before in-place serving existed, that mechanism had never actually run in
  production.
- **Secrets survive.**

And migrations do not run in a fleet-wide deploy step. Each scope migrates **on wake** —
the `migrateAndRecord` call from chapter 3 — inside its own serialization domain, the first
time anyone touches it after the version moved. A thousand scopes do not migrate at once;
each migrates when it is next used, and a scope that fails to migrate fails closed and
serves nothing, which is what stops it from rendering as healthy.

Stragglers are not left to chance: the platform sweep reconciles migrations as its first
phase, walks scopes that are behind the frontier, and flags the ones that keep failing.
Chapter 9 is that pass.

#### A promote repairs its own installs

Each scope records the version its provision hook last ran against. Once a scope is bound to
a new version — by a promote or a per-scope bind, never by an upload alone — the platform
sweep re-runs `/internal/reconcile` on every active install whose receipt is behind.

So whatever a new release's `onProvision` mints reaches existing installs too. Which is
exactly why **that hook must be idempotent**: it runs more than once over a scope's life,
and a hook that assumes it runs at provision time only will do the wrong thing the first
time a release changes what it mints.

A version is badged **code-only** or **schema-change** at publish, so you know before you
promote whether the schema is involved.

### Previews: a fork of the data, bound to the new version

Before a promotion whose migrations changed, you want to see the new version run against
real-shaped data without touching production.

Substrat can do this because the scope-host contract already runs identical module code on
two adapters. A **preview** is a **fork** of a scope's data bound to the new version,
reachable at its own URL.

The governing law is that **migrations are forward-only**. A snapshot taken today, at
production's frontier, bound to today's or a later version, rolls its migrations forward on
the copy. Rehearse, throw the fork away if it breaks, and production never saw it.

The inverse is invalid: pointing *old* code at *today's* data across a schema change cannot
work, because you cannot un-run a migration. So the rollback strategy is
**fork-before-you-promote** — snapshot right before binding the new version, and a bad
version leaves the prior one still runnable at its correct frontier.

A fork is a **governed dead end**. The export copies the kernel spine too, so no connector,
no schedule and no billing consumes from a preview — a test copy cannot send a real invoice.
Preview URLs are non-public: gated code running against real-shaped data.

### Backout

Because every app is one scope with its own database, recovery is a per-tenant primitive
rather than an environment-wide runbook. One tenant's rewind has a clean, self-contained
blast radius.

**PITR rewind — the first-hours backout.** Right before an upgrade migrates, the scope
bookmarks the instant (Durable Object point-in-time recovery, roughly 30 days of history).
If a promotion goes wrong, an audited rewind rolls that scope's data back to the bookmark,
time-boxed to about 24 hours unless forced. It is a *destructive, in-place* rollback of one
DO — the opposite lever to a fork, which is a non-destructive copy. It rewinds the whole
database, so a stale bookmark is refused rather than silently skipped.

One sharp edge, stated because it will bite otherwise: **data and version binding live in
different DOs.** Rewinding *past a bad migration* means also rebinding the version — two
coordinated rewinds, with no atomicity between them.

**Backup / restore — the considered path.** `substrat scope restore` loads a backup into an
existing hosted scope, replacing its data: a pulled `.sqlite`, a local scope file, or a
dump. This is the deliberate recovery when a time-boxed rewind is not the right tool.

### The whole shape

Push uploads a version. Admission asks whether it may run here, mechanically when you are
the only tenant exposed and with a human when you list it. Promotion points the one channel
and serves **in place**, so data carries forward and migrations run forward against it, each
scope on wake. Previews rehearse a risky version against a fork. PITR rewind and
backup/restore are the backout.

The staff gate lives at **publish** — the moment other tenants can run your code against
their data — and nowhere else in the loop.

---

## 9. The two clocks

Everything so far has been driven by a request. This chapter is the work nothing asks for:
retries, recurring jobs, reconciliation, reaping.

There are **two** background clocks, they run in different places, they do different work,
and the reference describes them in two pages that never sit next to each other. Getting
them straight is most of understanding how Substrat behaves when nobody is looking.

### Why an alarm and not a cron

Both clocks ship as Durable Object **alarms**. Three reasons, and the first one is
decisive:

**A dispatch-namespace script gets no crons.** A hosted vertical pushed into
Workers-for-Platforms does not honour `triggers.crons`. A DO alarm is the only timer such a
deployment can own. Everything else follows from needing a mechanism that works there.

**No overlap.** A cron fires on a schedule regardless of whether the last run finished, so
two passes overlap the moment one runs long. Here the next alarm is set **only after the
current pass settles** — so the interval is a **gap between passes, not a rate**. A slow
pass delays the next one rather than racing it.

**No configuration.** An alarm self-arms from code. There is no `wrangler.toml` entry to
forget in one environment and remember in another.

The first reason is also the limit of the argument, and it is worth being exact about which
clock it binds. Clock one runs inside a dispatched vertical, so the alarm is the only timer
available to it and there is no choice to make. Clock two runs on the platform's own
deployment, which is an ordinary worker and *can* hold a cron — and the hosted control
plane uses one, firing the same pass every fifteen minutes. So the no-overlap property
above is the alarm's, not the pass's: on the cron path a long pass can still meet the next
tick, and it is the pass itself that has to tolerate that.

And the loop is built never to die. A pass that throws whole is still caught, still reported
through `onPass`, and the alarm is **still re-armed** — the alternative being workerd's own
alarm retry, which would tighten the interval rather than keep it.

Arming is idempotent. `ensureArmed()` does not move an alarm that is already set, so it is
safe and cheap to call on every request, which is exactly how it is kept alive. Where a cron
*is* available, pointing `scheduled()` at `ensureArmed()` makes the cron a safety net that
recovers a lost alarm, never a second thing that runs passes.

### Clock one: the scope sweeper

**Where:** inside the vertical's own deployment, where its modules and its scopes' data are.
**Singleton name:** `scope-sweeper`.
**What it holds:** a roster in its own storage — `scope:<scopeId>` → `tenantId`.

This is the clock a vertical cannot do without, and the one most people mean when they ask
"what retries my failed delivery?".

One pass walks the roster, eight scopes at a time by default, and does two things per scope:

**`drainDue(tenantId, scopeId)`** — the executor retry driver from chapter 5. This is the
outbox's backstop: connector deliveries and executor effects that failed and are now past
their `next_attempt_at`. The pass returns `attempted` / `delivered` / `retrying` /
`deadLettered` summed across the roster.

**`runDueSchedules(...)`** — the recurring work a module declared in its manifest. Two
properties make that safe to run unattended:

- It runs under a **system actor**. Emitted events read as `{ system: '@your/module' }`, not
  as whichever admin last touched the data. Modelling a nightly job as a signed-in user is
  precisely the audit-trail laundering this exists to avoid.
- **`ctx.check` is still the only gate.** A schedule's declared permissions are granted to
  the module's system principal at provision, so the operation's own `assertAllowed` resolves
  the same way it does for any caller. A schedule can do exactly what it declared and no
  more — and revoking that grant for one tenant turns the schedule off for them, with no
  special "disabled" flag, because the operation simply fails its own check closed.

`cadence` is a **floor, not a promise**: a schedule fires no more often than `everyMinutes`,
and the sweep is what actually runs it, so sub-sweep cadences round up.

#### The roster is maintained by the routes, not a directory

The sweeper does not enumerate anything. `/internal/provision` and `/internal/reconcile` call
`noteScope`, `/internal/delete-scope` calls `forgetScope`.

That is what lets a **CP-less vertical** — one with no control-plane binding at all — have
recurring work despite having no directory to read. It knows its scopes because it was told.

And the alarm follows the roster: **it lapses on an empty roster**, and `noteScope` restarts
the loop when the first scope arrives. A vertical with no scopes burns no alarms.

### Clock two: the platform sweeper

**Where:** the platform's own deployment.
**What fires it:** either trigger, over the same `runPlatformSweep`. The adapter ships
`definePlatformSweeperDO` — singleton `platform-sweeper`, the alarm loop clock one uses —
for a deployment that wants one. The hosted control plane instead points its `scheduled()`
handler straight at the pass, on a fifteen-minute cron.
**What it holds:** nothing — it reads the directory.

This is the fleet-wide maintenance pass. One pass, in this order, each phase recording
per-unit outcomes and **stepping over** failures rather than letting one sink the pass. Every
unit of work also leaves a durable **sweep run** record, which is how "this schedule never ran"
becomes a visible verdict rather than an absence (chapter 10):

**1. Migration reconciliation.** First, deliberately. The drain phase below wakes scopes,
and waking migrates lazily — so running this after it would count every failed scope's
attempt twice per pass. It walks scopes behind the deployed frontier, retries them with
backoff, and flags the ones that keep failing. Scopes it leaves in a failed state are
**skipped by every later phase**: they fail closed, so touching them would only re-throw the
same migration error as noise.

**2. Executor drain.** `drainDue` over every active scope — the same call the scope sweeper
makes, here for deployments the platform drives.

**3. Platform-intent drain.** Pull and execute each active scope's pending intents from
chapter 5. Injected, because the kernel cannot reach a vertical's DO — the control plane
supplies a function that goes over the vertical's `/internal` surface.

**4. Provision reconcile.** The phase chapter 8 described: re-run `/internal/reconcile` on
every active install whose provision receipt is behind the version it now serves.

**5. Recurring schedules.** For each vertical that declares them, enumerate live scopes and
invoke each due operation. A fork or a snapshot is **skipped**, so a test copy never runs
real recurring side effects.

**6. Freshness expectations.** Record a verdict on event types that were expected to have
been seen recently — the "nothing ran" signal, which is otherwise invisible because absence
produces no row anywhere.

**7. Snapshot GC.** Delete expired forks. A preview with a TTL disappears here.

**8. Reap long-archived scopes.** See below.

**9. Reap tenants past their grace window.** After the scope reap, so a tenant's scopes are
gone before its row becomes a tombstone.

**10. Event drain to the lake.** Ship each active scope's undrained outbox rows to the event sink
(Tier 2, chapter 11), up to 200 per scope per pass, then stamp them drained. With no sink bound,
the phase does not run and nothing is marked. Draining copies events and never prunes the
outbox.

**11. Access-log drain and prune.** Ship staff access rows to durable storage, stamp them
drained, then prune what was shipped. The sink must be **durable before it resolves** —
everything downstream treats a resolved ship as proof the evidence survives outside the
directory, so a sink that buffers and returns early turns a retention policy into data loss.

### Reaping: the part that is not automatic

This is the mechanism the reference mentions in a single clause, and it deserves the space.

**Cloudflare never garbage-collects a Durable Object.** An archived scope's bytes persist
forever unless something explicitly wipes them. There is no platform-level cleanup to
inherit, and "we archived it" is not the same sentence as "it is gone".

So the platform sweep can do it — and it is **opt-in**:

```ts
runPlatformSweep(host, {
  reapArchivedAfterDays: 90,   // UNSET ⇒ the phase is skipped entirely
  reapDeletingAfterDays: 30,   // same, for tenants
});
```

Unset, the phase does not run. The reap is irreversible, so a deployment must **name a
retention window** before the sweep will ever destroy anything. The default is to keep the
bytes and let a human decide.

Three details you will meet in practice:

- A scope with a **null `archivedAt`** — archived before that column shipped — has no
  knowable age and is **never** auto-reaped. It must be reaped by hand.
- `reapScope` re-checks the `archived` status below the seam, so a row that changed
  underneath fails closed there rather than being reaped on stale information.
- The retention reap **forces past the bound-hostname guard**. That guard exists to stop an
  interactive per-scope mistake; a retention policy on an already-archived scope is a
  deliberate decision, not a slip.

The tenant reap composes the same seam: for each of the tenant's non-reaped scopes, archive
if needed, then reap, then clear the tenant's PII and config directory rows. The tenant row
survives as a **tombstone** — audit history, and a burned slug — and the admin log is kept
whole, because it is the compliance witness and is never swept.

What ends up gone: the storage. What remains: the fact that it existed, who ended it, and
when.

### Node, for completeness

Self-hosting on Node, neither DO exists. A server calls `startPlatformSweeper` at boot and
gets the same pass on an interval. The unit of work is identical — `runPlatformSweep` is
kernel code that holds no timer of its own. Only the thing that fires it differs.

That separation is why the sweep is testable at all: a test calls `runPlatformSweep` directly
and asserts the report, with no clock involved.

### Reading the result

Both sweepers return a report rather than logging and forgetting. `retrying` and
`deadLettered` are the numbers a health surface should show; `errors` is per-unit, so one
failed connector reconciliation does not hide behind a green pass.

A caller that ignores those learns nothing, which is exactly the failure mode the older
silent drain had, and it is why the reports exist in the shape they do. The next chapter is
about the reading.

---


## 10. Seeing what happened

The last chapter ended on reports that background work returns so that something can read
them. This chapter is about the reading. It covers what the platform records about a running
vertical, where each record lives, and which question each one can answer.

There are four instruments, at four different grains, and the most common mistake is to ask
one of them a question only another can answer:

| Instrument | Grain | Exact or sampled | Answers |
|---|---|---|---|
| **Traffic** | one datapoint per request, at the router | sampled | how much, how fast, how many errors |
| **Invocation logs** | one line per request, written by the vertical | exact while retained | what this request did, and what it printed |
| **Traces** | spans the runtime produces | sampled, opt-in | where outbound time went |
| **The spine** | one row per mutation, per delivery, per denial | exact, forever | what happened to the data, who did it, and what it set off |

The first three are about *requests*. The fourth is about *meaning*, and it is the one that
cannot be rebuilt from anything else.

### Traffic: the router counts

Every request passes the router (chapter 3), so that is where it is counted. Once dispatch
returns, in a `finally` so errors count too, the router writes one Analytics Engine datapoint:

```
index:   [tenantId]
blobs:   [verticalSlug, scopeId, surface, statusClass, rayId]
doubles: [durationMs, status]
```

The index is the tenant, so every read of this dataset is a read of one tenant. The dashboard's
chart is a sampling-weighted `sum(_sample_interval)` with weighted quantiles for latency, run
through the Analytics Engine SQL API.

**The data is sampled.** It is the right instrument for "is this app getting slower", and the
wrong one for anything that has to be exact, such as an invoice (chapter 12) or proof that a
particular request happened. A sampled count is an estimate, and it presents itself as one.

### Logs: the vertical writes its own line

This is the instrument with the least obvious design, and the reason for it is a platform fact.

Workers observability is keyed on the **worker script**. A vertical's script serves *every*
tenant that installed it. And a trace does not cross the router's dispatch hop into the
vertical. So the router knows the tenant and cannot label the vertical's log lines with it,
and the vertical's log lines, unlabelled, belong to nobody in particular. A tenant's
Observability page would be either empty or someone else's.

So the vertical writes the line itself. One middleware, mounted first:

```ts
app.use('*', invocationLog({ routerSecret: (env) => env.ROUTER_SECRET }));
```

It writes one JSON line per request to Workers Logs, in a `finally`, so logging can never fail
the request:

```
substrat: 'invocation'
tenantId, scopeId, vertical, surface
method, path          // the path only; the query string is dropped, so no OIDC code is logged
invocationId          // a ULID minted for this request
status, threw, durationMs
```

Workers Logs indexes JSON fields, so `tenantId` becomes a filter. Reading a tenant's logs takes
two phases. The first finds that tenant's invocation lines. The second uses Cloudflare's own
per-request `$metadata.requestId` to fetch every other line the same request printed, including
a stack trace from code that has never heard of tenants.

Three details come from getting this wrong once:

- **The tenant comes from `readRoutedNode`, never from a header read directly.** A forged stamp
  would put text the forger chose onto another tenant's dashboard. So `routerSecret` is
  required: without it every assertion fails verification and **nothing is written**. From the
  outside that looks the same as no traffic, which is why a bare `invocationLog()` is refused.
- **It must be the first registration.** Hono composes handlers in registration order, so a
  mount below some routes logs part of the surface and stays silent for the rest, which reads
  as "no traffic on those routes". `lint:invocation-log` checks the *order*, not just that the
  mount exists. The scaffold template ships with the mount in place.
- **It is not retroactive.** Only versions pushed after a vertical adopted it write lines.

### Traces: opt-in, and narrower than the word suggests

Automatic tracing is set per push, from a sampling rate the control plane holds. Today that rate
is on in the test environment and off in production. Nothing in the platform emits a span per
operation, permission check or engine call. What reaches the trace data is what the runtime
generates on its own, mainly outbound `fetch` and the Durable Object entry hop. The platform
reads it for one purpose: comparing a version's *observed* egress with its *declared*
allowlist (chapter 8).

A per-operation waterfall is designed and not built. That is part of why the next section
matters.

### The spine, read as an instrument

Chapter 5 described the outbox as the audit log. It is also the best observability data the
platform has. Every row is exact, typed, stamped by the kernel, and kept. The kernel ships
**sanctioned reads** over it, and the dashboard puts each one behind a button, so nobody has to
write SQL against the spine to use it.

**History.** `readHistory` returns one entity's events, newest first, with the fields a hand-rolled
`SELECT` would misread: the authorization chain, the impersonation stamp, the operation, the
deployed version. In the dashboard, opening a record under an app's **Data** tab shows this
timeline.

**Why?** Every event emitted while a delivery is in flight records `causedBy`, the id of the
event whose consumer emitted it. `walkEventCause` follows that link backwards. It says how the
walk ended (complete, cut, depth limit, or a missing link), so a short chain is never mistaken
for a complete one.

**What did it do?** `walkEventEffects` goes forward. It builds a tree from the delivery journal:
which consumers an event reached, whether each one handled it, is retrying, or gave up, and how
many attempts it took. This is where `_substrat_deliveries` shows up in the dashboard. It is not
a raw table of deliveries but the delivery rows hanging off the event that caused them.

**Same call.** This joins a data change to a request. The vertical host mints the
`invocationId` that the log line carries, passes it into `invoke`, and the kernel stamps it
on every event the call emits, including events that consumers emit in that call's
post-commit tail. `readInvocation` returns everything one call did. The same id is on the log
line, so a record's call can be matched to the request that made it. That match is by hand
today: the Logs view does not yet filter by invocation id.

That join has limits, and they are exact:

- It exists only for operations reached over the vertical host's HTTP operation route. Seeds,
  internal calls and events from before the column existed carry `null`.
- The router's datapoint carries Cloudflare's `rayId`, not the `invocationId`, so traffic does
  not join to events.
- **Denials and deliveries carry no invocation id.** A delivery joins through its event, and a
  denial joins to nothing but its time and actor.

**Facets.** `facetEvents` is a small group-and-count over one scope's outbox. It filters by type
and time window, groups by type, actor, operation, version, entity type, PII class, invocation,
or one payload field, and returns counts per bucket. Each bucket carries `lastSeen`, the time of
its latest event. That tells a path that **stopped** (it has old events) from a path that
**never ran** (it has none). Erased payloads are counted separately rather than folded into a
"null" bucket. This is the **Events** explorer.

**Flow.** `substrat push` records what each module *declares* it emits and consumes. The dashboard
compares that with what the outbox *shows* happening. Triggers lead to modules, modules to event
types, event types to the outside world, and anything declared but never observed is drawn
dashed. Next to it, per-operation health joins emitted events with the denial log, grouped by
operation. That is where a role missing a permission key shows up before a user reports it.

### Absence is a signal too

Everything above records things that happened. The hardest failure to see is a job that never
ran, because that failure leaves no row anywhere.

Two mechanisms produce the row:

- **Freshness expectations.** A manifest can declare
  `freshness: [{ eventType, within: { hours } }]`. The platform sweep (chapter 9) checks each one
  and records a verdict. "No `invoice.sent` in 26 hours" becomes a fact instead of a hunch.
- **Sweep runs.** Every unit of sweep work is recorded, whether it was a schedule, a freshness
  check or a connector poll. The Schedules panel turns those records into four verdicts: *On
  schedule*, *Overdue*, *Never run*, and *No sweep data*. The last two are deliberately different.
  Module code cannot forge these rows: `ctx.requestPlatform` refuses the kind the platform uses
  to write them.

### Where each one surfaces

For a customer's admin, in the **Dashboard**:

- **Observability** (a team page, filtered by app): *Traffic* (the router's data), *Health*
  (worst first), *Logs* (invocation lines), *Events* (facets), *Schedules* (sweep runs), and
  *Flow* (declared versus observed, plus per-operation health).
- An app's **Data** tab: a record's history, with *Why?*, *What did it do?* and *Same call* on
  each event.
- An app's **Settings → Integrations**: the platform-request journal, meaning what the platform
  did with each intent the app raised.
- An app's **Overview**: a status band, a traffic sparkline, and release markers on the chart.

For Substrat operators, in the **Console**: the fleet views, plus *Operations → Failures, Issues
and Sweeps*. Failures are grouped into issues by fingerprint, so a thousand identical errors
appear as one row with a count.

One property holds across all of them. **Every read that reaches into a scope on a tenant's
behalf leaves an access-log row** that names what was read. Looking at data is itself recorded,
which is where the next chapter starts.

### What this does not give you

- **A raw outbox or delivery-journal browser.** Events are reached through facets and through
  records, and deliveries through the event that caused them. The spine's tables appear under
  *System* in the table browser, but there is no dead-letter list across a scope. Finding every
  consumer that gave up this week is a query nobody has built into a view.
- **Exact request counts.** Traffic is sampled.
- **Cross-scope event questions.** Every read above is one scope. "Across all forty branches" is
  forty reads, or it is the lake.

---


## 11. The audit trail and the lake

"Audit log" is one phrase for several records, kept in different places, by different code,
under different retention. Asking which one is *the* audit log is how a compliance question
gets a confident wrong answer. So this chapter names each witness, then follows the one that
leaves the scope: the event stream, drained into a table that SQL can query across the whole
fleet.

### Four witnesses, and what each one saw

| Record | Lives in | Written by | Witnesses | Kept |
|---|---|---|---|---|
| **The outbox** | each scope | the kernel, inside the operation's transaction | every *allowed* mutation, with actor, authorization chain, impersonation, operation, version, and the call it belonged to | for the life of the scope, and copied to the lake |
| **The denial log** | each scope | the kernel, after the rollback | every *enforced refusal*: actor, permission, node, operation, impersonation | for the life of the scope |
| **The admin log** | the control-plane directory | every audited `HostAdmin` verb | every *privileged change*: who provisioned, suspended, granted, admitted, rewound, reaped, with before and after | **never swept**, and it outlives a reaped tenant |
| **The access log** | the directory, then R2 | every staff or delegated read | every *look*: who read what in which tenant, with how many rows | drained to R2, pruned per retention |

The delivery journal (chapter 5) is a fifth record, but it is an operational one. It says what
happened to an event, not who did something.

Read the table by columns rather than rows, and each record covers a gap the others leave:

- The outbox cannot see a refusal, because a refused operation rolls back its own events. That
  is why denials are written separately, after the rollback.
- Neither scope-local record can see a change made *to* the scope from outside, such as a
  suspension, a role grant or a rewind. The admin log exists for that.
- None of the three records a *read*. When a staff member or the dashboard opens a tenant's
  records, the access log is the only witness. It names the method, the tenant, the scope and
  the parameters, so "who looked at this customer's data" has an answer. Nothing in the
  product shows it yet. It is answered from the directory or from the R2 copies.

### The admin log, precisely

Each row holds `actor`, `action`, `tenant_id`, `scope_id`, `vertical`, `before`, `after`,
`caused_by` and `at`. When an executor wrote the row, `caused_by` is the event that set it off
(chapter 5), so an automated change links back to the domain event behind it.

Two properties are worth knowing exactly.

**It is never swept.** Grant provenance and bookkeeping-law retention both depend on it, so it has
no TTL and no retention phase. Its growth is handled by bounding *reads*: the HTTP surface pages,
and a cursor walks the whole thing. When a tenant is reaped, its scope data and personal
directory rows are destroyed and its admin-log rows are **kept**. The record of what was done to
a tenant has to outlive the tenant.

**The row follows the mutation, in a separate write.** An audited verb commits its change and
then writes its row. A failed log write fails the verb, but the change already stands, and a
crash between the two leaves a change with no row. That window is accepted and documented rather
than hidden. Two verbs deliberately reverse the order, because for them the missing row is the
harmful half-state: a point-in-time rewind records its intent *before* it destroys anything, and
an impersonation session is logged before the session is handed back.

In the dashboard this is the **Audit** page. It is a team page rather than an app tab, because
some entries, such as role changes and entitlements, name no app at all.

### The access log: looking is an act

Every read the control plane makes into a scope on someone else's behalf writes a row. That
covers the history, facet, cause, effect and same-call reads from chapter 10 and the denial
summary. This includes the dashboard's own delegated reads. A tenant's admin opening
a record in their own app is recorded, not just staff doing it.

The access log does not stay in the directory. The platform sweep's last phase ships unshipped
rows to R2 as newline-delimited JSON, stamps them as drained, and only then prunes them from the
directory. The order is the whole design. The sink must be durable before it returns, because
everything downstream treats a returned ship as proof the evidence exists outside the directory.
A sink that buffered and returned early would turn a retention policy into data loss. The R2
copies are pruned only if a retention period has been configured.

### Tier 2: the event stream leaves the scope

Chapter 2 made a trade: every scope is its own database, so no query crosses scopes. For
operational reads that is the right trade. For history it is a real cost. "Every invoice basis
exported across all tenants last quarter" should not be ten thousand reads and a fold.

So the platform keeps three tiers of data:

1. **Operational**: each scope's own SQLite. Live, exact, one scope at a time.
2. **Exact history**: every event from every scope, in one table, queryable with SQL.
3. **Telemetry**: sampled request data (chapter 10). Cheap, approximate, never evidence.

Tier 2 is **the lake**.

#### How an event gets there

```
_substrat_outbox (each scope)
  │  platform sweep, event-drain phase: read undrained → ship → stamp drained_at
  ▼
control-plane worker ── EventSink ──▶ Cloudflare Pipelines stream
                                          │  INSERT INTO sink SELECT * FROM stream
                                          ▼
                                 Iceberg table kernel.events, in R2 (Data Catalog)
```

Tracing each hop:

**The drain is a sweep phase.** It runs after the reaps, before the access-log drain, over every
active scope, up to 200 events per scope per pass. A scope with more reports itself `incomplete`
rather than looping, and the next pass continues. The control plane reads undrained rows over
the vertical's `/internal` surface, ships them, and stamps them drained.

**Read, ship, stamp is at-least-once, on purpose.** Stamping before shipping would lose events
on any crash in between. Stamping after means a crash re-ships, so the lake can hold the same
event twice. Every row carries its event id, which makes the duplicate recognisable: **dedupe on
`id` before counting anything.**

**Draining copies, it does not move.** The outbox is never pruned after a drain. The scope still
holds its full history for `readHistory` and every chapter 10 view. The lake is a second copy,
built for cross-scope questions.

**The schema is generated.** The lake's columns are derived from the outbox DDL by a tool, and
`lint:lake-schema` fails CI when the two disagree. So an envelope field added to the kernel
cannot silently fail to reach the lake. The pipeline's SQL is `SELECT *` for the same reason: a
record whose job is completeness should pass a new field through or fail loudly, not drop it
without a word.

```
id, type, schema_version, occurred_at,
tenant_id, scope_id, actor, entity_type, entity_id, pii_class,
subject_id?, payload?, authorization?, impersonation?,
operation?, version?, caused_by?, invocation_id?,
bytes
```

`drained_at` is not shipped, since it is a fact about the scope, not the event. `bytes` is added
by the shipper and is the row's serialized size. Chapter 12 explains why that column exists.

**Files roll on size or time**: parquet with zstd, at most 100 MB or 300 seconds, whichever comes
first. At real outbox volumes the timer always wins, so history reaches the lake about five
minutes after the drain ships it. The sweep itself runs every fifteen minutes. Freshness is
Tier 1's job.

#### Rebuilding it

A lake table can need recreating, for example after a schema change. Because the outbox is never
pruned, nothing is lost. **Redrain** clears the drained stamp on events stamped before a given
instant, so the next sweeps ship them again. It is a staff-only, audited admin verb, and the
script that drives it works one scope at a time in bounded batches.

Provisioning the lake is deliberately **not** automated in CI. The script that declares the
bucket, table, stream, sink and pipeline runs as the operator's own `wrangler login`, checks the
live configuration for drift, and refuses to recreate a table that holds snapshots unless the
operator names that exact table as the one whose history they are discarding.

### Querying it: R2 SQL

The table is Iceberg in R2, so R2 SQL reads it directly. Here is the query the platform uses to
measure per-tenant event volume, with duplicates removed first:

```sql
SELECT tenant_id, SUM(bytes) FROM (
  SELECT DISTINCT tenant_id, id, bytes FROM kernel.events
) GROUP BY tenant_id
```

The same shape answers audit questions that no single scope can: every event a given actor
produced across every tenant, every change made under impersonation in a month, or every call
from a version later found to be faulty.

Here is what **is not built**, stated where you would otherwise assume it was:

- **No query gateway.** Nothing in the product runs R2 SQL on anyone's behalf. A tenant cannot
  query its own lake rows, and no dashboard or console view is backed by the lake. Today the
  lake is queried by whoever holds read credentials for the bucket, which means operators.
- **No tenant scoping on reads.** Every tenant's events share one table, and the sink cannot
  partition, so a tenant-scoped read would have to be enforced by the missing gateway.
- **No erasure in the lake.** Payloads ship as they are in the outbox, with `subject_id`
  alongside. That column exists so "delete every row for this subject" can be expressed as a
  query, but nothing runs it yet. Erasure is handled in Tier 1. **Until lake erasure is built,
  an erased subject's payloads survive in Tier 2**, and a deployment that drains to a lake has
  to account for that.
- **No lake retention policy** is configured. Snapshot expiry and compaction are the catalog's
  defaults.

### Exports

A tenant's data can be exported in full: every scope's dump, plus directory rows and, in
unmasked mode, the tenant's admin-log entries. That is the portability export. It is a staff
route today, not a self-service one.

### Denials stay home

The denial log has a `drained_at` column and no drain. Denials stay in their scope. They are
readable per scope, summarized by operation in the dashboard's Flow view (chapter 10), and
listed in the console's per-scope panel, but they do not reach the lake. A fleet-wide
"what was refused" question is still a fold over scopes.

---


## 12. Metering and billing

Two different subjects share these words, and mixing them up leads to wrong designs in both
directions:

- **A vertical billing *its* customers.** A support desk charging for AI answers, or a
  field-service firm invoicing hours and parts. That is domain logic, so it belongs in
  engines and verticals.
- **The platform billing a *tenant* for using Substrat.** Apps installed, engines licensed,
  models called, storage held. That is platform logic, so it belongs in the control plane.

They share one principle, and it is the thread through this chapter: **count exactly, price
separately, and never let a price change a count.**

### Part one: a vertical meters its own customers

#### Quantities, not prices

The metering engine records *how much*. It has no currency, no rate, no plan, and no idea
what anything costs. That line is drawn deliberately. One customer rounds differently, another
offers a volume discount, a third changes price mid-month. None of those decisions can be
wrong for *every* vertical, so by chapter 7's test none of them belongs in an engine. What
would be wrong for every vertical is losing a unit or counting one twice, so that is what the
engine owns.

A **meter** has a key, a unit, and a kind:

- a **counter** sums what is recorded (tokens used, messages sent);
- a **gauge** records a level and takes the maximum within a window, carrying the last level
  forward into windows where nothing was recorded (seats, stored documents).

A meter's kind and unit are **frozen** once set. A counter that later became a gauge would
reinterpret every entry already written.

#### Recording, and the invariants that make it safe

```ts
// inside the vertical's own operation, in the same transaction as the work
recordUsage(ctx, {
  meter: 'ai.tokens.output',
  qty: String(outputTokens),
  subject: { entityType: 'conversation', entityId: conversationId },
  dedupeKey: `${turnId}:out`,
});
```

The engine is composed **by call** (chapter 7), and that choice matters here more than
anywhere. The usage entry commits in the same transaction as the work it measures. There is
no gap in which an answer was delivered and its tokens were not recorded, or the tokens were
recorded for an answer that rolled back.

What the engine refuses:

- **A replay with the same dedupe key** returns the original entry, marked as deduplicated,
  and emits nothing. **The same key with a different quantity** throws. That is not a
  replay but two different claims about one event, and one of them is wrong.
- **An entry dated inside a closed period** throws. Once a period is closed, its numbers
  cannot change underneath an invoice built from them.
- **An entry dated more than five minutes in the future** throws. The close horizon only moves
  forward, so an entry dated past a close that has not happened yet could end up counted by no
  period at all.
- **A negative gauge**, or an entry against an inactive meter, throws.

#### Closing a period

`closePeriod` totals every meter over a window and freezes the result as **period lines**.
Closes are monotonic: a new period may not overlap one already closed. It emits
`metering.period-closed` with the lines in the payload, fat as chapter 5 requires, so a
consumer never has to read back. **Nothing schedules a close.** When a period ends is a business
decision (calendar month, contract anniversary, on demand), so it belongs to the vertical.

#### Where the price comes in

The vertical holds the rate card, in its own table, with its own permission. The support-desk
reference vertical stores rates per meter with an effective date, and prices usage for display
by joining each entry to the rate in force when it was recorded. A rate change next month does
not reprice last month.

#### Handing it to invoicing

Invoicing is the other half, and it is composed **by event**. A vertical or engine emits a
billable event, and the invoicing engine consumes it into the customer's single open **billing
basis**: the set of lines a real invoice will be issued from. Today it consumes three events:
a completed work order, a placed commerce order paid by invoice, and a closed timesheet period.

Each line copies its price from the event payload at write time, so last month's work is never
billed at today's price. A basis holds one currency, checked on every write. Once exported it is
**immutable**, which is why invoicing has no in-scope exports (chapter 5): nothing may call in
halfway through and change a basis the accounts already hold.

Two gaps are worth stating plainly:

- **Metering does not feed invoicing yet.** Invoicing does not consume `metering.period-closed`.
  The intended shape is for the vertical to price the closed lines with its own rate card and
  emit them as a billable event. On the invoicing side that is an additive new consumed event,
  designed and not built.
- **Nothing consumes the export event.** No accounting connector takes an exported basis today.
  The one accounting connector that exists reads bookkeeping *from* the ledger and has no write
  path.

### Part two: the platform meters a tenant

#### Meter, don't bill

The platform's rule is **count everything exactly, store no bill**. Usage is recorded as
facts. Prices and margins are applied when someone reads, and nothing stores a computed
charge. So a pricing change never requires a data migration, and a count is never tangled
with a number that was a commercial decision.

The commercial design names four meters. Here is the state of each:

| Meter | Counts | State |
|---|---|---|
| **1. Base fee** | active tenants and active scopes | counted, live |
| **2. Engine licensing** | entitlements, grouped by key and plan, with expired ones counted apart | counted, live |
| **3. Usage** | platform-provided model calls, with tokens and list price | **counted**, and priced when read |
| | event history retained | measurable in the lake, and nothing reads it yet |
| | storage, API calls | **not counted** |
| **4. Network transactions** | orders flowing between tenants | not countable, because that flow does not exist |

Meters 1 and 2 are folds over the directory, computed on read and never stored. Operators see
them, and model usage, in the console's **Meters** view, with a per-tenant card on each tenant.

#### Entitlements: the licence, not the enforcement

An entitlement is a row naming a tenant and a key, optionally with a plan, a quota and an
expiry. The kernel enforces **presence and expiry**: a module whose key the tenant does not
hold is not registered, so its operations do not resolve (chapter 3). `plan` and `quota` are
**expression only**. A module can read them through `ctx.entitlement(key)` and act on them,
but **nothing in the platform enforces a quota**.

#### Model usage, end to end

Platform-provided models are the one part of usage that is complete from call to price:

1. A vertical calls a model through the **model host**. That happens *around* an operation, never
   inside one, because a model call is a multi-second network round trip (chapter 4).
2. The host resolves the provider with platform-held credentials, makes the call, and builds a
   **usage line**: token counts, the list price computed on the platform's side from a generated
   rate card, and five attribution keys (tenant, scope, vertical, version, operation).
3. The vertical records that line through its own operation, which raises it as a platform
   intent, atomically with whatever the answer changed.
4. The platform drain checks that the line's tenant, scope and vertical match the scope it was
   drained from, so a vertical cannot bill a model call to someone else, and writes it to the
   directory. The write is idempotent on the request id and kept for 400 days.
5. When staff read the summary, a margin is applied, 20% unless configured otherwise. Nothing
   stores the result.

Two numbers exist, and both are correct. The platform's list price plus margin is what the
platform charges the tenant. The vertical's own rate card is what the tenant charges *its*
customers. Reconciling them is the tenant's business.

One known hole: a vertical that holds the models binding can call it directly, outside the
model host, and that call is not metered.

#### Storage: the honest answer

**Nothing measures storage used by a scope or a tenant today.**

Here is what exists nearby, and why none of it is the answer:

- The Durable Object SQL API reports a database's size. Nothing collects it. Reading it means
  waking each scope, and no sweep phase does that yet.
- Attachment rows record their own byte size. Nothing sums them, and attachment bytes live in a
  blob store rather than in the scope.
- **The lake's `bytes` column** (chapter 11) records every event's serialized size as shipped.
  Summed per tenant after dedupe, it is exact, and it is the only per-tenant volume measure the
  platform has. But it measures **event history volume**, not database size. It excludes a
  scope's current rows, its indexes, its attachments, and every row an update overwrote. It is a
  defensible basis for billing history retention, and it is not a storage meter.

So a storage line on an invoice needs one of two things built: a sweep phase that collects each
scope's database size into the directory next to the other meters, or a decision that
event-history bytes are the unit being sold.

#### Requests are not billable either

The router's per-request datapoints (chapter 10) are sampled. They are good enough to draw a
chart and not good enough to put on an invoice. And a read emits no event, so the spine cannot
count API calls either. That is why "API calls" sits uncounted next to storage.

#### What a tenant sees

The dashboard has a **Billing** page, and today it is a placeholder, marked as not yet enabled.
No payment provider is integrated, no invoice is generated, and nothing takes payment. The
platform counts, and the operator bills from those counts outside the product.

### The shape, once more

In both halves the count is exact, deduplicated, and written in the same transaction or intent
as the thing it counts. The price is applied later, by whoever owns the commercial decision:
the vertical for its customers, the platform for its tenants. What is missing is also the same
in both halves: the hand-off from a closed count to a document someone pays.

---

## 13. Operating it

The last twelve chapters were how the system works and what it records. This one is how it
behaves when something is wrong, which is a different subject.

### Four surfaces, and three more

The pieces that turn "a vertical" into "a vertical serving a customer at a hostname" are four
separate deployments.

| Surface | Audience | Answers |
|---|---|---|
| [Control plane](https://substrat.net/platform/control-plane.md) | the platform | the shared directory every vertical registers against — tenants, scopes, roles, entitlements, the admin log |
| [Console](https://substrat.net/platform/console.md) | Substrat operators | *run the platform* — the whole fleet, every tenant, provisioning, the audit log |
| [Router](https://substrat.net/platform/router.md) | inbound traffic | `hostname → (tenant, scope, surface)`, then dispatch |
| [Dashboard](https://substrat.net/platform/dashboard.md) | a customer's admin | *run my org* — their tenant only |

The split that matters most is **Console versus Dashboard**: the same platform, opposite
audience and opposite blast radius. The Console is the operator's back office — all tenants,
staff SSO. The Dashboard is the customer's home — one tenant, customer sign-up. Neither is
called a "portal", which is reserved for a *vertical's* own end-user surface.

The Dashboard is itself a Substrat vertical, which is the useful kind of dogfooding: the
tenancy model either works for the platform's own product or it does not.

Its shape is worth knowing before you go looking for something in it. Each app has **four
tabs**: *Overview*, *Deployments*, *Data* and *Settings*. Anything about more than one app is a
team page in the left menu, narrowed with an app filter: *Observability* (chapter 10) and
*Audit* (chapter 11) sit under **Operate**, next to Domains, Integrations, Team and Billing under
**Configure**. The rule is deliberate: a fifth tab on the app is almost always a filter on a team
page instead.

Three more deployments sit beside those four:

- **The builder studio**: where a vertical is written with an agent, against its own workspace
  and snapshots.
- **The egress worker**: the hop every outbound call from a dispatched vertical takes, which
  enforces the version's declared allowlist (chapter 8).
- **The hosted issuer**: an OIDC provider that verticals can sign users in against. It runs as
  a vertical, with its own admin console and sign-in log, rather than as a special platform
  component, because authentication is a seam and not something the kernel owns (chapter 6).

### The lifecycle states, operationally

Chapter 2 listed them. Here is what each one *means* when you are the person deciding.

| State | Serves traffic | Data | Reversible |
|---|---|---|---|
| `provisioning` | no | being created | — |
| `active` | yes | live | — |
| `suspended` | **no**, fails closed | intact | yes |
| `archiving` → `archived` | no | intact | yes |
| `reaped` | no | **gone** | **no** |

Tenants have the parallel ladder, where `deleting` is the reversible grace state that makes
every scope beneath it inert without reclaiming anything.

Two things to internalise:

**Suspension is the live weapon.** It takes effect on the next request, because the router
does not cache routes and `getScope` re-checks the directory. When a customer must stop being
served *now*, this is the lever, and its immediacy is paid for with a directory read per
request.

**`archived` is not `deleted`.** The bytes are still there and cost money. Moving to `reaped`
is what frees them, it is irreversible, and — per chapter 9 — nothing does it automatically
unless a retention window has been configured. If your mental model is "archiving cleans up",
it does not.

### Four failures, four instruments

Backup is where people's mental models are usually wrong, because "backup" is four different
things here.

| What is lost | What covers it |
|---|---|
| A scope's data, wrongly changed | **Durable Object PITR** — ~30 days, continuous, per scope. A destructive rewind of the live app. |
| A scope, reaped | **The copy the reap takes first.** A reap is irreversible, so the control plane stores a full-fidelity dump *before any byte goes* and records its address on the admin-log entry. |
| A scope, wanted elsewhere | **`scope pull` / snapshots** — a non-destructive copy that leaves the live app alone. |
| **The directory itself** | **The scheduled directory backup.** |

That last row is the one the others cannot cover, and it is worth understanding why.

The directory is a single Durable Object. Lose it and **no scope can be found**, even though
every scope's data is sitting there completely intact. PITR does not help: PITR rewinds a
database that still exists, and having nothing to point it at is a different failure. And no
scope can rebuild the map from below, because a scope does not know its own tenancy, hostname,
or bound version.

So the control plane backs itself up: a scheduled dump of the whole directory — tenants,
scopes, hostnames, verticals, entitlements, identities, and the audit log — to an object store
outside the DO, once a day, keeping 30 copies.

Two details in that design are worth borrowing for anything similar. The cadence is derived by
reading the newest stored copy rather than from a separate timer, so a missed run is caught up
on the next pass rather than skipped. And retention is pruned only **after** a new copy lands,
so a failed backup can never be the thing that deletes the last good one.

RPO ≤ 24h, RTO ≤ 1h for a directory loss, and the restore is rehearsed rather than merely
designed — the round trip runs in the contract suite against both adapters: capture, diverge,
restore, keep serving.

**Restoring replaces.** The dump's contents *become* the directory; anything created since is
gone. A merge would silently interleave two histories of the same tenant, so replace is the
honest semantic, and the restore refuses a directory that still holds tenants unless the
request explicitly says to overwrite — a guard for the realistic hazard of a restore replayed
against a control plane that has already recovered.

What it does **not** cover, stated plainly: the backup bucket lives in the platform's own
account. This survives losing the directory; it does not survive losing the account. And a
restore does not bring back what was never in the directory — the staff roster, worker
secrets, or the key any sealed credential was sealed with.

### Where to look when something is wrong

Chapters 10 and 11 describe the instruments. What matters when something is wrong is which one
witnessed the thing you are asking about, because reaching for the wrong one is the usual
reason an investigation stalls:

| Question | Instrument | In the dashboard |
|---|---|---|
| Is it slow, or failing, for everyone? | router traffic (sampled) | Observability → Traffic, Health |
| What did this request do, and what did it print? | invocation log | Observability → Logs |
| What happened to this record, and who did it? | the outbox, via `readHistory` | the app's Data tab → a record |
| Why does this event exist? | `causedBy` | a record's event → *Why?* |
| Did the consumer ever handle it? | the delivery journal | a record's event → *What did it do?* |
| Everything this request changed | `invocation_id` | a record's event → *Same call* |
| Has this job stopped running? | sweep runs, freshness | Observability → Schedules |
| Why was this person refused? | the denial log | Observability → Flow (per-operation health) |
| Who changed this app's configuration? | the admin log | Audit |
| Who looked at this tenant's data? | the access log | no view yet: it drains to R2 (chapter 11) |
| Did the platform act on what the app asked for? | the platform-request journal | Settings → Integrations |

Add to those the sweep reports from chapter 9 (`retrying`, `deadLettered`, and per-unit
`errors`), which are the signal that background work is degrading rather than failing loudly.

### A short diagnostic index

Symptoms whose cause is rarely where you first look:

**"The deploy went green and production is unchanged."** A push is not a deploy. Check whether
the version was promoted, and whether the scope is bound to it.

**"One tenant is broken, everyone else is fine."** Very likely a failed migration on that
scope. It fails closed and serves nothing by design. The migration-reconciliation phase records
why; the scope's `migrationFailure` carries the version and the error.

**"The event fired but nothing happened."** Open the event's *What did it do?* tree, which is
the delivery journal. If the consumer is
in-scope, remember from chapter 5 that it **does not retry** — one failure is terminal for that
pair, and waiting will not fix it. If it is an executor, check `next_attempt_at`: it may be
backing off, or already dead-lettered at `maxAttempts`.

**"The app asked for something and it took a quarter of an hour."** The platform intent waited
for the sweep. The vertical is not flagging its responses, so the router never kicked a drain.
Wire `onPlatformRequests` when minting the stub (chapter 5). If the intent never settles at
all, find it in the platform-request journal (Settings → Integrations for a connector's
intents). A handler that keeps throwing leaves the intent pending for about a day and then fails
it, with the last error kept.

**"Observability shows no traffic for an app that is clearly in use."** Either the version
predates the invocation log, or the log is mounted below some routes, or it has no
`routerSecret` and verifies nothing. All three produce silence rather than an error (chapter 10).

**"Recurring work stopped."** Check the scope sweeper's roster. The alarm **lapses on an empty
roster**, and the roster is maintained by `/internal/provision` and `/internal/delete-scope`
rather than read from a directory. A scope that never called `noteScope` is not swept.

**"Permission denied and I do not know why."** The denial log gives the key and the node.
`explain` gives the chain. If the permission is tenant-level, remember it reaches the scope by
**projection** — and an absent projection is a deny, which is the safe direction but also a
real cause.

**"It works locally and fails deployed."** The adapters are contract-equivalent, so suspect the
things that are not module code: the router secret, a binding, an unset platform secret, a
grant that was never made. Both trust boundaries fail closed, which means the symptom of a
missing secret is a blanket refusal rather than a subtle misbehaviour.

### What is honestly not solved

Ending on the gaps, because a book that implies completeness is worse than one that names the
edges.

- **The directory is a single control-plane DO.** The tenant-root-DO plus global-D1 split is
  designed and not built. It is the obvious scaling and blast-radius question.
- **Per-jurisdiction DO ids are not built.** `eu` and `us` name guarantees whose enforcement is
  not there yet, which is why provisioning gates them. `global` is what every scope is today.
- **In-scope consumers do not retry.** Deliberate for v0, and worth re-deciding as verticals
  grow consumers that fail for transient reasons.
- **Dual-emit across a schema version is not available.** Postponed with a design, not
  abandoned — but no plan should promise a deprecation window until it exists.
- **Cross-scope reads are a fold, not a join.** Fleet-wide questions cost a read per scope.
  Projections and per-tenant D1 are the current answers, and the many-scope fan-out cost of
  permission projection for a very large tenant is an explicit open question.
- **The lake has no query gateway, no tenant scoping and no erasure.** Tier 2 is filled and
  queried by operators. A subject erased in Tier 1 still has payloads in Tier 2 until lake
  erasure is built (chapter 11).
- **Storage is not metered, and quotas are not enforced.** The platform counts installations,
  engine licences and model calls. It cannot yet say how many bytes a tenant holds, and an
  entitlement's quota is information a module may act on, not a limit the platform applies
  (chapter 12).
- **Denials and deliveries carry no invocation id**, so "same call" reaches a request's events
  but not its refusals.
- **Executors hold the scope's turn on the local SQLite adapter** and not on Durable Objects.
  The two hosts differ only under a slow connector, but they do differ.
- **Grant expiry transitions and facet recency are contract-tested on SQLite only**, because
  the DO host takes no clock. Both hosts run the same SQL; only one can be tested across the
  passage of time.

[What Substrat doesn't have (yet)](https://substrat.net/guide/what-substrat-lacks.md) is the maintained version of
this list.

### Where to go next

You now have the shape. The reference is the right tool from here:

- Building something — [Getting started](https://substrat.net/guide/getting-started.md), then the [todo
  walkthrough](https://substrat.net/guide/walkthrough-todo.md), then [The model](https://substrat.net/concepts/model.md).
- Choosing an engine — [What is an engine?](https://substrat.net/engines/index.md) and the five pages each one carries.
- Shipping — [Deploying a vertical](https://substrat.net/guide/deploying.md) and [Environments &
  previews](https://substrat.net/guide/environments-and-previews.md).
- Billing your own customers — the [metering](https://substrat.net/engines/metering/index.md) and
  [invoicing](https://substrat.net/engines/invoicing/index.md) engines.
- Operating — the four [platform surfaces](https://substrat.net/platform/index.md).

And if you are an agent rather than a person: [Agent rules](https://substrat.net/guide/agent-rules.md) is the page to
read before writing any code, and [llms.txt](https://substrat.net/llms.txt) is the index.