Week 30 · 20–26 July 2026
This is the week Substrat became something you deploy to rather than something you run. substrat push uploads a built vertical from a laptop or a CI job, the platform holds the only Cloudflare credential, and a private vertical goes all the way to production without anyone on the Substrat side in the loop. Around it: a tenant-facing dashboard where a customer signs up, names a team, installs an app from a marketplace and invites colleagues; test copies and snapshots, so a schema-changing upgrade has a rollback point; a hosted vertical that evaluates permissions from its own storage and needs no line to the control plane on the request path; sign-in it does not have to write; and a first connector, Scrive, that turns a signature request into a real e-signing document and records the signature back. Two new demos — Manyfold, a headless CMS, and an OIDC issuer — join Meridian, which became installable from the marketplace.
Deploy a vertical with one command
Until this week a vertical reached Cloudflare because someone ran wrangler with the platform's own token. Now it reaches it through the platform:
npm install -g @substrat-run/cli
substrat login # opens the browser; you sign in as yourself
cd my-vertical && substrat pushlogin runs a loopback OAuth flow through the control plane, so you are the actor on every push in the audit log; login --token stores a service credential for CI instead. push builds the vertical with wrangler --dry-run, derives the deploy manifest, and uploads the bundle to a dispatch namespace. The slug and display name come from a "substrat": { "slug", "name" } block in package.json or from the package name; the version defaults to the registry's latest, patch-bumped, so nothing is retyped and a stale number cannot slip in. substrat versions <slug> lists what you have pushed with each version's admission and channels; substrat promote <slug> --channel dev|staging --version <id> moves a channel; substrat whoami says who and which workspace. The CLI is published under Apache-2.0 — it holds no platform code, only a build and a POST.
Two things change behaviour. A vertical you push is stored as <workspace>/<slug> — the bare --slug you type is prefixed with your workspace's handle server-side, so every team can own a helpdesk; hostnames never carry the prefix. And because the first push of a slug claims it for whichever workspace resolved, push no longer falls back to the machine-wide login default: it resolves --tenant, then SUBSTRAT_TENANT, then a "substrat": { "tenant" } pin in package.json. The first interactive push offers to write that pin; a non-interactive push with none exits 1 rather than guess.
You no longer author a wrangler.jsonc either. A vertical declares substrat.runtimeNeeds in package.json — entry, needsNodeCompat, an optional pre-bundle build, and its own stores — and push derives the runtime config from that at the platform's pinned baseline, feeds it to the dry run and discards it. A hand-written wrangler.jsonc still works as the expert path; when both exist, runtimeNeeds wins and says so. Your own Durable Object stores and D1 bindings ride through; a bundle that declares a CONTROL_PLANE binding, a cross-script Durable Object or a service binding is refused with a 403 — that is the sandbox contract, and it is what makes a customer's code safe to run beside everyone else's. nodejs_compat and the platform's two shared secrets are forwarded and injected for you; a bundle the runtime rejects comes back as 502 with the runtime's own message rather than an anonymous 500.
Private verticals are yours end to end
This is the decision of the week. The old model gated every prod promotion on a Substrat staff member, which protected nothing for a vertical only its owner will ever install and left builders parked at "Admission: pending". Now the human gate moves to where the audience widens. A private vertical — owned by your workspace, not listed in the marketplace — lands admitted when pushed, every channel including prod is yours to promote, and a prod promote re-points your live installs to the new version (snapshotting first if the migrations changed). Every promotion is appended to the channel's history, which the dashboard's Go-live history panel rolls back from in one click. The loop from a merge to main is:
substrat push --promote prodwith --ack-permissions and --ack-migrations for a version whose permission or migration surface differs from the one it replaces. Listing a vertical in the marketplace is where staff review returns: a listed vertical's pushes land pending again and its prod promotion is staff-only, because from then on other tenants run it. Shared verticals never auto-rebind anyone's install.
Deploy from GitHub in one click
On the dashboard's Verticals page, connect the GitHub App, pick a repository and a branch, and Set up deployment writes a deploy credential as a repo Actions secret and commits .github/workflows/substrat-deploy.yml to that branch; the first run is the first push. The credential is a tenant-scoped push token — it travels in the same header the CLI uses, verifies into a builder principal confined to your own workspace's verticals, and cannot promote to prod or admit anything. Put that in a customer repo, never the platform-wide service token. The generated workflow installs dependencies from the lockfile it finds and runs Node 22, because the first real repo it was tried on failed on a bare checkout and — since a vertical registers only on its first successful push — the failure was invisible from the dashboard. A workflow committed before that fix needs the install step added by hand.
Your tenants get a dashboard
app.substrat.net is the customer's own surface — separate from the operator console, which stays on console.substrat.net so a platform-admin session cookie can never reach a tenant app. First sign-in bootstraps a tenant of your own and asks you to name your team; from there you install apps, invite people and manage what you build.
Teams. A user belongs to as many teams as they are invited to (a team is a tenant; the sidebar switcher is a cookie re-verified against real memberships). Invites are hashed, accept-required and emailed — a new adapter-email host package sends through Cloudflare Email Service, with a mock for dev. Delivery is best-effort: a committed invite is never rolled back if the mail fails, and the accept link is returned for manual sharing. An invited person who signs up from the link lands in the team, with the address prefilled; a signed-in member who opens an invite meant for someone else sees "this invite is for X" rather than silently switching teams. Members can leave; owners can delete the organization — type its name, every app is deprovisioned, the tenant becomes a tombstone. One thing changes behaviour: first login no longer creates a team from your email domain — it lands on a "name your team" step.
Apps and verticals are different pages. Create app offers the marketplace (published verticals) and Your verticals (what your team pushed, installable once a version is on prod). The Verticals page owns the supply side: your pushed verticals, their versions and channels, GitHub import and the CI scaffold. A new app binds <app>-<team>.global.substrat.run; the form previews exactly that.
What actually runs. An installed app is pinned to the version it was installed at. For a marketplace vertical, promoting a new version to prod moves the channel pointer and nothing else — the one exception is the private vertical above, whose owner's own installs follow its prod promote — so the per-app Deployments tab now shows the running version with a running pill and an Update to latest button that rebinds this one install, and the Overview shows the real version with an "update available" badge. A failed install is marked failed with the vertical's verbatim reason on a real Activity trail (created → active → failed → deleted), and Retry tears down the attempt and provisions again. Delete really deprovisions and releases the name, so delete-then-recreate with the same name works.
Configuring an install. The Env tab renders the envSpec your vertical declares in its manifest — grouped fields, placeholders, secrets write-only — and delivers the values to the running instance rather than only recording them. An Identity section on the install form picks the app's identity source: the built-in per-tenant store, one of your team's Auth Server apps, or any external OIDC issuer (more under sign-in below). The Data tab browses the app's tables and, new this week, gives you a read-only SQL console.
A read-only SQL console
Beside the table browser, type any SELECT (or WITH, VALUES, EXPLAIN) and run it with ⌘⏎. Read-only is enforced twice: a token scan that rejects a second statement and any write or DDL verb anywhere in the text — WITH … INSERT INTO is valid SQLite, so the first keyword alone proves nothing — and an adapter backstop (sqlite3_stmt_readonly on the pure adapter; on the Durable Object the query runs inside a transaction that always rolls back). Results are capped at 200 rows with a truncated flag, and the SQL you ran is what the access log records. Row editing is out of scope forever: a write here would bypass the event log.
Take a test copy before you upgrade
Migrations are forward-only, so "rollback" cannot mean running them backwards. It means keeping a copy. A scope can now be exported (a complete dump, spine included) and imported into a fresh scope at the same migration frontier, bound to the same version — a running copy, not loose rows, with its provenance (forkedFrom, forkedAt) on the record. On top of that:
- Snapshot before update. The dashboard's Update to latest has a "Snapshot data first" checkbox, on by default. A code-only rebind ignores it; a migration-crossing one takes the copy first, so a bad upgrade has somewhere to go back to.
- Test copies with a TTL. App → Snapshots → "Create test copy" with 1, 7 or 30 days, or keep until deleted. A copy gets its own hostname,
<app>--s<tail>.global.substrat.run, active immediately, behind the app's own sign-in with the source's users as of the fork. It receives no traffic of its own and auto-deletes on expiry; a control-plane sweep every fifteen minutes reaps what has expired. Copies are dead ends — nothing merges back. - Nothing leaves the deployment. A snapshot copies between sibling Durable Objects inside the vertical's own deployment; no scope bytes reach the control plane. A hosted vertical takes part by exposing
POST /internal/snapshotandPOST /internal/delete-scope(Callout, Meridian and Manyfold do).
The one path that deliberately moves data out is for staff: substrat scope pull <scopeId> writes .substrat/<tenant>__<scope>.sqlite, a real SQLite file in exactly the pure adapter's shape, so a local harness runs the identical vertical against production-shaped data. It is jurisdiction-gated (anything pinned tighter than global is refused), access-logged, and masked by default — PII-named columns become [masked] and JSON payloads are swept by key; --full is the break-glass and prints a warning.
A hosted vertical stands on its own
This changes what a hosted vertical is. Every permission check used to be a call to one global directory object, which meant every scope on the platform serialised through it and a customer's untrusted code would have needed a binding to it. Now the control plane is a write-time authority: it projects role definitions and tenant memberships into each scope's own storage when they change, and the scope evaluates permissions locally — fail-closed, so an empty projection denies. Scope-level grants are immediate; a tenant-level role change becomes visible across that tenant's scopes eventually, with a reconciliation sweep behind it. On Cloudflare this is scopeLocalPermissions: true on the host; a host built with no control plane at all is now valid, and the entitlement check on the request path passes because the SKU was enforced at provisioning.
Callout is the reference for the resulting shape, and Meridian and Manyfold follow it: the worker binds only its own SCOPE Durable Object (plus its own D1), trusts the tenant and scope the router asserts, serves its SPA from bytes bundled into the worker at build time (a pushed vertical cannot upload static assets), and exposes one platform-facing route, POST /internal/provision, that the control plane calls to create an instance. That route authenticates on PLATFORM_SECRET alone and fails closed when the secret is unset — the opposite of the router secret, where unset means "no router". Provisioning is two-phase: a scope is created as provisioning, the vertical is called, and only its confirmation activates it; getScope refuses anything not active. A harness that provisions must now call activateScope, and the directory row is written before the vertical call, so a stuck row is a work item a sweep can find rather than an orphaned object nobody can see. On Cloudflare a permission denial now returns 403, as it does on node; the error was being rebuilt across the Durable Object boundary and degrading to 400 on every deployed vertical.
Two adapter bugs found by doing this for real: the Cloudflare adapter split SQL on ; naively, so a semicolon inside a comment or string literal truncated the statement — green on every SQLite test, failing only on workerd, first in a Meridian migration and then in the kernel's own DDL, where every scope failed closed at construction. Both paths now use a SQL-aware splitter, and the shared contract-test migration contains every hard case. And the router reused a Durable Object stub across requests, so after each cold start the first request succeeded and every later one returned Cloudflare's 1101; stubs are rebuilt per request, and the test fake now enforces workerd's ownership rule.
Connect an e-signature provider
The first connector ships: @substrat-run/connector-scrive 0.1.0 turns the protocol engine's protocol.signatures-requested into a Scrive document, polls for completion and records the signature back into the scope. What it runs on is general:
- Connections. A per-tenant provider credential lives in a connection store keyed on
(tenant, vertical, provider), sealed by aSecretBox(AES-256-GCM with a key id, so rotation is a sweep). A host with noSecretBoxrefuses to store a credential rather than store it in the clear. Revoking a connection destroys the sealed blob, its connector state and its grants in one act. - Connectors.
registerConnector(id, eventType, handler)rides the same post-commit dispatch as executors. The handler gets the tenant, scope and vertical ambiently — there is no parameter through which to ask for another vertical's credential — andconnection(provider)returns the opened credential with a boundfetchthat lands health (lastOkAt,lastError) on that connection row. Module code still cannot reach any of this; a connector is host code. - A connection is a subject. To write back, the connection opens the scope as itself and holds an ordinary permission grant —
protocol:record-signature— checked on the same path as any principal, with the event actor stamped{ connection }. There is no allow-list and no bypass. Breaking for a custom permission checker:checknow takes aCheckSubjectrather than aPrincipalId;asPrincipal(id)covers the common case. - Deliveries retry, back off and dead-letter. A throwing executor used to escape after commit, so the caller was told the operation failed when the work order existed, and one poison event wedged every executor behind it. This changes semantics: an operation now succeeds even if its outbound effect has not happened yet — the delivery is journaled, backed off and retried, and after exhausting its attempts it is dead-lettered with the error kept, readable through
executorDeadLetters. - Connector state lives in the directory. Writing the provider's document id back into the scope deadlocks — a connector runs inside the scope's own post-commit dispatch — so
putConnectorState/getConnectorStatekeep a ledger beside the connection. That is what makes a retriedsignatures-requestedskip rather than create a second legal document. - Polling is the floor.
startPlatformSweeperdrains deliveries and reconciles live connections on a timer (node) or from a cron / alarm (Cloudflare). Nothing polls unless your deployment schedules it; dispatch works without it, signatures are never recorded back without it.
The connector was corrected against Scrive's real testbed on first contact (OAuth1 signing, not a bearer token; multipart upload; no status on create), and its mock now enforces the real encodings. The live BankID round-trip is still unverified — it is disabled on the testbed — and the reconcile fails closed on any shape mismatch, which is why the package stays 0.x.
Protocols can be signed asynchronously
For all of this to have something to sign, the protocol engine gained a signature request: open → requestSignatures → pending_signature, then signed once every party has signed, with cancelSignatureRequests back to open; in-app signProtocol still goes straight to signed. This changes behaviour: pending_signature is frozen — no fill, no rebind — where before an instance out at a provider stayed writable and its content could drift from what was hashed. A signatory may be a principal or an external party with no account. Content has a kind: checklist as before, or document, where the vertical owns the rows and binds (contentRef, contentHash) with a required hashRecipe so an auditor can reproduce it; the engine proves the signature was made over that hash and that it has not moved. Three new permission keys — protocol:bind, protocol:request-signature, protocol:record-signature — are held by no role in any demo; the last speaks for a provider, not a person, and granting it is how a deployment says what it trusts to speak for Scrive. The migration rebuilds the three protocol tables and backfills the frozen hash; every existing signature still verifies.
Sign-in you do not have to write
The platform apps moved to OIDC: the dashboard and console are relying parties against the platform issuer through a new shared oidc-rp package — authorization code with PKCE, JWKS verification, a signed session cookie, no auth database and no node:*. Signed-out visits go straight to the issuer now (the local sign-in card survives only as the retry screen), sign-out is always federated, and substrat login --fresh forces an account picker — before this, switching accounts was impossible because the issuer's SSO cookie silently signed the old user back in.
For a vertical, @substrat-run/vertical-auth gives the app one contract, an AuthProvider, and two implementations: an OIDC bearer provider that verifies a JWT against an issuer's JWKS (Supabase, Auth0, AuthHero, Keycloak — any issuer), and an IdentityDO, one Durable Object per tenant running a full credential store with a sub → principal directory. No shared AUTH_DB D1 across every install of a worker, which was wrong for hosting: one global email namespace, one principal binding. The owner seat is seeded at provisioning and the first login claims it — trust on first use with an explicit setup screen, after which self-service sign-up returns 403 and people join by invite: an admin creates a one-time link at a role, the invitee opens ?invite=<token>, creates their login and arrives already holding the role's permissions, resolved from the scope's own storage.
The identity source is now chosen at install. The dashboard's Identity section offers the built-in store, one of your team's Auth Server apps, or an external issuer. Picking a team Auth Server registers the app there automatically (dynamic client registration against the app's real hostname) and delivers the resulting config before the app goes active; a failure marks the install failed with the reason. The Auth Server is a new demo — an OIDC issuer with discovery, JWKS, password reset and its own admin — and it is multi-instance: one issuer per install, each with its own users and signing keys. Its database shows in the Data tab with credential columns redacted inside the Durable Object before anything crosses out.
Demos
Meridian (HR) became a hosted app. It installs from the marketplace, and a fresh empty instance is usable from zero: the installer becomes hr-admin and lands on an Admin section with a setup checklist — define leave types (with Swedish and Spanish statutory presets), add people, add projects — plus the per-period payroll export. The employment contract is no longer a checkbox labelled "signed": the articles (role, salary, occupancy, start date, notice) live in an append-only terms table, hr/issue-employment-contract binds them as a document and requests two signatures — the employer as a principal, the new hire as an external party who has no login on the day they sign — and hr/verify-contract re-derives the hash from Meridian's own rows. Meridian is also the reference wiring for the Scrive connector: the worker registers it, opens a connection holding only protocol:record-signature, and runs the sweeper, so a completed signature is recorded unattended.
Manyfold is new: a multi-scope headless CMS where a site is a scope, so one install serves many sites. Content moves draft → in_review → approved → published without skipping, revisions are append-only, publishing freezes the body under a content hash, and the delivery surface resolves references — a draft or archived target comes back explicitly unresolved. Content types are data authored in a model builder, and each change compiles to a reviewable migration rather than a live ALTER. It runs at manyfold.global.substrat.run with its own IdentityDO, owner claim and invites, and it is the second vertical on the CP-less pattern.
Callout is the first sandbox-clean vertical to go through substrat push, and the pattern the others copy. Its unauthenticated /api/seed — which created tenants and known demo logins on the deployed hostname — is gone.
For platform operators
- Production is on its own account. The control plane runs at
console.substrat.net, the router beside it; both deploy to a TEST environment on every push to main and to prod when a release publishes. Deploy scripts arecf:deployeverywhere (the olddeployname collided with pnpm's built-in), and a control-plane deploy now refuses to run while the target D1 has pending migrations — the first production 500s were an unapplied migration. The*.global.substrat.runwildcard is attached to the router, so tenant hostnames resolve live, andtools/set-platform-secretsgenerates each shared secret once and sets it under the right name on every service that shares it — aSERVICE_TOKEN/CP_SERVICE_TOKENname mismatch had cost a long detour. - Jurisdiction is
eu,usorglobal, never null, and defaults toglobal, which is the honest name for what every scope already is.euandusare refused with a 400 at the control-plane boundary until the mechanism that enforces them exists; the console shows them disabled. Residency lives in the hostname (*.eu.substrat.run), not in a column. - Secrets a pushed vertical needs are injected at push —
PLATFORM_SECRETandROUTER_SECRET— becausewrangler secret putcannot target a dispatch-namespace script. The control plane needsCF_API_TOKEN,CF_ACCOUNT_IDand aDISPATCHbinding; the dashboard needs the GitHub App secrets, aSECRET_BOX_KEYandPUSH_TOKEN_SECRET; invite mail needs Email Sending onboarded on a sending subdomain. - Registry hygiene.
DELETE /verticals/:slugremoves a vertical, refused while any scope is bound; block new installs hides a vertical from the catalog and refuses new instances while existing ones keep serving; deleting an app archives its scope and releases the slug and hostname; andtools/cleanup-orphans.mjs(dry-run by default) reclaims hostname rows, expired forks and dispatch scripts that a failed step left behind. - Observability. The router stamps every resolved request with its tenant into Analytics Engine and pushed verticals are uploaded with observability on, so the console's fleet view shows invocations, error rate and CPU percentiles per service with recent logs, and a builder's Verticals page has a Traffic panel for their own deployments. The API token needs Account Analytics: Read and Workers Observability: Read.
- The Data view reads a scope's bound version's deployment rather than the prod channel's — an install behind prod used to show empty tables.
Also
- Docs: a full refresh caught the site up — a Deploying a vertical guide, a Platform section (control plane, console, router, dashboard), pages for Callout, Handlebar and the shop demo, a Snapshots concept page and the Connectors section with its catalog; the permissions and identity concepts describe projection-on-write and OIDC as built. Every published package's npm page now links to its own docs page.
- Tooling: the repo typechecks on TypeScript 7, about ten times faster per package; contributors should point their editor's TS server at it.
- Design notes on record: deploy orchestration, the untrusted-builder trust model and the sandbox contract, the builder plane, scope-local permissions, previews and snapshots, the marketplace's push/publish tiers, observability, detachable vertical auth, and the CMS content model.
Released
| Package | Span |
|---|---|
@substrat-run/contracts | 0.7.0 → 0.16.0 |
@substrat-run/kernel | 0.7.0 → 0.16.0 |
@substrat-run/adapter-sqlite | 0.7.0 → 0.16.0 |
@substrat-run/adapter-cloudflare | 0.7.0 → 0.16.0 |
@substrat-run/control-plane-api | 0.7.0 → 0.16.0 |
@substrat-run/contract-tests | 0.7.0 → 0.16.0 |
@substrat-run/engine-workorder | 0.3.4 → 0.3.14 |
@substrat-run/engine-invoicing | 0.3.4 → 0.3.14 |
@substrat-run/engine-protocol | 0.3.5 → 0.4.8 |
@substrat-run/engine-booking | 0.1.1 → 0.1.11 |
@substrat-run/engine-invites | 0.0.3 → 0.0.13 |
@substrat-run/connector-scrive | 0.1.0 → 0.1.6 |
@substrat-run/cli | 0.1.0 → 0.5.0 |
@substrat-run/boundary-lint | 0.0.2 → 0.0.5 |
create-substrat | 0.0.1 |
@substrat-run/cli and @substrat-run/connector-scrive are new to npm this week, and create-substrat is reserved. The kernel group's nine minor bumps are the additive surfaces above; the one shape change is the permission checker's CheckSubject, and the protocol engine's minor is the signature-request migration.