10. Seeing what happened
The last chapter ended on reports that background work returns so that something can read them. This chapter is about the reading. It covers what the platform records about a running vertical, where each record lives, and which question each one can answer.
There are four instruments, at four different grains, and the most common mistake is to ask one of them a question only another can answer:
| Instrument | Grain | Exact or sampled | Answers |
|---|---|---|---|
| Traffic | one datapoint per request, at the router | sampled | how much, how fast, how many errors |
| Invocation logs | one line per request, written by the vertical | exact while retained | what this request did, and what it printed |
| Traces | spans the runtime produces | sampled, opt-in | where outbound time went |
| The spine | one row per mutation, per delivery, per denial | exact; retention depends on the record | what happened to the data, who did it, and what it set off |
The first three describe execution, including background work. The fourth is about meaning, and it is the one that cannot be rebuilt from anything else.
Traffic: the router counts
Every request passes the router (chapter 3), so that is where it is counted. Once dispatch returns, in a finally so errors count too, the router writes one Analytics Engine datapoint:
index: [tenantId]
blobs: [verticalSlug, scopeId, surface, statusClass, rayId,
operation, problemCode, principalKind, level]
doubles: [durationMs, status]The router knows the tenant, scope, surface and status by itself. Which operation ran, how it failed and who it ran as are known only inside the vertical, so the platform's entry hands them back on a response header, which the router writes into the datapoint and removes before the response leaves. The Requests histogram and its facet counts are read from these fields.
The index is the tenant, so every read of this dataset is a read of one tenant. The dashboard's chart is a sampling-weighted sum(_sample_interval) with weighted quantiles for latency, run through the Analytics Engine SQL API.
The data is sampled. It is the right instrument for "is this app getting slower", and the wrong one for anything that has to be exact, such as an invoice (chapter 12) or proof that a particular request happened. A sampled count is an estimate, and it presents itself as one.
Logs: the vertical writes its own line
This is the instrument with the least obvious design, and the reason for it is a platform fact.
Workers observability is keyed on the worker script. A vertical's script serves every tenant that installed it. And a trace does not cross the router's dispatch hop into the vertical. So the router knows the tenant and cannot label the vertical's log lines with it, and the vertical's log lines, unlabelled, belong to nobody in particular. A tenant's Observability page would be either empty or someone else's.
So a line carrying the tenant is written from inside the vertical's script, and the platform puts it there. At upload, the control plane adds its own entry module in front of the vertical's. That module wraps the vertical's default export with @substrat-run/vertical-host's withInvocationLog, so every request the script's fetch handler serves is stamped, whether or not its author thought about logging. Two kinds of script are uploaded unwrapped: one whose default export has no fetch to wrap, and one built on an older kernel that writes the line itself (see below). A vertical may also mount the same thing as middleware, first on its app, and the scaffold does:
app.use('*', invocationLog({ routerSecret: (env) => env.ROUTER_SECRET }));Doing both is safe. The middleware finds the platform's stamp on the request and steps aside, so the request is still at most one line, and what the operation route learns about the request lands on that line.
The stamp is internal: an id and a record for the request. What reaches Workers Logs is one JSON line for each request whose router assertion verifies. A request with no assertion, or one that does not verify, writes nothing (more on that below). The line is written in a finally, and a line that cannot be written is dropped rather than thrown, so logging can never fail the request:
substrat: 'invocation'
tenantId, scopeId, vertical, surface
method, path // the path only; the query string is dropped, so no OIDC code is logged
invocationId // a ULID minted for this request
status, threw, durationMs
level // error (5xx or a crash), warn (refused), info
operation, problemCode, principalKind
eventCount, eventTypes, entities // what the operation itself emitted
versionId // the pushed version that served itThat makes the line a record of the request as well as its proof of tenancy, and the dashboard's Requests view counts and filters it over the whole window.
Work that never arrives as a request writes the same line from the scope host, with one more field, kind. A consumer's delivery is consumer, with the event type, the attempt and how it ended: delivered, retrying, dead-lettered, or held inert on a copy of the scope. A schedule run is schedule, with how late it started. A request's line has no kind at all, so every older line still reads as what it was. The line keeps a failure's error code and none of its text, since an error message can quote the event it failed on. This part ships inside the vertical's own adapter, so a vertical writes it once it upgrades and pushes again.
Workers Logs indexes JSON fields, so tenantId becomes a filter. Reading a tenant's logs takes two phases. The first finds that tenant's invocation lines. The second uses Cloudflare's own per-request $metadata.requestId to fetch every other line the same request printed, including a stack trace from code that has never heard of tenants.
Three details come from getting this wrong once:
- The tenant comes from
readRoutedNode, never from a header read directly. A forged stamp would put text the forger chose onto another tenant's dashboard. SorouterSecretis required: without it every assertion fails verification and nothing is written. From the outside that looks the same as no traffic, which is why a bareinvocationLog()is refused. - When you mount it, it must be the first registration. Hono composes handlers in registration order, so a mount below some routes logs part of the surface and stays silent for the rest, which reads as "no traffic on those routes".
lint:invocation-logchecks the order, not just that the mount exists. The scaffold template ships with the mount in place. - It is not retroactive. A request that has already been served was never stamped, so there is nothing to backfill. An older version, though, is re-uploaded through the same entry when it is promoted or backed out to, so from then on it writes lines too.
- A bundle whose older kernel writes the line itself is left alone. That kernel predates the shared stamp and would not step aside, so wrapping it would log every request twice. Pushing on a current kernel moves it over.
Inside an operation, write your own lines with ctx.log rather than console.log:
ctx.log.warn('reply to {ticketId} bounced: {reason}', { ticketId, reason });The host stamps that line with the same tenant, scope, operation and invocation id, and with the template it was written from. Every line from that call site shares the template whatever the values were. Those two facts are what a log read needs to find a line's tenant without correlating it, to filter it by operation like a request, and to group lines into patterns exactly rather than by guessing which free-text lines look alike. The dashboard's log reads do not use them yet; that is the next step.
Traces: opt-in, and narrower than the word suggests
Automatic tracing is set per push, from a sampling rate the control plane holds. Today that rate is on in the test environment and off in production. Nothing in the platform emits a span per operation, permission check or engine call. What reaches the trace data is what the runtime generates on its own, mainly outbound fetch and the Durable Object entry hop. The platform reads it for one purpose: comparing a version's observed egress with its declared allowlist (chapter 8).
A per-operation waterfall is designed and not built. That is part of why the next section matters.
The spine, read as an instrument
Chapter 5 described the outbox as the audit log. It is also the best observability data the platform has. Every row is exact, typed, stamped by the kernel, and kept. The kernel ships sanctioned reads over it, and the dashboard puts each one behind a button, so nobody has to write SQL against the spine to use it.
History. readHistory returns one entity's events, newest first, with the fields a hand-rolled SELECT would misread: the authorization chain, the impersonation stamp, the operation, the deployed version. In the dashboard, opening a record under an app's Data tab shows this timeline.
Why? Every event emitted while a delivery is in flight records causedBy, the id of the event whose consumer emitted it. walkEventCause follows that link backwards. It says how the walk ended (complete, cut, depth limit, or a missing link), so a short chain is never mistaken for a complete one.
What did it do? walkEventEffects goes forward. It builds a tree from the delivery journal: which consumers an event reached, whether each one handled it, is retrying, or gave up, and how many attempts it took. This is where _substrat_deliveries shows up in the dashboard. It is not a raw table of deliveries but the delivery rows hanging off the event that caused them.
Same call. This joins a data change to a request. The vertical host mints the invocationId that the log line carries, passes it into invoke, and the kernel stamps it on every event the call emits, including events that consumers emit in that call's post-commit tail. readInvocation returns everything one call did. The same id is on the log line, so a record's call can be matched to the request that made it. That match is by hand today: the Logs view does not yet filter by invocation id.
That join has limits, and they are exact:
- It exists only for operations reached over the vertical host's HTTP operation route. Seeds, internal calls and events from before the column existed carry
null. - The router's datapoint carries Cloudflare's
rayId, not theinvocationId, so traffic does not join to events. - Denials and deliveries carry no invocation id. A delivery joins through its event, and a denial joins to nothing but its time and actor.
Facets. facetEvents is a small group-and-count over one scope's outbox. It filters by type and time window, groups by type, actor, operation, version, entity type, PII class, invocation, or one payload field, and returns counts per bucket. Each bucket carries lastSeen, the time of its latest event. That tells a path that stopped (it has old events) from a path that never ran (it has none). Erased payloads are counted separately rather than folded into a "null" bucket. A payload grouping counts only events whose PII class is none; events classed as personal data are withheld and counted in withheldPersonal. The class is per event, not per field, so an event classed none that carries personal data anyway is still grouped. This is the Events explorer. It is not the only way to read the outbox: the SQL console and the table browse sit behind the same access and are deliberately not narrowed this way. They read the app's data as stored, personal event payloads included, for the team that owns it, and every query is recorded in the access log (chapter 11). Grouping by actor lists principals, which are pseudonymous ids (#1821).
Flow. substrat push records what each module declares it emits and consumes. The dashboard compares that with what the outbox shows happening. Triggers lead to modules, modules to event types, event types to the outside world, and anything declared but never observed is drawn dashed. Next to it, per-operation health joins emitted events with the denial log, grouped by operation. That is where a role missing a permission key shows up before a user reports it.
Refused lifecycle transitions also have records. The process map can show where a transition stopped and which guard refused it. Audit can include permission denials beside allowed activity, with a link to the refused invocation. A failure to read refusals is shown as missing evidence, never as proof that everything was allowed.
The platform can also sample which declared output fields a response actually returned: present, null or absent. This is off unless the platform enables it. It records declared field names rather than values, and the read aggregates by operation and field. That helps distinguish an API field nobody receives from one merely present in its schema; sampled coverage is not proof that a field is unused. HTTP and MCP use the same observation surface.
Absence is a signal too
Everything above records things that happened. The hardest failure to see is a job that never ran, because that failure leaves no row anywhere.
Two mechanisms produce the row:
- Freshness expectations. A manifest can declare
freshness: [{ eventType, within: { hours } }]. The platform sweep (chapter 9) checks each one and records a verdict. "Noinvoice.sentin 26 hours" becomes a fact instead of a hunch. - Sweep runs. Every unit of sweep work is recorded, whether it was a schedule, a freshness check or a connector poll. The Schedules panel turns those records into four verdicts: On schedule, Overdue, Never run, and No sweep data. The last two are deliberately different. Module code cannot forge these rows:
ctx.requestPlatformrefuses the kind the platform uses to write them.
Findings: from an occurrence to something to resolve
A log line answers what happened once. Observability → Findings keeps track of problems that deserve follow-up: recurring operational failures, failed schedule invariants and stale freshness expectations. They are detected when the evidence is written, rather than by a separate periodic scan.
Each finding has severity, first and last seen times, evidence references and a lifecycle. A team can acknowledge, resolve or reopen it; a recurring problem can name the version under which it regressed. Suppress rules have a scope and an expiry, and changes are audited. Retention resolves quiet findings as stale rather than deleting their history. This is the useful next stop after “the logs show it failed”: who has seen the problem, and whether it has been dealt with.
Where each one surfaces
For a customer's admin, in the Dashboard:
- Observability (a team page, filtered by app): Pulse for traffic, health and schedules; Processes for declared versus observed flow; Logs for lines, requests, patterns and event facets; and Findings for problems with a lifecycle.
- An app's Data tab: a record's history, with Why?, What did it do? and Same call on each event.
- An app's Settings → Integrations: the platform-request journal, meaning what the platform did with each intent the app raised.
- An app's Overview: a status band, a traffic sparkline, and release markers on the chart.
For Substrat operators, in the Console: the fleet views, plus Operations → Failures, Issues and Sweeps. Failures are grouped into issues by fingerprint, so a thousand identical errors appear as one row with a count.
One property holds across all of them. Every read that reaches into a scope on a tenant's behalf leaves an access-log row that names what was read. Looking at data is itself recorded, which is where the next chapter starts.
What this does not give you
- A raw outbox or delivery-journal browser. Events are reached through facets and through records, and deliveries through the event that caused them. The spine's tables appear under System in the table browser, but there is no dead-letter list across a scope. Finding every consumer that gave up this week is a query nobody has built into a view.
- Exact request counts. Traffic is sampled.
- Cross-scope event questions. Every read above is one scope. "Across all forty branches" is forty reads, or it is the lake.