The durable account of an outage

Step 5 of disaster recovery: the record of what was unreachable, and for how long.

A gap in a host's beacons closes over itself the moment the host returns: the beacon prefix holds only the latest document per host, so a short outage leaves no trace there, and a resolver's timestamped refusals would exist only in a local log file. Two append-only blob families keep the record:

• state/host_silence/<host>/<started_at>.json — one record per gap, opened when the newest beacon crosses the silence threshold and closed by the first fresher beacon. started_at is the last moment the host was heard from, not the moment somebody noticed, so the duration measures the outage rather than the polling interval.
• state/reader_refusals/<host>/<at>.json — one record per refusal, carrying the refusing component's own sentence verbatim, so the operator greps for a string that exists in a source file. <host> is the subject of the refusal, not the machine that refused: a resolver on one host failing to reach the authority on another is evidence about the authority's host.

A refusal record is written once per distinct sentence per host, reader and reason, and a refusal counts as recorded only once its record is in the store. While the store itself is unreachable or refuses the write, the refusing command says so on its own stderr, without replacing its own error:

stado: the resolver refusal about <host> (authority_unreachable) was not recorded in the fleet store: <the store's cause>

The next identical refusal writes the record again, so the outage is on record as soon as the store is back; a record the store never took is never silently counted as taken.

One write per refusal stands at a time. While a record is being written and the store has not answered, the same refusal raised again (a resolver whose directory is stale refuses every resolution it serves) opens no second write and no second connection to the store; a refusal with a different sentence is written once the standing write ends. A process that holds no store opens the fleet store once for all its refusals, so a store host that stops answering holds one request of each process's refusals, never one per refused request.

These records are where an account of "what was unreachable, and for how long" comes from. They live under state/ because that prefix is on the object gateway's allowlist; a write outside it is refused (see operations).