Disaster recovery

A dependency is down — a provider, a billing account, a storage backend, a host — and you need to know what is affected, move what must move, and keep a truthful record of what happened. This page is the order an operator acts in. Individual flags live in the cli reference; day-two procedures in runbook.

1. Assess the blast radius before you touch anything

stado blast-radius is a side-effect-free assessment of one failed dependency:

stado blast-radius --dependency gcp --json

It reports the dependency's live resources, auth, consumers, configured storage, and DR coverage, and it deliberately keeps those failure domains separate instead of collapsing them into "the queue is empty": primary and backup configuration, Skarbiec credentials, live cloud resources and caller/runtime IAM, downstream consumers, and backup namespace coverage each get their own answer. Provider probes are independent and paginated, so one disabled API cannot hide the remaining project assessment.

This is an impact assessment, not a complete object inventory of a host's local A and B roots. Walking those roots before writes are fenced would mix objects from different instants and could make the later authority choice look proved when it is not. The host transaction in step 4 therefore discovers the actual serving roots first, fences every writer, and only then captures its complete inventories and byte evidence.

The command never selects a backup for you. Queue state contains CAS locks, leases, and moving job records; a transparent read fallback can make two schedulers dispatch the same work from divergent stores. Promotion must fence writers first, then select one backend for every participant — which is the next step.

2. Transactional recovery: fence, migrate, verify, cut over

stado recovery migrate is the fenced, provider-neutral cutover: drain, copy, verify, cut over selected services, and optionally resume.

stado recovery migrate \
  --from gcs --from-bucket wisent-queue \
  --to local --to-path ~/.stado/local-storage \
  --writer control-host:stado-coordinator \
  --activate control-host:stado-coordinator \
  --enable-provider local \
  --dry-run

Drop --dry-run to execute; with it the command validates and prints the plan, performing no network or filesystem writes and no billing change.

What fencing guarantees:

• Every source writer named with --writer HOST:SERVICE is stopped before anything is copied. Omitting --writer requires --source-offline, an explicit assertion that no unlisted source writer can run — including schedulers, Cloud Functions, Cloud Run jobs, coordinators, monitors, and agents.
• Every service named with --activate is fenced before it is restarted on the destination.
• The queue stays paused at every failure boundary. Without --resume, the destination stays paused even after a fully successful cutover; resuming dispatch and claims is a separate, explicit decision.
• Only explicitly named services and compute providers are switched. --enable-provider is the complete post-cutover allowlist, and gcp is rejected in it.
• An optional GCP billing window (--manage-gcp-billing) is opened only around source fencing, copy, and verification, and closed before any workload is resumed. --confirm-billing-window must exactly repeat --gcp-project before a billable API call is made.

What refuses to proceed: the same fenced-transaction discipline as stado placement move, whose contract is the registry placement profile — concrete units per host, stop/start order, durable files, loopback health probes, and routing units. The command claims the profile through registry CAS, fences the source, copies state only after writers stop, activates and probes the destination, then commits the service declarations with a second CAS. Every failure before that commit restores destination files, routing, and source services. In particular, a required state file that is present when the move is planned but missing when read after writers stop fails the transaction with required state <path> disappeared after fencing — the world changed under the transaction, and the command refuses to cut over without the state rather than committing a move that silently lost it. A transaction record that vanishes before the second CAS fails the same way: placement transaction disappeared before commit.

3. Queue-state migration between backends

stado storage moves queue state between storage backends — the billing-outage migration path when the control plane must leave a provider:

stado storage copy --from gcs --from-bucket wisent-queue \
  --to local --to-path ~/.stado/local-storage --dry-run

Omitting --prefix copies the whole canonical prefix set. Verification is a separate read-only pass that compares two stores object-for-object and copies nothing:

stado storage verify --from gcs --from-bucket wisent-queue \
  --to local --to-path ~/.stado/local-storage

stado storage backup copies the active queue store to the configured disaster-recovery store; ls, stat, and cat inspect objects without writing. recovery migrate composes these with fencing and cutover; use bare storage copy only when you have established by other means that no writer can run.

4. What the backup store is — and is not

The configured backup backend is a disaster-recovery write mirror and a conditional read fallback, not a second write authority. Primary mutations commit first and are mirrored afterwards. Ordinary clients may retry a read that failed at the primary against the backup, but a successful absent answer from the primary remains authoritative. The object API server and other authority-sensitive readers use primary-only mode and return the primary error. No read path promotes the backup to writer. Cross-provider DR remains the explicit fenced migration in steps 2–3.

For the fixed ~/.stado/local-storage (A) and ~/.stado/local-backup (B) roots on a host, use the narrower resident transaction. It requires the running object API to report which of these exact paths it actually constructed as its primary and mirror; a different root is refused rather than substituted.

stado host storage-root-reconcile TARGET \
  --transaction ID --phase run --json
stado host storage-root-reconcile TARGET \
  --transaction ID --phase resume --json
stado host storage-root-reconcile TARGET \
  --transaction ID --phase status --json
stado host storage-root-reconcile TARGET \
  --transaction ID --phase rollback --json
stado host storage-root-reconcile TARGET \
  --transaction ID --phase finalize --json

run and resume address the same durable ID; resume never creates a second checkpoint or transaction identity. status only reads the receipt and the current native owner. rollback restores the exact prior route before the data-activation boundary, and finalize records completion only after the ordinary coordinator has finished typed lifecycle cleanup.

Stado Desktop exposes the same five phases from the selected host's Storage reconciliation sheet. Each action uses the sheet's transaction ID, quotes the corresponding stado host storage-root-reconcile ... --json invocation, and interprets the same durable receipt as the CLI. Switching between CLI and Desktop does not create a second workflow: use the same target and transaction ID when resuming or inspecting an operation.

Preflight identifies the real runtime

Before changing service or queue state, the running object API must report its constructed A and B roots, which root is primary, which root is its mirror, its serving PID, and the write-fence protocol it implements. The transaction also captures the object API executable and requires it to match the staged binary for the exact Stado release declared for the target. A plist path, a configured default, or a listener alone is not proof of the storage authority or runtime.

Before a lifecycle fence exists, a new run or resume selects that exact Stado release from the current canonical registry, not from an older captured target declaration. If the registry cannot be read, the attempt refuses to fence writers; it does not substitute a cached declaration. Once the lifecycle fence exists, every later phase remains pinned to the release and runtime captured in that fence.

Fence first, inventory second

The resident owner pauses and drains the queue, stops the other writers, and then acquires the reconciliation's exclusive filesystem lock against the shared lock used by ordinary object writers. It records a durable write-fence intent before treating the fence as held. The object API can continue serving reads in this interval, but object writes are refused with HTTP 503 and the name of the transaction holding the fence. The public status API exposes the same condition as storage.write_fence and reports the constructed backup as storage.backup.

Only after that fence is effective does Stado capture complete A and B inventories and compare authenticated HTTP reads byte-for-byte. An inventory taken earlier could race a queue claim, lease renewal, or mirrored write and is not preflight evidence. If B differs from A and the API's constructed authority is A, the transaction refuses a B-winning copy. This also protects A's object metadata when the body bytes happen to match.

The object API is stopped after those read-path comparisons and before the immutable physical checkpoints are made. Checkpoints preserve physical metadata as well as contents, including the recorded modes and ownership needed for exact restoration. HTTP comparison answers a different question: whether clients receive the same bytes. It deliberately ignores filesystem mode, uid, gid, and other physical metadata. A byte match is therefore not a substitute for a full physical checkpoint, and a metadata difference alone is not an HTTP-byte conflict.

The additive data step copies only the qualified ecosystem/ objects and matching metadata sidecars selected by the transaction. A-only objects remain and B is not modified. The lifecycle fence captures both the exact prior primary and the exact prior mirror; rollback never assumes that B was the sole prior route.

Activation, rollback, and fence release

Before any checkpoint exists, rollback can unwind a preparation whose queue was drained and whose placement leases were all captured. It restores the captured prior route and service state, releases any write fence, and restores the queue. Earlier preparation is refused with preparation rollback requires the recorded drained queue and complete placement leases; it has not acquired the storage write fence. An interrupted preparation rollback remains a rollback when the same transaction is resumed.

After checkpointing, rollback remains possible only before data commit. It restores the captured prior route, not an assumed B-only layout. The source roots are checked against their saved inventories; the immutable checkpoints remain evidence, and a partial additive copy in inactive A is retained rather than silently replacing the whole root.

Forward activation proves the constructed A primary and B mirror and compares the complete qualified API byte set before releasing the write fence. A checkpointed rollback proves the captured prior route and its corresponding byte set. A preparation rollback instead reports constructed-runtime-without-data-mutation: no data-copy step occurred, so this is not a claim that a full byte comparison ran. After the applicable proof, Stado releases the write fence, restores other writers, and restores the queue.

Repeating an active run or resume observes the existing owner instead of starting a competitor. An incompatible action is refused, for example native reconciliation is already executing run; cannot accept rollback; the refusal does not queue rollback behind a forward activation. If an owner was recorded as executing, status reports it that way only while the native manager still reports the same PID. Otherwise it reports interrupted, or unobserved when the manager cannot be read, while retaining the stored value as recorded_status.

Read the whole receipt

The lifecycle-fence schema is version 5. Its roots object records primary, backup, prior_primary, prior_backup, and runtime; those captured values, not conventional directory names, define activation and rollback. write_fence records status, intent, acquired_at, and released_at. A missing released_at means the fence has not been released. preflight_evidence is nullable until fenced inventories and API comparisons have actually been captured; null never means that both roots were compared or found equal.

A mutating command may return accepted as soon as the resident native owner accepts the operation. That is a launch acknowledgement, not success, activation, restoration, cleanup, or completion. Continue reading status for the same transaction ID: reconcile owner observation with recorded_status, inspect the preflight evidence and root identities, and confirm the final A byte proof and released write fence. On the forward path, only finalize after ordinary coordinator cleanup writes complete. For a rollback, require the durable restored/rolled-back outcome and released fence; an accepted rollback is no more complete than an accepted run.

On macOS, saved autostart state accepts both launchd's true/false overrides and its disabled/enabled names. Exact native-unit restoration reads back permissions, owner, group, and digest. Snapshot access may use an already available sudo -n capability when a captured root contains files the owner cannot read directly; it never asks for a password or changes the live roots' permissions.

See configuration for the storage layout and stado host storage-root-reconcile for the generated option reference.

5. The durable account of the outage

A gap in a host's beacons closes over itself the moment the host returns: the beacon prefix holds only the latest document per host, so after the 2026-08-19 control-host tailnet drop the product could not say the six-minute outage had happened, and the resolver's true, timestamped refusals existed only in a local log file read by nobody. Two append-only blob families now keep the record:

• state/host_silence/<host>/<started_at>.json — one record per gap, opened when the newest beacon crosses the silence threshold and closed by the first fresher beacon. started_at is the last moment the host was heard from, not the moment somebody noticed, so the duration measures the outage rather than the polling interval.
• state/reader_refusals/<host>/<at>.json — one record per refusal, carrying the refusing component's own sentence verbatim, so the operator greps for a string that exists in a source file. <host> is the subject of the refusal, not the machine that refused: a laptop resolver failing to reach the authority on the Mac mini is evidence about the Mac mini.

These records are where a post-incident account of "what was unreachable, and for how long" comes from, and they live under state/ because that prefix is on the object gateway's allowlist — the first cut wrote elsewhere and every write was silently refused while stale reads kept flowing (see operations).

The order rule

Three rules from this fleet's own incidents, in force everywhere above:

• Read the unit before you cycle it. stado service show prints what a managed unit actually runs — its program, arguments, and unit file — and repair signals only the pids a probe actually found; nothing re-derives a target from a pattern. A loaded unit must prove its live process matches the declared program before Stado adopts or restarts it.
• Prefer in-place restart. service restart and service ensure kick a loaded job in place and never unload it, so there is no window in which the job does not exist. The old bootout-then-bootstrap order could fail with the unit left unloaded — a partial failure strictly worse than never having run the restart — and did, on the always-on host.
• A failed repair is a reason to stop. Daemon restarts send TERM only, with no escalation: a control-plane daemon that ignores TERM is a finding to report, not a reason to try SIGKILL on the process holding the fleet's authorization state. A failed kick reports restart_failed with the exact privileged command an operator could run, and stops. Autonomy does the same: unproven ownership, an unavailable probe, and failed repair produce durable reconciliation records and alerts, never a guessed deployment.