Registry and store authority recovery

Runbook

The registry copy refuses a lower directory generation

stado resolver status names the refusal kind directory-rollback, or stderr says [registry-cache] not recording the last-known-good registry in <path>: the registry read names service directory generation <old>, below generation <new> this host's last-known-good copy holds; …. The copy keeps <new>. An authority answering <old> is either the former authority after placement move --authority took the role (the copy is right; move the store or the role through the new authority) or a real rollback of the authority's store (the copy is the false one). Only in the second case run stado resolver accept --generation <old> on this host; the resolver adopts exactly that generation on its next refresh and records it in the copy. See the registry authority and its copies.

The registry authority's host keeps failing, and its services must leave it

Start with what this host believes: stado registry pull --last-known-good --path service_directory.authority.target names the authority it asks, and stado resolver remembered show says whether a read of it less than one refresh interval ago got no snapshot (every command then goes to its copy at once; stado resolver remembered forget makes the next one ask again), and stado placement holds <old> lists what the registry places on the failing host: its profiles (the vault's marked), roles, the authority and fixed routes. Prepare what no command can decide for you:

1. A service that is a fixed route, not a profile (stado registry pull --path service_directory.services.<service>.placement_profile prints null), is given one: stado placement profile draft <name> --service <service> [--probe <service>=<url>] [--state <path>] | stado placement profile declare --file -. --probe is needed only when the catalog declares no GET answering request for the product. A route the host Stado process serves (the fleet object API, com.wisent.stado) is not given a profile: the role move carries it.
2. A profile declared before the catalog named its products' stores is given them: stado placement profile sync <profile> adds every store the catalog declares that it does not carry (Brama's donated subscriptions, for instance). The evacuation's --standby does this in the run; without it the check refuses such a profile and names this command.
3. A release-controlled product whose install_root is written under the old host's home (stado registry doctor names it foreign-install-root) would install on the new host outside its home. The evacuation's --portable-roots writes each such root as the same path under {home} before anything moves (alone: stado release policy root repair <product>); without it the check refuses.
4. Both hosts run a Stado that carries registry push --fence-transaction and resolver follow: the fleet object API leaves the old host with the registry authority through the old host's own fence, every moved step asks the old host's resolver to follow before this one, and the check refuses a Stado without either (<old>, whose resolver this host's resolver reads the registry from, runs a Stado without resolver follow, …).

Then, on the new host, empty the old one in one command, first with --check, which runs every step's check and changes nothing:

stado placement evacuate --from-host <old> --to-host <host> --reason <why> \
  --vault-owner-uid <uid> --authority --standby --profile-fixed-routes \
  --repin-schedules --drop-release-targets --portable-roots --check
stado placement evacuate --from-host <old> --to-host <host> --reason <why> \
  --vault-owner-uid <uid> --authority --standby --profile-fixed-routes \
  --repin-schedules --drop-release-targets --portable-roots

Run it while the old host still answers: the steps run in an order (install roots under {home}, vault, profiles, the fleet object API carrying the authority), each once this host's resolver holds the step before, in which the new authority writes a store it serves. When the old host is the registry authority and no longer answers, the check refuses, since nothing can be checked through a silent authority, and the run takes the authority on the new host first and checks every other step through it — unless the silent host also serves the fleet object API this host's registry store is: then nothing can move without it, and the run refuses before it promotes anything (the registry authority <old> does not answer (<cause>) and also serves <roles>, the store a new authority would write, so neither can move without it; evacuate while <old> answers).

--repin-schedules pins the schedules pinned to the old host to the new one once it is empty, so they stop firing where nothing runs any more, and --drop-release-targets stops release control delivering there (a product released only to the old host is refused first, naming stado release policy remove). --profile-fixed-routes does step 1 in the run for every fixed route the catalog gives a probe for (Weles' admission among them); a route it cannot draft stays, named with the draft's cause. --standby gives each profile the new host does not fully stand by for (not named, or a unit file missing there — the vault's profile included) that host first (stado placement standby <profile> --host <host> --reason <why> delivers its programs and stages its units) and checks its move, before anything moves; the check runs stado placement standby --check for those profiles and lists them as needs_preparing (with the catalog stores it would add, the gap stado registry doctor names as profile-lacks-catalog-stores and stado placement profile sync --all closes). A tree the new host's agent has yet to roll out stops the run before anything moves; run it again once it has.

It moves, in this order, the fleet vault (stado credentials vault move, which rotates the owner to <uid>, a key the new host's keyring holds, before its profile moves), every profile, and the fleet object API (stado placement role move --with-authority, which fences its store, copies it, and installs one registry naming the new host both the route's active host and the registry authority: the old host's store holds the registry, so no commit could land through it once fenced). When the old host serves no role, the registry authority moves last (stado placement move --authority: handed over through the old authority while it answers, taken from this host's copy while it does not). A step that fails stops it, naming what moved and what still stands; stado placement history keeps every step's receipt. A unit already running on the new host with none of its profile's required state (a Skarbiec started with no vault file) is stopped by the move before it writes the state; one holding state is refused as a second copy. Once every step answered, the evacuation reads the registry again and is refused when it still places anything on the old host; then the schedules are repinned (and read back) and the release targets dropped, through the new route: the object API's move asks this host's resolver to read the registry at once and waits until it holds the new generation. A run stopped anywhere is finished by running the same command again. Each step is also its own command, with its own --check. stado placement holds <old> says what still stands on the old host before and after: it reports the host emptied once it holds no profile, no role of the host Stado process and not the registry authority, and also lists the release-control products still released there and the schedules still pinned there.

stado resolver status on the new authority must not report this host, <host>, is the registry authority by its last-known-good copy, but the registry store it reads … is <service> on <other host>: until the object API has moved too, readers there read a store the authority does not hold. See the registry authority and its copies, placement, the vault move and holding writes while a store is copied.

Reads look healthy while writes fail

Storage is single-writer with automatic read failover: mutations commit to the configured primary and mirror to a read-only disaster-recovery backend, a failed primary read may be served from the backup, and the backup is never promoted to writer. The trap is the asymmetry: every write can be failing while reads keep answering — from stale backup data. This ran in the field: autonomy writes were refused for days while the local backup backend kept serving stale reads, which is why stado optimize status still printed a confident forecast.

So when writes fail, distrust every fresh-looking read and check its own timestamp through the fleet plane: reported_at on beacons (stado service list prints the age when it is past threshold), inventory freshness in stado optimize status. A confident answer with an old reported_at is a stale read, not health. There is one object plane — the configured Stado backend; the writer has no cloud CLI, provider SDK, direct bucket URL, or cross-backend fallback. See disaster-recovery.