Delivered Stado startup failures

Runbook

A delivered Stado stops on every start

A stado serve whose stop proves its install failed cannot serve on this host however often launchd or systemd restarts it, and every adapter of the host (Brama, Skarbiec, the object API) goes with it. Two stops prove it, and no count or window of Stado's own: a component stopping with a configuration failure (the resolver's … Retrying cannot resolve it., say) is this release's rules refusing the configuration the release before it served, on its first stop; any other failure once the same build already stopped on this host before, as ~/.stado/serve-stops.jsonl keeps it — the measurement that also raises the crash loop's Oko defect. The release does not wait for anyone: the failing stado serve records itself as failed on this host in ~/.stado/releases/stado/<version>/<platform>/failed-on-this-host.json (version, pid, time and the cause: this release's rules refuse the host's configuration: <failure>, or stado serve <version> stopped <n> time(s) on this host before this one, the first at <time> with: <failure>; this one with: <failure>), reinstalls the release that ran before it — the same release local restore --previous install — and ends, so the init system starts the restored release. Its stderr says stado serve pid=<pid>: Stado <version> failed on this host (<cause>); recorded as failed at <record> and reinstalled Stado <previous>, the release that ran before it, and every enabled alert channel carries the same sentence. A stop whose earlier stops cannot be read is not measured against them: stderr says stado serve's earlier stops cannot be read, so this stop is not measured against them: <cause> and nothing is rolled back on a guess.

A release recorded as failed is never installed on this host again, by product sync, release local install or release local restore: each refuses with Stado <version> failed on this host and is not installed here again: <record>; the record is <path> — remove it once the cause is repaired. So a delivery cannot reinstall the release that just failed, and two releases that both fail here (the configuration itself changed, or the host breaks both) do not restore each other in turn: the second one's rollback is refused, it says this release failed on this host and the release before it could not be reinstalled: <refusal>, and the host stays down with both records naming why. When no rollback target exists (no stado.previous, or no retained release stages it) the same sentence names that; see object API recovery.

The record also travels to the rest of the fleet. Each health pass publishes the host's failed releases in its beacon as failed_releases: [{version, platform, record}] ({unreadable: <cause>} when the retained releases cannot be read). A delivery to any host (release local install, so product sync too) reads every beacon first and refuses a version another host published as failed: Stado <version> failed on <host> and is not delivered to another host: <record>; <host> drops it from its beacon once its record is removed there. When the beacons cannot be read the delivery is refused with cannot read whether another host published Stado <version> as failed, so it is not delivered here: the fleet's beacons could not be read: <cause>. A restore does not read the beacons: it runs while this host's own services, the beacon store among them, may be down, and it reinstalls a release that ran here before.

stado release local list reads every release the host retains and what each is to it: installed, rollback target (the one restore --previous reinstalls) and failed on this host: <record> with the version, pid, time and cause the failing release wrote. --json prints {platform, releases: [{version, installed, rollbackTarget, failed}]}. A record that is not JSON, or a file it cannot read, is refused naming the path. Stado Desktop's host release operations run the same two commands: Read this machine's retained Stado releases and any that failed here (release local list --json) and Reinstall the Stado release that ran before this one (release local restore --previous), each showing the command's own answer or refusal.

stado release failed reads every host's beacon and lists, per host, each Stado release it recorded as failed: <host> Stado <version> failed and was rolled back: <record>, the host named by its beacon slug. A host whose beacon could not read its own retained releases is listed as <host> its beacon could not read its retained releases: <cause>, never as a host with no failure; with no failed release anywhere it says no host of the fleet publishes a failed Stado release. --json prints {hosts: [{host, failed: [{version, platform, record}]} | {host, unreadable}]}. When the beacons cannot be read it is refused with the fleet's failed Stado releases cannot be listed: the hosts' beacons could not be read: <cause>. Stado Desktop's Releases screen shows the same answer in Releases rolled back on a host: one panel per failed release with its host, version, platform, time and cause, and the command's refusal when it is refused.