Runbook

Something in the fleet looks wrong — which command do you run first, and what does its answer mean? Each entry below starts from the symptom, names the first command, says how to read it, and names the next one. Flag-by-flag detail lives in cli.

A host went quiet

Read host diagnosis.

A service reads missing, failed, or unknown

Read service diagnosis.

A public download fails while Funnel configuration looks enabled

release origin is unreachable means the public release route did not receive an HTTP response from its configured origin. A published release, a reachable private address, or tailscale funnel status showing an enabled listener does not prove that the public Funnel path served the request.

Read the failed host's retained Tailscale messages through host exec. Choose the command for that host's operating system, not the machine running Stado:

# macOS: Tailscale app, network extension, and daemon messages
stado host exec <target> --json -- log show --last 1h --style compact --info --debug --no-pager --process Tailscale --process IPNExtension --process io.tailscale.ipn.macsys.network-extension --process tailscaled

# Linux: the tailscaled systemd journal
stado host exec <target> --json -- journalctl --unit tailscaled --since -1h --no-pager --output short-iso

These fixed commands read the previous hour of already retained native logs. They do not start a live stream, enable logging, restart Tailscale, change Funnel settings, or make a test connection. The JSON receipt preserves the host, actual argv, native stdout and stderr, exit code, and failure sentence. Read the timestamps and the failing process's own message before naming the cause. Empty output is an absence of retained evidence, not a healthy Funnel verdict; an access refusal or unavailable logging program remains a failure.

The exact command must be supported by the Stado client running it. Changing the time window, adding a process or unit, or requesting a logging mutation returns '<command>' is not an approved host-exec command before contacting the host. An older client returns the same refusal and lists its supported commands.

In Desktop, the same read is under Hosts → Retained Tailscale logs; see Read retained Tailscale logs. Neither a completed read nor a successful publication supplies a public download verdict.

A write to the object API answers 401

The sentence is 401 {"error":"unauthorized or non-immutable release write"}, and it names neither the prefix nor the grant. The object gateway authorizes a write by matching its key against the configured namespace's prefix allowlist; a key whose prefix is outside that allowlist can never authorize, whatever token you present.

First command:

stado config show

Read object_api_namespaces: each namespace carries its verifier item and its prefix_policies, each with a prefix and the actions it allows. If the key you are writing does not start with a listed prefix, that is the whole diagnosis. This is exactly why every autonomy object and every host-silence record is rooted under state/: no namespace declared autonomy/ or host_silence/, so the whole layer's writes were refused with this sentence, and state/ is a canonical prefix that is authorized wherever the queue prefixes are. The fix is to put the object under an allowlisted prefix — state/ for fleet state — not to widen the token. A verifier item that cannot be read answers 503, not 401, so a 401 is always a scope or bearer answer.

Every object read answers 503 after a Stado delivery

The sentence is 503 {"error":"object authorization unavailable"} on every stado storage, registry and release read, and it started when a new Stado reached the control host. The object API checks the host's configuration with the rules of the binary that serves it; a release whose rules the host's configuration does not satisfy closes every boundary at once. The registry, the release records and stado release rollback are behind that same API, and stado release version converge refuses to downgrade a host that runs a newer version than it declares, so none of them can undo it.

First command, from any client that still reaches the host:

stado release version show --host <TARGET>

A host-ahead verdict on stado names the version the host declares. Every delivery keeps the archive it installed under ~/.stado/releases/stado/<version>/<platform>/ on the host, so that version is still there. Build the current Stado on the host and run its release local restore there:

stado host run deliver <TARGET> <stado checkout> .stado/work/runs/<id>/stado
stado host run build <TARGET> --manifest-path <HOME>/.stado/work/runs/<id>/stado/stado-rs/Cargo.toml --bin stado
stado host run attach <TARGET> --program <HOME>/.stado/work/runs/<id>/stado/stado-rs/target/release/stado --arg release --arg local --arg restore --arg --version --arg <declared version>

release local restore installs the retained archive through the same checked install and reader convergence as release local install: the restored binary validates the host's configuration before anything is replaced, and every unit running the replaced binary is moved to the installed file. An unknown version is refused with the list of versions the host retains. When the broken Stado cannot say which version ran before it, --previous in place of --version <declared version> reinstalls the retained version whose staged stado is byte for byte ~/.stado/bin/stado.previous — the binary every install moves aside when it replaces stado, and whose release reclaim never removes. It is refused, naming the file, when that rollback target is absent or no retained release stages it. The host is recovered when stado storage stat stado://probierz/registry.json answers present and version show reads in-sync.

A delivered Stado stops on every start

Read startup diagnosis and recovery.

Reads look healthy while writes fail

Storage is single-writer with automatic read failover: mutations commit to the configured primary and mirror to a read-only disaster-recovery backend, a failed primary read may be served from the backup, and the backup is never promoted to writer. The trap is the asymmetry: every write can be failing while reads keep answering — from stale backup data. This ran in the field: autonomy writes were refused for days while the local backup backend kept serving stale reads, which is why stado optimize status still printed a confident forecast.

So when writes fail, distrust every fresh-looking read and check its own timestamp through the fleet plane: reported_at on beacons (stado service list prints the age when it is past threshold), inventory freshness in stado optimize status. A confident answer with an old reported_at is a stale read, not health. There is one object plane — the configured Stado backend; the writer has no cloud CLI, provider SDK, direct bucket URL, or cross-backend fallback. See disaster-recovery.

A job is stuck

First commands:

stado status <job-id>
stado machine logs <job-id> --cursor 0 --limit 1048576

status gives the queue's view; machine logs pages the canonical command log by byte cursor.

Then check proof of life before concluding anything is dead. The running document's lease_expires_at is its worker's own promise: the time by which it said it would renew (its poll period plus the lateness and write time of its newest renewal, so one renewal a store held for hours does not make the worker promise hours from then on). While that time is ahead, the job is alive. A lease_expires_at hours ahead of a job whose work directory shows no process is a worker that died right after such a renewal. Once it has passed, a pulse at status/<job_id>/heartbeat written after it means the renewal is failing and the worker is not, and a blob under the job's checkpoint prefix written after it means a multi-gigabyte upload is starving the renewal. Only a job whose promise passed with neither write after it is requeued (worker lease expired; <n>s past its worker's promise). A job with no lease_expires_at was claimed by an older Stado and is kept; the reaper counts it as unpromised.

To classify a failed job fast, grep its stdout against the failure mode quick-grep table. For the job lifecycle itself, see jobs.

ensure refuses

stado service ensure is idempotent and honest: each of its refusals in the field names its own repair.

• A per-login unit on an always-on host. The declaration puts the unit in launchd's user domain, but nobody is logged in graphically on an always-on host, launchd builds no gui/<uid>, and the system domain is the only one that host can load a unit into. The finding is printed as one sentence ending in the one privileged install command (sudo /usr/bin/install -m 644 -o root -g wheel <plist> /Library/LaunchDaemons/...) to run on the host. stado service list and stado registry doctor report the same finding before any restart trips over it.
• A LaunchAgent loaded in the background domain. When the account does not hold a graphical session, a LaunchAgent goes to user/<uid>, and the report says why: <account> does not own /dev/console (it belongs to <owner>), or (it cannot be read (<stat's words>)) when the console's owner could not be read, or <account> owns /dev/console but launchd answers no gui/<uid>. An unreadable console is never reported as one belonging to nobody.
• A loaded unit running a different program. The refusal reads <domain>/<unit> is loaded and runs [<declared argv>], not [<argv>]; retire it first. launchd holds the definition it bootstrapped, so rewriting the plist under a live job changes nothing an operator can see; ensure refuses rather than silently overwrites. Retire the unit (stado service retire <unit> --host <target>), then ensure again.
• Registry push refused on directory generation. A registry write replaces the whole document, so a push that would delete top-level keys, or whose service_directory.generation would go backwards (which would make every stale cached directory start looking current), is refused. The refusal spells the repair: re-pull, re-apply the edit, and push again; --force only if the deletion or the older directory is genuinely the intent.

A release did not land on a host

Read release diagnosis.