Host and service diagnosis

Runbook

A host went quiet

First command:

stado host ping <target>

ping probes two independent signals — ssh reachability and health-beacon age — and its verdict is the worse of the two (ok, stale, down). The exit status follows the combined verdict, so a box that answers ssh with a five-day-old beacon fails this command; that split exists because a host once answered ssh perfectly for five days while its beacon writer was wedged. Read it as two different repairs: ssh up with a stale beacon is a host that is running and not reporting; both signals down is a host that is unreachable.

ping reads beacons from the fleet store, so it opens that store first. A store that does not open — an object API that refuses the connection — ends the command at once, non-zero, with stado host ping <target>: the beacon store did not open: <cause>; it does not go on to resolve or reach the host behind an answer it already has. The cause names the object API's address; ping again once it answers there.

Next command:

stado host link <target>

link is the host's own account of why it went quiet: beacon age, the tailnet path and endpoint it last published, its last sleep and wake, recent interface changes, the silences recorded against it, and what readers refused because of them. A silence record opens once the next publication the newest beacon promised (next_by: its publisher's period plus its last collection) has passed — the same verdict host ping reads, so every command agrees — and its started_at is the last moment the host was heard from, so the duration is the outage, not the polling interval. Reader refusals are counted by reason — directory_cache_stale, authority_unreachable, beacon_stale — from the host's newest beacon on (refusals: none since the newest beacon at <instant>; reader_refusals.since in --json): a refusal written before the host last spoke is answered by that beacon and stays in the store and in the silence it belonged to. A host never heard from counts every refusal on record. A reader writes a refusal when its sentence changes, and each record carries the refusing component's own sentence verbatim, so the string you grep for is one that exists in a source file. The exit status follows the verdict; blockers are named in the report.

On a Linux host the beacon reads both from the journal's own fields (journalctl -o json), never from message wording. Sleep is the entry where systemd begins the start job of systemd-suspend.service (JOB_TYPE=start, no JOB_RESULT), wake is the entry where that job ends (JOB_RESULT set). An interface change is every kernel message in the beacon interval that the journal attributes to a network device (_KERNEL_SUBSYSTEM=net), whatever its driver writes; interface_changes[].detail is that message. To see what a host will publish, run stado host beacon collect on it; the journey cargo test --test host-health-linux-link -- --ignored with STADO_TEST_LINUX_LINK=1 checks the published changes against the host's own journal.

On a macOS host sleep and wake come from pmset -g log's own transition kinds. configd states link, carrier and Wi-Fi changes only in sentences, so the beacon reads configd's unified-log entries for the interval, masks their digits, and asks a model through Oko's schema-bound call (oko gateway object) which sentences record an interface change; the answer may name only sentences it was shown, and every entry with such a sentence is published, verbatim, in interface_changes[].detail. When the judgement cannot be obtained, or log show writes a line that is not JSON, the beacon publishes no interface changes for that interval (the reading is absent, not empty) and names the cause in the block's unread.

Every datum of the link block that a probe could not read stays null and its cause is published in the block's unread list: the probe that exited non-zero with its exit status and stderr, a journal or unified-log line that is not JSON, a timestamp neither log format reads, a tailnet answer that is not JSON, or a beacon unit that declares no WC_HEALTH_INTERVAL_SECONDS. A null with no unread entry is a datum the host has no tool for (no tailscale, no pmset, no journalctl). stado host link <host> prints each cause on an unread: line and carries the list as link_unread in --json; unread: not reported by this beacon means the host's Stado predates the field, and a link block that is not the block's shape is itself reported there with the parse error.

For a host that answers while its beacon is stale, link also reads the managed beacon publisher's own declared log. If beacon_publisher.code is verifier_unavailable, run:

stado repair stado --step link <target>

The repair resolves the object API authority from the service directory, reconciles its existing least-privilege verifier grant without rotating the bearer or restarting a service, then waits for the host's normal publisher to write a newer beacon and closes the silence. It refuses every other publisher code; use the returned diagnosis rather than trying a restart.

A service reads missing, failed, or unknown

First command:

stado service list

list answers from the latest health beacons alone — no ssh — so it still reports on hosts that are currently broken. STATE is the host's own word about its unit; OBSERVED is when anybody last confirmed the service from outside. A beacon past the next publication its host promised turns every unit on that host to unknown, with the age and the promise spelled out in DETAIL: health beacon is <age>s old, past the next publication its host promised by <next_by>; unit state is unknown. Read unknown as "the host said nothing usable", which is deliberately not the same answer as missing.

Next commands:

stado service status <name> --lines <N>
stado service logs <name> --host <target> --lines <N>

status adds best-effort host reads for units whose beacon state is failed: launchd's last exit status, and with --lines the last N lines of the unit's stderr; without --lines the failure block prints stderr: not read and the two commands that read it. Those reads degrade to a note, never to a failed command. logs tails the unit's log over the approved channel, as many lines as --lines asks for.

The autonomy loop will also act on this without you. Every stado optimize run and scheduled tick joins the beacon's unit state with a fresh endpoint sweep: a failed or missing unit is reasserted through the idempotent service ensure path, a live process running a stale copy of its own declared binary is kicked in place, and a process executing a binary the unit never declared stays refused as identity_unresolved. The verdict lands in state/autonomy/services/latest.json and an immutable state/autonomy/services/runs/<timestamp>.json; stado optimize status prints the latest report. Whether the plan is only recorded or actually executed depends on the autonomy mode — see autonomy and the missing service reconciliation table.