Beacon

How does the fleet know a host is alive without asking it? Each host publishes a small health document on a timer; every liveness verdict in Stado is a read of that document's contents and age, never an ssh probe.

What it is

A beacon is one JSON document per host: the host's own slug, a reported_at timestamp, a disk line, and a units{} map with one entry per declared unit, each carrying its state (stado host beacon collect). The unit list is not fixed anywhere else — "the labels this host is judged on are the ones the registry declares for it", matched by target name or declared hostname with the .local suffix dropped. For each label the collector asks launchd in both the GUI and system domains, or systemd, and a live process outranks history: the previous run's exit code is read only when nothing is running now. A unit loaded with no process whose last run exited nonzero is failed, and its detail is the supervisor's own account of it, <unit> runs no process: its last run exited <code> and its supervisor has started it <runs> time(s) — a release that cannot serve, restarted without end — which stado service list and Stado Desktop's Managed services show beside the state.

On Linux, before publishing, the host reconciles each non-active unit with systemd's own systemctl show state, so a timer-driven oneshot that ran or is running reads scheduled and a unit systemd reports active reads active. When that read cannot run or fails, the collector's state stands and the unit carries lifecycle_unread with the command's cause, so a state left unreconciled is told apart from one systemd confirmed. A trigger (the timer of a oneshot) whose state cannot be read is listed in triggers_unread with each read's cause, rather than skipped as if the unit had no such trigger.

A failed unit is also a crash loop Oko is told of once. The collector keeps each failing unit in ~/.stado/failing-units.json under its label and, for a loop no defect was recorded for yet, runs oko defects record --product stado with the service, the host, when the loop was first read and the verdict. What Oko answered joins the unit's detail as Oko defect: recorded: …, or as why it did not record (Oko defect: oko defects record could not run: …, Oko defect: oko defects record ended …), in which case the next pass tries again; stado service list and Stado Desktop show it beside the state. A unit read not failing ends its loop, so a later failure is a new defect; a unit the pass could not read keeps its loop as it was. crash_loops_unrecorded is null, or why the loop file could not be read or written, in which case no unit's loop was raised. A unit the registry no longer declares on the host drops its loop from the file on the next pass.

A beacon's roles block names the roles held on the host and the Stado role processes nothing accounts for. Only the host process's own health pass ends those, and its beacon lists the ones it ended under ended_strays. stado host beacon collect, with or without --publish, lists them under unended_strays and ends none: run under another HOME, as a test or a second account does, it finds no role lock for the machine's own resident and would otherwise take that resident for a stray.

A host's beacons are published by the --health-interval-seconds role of its stado serve process, at the period that option declares.

One process per role, and the roles block

Every long-running role of stado serve (the host process, with or without --worker, --api, a resolver) takes a lock at start, ~/.stado/roles/<role>.lock, holding its pid, program and start time, for as long as the process lives. A second start of the same role on the host is refused before any listener binds: role serve already runs on this host: pid <pid>, program <path>, since <time>; one process holds a role, and a second start of the same role is never admitted beside it. A process whose program is not the installed Stado (~/.stado/bin/stado) is refused the same way, naming both programs: a checkout's build or a copy of the binary never serves a role beside the installed one.

Each health pass then reads the host's process table for Stado role processes nothing accounts for — a copy or a checkout's build running a role, a process whose program no longer exists on disk, stado agent from a release that had that role, a second holder of a role — ends each one (SIGTERM, then SIGKILL while the kernel still reports it), and publishes what it found in the beacon's roles block: held, one entry per role with its pid, program and since, and ended_strays, one entry per process ended on this pass with its pid, command, how long it had run, the reason (not_installed_program, program_gone, retired_role, duplicate_role) and a detail sentence. stado service list --json carries the same block on every unit row of the host, so a stray is read from the beacon it was ended on, never from a process list taken over ssh. The process table is read with ps; a host whose ps cannot be run publishes the beacon without the block and says why on the host process's stderr.

The next publication a beacon promises

A beacon the host process publishes (stado serve --health-interval-seconds) carries next_by: the time by which that process will publish again, its own period plus how long its last collection took. stado registry doctor reports stale-beacon once that time has passed, and also for a beacon that states no such time — one handed in by a one-shot stado host beacon publish or beacon collect --publish, which promises nothing. A beacon written before next_by existed is held to the stale_after_seconds it stated. No reader applies a window of its own.

A worker's capacity broadcast (capacity/<consumer>.json) carries the same field. The worker promises its next broadcast within its --poll-seconds plus how long its last write took; while its tick works, its heartbeat republishes the last measured snapshot for as long as the call the tick is in has run no longer than that call has ever taken to return in this process. A tick in a call slower than anything it has done before, or in a call it has never finished, stops being spoken for, the promise passes, and every reader (host gates, the scheduler, makespan placement, the live GPU ladder, the build fallback) stops counting the host. A cloud consumer's row whose promise has passed is deleted where it is found; a local host keeps its last row as the evidence that it went quiet.

What a pass reads of its own disk, and the scans block

A health pass reads, for every declared unit, the init system's state and launchd's event log for that unit. The event read is the costly half (one log show per unit, seconds each), and a pass once a minute repeated it for units that had not changed. The host process now reads the log for a unit only when the unit is not as it was at this process's previous read — its pid, launchd run count or state changed — and the host is under its scan load ceiling; otherwise the previous read's events are carried, and the unit's beacon entry says so in events: ok for a read, carried: the unit is as it was at the previous read, or carried: the host is above its scan load ceiling, the previous read stands. The first read of a unit in a process reads the log whatever the load. The read itself runs in the host's background band (taskpolicy -b on macOS).

Every scan of the host's own disk — the beacon's event reads (unit_events), a preview of the janitor's build-cache walk (build_caches, Space) and the inventory walk a space report runs on its own host (disk_inventory) — is admitted against one ceiling: the one-minute load average must not be above host.scan_load_ceiling (WC_HOST_SCAN_LOAD_CEILING), or above the host's logical core count when nothing is declared. A janitor pass on a volume at the full threshold walks whatever the load, since that walk frees the disk, and its reading counts as an admission. A beacon about this host carries a scans block: last, the reading the latest admission was decided on (load, ceiling, ceiling_source: declared or measured_cores; null while no scan has asked), and skipped, one entry per scan standing skipped with the reading that refused it, since the first refusal of the run and latest the newest; an admitted scan leaves the map. stado host health <target> prints it as one line: scans: load <load> against ceiling <ceiling> (<source>); <scan> skipped since <time>, or scans: no scan has asked yet before any scan asked, and scans: not reported for a beacon from before the block; stado host link <target> prints the same line as scans: and carries the block as scans in its --json document (null for a beacon from before the block), which is where Stado Desktop's Hosts inspector reads it. A scan that is skipped is not silently absent: the skip is published every pass until the scan runs again.

Who declares it

The host itself. The host process collects local disk and service state and publishes it through the scoped host-health API; stado host beacon publish FILE hands in a document collected elsewhere — an authenticated PUT requiring STADO_HOST_HEALTH_API_URL plus the dedicated stado-host-health-beacon Skarbiec grant. Missing routing, an unreadable or over-broad grant, an insecure non-loopback HTTP URL, failed authorization, and backend errors all leave the prior beacon untouched and return failure; there is no cloud CLI, provider SDK, direct bucket URL, ambient credential, or cross-backend fallback in the writer (operations).

Who observes it

stado host health <target>, stado host beacon list, stado registry doctor, and the read side of the service layer: stado service list joins the declared managed set against the latest beacons and issues no ssh at all. Independent host-beacon reads start together and retain registry order in the result, so one slow store read does not postpone every other host. stado service status <name> first narrows the declarations and reads beacons only for hosts that declare that service; a status lookup does not fetch unrelated hosts or build their unit rows. The beacon join reports unknown when a host has gone quiet; service status of a failed unit separately attempts its exit and stderr reads, carrying those readers' refusals as failure evidence (stado-rs/src/deploy/service/model/managed/status.rs). The autonomy cycle's service reconciliation reads the same documents (operations).

The host's declared service answering probes reuse one HTTP client and connection pool across beacon passes. After a standing-service recovery attempt, Stado cancels its in-flight question before asking again; a hung request cannot retain another task and connection after each attempt. A completed declaration pass also cancels questions and drops answers for services no longer declared on this host. An HTTP client initialization failure appears as a refused cause in the unit's answering reading, not as a healthy process or an absent question. See Service.

For a declared service.answering question, stado service list and stado service status <name> print an answering: <host> <unit>: <reading> line beneath their table. Their --json rows carry the same reading as answering: asked, waiting (with elapsed wait and possibly last), stood (with the host's declared bound), unreadable with the actual catalog, directory or resolver cause that prevented asking, or finished with a result whose state is answered (HTTP status, the HTTP library's successful verdict, elapsed time and the response detail on failure) or refused (the actual transport or request-construction cause). Stado Desktop's Managed services table marks a standing, unreadable, refused or unsuccessful HTTP answer, or one with no HTTP status even if it claims success (including the last result while a new question waits), when process state is active. A malformed answering value, including an answered result without its successful verdict, remains in the CLI's raw JSON row alongside answering_error; the plain table prints answering-error with the decode cause and observed value. Desktop keeps the other fleet rows visible and shows that row-local cause in its table and inspector, rather than treating a missing verdict as a failed or successful HTTP answer.

The beacon's reported_at still governs freshness: a stale reading is historical evidence, not proof that the API answers now.

The read-only real-host journey tests/service_answering/cli.mjs requires STADO_BIN, STADO_TEST_SOURCE_REVISION, STADO_ANSWERING_HOST and STADO_ANSWERING_NAME for a committed build and a declared service. It reads the actual host beacon, a fresh stado host ping verdict, service status and fleet list. It requires a finished answer or a waiting question with a last answered HTTP result whose successful verdict is true, and checks an unknown-name refusal. asked, stood, unreadable, refused, or an HTTP error answer do not prove the API works; a stale beacon does not either. Its command exits, raw responses and hashes are kept under .build/service-answering-cli-*.

Readers resolve a target to its beacon object through the slug rule in monitor/host_health.rs — target name, declared hostnames, and their first dot-labels, in order.

Where it lives in the store

host_health/<slug>.json, inside the deployment's namespace — the latest document only, one per host (monitor/host_health.rs). Because the prefix holds only the latest document, a gap closes over itself the moment the host comes back; the durable record of the gap is the silence record, state/host_silence/<host>/<started_at>.json, opened when the newest beacon crosses the threshold and closed by the first fresher beacon (monitor/host_silence/transitions.rs).

Silence, and what stale means

A host counts as silent when its newest beacon is older than the fleet-wide threshold: STADO_SILENCE_THRESHOLD_SECONDS, default 300 seconds — five minutes, because at a one-minute publication interval three consecutive misses is a host that has stopped talking and one miss is a slow pmset call. A value that is not a positive integer falls back to the default rather than disabling the detector (monitor/host_silence/transitions.rs). No beacon at all is silent — a host that has never published is not a host that is fine. A beacon stamped in the future is not silent: clock skew on the publisher is not an outage.

A stale beacon cannot describe the present, so nothing derives a confident verdict from it. A unit read through a stale beacon is unknown, never active and never missing (deploy/service/model/managed/status.rs; operations: "a stale beacon is unknown, never missing"), and staleness never authorizes a host mutation, with the single beacon-unit exception operations documents. Unit states a fresh beacon can report: active, inactive, failed; missing means the beacon exists and does not carry the unit; unknown means no beacon or an empty state (deploy/service/model/vocabulary.rs).

Commands that act on it

stado host beacon publish <file-or-dash>
stado host health <target>
stado host beacon list
stado host ping <target>
stado host link <target>

host link answers "why did this host go quiet": beacon age, recorded silences, and the reader refusals filed against it. Full flags in the cli reference.

Not to be confused with

• A probe. A beacon travels outward from the host; nothing dials in. Endpoint reachability is a separate observation (stado service verify).
• A silence record. The beacon is the latest state; the silence record is the durable history of its gaps (observation).
• The registry. The registry declares which units the host is judged on; the beacon reports what launchd or systemd actually says about them.