Missing service reconciliation

The coordinator runs a service reconciliation stage during every stado optimize run and scheduled autonomy tick. It joins two independent facts: the unit state in the newest host beacon and a fresh stado service verify reachability sweep from the declared consumer hosts. The result is written to both state/autonomy/services/latest.json and an immutable state/autonomy/services/runs/<timestamp>.json; stado optimize status prints the latest report.

Every autonomy object is rooted under state/ because the object gateway authorizes a write by matching its key against the configured namespace's prefix allowlist. No namespace declares autonomy/, so the whole layer's writes were refused with 401 unauthorized or non-immutable release write while the local backup backend kept serving stale reads — the same defect state/host_silence/ was moved to fix.

The fleet-wide host-silence threshold is also the service-beacon freshness threshold. A missing or stale reported_at changes the service state to unknown; it never authorizes a host mutation — with one exception. The beacon unit's own death is what makes every other unit unknown, so a silent host's declared beacon unit is reasserted through the idempotent service ensure path over the host channel: the channel answering is the evidence that repair is possible. Ensure restarts a matching definition in place and reloads a changed definition only after preflight, with the loaded result read back afterwards. Once the beacon publishes again, the rest of the host becomes repairable from real evidence. A failed unit goes through the same ensure path as a missing one. For a host-probed unit, a live process running a stale copy of its own declared binary is kicked, while a process executing a binary the unit never declared stays refused as identity_unresolved. For a fresh beacon that omits a declared unit:

Endpoint evidence	Reconciliation
observed	Probe the declared unit. Stado adopts a corrected path or unit record only when the unit is loaded and its live process matches the declared program. If ownership cannot be proven, Stado records identity_unresolved, alerts once on the transition, and refuses to create a duplicate.
unreachable	Run the existing idempotent service ensure path. It creates a missing unit, restarts a matching definition, or performs a guarded definition reload. It checks the running postcondition and updates the registry when the host selected a different valid unit path.
not declared in the service directory	The unit has no endpoint to disprove, so the host channel is the evidence: the unit is probed on the box, a loaded unit must prove its live program before adoption, and only a unit the host itself reports absent is ensured.
unverified	Record endpoint_unverified, alert once on the transition, and make no change because endpoint absence was not proven.

A product runs as one process per host, under its one catalog unit com.wisent.<product>. Any other unit on a host that runs that product's program is its predecessor, and Stado finds it from what it runs; the catalog names no unit, and a catalog that still carries retired_units or role_units is refused. A unit runs a product's program when its declared or running program lies in the product's own service tree (~/.stado/services/<name>/), is the exact program the catalog declares, or is a file named after the product wherever it was installed.

For every product but Stado the predecessor is replaced whole: stado service ensure <product> retires it before it starts the product's unit (and gives it back if that unit does not come up), withdraws its registry declaration in the same write that records the product's unit, and the reconciler retires it on every pass while the product runs there. For Stado the unit's work is one or more roles of stado serve, read from the unit's own command line by Stado's own command definitions (serve options and the one-pass product sync and beacon commands a native timer repeats), or matched to the live process's runner root, edge program or forward destination. It is retired only once the live com.wisent.stado process is proven to run every such role; the API listener's units are retired by that process's own takeover at API start, and the resolver's by the listener handoff. Until then the pass records the unit kept with the reason, and a unit that runs the Stado program with no role it can be matched to is reported kept with it runs <command>, the stado program, and no role of com.wisent.stado does that work.

A missing declaration of such a unit is never repaired. The pass records it as retired, and stado service ensure refuses to declare one with the same sentence: <unit> is retired: it runs the <product> program, whose work runs inside the one <product> process (<unit>), which retires it on this host; deploy <product> instead. A release_unit_image_revisit label filed under a product that is no product's unit is a label that product ran under before: stado registry set refuses it with the same sentence, and the next registry write Stado makes rewrites it to that product's one unit. A label that is another product's unit, or a block still invalid after that rewrite, is left as it is; the write proceeds when it does not change the block and prints `release_unit_image_revisit`: <fault>.

stado registry doctor holds a registry that still declares such a unit to the same contract. The unit is supposed to be absent, so the declaration is reported as retired-unit-declared rather than as missing-plist or unit-not-active, which would call the intended state a failure and point at the repair Stado refuses. The row names the product whose process replaced the unit and the command that forgets what is left of it: stado service retire <unit> --host <host>. When the product whose program the declaration runs cannot be read, the doctor cannot tell whether the unit is retired: it reports retired-unit-unjudged with the cause and judges the unit's own state as for any other unit, so a missing-plist or unit-not-active row beside it may be the intended absence.

A unit in the fleet's label namespace that no product's program explains is judged on its own (retire_stray). One that runs nothing and either last exited non-zero or is held by no launchd domain any more is retired, its file left in place; one that is loaded and live, or idle between scheduled runs, is never stopped by the pass and is reported as undeclared_live with its program, until it becomes a product's one unit or is removed. A program under ~/.stado/ is left alone. On a Linux host the scan reads the systemd unit files in /etc/systemd/system and ~/.config/systemd/user.

GET /api/service/reconciliation serves the latest report, with the same policy-read authorization as GET /api/registry.json. A store with no report answers HTTP 404 with no service reconciliation is recorded in this store: the autonomy pass writes state/autonomy/services/latest.json when it runs; an unreadable report answers HTTP 503 naming the read that failed. Stado Desktop's Services screen shows the report under Reconciliation: the time it was written, the mode, the tally, and every row with its host, unit, action, classification and the pass's own detail, including each kept reason and each undeclared_live program.

Every repair renders its unit through the same resolution chain service ensure uses: the host's registry declaration, then the shipped Wisent catalog, then the declaration bundled with the build. A declaration that names only a unit path cannot be reinstalled from the document; the repair records declaration_incomplete and alerts until the registry entry carries its program and args, which is the durable fix — read the truth with stado service show <name>, write it into the entry, and every future repair renders from the document.

AutonomyMode::Report records the same plan without executing it. EnforceSafe and EnforceOwned execute the reversible repair actions, bounded by max_actions_per_tick, the emergency pause, the circuit breaker, and a per-service mutation lease. Only a mutation that failed on a host feeds the circuit breaker; declaration_incomplete and identity_unresolved are refusals computed before any host command runs, and a refusal must not starve the healthy repairs behind it. Recovery-managed units stay with the fixed host-recovery program and are never silently converted into registry services; the beacon exception asserts the unit without writing the registry for them.

Known gap: a timer-driven oneshot beacon (Linux host-health-beacon.timer → oneshot service) is a unit shape service ensure cannot yet express — it asserts a running unit, and a oneshot exits by design. Declaring it as a plain service would create a restart loop, so such hosts stay with their installed timer, whose oneshot publishes through stado host beacon publish FILE; a timer an older build installed still runs it as host publish-beacon, and stado serve recognises that spelling as the health role too.

A stado serve that fails — a component stopped, or its start could not finish — appends the stop to ~/.stado/serve-stops.jsonl before it exits: {version, revision, at, failure, reported, report}, with the version and source revision of the binary that stopped. A build that stops again on the same host is a failed install, not a restart to wait out: on its second stop Stado runs oko defects record --product stado once for that build, titled “Stado VERSION stops on every start on HOST: FIRST LINE OF THE FAILURE”, found in stado serve VERSION on HOST (a host Stado serves on holds no checkout Oko could verify a revision against, so the revision is the evidence's first line), with every stop and its failure as evidence, and keeps what Oko answered on that stop (report): recorded: … marks it reported; oko defects record ended STATUS: STDERR or could not run: CAUSE leaves it unreported, says the cause on stderr, and the next stop of the build tries again. Oko is run from ~/.stado/bin/oko, where a Stado product install places it, because the service unit that runs stado serve has no login PATH; a host without Oko there reads oko defects record could not run: ~/.stado/bin/oko is installed at none of its declared paths: HOME/.stado/bin/oko. Once a stop is reported, later stops of the build are kept and raise nothing, so a crash loop is one defect, not one per restart; a new build starts over. No count or window is Stado's: the second stop of a build is the measurement. A stop that cannot be kept is said on stderr as stado serve's stop could not be kept: CAUSE beside the service's own failure, which it never replaces. The defect is read where every defect is, oko defects list and Oko Desktop. stado resolver status shows the newest build that stopped on the host under serve_stops (--json: {version, revision, stops, first_at, latest_at, latest_failure, report, held}, null when stado serve never stopped there, {unreadable} with the cause when the kept stops cannot be read), printed as serve stops stado VERSION (revision REV) stopped N time(s), first T, latest T: FAILURE; defect: REPORT (the recorded defect, or while none is, why the newest attempt failed), followed by serve stops revision REV is held from source installs until the checkout moves past it when held is true (a stop of that revision raised the defect, the hold below); Stado Desktop's route operations read the same report as "Read this host's resolver".

A revision whose stops raised that defect is held from source installs on the host. After stado release local restore --previous puts a working build back, product sync's next pass would build the canonical checkout again — the commit that stopped on every start — so a source install of a revision with a reported stop in ~/.stado/serve-stops.jsonl is refused before anything is exported: PRODUCT: revision REV is held: stado serve VERSION built from it stopped N time(s) on this host (PATH), the latest with: FAILURE; REPORT. A later commit is built as usual. A stop file that cannot be read refuses the install with whether revision REV is held cannot be told: CAUSE rather than building past it. The hold lifts by itself when the checkout moves to a commit with no reported stop; nothing has to be cleared by hand. STADO_BIN=<built stado> STADO_PRODUCT_TEST_PRODUCT=<product with a stado-release service recipe> STADO_PRODUCT_TEST_SOURCE=<its checkout> node tests/product/held/revision.mjs runs both refusals in isolated homes under .build/product-held-revision-* and keeps the commands, exit statuses and refusals in report.json.

A product process that no unit owns is outside this whole loop: no beacon names it and no ensure repairs or ends it. stado service list --unowned finds them on every local host. A process is a product process when it executes out of a managed root ($HOME/.stado/services, $HOME/.stado/work and each catalogued product's install root) or when its working directory is under one. The second rule finds a program started inside a root with a relative command line, such as bash release/build.sh or node node_modules/..., whose command line names no path under the root. Every candidate carries a verdict and its evidence: owned with the pid launchd or the systemd .service cgroup that claims it, or unowned. Each root also prints how many pids it matched, so an empty table says why it is empty.

stado service reap --host <host> --command '<program>' ends the copies of one named program on one host that no declared unit holds. It considers the same candidates as --unowned and keeps every process a declared unit holds: the pid launchd or systemd reports for the unit, that pid's descendants, and on Linux every process in the declared unit's cgroup, so a child a double fork left without a parent is still kept. Kept rows are printed and never signalled. Every other matching row reads would_end until --apply sends it SIGKILL and waits for it to exit. A row that survives reads still_running and the command exits non-zero. An empty --command is refused with a command substring is required.

Useful operator views:

stado service list
stado service list --unowned
stado service reap --host <host> --command '<program>'
stado service verify
stado optimize status
stado optimize run