Service lifecycle: ensure, adopt, retire, converge

The four commands that change a service's placement on a host.

Ensure

stado service ensure compares the desired unit with the on-disk definition, launchd's retained program and arguments, and the running executable. It leaves a matching running unit alone and installs a missing one. One pass reports created, restarted, already_correct, converged, or reloaded. A running process counts as matching only while the image it executes is the file installed at the declared program: an install that replaces the program in place (a new file renamed over the path) leaves the path and the command name unchanged while the process keeps the old inode, and ensure compares the two inodes (lsof -d txt on macOS, /proc/<pid>/exe on Linux) and restarts such a unit instead of answering already_correct; an image whose inode cannot be read is not taken as evidence of a replacement. A declaration that names its own label is rendered at that label, so an existing unit is reinstallable from the document without becoming a second service competing for the same port. Ensure is also the autonomy cycle's repair path for a proven-missing unit, and the one repair permitted on a silent host's own beacon unit, because the channel answering is the evidence that repair is possible (operations). The real journey tests/service/replaced_program.mjs replaces an isolated unit's program in place and requires ensure to restart it onto the installed inode, then to answer already_correct.

On macOS a matching loaded definition can be restarted in place. A changed plist or a different program or argument vector retained by launchd requires bootout and bootstrap after executable and plist preflight; kickstart would reuse the old definition. Success requires launchd readback and the running executable to match. An unreadable or malformed retained definition is refused. If activation or readback fails, ensure attempts rollback only when it retained a genuinely different prior unit file. An already-desired plist is not retried as a supposed rollback; the failure explains why no distinct prior definition was available.

After changing a unit, ensure waits up to 30 seconds for an actual authoritative registry read before recording completion. This also covers a unit that serves the registry itself: a new PID can appear before its object API is ready. Only retryable read failures are retried; the host action and conditional registry write are never repeated. Permanent refusals return immediately. If registry or audit recording fails, the error states the completed action, running PID, and original cause, including for --json callers; it does not pretend that nothing happened on the host.

service deploy and a standby stage make the account's unit directory (~/Library/LaunchAgents with ~/.stado/logs on macOS, ~/.config/systemd/user on Linux) before they write the unit. A directory that cannot be made ends the step as write_failed: <mkdir's own words> rather than as a later write failure with no cause. service ensure, when it rewrites a unit, refuses the same way with cannot create the log directory:, cannot create LaunchAgents: or cannot create the systemd user directory: followed by mkdir's own words.

One product, one process per host

A Wisent product runs as one process on a host. The shipped service catalog names each product's one unit and the program it runs, and service ensure and service deploy refuse, before the host is touched, a unit that would start that program under any other label: it would be a second process of the product. The file name of the program decides, whatever tree it was installed into, so a copy of skarbiec in another directory is still Skarbiec. A file named as the product is its own binary too, also for a product whose catalog unit starts a launcher (Brama's unit runs start-with-skarbiec), so a helper unit that runs brama directly is refused like one that runs the launcher. The real journey tests/service/second_process.mjs deploys, on an isolated test host, a unit running a launcher-started product's binary and one running a directly started product's binary, and requires both refused before anything reaches the host. The refusal names the product and its one unit; <label> and <program> are the unit and the resolved program path:

<label> would run <program>, a second skarbiec process beside its one unit com.wisent.skarbiec; a product runs as one process per host, so move this work into skarbiec and deploy skarbiec instead

Nothing is written to the host or the registry. Work a product needs done belongs inside its one process, not in a helper unit beside it.

A declaration that already ran the product's program before this rule is still repairable, because consolidation starts from exactly that state: the product absorbs the unit's work, and Stado finds the unit on its host from the program it runs and then never deploys or repairs it again (operations); no catalog entry names it. A declared unit that ran something else may not switch to a product's program; that is the same second process reached in two steps, and it is refused with the same sentence. service adopt records a unit that already exists and starts nothing, so it is not refused here.

Because every role shares the one process, a role that ends ends them all: a resident component that returns stops stado serve with its cause, so the host's init system restarts a service that is down rather than keeping a partial one. The worker role therefore holds, instead of ending, on the one condition that is not its own fault and that its ticks already pass over: the fleet store not answering. Binding the store at start reads the fleet's storage layout from the object API; when that read fails with the transport's or the held route's cause, the agent log says the agent did not start: the fleet store did not answer (<cause>); the next poll tries again and the start is tried again at the declared poll, while the resolver and every other role keep serving. Ending the agent there ended the process with the resolver and each adapter forward of the host, and launchd started it again into the same unanswered store: a restart loop that took the host's Brama and Skarbiec adapters down for as long as the store stood. Any other start error — an accelerator or VRAM that cannot be read, a Vast bridge without its declared parameters — still ends the agent visibly. Real journey tests/worker_start/held_store.rs.

The one unit is named com.wisent.<product>

Each catalog product uses one canonical service identity: com.wisent.stado, com.wisent.skarbiec, com.wisent.brama, com.wisent.weles, com.wisent.probierz, com.wisent.skrzynka, com.wisent.oko, com.wisent.transcript-lake. The catalog refuses any other service.unit, so stado registry set <product> --service-file <file> and every command that loads the catalog fail with:

<product>.service.unit: <label> is not com.wisent.<product>; a product runs one service named com.wisent.<product>

The units a product's process replaced are not declared anywhere: a catalog that carries retired_units or role_units is refused with <product>.service.<key>: the catalog does not list unit names; the units a product's one process replaced are found on each host from the program they run, so remove <key>. On the host, every unit that runs the product's program under another label is its predecessor (a program in the product's own service tree, the exact program the catalog declares, or a file named after the product wherever it was installed). stado service ensure <product> --host <target> retires those it replaces whole before it starts com.wisent.<product>, because the new unit binds the same port the old label holds. Each retired unit is printed on stderr as <target>: <product> retired <unit> before starting: retired (<detail>). If one cannot be retired, the new unit is not started and the command fails with <target>: <product> was not started, because the units it replaced could not all be retired and would run beside it: <unit>: <error>; a host whose units cannot be read refuses with <target>: the units <product> replaces could not be read, so none was retired: <error>. Units whose work is a role of com.wisent.stado (another stado serve unit, or a native timer repeating product sync or a beacon command) are not touched before the start; their roles are read from their command lines and retired only after the live process proves the work is held.

A unit whose work has not moved into the product is not its predecessor, and Stado does not guess: a unit that runs no product's program is never retired as one. An undeclared unit in stado service list --undeclared is not proof that its work has moved; the reconciler retires such a unit only when nothing runs it, and reports a live one as undeclared_live.

Before it retires anything, ensure checks that the unit's program is on the host. When it is not, nothing is retired or started and the command is refused (error_code=refused):

<target>: <product> runs <program>, which is not on the host; deliver the product's qualified service release to <target> before ensuring it. Nothing was retired or started

The current stado product install <product> --surface service --host <target> does not transfer its locally placed files to a remote host. Do not use its local installation receipt as proof that the target has new bytes. Use the qualified release delivery and inspect the target's active image before retrying ensure.

When the host already declared the product under an older label, the registry write that records com.wisent.<product> also points every reference to the old label on that host at the new unit: the host's unit in each placement profile (placement_profiles[].hosts.<target>.units.<service>) and every service-directory entry active on that host whose managed_service named the old label. This keeps placement and routing references consistent with the recorded unit. If recording fails after a host action, the command reports that completed action and the registry failure separately.

Stado's own API listener is taken over by the process itself: once com.wisent.stado has prepared its object store and before it binds, it retires every unit on its host that runs the Stado program as the API listener under another label (serve --api), provided each one serves the same local root. A scan of the host's units that fails is logged and retires nothing. The predecessor's root is resolved the way its own process resolves it: WC_STORAGE_BACKEND and WC_LOCAL_STORAGE_PATH from the unit's environment, else the storage section of the config file it starts with (STADO_CONFIG, then ~/.config/stado/config.json and ~/.stado/config.json under its HOME), else local and ~/.stado/local-storage. With no storage variables, the selected configuration and defaults still determine the root. A different root is refused before startup, and the log names both resolved roots:

[stado] predecessor <unit>: failed (it serves backend "local" root "<predecessor-root>" by its config file, this process serves <desired-root>; moving the object store is the storage-root step of `stado repair stado --step storage-root --target <TARGET>`, not a takeover)
serve refusing to start beside predecessors that could not be retired: <unit>: …

A predecessor whose state the init system does not give (a sudo -n the host refuses, a manager that does not answer) is not proven to serve the same root and is refused the same way, naming the command and how it ended:

[stado] predecessor <unit>: failed (its state could not be read, so it is not proven to serve this root: /usr/bin/sudo -n /bin/launchctl print system ended with exit status: <status>: sudo: a password is required; …)

If the init system cannot say which pid runs com.wisent.stado itself, the process retires nothing and logs why.

Moving the store is the storage-root step of stado repair stado --step storage-root --target <TARGET> --apply, never a takeover.

Adopt

stado service adopt brings a unit that already exists on the host under management. Adoption requires proof: Stado records a corrected path or unit record only when the unit is loaded and its live process matches the declared program; unproven ownership is recorded as identity_unresolved rather than guessed, and never duplicated (operations). The registry record is built from what the host actually reported — the resolved unit id, path, and init system — not from what the operator hoped (stado-rs/src/deploy/service/ops/files/records.rs, record_from_report).

Retire

stado service retire <unit> --host <host> stops the unit, then withdraws its managed record and every logical directory route on that active host naming the unit in one versioned registry write. If a replacement product unit is already declared on the host, its directory routes pass to that unit instead. The directory generation advances so readers can detect the change. Retire leaves the unit file on disk; stado service remove also deletes that declared file after the registry write. Removing the last declaration drops services entirely (cli).

A failed host stop leaves the declaration and routes in place. If the registry write fails after the stop, the command says that the unit stopped but remains declared; it does not report a completed retirement. A failed file deletion reports the service as retired and names the file it could not delete. Units sourced from recovery are never written into the registry by these commands (operations).

Converge

stado release version show --host <host> --json is the read-only comparison of declared, installed, attested, and running bytes. stado release version converge closes only the drift it can prove. For Stado, the global product release owns units that execute $HOME/.stado/bin/stado; every registry declaration that executes Stado from an independently installed $HOME/.stado/services/... tree is completed afterwards from the same catalog-verified archive through service update. Private readers are selected from the registry, not from a fixed label list.

The archive stays in the version/platform release tree so a partial failure is resumable. An already-attested global root is not reinstalled or kicked merely to repair one old private reader; an absent or corrupt retained copy is fetched and verified without root activation. service update is idempotent for a private tree already on that archive and image refresh --if-needed leaves its matching process running. A stale tree is installed and its process image must be proved. Any install, lifecycle, or image-proof failure remains a failed release version converge, and the queue agent keeps its existing installed-release handshake rather than being interrupted mid-slot.

Failed private updates retain their child JSON or full stdout and stderr in stado-readers.detail; the enclosing report does not replace that cause with a closing brace or generic status. Stado Desktop calls the authenticated typed API for both report and apply; it does not launch a second CLI process. GET /api/service/converge?target=<host>[&binary=<name>] requires converge-read; the corresponding bodyless POST requires converge-apply. Both call the same implementation as stado release version show|converge.

A completed request returns HTTP 200 with the full report and exit_code, including a failed delivery. Desktop retains that complete JSON and exit code through refresh, including fields it does not recognize. Missing or action-ineligible credentials return 401, unavailable verifier state returns 503, and malformed query parameters or a request body return 400. A failure before a report exists returns 503 with SERVICE_CONVERGE_FAILED and its cause. See Registry API access for the separate client bearer and server verifier grant.

Reader resume requires the target receiver to be built from a release containing the retained-archive command arguments. A numeric version banner alone does not establish that source identity. A receiver built without the contract rejects those arguments and the apply remains failed; normal root delivery of a release containing this fix must happen first. No compatibility shim converts an older source into success.