Host channel: inventories and repairs

The host channel also carries these host-side inventories and repairs.

Cargo home inventory

stado host inventory <target> --json reports the managed account's fixed $HOME/.cargo and $HOME/.cargo/bin paths in cargo.home and cargo.bin. Each row gives its type, metadata state, octal mode, numeric UID/GID, byte size, modification epoch, and symlink text; no file body is opened. Cargo home may itself link to a mounted cache: Stado preserves that link's metadata and inventories its fixed bin child.

cargo.entries contains the direct bin members, including hidden names. entries_seen counts every member seen, while entries_complete and complete are false after truncation, failed traversal, unreadable or malformed metadata, or a sanitized name or link target. A missing path is different from a path whose metadata could not be read. An incomplete empty list never proves that a binary is absent.

stado host inventory <target> --json

Stado Desktop exposes the same report in Hosts → Cargo home, with Read Cargo inventory and Refresh Cargo inventory for one selected declared host. It uses authenticated GET /api/host/inventory?target=<name> rather than spawning the CLI. Changing the endpoint or identity discards pending and cached results. The screen shows the actual refusal and marks incomplete membership explicitly.

The HTTP route requires the existing registry policy-read authorization. A missing, repeated, empty, or extra query argument returns HTTP 400 with exactly one non-empty target is required; unavailable registry authorization returns HTTP 503 with registry authorization unavailable. Successful HTTP delivery still carries the host operation's own status and error: only status: inventory together with cargo.complete: true proves that this Cargo inventory was complete.

For a host-channel failure, RUST_LOG=stado::deploy::host_channel=trace enables the shared transport's detailed diagnosis. Stado preserves the remote process's real exit status and stderr; complete stdout does not turn a failed command into success.

An env key can have an owner other than the operator

A managed env file may be reconciled by something already running on the host, and a write to a key that something else owns does not survive. When the read-back after env set finds a different value, it names the forward marker whose contents match what replaced the write: the repair is to correct the declaration that owns the key, not to write the file again. stado service list --undeclared enumerates every unit a host has loaded and what each one runs, which is how such a writer is found when no marker explains it.

A one-shot script under KeepAlive is not a migration, it is a reconciler nobody declared: KeepAlive suits a service that is meant to keep running, and a script that exits when its work is done needs StartInterval or no keepalive at all. A loop that repairs by restoring pins the past in place, and keeps a host failing on a defect that has already been fixed upstream. stado service unit logs <label> --host <target> --lines <N> shows such a cycle verbatim.

The fleet-shape sweep (stado doctor, check job-runs-are-work-not-a-loop) finds such a job without counting its starts. The loaded-unit read takes each fleet unit file's launch policy — scheduled when StartInterval, StartCalendarInterval, WatchPaths, QueueDirectories or StartOnMount starts it, keepalive when KeepAlive is set and not false, once otherwise, unread when no fleet unit file was read — and a keepalive job whose last exit launchd reports as a success is reported as "it ended by itself with success after N run(s); its unit file keeps it alive and names no schedule, so launchd starts this one-shot again every time it finishes", with stado service unit show and stado service unit logs as the next reads. A failing last exit is reported as before, with its run count; a last-exit status launchd states as neither a number nor - is reported as such, naming what launchd said, with stado service unit show as the next read. On Linux the policy is unread: systemd reports a successful ExecMainStatus for a main process that has not exited yet, and its NRestarts already counts only the restarts systemd made on its own.

Retiring an undeclared reconciler, and putting it back

stado service adopt <label> --host <target> claims an undeclared unit launchd has loaded; adoption probes the host first and records what the host reported. stado service retire then boots the label out of every domain it was loaded in and disables it there, which is what makes the retirement survive the next graphical login. The host confirms the postcondition with launchd's own no job at … answer.

One line reverses it, and it is ensure rather than adopt because adoption alone would re-declare a label launchd is still refusing to load:

stado service ensure <label> --host <target> --from <program path on the host> --reason "<why it is restored>"

That is faithful when the unit's argument vector is the program and nothing else: ensure finds the installed plist already declares what it would render, leaves the file alone, and takes its launchctl enable + bootstrap path — the one branch that undoes a disable. Pass no --arg: a mismatched vector would make ensure rewrite the plist instead of restoring the one that is there.

Proof that a retired writer actually stopped is two reads of the file it was rewriting, far enough apart to span many former cycles, returning identical assignments, while the unit's own log gains no line. stado service list --undeclared still lists the label, because the plist is still on disk, with no pid and no exit status.

Getting an unversioned host file back, byte for byte

env show cannot do it, by construction. It replaces every quote, every backslash and every byte outside printable ASCII with ? and clamps long values, because its job is to let an operator judge a file without a secret crossing the channel. That is the right trade for a configuration reader and the wrong one for a program.

stado service file fetch <service> --host <target> --source-file <path> --dest-file <local path> is the byte-exact counterpart. The host hashes the file itself, the bytes travel base64 inside the same encrypted channel's response, and the digest is recomputed locally over the decoded bytes — two independently computed SHA-256s, because a payload that lost a chunk decodes into something shorter and perfectly valid, so a length can never prove a transfer. A mismatch writes nothing and exits non-zero. $HOME confinement and symlink refusal are env show's prelude word for word, -L tested before -f; a file past the one-megabyte limit is refused whole rather than truncated.

Converging a host onto the release it actually verified

Retiring a competing writer is what makes a file writable; it is not what fixes the host. Read the host's own release receipts — the deployment's recorded coordinate and archive digest — and move the configuration onto those coordinates with stado service env set, each write confirmed by its own read-back. Never point a production host at bytes no published coordinate attests: a staged archive with no sidecar digest, manifest or provenance is left alone, and convergence stops at the newest release the host has verified and installed.

A dead unit reported as running, and who owns the worker

service show reads ProgramArguments out of the plist and asks launchd nothing, so its word is declares, and stado service serving <name> --host <target> answers the question that matters: is the DECLARED unit the process on its own port.

Ownership there is decided by launchd label and never by argv, because two units can execute an identical argument vector and argv matching would credit the survivor to whichever one was asked about. The pid holding each port is walked up its own parent chain until a pid appears in launchctl list, since a launcher script is the job and the server it starts is the child that holds the socket. A label that cannot be read — a system LaunchDaemon is invisible to an unprivileged launchctl list — is unknown, never "nobody owns it". The ports judged come from the service directory's declared endpoint or from --port, and deliberately not from the unit's env file, which names every endpoint the unit calls as well as the one it serves.

The report also carries unit_ports: every port a process whose parent chain resolves to the unit's own label listens on, declared or not, empty when the socket table was not read or no owner resolves to the unit. stado service restart reads the same report back to back after its kickstart and answers when the unit serves its declared port; it refuses as not_serving when launchd holds no process for the unit, when launchd started a second process, or when the unit's own process listens on a set of other ports that two reads in a row found unchanged without the declared one — <host>: <unit> serves ports <ports> but not port <port>, the port its service directory entry declares on this host: correct that endpoint or the unit's configuration. A directory entry that drifted from what the unit binds is therefore named instead of being waited on.

A product's own release deployer may bootstrap a per-login unit while the registry declares a system daemon for the same service, and the two then collide on one port. The registry must describe what actually runs: adopt the unit the release creates (with --host-heuristic so the placement carries, and the product's onboarding metadata re-attached) and retire the rival. Reversing it is two declarations and a checked restart, because retire deliberately keeps the unit file while withdrawing its registry entry:

stado service adopt <label> --host-heuristic always-on
stado service onboarding set <label> --host <target> --product-id <product> --display-name <name> --repository <owner/repo> --surfaces <surfaces> --first-success-fact <fact>
stado service restart <label> --host <target>

retire handles the unit's real domain instead of assuming a per-login job. A system LaunchDaemon is stopped through the host account credential, then both system/<label> and its recovery job are disabled with privileged launchctl; a Linux user unit is stopped, disabled, and runtime-masked so an older coordinator cannot revive it from a stale read. service remove composes the same fenced retirement with deletion of the exact managed unit file, while retire keeps that file for an explicit rollback.

Repairing a macOS GitHub runner's apphost signatures

Managed runner installation preserves the signatures shipped in GitHub's checksum-pinned archive. For an already adopted service that directly starts runsvc.sh, the same repair is available through the CLI and the Services inspector's Repair GitHub runner runtime action:

stado service runner-runtime repair <runner unit label> --host <target> --json

The repair retains the installed version and registration, verifies the official archive's SHA-256 and both replacement apphosts, and replaces only those files. It does not restart the unit: GitHub's existing listener retry loop owns the next launch. An intact signature returns runner apphost signatures are intact; no files changed. That result describes the files, not a working runner. Use service logs to read the listener's actual failure.

Linux services and units that do not directly launch runsvc.sh are refused. Missing registration, an ambiguous version, a missing release digest, a digest mismatch, or a failed signature check stops the repair before activation. The JSON receipt names the target, unit, runner root, output, and restarted: false; Desktop displays the same output or refusal.

A live process still executing a binary that was replaced underneath it

A launchd unit's process goes on executing the image it started with. Replacing the file the unit declares does not move it: self_update::recycle_replaced_units cycles units only inside the invocation that replaced their bytes, matches argv[0] by string equality, skips its own pid, defers any unit whose argv runs serve --worker, and logs a failed kickstart without ever coming back to it. A long-running process can therefore keep executing an inode its declared path no longer holds, and fail the same way for days, until something unrelated restarts it.

registry doctor reports it as stale-unit-image when the running and declared files are different inodes, and unread-unit-image when the question could not be asked. The identity is (st_dev, st_ino) and never a path, because a path is exactly what does not change; links: 0 distinguishes an unlinked image, where no copy of the running build survives to be diffed, from a replaced one that still exists somewhere. Both readings are local-only: which file a pid executes is answerable only on the machine holding that pid, so every other host gets an explicit unmeasured row rather than a silent pass. A replacement is not a finding while its installer lives: release local install, release converge-local-readers and self-update write .<name>.replacing beside the file, holding their pid, from the first byte written until every unit on it has been recycled, and remove it when they finish. A unit still on the old image once that process has exited — finished, failed or killed — is stale, however recently the file was written.

A unit whose declared program is a launcher script (a file beginning #!) runs the script's interpreter, and the script often execs the program it starts, so the running image is never the script's own file: a launcher whose running image merely differs from the declared file is not a finding. A launcher running an unlinked image (links: 0) still is, since that process serves bytes no file holds any more, and a compiled program whose declared file was replaced stays stale-unit-image.

stado service image refresh <label> is the operator verb: it refuses a unit that is not stale and names the identity it found, restarts through launchctl kickstart -k, then re-reads the identity. That second read is the whole discipline: launchd re-execs the declared PATH, so a process under KeepAlive can respawn straight back onto the same unlinked inode, and a restart that did not change the image exits non-zero rather than reporting success.

Letting the release agent do it, one unit at a time

The policy is a top-level registry key, release_unit_image_revisit, and its shape is exact:

{
  "release_unit_image_revisit": {
    "schema_version": 1,
    "targets": {
      "<host>": {
        "state_dir": "/absolute/host/release-state",
        "products": { "<product>": ["com.wisent.example.unit"] }
      }
    }
  }
}

Every listed label authorizes the selected host's release agent to restore that unit's declared image on its normal tick. Absent means off: without the policy, the pass returns before reading a process table, unit file, lock or disk. Read the selected registry to determine whether a deployment enables it.

Why top-level and not a release_control field. Every release_control struct carries deny_unknown_fields, so a document holding a key an older build does not model is refused outright by that build — not ignored — and a fleet running two builds can be left with no document both accept. A top-level key is not modelled by Registry, so it rides in Registry::extra and round-trips verbatim through every read and write: older builds preserve it and ignore it, this build reads it.

Seven properties, and each one is a bound rather than a detail:

• Exact labels, owned per target. The policy authorizes only its named (target, label) pairs. A pair claimed by two products is refused. A product need not have a blue-green policy in release_control; however, any positive ownership declaration in the product catalog or an adopted service's onboarding.product_id must agree. A declared_only onboarding placeholder is not a runtime ownership witness, and absence from a catalog does not manufacture a contradiction.
• Darwin only. Every target named in the block must have a release_platform beginning darwin-, because the restart goes through launchctl; another platform is refused at the document even when its products map is empty. Left to runtime, every authorised label on a Linux target would fail and record RestartRefused. That record bars the identity pair, so it is not a hot loop — but each replacement of the declared file expires the row and buys one more launchctl call that cannot succeed for the same reason as the last, so the host spends one futile restart per release indefinitely and records each as a repair considered.
• At most one unit per reconcile invocation. Each scheduled invocation may act on one eligible unit. The host lock prevents overlapping invocations from acting on the same identity pair. It does not impose a host-wide time or generation limit; a later sequential invocation may act on another eligible unit.
• Never a unit that recycles itself. The exclusion is self_update::defers_to_release_handshake — the argv runs serve --worker — reused rather than restated, because the host process is one of the units that goes stale and two spellings of that rule would eventually disagree.
• The attempt is written before the restart, and a failed one is not retried. An Attempting record is committed to a host-wide ledger in the target's state_dir and the restart is refused outright if that write fails, because a record written only after the outcome is lost by any crash in between and the next tick would kickstart the same unit again. The observed result replaces it. A record that did not reach the declared file bars that unit while BOTH identities are still the pair it was made against — so a replaced declared file or a unit something else cycled makes it eligible again, and no wall clock is involved. A surviving Attempting record means the pass stopped between recording intent and recording a result, so whether launchctl was invoked at all is unknown, and it bars for that reason.
• One host, one ledger, one lock. Each target carries one state_dir, and the label ownership map is computed from every product in the block before --product is applied, so two product-scoped agents share one ledger and cannot each spend a restart on the same unchanged identity pair. A non-blocking lock covers observe, record, restart, settle and record. What it prevents is overlap: two reconciles running at the same time would each see the same stale unit against the same unchanged identity pair and each spend a restart, neither having seen the other's ledger write. It is not a rate limit and defines no time window — sequential invocations are separate ticks and each may act on one unit, a different one because the unit already handled is afterwards either on its declared file or barred by its own record.
• The exclusion is decided from the observation, not a second read. observe_unit_image_scan is the one plist/process/image pass: it matches a unit's whole declared ProgramArguments against the live process table and returns an internal enriched row carrying that exact vector beside the stable public UnitImageObservation. The public observe_unit_images view moves out only the stable observation, while revisit planning keeps the captured argv and reads the subcommand from it rather than re-opening the plist. The decision about whether a unit may be touched therefore comes from the same moment as the pid and image being acted on. Re-reading would introduce the very window this feature exists because of: a replacement landing between the two reads.

What an operator sees stays in the vocabulary they already read. The tick reports the unit, registry doctor's own kind, and the outcome word service image refresh uses; there is no new severity word. The stale-unit-image row for an authorised unit gains a clause naming what the agent already tried and what came back — outcome-specific, so a refusal says the identity was not re-read, and a surviving Attempting record says only that intent was committed and no result was written, so whether launchctl was invoked at all is unknown. A ledger that cannot be read, or a contract that does not resolve, is stated on that row too: the agent will not act, and a doctor that dropped the reason would report the stale unit while omitting why nothing is coming for it. A healthy host emits nothing per tick, and an authorised label the image pass never returned — a typo, or a unit this host never installed — is reported rather than passed over, because "nothing to do" and "nothing found" must not read alike.