Host channel

The host channel resolves the target from the canonical registry, checks its pinned host identity, and runs a fixed operation through Stado. Anything absent from the command allowlist belongs in a checked-in Stado operation; it does not justify raw SSH.

Use stado host exec <target> -- <allowlisted-command> for fixed diagnostics. Mutations use stado service file sync, stado service secret sync, stado release promote, or stado repair stado --step object-verifier. Secret values stay on the target, outside remote arguments and command output.

Provider sign-ins and chargeable probes use Brama's account-bound CLI, API or Desktop, not host-exec. See the account boundary.

Set RUST_LOG=stado::deploy::host_channel=trace when the same operation needs OpenSSH's own transport diagnosis. Stado adds -vvv to the shared SSH invocation only while that existing tracing target is enabled; normal argv is unchanged, and the debug stream remains verbatim in the operation's existing stderr and status receipt.

The host-side halves

An operator verb that must read or change a host runs its host-side half there, over the host channel, with that host's installed Stado. Each half is a visible verb of the object it acts on, so stado <object> --help on the host lists it, its argv parses on every host that carries the release, and a host whose installed Stado predates it answers with clap's unknown-subcommand error naming stado release version converge --host <host> as the way up:

Operator verb	Host-side half	What it does there
host repair (object API)	host local run LOCK -- PROGRAM	runs the recovery program under the storage-root lock
host repair (object API)	host local api paths, plist render, plist equal, boundary ready, route, skarbiec plan, skarbiec ownership, skarbiec upstream, skarbiec proxy	the readers of config, launchd definitions, route and Skarbiec release state
host repair (release store)	host local repair --config C --product P	returns one release-catalog coordinate to the managed account
host backup-audit	host local audit --backup B --primary P --namespace N --reclaim yes/no --apply yes/no	classifies the replica against the primary and prints the marker lines; an entry it cannot read is reported with the read error (<label> is unreadable (<cause>): <path>), a same-size pair left same_size_unproven and a delete_failed reclaim row carry the read or unlink error as their last field, and a marker row the reader cannot read (missing fields, a size or count that is no number, a path or namespace that is not hex-encoded UTF-8, an unknown class, outcome or phase, an unnamed root or reason) marks the audit unavailable with every such row's cause and line, in the order printed
service ensure, service directory consumer add, resolver api reassign	host local port	binds, releases and prints a loopback port the host hands out
host storage-root-reconcile	host local reconcile phase, step launch, step unit snapshot, step unit restore, step listener closed, step object runtime, step store served, worker	one receipt phase, one host step, or the resident worker of the transaction
credentials item show	credentials item summarize	reduces a skarbiec get --json document on stdin to each field's length and SHA-256
credentials token mint --token-item	credentials token register SKARBIEC ITEM FIELD -- grant issue …	registers a vault field as a bearer on the owner
credentials token sync	credentials token custody OPERATION VAULT CONSUMER FILE	exports, installs or checks one bearer against the grant
space file retire, space file remove	space file local retire, space file local remove	archives or unlinks the file on the host itself
service converge, service handoff	service roles --pid PID / --resolver-state	the stado serve roles a process runs, or what the resolver published
service update	service image refresh LABEL --if-needed	puts a unit back on the file its definition names
service env set, service env unset (systemd)	service env write --path-b64 … --key-b64 … [--value-stdin] --uid N	sets or removes one Environment= key, the value on stdin
a product's delivery argv	release local install --member M [--name N]	installs the delivered archive's member on the host
release local restore, the serve rollback	release local restore --version V / --previous	reinstalls a retained Stado release
the CLI recipe's after_install	release local converge --archive A --sha256 S	recycles every unit still on the replaced image
the delivery queue job	release delivery worker --request R	applies one delivery request on its target
Weles, the adopt handoff, credentials host lookups	release active binary PRODUCT --json / release active dir PRODUCT RELATIVE	the executable this host runs for a product, or a path under its release directory

None of these names joins a verb to a qualifier with a hyphen, and none is hidden from help: a hidden or hyphenated entry point is refused at the write by Tama's block-hidden-subcommands.

Bounded vault bearers

Select a declared host in Stado Desktop and open Fleet › Hosts › selected host › Bounded vault bearer to run the same host operation without switching to a terminal. The form names the consumer, comma-separated exact action:item[#field] capabilities, audience, and lifetime in seconds. Replace an existing capability set maps to --replace-capabilities; leave it off to make a different existing grant a refusal rather than an implicit widening or replacement.

Mint a new bearer asks the target's live Skarbiec vault to create the bounded bearer and store its hash. By default the sheet requests non-secret metadata, so the generated plaintext is discarded. Turn on Show generated bearer before review to opt into the CLI's one-time raw output; after success Desktop shows the value in its existing sensitive copy block and clears its displayed copy when the sheet closes.

Register a stored bearer instead adds an existing owner-vault item and field; the field defaults to token. Only that item/field coordinate crosses the host channel. The existing field value does not enter the process argument vector or ordinary output, and the operation does not rotate it. Raw output is available only for newly generated bearers.

Metadata success shows the selected target, the command's token_minted or token_registered status, consumer, audience, expiry, workload-binding state, exact capabilities, and the stored source coordinate when one was used. The explicit raw mode shows the generated bearer together with its target and requested grant fields instead. Host-channel, vault, item, field, audience, capability, expiry, and existing-capability conflicts are shown using Stado's actual refusal detail rather than a generic command-failed message.

An unmanaged executable is retired with stado host retire-file <target> <absolute-path> --product <product>. Add --dry-run for an exact-path preflight: it reports a transaction token, exact planned destination, byte count, mode, and SHA-256 without creating a directory or moving the source. A reviewed apply carries that receipt back with --transaction, --expected-sha256, --expected-size, and --expected-mode; all four must be supplied together, and Stado refuses before destination creation when the current source differs or the transaction token is invalid. Stado Desktop always uses this receipt-bound form. User executables must be owner-owned regular non-symlink direct children of $HOME/.stado/bin, $HOME/.local/bin, or $HOME/.cargo/bin; they move atomically into the product backup tree. One exact root-owned /Library/LaunchDaemons/*.plist is also accepted under the target's approved sudo grant and moves atomically to a non-loadable sibling. Retire a legacy plist and its convenience binary as two separate reviewed operations.

Retiring an undeclared init-system unit

stado service bootout <exact-unit> --host <target> [--domain system|user] is the declaration-independent stop path for a unit found by service list --undeclared or service unit show. It addresses the exact launchd label or systemd unit name supplied; it does not derive a prefix or add or remove a .service suffix. With no domain, Stado preserves system-first precedence and checks the calling account only when the system manager holds no exact unit by that name. Pass --domain user when the same or a related canonical name must remain running in system scope.

On Linux, bootout uses the target account's explicit systemd user bus for user scope and the existing non-interactive privilege path for system. It runs systemctl disable --now for only the exact requested identity, then refuses unless that identity reads back inactive and not enabled. An absent identity is a retry-safe absent; manager, privilege, disable, identity, and postcondition failures are refused. On Darwin, the existing system or per-login launchd bootout and absence checks are unchanged. Neither platform branch deletes the unit definition; file removal remains a separate stado space file remove operation.

Unit identity is exact: retiring a selected legacy user unit does not select another label or the corresponding system unit. Read the undeclared-unit report, then pass that identity and its intended domain unchanged.

Keeping a service's non-secret environment

stado service ensure <name> --host <target> --env NAME=VALUE --reason <reason> records each non-secret assignment in the managed service's env map and renders it into the host unit. Repeat --env for multiple keys; the last assignment wins. Recorded values override catalog defaults and survive subsequent ensure calls and automatic repairs. $HOME, $STADO_HOST, and $STADO_PLATFORM expand against the target, not the caller. Credentials still use secret sync, never --env.

When the rendered environment differs, ensure preserves a genuinely different prior unit file and loads the desired definition. On macOS this requires bootout and bootstrap: kickstart alone would reuse launchd's old environment. Ensure also compares the desired program and arguments with launchd's retained definition, not just the plist. A retained difference takes the same preflighted reload path and must read back as the desired program, arguments, and live executable before success is reported.

An unreadable or malformed retained identity is refused. If activation or readback fails, ensure attempts rollback only when a distinct prior unit file was preserved, and still reports failure. An already-desired plist is not reactivated as rollback; the failure states that no distinct prior definition was available.

For an independently managed instance outside a release policy's target map, registry doctor accepts pinned environment only when both the registry record and the local unit file contain the product's exact required values. A remote unit that was not read is not treated as agreeing.

Run stado host exec <target> -- stado registry doctor to measure those host-local facts through Stado. The fixed read-only command uses the target's installed binary and returns its own registry and executing-image findings; running the doctor on a workstation cannot establish them for another host.

When a host accepts a command and answers nothing — the registry authority's stado resolver snapshot, its own registry read — stado host exec <target> -- stado waits --json lists what every Stado process there is waiting on, from that host's own register of open waits: the kind (a lock, the object API, another host, a child process), what, where and since when. The fixed verb reads the register and removes only waits whose process has ended; it opens, closes and signals nothing.

When a workstation's own resolver refuses a request because another host holds it unanswered, stado host exec <target> -- stado resolver status --json reads what that host's resolver and object API publish about themselves: every wait an adapter stands on with its phase and age, the vault consultations the object API holds under each key, and the blockers each raises. It reads the files the resolver already wrote and changes nothing; the workstation's refusal names the held request, and only this answer says what it stands on.

Release-controlled placement handoff

A service placement unit has one exact lifecycle. Stado-managed units retain the existing shape:

{"name":"skarbiec","unit":"com.wisent.skarbiec","path":"/Library/LaunchDaemons/com.wisent.skarbiec.plist","kind":"launchd"}

A logical service whose process lifecycle belongs to the release controller instead carries only:

{"name":"skarbiec","controller":"release-control","product":"skarbiec"}

The external shape must appear in every host template for that service and name one release product. The service directory still owns its route, active host, endpoint, consumers, and generation; the placement profile still owns state, routing units, and probes. Routing units remain ordinary Stado-managed units. Static validation requires the route's active host, endpoint, active-host probe, product, release target, and stable bind to agree; it rejects a target service row or any legacy launchd restore field on every target of the owning product. Resolution and verification therefore keep using the route without pretending the external process is a target-managed unit.

Install 0.15.24 or newer on every registry reader and agent before publishing this shape. Then run stado service handoff <service> --host <active-host> --product <product> --json once. The command proves the committed desired release, active signed executable, stable proxy, readiness, inactive legacy unit, legacy file digests, and absence of an executable caller before one generation-bound compare-and-swap externalizes all templates, removes the active target row, removes the legacy restore identity, and advances the release and directory generations. It preserves the route, endpoints, consumers, profile state, routing units, and probes. Every placement mutation refuses a release-controlled member before a transaction or host action. A handoff invocation first fsyncs a version-scoped prepared receipt under ~/.stado/work/service-release/<product>/<version>/, then performs the sole registry compare-and-swap and immediately advances that receipt to registry_committed. If the caller is interrupted, rerun the identical command: a matching prepared receipt is reused only after the same intent, exact legacy files, active release, and still-managed lifecycle are re-proved; a registry that already carries the intended handoff causes the command to skip the successful CAS, reacquire the same per-service lease, and finish the reconciler-report fence. Do not delete or replace the receipt, issue a different handoff, or infer completion from the registry shape alone; handed_off with a satisfied fence is the retirement boundary.

After the successful handoff, retire the reported plist and convenience binary separately with host retire-file, plist first. Each handoff receipt supplies the unique transaction, SHA-256, byte count, and four-digit mode passed to the mutating command's four explicit binding flags; do not perform another dry-run or an unbound mutation. Each retirement reports retired or retry-safe absent, so a failure after the plist move preserves an exact partial-cleanup record and the binary operation remains independently resumable. Rollback after handoff is not a binary downgrade: it requires a new generation-bound registry change, restoration of both archived files, and proof that release-control is no longer serving. Older readers cannot parse the external shape and must not be reintroduced while it is published.

A product whose catalog service names its one unit (com.wisent.<product>) is not served blue-green. stado release catalog enroll <product> writes its rollout policy first, before the publisher and secrets steps: it creates the policy with replace, or converts a blue-green policy it already has, so the strategy becomes replace and every target loses stable_bind and candidate_ports (<product>: rollout policy converted from blue-green to replace of its one unit <unit>). Declare the unit with stado service ensure <product> --host <host>; the write that records it withdraws the declarations on that host of every unit that runs the product's program under another label and points the placement profiles and directory entries that named them at the new unit. On a host that still serves the product blue-green, the release agent then hands the port to the product's catalog unit once launchd or systemd holds that unit, and once either the release proxy holds the port the service directory names for the service, or another process — the unit — already listens there. It stops the proxy, ends the blue-green release processes its state records, sweeps any it leaked, and clears the state; its log says handed <host:port> to <unit>: stopped the release proxy and <n> blue-green release process(es) (<version> pid <pid> port <port>, …). Until then it stops nothing and logs why: the unit is not loaded, or nobody listens on the unit's port while the proxy holds another one.

A rollout policy must name a service the directory declares. When stado release catalog enroll <product> enrolls a service product the directory does not know yet, it declares the product's fixed route in the same registry write as the policy, as stado service declare would: the policy's host as active_host, the loopback endpoint at the manifest's runtime.port, managed_service naming the product, the consumers listed in the manifest's runtime.consumers, and a declared-only record of the service on that host, which the first delivery replaces. A manifest without runtime.consumers is refused with <product>: the service directory declares no route for <product>, and .wisent-release.json names no runtime.consumers to declare one with. A product the directory already declares keeps its route.

To stop releasing a product to one host, run stado release policy remove-target <product> --target <host>; to stop rolling it out by release control anywhere, run stado release policy remove <product>. Each is one verified registry write. stado release policy show <product> [--json] prints the policy as the registry holds it, in the {product, policy} shape stado release policy apply --file reads (strip desired and previous, which only release promote sets, before re-applying an edited copy), and stado release policy list names every release-controlled product with its targets and desired release. The last target of a policy is refused with <host> is the last target of <product>; … stop releasing the product by release control altogether with stado release policy remove <product>, and an unknown product or a host that is not a target is refused with the names that are. On every host the policy no longer names, the release agent retires what it left there: the release proxy that product's state file owns (stopped by state path, whatever its port), the release processes it launched — only processes running out of ~/.stado/services/<product> with the agent's launch marker — and, on the pass that finds nothing of the product running, its state and proxy files. Its log says [release-agent] <product> is no longer released to this host: retired its release proxy and ended <version> pid <pid> port <port> and then … nothing of it runs here; its release state was removed.

Ordered connection paths

ssh remains the preferred host-control destination. A target may also declare up to 16 ordered ssh_fallbacks; each fallback has a stable lowercase name and an SSH destination:

{
  "name": "example-host",
  "ssh": "operator@example-host.example",
  "ssh_fallbacks": [
    {"name": "nebula", "destination": "operator@example-host.nebula.example"},
    {"name": "tailscale", "destination": "operator@example-host.tailnet.example"},
    {"name": "lan", "destination": "operator@example-host.lan.example"}
  ]
}

The names describe routes, not transport implementations. Nebula, Tailscale, WireGuard, ZeroTier, a private LAN, and a public address all provide an IP path; Stado still supplies the host identity, credential, fixed operation, and audit boundary above that path. Public Tailscale Funnel endpoints remain service publication and do not become host-control routes.

When a target has more than one path, Stado authenticates a side-effect-free true command in declaration order and sends the real operation exactly once through the first path that answers. A one-path target keeps the original single connection attempt. stado host link <target> probes every declared path and reports the selected path, so a working primary cannot hide a broken fallback.

The stado host exec receipt says which of those paths carried the command. used_connection names the route: kind is ssh or local, name is the declared path name, and an SSH route also carries its destination. A target that IS the machine running the command reports kind: local and name: local, because no declared path was used and naming the preferred one would be an invention.

stado host exec --json gpu-host -- uptime

{
  "ssh": "root@gpu-host.invalid",
  "ssh_fallbacks": [{"name": "tailnet", "destination": "root@gpu-host.tailnet.example"}],
  "used_connection": {"kind": "ssh", "name": "tailnet", "destination": "root@gpu-host.tailnet.example"}
}

ssh and ssh_fallbacks are declarations; used_connection is what happened. A receipt carrying only the declarations cannot separate a healthy preferred route from a dead one whose fallback rescued the operation, and that difference is what an operator reads the receipt for. The other host reports — host ping, host disk, host uptime, host inventory, host release, host cleanup, service env show and the inference and stream reports — still publish the declarations alone and do not yet name the route that carried them.

Stado Desktop reads the same receipt: the Hosts inspector's Retained Tailscale logs panel shows the route under Route, between the reported host and the reported command, and says that this Stado did not report the route when the receipt predates the field rather than naming a declared path it cannot vouch for.

Manage the declarations without editing the registry document by hand:

stado registry host path list example-host
stado registry host path set example-host nebula --ssh operator@example-host.nebula.example --priority 1
stado registry host path set example-host primary --ssh operator@example-host.example
stado registry host path remove example-host nebula

set and remove accept --json; Stado Desktop uses those typed receipts rather than parsing terminal sentences. In the Desktop Hosts inspector, Host-control routes shows every declared destination and probe answer, marks the route Stado selected, and opens the same set/remove commands behind a review. Beacon network path remains separate because it describes how the host published its beacon, not how Stado reaches the host.

A refused path change answers in the format the caller asked for and states its own code. Registry validation refusals — a host identity declared by both ssh and an ssh_fallbacks entry, for instance — carry error_code: refused with retryable: false and the validator's exact sentence, and a --json caller receives that as one document rather than prose. A deterministic refusal reported as unknown, under "we could not attribute the failure", sends an operator to inspect credentials for a change the registry rejected on its own declared rules.

When a host answers through one of those routes but its beacon is stale, stado host link <target> also reads com.wisent.host-health-beacon's declared log and returns a beacon_publisher diagnosis. The exact verifier_unavailable diagnosis is repairable with stado repair stado --step link --target <target> --apply: Stado resolves the stado-object-api authority from the service directory and reconciles that authority's verifier, so the token of the item playing role host-health-api is readable there under the existing least-privilege verifier grant, without rotating either bearer. It then reads the host's beacon once: a newer, fresh beacon closes the open silence; otherwise the step refuses, saying the repair is proven by the host's next beacon, read with the same check. It restarts no service and refuses every other publisher diagnosis rather than guessing.

Stado Desktop shows Repair beacon publication only for that repairable diagnosis. The action runs the same command and keeps its success or refusal on the Hosts screen; a stale host with a different publisher failure remains diagnostic-only until its own exact repair exists.

host exec accepts only fixed, allowlisted argument vectors, not arbitrary operator-supplied paths or shell expressions. Its fixed diagnostics include directory metadata, selected configuration files, and bounded unit logs. A managed unit's owner-controlled env file — the one a launcher .-sources, not the one the unit file declares — is read with stado service env show <service> --host <target> --env-file <path>, through the same channel and the same $HOME confinement stado service env set writes through; without --env-file, env show reads the environment the unit file declares. Only values whose shape cannot carry a credential cross the channel: endpoints without userinfo, query or fragment, host:port, plain numbers and variable references, because those are what an operator must verify. Every other value — a URL carrying userinfo, a word, a boolean, a token — is withheld on the target whatever its key is called, because a key's name does not say whether its value is a secret; --reveal <KEY> shows one key's value on purpose. An unparsable line is shown unless it contains =. stado service env check reconciles the loopback endpoints that file declares against the target's own socket table and exits non-zero when a declared dependency is dead.

A configuration surface Stado can write and cannot read is not a boundary, it is a blind spot: a write to a managed unit's endpoint can be reverted by another writer, and without a read-back no command shows either fact. stado service env set therefore reads the key back through the same channel after writing it and exits non-zero unless the file's effective assignment holds what it wrote. The comparison happens on the target, so a secret is verified exactly without its value returning.

The pool a gateway routes over, read on its host

brama subscription list on a workstation reads the pool of the gateway Stado's directory hands that machine, and that route is the machine's own resolver adapter: with the resolver stopped, or on a machine that declares no adapter for Brama, the pool of a serving gateway could not be read at all, and the operator's purchased accounts were compared against nothing. Three fixed reads answer it on the host that serves the gateway, through this channel:

stado host exec <target> -- start-with-skarbiec subscription list --json
stado host exec <target> -- start-with-skarbiec subscription list --json --refresh-usage
stado host exec <target> -- skarbiec list

The first two run Brama's own CLI through the release launcher the gateway's unit runs (~/.stado/services/brama/current/<platform>/bin/start-with-skarbiec, resolved on the host from the platform directories a release lays down), so they read the vault the gateway routes over under the same Skarbiec authority and workload identity, and answer the gateway's own pool document: every member under subscriptions with its provider, account, standing, usage window and last failure, and accounts.capacity per real account. --refresh-usage first reads each provider's current usage report; neither starts a sign-in or a model request, and neither takes a membership document, a gateway origin or a bearer. The third lists the vault's live item metadata on the host that holds it — id, tags, revision, no field and no value — which is the inventory the pool is made from (the items carrying Brama's tags) and the inventory a workstation reading the fleet vault through its own adapter cannot ask for while that adapter is unserved. A spelling with anything else (--apply, --gateway, --all) is refused by the allowlist, naming these.

Whether a builder's Docker daemon answers

A Supabase schema release's post-build test starts its scratch database through the Supabase CLI, which needs a running Docker daemon on the builder. Whether one answers there is a fixed readiness read:

stado host exec <builder> -- docker info

It runs the Docker CLI from the paths the fleet installs it at (/usr/bin/docker, /usr/local/bin/docker, /opt/homebrew/bin/docker), takes no argument beyond info, and starts, stops or pulls nothing; Docker's own answer names the server and its state, and a daemon that does not answer is Docker's own refusal.