Operations

Host health publication

stado host beacon collect builds the document a host publishes about itself: every unit identity the registry declares for that host, the state its own init system reports for each one, the free space on the volume the fleet writes to, and the host's memory readings. --publish hands the document to the same route stado host beacon publish FILE takes for a document collected elsewhere; without it the collection is printed and nothing is sent.

A unit's state is active, failed, inactive when every domain answered that nothing is there, or unreadable with the cause in detail when a domain refused the read or the read failed. Those are three different facts and they stay apart: the collector this replaced read each label with launchctl print … 2>/dev/null || true and published inactive whenever the read produced nothing, so every system-domain daemon on a Mac without passwordless sudo — the always-on gateway among them — was published as not loaded, and stado service status, registry doctor and Stado Desktop each repeated it. Reading a launchd domain or a systemd unit needs no privilege, and neither reader asks for one now.

Every read a host makes of its own registry entry - the unit states, the disk inventory, the cache census, the link trace - runs on the host itself, never over SSH to itself. The entry is recognised as this machine by the one identity rule a target has: its name, each declared hostnames alias and the host part of each declared SSH path, compared to the kernel's own hostname with the mDNS .local suffix dropped and case ignored. It is the same rule stado host beacon collect finds the host's own entry by, so the entry a host collects a beacon for is the entry whose commands run locally. stado host link <target> --json says which route a command took: selected_connection is local for this machine's own entry, declared by name alone, by alias or by SSH host; the real test tests/host_health/beacon/own_channel.rs drives all three shapes. When the two readings used different rules, an entry declared by name only was found for its beacon and then reached over SSH to the same box for every script of the pass, through sshd and a login shell, on every health interval.

A GitHub runner unit is judged by its listener, not by its wrapper. What the init system holds for a runner is runsvc.sh, start-runner.sh or a reconcile script; the process that takes jobs is its child, the runner release archive's bin/Runner.Listener. The install is taken from the unit's own launch chain, in this order: its program's directory (or the parent of its bin/) holds GitHub's .runner registration file; or the launcher names the install it selected in .stado-runner-install beside itself; or the unit's running process works in a registered directory directly inside that directory. wisent-backend's release runner uses the record: its launcher selects vendor-<version>-layout-<n>/, writes that path to .stado-runner-install, changes into it and execs run.sh. The record matters because run.sh exits 0 when the listener stops for good, and launchd then shows a loaded unit with no process to follow. A registration that merely sits next to a program does not make its unit a runner. When a runner unit reads as active and its install's listener is not in the process table, it is published failed, and detail names the listener it looked for and ends with the last lines of the newest _diag/Runner_*.log — the line that says why the listener exited or never started. A directory, registration file, install record, working directory or process table the collector cannot read, or a record naming a path outside the launcher's directory, is appended to detail, and the state stays what the init system reported.

On a fleet host the publisher is the host's one Stado process: the catalog's com.wisent.stado declares --health-interval-seconds 60, and the process collects the document on that cadence and writes it as host_health/<host>.json into the fleet store its queue roles already use — the store stado host beacon list and stado service list read. It needs no beacon URL and no beacon grant, so a host cannot go unknown because its unit environment named no route. A local API the same process may serve (--api-local-store) is not that store and receives no beacon.

stado host beacon collect --publish and stado host beacon publish FILE require STADO_HOST_HEALTH_API_URL plus the dedicated stado-host-health-beacon Skarbiec URL/consumer/grant metadata. They resolve only the token of the item playing role host-health-api (the item tagged stado:role:host-health-api) and send the document to authenticated PUT /api/host-health. The control plane stores host_health/<host>.json through its configured Stado backend, so Azure and local outage profiles write to Azure Blob and local storage respectively.

Missing routing, an unreadable or over-broad grant, an insecure non-loopback HTTP URL, failed authorization, and backend errors all leave the prior beacon untouched and return failure. There is no cloud CLI, provider SDK, direct bucket URL, ambient credential, or cross-backend fallback in the writer.

The two commands above read the non-secret API/Skarbiec origins from their environment (/etc/stado/host-health.env on systemd hosts); the opaque Skarbiec grant stays owner-only at ~/.stado/host-health-beacon-skarbiec-token.

Reading admission and diagnostic failures

stado host gates HOST --json reads the registry, current disk usage, janitor state and snapshots, effective agent storage configuration, storage client, published capacity and queue. Each read retains its result or concrete error and reports its operation, source, state, start and finish time, elapsed milliseconds and error in observations. Elapsed time describes the read; the host-gate observer does not use it to cancel the operation. Independent reads run together. A failed host read does not remove a completed capacity or queue reading.

complete: false means the diagnostic could not finish; claiming is then null, and the command exits non-zero. Observation states distinguish complete, partial, absent, cached, error and skipped. A cached registry is identified as historical, not presented as a successful current read. An unread source is not the same thing as an absent object.

agent_declared_not_loaded is a blocker when the host's newest beacon does not report its declared agent unit active. A beacon that carries no units map says nothing about the agent, so it is that blocker too, naming the beacon's object path, and a unit the beacon lists without a state is reported with the entry the beacon carried rather than as an empty state. A host with no beacon at all is the separate no_capacity_publication silence.

The memory: line reports what memory this host has left — available X of Y GiB, swap N% used — and memory evidence: names where that reading came from: capacity_publication while the host's agent is publishing, and host_memory_measurement when it is not and the command read vm_stat and sysctl (or /proc/meminfo) off the host itself. Memory is a reading only and never a blocker. The second source is what makes the line useful on the host an operator looks at when it has stopped answering: the one whose agent has no memory left to publish with would otherwise print memory: not observed.

A figure the host's capacity publication does not carry — free or total cores, RAM or VRAM, the running job count, its age, a reservation's fields — prints as not reported on the capacity: and reserved: lines, and a reading this command could not take (disk use, memory, its evidence, the local snapshot count) prints as not observed; neither is ever printed as zero, - or ?. Figures print as the host published them, and waiting jobs are named by their whole job id.

The queue read walks the priority index — queue_priority/, which names queued jobs and nothing else — and returns the pinned jobs it finds, oldest first, up to 200. It is partial when that window filled, with the detail 200 pinned jobs filled the window; more may be queued for this host; a partial read still decides the verdict. Reading every object under queue/ instead would include one transition sentinel per job that ever left the queue, and on a busy queue every host's read would time out and every verdict would be claiming: unknown. The coordinator's reaper retires a sentinel once its job has settled in a terminal prefix and the object is a day old, at most 100 per prefix per tick, and logs reaper: queue/ settled sentinels retired=N kept=M inspected=K when it removed any.

When placed workloads hold the host, the text carries a reserved: line — N placed workload(s) hold C core(s), R GiB RAM, V GiB VRAM; the capacity line is net of them — and one line per hold naming the kind, product, holder and acquisition time; capacity in the JSON carries running_workloads, reserved and reservations. The accelerator: line says who holds the card, from the same publication (space).

disk.free_bytes is measured by the host's filesystem reader; free_gb is its GiB display value. used_percent, headroom_gb (what may still be written before the disk-full rule deletes; negative past it), full_percent (80) and full judge that measurement against the rule. These fields are null when the measurement is unavailable, never zero. pressure_source distinguishes the agent's current capacity_publication from host_disk_measurement; capacity.diagnostics retains the agent's actual admission reason and reported state. Low free space and the agent's decision are different readings. Neither a slow queue read nor an unavailable disk reader proves that the disk is full.

Stado Desktop shows the same results in Hosts → selected host → Gates: completed measurements, unknown admission when reads are incomplete, each source and error, and the agent's published diagnostics. Reading this screen or running host gates does not clean a disk, restart a service or change policy.

Missing service reconciliation

Every autonomy pass reconciles each host's declared services against its newest beacon and a reachability sweep, retires units a product's one process replaced, and reports what it did: Missing service reconciliation.

Every command below enters through Stado. Provider diagnostics belong inside the corresponding adapter and are unavailable unless that provider is explicitly enabled in the selected profile.

Storage root authority handoff

Inspect, run, resume, roll back and finalize a storage root handoff, from the CLI or Stado Desktop, and what each answer means: Storage root handoff and space reclaim.

Space reclaim

The one built-in rule every host's space reclaim follows, when a pass runs and how to operate it: Storage root handoff and space reclaim.

Isolated GitHub pre-check runners

Install, inspect and remove the isolated pre-check runner, and what it may reach: Isolated GitHub pre-check runners.

Common queries

The commands that read the fleet, the queue, quota and billing, a local agent's state and one job end to end: Common queries and failure modes.

Failure mode quick-grep

The failure modes, what each looks like and where to look: Common queries and failure modes.

Recover the Stado binary on a host

When a host's Stado binary or resolver is missing or broken, so declaration-driven delivery cannot run, follow the one break-glass procedure: Recover the Stado binary on a host.

Release / publishing

Release operations enter through Stado:

stado release catalog sync --root /path/to/registered-checkouts
stado release catalog audit
stado release submit --source /path/to/product --version <exact-version> \
  --channel candidate

Submit requires a clean committed tree but does not contact its remote. It archives the exact committed tree, publishes the create-only source object, records source and manifest identity in the Stado catalog, and creates one provider-neutral queue job pinned to a registry builder whose release_platform matches the recipe. Queue state and status/<job>/output/ remain the authoritative work and transport records.

Inspect or resume a run by repeating the same submit command. Its identity is derived from product, version, channel, source digest, and manifest digest; the durable stado://release-runs/<id>/run.json shows job IDs, output coordinates, delivery state, and failure. A terminal successful platform output is read from JobStorage and published, never rebuilt.

The release authority is configured by item name and trusted key ID:

release.signing_key_item = stado-release-signing
release.signing_key_id   = stado-release-2026-08

The Skarbiec key-pair item contains the base64 PKCS#8 value in private_key. Key bytes stay in Skarbiec. Build and delivery secrets are only checked-in item#field references and must also be permitted by agent.skarbiec.secret_fields.

Canonical publication is immutable and ordered:

release.tar.gz -> qualification.json -> release.sig -> release.json

The signed manifest is the commit marker. Required delivery jobs run only after it exists and consume its exact URI and digest. Optional mirrors do not gate canonical success. Runtime products then use the existing registry.release_control generation CAS and release-agent reconciliation. deployment.json is written only after every declared target reports the promoted version, artifact digest, and manifest digest exactly.

No Git forge credential, hosted workflow, provider repository checkout, or direct provider API is required by this chain. Existing external adapters may mirror completed releases, but their availability cannot change source, qualification, signing, desired state, or observed rollout truth. See release.md for the strict manifest and catalog contract.

Bringing up a new local box

# Install a verified Rust release and resolved profile first.
./install-stado.sh
export STADO_CONFIG="$HOME/.stado/config.json"
stado config validate
stado doctor --fix-hints

# Writes the one com.wisent.stado unit running `stado serve` with every role
# the registry declares for this host.
stado bootstrap --local --target <registry-target>

# Health publication additionally requires the dedicated
# stado-host-health-beacon Skarbiec grant and non-secret Stado/Skarbiec origins
# described above. It never requires a cloud login.
stado host health <registry-target>