Release diagnosis

Runbook

A release did not land on a host

First command:

stado release version show --host <target>

Verdicts: in-sync, host-behind, host-ahead, host-missing, and unknown for a binary whose installed version could not be read — unknown is never folded into another verdict, so an uninstalled reporter cannot masquerade as drift. version show exits non-zero on drift alone; stado release version converge --host <target> delivers the declared version and exits non-zero unless every binary in scope is confirmed in-sync.

When the release run itself ended Failed after its platforms were published (a delivery that failed on one host, a promotion that did not answer), converge no host by hand. stado release resume <run> reads every published coordinate back, puts the run into delivering in the same write that clears its failure, and only then queues a new attempt for each failed delivery. A delivery its host claimed while the run read Failed refused itself with refusing the stale coordinate: the run is Failed, not delivering; that attempt is one of the failed deliveries the resume queues again.

If the host keeps refusing a version it was already given, check quarantine. The release agent quarantines a digest that failed to become ready, because a candidate that dies in ninety seconds must not respawn in a loop.

stado release quarantine list <product> --target <target>
stado release quarantine clear <product> --target <target> \
  --digest <digest> --reason "<why this digest gets another chance>"

Read the CAUSE column before reaching for clear. A cause is what the agent observed where it failed — a manifest without rollback compatibility, a candidate pid gone while another process listens on its port, a readiness request its HTTP client gave up on, a product's own wisent-errors envelope — never a sentence read out of the candidate's log; a record with no such observation reads unclassified. A cause that names the candidate — credential_store_unreadable, credential_cannot_serve, capability_routes_unmapped, capability_redemption_refused, rollback_compatibility_undeclared — is held from its first retained quarantine, not after a retry count. A live predicate proving recovery releases the next candidate; an unavailable check keeps the hold and reports its failed command and output. Without a predicate, clear the retained cause explicitly. release status and Desktop Releases show the same detail; a new version alone does not prove recovery.

See quarantine admission qualification for the real CLI journey and its isolated release-fixture requirements.

stable_bind_occupied is such a wall and names a port, not a release: some other process holds the stable bind or a candidate port, so the candidate exits without binding. The agent asks the kernel who holds the bind before it spawns anything, reports that holder by pid and command, and spawns nothing. Name the holder with stado service serving <product> --host <host> and the loopback ports the host's resolver claims with stado resolver status; the registry refuses a resolver adapter or api_bind that claims a port a release target on that host declares, so the repair is to move whichever declaration is wrong and then clear the digest. When the resolver's own API holds the port, stado resolver api reassign --target <host> asks that host for a loopback port it hands out now (stado host local port run there), records it as targets.<host>.service_resolver.api_bind under the generation it read, and prints the previous and new bind (--json adds the generation); the host's resolver rebinds when it next reads the registry, and consumers ask the resolver at every start, so none keeps the old address.

readiness_probe_unanswered and release_process_vanished are not that. A host that could not answer the readiness probe, or that reaped the process, said nothing about the bytes it failed to run, so the agent retires that record itself on a later tick, writes its own line in the same audit trail beside the host's rollout state — with the memory a new allocation could obtain and the free space on the state directory's volume at that moment — and rolls the desired digest out again. A digest the agent already retried is retried again only once the host has more of either than at every earlier retry of it: a retry that failed with that much room says the release needs more, and waiting longer changes nothing. Until then the state document's detail names the last retry and both readings, then and now; an audit line written before readings were kept does not hold a retry back. quarantine list --json and release doctor --json carry agent_retires per row, Stado Desktop's Releases screen shows those rows as The agent retires this one, and release status prints the same account in the state document's detail.

Without automatic retirement, one missed probe on a host that briefly ran out of disk would keep a release quarantined indefinitely, and every command that resolves a release-controlled binary would answer no observed active release (phase Quarantined) — credential reads, grant reads, and release catalog declare-publisher with them.

clear starts nothing and kills nothing: it removes one map entry, and the agent's next tick finds the desired digest no longer quarantined and rolls it out on its own. --reason is required and recorded in the audit trail beside the host's rollout state. See primitives/release.