When the stable bind is gone

A release-controlled product declares one consumer-facing stable_bind and two candidate_ports under release_control.products.<product>.targets.<host>. Read those values from the selected registry; do not copy a port from another deployment. Consumers use the stable bind. The release agent brings up a candidate, probes its readiness_path, and switches the stable proxy only after that candidate qualifies.

The release agent is the only thing that publishes a stable bind. The legacy launchd daemon named by legacy_launchd_label does not: in a settled blue-green state that daemon is the live candidate, so restarting it moves nothing and only interrupts the running service. stado repair stado --step host reports candidate_live:<port> for exactly that case and touches nothing.

Why a namespace declared without its grant takes the stable binds down

The agent learns which ports to publish from release_control in the canonical registry, which it reads through the object API. The object API gates every non-release object read on its object authorization boundary, and that boundary is open only while the host's object verifier holds a read on the Skarbiec item of every namespace in object_api.namespaces. One namespace declared without its item in the grant closes the whole boundary, and the host's log says so:

object authorization boundary revalidation failed: Skarbiec deployment configuration:
object verifier grant item set mismatch (missing=[<namespace-credential-item>], unexpected=[])

The object and release verifiers keep no credential through a failed vault read: the request is answered 503 object authorization unavailable. A last-known-good registry may keep existing processes serving while fresh authority reads fail; its health is not proof that authorization recovered. The release agent reports when it uses its last-known-good registry and checks the authoritative store again on each tick. Diagnose the verifier's actual refusal rather than restarting a process that may currently preserve the serving path.

The repair

Read the boundary's own reason first, because it names the item:

stado service unit logs com.wisent.stado --host <host> --lines 300 | grep 'object authorization boundary'

Then declare the missing namespace on the machine you are running from — reconcile-object-verifier computes the item set from the local configuration, so a namespace that exists only on the host can never be satisfied from elsewhere — and reconcile the host's grant:

stado config set object_api.namespaces.<ns> '<the same JSON the host declares>'
stado repair stado --step object-verifier --target <host> --json

Read the exact grant report, then prove recovery through the real consumers. An exact: true grant result alone is not evidence that the stable bind is serving. Verify in this order:

stado storage stat stado://<queue-namespace>/registry.json      # state present, no 503
stado host exec <host> -- lsof -nP -iTCP -sTCP:LISTEN           # every stable bind held
stado service unit logs com.wisent.stado.release-agent --host <host> --lines 200  # no infra_down loop
stado service verify --host <host>

stado host config set warns when a declared namespace names an item not covered by the local configuration. Reconcile the actual declarations rather than adding an unrelated credential.

The fallback, and its reversal

The release agent first asks the authoritative registry. If that read fails, it reads this host's last-known-good registry and reports both the authority failure and the recovery source. An unreadable cache reports its own cause; a cached document identifying a different host is refused.

This fallback preserves the declared release-control inputs. It does not rewrite consumers to a candidate URL or select a port from an example. Each later tick tries the authority first, so recovery returns to the authoritative declaration without a manual configuration reversal. Read the actual stable-bind and consumer results before accepting recovery. Do not restart a serving candidate merely to recreate its separate proxy.

What now prevents the recurrence

• The release agent falls back to this host's last-known-good release_control when the authority cannot be read, the way the resolver already did for service_directory, and says release agent recovery: … when it does. A closed boundary no longer takes the stable binds with it.
• stado release version converge polls every declared stable bind of the units it restarted, for up to 120 seconds, and reports stable_binds with a verdict per port. It refuses to report ok while one is absent, so a roll can no longer succeed on paper through an outage.
• stado repair stado --step host reports every declared stable bind as already_bound, candidate_live:<port> or refused:<reason>, and bootstraps only a declared bind that nothing at all is holding.