On this page
A service that accepts and never answers
A service that accepts and never answers is recovered where it runs
For an operational reading of a standing request, see diagnosing an unanswered service.
A service that accepts every connection and answers none passes a readiness
read and fails everyone: its /health can answer while the path the service
exists for holds every request, every client's resolver records the waits,
and nothing on the host that runs the service acts, because nothing there
asks the question the clients are asking. A product's catalog service
declares that question as answering: the method and path of one request
on the service's own API, the JSON body to send, and the repair step of
its repair declaration (stado-rs/data/catalog/repair-catalog.json) to run
before a restart. Brama declares POST /v1/chat/completions. Skarbiec
declares GET /readyz with the crypto and audit-lock repairs: its
readiness decrypts the vault's canary item, so a gpg standing behind a held
keyring lock leaves the question unanswered, while a listing such as
POST /v1/items/list decrypts nothing and answers as fast as ever.
Stado declares
GET /api/release/object?uri=stado://releases/answering-probe with the
object-api repair: the public release route answers before host trust and
Skarbiec and reads the store, so a healthy host answers at once that the
object is absent, while an object API whose store holds every read leaves the
question unanswered and its declared repair runs.
service:
answering:
method: GET
path: /readyz
repair:
- crypto
- audit-lock
On every beacon pass the host process sends the question to the service's
recorded listen port on this host. The request runs to its end in a task of
its own and its start is recorded, the way the resolver records every wait;
the next pass reads the record. A question unanswered longer than this host's
declared directory refresh interval (service_resolver.refresh_seconds, the
same measure the resolver applies to a held adapter) is a service that
stands: the host runs each declared repair step through the same path
stado repair <service> --step <name> --target <host> --apply takes, then
the restart through the same path stado service restart takes, forgets the
question so the next pass asks the new process afresh, attempts an alert on
the configured channels, and appends the recovery to
~/.stado/service-recoveries.jsonl when writable. No manual step: the host
that runs the service acts first.
A refused restart remains a refusal in the beacon. Alert delivery is
attempted on the configured channels, with each channel's outcome retained;
the answering beacon does not present the attempt as recovered.
The alert says recovery was attempted, names the repair and restart outcomes, and never claims the register already contains the event: the append happens after the alert and may refuse. The beacon reports that refusal with its file operation and cause.
Once the declared repair steps and restart succeed, the host retains that
attempt for the continuous failed-answering episode. A later standing reading
reports already_attempted with the last recovery instead of repeating the
restart. The unanswered question is retired so the next beacon can ask afresh
and observe an answer.
A successful API answer ends the episode; a later failure can recover again.
A refused repair step or restart does not count as complete recovery: the
next beacon retries it; the beacon and alert name each refusal, and a writable
register retains it.
Replacing the managed unit or changing its declared listen port, method,
path or body retires the old in-flight question and answer. The next pass
asks the new endpoint rather than treating an old wait or HTTP result as its
state. Changing the repair steps starts a new recovery episode if the
question still stands, so an earlier attempted repair does not suppress the
new declaration. Removing the question or being unable to read its directory
endpoint retires its pending request and last answer; the beacon names an
unreadable endpoint instead of carrying stale readiness.
stado service status <service> --json and stado service list --json carry
the beacon's answering.recovery beside the failed API question; the plain
status prints the same answering block. Stado Desktop → Services → Inspector
shows the attempt, its repair-step refusals, restart cause, register refusal,
and failed alert deliveries on the service row.
An absent recovery for a standing question is shown as missing evidence,
not as a successful restart. stado service recoveries --service <service>
reads the recoveries the host could persist. If a restart ran but the register
append failed, answering.recovery keeps the observed restart and
recording_refusal names the file operation and cause. The host does not
restart the same failed episode again merely because that record could not be
persisted; the recovery-history command cannot prove the attempt, so inspect
the beacon's refusal.
When the host that runs it does not act
A host that runs an older catalog never asks the question that would find
its service standing, and a host whose own Stado stands on the service it
would judge never reaches the judgement. The other hosts see it anyway:
their resolvers hold the service's requests in phase: answer and refuse
new ones. So every host's beacon pass also reads its own resolver's waiting
list (~/.stado/resolver-waiting.json): a service on another host that has
held this host's requests for the first byte of an answer longer than the
declared refresh interval triggers a recovery attempt from this host: the same
repair steps and restart run on the host that serves it over its
registry SSH channel. The steps are fixed host payloads and the restart is
the unit's own init system command, so neither needs that host's Stado,
which may be the very process that stands. A wait still opening its channel
says the host or its SSH daemon does not answer, not the service, and is
not acted on.
One standing is acted on once after its declared repairs and restart succeed.
The record keeps stood_since, when the oldest held request was accepted,
and the complete answering declaration used for that attempt. A later pass
reports already_recovered only when the same oldest request still stands,
the managed unit and current declaration match that record, and every
declared repair step was applied. A changed declaration or an older record
without its snapshot cannot prove the current attempt; the peer acts again.
A refused repair step or restart is retried on the next pass. A restart ends
the held connections with the process that held them, so a service that
stands again is a new standing.
The beacon carries peer_recoveries: one block per service and host with since,
waited_seconds, bound_seconds, held (how many requests stand),
oldest (the oldest wait's sentence: client, request line, endpoint) and
one of recovery, already_recovered, or not_recovered with the reason
(the catalog declares no answering question or no such service, the
registry declares no unit for it on that host, the register cannot be
read). A waiting list that cannot be read is carried as unreadable.
If a peer's register append fails after a restart, its beacon names
recording_refusal and retains that outcome for the same oldest held request
while the observing host process runs. That is not durable history; if the
observer restarts before the register becomes writable, a later pass may
attempt the standing again.
The beacon carries each probe under the unit's answering: asked (the
first question was just sent), waiting with since, waited_seconds and
the newest finished probe, stood with since, waited_seconds and
bound_seconds plus the recovery record, or finished with answered
(status, the HTTP library's successful verdict, elapsed_ms, at, and
the actual response detail when the HTTP status is not successful) or
refused (the transport's own cause). A fast HTTP 503 is an answer,
not a standing request: its response detail diagnoses the failed readiness
operation, but the standing-recovery path does not restart the service for
that status. The CLI's service list/status prints the beacon's failed
successful verdict and response detail. Desktop's Services fleet table
and inspector mark the failed answer as needing attention and show that
detail; an older beacon missing successful is not shown as healthy.
A port the directory does not record, or a resolver that declares
no interval, is said in the block as unreadable, never guessed. A service
whose catalog entry declares no question carries no block.
stado service recoveries [--service NAME] [--json] reads this host's
register, oldest first: when, which service on which host and its unit, the
question that stood and the exact answering declaration used (older rows
lack this snapshot), since when and for how long against the bound, the
requests this host's resolver held unanswered to the service at the time,
the ordered repair-step reports (empty when none is declared; older rows
may hold one report or null), what the restart answered and whether it
ended well, what each alert channel answered, and for an attempt on another
host's service observed_by, the host whose resolver observed it. The plain
line derives its words from what the register holds: an attempt from another
host says recovered from <host>, whose resolver held its requests only when
the restart ended well, and tried from <host>, whose resolver held its requests, and not recovered when it was refused; a resolver waiting list
that could not be read is said as the resolver waiting list was unreadable (<cause>) instead of being counted as one held request.
A register that
is not there holds no recovery; one that cannot be read, or holds a line that
is not a recovery, is refused naming the file and the line. The catalog
refuses an answering whose method is not in capitals, whose path does not
start with /, whose repair is not a list of nonempty step names, or that
carries a field outside method, path, body, repair. Omitted or empty repair
steps are valid when only the restart is declared. Stado Desktop's Routes
operations carry Read this host's service recovery attempts, including
refusals, which runs the same history command on the selected host.
The internal probe exercise tests/service_answering/stands.rs (service-answering-stands)
asks the question of a real listener on this machine that accepts and never
writes a byte and reads it as standing once the fixture registry's declared
refresh interval has passed, asks it of one that answers successfully and
another that promptly returns HTTP unavailable with a diagnostic body,
classifies the latter as unsuccessful without losing the body, forgets the
standing one and sees the next pass ask afresh, and writes a recovery to a
register under an isolated home, reads it back whole and sees a line that is
not a recovery refused. It does not run a managed host's repair or restart.
tests/service_answering/cli.mjs reads an actual fresh host beacon through
stado host health, service status and service list, plus the unmanaged
service refusal. It requires a declared live host/service and a clean source
revision, but does not exercise a declaration change or standing recovery.
The resolver-list journey tests/service_answering/peer.rs
(service-answering-peer) publishes a waiting list as a running resolver does
under an isolated home. It shows that an undeclared standing service is
not_recovered with its cause, while a wait still opening its channel, a wait
on this host's service, and a wait younger than the bound are not standings.
A list whose publisher no longer runs names nothing, and a malformed list is
unreadable naming the file. It does not qualify a peer repair or restart:
that requires a managed service on a real other host, its current declaration,
and the resulting recovery and service state.
Source: this website