Remote commands and the authority's snapshot

How a remote command reaches a host whose service directory route is held, and what the registry authority's status and snapshot answer from.

A remote command rides the host channel to the host it names. The channel is a new SSH connection, not the resolver's session: a resolver session to a host can stand while the host answers a new session at once, and refusing every command to such a host left its one diagnostic unsent. Every command that rides the channel (host exec, host uptime, host ping, service logs, a delivery, a reclaim) therefore reads the published list first, and when this host's resolver holds connections to that host unanswered longer than the refresh interval it asks every declared path to the host, even a single one. A path that answers carries the command, saying stado: <host> answers over <path> (<destination>) although this host's resolver holds it: <held>; the command goes over that channel on stderr, where <held> is this host's resolver holds <N> connection(s) to <host> unanswered longer than the <interval>s refresh interval it declares (services <names>): <the held wait's sentence>; stado resolver status lists every one. When no path answers the refusal is <held>; target <host> has no reachable SSH connection path (<each path's failure>), and when the channel key cannot be read from the vault it is <held>; and the key <host>'s command channel authenticates with cannot be read: <cause>, so the vault's error never stands in for the host's state. A published list that cannot be read refuses with whether <host> answers cannot be told before sending to it: <cause>. A command to this machine itself rides no channel and is not read against the list. The real test tests/service_directory/held/held.rs sends host uptime to the held fixture's target from a machine that is not it and reads the held reading beside the key failure.

A declared repair is the exception: stado repair <service> --target <host> --apply sends its steps to a host whose services stand, saying stado: sending the declared repair of <service> to <host> although it stands: <the held sentence> on stderr, because that repair (stado repair skarbiec --step crypto, Skarbiec's own recover-daemons) is what clears the wedge holding them, and refusing it would leave the host with no product path out.

The registry read every host command starts with asks the same of the registry authority before it runs stado resolver snapshot there over SSH: an authority whose services this resolver holds unanswered past the refresh interval is not asked, with the same held reading (this host's resolver holds <N> connection(s) to <host> unanswered longer than the …; whether the registry authority <host> answers cannot be told before asking it: <cause> when the list cannot be read), and the read goes on to the store and then the last-known-good copy, announcing which. A snapshot command sent to such a host reads the same vault and stood with it, and every host command stood behind it before sending anything of its own.

Every command run on the authority over SSH — stado resolver snapshot, a registry write, a key or route step — passes each line it writes to stderr on to the caller's stderr as it arrives, as <destination>: <line>, and keeps the whole of it for the refusal when the command fails. The authority's Stado announces its own waits there (czekam: …, koniec czekania: …), so a caller standing minutes on the authority reads what the authority itself stands on instead of only how long it took.

A request already sent when the route it rides stops answering is not waited on without end either. Every request that rides a resolver route — an object API request (the registry read behind every command, stado resolver status among them), a Skarbiec request, and the authority's stado resolver snapshot over SSH — reads the published list again at every refresh interval the target declares while it is unanswered; once a connection on the adapter it rides, or to the host it is sent to, has waited longer than that interval, its own included, the request is dropped, which closes its connection and gives its place back, and the read is refused: <url> request abandoned unanswered: this host's resolver holds <N> connection(s) on that adapter unanswered longer than the <N>s refresh interval it declares, … for an adapter, <host> request abandoned unanswered: <the held reading> for a host, with the exit code of a store that does not answer. The authority's snapshot over SSH stands in no list of this host's — the SSH connection opens in a second and the remote command answers nothing — so a snapshot read that outlasts the interval with nothing published held is abandoned too: <host> request abandoned unanswered: it stood longer than the <N>s refresh interval this host's resolver declares; the host's own stado resolver snapshotis what answered nothing:stado host exec -- stado waits --jsonlists what every Stado process there waits on, andstado service logs com.wisent.stado --host says what its unit is doing, and the reader goes on to its store or its last-known-good copy, announcing which. When the host runs a Stado older than the release that declared stado waits (or any approved Stado verb), stado host exec says so instead of ending on the usage error: the Stado installed on <host> does not take : it exited with the usage status, so the release it runs is older than this Stado <version>, which declares the command; stado release statusshows the release each host was asked to run and the one it runs; exited <code>; its last line was: <line>. The journey tests/host_silence/silent_authority/main.rs runs it against a real OpenSSH server on a loopback port that runs the real stado as the authority: it answers one read, is stopped with SIGSTOP and then accepts every connection and sends nothing, as a hung host does. stado resolver status therefore ends with the address it stood on instead of standing for hours on the registry read of the resolver it diagnoses. A client reaching a route without a resolver adapter, or on a host that declares no resolver, has no such reading and waits as before. No other bound exists: the one measure is the resolver's own refresh interval.

The authority line of stado resolver status says how long the authority took to answer: authority <target> source=ssh reachable generation=<N> answered_after=<seconds>s (--json: authority.answered_after_seconds, null for the local authority, whose document is in hand). An authority that answers only after longer than the refresh interval the target declares is a blocker, the registry authority <target> answered resolver snapshot over SSH only after <seconds>s, longer than the <interval>s refresh interval this target declares: that wait is the cause of the stale snapshot and of every relay held to that host, so the verdict is degraded and the line names it instead of reporting the authority as reachable alone.

What the authority's snapshot answers from

stado resolver snapshot on the registry authority does not read the registry store. The authority's store is its object API, which authorizes each read through Skarbiec, so a Skarbiec that stopped answering left the snapshot, and every registry read of every host behind it, standing. The command asks the authority's running resolver for the generation it holds in memory, at the API address that resolver publishes in its state file (api_bind in ~/.stado/resolver-state.json), in one registered wait on that loopback address (stado waits lists it while it lasts). The resolver reads the store itself on its declared refresh interval and keeps answering the generation it last read while a refresh fails. The command prints the same {"store_version", "document"} every reader already reads, and is refused at once, so the caller goes on to its own last-known-good copy, when:

• no resolver has published its state on this host, or the process that published it is not running;
• the running resolver publishes no API address (a resolver older than this behaviour);
• this host is not the registry authority (<host> is not the registry authority; its resolver holds a copy read from <authority>);
• the held generation is older than the declared max-stale (the registry generation <N> this resolver holds was loaded at <time> (<seconds> s ago), past the declared max-stale of <seconds> s; every refresh since failed), and stado resolver status on that host names the failed refresh.