The registry authority, its copies and held adapters

How a service directory resolver reads the registry authority and its last-known-good copy, moves the authority, and treats an adapter whose service holds its requests.

A resolver whose registry authority is another host reads its first snapshot from that host over SSH before it binds anything. An authority that accepts the connection and answers nothing does not keep the resolver in state starting with no listener: the first read is given the refresh interval the target declares; past it the resolver says stado resolver recovery: the registry authority's first snapshot did not answer (<the abandoned read's sentence>); serving generation <N> from the registry store immediately while authority retries continue, binds its adapters from the registry its store (or its last-known-good copy) already holds, publishes serving, and keeps asking the authority on every refresh. While that first read stands, the published state says what it waits on: reading this host's registry store before binding any port, then reading its first registry snapshot from the authority <host> before binding any port, so connect and stado resolver status name the read (… holds no listener: reading its first registry snapshot from the authority <host> …) rather than report that nothing was published. tests/service_directory/held/silent/ runs the real resolver against a real OpenSSH server as the authority, which greets once and is then stopped with SIGSTOP so it accepts every connection and never speaks, and relays a client through the adapter it bound to a real object API.

A refresh whose authority does not answer reads this host's last-known-good copy too. When that copy's service_directory.generation is above the one the resolver serves, the resolver adopts it, says stado resolver: the registry authority did not answer (<cause>); adopting the newer registry this host was handed, directory generation <N>, and from then on asks the authority that copy names. This is how a registry naming a new authority reaches a host whose old authority is down: it is written into the host's copy, and the resolver takes it on its next refresh without a restart. The generation is the fence: a later answer of the old authority carries a lower generation and is refused as service directory rollback rejected: generation <old> < <new>. A copy that cannot be read is named beside the authority's own refusal. When it holds no higher generation, the refresh fails with the authority's refusal, as before.

The copy is written on each host with stado registry push PATH|- --last-known-good [--json]. It validates the document as every push does, then refuses when the authority answers (the registry authority answers; a registry it would serve is pushed through it, without --last-known-good), when this host is the authority or keeps no copy naming one, and when the document's service_directory.generation is not above the copy's (… not above the <held> this host's last-known-good copy holds; a copy is never rolled back). Otherwise it writes the copy with the store version handed-over and prints handed <path> to this host's last-known-good copy at service directory generation <N>; the authority did not answer: <cause>, or under --json {"state": "handed-over", "directory_generation", "authority_unanswered"}. It cannot be combined with --if-generation, --force or --allow-empty-fleet.

stado placement move --authority --to-host <host> [--json], run on <host>, takes the registry authority role while the declared authority does not answer. It refuses on any other host, when <host> is already the authority, when this host keeps no last-known-good copy, and while the authority answers (the registry authority <old> answers; move the role through it with stado placement move on <old>). Otherwise it promotes this host's copy: service_directory.authority names <host> and this program's own path as the command resolvers run there, and the directory generation advances by one. The promoted document is validated against the copy, written into this host's copy (its resolver adopts it and reads its own store from then on), written into the store this host reads, and handed to every other registered host except the old authority with registry push - --last-known-good over that host's SSH paths. The receipt names the new authority, the fenced one, the generation, the authority's own refusal, the store's version or error, and each host's receipt or error; its state is taken only when the store and every host took it, and otherwise the command fails naming the store's result and every host that was not handed the registry.

The object API publishes what it stands on at the vault the same way, in ~/.stado/resolver-vault.json: per bearer key (an object namespace, a release publisher item, the host-health role) the czekam line of the consultation holding it with open_since, and the blad czekania line of the last one that failed, plus the keyring_lock_repair it ran (holder, started, outcome) when a failure named the process holding the vault's keyring lock. It is the document /api/state.json serves under vault, written on every change, so it answers with the process stopped. stado resolver status lists it under vault and turns three states into blockers: a key whose last consultation failed (the object API's last consultation of the vault under <key> failed, so requests under <key> are refused with that line while one of them consults again: …), a key held longer than the refresh interval (… has stood <N>s, longer than the <N>s directory refresh, and every request under <key> is refused behind it: …) and a keyring-lock repair still running (… runs since <started>: the vault names that holder, so consultations fail until the lock is released); the text report prints the repair as vault keyring-lock-repair holder=… started=… outcome=…. A document that is there and cannot be read is a blocker naming the file. tests/storage/vault_consultation/ proves the file follows the live document through a held keyring and a revoked grant.

The resolver itself refuses the same way. A new connection accepted on an adapter whose published list already holds one unanswered longer than the refresh interval is not relayed: before anything is read from the client the adapter writes the refusal with the held wait's own sentence (not relayed: this host's resolver holds <N> connection(s) on that adapter unanswered longer than the <N>s refresh interval it declares, …) and closes, so a client that does not read ~/.stado/resolver-waiting.json — a curl in a git hook, a program on another machine — learns at once what it would otherwise learn never. The held connection keeps its place in stado resolver status; no clock ends it on its behalf.

One thing does end it from the resolver's side: the adapter's listener running out of file descriptors. Every held connection is a descriptor the resolver keeps for a request nothing answers, and an adapter whose accept fails with Too many open files serves nobody at all. On that one accept error the resolver ends every relay that has waited longer than the refresh interval the target declares, answers each such client with an HTTP 502 whose body is this host's resolver ran out of file descriptors accepting on <bind> (<accept error>): this connection had waited longer than the <N>s refresh interval for an answer that never came, so the resolver ended it to accept again, logs each shed wait as stado resolver adapter service=<s> consumer=<c> shed the wait of <client> to <endpoint> on <host> held since <instant> and the count as shed <N> held relay(s) after <error> and accepts again; <M> held wait(s) stand on channels still opening, gives their places back, and accepts again. A wait still opening its channel to the host (phase: open) stands on the SSH session, not on a relay, and is not shed. When no held relay is older than the interval, the adapter ends as before and stado resolver status shows failed with <bind> accept failed: <error>; no relay older than the <N>s refresh interval could be shed (<M> held wait(s) stand on channels still opening), so the descriptors are held by something this adapter cannot free.

A held adapter whose service lives on another host is first asked whether its held waits stand on the SSH session the new connection would ride. Every connection to one service endpoint on a host shares one native session, and that session can stop answering its channels while the host answers a new session at once. When the youngest held wait began after the current session answered, the session itself stands: the resolver opens a new session to the same host in its place, and when it answers the connection rides it, said on stderr as relaying although the adapter holds connections: the session to <host> for <endpoint> stands, … while a new session to <host> answered: new connections ride the new session and the held ones stay on the standing one until their clients leave. When the held waits all began before the current session answered, they ride one already replaced and the connection rides the current session. Only when no new session opens is the connection refused, with the held sentence followed by the session to <host> for <endpoint> stands with them and a new session in its place did not open: <cause>; a held adapter to a service on this host is refused as before. A service that holds its requests on every session costs one new session per refresh interval, since a wait counts as held only after it. The held waits a replaced session keeps carry session_replaced_at in ~/.stado/resolver-waiting.json and in stado resolver status waiting_opens, and the route's blocker ends with the SSH session it rides was replaced at <instant> by a new session to <host> that carries new connections, so it stands only until its client leaves. Such a wait no longer refuses a caller: held_at, the host-channel reading and the in-flight reading skip it, since the route it stood on is carried again.

The resolver does not wait for a connection to arrive before replacing a standing session: a caller that reads the waiting list refuses itself before it connects, so that connection would never come. On every refresh interval it reads its own waits, and for each route held past the interval on a session it has not replaced it opens a new session to the route's host in its place, saying stado resolver service=<name> consumer=<consumer> the session to <host> for <endpoint> stands, … while a new session to <host> answered: … on stderr, or the same prefix followed by the reason no new session opened. This runs in its own loop, so a host that answers no new session never holds the directory refresh.

A relay to another host that fails says on stderr how long it ran and whether the SSH session under it survived: stado resolver adapter service=<name> consumer=<consumer> rejected connection: proxy to <host>:<port> failed: <cause> after <n> ms; the SSH session to "<host>" is still open, so the service end closed this one channel when the service on that host dropped this one connection, or … is closed, so every channel on it failed together when the session died under every channel. The first points at the service on the destination host; the second at the session, which the resolver opens again for the next connection. The real journey cargo test --test service-directory-held -- closed_channel:: runs a real SSH server that closes every forwarded channel while its session lives, and reads the first sentence from the resolver's log.

A destination dials SSH's own port unless it names another the way known_hosts writes a server off that port: <user>@[<host>]:<port>. A native session (the resolver's relay and the registry authority read) dials <port> and trusts the host key known_hosts holds for [<host>]:<port>; a command that rides OpenSSH's ssh (the host channel, recovery, reboot, account and bootstrap commands) is given the ssh://<user>@<host>:<port> URI ssh reads. A port that is not a port is refused with SSH destination <destination> names port "<word>", which is not a port. The real test tests/service_directory/held/session/ runs a real SSH server on such a port whose first session carrying a channel stands while every other forwards to a service that answers: a first client's request stands on the resolver's session, and once it has waited past the refresh interval a second client on the same adapter is answered over a new session, the resolver says so, and the held wait carries session_replaced_at in the published list and in resolver status.

A route whose endpoint on a host is one of that host's resolver listeners — an adapter's bind, or the resolver API — is never relayed. Relaying it would send the connection back into the same process: accepted, forwarded to the same socket, accepted again, and held with nothing to answer it, which from the client looks like a healthy port with a hung service. The resolver refuses such a route when it resolves it, before any channel is opened, and the client receives the refusal as an HTTP 502 at once: service <name> resolves to <url>, which is adapter <service> for consumer <consumer> (<bind>) of this resolver on <host>: the service directory must carry the address the service listens on, not the stable address published for its clients; relaying it would loop back into this process and no client would ever be answered. localhost and the loopback address name the same listener. The resolver keeps serving; stado registry doctor names the same pair in the document so it can be corrected. The real test tests/service_directory/held/self_loop.rs drives it.

No clock cuts a relayed request. A request the service never begins answering stays in the published waiting list as phase: answer, where stado resolver status names it and every caller that reads the list before sending (the object API client, Skarbiec requests, Tama's judge client and stado service directory connect) is refused at once with the held wait's own sentence — for as long as the client that sent it waits for the answer. A client that closes before the service's first byte comes down (a hook process that ended, a caller whose turn was over) has nobody left to answer: the resolver ends that relay, closes its channel to the service, gives the place back and says request to <endpoint> on <host> abandoned by its client before an answer; the channel is closed; <N> open(s) still waiting, so a caller that died with its request sent does not hold the adapter against every later caller. A client that closes its sending side after the answer began is still read the rest of the answer. The former route field answer_seconds is retired: registry import refuses a route that declares it (unknown field \answer_seconds`) and stado service declare --file refuses a file that names it ('answer_seconds' is retired: a resolver never cuts a relayed request by a clock; …; remove the key). The real test tests/service_directory/held/held_answer.rs` holds a request against a service that never answers, reads it from the published list, closes the client and reads the list empty and the next caller relayed.