Held connections on a resolver's adapters

Held connections

The object API publishes what it stands on at the vault the same way, in ~/.stado/resolver-vault.json: per bearer key (an object namespace, a release publisher item, the host-health role) the czekam line of the consultation holding it with open_since, and the blad czekania line of the last one that failed. It is the document /api/state.json serves under vault, written on every change, so it answers with the process stopped. stado resolver status lists it under vault and turns two states into blockers: a key whose last consultation failed (the object API's last consultation of the vault under <key> failed, so requests under <key> are refused with that line while one of them consults again: …) and a key held longer than the refresh interval (… has stood <N>s, longer than the <N>s directory refresh, and every request under <key> is refused behind it: …). A document that is there and cannot be read is a blocker naming the file. tests/storage/vault_consultation/ proves the file follows the live document through a held read and a revoked grant.

The resolver itself refuses the same way. A new connection accepted on an adapter whose published list already holds one unanswered longer than the refresh interval is not relayed: before anything is read from the client the adapter writes the refusal with the held wait's own sentence (not relayed: this host's resolver holds <N> connection(s) on that adapter unanswered longer than the <N>s refresh interval it declares, …) and closes, so a client that does not read ~/.stado/resolver-waiting.json — a curl in a git hook, a program on another machine — learns at once what it would otherwise learn never. The held connection keeps its place in stado resolver status; no clock ends it on its behalf.

One thing does end it from the resolver's side: the adapter's listener running out of file descriptors. Every held connection is a descriptor the resolver keeps for a request nothing answers, and an adapter whose accept fails with Too many open files serves nobody at all. On that one accept error the resolver ends every relay that has waited longer than the refresh interval the target declares, answers each such client with an HTTP 502 whose body is this host's resolver ran out of file descriptors accepting on <bind> (<accept error>): this connection had waited longer than the <N>s refresh interval for an answer that never came, so the resolver ended it to accept again, logs each shed wait as stado resolver adapter service=<s> consumer=<c> shed the wait of <client> to <endpoint> on <host> held since <instant> and the count as shed <N> held relay(s) after <error> and accepts again; <M> held wait(s) stand on channels still opening, gives their places back, and accepts again. A wait still opening its channel to the host (phase: open) stands on the SSH session, not on a relay, and is not shed. When no held relay is older than the interval, the adapter ends as before and stado resolver status shows failed with <bind> accept failed: <error>; no relay older than the <N>s refresh interval could be shed (<M> held wait(s) stand on channels still opening), so the descriptors are held by something this adapter cannot free.