Where the service directory lives

Where the service directory is stored, and how a host's resolver serves, relays and holds its routes.

The service_directory block of registry.json in the configured canonical backend, served as versioned snapshots by the authority. Each host materializes its own view as forward markers under $HOME/.stado/forwards/<service>.local, and that host's resolver keeps them: it writes every marker the directory declares for the host when it starts, before resolver status reports serving, and again on every refresh, rewriting only a marker that is missing or holds another address. The resolver logs each write, and, once that host's markers follow a newer directory a refresh loaded, the generation it loaded:

stado resolver wrote forward marker <service> -> <url>
stado resolver loaded directory generation <N>

A service the directory gives the host no address for gets no marker, and a marker the directory does not declare is left alone for service directory publish --prune. Before the resolver kept them, markers existed only where an operator had run service directory publish by hand, so a consumer such as Weles could fail at start on a host the directory placed its dependency on. service directory publish writes the same set on demand and reports the fossils; stado host inventory reconciles marker against socket table and marker against directory as deliberately independent axes.

On a host whose configured store is another host's object API, every registry read — stado registry self, database resolve, service directory connect, the registry behind a host command, the storage layout — takes the authority's versioned snapshot over that registry-owned SSH transport before the store, in one place (the registry store itself), because the object API authorizes each read against the vault on its host and a vault that stops answering holds every such read with it. The authority is located through this host's last-known-good copy; a host that keeps no copy, or is itself the authority, reads its store. A snapshot that cannot be read is said on stderr with the authority's own sentence (stado: the registry authority's snapshot could not be read (…); reading the registry store at … instead) and the store is read, which the store read's own czekam line then names. Writes go the same way: on a host whose authority is another target, the compare-and-swap of every registry write (registry set, registry push, registry host …) runs on the authority as stado registry push - --if-generation <the generation this host read> --force --allow-empty-fleet --json over the same SSH transport, with the document on standard input; the guards ran on this host against that generation, which the authority's swap is conditional on. A write therefore never waits on this host's own resolver, so a resolver that refuses to start cannot hold back the registry change that lets it start. A moved generation is still exit 75; a write no SSH path carried says registry write through authority <target>: no SSH connection path answered (<path>: <cause>; …), and one the authority refused says registry authority <target> refused the write with exit <code>: <its stderr>. Every write is then read back the way every reader reads it, here through the authority's snapshot, and confirmed when the generation the swap returned holds the document sent, compared as a document, not as bytes: the snapshot carries the document re-serialized. A write that cannot be confirmed says registry write of generation <generation> at <location> is unconfirmed: and why: no object read back, another generation read back (another write landed since, or the read came from a copy behind the store), something that is not JSON, or a different document. registry push prints replaced=<the generation it replaced>, or created when it created the registry, whose JSON receipt then carries no replaced.

A resolver serves every route it can and names each one it cannot; one route it cannot serve does not stop it or any other route. On the authority host, whose store is storage.stado.url, an adapter declared on that same address can neither bind while the object API holds it nor read the registry while it does not, so that one adapter is left unserved with this resolver is declared to serve <bind> for service … and the registry it must read first is configured at storage.stado.url = …; every other route is served. An adapter whose port something else holds is left unserved with its bind error (could not bind <bind>: <cause>. Something else already holds this resolver port; find it with lsof …). Each is published in ~/.stado/resolver-state.json beside the served adapters with its reason (unserved), the next directory change tries it again, and stado resolver status reports the resolver degraded with the resolver does not serve the <service> adapter for consumer <consumer> at <bind>: <reason>. Only a resolution API that cannot bind still stops the resolver. A host whose authority is another target is not refused the store adapter at all: its resolver reads every snapshot from the authority over SSH, so a store address it serves is the route it hands its own clients, and the workstation keeps every adapter.

When the object API cannot serve the registry, the verbs that only read the directory — connect, endpoint, bind, show and profiles — answer from this host's last-known-good copy at ~/.stado/cache/registry-last-good.json. Each such answer first prints one line on stderr with the copy's age, its read_at, its generation and the authority's own refusal, for example reading the last-known-good registry copy from 44135s ago (…) because the authority did not answer: Stado object API error HTTP 503: {"error":"object authorization unavailable"}. A host whose store is a local directory has no such copy, and one that has never read the authority has none either; both exit with the authority's error. publish, consumer add, consumer remove and placement move never read the copy: a routing change committed against a stale generation is what the authority exists to refuse.

A wait leaves ~/.stado/resolver-waiting.json (described below) as soon as its client closes, so a service whose clients give up before it answers never shows an old wait there. Every request that ends is therefore also recorded per service in ~/.stado/resolver-answers.json: last_answered_at, when the service last sent the first byte of an answer, and, while it has answered none since, first_unanswered_at and last_unanswered_at, when the first and the latest request ended without one. A service with first_unanswered_at set has taken every request since that instant and answered none, however young each wait was when its client left. The first answer clears both.

A resolver adapter accepts its client at once; what it relays may never be answered. The resolver publishes every such wait beside its state, in ~/.stado/resolver-waiting.json: a channel open the destination host has not answered (phase: open), and a request that went up an open channel whose service has not sent its first byte back (phase: answer), each with the service, consumer, the adapter's bind, the destination host and endpoint, since, when the client began waiting — the accept of its connection for a request, since the client has held that connection from then on, through the channel open before its request went up; the start of the open for a channel open — client, the loopback address the client connected from, and for a request, request, the request line the client sent first (GET /api/object?uri=… HTTP/1.1) when its first bytes carried a whole printable line: the one fact that names which operation the holding process is waiting on, so many waits from one process each say what they asked for. A resolver publishes the list as it starts, empty, so the waits a previous resolver left are not read as held by this one, and a reader finds a list rather than none. stado resolver status lists them under waiting_opens with waited_seconds, and one that has waited longer than the refresh interval the target declares is a blocker: service <name> for consumer <consumer> at <bind> has waited <N>s for the first byte of an answer on an open channel to to <endpoint> on <host>: the client holds an accepted connection that receives nothing; ask that host what answers on <endpoint> held by the client at <address>:lsof -nP -i : names the process still holding it. The client address is what ties a held connection to the process holding it: a hook that ended hours ago is told from one still waiting by whether anything still holds that port, and a list an older resolver published, without the address, names no holder. Waits on one route — the same service, consumer, bind, phase, endpoint and host — are one blocker, since they stand behind the same thing: a route held for hours keeps every connection its clients open, and the blocker reads <N> connections on one route, the oldest: <that wait's sentence>; a route holding one connection reads that sentence alone. waiting_opens in the JSON still lists every wait. A request that would go through such an adapter is refused before it is sent, with that sentence, instead of standing behind the held connection for as long as it stands: every object API request (Stado object API not asked: GET <url> not sent: this host's resolver holds <N> connection(s) on that adapter unanswered longer than the <N>s refresh interval it declares, so a request sent now would stand behind them: …), every Skarbiec request (Skarbiec not asked: …), and the probe connect makes of the route it hands out (<service> is placed on <host> and did not answer at <url>: this host's resolver holds …), each with the exit code of a store that does not answer. The refusal ends when the service answers the held request or the connection ends, and stado resolver status says which it is waiting for. A waiting list that is there and cannot be read is itself the refusal, naming the file.

The same list is read before a command sent to a host over its fleet channel; a host whose services it holds is not refused for that, but asked over every declared path, as a remote command describes.

A route through this host's own resolver adapter that nobody answers at all (connection refused on the adapter's bind) is the resolver's doing, not the service's. connect reads what the last serve --resolver on this host published (~/.stado/resolver-state.json, the file stado resolver status reads) and says why beside the refusal: when that resolver holds no listener, <service> is placed on <host> and did not answer at <url>: <connection refused>; this host's resolver (pid <pid>, state failed at <time>) holds no listener: <the resolver's own reason>; when it serves other routes but left this one unserved, …; this host's resolver (pid <pid>) does not serve <bind> for <service>/<consumer>: <its reason>. A resolver that published no reason is said to have published none. The state counts only while the process that published it still runs: the file outlives the resolver that wrote it, so a dead resolver's listening: true is never read as ports someone holds while its replacement is still starting. Whether it runs is read from ~/.stado/resolver-state.lock, which the resolver locks from its first publication until it exits and in which it records its pid, never from the pid alone, which the system hands to another process once the resolver is gone. A lock any reader can take means no resolver runs; a held lock counts for the published state only when its holder is the pid that published it. When that process is gone, connect, stado resolver status and every program reading the published adapter say the resolver that published this host's state (pid <pid>, state <state> at <time>) is not running, and no resolver has published since; its listeners went with it. The reason is the one to act on: a configuration contradiction or a held port stays until the configuration, the holder or the installed build changes, and no retry of the connect changes it.