What a machine-interface command is waiting on

How a command of the machine interface says what it waits on while it waits, and refuses a request that would stand behind the vault.

What a command is waiting on

A command that waits on the fleet store, a lock, a host or a child process says so on stderr before it waits, so a command that has been quiet for twenty minutes names the request it stands on instead of looking hung. Every wait writes up to three lines, each one write ending in a newline:

czekam: <what>; gdzie: <where>; rodzaj: <kind>; od: <UTC time, RFC 3339>; id: <pid>-<n>
koniec czekania: <what>; trwalo: <seconds>s; id: <pid>-<n>
blad czekania: <what>; gdzie: <where>; trwalo: <seconds>s; przyczyna: <cause>; id: <pid>-<n>

• czekam is written before the wait begins. koniec czekania follows when what was waited for arrived; blad czekania when the wait failed, with the operation's own error and every cause under it.
• rodzaj is one of object-api (a fleet-store read, write, listing or stat), blokada (a lock another process holds), host (a channel to a host), proces (a child process), baza (a database), siec (another network service), terminal (a key or a line typed on the terminal the program runs on), zgoda (the operator's answer to a grant a program asked for) or sesja (another agent session's work). The last three are waits nothing answers by itself; Oko's wait watch acts on them by their kind (oko sessions waits).
• id is the waiting process and the wait's number in it. Waits interleave, so the id pairs a start with its end. A czekam line with no end under its id, from a process still alive, is a wait still open: it has lasted from od until now.
• Fields are separated by ; and each field after the first is name: value. A value never holds ; or a line break; they are replaced by , and a space.
• An object API request reads as its method, path and decoded query, and where it waits as object API and the store's origin, for example czekam: GET /api/object uri=stado://probierz/registry.json versioned=true; gdzie: object API <store origin>; rodzaj: object-api; ….

Nothing limits how long a wait may take and nothing repeats it: the lines only make it visible. Stdout, and with it every --json envelope, never carries them. Oko's oko sessions agents reads the same lines from an agent session's tool output and terminal, and shows each open wait with its age.

A blad czekania line is the end of that request, not a pause before the next one. A dependency that refuses the connection — nothing listens at the address, as with an object API that is not running — has answered: an object or release read ends the command at once, non-zero, with the operation, the address and the refusal as its cause (the object API refused the object GET after 0 bytes: …), and sends no second request into the same refusal. A ranged read whose range is cut short or whose connection drops ends the same way, naming how many bytes arrived.

Every open wait is also a file in a register on its machine, $STADO_WAITS_DIR or ~/.stado/waits/<pid>-<n>, holding its czekam line; the file is written before the line and removed when the wait ends or fails, so a wait is visible however the waiting program's stderr was filtered. stado waits reads the register:

stado waits
stado waits --json

register <directory>: <n> open, <m> ended
  open  czekam: …; id: <pid>-<n>
  ended czekam: …; id: <pid>-<n> (process <pid> no longer runs; removed from the register)

--json prints {"register": <directory>, "open": [...], "ended": [...]}, each wait as {"id", "pid", "line"}. A process killed while it waited never removes its file; the reading that finds such a wait reports it under ended, with the line it held when its process died, and removes its file, so the next reading holds only waits a running process stands in. The refusals, each with the exit status of its class:

the register of open waits cannot be read: neither STADO_WAITS_DIR nor HOME is set, so the register of open waits has no directory
the register of open waits cannot be read: <file> is not named <pid>-<n>, so it is no wait of this register; remove it
the register of open waits cannot be read: reading <file>: <cause>
the register of open waits cannot be read: listing <directory>: <cause>
the register of open waits cannot be read: process <pid> no longer runs, so its wait <id> ended, but removing <file>: <cause>

A wait a program does not perform in Rust itself — a grant dialog in a harness, a terminal read a supervisor sees standing, another session an executor waits on — is registered for the waiting process by whoever knows of it, with the same line:

stado waits open --kind <kind> --what <what> --where <where> --pid <pid> [--awaits <kind>:<id>] [--json]
stado waits close <id> [--json]

open writes <pid>-<n> into the register, n being the moment of registration in nanoseconds so it never meets the numbers that process counts its own waits by, and prints registered <id>: process <pid> waits on <kind> for <what> at <where> or, with --json, {"id", "pid", "kind", "what", "where"}. The wait is listed like every other and ends when its process does, or with close, which prints closed <id> or, for a wait already gone, <id> was no longer in the register: it had ended or its process was gone ({"id", "removed"} with --json). For example, a harness about to show a grant dialog for a build runs stado waits open --kind zgoda --what build:stado --where "grant dialog" --pid <its pid>, and closes it when the dialog is answered. Refused, with exit status 2:

stado waits open --kind takes one of the register's kinds, siec, object-api, host, blokada, proces, baza, terminal, zgoda, sesja; got <word>
stado waits open --what names what the wait is for and where; it was empty

and with the exit status of its class, the register of open waits cannot be written: <id> is not a wait id: an id is <pid>-<n>, as stado waits lists it, … creating <directory>: <cause>, … writing <file>: <cause> or … removing <file>: <cause>. The real journey cargo test --manifest-path stado-rs/Cargo.toml --test wait-registered registers a zgoda wait for its own process and a terminal wait for a child it kills, and checks the listing, the close and every refusal above.

A wait on a lock file whose file names its holder as "pid": <n> — every product installation's surface lock — is also an edge of the machine's wait-for graph: beside the register, <register>.graph/<id> (~/.stado/waits.graph/<id>) holds the wait's resource, here lock:<path>, until the wait ends. A lock is one kind of resource the graph knows; a wait can await any of them, each written <kind>:<id>:

kind	what it is	who holds it
lock	a lock file	the "pid" its own file names, and any recorded holding
host	a fleet host a process keeps a channel to or waits to reach	a recorded holding
service	a service by name	the process serving it, as recorded
revision	a product revision, <product>@<revision>	the operation delivering it, as recorded
unit	a service-manager unit	the process managing it, as recorded
port	a listening port, <address>:<port>	the process bound to it, as recorded

What a process holds beside a lock file's own record lives beside the register too, <register>.holds/<pid>-<n> (~/.stado/waits.holds/…), one resource per file. A Rust program records its own holding with stado_wait::hold(&resource), released when the answer is dropped; any other program records it for a process that holds something:

stado waits hold --resource <kind>:<id> --pid <pid> [--json]
stado waits release <id> [--json]
stado waits holds [--json]

hold prints recorded <id>: process <pid> holds <kind> <id> ({"id", "pid", "resource"}); release prints released <id>, or <id> was no longer recorded: it had been released or its process was gone ({"id", "removed"}); holds lists every holding of a running process ({"holds": [{"id", "pid", "resource"}]}) and removes those of processes that ended. A wait registered for another process awaits a resource with stado waits open … --awaits <kind>:<id>, which adds "awaits" to its answer and , awaiting <kind> <id> to its line.

Before a wait on a resource starts — stado_wait::begin_on, blocking_on, until_on, blocking_on_lock, or stado waits open --awaits — Stado follows the graph from the resource's holders: each holder, every resource that holder waits on, those resources' holders, and so on. When a chain reaches the waiting process itself, or a process whose completion needs it (one of its ancestors, which waits for it to end), the wait could only stand until someone killed a process by hand, so it never starts. Its line says why:

czekanie odmowione: <what>; gdzie: <kind> <id>; przyczyna: a wait cycle: process <pid> (<command>) waits on <kind> <id>, held by process <holder>; process <holder> (<command>) waits on <kind> <id>, held by process <next>; …; process <last holder> needs process <pid> to end, so none of these waits can end

and the command fails: a lock wait with waiting for <lock> was refused: a wait cycle: …, stado waits open --awaits with the refusal class's status and the wait was not registered: it would close a wait cycle: …. An installation started by a step of an installation holding the same surface's lock is refused at once this way; so is an installation whose lock's holder waits on a lock its own parent holds, and an installation whose converge waits on a service whose process waits for a product revision that installation holds. A resource no running process holds ends every chain through it, so a wait whose chains end there starts as before. A graph that cannot be read does not stop a program's own wait; graf czekan nieodczytany: <kind> <id>; przyczyna: <cause> stands beside its line, and graf czekan niezapisany: id: <id>; przyczyna: <cause> when its edge or a holding's release cannot be written. stado waits open --awaits asked for the check, so a graph it cannot read is its refusal: the register of open waits cannot be written: the wait-for graph cannot be read: <cause>. Each refusal of a resource names what is wrong, with exit status 2:

stado waits hold --resource: "<text>" is no resource: a resource is <kind>:<id>, kind one of lock, host, service, revision, unit, port
stado waits open --awaits: <word> is no resource kind; a resource is <kind>:<id>, kind one of lock, host, service, revision, unit, port
stado waits hold --resource: a <kind> resource needs an id that is not empty and holds no control character; got "<id>"

and release of something that is not an id fails with <id> is not a holding id: an id is <pid>-<n>, as stado waits holds lists it.

A cycle can still form after its waits began, when a resource passes to a new holder. stado waits cycles names every cycle standing among the waits of running processes, once each:

stado waits cycles [--json]

With none it prints no wait cycle stands among this machine's waits and exits 0. With any it fails with the refusal class's status and <n> wait cycle(s) stand on this machine; end one process in each: <cycle> | <cycle>; --json prints {"cycles": [{"links": [{"pid", "command", "awaits", "held_by"}], "closed_by", "said"}]} on stdout either way, each awaits as <kind>:<id>. A graph it cannot read fails with the wait-for graph of this machine's waits cannot be read: <cause>. Stado Desktop's Waits section reads the same commands with Read wait cycles, showing each cycle's text, and Read held resources, showing each holding as process <pid> holds <kind>:<id>. The real journey cargo test --manifest-path stado-rs/Cargo.toml --test wait-cycle-install-lock holds a product's real surface lock and runs stado product install … --wait three ways: under a lock its own parent holds (refused, naming the cycle), under a lock a process that waits on nothing holds (the wait stands, no cycle listed), and through another install's wait (refused, then a standing cycle that stado waits cycles names). The real journey cargo test --manifest-path stado-rs/Cargo.toml --test wait-cycle-resource runs stado waits hold | open --awaits | cycles | release | holds for itself and a child: the child holds service:example-api and waits for revision:example@next; a wait of the test on the service while it holds the revision is refused naming both links, one while nothing holds the revision is registered, a cycle formed by holding the revision afterwards is listed and ends with its release, and each malformed resource and id is refused as above.

The lines come from one library, the stado-wait crate in Stado's repository (stado-rs/wait), which Stado re-exports as stado::wait and which any other Rust program of the fleet depends on by revision instead of copying it. Its calls take what is waited for, where, and the wait itself: until for a future, blocking for blocking work, begin with done, failed or settle for a wait that spans a loop, send, request and request_blocking for an HTTP request (method and path only, never the query), output, status and their _async forms for a child process, and sdk for an AWS SDK call. A database is waited on by name and credential item, and a child whose command line carries a credential (psql with its URL) by the statement it runs. Tama's block-silent-waits refuses a write that adds a Rust request, child-process wait, file lock or TCP connection outside these calls. The HTTP and AWS SDK waits are the crate's http feature and the cancellable and tokio child waits its async feature, both on by default; a program that waits only on blocking work depends on it with default-features = false and takes neither reqwest nor tokio, keeping blocking, begin, output, status, child_output, the register of open waits and the wait-for graph.

A request that would stand behind the vault is refused with the line

The object API compares every bearer against Skarbiec: a namespace bearer's version on each object request, the bearer itself when the version moved, the release publisher's, the host-health role's and the service and machine clients' on each request of theirs. Each of those reads is a wait the server writes as above, under a key (the namespace, release:<item>, host-health, service:<item>, machine:<item>). A request is never queued behind another's read:

• A namespace bearer read holds its namespace while it runs. A request that finds its key held is answered 503 at once, with the body {"error":"object authorization unavailable","cause":"czekam: …"} — the czekam line of the read it would have stood behind, so the caller reads what the server waits on, where and since when.
• Once a read under a key failed, every later request under it is answered 503 with that failure's blad czekania line as cause (the vault's own answer in przyczyna, for example HTTP 403 or HTTP 503 with gpg's waiting for lock (held by <pid>)), except the first, which reads again while holding the key; its success clears the failure.
• A request gated on a closed authorization boundary answers the same body with the boundary's label, its verifier's last sentence and when that verdict was reached as cause. The error word names the boundary (object, release, service, registry, host-health); the machine routes carry the same cause in their AUTH_UNAVAILABLE message.

GET /api/state.json carries the same state under vault: consultations.<key>.open is the czekam line of the read holding the key and consultations.<key>.failed the blad czekania line of its last failed read, each null when there is none; keyring_lock_repair is the declared skarbiec crypto repair the vault host's own object API last ran because a vault answer named the process holding the keyring's lock — its holder, started and outcome (null while it runs) — or null (Declared repair). stado doctor's gateway-auth row names the same holder and the repair when the vault answers its verifiers that way. Nothing here times a wait out: the server's own read still runs to the vault's answer, and only the requests that would have queued behind it are refused.