Moving the fleet vault

The profile that carries skarbiec, the fleet vault, is moved by stado credentials vault move --apply, and its move is fenced differently. The registry authority authorizes every write to its own store with a bearer whose version it asks the vault for on each request, and refuses when the vault does not answer. A move that stopped the vault's unit on the authority's host could never commit: its last step, the registry write, asked the vault the move had stopped, and the move rolled back after every state was copied.

So the source's vault is not stopped. The step hold the vault's writes runs skarbiec fence arm --transaction <T> on the source, against the vault file the source declares. Skarbiec then refuses every write, naming the transaction, and keeps answering every read. The states read after it are the last ones. Routes, the destination's units and its probes follow as for any profile.

Then, still before the commit, authorize <source>'s resolver key on <destination> mints the source resolver's SSH key on the source when it has none ($HOME/.stado/resolver-ssh-key; the private half never leaves the source) and appends its public half to the destination account's authorized_keys, as stado route key <source> --to <destination> does. A source that is the registry authority never opened a session to another host, so it held no resolver key, and without this step its resolver and the reach below fail with SSH authentication failed for <user>: no identity is available. A refusal here rolls the move back.

Then read the destination's vault from <source> runs stado host local vault-reach --host <destination> --endpoint <url> on the source with the source's own Stado. <url> is the destination's skarbiec endpoint in the service directory. The step reads every object API bearer's revision (/v1/items/revision, field token of each object_api.namespaces.*.item) as consumer stado, with the source's own grant, over a native SSH channel to the destination. That is the same channel the source's resolver adapter opens once the commit sends it there. The session must open within the resolver's refresh interval, and each read must answer within that interval less the time the session took to open (read_bound_seconds in the receipt): the object API makes one read per request, the resolver gives each request up alone, and the request that opens the session pays both. The reads are not bounded together, since one bound for all of them refused a channel whose every read answered in a fraction of a second only because many namespaces are declared. The destination's probe asks only GET /health on its own loopback, which the vault answers without reading anything, so it cannot stand for this read. A source that cannot read the destination is refused before the commit, and the move rolls back: <source> did not read the vault at <url> on <destination> over the channel its resolver will use once the move commits: <cause>; reads: [...]. Each read in the receipt names its namespace, item, seconds and answer. The receipt also names paths (each SSH path to the destination, in the order they are tried), paths_seconds (choosing them, which reads the tailnet's status, as the resolver does), session_seconds (the SSH session opening) and attempts (each path tried alone, in the order the resolver's tunnel tries them, with its seconds, whether it opened and its error). When a bound expires, the cause names the phase it expired in:

Cause begins	Where the time went
`choosing the SSH paths to <destination> (the tailnet status read) had not finished`	`tailscale status --json` on the source did not answer in time; the resolver pays the same read
`the SSH paths to <destination> were chosen after Ns; <path> failed after Ns: <error>; … <path> was still connecting`	the earlier paths failed as named, and the last named path had not completed its SSH handshake in time
`the SSH session to <destination> opened after Ns, then K of M reads answered and the read of <namespace> (<item>) was still unanswered Ns later`	the channel opened and the destination's vault did not answer the named read within the per-read bound; the reads after it are not made

A tailnet path is left out before any attempt when the source's own Tailscale carries no traffic (this machine's own Tailscale node key expired at <T>, or this machine's Tailscale is <state>, not Running) or the destination's node key expired; each one left out is named on stderr.

A source whose Stado does not know the verb is refused with answered no receipt … install this release there first.

Then commit the placement, which the authority can authorize because the vault still answers. Only after the commit:

the source's resolver follows the commit runs stado resolver follow on the source, so every consumer there that reads the vault through its resolver adapter is sent to the destination.
retire <unit> retires the source's units.
lift the retired copy's fence, once every unit is retired: with nothing serving the source's copy, a fence left there would refuse every write to the vault if it ever moved back to that host.

Unlike every other service, skarbiec is not refused by the resolver while its own move's transaction is open: each resolver adapter keeps sending its consumers to the active host, the fenced source, until the commit retargets it. The registry authority and the object API read the vault through such an adapter when their host declares secrets.skarbiec.url stado://service/skarbiec?consumer=stado; refused there, the move's own claim cut them off from the vault every store write is checked against, the claim's receipt never came back, and the transaction stayed open.

The move is committed whatever these steps answer. Each one that fails is listed under unretired_source in the receipt (on stderr in text), with what still stands on the source: its units keep answering reads with every write held, so a consumer still sent there reads what was moved and writes nothing that would be lost. A move that fails before the commit lifts the fence (skarbiec fence lift --transaction <T>) instead of restarting units that never stopped. Read entries the source's audit journal records between the last copy and its retirement stay in the source's journal.

The preflight, and so --check, refuses such a move before anything changes:

when the source's Skarbiec has no write fence: <source>: its Skarbiec cannot hold writes for a move (<error>); … roll out a Skarbiec release that has skarbiec fence on <source> first;
when the source's skarbiec unit file does not name that Skarbiec: <source>: unit <label> (<path>) does not run <skarbiec>, the Skarbiec whose write fence the move arms, …;
when the source's own stado registry doctor (run there over its channel, the only machine that can read which file a process executes) reports stale-unit-image or unread-unit-image for that unit: <source>: its own registry doctor reports <finding> for the vault unit: <detail>; a process not executing its installed Skarbiec ignores the write fence, so put <label> on its installed file (stado service image refresh <label> on <source>) before moving;
when the source's own Stado reads the vault at the unit the move retires (its resolved skarbiec_url names the host and port of the source's skarbiec probe): <source>: its Stado reads the fleet vault at <url>, the skarbiec unit this move retires, …. The way out is to point secrets.skarbiec.url at the resolver adapter and reload the serving process in one step, run on the source (stado host config set <source> secrets.skarbiec.url stado://service/skarbiec?consumer=stado --reload-service stado). Restarting a system daemon needs the host account's password, which the source's own owner vault holds. Run from another host, the reload is refused when the stado grant cannot read that item: the value is written, but the running process keeps the old address until the reload succeeds. The other way out is to move the registry authority and the stado-object-api role off the source first.

Flag-by-flag detail is in cli.

Source: this website