Lending to another Stado fleet

Another Stado deployment can use this fleet's idle hosts. While our own queue has nothing for a host, the host runs a Stado worker against the other fleet's queue; the moment our work appears, that worker stops taking new jobs, lets the other fleet's jobs it is running finish, and exits. The commands are the stado market group with --provider stado, the same operations Vast.ai has, and the other fleet is declared once with stado market peer.

Nothing here moves money. Every job our host ran for the other fleet is in that fleet's own job records, with its start, its completion and the host that ran it. The price a host was offered at turns those seconds into an amount, and both operators read it from the same records.

What the other fleet hands over

The other fleet's operator gives ours three things:

what	where it goes
the origin of the object API that serves its queue	--origin
the object namespace its queue lives in	--namespace
a bearer for that namespace	a Skarbiec item, by role

The bearer is stored in our vault under a role our operator chooses, in the item's token field. The item document is read on stdin only, and its kind must be the --type:

printf '%s' '{"schema":"skarbiec.item.v2","kind":"token","fields":{"token":"<bearer>"},"context":{}}' | stado credentials item put --host <vault-host> --role <role> --type token

Declaring the other fleet

stado market peer add <name> --origin https://objects.example.com \
  --namespace <namespace> --credential-role <role> [--ca-file <pem>]
stado market peer edit <name> [--origin URL] [--namespace NS] \
  [--credential-role ROLE] [--ca-file PEM | --clear-ca-file]
stado market peer list
stado market peer show <name>
stado market peer remove <name>

The declaration lives in this fleet's store under state/market/peers/, so every host of the fleet lends under the same one. It never holds the bearer. --ca-file names the PEM authority of an origin signed by a private authority, such as an object API published on a tailnet; without it the system roots are trusted. The origin must be HTTPS, or HTTP on loopback or a tailnet IP address, the same rule every Stado store URL meets; that is judged when the store is opened.

remove retires the declaration together with every host's offer to that fleet, and is refused while an auto-list loop on this host holds it.

Lending a host

stado market readiness --provider stado --peer <name>
stado market follow --provider stado --peer <name> \
  --idle-window-s <seconds> --poll-interval-s <seconds> \
  --price-gpu <usd> --max-duration-s <seconds>

readiness asks three authorities in order: the Skarbiec channel this host holds, the vault that would hold the item playing the role, and the other fleet's store, with the bearer, for one name of its queue. It exits non-zero unless that store answers. --json prints a stado.peer-readiness.v1 document.

verdict	what it means
ready	the other fleet's store answered with our bearer
peer_refused	the bearer resolved and that store did not answer with it
item_absent	no item in the vault plays the role
not_authorized	the item exists and this consumer may not read token
no_channel	this host holds no Skarbiec bearer at all
unknown	the bearer did not resolve and the vault could not be read
The document carries skarbiec_error and its class skarbiec_failure
(infra_down when the vault did not answer, refused, not_found, config).
Without --json, a verdict that cannot lend exits with its own class:
refused for peer_refused, the channel's class for unknown, and config
for what this host must provision. Stado Desktop's Peer fleets panel shows the
Skarbiec line and its failure class under the verdict.

follow reads our queue every --poll-interval-s, as the Vast bridge does: idle means nothing is queued and no running job names this host. Once the host has been idle for --idle-window-s, the offer is listed and the other fleet's queue holds work, it starts a lending worker:

• stado serve --worker --standalone --kind stado-peer --idle-shutdown, this same Stado program, as its own process;
• with a home of its own under ~/.stado/market/stado/<name>/homes/ and an environment that keeps nothing of ours but PATH;
• configured with only the other fleet's store, so it reads none of our registry, vault or queue.

Jobs it claims record worker_allocation.kind stado-peer and our host's name, which is how both fleets tell lent work apart.

The worker is asked to drain when our work runs on this host, when work only queued for our fleet is worth more per GPU-hour than the peer pays (the job's stated income, else the autonomy policy's budgets.internal_usd_per_gpu_hour; a tie keeps lending, and work whose value cannot be read drains it, the log saying what to declare), when an operator unlists the host, when an inference reservation names no board (or cannot be read), or when the session has run --max-duration-s seconds (0 leaves it open-ended). A reservation that names its board does not drain it: the worker is started with STADO_INFERENCE_RESERVATION pointing at this host's reservation, so its own admission gate withholds that board and claims the other fleet's work only on the host's other boards, and the loop prints which board the reservation holds. Draining means it claims nothing more and exits once the jobs it is running have finished, so a job of the other fleet is never cut off halfway. A worker that runs out of eligible work ends on its own. When a session ends, the worker's log is kept under ~/.stado/market/stado/<name>/logs/ and its home is removed.

--dry-run starts no worker and writes nothing. It prints what the loop would decide; --once evaluates one poll and exits.

The loop holds the role lock market-stado--<name> for as long as it runs. A second loop for the same fleet on the same host is refused, naming the holder. A loop that starts and finds a session an earlier loop left running asks that session's worker to drain.

List, unlist, status, monitor

stado market list --provider stado --peer <name> --price-gpu <usd>
stado market unlist --provider stado --peer <name>
stado market status --provider stado [--peer <name>]
stado market monitor --provider stado --peer <name> [--bucket <namespace>]

list opens this host to the other fleet again, at the price given. The loop reads that switch on its next poll. list is refused when no loop on this host holds that fleet, because the offer would lend nothing. unlist closes the host; a running session is drained at the loop's next poll. Each host's offer is kept in this fleet's store under state/market/offers/ and records the price, who changed it last, what the loop last observed, and the current or last session.

status --peer reads the other fleet's completed jobs run by each of our hosts that holds an offer. For each host it reports the jobs, the measured seconds, the jobs that cannot be measured with the reason, and the amount at that host's price. status without --peer reads our own completed jobs that peer workers ran here, which is the buying side of the same records.

monitor prints one snapshot: our queue on this host, the other fleet's queue (whether it is empty, and how many of its jobs run), and this host's offer.

The screen

Stado Desktop's Earning screen has an Other Stado fleets panel. It declares, changes and retires a fleet, shows each fleet's readiness verdict and the commands that fix it, shows each host's offer and what it lent, and has the controls for offering a host and withdrawing it. Each control asks for confirmation and shows the exact command first. The panel runs the same commands and shows their sentences as written.

Refusals

sentence	meaning
stado market <verb> --provider stado needs --peer NAME, the declared peer fleet; …	--peer is missing
role market-stado--<name> is not started from the installed Stado: …	the loop runs from a program other than the installed ~/.stado/bin/stado
--idle-window-s and --max-duration-s are seconds and cannot be negative	a negative window or duration
--price-gpu … is not a price; give US dollars per GPU-hour	a negative or non-finite price
stado market peer edit <name> names nothing to change: …	edit with no field
--peer names a Stado fleet and is taken only with --provider stado	--peer was given to another provider
--price-disk and --price-min-bid are Vast.ai's prices; a peer fleet is offered at --price-gpu alone	a Vast-only price was given
no peer fleet named <name> is declared (…); …	the name is not declared; the sentence lists the declared names
peer <name> is already declared; …	add of a name that exists
peer name … must be lowercase letters, digits and inner hyphens	the name cannot be a key, a lock or a directory
no follow loop holds peer <name> on this host, so an offer would lend nothing; …	list without a loop
peer <name> is held by the follow loop pid … on this host since …; …	remove while a loop holds the fleet
no bearer for peer <name> on this host: …	the role's token did not resolve; readiness says which step is missing
role market-stado--<name> already runs on this host: …	a second loop for the same fleet
lending needs --poll-interval-s: …	a loop that would start a worker was given no period