A job from the claim to its proof of life

Queued → running: the claim

Job state is a set of storage prefixes, not a mutable status field. The queued-to-running transition is a create-if-absent write of running/<id>.json, so it has exactly one winner — two agents scanning the same queue cannot both start your job. Every other prefix transition writes the new record before deleting the old one; readers tolerate the retry window and resolve terminal state first.

Who wins the claim is a scheduling decision. The scheduler orders the queue by priority and creation time and admits only targets whose capabilities, policy, deadline, resource envelope, and provider fence match. Every agent broadcasts its live CPU, RAM, disk, running-job, and per-device accelerator capacity to capacity/<consumer-id>.json; the cloud scheduler reads these to decide whether to yield a queued job to a free local consumer instead of dispatching a paid VM, marking jobs with the highest $-saved-per-GB-of-local-VRAM for local pickup first. So a job you expected on a cloud GPU may legitimately run on an idle local machine — see providers for the provider model. --pinned-host opts out: only the named consumer may claim the job.

Workers have no configured job-slot count. On each poll a worker subtracts the resources used by running jobs and by the live reservations placed workloads hold on it, then admits queued jobs until the live CPU, RAM, VRAM, disk, or policy budget is exhausted. A job without explicit CPU or RAM sizing reserves one CPU core and 1 GiB of RAM; --exclusive starts only when the worker is idle and prevents further admission until that job finishes.

One tick now: stado scheduler tick

stado scheduler tick [--provider <PROVIDER>]... [--billing] [--json]

The coordinator runs its tick on its own cadence. stado scheduler tick runs that same tick once, in this process, and answers when it ends: queued cancellations are settled, due schedules fired, expired leases reaped, jobs assigned, and for each provider arm running jobs checked, dead agents reaped and queued jobs dispatched; terminal runs are reaped last. --provider runs only the named arms (repeat it for several); without it every arm of providers runs. --billing also runs the billing collector, as the cloud coordinator does. The answer is scheduled (machines started), providers (the arms that ran) and billing; each pass logs on stderr as [tick] …, [scheduler] … and [azure] …, the lines a coordinator writes.

It refuses before the tick starts:

• a --provider the capability catalog has no compute provider for, with exit 2: --provider "<name>" is not a compute provider of the capability catalog; stado capabilities compute lists them;
• a provider that is fenced or not enabled, with PROVIDER_DISABLED or PROVIDER_NOT_ENABLED as live instances describes;
• an enabled provider with no tick arm here (local, or one whose client could not be built; the [tick] line above names why): provider <name> has no tick arm on this deployment: …;
• a store or workload grant that cannot be opened, with the store's or the vault's own error;
• another tick running over the same store, refused as refused: another tick holds this store: pid <pid> on <host> since <time> (state/coordinator/tick-lock.json); it ends when that tick does.

One tick at a time runs over a store. Every tick, the coordinator's and this verb's, first creates state/coordinator/tick-lock.json naming its process and deletes it when it ends, whether it passed or failed. A record whose process on this host is gone (a tick that was killed) is taken over; one held on another host cannot be observed from here, so the refusal says to remove it there or with stado storage rm state/coordinator/tick-lock.json once that tick no longer runs. A resident coordinator whose tick finds the store held logs tick skipped: <that sentence> and ticks at its next interval.

A pass that fails ends the command non-zero with that pass's error, as it ends the coordinator's tick: for Azure, no Azure location placed <name>: … or a NIC that could not be removed, as where a VM is placed says.

The real CLI journey is node tests/azure/placement.mjs (needs STADO_BIN, SKARBIEC_BIN, OKO_BIN). Each case is an isolated deployment under .build/azure-placement-* with its own HOME, local store, a real Skarbiec holding the billing and workload items, Oko reading a rulings register of the case that names an Azure credit floor, and a loopback listener declared as the Azure cloud. One queued T4 job, two locations; the first answers the NIC create QuotaExceeded. The second answers the VM create SkuNotAvailable (the NIC is deleted, 404, and nothing is placed), HTTP 500 with no code (refused naming no Azure location placed and that refusal), SkuNotAvailable with the NIC delete answering 409 (refused naming the VM, the location and both refusals), or an operation that ends Failed with SkuNotAvailable (read as that code). While that last tick waits on its operation, a second stado scheduler tick on the same store is refused naming the first tick's pid and the lock record, and once the first ends no lock is left. Every command, its pid, status, output and every request are kept in its report.json.

Running: heartbeats and proof of life

A running job's evidence lives under status/<id>/. The job's per-job heartbeat at status/<job_id>/heartbeat is written by the running job itself, independent of the agent's capacity-broadcast loop. That independence is the point: a long training subprocess can starve the broadcast loop until it looks stale, and the reaper's heartbeat guard uses the per-job heartbeat as the second signal — if any job on a VM heartbeats fresh, the reap is deferred.

Even the heartbeat can starve: a multi-GB checkpoint upload saturates outbound network and delays the small heartbeat write while the job is demonstrably alive. The guard therefore also accepts a fresh checkpoint write as proof of life — the newest blob under the job's checkpoint prefix stays fresh throughout an upload. A genuinely dead job writes neither, and is requeued once both signals age out. A coordinator-side read or listing failure is never treated as proof of death; the guard fails safe and defers.

What this means for you: a busy job is protected without any action on your part, and a job whose worker really died is requeued automatically rather than stranded.

A maintenance job that ends by stopping the agent running it is submitted with --terminates-agent: the agent vanishing is its success, recorded completed rather than requeued onto the next agent generation. A resident service, a job that runs until it is stopped, is submitted with --resident (also on stado schedule create): a worker handing off to an installed release puts it back to the queue without spending a yield, and the next worker runs it again; any other job holds that handoff until it ends. Only declarations count: Stado never reads the command text. stado job rerun carries both to the new job.

Watch a job while it runs:

stado job watch <job-id> --follow

--follow holds on the store's change watch, here or through the object API (self-hosted storage), and refuses before it waits when the store cannot hold.