Job

What is the durable object behind stado submit, and how does the fleet agree on who runs it? A job is one queue object whose prefix is its state, claimed by exactly one worker through a create-if-absent write.

What it is

A job is a JSON object in the canonical backend. Its lifecycle is spelled by the prefix it lives under:

Prefix	State
queue/<id>.json	Pending; the submission ledger.
running/<id>.json	In flight, with owner, instance reference, and start time.
completed/<id>.json	Successful terminal state.
failed/<id>.json	Unsuccessful terminal state, with bounded error classification.
cancelled/<id>.json	Durable operator cancellation record.
status/<id>/...	Heartbeats, redacted output, result manifest, artifact evidence.

The queued-to-running claim is create-if-absent and therefore has one winner. Other transitions write the new record before deleting the old one; readers tolerate the resulting retry window and resolve terminal state first, and writers are idempotent and fenced by the expected generation.

Who declares it

A submitter, through stado submit (or a batch file) into the configured canonical queue backend, via the authenticated Stado machine/object boundary. The job spec — provider constraint, spot policy, cost cap, resource envelope — travels with the object; stado job rerun resubmits a job's exact spec under a new id.

Who observes it

The Rust scheduler reads a bounded provider-neutral queue window, orders by priority and creation time, and admits only targets whose declared capabilities, policy, resource envelope, and provider fence match. Agents claim through the atomic storage primitive before writing runtime state or starting a process.

A running job is alive while the promise its worker wrote holds. The claim and every renewal write lease_expires_at into the running document: the time by which the worker says it will renew again — its --poll-seconds, plus the longest it has measured a renewal landing late in this process, plus how long its last renewal write took. The renewal is a compare-and-swap on that same document, so a reaper that read the job a moment earlier loses its move.

Once the promise has passed, two writes still prove the job alive:

• The per-job pulse at status/<job_id>/heartbeat, written right after each renewal. A pulse newer than the passed promise means the renewal is failing, not the worker.
• A blob under the job's checkpoint prefix newer than the passed promise — a multi-gigabyte checkpoint upload starves the small renewal while the job is demonstrably working.

Only a job whose promise passed with neither write after it is reaped: requeued once with worker lease expired, failed on the second expiry. A job claimed by a Stado from before promises carries none and is kept (counted as unpromised in the reaper's summary): nothing says when it should have spoken. The same verdict protects a VM: a cloud VM is not deleted while any job on it is alive by it.

A job requeued while the worker that claimed it still runs its process is not started a second time beside it. On the slot's next tick its lease renewal finds no running document; the slot finds the job in queue/, moves the record back to running/ as the attempt already under way — the same started_at, a fresh lease its heartbeat then renews — and the agent log says <job>: already running here as process group <pid> for <n> s: its lease lapsed while it ran and it was put back in the queue; the running process took the job back and no second process was started. This happens whatever the host's admission gates decide, and the claim scan never starts a queued job one of its own slots runs. If another worker or a cancellation took the queued record first, the line says so instead and nothing is started. The real test is tests/scheduler/readopt.rs (--test scheduler readopt::).

A worker that starts on a host runs nothing yet, so every running job whose record names that host's worker (instance_ref local@<host>) was left by the worker it replaced. Before its first tick it asks, for each such job, which processes still have the job's work directory open (lsof). A job no process runs is settled as a lapsed lease is, at once instead of when the dead worker's promise passes: requeued with its worker was replaced while it ran (failed on the same loss again, completed when its release output already verifies), logged as init: <job>: requeued (its worker was replaced while it ran; started <time> by the worker this one replaced on local@<host>; restart <n>). A job a process still runs keeps its lease: init: <job>: left by the worker this one replaced, but still run here by pid <pid>; it keeps its lease. The real test is tests/scheduler/restart.rs (--test scheduler restart::).

Operators observe with stado status and stado job watch.

stado machine status <job-id> answers the same question for a program: besides state, each job carries terminal, which is true once the job has stopped moving — Stado decides it from its own terminal set, so a reader no longer keeps a copy of which state words mean "finished". Oko's routines read that field before they go looking for a routine's artifacts.

What its shell is given

A local agent starts the command with /bin/sh -c, an environment cleared to the execution-runtime variables (paths, locale, proxies, CA bundles, accelerator variables), and these:

• STADO_BIN, the agent's own program, so a job that calls Stado back (to upload what it produced) runs that program instead of guessing where one is installed;
• WC_JOB_ID, WC_JOB_STARTED_AT, WC_ARTIFACT_INPUTS_JSON and WC_ARTIFACT_INPUTS_FILE, and CARGO_BUILD_JOBS set to the cores reserved;
• GIT_TERMINAL_PROMPT=false and CARGO_NET_GIT_FETCH_WITH_CLI=true, so git never waits for a terminal and Cargo fetches through git. No transfer is ended by a clock: a fetch that stops moving stays visible in the job's output and is cancelled by name.

Where it lives

Under the queue prefixes above, in whatever backend the selected STADO_CONFIG names — the same canonical prefixes regardless of backend. See object-store.

Commands

stado submit "python train.py"
stado status <id>
stado job watch <id>
stado job rerun <id>
stado results <id> <output-dir>
stado cancel <id>
stado queue pause

The end-to-end submission workflow is jobs; flags are in cli.

Not to be confused with

• A lease — ownership of one mutation by one process. The queued-to-running claim is create-if-absent; the claimed job is held to its worker's own promise, not to a window of the reaper's.
• A service — a unit a host must keep running indefinitely. A job terminates.
• A schedule — stado schedule submits jobs on a cron; each submission is an ordinary job object.