Jobs

You have a command to run on fleet compute: how does it get from your shell to a worker, what happens while it runs, and what do you do at each state? This page follows one job through its whole lifecycle. First submission is covered in quick-start; flag-by-flag detail lives in cli.

A detached agent session is one of these jobs. stado workload start jeden-session submits it for you, sized from the kind's declared reservation and pinned to the host its placement chose, so everything on this page — the claim, the heartbeats, the reaper, stado job watch, stado results and stado cancel — is how you follow and stop it. List the ones the fleet holds with stado workload sessions, and read workloads for the placement itself.

Submit

stado submit --run-id train-topic "python train.py --epochs 3"

Every submission carries --run-id, the caller's own stable name for the run: repeating the same run id and request answers the jobs already submitted instead of queueing them twice, so a retried submission is safe. The job enters queue/<id>.json in the configured canonical backend. Routing and sizing are declared at submit time: an optional provider constraint (--provider, with --pin-provider or the default --any-provider), spot policy, a hard --max-cost-per-hour cap, priority within the FIFO bucket, a hard RFC 3339 --deadline-at, and the resource envelope (--gpu-type, --vram-gb, --machine-type, --exclusive). Inputs are declared, not ambient: a git repo pinned to an exact commit (--repo plus --repo-ref), pinned artifacts (--input-artifact), and scoped workload secrets as --secret-env ENV_NAME=SKARBIEC_ITEM#FIELD.

Priority

--priority accepts the nonnegative range of the job model's signed integer, not a limit chosen by the index's decimal formatting. Higher priorities are claimed first, then older jobs. Reorder an existing queued job without resubmitting it:

stado job set-priority <job-id> <priority> --json

A negative value is refused as job priority <value> must be nonnegative; submission reports priority <value> must be nonnegative. A job no longer queued is refused with its current state and is not resurrected. Desktop Queue → Change a queued job's priority invokes the same native operation, shows its mutation confirmation and retains the same receipt or refusal.

The priority index encodes the full integer in descending order. Existing queue documents remain authoritative: the normal backfill rewrites their keys and removes superseded entries before marking the new coverage complete. An index entry that disagrees with its job's current key cannot admit it. Deliver the new queue readers and writers together; an older writer does not understand the new index order.

The real CLI journey is node tests/queue/priority.mjs. It needs STADO_BIN, STADO_TEST_SOURCE_REVISION, STADO_TEST_PRIORITY_NEGATIVE (a negative integer), and the host's PATH. It creates an isolated local deployment under .build/queue-priority, submits a job naming no priority and reads the default stado submit gave it, submits jobs beyond the old decimal range, promotes and demotes them, checks persisted order and negative-value refusals, then cancels them and checks terminal-state refusal. It retains the source and binary identities, command streams, statuses, the default priority and final state in report.json.

Nothing about the machine is assumed. A job that may run on a rented machine states its disk with --boot-disk-gb <GB>; a job that states none is never the reason a machine is rented (the dispatcher logs REFUSING to rent agent machines … none of its queued job(s) states a boot disk) and waits for a registered host. A rented machine gets the largest disk any job in its bucket stated. --yieldable needs --max-yields <N>, how often the worker may yield the job before it is protected; a job that states no budget is protected from the start. stado schedule create --boot-disk-gb and the machine interface's boot_disk_gb field state the same disk for scheduled and machine jobs. A non-positive value is refused by name.

Nor is the run time. Placement prices a job, holds it to its deadline and orders the local pack by how long it runs, and the only two sources of that are the submitter's --runtime-seconds-estimate <SECONDS> (also on stado schedule create, and the machine interface's runtime_seconds_estimate) and the median of completed runs of the same model on the same hardware. A job with neither is refused a priced placement (no run time for this job on <target>: no runtime_seconds_estimate was stated and no completed run was measured), left unassigned by the makespan matcher, and packed locally only after every job whose run time is known. A non-positive estimate is refused by name; a rerun keeps the original's. Priority changes ordering, not the runtime evidence: a high-priority job with neither source also remains unassigned. Any previous derived assignment is cleared; eligible workers may still claim it through the normal priority queue. The makespan diagnostic reports every skipped model/task group with its count, ordered by count. It does not hide groups outside a fixed display window.

The real regression is cargo test --manifest-path stado-rs/Cargo.toml --test scheduler -- sizing::runtime::, from the Stado checkout. Supply STADO_SOURCE_REVISION, STADO_TEST_WORKER_POLL_SECONDS, STADO_TEST_COORDINATOR_INTERVAL_SECONDS, STADO_TEST_POSITIVE_PRIORITY, and STADO_TEST_RUNTIME_SECONDS. It starts an isolated worker, retains its actual capacity publication, stops that worker before submitting jobs, then runs a real coordinator. A declared-runtime job must be assigned to that published consumer while both ordinary and priority jobs without runtime stay unassigned. If admission fails or the measured publication expires before assignment, the journey fails, not passes. Commands, process exits, logs and persisted before/ after state are retained in its .build/z/ report. No provider is simulated.

A batch is one submit over a file of commands — one command per line, with blank lines and # comments skipped. With --batch the positional command is ignored; every submitted job carries the same batch id, and stado status filters by job id or batch id substring, so a batch is inspected as one unit.

stado submit --run-id nightly-batch --batch commands.txt ""
stado status <batch-id-substring>

Submit profiles

A profile is a named, reusable submit spec. --profile NAME applies one from the bundled profiles directory (or $WC_PROFILES_DIR); explicit CLI flags override profile fields. A WC_PROFILES_DIR that names no directory is refused with WC_PROFILES_DIR names <path>, which is not a directory, and one that is not text with WC_PROFILES_DIR cannot be read: <error>, rather than applying the bundled profile of the same name.

stado profiles              # list available profiles
stado profiles <name>       # show one profile's JSON
stado submit --run-id <run id> --profile <name> "python train.py"

Queued → running: the claim

Job state is a set of storage prefixes, not a mutable status field. The queued-to-running transition is a create-if-absent write of running/<id>.json, so it has exactly one winner — two agents scanning the same queue cannot both start your job. Every other prefix transition writes the new record before deleting the old one; readers tolerate the retry window and resolve terminal state first.

Who wins the claim is a scheduling decision. The scheduler orders the queue by priority and creation time and admits only targets whose capabilities, policy, deadline, resource envelope, and provider fence match. Every agent broadcasts its live CPU, RAM, disk, running-job, and per-device accelerator capacity to capacity/<consumer-id>.json; the cloud scheduler reads these to decide whether to yield a queued job to a free local consumer instead of dispatching a paid VM, marking jobs with the highest $-saved-per-GB-of-local-VRAM for local pickup first. So a job you expected on a cloud GPU may legitimately run on an idle local machine — see providers for the provider model. --pinned-host opts out: only the named consumer may claim the job.

Workers have no configured job-slot count. On each poll a worker subtracts the resources used by running jobs and by the live reservations placed workloads hold on it, then admits queued jobs until the live CPU, RAM, VRAM, disk, or policy budget is exhausted. A job without explicit CPU or RAM sizing reserves one CPU core and 1 GiB of RAM; --exclusive starts only when the worker is idle and prevents further admission until that job finishes.

Running: heartbeats and proof of life

A running job's evidence lives under status/<id>/. The job's per-job heartbeat at status/<job_id>/heartbeat is written by the running job itself, independent of the agent's capacity-broadcast loop. That independence is the point: a long training subprocess can starve the broadcast loop until it looks stale, and the reaper's heartbeat guard uses the per-job heartbeat as the second signal — if any job on a VM heartbeats fresh, the reap is deferred.

Even the heartbeat can starve: a multi-GB checkpoint upload saturates outbound network and delays the small heartbeat write while the job is demonstrably alive. The guard therefore also accepts a fresh checkpoint write as proof of life — the newest blob under the job's checkpoint prefix stays fresh throughout an upload. A genuinely dead job writes neither, and is requeued once both signals age out. A coordinator-side read or listing failure is never treated as proof of death; the guard fails safe and defers.

What this means for you: a busy job is protected without any action on your part, and a job whose worker really died is requeued automatically rather than stranded.

A maintenance job that ends by stopping the agent running it (upgrade the agent, then restart it) is submitted with --terminates-agent. The agent vanishing is then that job's success: the reapers record it completed instead of requeuing it, which would run the stop again on the next agent generation. Only the declaration counts; Stado does not read the command text, so a command that stops the agent without the flag is requeued like any other orphan, and one that merely mentions kill is not taken for a maintenance job. stado job rerun carries the declaration to the new job.

Watch a job while it runs:

stado job watch <job-id> --follow

--follow polls the log until the job reaches a terminal prefix.

Terminal states and results

completed/ records success; failed/ records failure with bounded error classification. If the job declared --verify, that command must exit 0 after success — a non-zero exit reverses COMPLETED to FAILED, catching silent-success failure modes. Output always lands under canonical status/<id>/output/; --output-uri adds a second stado:// destination.

stado results <job-id> ./out

The download includes the result manifest recording artifact size and SHA-256. A failed job remains inspectable and may still publish logs and partial artifacts.

A job runs in its own process group. When the agent first sees the job's process exit, it ends whatever is still running in that group — a download or a server a step started in the background — once, and logs <job>: ended the processes its job left running in process group <pgid> when there was something to end. Nothing a job started outlives it. Processes that earlier agents left behind are found with stado service list --unowned and ended with stado service reap --host <host> --command <program> --apply, which never signals a process a declared unit holds.

Cancel

stado cancel <job-id>
stado cancel <job-id> --terminate

Cancel works on a queued or running job and writes a durable record to cancelled/. Without --terminate, a cancelled job's cloud VM keeps running — and billing; --terminate also deletes the instance the job holds.

Rerun

stado job rerun <job-id>

Rerun resubmits a job's exact spec under a new job id, from any lifecycle prefix — the original record is untouched.

Lifecycle table

State (prefix)	Who sets it	What an operator does
queue/	stado submit (or the coordinator, for a due schedule)	Wait, or check why it is not claimed: eligible worker running, queue not paused, capacity fits, deadline not expired.
running/	The winning agent, via the create-if-absent claim	stado job watch <id> --follow; nothing else — heartbeats and checkpoint writes protect the execution.
running/ (stale heartbeat and checkpoint)	The reaper requeues to queue/	Nothing; the job restarts from its last checkpoint.
completed/	The agent, after the job (and any --verify) succeeds	stado results <id> <dir>.
failed/	The agent, on job failure or a failed --verify	Read the log and error classification; fix; stado job rerun <id>.
cancelled/	The operator, via stado cancel	Confirm --terminate was used if the job held a VM.

Recurring jobs

A schedule submits a fresh job on a cron expression, with the same routing, sizing, and secret-reference contract as a direct submit. Schedules live in configured Stado storage and are evaluated every coordinator tick.

stado schedule create --cron "0 2 * * *" "python nightly.py"
stado schedule list
stado schedule pause <schedule-id>
stado schedule resume <schedule-id>
stado schedule run <schedule-id>

A schedule records who declared it, as a submission records its submitter: USER, else LOGNAME, else the account this process runs as in the system's user database. A variable that is not text is refused with USER cannot be read: <error>, and an account the database does not hold with neither USER nor LOGNAME is set and the user database holds no account for uid <uid>, instead of storing a schedule created by nobody. Every other record that names who acted (autonomy pause and resume, resources adopt, release quarantine clear, a service ensure audit and a space reclaim audit) reads the same account and refuses with the same reasons.

The default --overlap-policy skip does not fire while the prior instance is still queued or running, and the coordinator's log says what that instance is doing at every tick it skips: schedule <id>: skip fire: prior job <job> is still running, started <time> (<n> s ago), worker lease until <time>, on local@<host>, with put back after: worker lease expired (restarts <n>) when a reaper requeued it. rm deletes a schedule without affecting jobs it already submitted; run fires once immediately regardless of the next run time; resume recomputes the next run from now.

What it cost

Cost reporting is computed from observed wall-times, per job and per batch:

stado cost report                  # $ spent per target_kind and per model, from completed jobs
stado cost estimate commands.txt   # project total $ for a batch file using observed per-job cost

stado cost also carries allocation, forecast, anomalies, and savings; budgets and the wider cost model are covered in costs.