Recurring jobs

A schedule submits a fresh job on a cron expression, with the same routing, sizing, and secret-reference contract as a direct submit. Schedules live in configured Stado storage and are evaluated every coordinator tick.

stado schedule create --cron "0 2 * * *" "python nightly.py"
stado schedule list
stado schedule pause <schedule-id>
stado schedule resume <schedule-id>
stado schedule run <schedule-id>

A schedule records who declared it, as a submission records its submitter: USER, else LOGNAME, else the account this process runs as in the system's user database. A variable that is not text is refused with USER cannot be read: <error>, and an account the database does not hold with neither USER nor LOGNAME is set and the user database holds no account for uid <uid>, instead of storing a schedule created by nobody. Every other record that names who acted (autonomy pause and resume, resources adopt, release quarantine clear, a service ensure audit and a space reclaim audit) reads the same account and refuses with the same reasons.

The default --overlap-policy skip does not fire while the prior instance is still queued or running, and the coordinator's log says what that instance is doing at every tick it skips: schedule <id>: skip fire: prior job <job> is still running, started <time> (<n> s ago), worker lease until <time>, on local@<host>, with put back after: worker lease expired (restarts <n>) when a reaper requeued it. rm deletes a schedule without affecting jobs it already submitted; run fires once immediately regardless of the next run time; resume recomputes the next run from now.

Declarations held on this host

A host declares its schedules while the fleet store does not answer. When schedule create meets a store that answers a server error, a request the resolver holds unanswered, or a transport failure, it keeps the declaration on this host at ~/.stado/schedules/held/<schedule-id>.json and exits 0 with held schedule <id> on this host at <file>: the fleet store did not answer (<refusal>); the next schedule command that reaches it writes it there (--json answers {held, refusal, schedule}). Any other refusal of the store still ends the command. Every schedule command that reaches the store writes the held declarations into it first, as schedule create would, and says stado: held schedule <id> written into the fleet store; <file> removed on stderr (was already declared in the fleet store when the store holds the same declaration under a retained id, brought the stored schedule to its declaration: changed <fields> when it was declared with converge); one the store refuses (another schedule under its id) stays held with stado: held schedule <id> stays at <file>: the fleet store refused it: <refusal>. schedule list shows held declarations beside the store's, marked H with their file, time and refusal (--json adds a held object to each); when the store does not answer it lists what is held here and exits with the fleet store did not answer (<refusal>), so only the schedules held on this host are listed.

A held fleet declaration is not a host-local routine: it waits for the fleet store. Use the host store explicitly for work that must run without it.

Host-local routines

A host also keeps its own store, ~/.stado/host-store, with the same schedule and job records as the fleet store, for routines that keep that one host working. schedule create --host-store declares a routine there, never in the fleet store, and needs no answer from the fleet store:

stado schedule create --host-store --cron "0 2 * * *" "your-maintenance-command"
stado schedule list

schedule list lists the host store's routines beside the fleet store's schedules, marked R in the EN column (r when paused) and counted as <n> routine(s) of this host's own store (R), fired and run by its stado serve --worker; --json marks each with host_store: true, and an overdue one is judged against the host pass's last sweep. When the fleet store does not answer, the host's routines are still listed. A host store that cannot be opened (no Stado directory under HOME) is said after the listing and the command exits with that refusal. schedule show, edit, pause, resume, rm and run take a host routine's id too: they ask the host store first and act on the routine there, so they work while the fleet store does not answer, and an id the host store does not hold goes to the fleet store as before.

The Stado CLI installation declares one such routine on every machine it installs on: the submit queue's resident pass, "$STADO_BIN" change run every ten minutes (who runs the queue).

Its worker (stado serve --worker) retains its normal fleet configuration. It runs host routines itself on its poll, before and independently of the fleet store it binds: each poll fires every due schedule held there (overlap policy included) and records the sweep, records the end of every routine job it started, moves to failed every running host job no child of it runs whose lease has run out (the worker that ran it stopped before recording its end; its lease ran out at <time>), so a worker that stopped mid-job never leaves an overlap-skip routine waiting forever, deletes the cleaned sentinel every finished job leaves under queue/ and running/ once the job has settled in a terminal prefix (said as retired N settled sentinel(s) under <prefix>/; a failed sweep as retiring the settled sentinels under <prefix>/ failed), so the host store does not keep two records per fired routine for good and each pass's listings stay the size of the live work, renews the leases of the jobs it runs, and starts every queued host job as a child shell running the job's command, its output appended to ~/.stado/host-store/logs/<job-id>.log. A host job runs as a fleet worker runs a job: in its own directory, ~/.stado/host-store/work/<job-id>, with WC_JOB_ID, WC_JOB_STARTED_AT, STADO_BIN, and the firing schedule's STADO_SCHEDULE_ID, STADO_SCHEDULE_CRON and STADO_SCHEDULE_NEXT_DUE_AT (its next firing, the moment the run must end before) set, and whatever it leaves in that directory's output/ is kept in the host store under status/<job-id>/output/ when it ends, where stado artifacts <job-id> against the host store reads it; a routine that hands back its receipt as an artifact works the same from either store. Once the output is kept the work tree is removed, so the host store holds no tree per fired routine; a job the pass fails because its worker stopped mid-run has its output kept and its tree removed the same way. A child that exits moves its job to completed, or to failed with the command ended with <status> in error; output that could not be kept fails the job with the command ended with <status>, but its output in <dir> was not kept (<error>); its work tree stays there for inspection, and a tree that could not be removed after its output was kept is said in the log as <job-id>: its output is kept, but its work tree <dir> was not removed: <error>.

The schedule's immutable submission plan, replay lookup, claims and queue writes all use that same host store. They do not reopen the configured fleet backend while submitting a due routine. The library operation JobStorage::submit_batch requires the request's bucket to equal the selected store's bucket; otherwise it refuses with submission bucket <requested> does not match selected store bucket <selected> before writing a plan or queue entry. Ordinary stado submit still selects the configured backend.

Diagnose an outage

Nothing in the host pass reads the fleet store, so a fleet store that does not answer stops no host routine. Its log lines start [host-routines]; a store that cannot be opened is said once as the host store cannot be opened, so no host routine runs: <error>, and every failed step names itself (firing due schedules failed, the sweep record was not written, renewing the lease of <job> failed, starting <job> failed, <job> ended but was not moved to <state>).

A routine whose next run passed and did not fire is named in the listing itself: schedule list reads the coordinator's last recorded schedule sweep once and, for each enabled schedule still waiting past its next run, adds the reason schedule show <id> gives as overdue (--json) or as an overdue: <reason> line under its row: its occurrence is reserved and not yet enqueued, no sweep was recorded in the store, the last sweep failed with its error, the coordinator has not swept since a time on a host, a sweep ran after the due time and did not fire it, or the recorded sweep time is unreadable. Each sweep records when its sweeper promises the next one (next_by: its start plus two of its periods, the coordinator's interval or the worker's poll; none for a one-off stado scheduler tick), and a schedule that fell due after the last sweep is not overdue until that promise has passed: the reason then names it (… which promised its next sweep by <time>: that coordinator has not ticked since …). A sweep recorded by an older Stado promises nothing and is judged as before. A schedule that is due, paused or held has none. Oko's oko routines list passes the same field through for its own routines, and Oko Desktop's Automation screen shows it under the next scheduled time.

A fleet-store refusal alone does not prove the routine ran. Inspect the routine's actual job state in ~/.stado/host-store/completed/ or failed/ and its log under logs/; match the job's schedule_id to the declaration. A healthy worker process or a listed schedule is not execution evidence.

Real qualification

Run the existing isolated product journey from the Stado checkout:

kompilator test --manifest-path stado-rs/Cargo.toml --test scheduler -- \
  host_routine::a_host_routine_fires_and_runs_while_the_fleet_store_does_not_answer \
  --nocapture

It declares a genuine host-local schedule through the CLI, starts the actual standalone worker against a loopback endpoint with no listener, observes the real fleet connection refusal, and requires the schedule's completed job and its command output in the host store. Each run retains command exits, source revision, worker output and report.json under .build/z/<run>/. It does not change the operator's configuration, schedules or service process.