Held-back jobs
A job's state says where it is, not why it stays there. A host can leave a job
where it is rather than run it: declined (its agent
grant cannot resolve a secret the job names), its apt packages not installed,
not charged to the build budget, not claimed for want of a lease promise, its
claim refused by the store (the job stays in queue/), or run to its end with
the store refusing the terminal move (it stays in running/ and the slot
retries the move every tick). The job's state says none of that, so each host
records its newest such decision in the job's status record,
status/<id>/held/<host>.json (job_id, host, decision — declined,
packages-uninstalled, uncharged, unclaimed, claim-refused,
terminal-unrecorded — reason, at). stado job watch prints them
(held_back under --json, one held back by <host>: <decision> (<reason>)
line otherwise).
stado job watch <job-id> --follow --until-held-back --json
--until-held-back also ends the follow when a host records such a decision,
including one recorded before the follow began, and exits non-zero:
following <job-id> until it ends or is held back: a host recorded that it leaves the job queued, so it does not run there; observed: the job is queued; <host> declined it: <reason>
Fix what the host named (its grant, its packages, the build budget or its
store), or let another host take it; --follow alone keeps holding until the
job ends. tests/release/held_back.rs submits a job naming a secret the
isolated worker does not allow, and reads the decline from the follow and the
job's status record while the job stays queued.
Source: this website