Held-back jobs

A job's state says where it is, not why it stays there. A host can leave a job where it is rather than run it: declined (its agent grant cannot resolve a secret the job names), its apt packages not installed, not charged to the build budget, not claimed for want of a lease promise, its claim refused by the store (the job stays in queue/), or run to its end with the store refusing the terminal move (it stays in running/ and the slot retries the move every tick). The job's state says none of that, so each host records its newest such decision in the job's status record, status/<id>/held/<host>.json (job_id, host, decision — declined, packages-uninstalled, uncharged, unclaimed, claim-refused, terminal-unrecorded — reason, at). stado job watch prints them (held_back under --json, one held back by <host>: <decision> (<reason>) line otherwise).

stado job watch <job-id> --follow --until-held-back --json

--until-held-back also ends the follow when a host records such a decision, including one recorded before the follow began, and exits non-zero:

following <job-id> until it ends or is held back: a host recorded that it leaves the job queued, so it does not run there; observed: the job is queued; <host> declined it: <reason>

Fix what the host named (its grant, its packages, the build budget or its store), or let another host take it; --follow alone keeps holding until the job ends. tests/release/held_back.rs submits a job naming a secret the isolated worker does not allow, and reads the decline from the follow and the job's status record while the job stays queued.

Source: this website