On this page

Run a quality check on the fleet

stado quality check --place runs the committed-source check of one revision on a fleet host with room for it, instead of on the machine that asked, and follows it to its end:

stado quality check --root /path/to/product --revision COMMIT --place
stado quality check --root /path/to/product --revision COMMIT --target HOST
stado quality check --root /path/to/product --place --json

Without --revision it checks the revision an installation of the checkout builds, as the check here does. --target HOST places the check on that registry target and implies --place. --purpose DEFECT is passed on to the check on the host. --place and --target cannot be combined with --pre-push, which compiles nothing and stays on the machine that pushes.

Where the check runs

The host is chosen from what each host's own agent last published about itself (its capacity publication). A host is offered the check only while all of these hold:

  • its publication is live, inside the next publication its agent promised (next_by), as of the moment the publications were asked for: a listing of the fleet store can take minutes on a loaded host, and a host that kept its promise until the read began is not read as silent because the read was slow;
  • its Stado runs the work: the publication's placed_work lists quality-check;
  • it runs no CI job: ci_jobs_running is 0. The agent counts the processes its CI runners start for each job they take (Runner.Worker). When the process table cannot be read, ci_jobs_running is null and ci_jobs_unread says why.

A host that accepts placements now is admitted. A host that refuses only because it is busy is waiting: its CPU, its RAM headroom, an exclusive job, its reservations, its janitor's turn or a CI job. It still takes the check once that clears. Any other answer refuses the host: no publication, a stale one, a Stado that does not run placed checks, an unread CI count, or another admission reason. Admitted hosts come before waiting ones. Among hosts that stand alike, a machine whose registry role is interactive (the one an operator works at) comes last, and more free cores come before fewer. Every host's verdict is printed on stderr (stado quality check: placement: <host> admitted, … waiting: <reason>, … refused: <reason>), and --json carries them as placement.

When every host is refused, nothing is queued, and the refusal ends with what credit-funded capacity says. That is the Azure credit gate's decision on the balance read at that moment:

no fleet host can run the check of <product> <revision>: <host> refused: <reason>; …; no Azure machine is rented for it: <the gate's refusal>

With no floor ruling recorded, the gate's refusal is a VM create or start is refused: no standing ruling of the operator names an Azure credit floor …. With the credits above the floor it says the Azure credits are above the operator's floor, but placed quality-check rents no cloud machine; it waits for a fleet host. A placed check does not rent a cloud machine yet.

One check per host, one check per product

The check is a queue job pinned to the chosen host and marked exclusive. Its host claims it only while it runs no other job, and claims nothing beside it while it runs, so a host compiles one check at a time and later checks queue behind it. The job asks for the CPU cores the host published free when it was chosen (at least one), and the host's agent hands that number to Cargo as CARGO_BUILD_JOBS.

A product has one check in flight. Once a new check's job is queued, every check of the same product for another revision that is still queued or running is cancelled:

stado quality check: cancelled job <job> (<queue|running> on <host>), the check of <product> <older revision> this check of <revision> supersedes

The command following the cancelled check then ends non-zero: the placed check of <product> <revision> on <host> (job <job>) ended cancelled. A check of a revision that is already queued or running is not queued again. The command says <product> <revision> is already <queued|running> as job <job> on <host>; following it and follows that job.

The real journey tests/quality/placed.rs (kompilator test --manifest-path stado-rs/Cargo.toml --test scheduler -- quality_placed::) runs it end to end in an isolated deployment. The real stado serve --worker claims the placed jobs. A fixture product commits a revision whose gate keeps its check running and a later revision whose gate passes. The second check must cancel the first job, run after it and pass. The first command must end non-zero saying its check ended cancelled, and the store must hold the first job under cancelled/ and the second under completed/. The evidence is kept in report.json beside the deployment under .build/z/.

How the revision reaches the host

The revision travels as a Git bundle: the revision's history and the checkout's origin/main (its HEAD when it tracks no origin/main). The bundle is kept in the fleet store at runs/quality/<product>/<revision>.bundle until the host takes it, and a superseded check that was never claimed takes its bundle with it. On the host the job runs:

"$STADO_BIN" quality check --bundle runs/quality/<product>/<revision>.bundle --revision <revision>

STADO_BIN is the worker's own Stado, which its agent sets for every job it starts; on a host that is the installed one.

That rebuilds the repository under ~/.stado/products/placed/<product> on the host: the bundle's objects, its origin/main, and the bundle named as origin. The bundle is removed from the store, the same check a checkout gets runs over that repository, and the area is removed when the check ends. Because the path is the same for every check of the product on that host, every check there builds with the same Cargo build directory, and only what changed compiles again. A key that is not runs/quality/<product>/<revision>.bundle is refused as a usage error. A bundle the store no longer holds is refused: the fleet store holds no source bundle at <key>: a newer check of the product superseded this one, or it was already taken; place the check again with stado quality check --place --revision <revision>. A bundle that does not hold the revision is refused as placed-bundle-commit-absent, naming the git rev-parse --verify --quiet <revision>^{commit} that resolved nothing and the refs the bundle carried. Git is given the bundle's path as the system's bytes, so a home whose path is not UTF-8 works as any other. Git that does not start is placed-git-unstarted, with the system's reason (NotFound when no git is on the PATH, which it names); git that ran and failed, a store object at the key that is no bundle among them, is placed-git-refused, with the command, its status and both streams.

Following it

The command holds the job on the queue store's change watch until it ends: the store's directory on this machine, or, on a machine that reaches the store through the object API (storage.backend self-hosted or stado), that API's held listing of the store the registry's storage host keeps (GET /api/object/watch). Nothing waits on a timer. On a machine that reaches the store through an S3, GCS or Azure client, Stado's own /s3 route among them, the store gives no change notification, so once the job is queued the command ends job <job> could not be held to its end: … HOLD_UNAVAILABLE … naming that store; the job keeps running, and stado job watch <job> reads it. When the job ends the command reads its log from the same store and prints it. The command exits 0 only when the job completed. Otherwise it ends with the placed check of <product> <revision> on <host> (job <job>) ended <state>: <error>; its log is above and in stado job watch <job>. A watch that fails while it holds ends it HOLD_FAILED. With --json the log goes to stderr, and stdout carries {"status", "product", "revision", "target", "job_id", "placement", "superseded", "ending": {"state", "error"}}.

What the capacity publication states

Every agent's capacity publication carries the two fields placement reads. ci_jobs_running is the count of CI jobs running on the host, or null with ci_jobs_unread when the count could not be taken. placed_work lists the placed work kinds this Stado runs, today ["quality-check"]. A host whose Stado predates them is refused for placed checks, naming its Stado version: <host>'s Stado <version> does not run placed quality-check; install a Stado that does with stado product install stado --surface service --host <host>.

In Stado Desktop

Releases → Available operations… → Check committed gates and the built CLI command tree carries two more fields. Run the check on the fleet host with room for it passes --place, and Registry target to place the check on passes --target. The operation runs the same command on the selected Stado API host and shows its placement lines, the job's log and its ending or refusal as the CLI prints them.

Source: this website