The delivery routine and the compiler cache

The delivery routine

Stado installs the products a repair has to reach — Stado, Tama, Oko, in the order you declare — itself, from the main of their checkouts, and then pushes every repository, so a delivery never waits on anyone's turn. Declare the plan once, and schedule the routine on the cadence you choose; nothing is built per commit.

stado ci plan set --purpose <defect> --push --step stado:cli:/Users/me/Wisent/stado --step tama:cli:/Users/me/Wisent/tama --step oko:cli:/Users/me/Wisent/oko
stado ci plan show
stado schedule create --cron '0 * * * *' 'stado ci run'
stado ci run --json
stado ci status stado
stado ci plan clear

Each --step is PRODUCT:SURFACE:CHECKOUT, the checkout named absolutely and holding a git checkout; a step named twice, a relative checkout or a missing one is refused by name. --purpose names the open Oko defect the installs repair. The plan lives at ~/.stado/delivery-plan.json (STADO_DELIVERY_PLAN), the routine's state at ~/.stado/delivery-state.json (STADO_DELIVERY_STATE).

stado ci delivery [--json] says whether the path that delivers repairs stands on this machine, reading only: the routine's state (routine: plan, running step and for how long, refused step with its refusals and defects, last pass), who holds the routine's pass lock (pass_lock_holder, the pid <n> since <time> a pass records beside delivery-state.lock when it takes it, or null), which product operations hold build trees (build_holds), and this machine's Stado unit as its init system holds it (unit: label, loaded, state, pid, runs, last exit code and why it is failing, or read_error naming why it could not be read — never taken for a healthy unit). Only the unit's state is read, never launchd's event log, which log show reads for minutes on a busy host while Oko's gates and oko sessions agents wait on this answer; events says so (not read: this reader asks for the unit state only). stalled names every way the path is down, one sentence each: a refused step with its refusals, a step recorded as installing that no pass holds (the pass ended without an outcome), and a failing unit; it is empty while the path stands. blocking_defects carries the Oko defects recorded for the refused step. unrecorded names each refused write in Oko's register about a step's last checked commit or a commit whose repairs are still kept (main of <step> at <commit>: …), the refused step (step <step> refused at <revision>: …) or a step's last install (step <step> installed at <revision>: …), printed as register write refused: …. Oko reads it: oko work next puts the defect that blocks a down path first for every executor, and the reflection routine records one against Stado when none is open. A delivery state that cannot be read is the refusal, naming the file.

The routine installs and builds, so it asks the operator's rulings first. Both stado ci plan set and every stado ci run ask Oko's one admission decision, for every step and for the plan's purpose, for both actions a step's install runs: oko admit build --product PRODUCT --purpose DEFECT and oko admit install --product PRODUCT --purpose DEFECT. Each admission is kept in the routine's environment (STADO_ADMITTED_BUILD, STADO_ADMITTED_INSTALL), so the stado product install a step starts inherits both and never asks again without the purpose. Stado keeps no copy of any ruling. While a ruling forbids building or installing, for example until every task and defect is closed, a consent the operator gave only for one purpose (to repair CI/CD) admits the routine for as long as that defect is open. Any other purpose is refused, and so is the same purpose once its defect closes. The refusal quotes the ruling Oko names: declaring the delivery plan is refused: build of <product> is not run: the operator's rulings do not allow it now (<Oko's refusal>) (and the same for install of <product>), or stado ci run is refused: …. A refused declaration writes no plan. A refused pass takes no lock, installs nothing and writes no state, so a scheduled stado ci run does nothing until the rulings admit it. A plan file written before plans carried a purpose cannot be read and is refused naming the file; declare it again with --purpose.

One pass (stado ci run) first runs the main check: for every step, each commit of its checkout's main after the last one it checked, oldest first (only main HEAD the first time, and when main was rewritten so the last checked commit is no longer behind it), runs the product's own declared gates — every kind its .wisent-release.json declares — over that exact committed tree, exported beside the checkout, on the host that runs the pass, under the plan's purpose the rulings admitted. Each commit is recorded green or red with every refusal (the gate's own words, so a compiler error names its file and line) in ~/.stado/ci-main.json (STADO_CI_MAIN_RECORD), as soon as it is checked. A red commit after a green one is recorded in Oko as one defect per session whose recorded repair the commits since the last green one brought (or one unowned defect), --found-in <repository>@<commit>, every refusal in its evidence, so Oko attributes the break to that commit; a red commit after a red one notes on those defects, and the next green one records itself as their repair.

The pass then runs stado product install PRODUCT --surface SURFACE --source-commit <commit> --wait at each step's last green commit, never main HEAD, for every step not yet installed at it; a step already installed there is said and skipped, and a step with no green commit yet stops the pass: <step> has no commit of main the main check found green; it and every later step wait. The install runs every quality gate to its end, so its refusal names every gate that refused the revision at once (<product> quality failed, every gate run to its end; build evidence: <path>: followed by quality "<gate>" refused the revision: … per gate, and could not run on this host for a gate whose program the host lacks). A refused step stops the pass, because the later steps build on it: the state keeps it with every refusal, Oko gets one defect per session whose recorded repair (oko defects repaired) the refused revision brought since the step's last delivered revision — owned by that session, every refusal in its evidence, or one unowned defect when the revision carries no recorded repair — and the command exits non-zero. While no newer commit of that step is green, a pass installs nothing and says <step> refused at <revision> and no newer commit of its main is green; waiting for a commit that repairs: <defects>. Once one is, the pass resumes at that step: a refusal that remains is noted on the same defects, and the revision that installs is recorded on them as their repair (oko defects repaired <defect> --revision <repository>@<revision>). Each of these writes — a defect, its assignment, a note, a repair — may be refused by the register (Brama gives no judgment and no attestation stands in, a step it asks for first). That ends nothing: the commit stays green or red, the refused step is kept, the install delivered. Each refused write is said on stderr as stado ci run: <step> is green at <commit>, and oko defects repaired <defect> --revision <repository>@<commit> was not recorded: <the register's sentence> (is red at, is refused at, is installed at for the others), kept in the unrecorded list of the commit, refused step or delivery it was about, said in the pass (<commit> green, its register writes refused: …, <step>: its register write was refused: …) and listed by stado ci delivery. A refused repair also keeps its defect in that commit's or delivery's unrepaired list, and every later pass records it again until the register takes it. A defect the register says open ones already restate stands on them: the evidence is noted on each and their ids are kept. Authors that cannot be read leave the defect unowned, saying why; a refusal after one whose defect was never recorded is recorded anew. One pass runs at a time: a pass that finds another running prints another pass is installing <step> at <revision> since <time>; this pass ends. A plan that is not declared is refused naming stado ci plan set.

When every step is installed and the plan was declared with --push, the pass then runs git push origin refs/heads/main:refs/heads/main in every checkout of the plan, each once, through the push gates it just installed, and keeps each outcome under pushed (by checkout: the revision, when, and git's refusal for a push the gates refused); a refused push exits non-zero naming every one. A plan declared without --push installs and pushes nothing upstream, and the pass says the plan declares no push.

stado ci status PRODUCT carries the routine under delivery, and stado ci delivery under routine: the plan, the step running with its revision and age_seconds, the refused step with its refusals and defects, the revision each step last delivered, each checkout's push, what the last pass said, and main: per step its last_checked and last_green commit and every commit checked, each with revision, at, green, refusals, defects, unrecorded and unrepaired; the plain form prints <commit>'s repair of <defect> is still to be recorded in Oko under the step for each kept repair (a delivery's as its repair of <defect> …). stado ci delivery names a step whose last checked commit is red among the ways the path is down: main of <step> is red at <commit> (installs stay at <last green>): <refusals>. Stado Desktop's Products screen runs the same commands: Declare the products the delivery routine installs, in order, Read the delivery routine's declared plan, Remove the delivery routine's plan, Run one pass of the delivery routine: check every new commit of main, install each step's last green commit, Read the delivery path and the main check of every step and Read what the forge's continuous integration and the delivery routine last did.

The real journey tests/ci/delivery_run.mjs declares a plan without --push over a catalogued product's canonical checkout with the plan and state kept for the run alone, runs a pass and reads the state it wrote (the step delivered at its last green commit with no push, or refused with its refusals and defects), runs a second pass and requires it to install nothing again, requires the checkout's view of its origin unmoved, and requires the refusals of a relative checkout, a missing checkout, a step named twice and a pass with no plan. Its passes run with no OKO_ATTESTATION, as the routine does; with a red record of the last green commit's parent naming a defect recorded while Brama was unreachable, and a refused install there naming one the register does not hold, the next pass must end cleanly, installed, with both refused repairs (the first has no judgment) kept in unrepaired, said and listed by stado ci delivery, and a later pass run with an attestation must record the first repair and keep the second. It needs STADO_BIN, OKO_BIN, STADO_DELIVERY_TEST_STEP and STADO_DELIVERY_TEST_PURPOSE. The real journey tests/ci/delivery_ruling.mjs asks Oko for two purposes (STADO_DELIVERY_TEST_PURPOSE, STADO_DELIVERY_TEST_OTHER_PURPOSE) and requires Stado to answer as Oko does. Where Oko refuses, declaring is refused quoting the same refusal and writes no plan, and a pass over a plan already on disk is refused the same way and writes no state. Where Oko allows, the plan is declared with its purpose. The real journey tests/ci/main_check.mjs commits a Rust crate whose one gate is cargo check --offline into a checkout of its own, keeps the plan, state, record and Oko register for the run alone, and requires: its first commit green and last green; then a commit removing a function green and the next one, still calling it, red with cargo's error naming src/lib.rs:<line> and the function, recorded in Oko as found in that commit, the last green staying the earlier commit and stado ci delivery naming the red main; then a repairing commit green, last green, and recorded on the red commit's defects as their repair.

The compiler cache

Every Cargo build Stado runs compiles through Kache, a compiler cache that keys each rustc invocation by the content of its inputs. Each build has a target directory of its own, an install run or a release job's scratch tree, so without a cache every build compiled every dependency again. With it, a crate this host compiled once, for any product or run directory, is restored in every later build.

The tool and its exact version are declared once, in stado-rs/data/work/compiler-cache.json. Stado sets Kache as Cargo's RUSTC_WRAPPER on the commands it starts itself:

• stado product cargo build|check|test|run (not metadata) and the owner-local stado product install of a Cargo recipe;
• stado product cargo stage on a release builder;
• every step of a release builder job whose source has a Cargo.toml at its root or one directory below it, including scripts those steps call;
• stado host run build, on the target host.

It changes no Cargo configuration of the account; a cargo you type yourself is not wrapped. A host without the declared version installs it with cargo install --locked --version <declared> kache before the first compile. The owner-local result record cargo/<invocation>/result.json — in the checkout's build area ~/.stado/products/builds/<the checkout's path below the home> when stado product cargo runs by hand, inside WISENT_OUTPUT_DIR in a builder job; its private scratch is cargo-scratch/<invocation> of the same build area, removed when the command ends — carries compiler_cache with the executable and version that was used, and a builder job's log prints [release-worker] compiler cache: <path> <version>.

Read every retained Cargo receipt of a checkout without starting another build:

stado product cargo --manifest-path ./rust/Cargo.toml results --json

To read retained receipts from every canonical checkout in the selected workspace without choosing a manifest or starting a build:

stado product cargo --all --json results

The results answer names each receipt path, state, operation, source records, failed operation and error. A missing receipt directory means no executions have been retained for that checkout. A receipt is a directory named by its invocation's UUID; the installation's Cargo cache in the same area (cargo/<platform>/<lock digest>/<revision>/<recipe>) is not one and is not read. An invocation that has written no result.json yet — it is running, or it ended without one — is listed with state unfinished and that cause, and the other receipts are still read; an unreadable receipt refuses with its path and filesystem or decoding cause. --revision and forwarded Cargo arguments do not apply to this read and are refused. A compilation records the full output streams beside result.json as cargo.stdout and cargo.stderr, with their paths and the compiler's file, line, code and message findings in the receipt. Inspect the failed operation and findings before attributing the failure to a commit; an earlier preparation failure has no compiler finding.

--all selects the workspace's canonical checkout directories and returns one results array with the same receipt paths and source records. It cannot be combined with --manifest-path or used for a compiling Cargo operation. An unreadable workspace, checkout entry or retained receipt refuses with the failed path and cause rather than silently omitting that build. Oko's compiler scorecard reconciles this workspace-wide answer across products; a missing source revision or transcript author remains an explicit pending attribution, not a clean model score.

A build whose cache cannot be installed fails at that step:

cargo install --locked --version <declared> kache failed (<status>), so no build runs through the compiler cache stado-rs/data/work/compiler-cache.json declares: <cargo's error>

A build is never compiled without the cache in its place. To change the version, change the declaration; every host installs it on its next build.

stado product compiler-cache status --json
stado product compiler-cache ensure
stado product compiler-cache remove

status compares the installed Kache with the declaration and reports state: ready, absent, other-version, or unreadable (it exists but --version failed, with the error). It changes nothing and exits nonzero unless the state is ready. ensure installs the declared version when it is absent or another and reads the version back. remove runs cargo uninstall kache and refuses with kache is not installed at <path>; nothing to remove when it is not there. Kache is used only where cargo install places it: $CARGO_INSTALL_ROOT/bin, else $CARGO_HOME/bin, else ~/.cargo/bin. A Kache on PATH from another source is not the declared one and is not used. Another host's cache is read or brought to the declared version over the fleet channel with stado host compiler cache TARGET status|ensure|remove [--json], which runs that host's own stado product compiler-cache and prints its report.

Kache keeps its store in the account's cache directory (~/Library/Caches/kache on macOS, ~/.cache/kache on Linux). Run kache stats on a host to read its hits and size.

A build on a host

stado host run build TARGET --manifest-path PATH --bin NAME [--json] builds one binary of a source tree delivered below the target account's ~/.stado/work/runs. It is refused before anything is compiled when the target's volume is already at the disk-full threshold: <target>: its volume is <used>% used, at the <threshold>% disk-full threshold; a release build would take the host's services down …, with stado space report <target> to read what fills it. Every build shares one Cargo target, ~/.stado/build-cache/host-build/cargo-target, which the build-cache cleaner owns, and only the built executable is copied into the run directory's target/release/, so a run directory holds source and that one file.

The outcome says what the build took from the volume. The text form ends with <target>: the build took <n> GiB of free space on its volume, from the host's df reading before and after; --json carries free_bytes_before and free_bytes_after. A host that does not answer the second reading leaves the build's result standing, prints the volume's free space after the build was not read: <error>, and --json carries that error in free_bytes_after_error.