Native builds

A build compiles one committed source on every platform its .wisent-release.json declares and keeps the result under its own id. A release consumes a build that passed. The two are separate commands with separate durable records: stado build compiles and reports, stado release submit signs, publishes, delivers and promotes, and it refuses a build that is still waiting or has failed.

A build never signs an artifact, changes release_control, or promotes a release. Its identity is the product, the version the commit declares, the digest of the committed tree and the digest of the manifest — no channel — so one build serves a candidate release and the stable release that follows it, and submitting the same commit again resumes the same build instead of compiling it twice. When Stado itself is delivered, the installer records the installed version beside the managed binary. Each managed queue agent finishes its active jobs, detects that its loaded version differs, and exits through its declared KeepAlive or Restart=on-failure policy; the supervisor then starts the installed release without unloading the unit or interrupting a job. Before executing an artifact, every delivery verifies both the newest submitted source in the product catalog and the exact published platform coordinate in its still-delivering run. This fences an older queued delivery without assuming the product has a separate release-control rollout policy. The Stado delivery job starts its worker from the digest-pinned candidate archive and that worker uses itself for release local install; a broken older installed worker therefore cannot prevent the release that repairs it. A non-Stado archive contains that product rather than another Stado binary, so its delivery runs the installed Stado worker against the digest-pinned archive.

Repeating stado release submit resumes the same release run. An initial platform submission keeps its original stable job identity and output URI. Before replacing a failed platform attempt, the coordinator reads the original job: queued, running, completed, and uploaded jobs keep that identity and their existing output. A publication or storage-read failure does not start another build. A failed job also keeps its identity when its durable build receipt says passed: the bootstrap's upload can fail after compilation and qualification succeed, while the native finalizer subsequently preserves the output. If canonical output is missing, resumption reads the retained receipt from the recorded builder through Stado. It verifies the receipt's run, job, product, version, platform, builder, source and manifest bindings, then asks that builder's installed Stado to upload its retained command log, archive and receipt to both the attempt and canonical output locations. The receipt is uploaded last; publication checks the storage write receipts, reads the canonical receipt back, and verifies the actual archive's SHA-256 before using it. A mismatch returns release job <JOB> returned a receipt that does not match this run: <FIELD> is "<RECEIPT>", not "<EXPECTED>"; …, naming every field that disagrees, an absent artifact with the archive's digest, and a status other than passed with the failure the receipt states. A missing archive returns release job omitted archive. Neither replaces a passed build. The original job keeps its recorded terminal state and cause; recovery does not rewrite that history.

The worker also checks for its own exact completed output before executing a replayed request. It reuses a passed receipt only when the immutable request, including its inputs and secret references, matches and the retained archive still has the recorded size and SHA-256. That replay returns the existing build instead of running the compiler again.

The stored platform request fixes its builder and input archives. Resumption checks that request against the release's source and manifest, preserves its exact bytes, and uses the already-staged inputs instead of fetching them again. An existing queue plan retains its recorded consumer and does not require a new live-builder selection. If no queue plan exists yet, the saved builder must still pass normal admission; another machine is not silently selected. An altered request returns immutable queue object differs: PATH without overwriting the saved request or creating a replacement build.

stado release status stado --json shows the recorded release runs, including each platform's builder, job ID, state, and failure. A recorded platform failure can describe a publisher read error rather than a failed worker; resumption checks the job and its build receipt before deciding whether another attempt is needed. A published platform can retain a failed job whose passing build receipt and archive were recovered; the qualification receipt records the successful build and the job record preserves the failed upload.

Every saved run is also filed under its product (runs/release-pipeline-by-product/<product>/<run id>), so a question about one product's runs, such as stado release status tama or a delivery's check for the newest published run, reads only that product's run objects. Runs saved before the index existed are filed by the next stado release newest, which already reads every run once; until then a product question reads every run object and says so on stderr ([release runs] runs are not indexed by product yet …).

When a required delivery fails, the submit command names the delivery and its queue job, followed by that job's recorded failure and last command output. The same detail is retained in the run's delivery record. Use stado machine status JOB_ID and stado machine logs JOB_ID to inspect that exact job; no new build or delivery is needed to retrieve its failure.

A cancelled unfinished build, or a failed build without a passed receipt, receives a retry identity derived from the release run, platform, and prior terminal job ID, with its own output URI. A cancelled replay is different when its retained passing receipt was completed before that replay started: resumption can publish the earlier completed build without undoing the later cancellation. A passing receipt from the cancelled attempt itself does not override cancellation. A crash before or after the replacement platform record is saved reuses the same attempt; only another recorded terminal job failure can chain to another retry. An unreadable job or receipt returns the storage or decoding error. A missing job returns release job JOB_ID was not found in recorded states; refusing a replacement without terminal failure; neither condition creates a replacement build.

Local queue jobs execute from the agent owner's ~/.stado/work/jobs/wc-<job-id> tree, not a temporary directory. Admission resolves the physical home, then creates or opens .stado, work, and jobs one component at a time as owner-only directories with O_DIRECTORY|O_NOFOLLOW; component symlinks are refused. The serialized old-agent bridge repeats that refusal and changes permissions or creates the next child only from a held cwd whose physical path it just verified. A release submission remains able to repair an older agent: its deterministic bootstrap moves that agent's already-materialized job into the persistent root before the worker starts and preserves a compatibility symlink for the old agent's log and artifact upload. It also pins TMPDIR, TMP, and TEMP beneath that persistent tree so a pre-migration worker cannot put its release scratch back in OS temp. After a successful worker, the bridge uses the checked storage writer to upload and read back the command log and archive at both canonical status and exact attempt URIs, then publishes the attempt's receipt.json last as the completion marker. Each writer receipt must match the quiescent local source's SHA-256 and byte count. Its JSON and lifecycle-watch responses use random 0600 files under the validated work tmp directory, are removed after parsing, and leave only their proof lines in the already-open command log. A failed write or read-back turns the workload into a failure instead of allowing a successful terminal record. For a worker that already failed, the bridge still proves the command log at both destinations and, when an owned regular receipt.json exists, publishes that failed receipt to canonical then attempt storage. Evidence errors are appended to the safe log without replacing the worker's original nonzero exit code; the legacy link remains for the old agent's canonical finalization. The bootstrap prints both physical workdir and effective temporary root before invoking the worker, so an operator can prove the retry left /tmp from the canonical job log. Queue-workdir reclamation runs only through the janitor under its exclusive admission lock; every pass reserves a bounded scan share for terminal old-agent links independently of canonical workdir enumeration. Each agent tick observes all local workload exits and releases their cleanup holds before reading remote policy or finalizing any slot. A slow upload or queue read for one job cannot retain the holds of siblings that already exited; live workloads remain protected. If the canonical tree is ever absent or replaced while a job is live, the agent emits workdir_missing with its exact expected path at heartbeat and finalization, then retains that marker and the real workload exit code in the terminal job error. An existing empty command log remains the distinct wrote no output case.

stado release redeliver PRODUCT RUN_ID DELIVERY --retry-token TOKEN is the operator recovery path for one delivery from the exact newest completed run. It does not publish a new candidate or move a channel. The retry token identifies a fixed per-run CAS transaction and stable queue submission, so repeating the same command resumes after interruption. A different token is refused while a transaction is active. Success replaces that delivery's job, output, state, and receipt evidence before restoring the run's prior state; failure preserves the previous passed delivery while recording the terminal failure and restoring the run state.

stado host config set TARGET KEY VALUE migrates an older deployment profile through the installed stado config migrate before applying the field. The migration preserves the exact prior file beside the profile and refuses newer schemas. No separate migration command is needed; --reload-service SERVICE activates the change through the declared service policy.

Owner-local builds covers owner-local source selection and qualification, and canonical Swift sources and editor indexing.

Owner-local Apple signing credentials

stado product reads the fleet's signing material through Skarbiec by default, from the one vault item that plays the role macos-development-signing (it carries the tag stado:role:macos-development-signing), so no private PEM text is copied into a command and no item is named. Another role is named explicitly:

WISENT_CODESIGN_ROLE=<role> \
WISENT_CODESIGN_ISSUERS_FILE=/path/to/apple-issuers.pem \
stado product install transcript-lake --surface cli --json

The item in the role supplies certificate and private_key. The optional issuer file contains public PEM certificates, not another private key. Overlapping issuer certificates are imported once, even when their PEM wrapping differs. Stado checks the key against the certificate, uses an owner-only, short-lived keychain under ~/.stado/signing, takes only its own entry off the search list, and removes its private files when the operation ends; one scope signs on a host at a time (signing).

An existing publisher can instead supply both WISENT_CODESIGN_CERTIFICATE_PEM and WISENT_CODESIGN_PRIVATE_KEY_PEM. Selecting a role and nonempty PEM input together is refused. No item in the role, two items in it, or an absent field reports the role and the actual Skarbiec read error; an unreadable issuer file reports its path and filesystem error. Invalid certificates and mismatched keys fail before installed bytes are replaced. An explicitly selected credential never falls back to another signing identity.

A product's catalog installation recipe may instead require signing_keychain, a local path on the machine running the installation. Skarbiec's CLI, service and desktop recipes declare $HOME/Library/Keychains/login.keychain-db; they do not read their own unavailable vault for signing. On a Mac without that file, stado product install skarbiec --surface cli --json refuses naming the missing keychain before exporting or building. An explicit role, PEM pair or different keychain does not override the recipe. A successful source installation records signing.source, signing.keychain and observed signing.signatures per native installed path in the JSON receipt; stado product status skarbiec --surface cli --json reads the same record. An exact published release is verified and placed without local signing.

This owner-local input does not widen a queue worker's Skarbiec grant or move publication credentials into the control plane. Queued releases still use their declared secret references and grants. Use stado product signing inspect PATH --json to read the resulting identifier, Apple team, authority and designated requirement. stado product signing report and Desktop Settings show the same installed signature diagnostics on the selected host; neither resets privacy permissions.

Request failure diagnostics

A storage request failure includes the HTTP client's complete cause chain, not just error sending request. Read the failed operation and URL together with the underlying cause: for example, Connection refused (os error 61) means the connection was refused, not that compilation failed. The operating system's error numbers differ between platforms. A TLS failure, or a connection the operating system abandoned, reports the cause the HTTP client supplied rather than being relabelled as a connection refusal.

The same message reaches the CLI, a release run's saved failure when that record can be written, and the native Desktop command API's stderr. Stado Desktop's Releases screen displays the recorded failure without shortening its cause chain. If recording the failure also fails, the command keeps the original error followed by failed to persist release failure and the second error; the stored run may therefore not contain the latest failure.

This change adds no request, retry, rebuild, or field to stored records. Existing failure classification uses the richer message. It cannot reconstruct causes discarded by an older binary. Inspect the retained command output from that run before attributing an older generic message to a specific network fault.

Release publication authority

Which publisher may publish a release and how its writes are authorized: Release publication and the active executable.

Wait for the stable release endpoint

Reading the stable release endpoint once a release is published: Release publication and the active executable.

Resolve the executable that is actually active

Finding the executable a host actually runs: Release publication and the active executable.

Private Cargo build inputs

Private Cargo sources a build needs and how they are supplied: Private Cargo build inputs.

Build one product

Building one product and the products its post-build tests run: Building, reading and releasing builds.

Read a build

Reading a build's state, its platforms and its failure: Building, reading and releasing builds.

Build the whole workspace

Building every product of the workspace at once: Building, reading and releasing builds.

Release a build

Releasing a finished build: Building, reading and releasing builds.

What the forge's continuous integration last did

Reading what the forge's continuous integration last did for a product: Building, reading and releasing builds.

The delivery routine

The routine that delivers every built release: The delivery routine and the compiler cache.

The fleet's daily build budget

The fleet's daily build budget and a recorded user exception: Building, reading and releasing builds.

Failure boundaries

Platform state	Meaning
no job for a platform	no builder could take it yet, or the build was refused before queueing
building	the job is queued or a matching worker claimed it
failed	the quality gates, the build command, packaging, signing or the receipt itself failed; the reason is on the platform line
passed	the job wrote a passing receipt naming this build and its archive

A successful process exit without a passing receipt and an archive is a failed platform. A host merely declaring a platform is insufficient: a live worker must publish capacity for that platform. The artifacts and command log of every job stay under status/<job-id>/output/ and download with stado results <job-id> ./build-results.

A platform whose secret_env asks for a role no item the stado consumer can see plays is refused before anything is queued or charged: a builder resolves a job's secrets as that consumer, so no host could ever claim the job. A secret_env entry is ENV: "role#field"; the role is what the secret is for, and the vault decides which item plays it (Skarbiec and Stado). The refusal names each environment variable and why its role cannot be selected:

Error: <product>'s darwin-arm64 build asks for roles the stado consumer cannot select, so no builder could ever claim it: MACOS_CERT_P12: no item visible here carries stado:role:<role>; store the secret for role <role> with `stado credentials item put --host <vault owner> --role <role>`. Store the secret with `stado credentials item put --host <vault owner> --role <ROLE>` and grant it to stado before building.

The check reads ids and tags only, never values, through the same listing stado credentials ls prints. A builder's declared agent.skarbiec.secret_fields is not proof: the declaration can name a role no item plays. A job already queued for a role its builder's grant cannot resolve is shown by stado host gates <builder> as a declined: line.

The quality gate names whose revision refused

How a refusing quality gate names the revision it read: Building, reading and releasing builds.

Disk pressure on build hosts

stado space report TARGET --json includes Linux inventory under the managed home, /home, /mnt, /var, and /opt. stado host reclaim TARGET --apply --reason TEXT also recognizes the former ~/.stado/work/platform-matrix-cargo-target cache. It refuses linked, unrecognizable, younger-than-one-hour, or live-held trees; unrelated untagged directories remain outside that cleanup. host build-caches reports no-cache-tags, root-protected, or scan-failed rather than making those cases look like a successful empty scan.

The compiler cache

How builds share compiled output through the compiler cache: The delivery routine and the compiler cache.

Desktop

Stado Desktop's Releases screen carries the same operations as the CLI: build a source, build every product, read and list builds, and release a passed build.