Release and Compatibility

This document defines how a Stado source revision becomes an identifiable, immutable, upgradeable, and recoverable product release.

Canonical version

stado-rs/Cargo.toml is the only source of the product version. Binaries expose that version through Clap, release jobs read it from the manifest, and every published object uses it in its immutable coordinate.

A release tag has the form:

stado-v<semver>

The tag version must equal the Cargo package version exactly after removing the stado-v prefix.

Stado follows Semantic Versioning:

• patch: compatible defect, security, and operational corrections;
• minor: backward-compatible public capabilities or optional fields;
• major: an incompatible CLI, machine API, persisted-state, configuration, or provider contract.

Before 1.0, an incompatible preview-integration change may occur in a minor release only when the release notes identify the affected adapter, migration, and rollback boundary.

Stado-native release pipeline

Every product carries one strict .wisent-release.json v1 contract. A product that is not released declares only schema_version, product, releases:false, and a reason. A releasing product declares its version source, platform recipes, argv-array quality and build commands, stage mapping, promotion policy, immutable inputs, and post-publication deliveries. A runtime contract is required only for products reconciled onto registry hosts. Repository URLs, branches, hosts, buckets, tokens, signing-key bytes, and provider credentials are forbidden from that file.

The product version is selected by AutoVersion and passed explicitly:

stado release submit --source DIR --version V --channel candidate

Stado verifies V against the manifest's checked-in json, regex, or text version source. By default it requires a clean checkout and captures its full HEAD commit before reading the manifest, version and archive. It never contacts a Git remote.

To publish an exact committed source while other work is in progress:

stado release submit --source DIR --commit FULL_GIT_SHA --version V --channel candidate

--commit names a 40-character lowercase Git commit already present in DIR. It reads Git objects directly: no checkout, clone, worktree, stash or index change is performed. The manifest, version source and archive all belong to that commit, even if the current branch, staged edits, working files or untracked files differ. --version must match that committed source, not the current working file. Omitting --commit still refuses a dirty checkout with release source must be a clean committed Git tree. Malformed commit names are refused with --commit must be 40 lowercase hexadecimal characters; a tree or blob is refused with --commit must name a Git commit object. Missing objects and committed files retain Git's actual read error. These failures occur before any build is queued.

The ownership chain is:

committed tree
  -> deterministic source.tar.gz
  -> stado://sources/<product>/<sha256>/source.tar.gz
  -> fleet queue quality/build job per platform
  -> status/<job>/output/{receipt.json,release.tar.gz}
  -> signed stado://releases/<product>/<version>/<platform>/
  -> registry.release_control desired CAS
  -> release_agent reconciliation
  -> exact-digest deployment receipt

The source object is create-only and carries metadata for its exact Git commit, source digest, and pipeline-manifest digest. A durable stado://<queue-namespace>/runs/release-pipeline/<id>/run.json joins those identities to platform job IDs, canonical output prefixes, delivery jobs, state, and failure. Repeating submit with the same inputs resumes that run and does not rebuild a platform whose published output is already recorded.

When the checkout has changed, resume the recorded run instead of submitting the new checkout under an already published version:

stado release status PRODUCT --json
stado release resume RUN_ID --json

Use the full run_id from the status JSON. resume reads no local Git tree or local product manifest. It verifies the stored run identity, manifest coordinate and manifest digest, then uses the same build, publication and delivery processing as submit. Queued and running jobs keep their IDs; published platforms are verified rather than rebuilt. Failed or cancelled deliveries get an idempotent replacement keyed by their previous terminal job, so repeating an interrupted retry does not create a second replacement. A stale delivery summary is checked against the queue.

Malformed IDs are refused with run ID must be 32 lowercase hexadecimal characters. An unknown ID reports release run <id> does not exist; a changed stored manifest reports release run manifest digest mismatch. These refusals do not replace the recorded source. Other failures retain the actual operation's error in the same run and in command output.

Stado Desktop's Releases screen has Resume selected run. The review dialog shows the original source revision and exact command before execution; Release resume result retains stdout and stderr, including a refusal. The request and read-back use the selected Stado API endpoint through POST /api/operator/run, with the existing RUN_MUTATION confirmation; Desktop does not launch a local CLI for resumption. The native API's ordinary 300-second request bound applies. If that bound expires, the recorded run and jobs remain resumable with the same command. Resuming already published platforms needs their verification keys, not a private signing grant; a platform still requiring publication needs that grant.

Available operations… → Submit a release source on the same Releases screen sends the repository path, optional full commit, version and channel through the selected Stado API. The repository path is on that API host, not implicitly on the desktop computer. Leaving the commit blank requests the clean-HEAD behavior. The form displays the exact command and requires review before submission; its receipt retains the process result, stdout and stderr. The same API request bound applies, and the recorded run can be continued with Resume selected run.

Platform output coordinates are canonical lowercase identifiers. Each recipe names the verified fleet runner_platform (darwin-arm64 or linux-amd64), quality argv in order, one build argv, staged source-to-archive paths, and optional secret_env references of the form ENV=item#field. Builders receive no repository coordinate or repository token. They materialize only the exact source URI and declared immutable inputs. Inputs are digest-checked and either extracted under WISENT_INPUTS_DIR or mounted there as their original archive, as declared by each input's extract field. Builders write output through the queue's existing canonical job-output collection.

A recipe may also declare min_free_gb, the free space its build needs on the work volume. The worker reads the volume holding the checkout through the host's own df -Pk and refuses before the first quality gate when the space is not there, naming the requirement, the reading and the path. A recipe that declares nothing is not measured. The refusal is the whole point of the field: the Stado 0.20.3 Darwin build compiled 616 crates on charless-mac-mini and died on No space left on device (os error 28) while rustc wrote metadata, so the requirement was readable only as a linker error inside a 30 KB job log, twenty minutes after the job started. stado space reclaim <host> --apply makes the room; the refusal says how much is missing.

A recipe field is read by the worker installed on the builder, not by the coordinator that queues the job, and the manifest refuses fields it does not know. A product therefore declares a new field only after a release carrying its reader is installed on every builder of that product's platforms: Stado's own manifest declared min_free_gb while every fleet builder still ran a worker that predates it, and the 0.20.4 darwin job died at the manifest with unknown field \min_free_gb``. The reader ships first; the declaration follows it.

The builder also measures what its build actually wrote. When the build step ends, before the scratch tree is removed, the worker records the tree's bytes and the free bytes left on its volume in scratch.json beside the receipt, and a failed build's receipt and exit sentence carry the same two numbers (the build tree held 14.8 GiB and its volume had 0.0 GiB free (ran out of disk)). The record is its own output leaf, never a receipt field, so a reader older than it is not asked to parse it.

Placement reads that evidence. Before a build job is queued, the coordinator looks up the newest scratch.json for the same product and platform and judges every live builder's capacity publication against it: a host whose published free disk is less than that scratch above the host's own low watermark is refused with release_scratch_short, naming the free disk, the measured scratch, the host it was measured on, and the low watermark, and nothing is queued. A build that failed for lack of disk is a floor, reported as wrote at least … before running out of disk. Among the builders that may take the job, the one publishing the most free disk goes first; name order decides only ties. A product no measuring builder has built yet is placed without a scratch verdict — silence is not a requirement — and its first build leaves the record the next one is judged by.

The keys in stage are relative to WISENT_OUTPUT_DIR, not the source checkout. The worker normally binds that output directory to source/.wisent-output. For a build that uses --target-dir .wisent-output/stado-rs/target, the release archive stages stado-rs/target/release/stado from WISENT_OUTPUT_DIR as its stado member. Putting .wisent-output/ in that key repeats the directory prefix and packaging fails even after compilation succeeds. Stage values name paths inside the release archive.

Publication writes the archive, immutable qualification receipt, signature, and signed manifest last. The signing key is read from the configured Skarbiec item field private_key; only the item name and trusted key ID are configuration. For the default configuration the item is stado-release-signing, never a secret value in a manifest or command line.

Post-publication deliveries are queue jobs consuming the canonical archive URI and digest. Required package or product channels gate completion; optional deliveries are adapters whose failure is retained without changing canonical release success. GitHub may be one such optional adapter, but no GitHub-hosted step is part of the source, qualification, publication, promotion, or reconciliation contract. The repository's existing deploy.yml invokes the same Stado CLI convergence after publication; it is not a second release contract.

A delivery that installs software on a fleet host declares that host in its optional target field and is pinned to run ON it, where stado release install-local verifies the delivered archive against the contract digest (WISENT_RELEASE_ARCHIVE, WISENT_RELEASE_SHA256) and extracts the named member. For the Stado product it first moves the already-downloaded, verified archive into $HOME/.stado/releases/stado/<version>/<platform>/stado-reader-convergence.tar.gz. It installs changed root bytes under $HOME/.stado/bin by rename with a dated backup. When those installed bytes already equal the verified payload, a required-delivery retry leaves the root untouched while repairing the attestation copy and release-version handshake, then reruns the image-idempotent global reader pass: processes already mapped to the installed inode are not kicked, while stale global reader images are reconciled. In either case private updates execute through the explicit installed $HOME/.stado/bin/stado, never through the delivery process image that an earlier rename may have retired. The same command gives every registry-declared Stado reader whose executable lives under $HOME/.stado/services/ that exact archive through the existing idempotent service update --from-archive --refresh-image path.

The macOS image-reconciliation pass enumerates every loaded label and domain, its declared executable and running command, then verifies the kernel image. A label launchd holds without running it has no image: the inventory reports it with an empty pid, whether launchctl list wrote - for it or a domain table wrote 0, and the pass leaves it alone to pick up the installed binary when launchd next starts it. Only a live pid reaches the image read, so an idle unit that declares the installed binary never fails a delivery. It does not collect script-variable requirements, process ages or PATH-version diagnostics for this operation. Those remain available in the full service inventory; no labels are excluded from the installation's ownership check.

Routine service converge --apply owns the same reader completion contract. Root release activates and restarts only units that execute the global install; it never compares or kicks an unchanged private executable as though it were the new global file. If the global root is already byte-attested while a private reader is old or a prior private update failed, convergence reuses the retained archive. When that copy is absent or corrupt it fetches and verifies the exact declared archive without extracting, reinstalling, or restarting the root. Already-current private trees and process images remain running; stale ones are installed, refreshed, and proved, and any private failure keeps the whole convergence failed. The queue agent alone retains its release-handshake deferral. If root delivery fails, that invocation does not attempt the private-reader pass.

A target that declares no services has no private-reader leg to run, and that leg is recorded as skipped — a third delivery status beside completed and failed. Reader trees are per service: a host that declares none carries no tree to install, refresh or prove, so there is nothing for the pass to do and nothing in that to call a failure. The pass used to run anyway and report failed on every apply against such a host, with cannot read registry unit ownership: no registry document at :registry.json — a delivery that had installed exactly what it was asked to install, marked failed for finding no services on a host that declares none. skipped is what separates "there was nothing to do" from "the thing was not done", and only the second may fail a run.

When one private update exits non-zero after emitting JSON, its complete decoded or captured stdout and stderr remain in the existing stado-readers.detail field. The enclosing converge report does not reduce that evidence to the child's last line.

This partial-state resume contract requires the target's global receiver to be built from a release containing the retained-archive arguments. A numeric version banner alone does not establish that source identity. A receiver built without the contract rejects the hidden archive and digest arguments, so convergence remains failed and cannot report the private readers complete. Deliver a release containing this fix through the normal root path first; there is no compatibility shim.

Installation is a local file operation: no delivery needs ssh, Remote Login, or any other login service, and a failed delivery is re-enqueued on resume — a recorded failure never lets a run complete past a required delivery. A resumed submit retries exactly the failed legs and deliveries; published platforms are verified, never rebuilt.

promotion.reconcile:true is reserved for registry-hosted runtime products and requires a runtime contract. Package, web, mobile, source, and archive products use reconcile:false, omit runtime fields, and complete through their required delivery receipts.

Native release verification runs the real clean-source, cancelled-build, resumption and committed-source journeys. Their Skarbiec fixture selects SKARBIEC_BIN, then an installed executable on PATH, then ~/.stado/bin/skarbiec; it requires the current grant issue interface. Verification does not download a legacy broker which those tests never use.

Release-channel evidence and publication verification

The public release channel is an immutable byte-serving boundary, not a Git tree and not the fleet job store. Every released object has an exact coordinate under:

stado://releases/<product>/<version>/<platform>/

STADO_API_URL, otherwise the deployment's api.url, selects the canonical HTTP origin. It is an endpoint, not a choice of network provider: direct HTTPS, a reverse proxy, a tunnel or a provider-managed edge can carry the same release requests. Repository workflows read the STADO_API_URL repository variable, defaulting to https://stado.wisent.com; they verify served bytes, not the state of a particular tunnel program.

stado host recover-object-api <target> repairs the target's local object service. It does not enable, replace or reconcile public ingress. The public connection belongs to its declared web or gateway configuration and is checked by the release read-back.

The dashboard serves those objects through GET /api/release/object?uri=.... The route answers for one coordinate at a time; there is no release-listing endpoint. A caller must distinguish three states:

• present — the channel served the object;
• absent — the channel answered authoritatively that the coordinate does not exist;
• unreachable — the channel did not answer, so existence is unknown.

stado storage stat preserves that distinction for release URIs. A network or control-origin failure is infra_down; it must never be treated as absence and must never authorize regeneration or publication.

When the public gateway exhausts its upstream attempts, its HTTP 503 body names the operation, method, logical URI, selected origin, attempt count and cause. stado storage stat --json retains that body in an unavailable receipt and exits 69; a failure before an HTTP response is unreachable. stado storage get retains the cause in its failure diagnostic, exits 69, and does not save the error response as the requested object.

What the version gate compares

The version gate does not trust a checked-in released-surface.json. A repository file can describe intended publication, but it cannot prove which bytes the channel serves. The gate's programs live inline in .github/workflows/version-check.yml, which materialises them at run time under $RUNNER_TEMP/stado-version-check/ — surface.py reads a binary's advertised command list, baseline.py derives the baseline, version_check.sh compares the candidate against it, version_check_proof.sh proves the gate refuses dishonest inputs, and semver_at_least.py and unreachable_modules.py cover the version comparison and crate-root reachability. The repository has no scripts/ directory: the job deliberately needs no file out of the tree it is judging, and a checked-in copy of a gate it does not run is a program nothing executes. The baseline generator (baseline.py --best) derives its baseline from the newest complete Stado release for which both required native platforms are present:

1. ask the release channel for the linux-amd64 and darwin-arm64 manifests;
2. download each archive named by those manifests;
3. verify every archive against the manifest SHA-256;
4. extract the Stado binary from each verified archive;
5. execute the extracted binaries to derive their advertised command surface;
6. require both native artifacts to describe the same surface.

The resulting document records the immutable manifest coordinate as its source. The version gate (version_check.sh, given the candidate surface) asks the release channel to confirm that coordinate before comparing it with the candidate binary. A stale local binary, a prose claim, a Git tag alone, or a committed generated file is not release evidence.

The workflow sets STADO_API_URL to the declared control origin and obtains its read credential from Skarbiec. It does not override Stado storage variables to point at a runner-local service: doing that would make the candidate and the baseline answer questions about different stores.

What publication proves

Before the first publication write, stado release submit ensures the managed stado-object-api dependency is available. Publication then writes each immutable release object to the canonical channel, reads that same coordinate back through Stado, and byte-compares the returned content with the source. A successful write without a successful read-back is a failed publication.

The signed release manifest remains the commit marker and is written last. Therefore a complete release means all of the following are true:

• every declared object write succeeded;
• every published coordinate was served back;
• every served byte sequence matched its source;
• the manifest was published only after those checks;
• a later version gate can reconstruct the released command surface from the channel bytes alone.

If the control origin is unreachable, the workflow stops with infra_down. It does not replace the canonical channel with local state, reuse an old baseline, or report the release as absent. The failure that led to these checks and the shipped correction are recorded in stado#97.

One version, one build

An immutable coordinate is bound to exactly one source revision before any artifact byte is written into it. On 2026-09-02 stado 0.13.46 for darwin-arm64 was published twice, from two source revisions — 446ad490 through the signed pipeline and 641a52b2 through a tag train. Release objects cannot be rewritten, so that version could never afterwards mean one build, and the only remedy was publishing 0.13.47.

stado release claim-coordinate <PRODUCT> <VERSION> <PLATFORM> --source-commit <REVISION> [--json] performs that binding. It writes two create-only records:

stado://releases/<product>/<version>/source-revision.json
stado://releases/<product>/<version>/<platform>/source-revision.json

The version-scoped record is the arbitration point every publisher and every platform shares; its create-only write closes the race where two publishers each saw no sibling claim and then claimed different platforms from different commits. Each record carries schema_version, product, version, the platform where it is platform-scoped, and source_revision.

The command reports one of three outcomes: claimed, when this revision now owns the coordinate; confirmed, when the coordinate already attests this same revision, so republishing one build is not an error; and a refusal, when the coordinate attests a different revision.

A refusal is a decision this product made, so it carries error_code: refused, retryable: false and failure_point: cli.release.claim-coordinate, and a --json caller receives it as one document. The exact sentences:

• <product>/<version> already attests source revision <held>, and this publisher carries <candidate>. Release objects are immutable: publish a new version
• <product>/<version>/<platform> already attests source revision <held>, and this publisher carries <candidate>. Release objects are immutable, so one version can never mean two builds: publish a new version instead of writing a second build into this coordinate
• <product>/<version> already carries platform <platform> from source revision <held>; this publisher carries <candidate>. A version's platforms are one build: publish a new version

stado release promote refuses a channel move whose platforms disagree, with release platforms were not built from one source revision, and stado doctor reports a version whose version-scoped and platform-scoped claims do not attest one revision. stado-rs/tests/release_claim/ drives the real binary against an isolated object store: it reads both persisted records, repeats the identical claim without changing them, and proves a second revision is refused with the stored bytes untouched.

Product catalog

Stado owns product release policy independently of repository hosting:

stado release catalog sync --catalog /path/to/release-catalog.json
stado release catalog audit

Sync imports the fleet's reviewed central catalog, including explicit releases:false manifests, refuses missing or duplicate product names, and CAS-updates stado://system/release-catalog/<product>.json. --root ROOT remains available for bootstrapping a catalog from local registered checkouts. Submit records the strict manifest together with its immutable source identity before queueing work. Audit reads only Stado catalog objects and refuses malformed, duplicate, or silent catalogs; it does not enumerate a Git forge or require forge tokens.

Compatibility matrix

Contract	Compatibility rule
Human CLI	flags and commands may be added compatibly; removal or semantic reversal requires a major release
Machine JSON	clients send and receive a supported schema_version; additive optional fields are compatible
MCP	protocol version and tool schemas are explicit; mutation authority is never added to the read-only server implicitly
Job JSON	unknown additive fields are preserved where round-trip ownership requires it; incompatible required fields require migration
Queue/storage layout	readers reject unsupported future schema versions; writers never silently downgrade canonical state
Configuration	the root schema version is required; migration produces a new document and preserves the prior file for rollback
Coordinator/agent	a release manifest declares the minimum compatible agent; dispatch refuses an incompatible agent
Provider adapters	implemented capability does not imply stable support; the released capability catalog and live evidence define support

Upgrade procedure

1. Read release notes and compatibility range.
2. Record current exact version and platform.
3. Run stado doctor --fix-hints.
4. Pause new work and drain running work when the release changes state or execution contracts.
5. Verify the configured backup destination.
6. Copy and verify canonical state when a schema migration requires a recovery point.
7. Download the exact candidate through the public release route.
8. Verify manifest identity and SHA-256.
9. Install atomically while preserving the prior binary.
10. Restart the selected canary service.
11. Verify version, health, queue visibility, and one representative workload.
12. Resume dispatch only after the candidate evidence is clean.

Rollback procedure

Rollback uses the exact previously recorded release coordinate; it never rebuilds source in place.

1. Pause dispatch.
2. Stop the affected service.
3. Restore the previous verified binary atomically.
4. Restore the previous configuration file if configuration migration occurred.
5. Restore canonical state only when release notes explicitly state that the new writer produced an incompatible layout.
6. Restart and verify version, storage reachability, queue counts, and agent compatibility.
7. Resume after one successful local workload.

If a migration is forward-only, the release notes must say so before installation and the rollback procedure must restore a pre-migration state copy.

When Stado itself cannot deliver

The upgrade and rollback procedures above assume the host's Stado binary and resolver still work. If either is missing or broken, do not turn those steps into an ad-hoc SSH copy. Use the single versioned host-recovery procedure: it restores the registry-selected release object API before its first catalog read, fetches the exact canonical signed artifact without the local resolver or a remote Stado binary, verifies manifest/signature/SHA-256, installs atomically, and restores the preserved binary automatically if the remote resolver probe fails. Routine declaration-driven delivery remains host release; the recovery path exists only to restore Stado itself.

Release notes contract

Every candidate and stable release records:

• source commit and immutable artifact coordinates;
• user-visible changes;
• fixed failures and security changes;
• CLI and machine-contract changes;
• configuration and persisted-state migrations;
• provider capability changes;
• supported platforms;
• known limitations;
• required operator actions;
• upgrade and rollback instructions;
• acceptance evidence and exclusions.

User-visible history lives in CHANGELOG.md. Incident detail is operator material and stays out of this repository.

Promotion gate

A candidate may become stable only when:

• the version/tag/source relationship is exact;
• all required platform artifacts and manifests exist;
• checksums verify;
• onboarding succeeds from a clean supported environment;
• core contract suites pass;
• every stable integration has current live acceptance evidence;
• upgrade and rollback succeed on a canary;
• release notes describe all compatibility and migration effects;
• an operator explicitly promotes the immutable candidate.