Operations

Host health publication

Linux and macOS writers collect local disk and service state, then call stado host publish-beacon FILE. The command requires STADO_HOST_HEALTH_API_URL plus the dedicated stado-host-health-beacon Skarbiec URL/consumer/grant metadata. It resolves only stado-host-health-api/token and sends the document to authenticated PUT /api/host-health. The control plane stores host_health/<host>.json through its configured Stado backend, so Azure and local outage profiles write to Azure Blob and local storage respectively.

Missing routing, an unreadable or over-broad grant, an insecure non-loopback HTTP URL, failed authorization, and backend errors all leave the prior beacon untouched and return failure. There is no cloud CLI, provider SDK, direct bucket URL, ambient credential, or cross-backend fallback in the writer.

The systemd unit reads non-secret API/Skarbiec origins from /etc/stado/host-health.env; the launchd template carries the same non-secret routing metadata. Both keep the opaque Skarbiec grant owner-only at ~/.stado/host-health-beacon-skarbiec-token.

Reading admission and diagnostic failures

stado host gates HOST --json reads the registry, current disk usage, janitor state and snapshots, effective agent storage configuration, storage client, published capacity and queue. Each read has a ten-second budget and reports its operation, source, state, start and finish time, elapsed milliseconds, budget and error in observations. Independent reads run together. A failed host read does not remove a completed capacity or queue reading.

complete: false means the diagnostic could not finish; claiming is then null, and the command exits non-zero. Observation states distinguish complete, absent, cached, error, timed_out and skipped. A cached registry is identified as historical, not presented as a successful current read. An unread source is not the same thing as an absent object.

disk.free_bytes is measured by the host's filesystem reader; free_gb is its GiB display value. below_watermark uses the unrounded measurement. These fields are null when the measurement is unavailable, never zero. pressure_source distinguishes the agent's current capacity_publication from host_disk_measurement; capacity.diagnostics retains the agent's actual admission reason and reported state. Low free space and the agent's decision are different readings. Neither a slow queue read nor an unavailable disk reader proves that the disk is full.

Stado Desktop shows the same results in Hosts → selected host → Gates: completed measurements, unknown admission when reads are incomplete, each source and error, and the agent's published diagnostics. Reading this screen or running host gates does not clean a disk, restart a service or change policy.

Missing service reconciliation

The coordinator runs a service reconciliation stage during every stado optimize run and scheduled autonomy tick. It joins two independent facts: the unit state in the newest host beacon and a fresh stado service verify reachability sweep from the declared consumer hosts. The result is written to both state/autonomy/services/latest.json and an immutable state/autonomy/services/runs/<timestamp>.json; stado optimize status prints the latest report.

Every autonomy object is rooted under state/ because the object gateway authorizes a write by matching its key against the configured namespace's prefix allowlist. No namespace declares autonomy/, so the whole layer's writes were refused with 401 unauthorized or non-immutable release write while the local backup backend kept serving stale reads — the same defect state/host_silence/ was moved to fix.

The fleet-wide host-silence threshold is also the service-beacon freshness threshold. A missing or stale reported_at changes the service state to unknown; it never authorizes a host mutation — with one exception. The beacon unit's own death is what makes every other unit unknown, so a silent host's declared beacon unit is reasserted through the idempotent service ensure path over the host channel: the channel answering is the evidence that repair is possible. Ensure restarts a matching definition in place and reloads a changed definition only after preflight, with the loaded result read back afterwards. Once the beacon publishes again, the rest of the host becomes repairable from real evidence. A failed unit goes through the same ensure path as a missing one. For a host-probed unit, a live process running a stale copy of its own declared binary is kicked (the four-day stale-agent incident), while a process executing a binary the unit never declared stays refused as identity_unresolved. For a fresh beacon that omits a declared unit:

Endpoint evidence	Reconciliation
observed	Probe the declared unit. Stado adopts a corrected path or unit record only when the unit is loaded and its live process matches the declared program. If ownership cannot be proven, Stado records identity_unresolved, alerts once on the transition, and refuses to create a duplicate.
unreachable	Run the existing idempotent service ensure path. It creates a missing unit, restarts a matching definition, or performs a guarded definition reload. It checks the running postcondition and updates the registry when the host selected a different valid unit path.
not declared in the service directory	The unit has no endpoint to disprove, so the host channel is the evidence: the unit is probed on the box, a loaded unit must prove its live program before adoption, and only a unit the host itself reports absent is ensured.
unverified	Record endpoint_unverified, alert once on the transition, and make no change because endpoint absence was not proven.

Every repair renders its unit through the same resolution chain service ensure uses: the host's registry declaration, then the shipped Wisent catalog, then the declaration bundled with the build. A declaration that names only a unit path cannot be reinstalled from the document; the repair records declaration_incomplete and alerts until the registry entry carries its program and args, which is the durable fix — read the truth with stado service show <name>, write it into the entry, and every future repair renders from the document.

AutonomyMode::Report records the same plan without executing it. EnforceSafe and EnforceOwned execute the reversible repair actions, bounded by max_actions_per_tick, the emergency pause, the circuit breaker, and a per-service mutation lease. Only a mutation that failed on a host feeds the circuit breaker; declaration_incomplete and identity_unresolved are refusals computed before any host command runs, and a refusal must not starve the healthy repairs behind it. Recovery-managed units stay with the fixed host-recovery program and are never silently converted into registry services; the beacon exception asserts the unit without writing the registry for them.

Known gap: a timer-driven oneshot beacon (Linux host-health-beacon.timer → oneshot service) is a unit shape service ensure cannot yet express — it asserts a running unit, and a oneshot exits by design. Declaring it as a plain service would create a restart loop, so such hosts stay with their installed timer, whose oneshot publishes through stado host publish-beacon FILE.

Useful operator views:

stado service list
stado service verify
stado optimize status
stado optimize run

Every command below enters through Stado. Provider diagnostics belong inside the corresponding adapter and are unavailable unless that provider is explicitly enabled in the selected profile.

Storage root authority handoff

Inspect a durable handoff before choosing an action:

stado host storage-root-reconcile <target> --transaction <id> --phase status --json

The CLI and Desktop enter the same storage reconciliation report/outcome implementation. Desktop sends GET /api/host/storage-root-reconcile?target=<host>&transaction=<id>&phase=status for Status and POST to the same path for Run, Resume, Rollback or Finalize; neither method accepts a body. GET requires the independent storage-reconcile-read registry action and POST requires storage-reconcile-apply.

HTTP 200 means a product report exists, not that the requested phase succeeded: the response preserves exit_code, refusal and the complete report. Malformed, duplicate, empty or unknown query fields, a body, or the wrong method for a phase return 400. A missing or unauthorized action returns 401; unavailable verifier state returns 503. Failure before a product report exists also returns 503, with STORAGE_RECONCILIATION_FAILED and the actual error. Desktop captures the selected dashboard address and host when its sheet opens, shows the equivalent CLI command without launching it, and retains the HTTP status, raw API bytes, product exit, refusal and full report after errors.

Run, Resume, Rollback and Finalize first return accepted after the target-resident native worker owns the globally locked operation. Acceptance is not completion. Neither the API nor Desktop issues Status or another phase automatically; explicitly read Status for the same transaction ID after every accepted action.

Run and Resume automatically capture the queue and native-service state, acquire placement leases, pause and drain the queue, stop storage writers while retaining the transport and current runner, acquire the storage-write fence, and checkpoint both complete physical roots before constructing the additive union on fixed primary root A. Checkpoint and apply are internal stages, not operator-selectable phases.

The object API's captured loaded route at the fenced preflight pins one conflict winner for the whole transaction. If it serves A, A's bytes and metadata win every shared-path conflict and only B-only objects are imported. If it serves B, B wins shared-path conflicts and A-only objects remain in the union. The immutable checkpoint receipt records that source-selected authority; data application, post-commit correlation and the effective lifecycle snapshot must all agree with it.

Typed lifecycle handling examines records that become newly authoritative: B-only lifecycle records when A was serving, or A-only records when B was serving. This prevents an imported queued job, retained outcome, or live lock from bypassing the same classification that guards the rest of the handoff.

Before the data-commit boundary, Rollback is resumable and restores A to its exact physical checkpoint, including removal of transaction-created imports and now-empty directories, before restoring the captured route and native service state. After that boundary, Resume the same transaction to complete its recorded activation rather than starting a second transaction or reversing the winner.

Run or Resume owns the remaining core stages through declared-runtime activation, exact native-service and queue restoration, write-fence release and an activated_pending_lifecycle receipt. Ordinary lifecycle handling must then resolve the typed queued cancellations and retained outcomes. Select Finalize only after that cleanup; it validates the final typed observations and records complete, which a later explicit Status read proves.

Disk cleanup

A host's targets[].disk_cleanup declaration selects its mode, watermarks, per-pass budgets and cleaners. Each cleaner implements the eligibility rules for the state it owns. A local target with no policy is measured against DiskCleanupPolicy::reporting_default, which reports and never deletes.

"disk_cleanup": {
  "mode": "enforce",           // off | report | enforce; only enforce deletes
  "check_interval_seconds": 3600,
  "low_free_gb": 100,          // a pass does nothing above this much free space
  "target_free_gb": 200,       // and stops as soon as this much is free
  "max_items_per_pass": 500,
  "max_bytes_per_pass": 274877906944,
  "max_scan_items": 100000,    // ceiling 200000
  "max_pass_seconds": 300,     // optional; absent means the janitor's own 30
  "cleaners": {
    "build_caches": { "min_age_seconds": 86400, "root": "~/Documents/CodingProjects" }
  }
}

What a pass may delete

Each cleaner owns one kind of reclaimable state and refuses everything else:

Cleaner	Removes
build_caches	a directory whose own build tool declared it regenerable
huggingface_cache	model blobs no surviving snapshot references
weles_recordings	recordings admitted by the declared age and upload-proof policy
chromium_clones	the bundle clones macOS makes to validate Chromium at launch
queue_workdirs	job trees whose job is neither queued nor running
backup_twins	local replica objects the store it mirrors already holds
release_store	unneeded release payloads whose exact source has a completed publication record

build_caches is the one that answers for a developer host, and its rule is the Cache Directory Tagging Standard rather than any judgement of ours: a directory is eligible only when it contains a CACHEDIR.TAG whose first line is the standard signature, that file is a regular file owned by the same user, the directory's own mtime is older than min_age_seconds, it sits on the same device as the scan root, and it is reached without following a symlink. Cargo writes that file into every target/, and cmake and many other tools write it too, precisely so a cleaner may remove the directory without asking. A directory without the tag is never a candidate, which is why source trees, .git and node_modules are not touched.

Some roots are refused even when tagged, because a tag one level above them would let the youngest cleaner overrule an older one whose rules are stricter: the janitor's own state directory, ~/.cache/huggingface/hub, Cargo's package registry, and the declared weles_recordings and huggingface_cache roots.

release_store retains versions named by a host, a registry declaration, a configuration pin or an active or young pipeline run. It also keeps the newest versions selected by keep_newest and the newest installable release of each publication family. An absent active-run pin is not proof that publishing ended: the tag workflow can still be uploading outside the queue.

Before reclaiming any remaining version, Stado requires a readable source-revision.json and a completed or reconciled pipeline run for that exact source. source_identity_unverified and publication_completion_unverified explain the respective refusals in cleaners.release_store.skipped. A failed run is not sufficient evidence. installer_publication_untracked means an installer publication is not covered by the signed pipeline's completion record and remains retained. The CLI reports these reasons, and Desktop's Disk screen shows every reported cleaner with its exact skipped reasons and counts.

Eligible payloads are removed without deleting the version or platform source reservations. Cleanup therefore cannot make an already used version available to a different source revision. Tag-only releases remain retained until their publisher supplies completion evidence; disk pressure does not change that rule.

When a pass runs, and how far it gets

Pressure and explicit cleanup-to-target requests use bounded passes toward target_free_gb. An unfinished pressure pass keeps that goal after free space crosses the lower watermark. Ordinary non-pressure observations retain their interval gate. Item, byte, scan and time budgets still bound each pass; caps records which bound was reached.

The build-cache walker persists its remaining directory frontier and position inside the current directory. build_caches_resume_from names the next visit. The backup-twin cleaner likewise persists backup_twins_cursor, so retained objects at the start of a large replica cannot prevent later duplicates from being checked. Completed traversals clear their checkpoints; a changed policy or root invalidates them. Resuming changes scan order, never deletion authority. Every replica is checked against its primary again, and same_file_as_primary retains an object when both paths identify the same file.

Two writers run the same pass against the same policy, lock and state file: the disk-cleanup --watch unit on its own timer, and the queue agent on every tick. Every report names which one produced it in writer, together with the writer_version that produced it, because an operator reading a host's last outcome is otherwise reading whichever writer wrote last.

An outcome is one of never_run, invalid_or_unavailable_policy, lock_busy, interval_noop, healthy_noop, report_only, lock_recovery_report_only, blocked_running_jobs, reclaimed_target, reclaimed_progress, cap_reached, partial_error or no_eligible_items, and caps says which budget stopped the walk. lock_recovery_report_only means an overdue lock was replaced but its retired inode is still locked: Stado persists the scan and deletes nothing until the kernel confirms that predecessor is gone.

Operating it

stado space report <target> --json
stado space cleaners list <target> --json
stado space reclaim <target> --stage registry_cleanup --dry-run --json
stado disk-cleanup --once
stado disk-cleanup --dry-run
stado space reclaim <target> --apply --reason "reviewed cleanup" --json

The preview writes no cleanup state and deletes nothing. An applied remote reclamation records its reason and result on the affected host. space report retains the full recorded pass under cleanup_state.report, including skipped reasons and exhausted limits; directory sizes alone are not used to claim that cleanup can or cannot reach the target. The native Desktop API exposes the same declaration editor and reports. See Space for fields, guarded root overrides and refusal sentences.

install-disk-cleanup is also the repair. The unit is KeepAlive, so launchd restarts the process by itself, but a job whose spawn keeps failing writes nothing to its own log and is only visible in launchctl print: check state, runs and last exit code there before concluding the janitor is idle rather than refused. Re-running the installer rewrites the unit and bootstraps it again, which is the declared way to clear that state.

Isolated GitHub pre-check runners

The runner lifecycle enters only through Stado:

stado host precheck-runner install <registry-target>
stado host precheck-runner status <registry-target>
stado host precheck-runner remove <registry-target>

Stado resolves the host address and release_platform from the canonical registry. install exchanges GITHUB_TOKEN.value through Stado's admin-scoped Skarbiec coordinates for a short-lived organization registration token, transports the installer and token on host-channel stdin, verifies the pinned Actions Runner archive, and installs the OS service. It also makes the Brama host's Skarbiec reconcile missing routes from its live vault, resolves agent:probierz to the request-signing item and field selected by Skarbiec, reads that field on the host without putting its value in argv, and installs the resolved value through stdin as the runner-owned, mode-0600 .stado/kronika-agent-auth-secret; the non-secret probierz agent ID is published as $runner_root/routes/kronika-agent-id beside $runner_root/routes/brama.url. Kronika runs the audit, while Probierz is the Brama client identity authorizing that product workflow. remove uses a short-lived removal token before deleting the service, account, files, and network rule.

The runner has one unprivileged stado-precheck account and a root-owned pre/post-job cleanup hook. Workspaces, diagnostics, package and toolchain caches, application caches, and .stado are runner-owned; the rest of the installation is root-owned and not writable by jobs. An nftables UID rule on Linux and a PF user rule on macOS reject loopback, RFC1918, link-local, unique-local, and CGNAT/Tailscale ranges while leaving public GitHub and package endpoints reachable. The one exception is the exact loopback Brama port published by Stado for the authorized kronika consumer. Those CIDRs are protocol network classes compiled into Stado, not fleet host addresses; fleet destinations remain registry data.

GitHub runner group stado-precheck grants access to an explicit repository list. Stado keeps public-repository admission disabled until stado host precheck-runner repository-add <repository> admits a named public repository; those workflows must refuse pull requests whose head repository differs from the base repository before GitHub assigns the job to this runner. Eligible Linux jobs use runs-on: [self-hosted, Linux, X64, stado-precheck]; eligible macOS jobs use runs-on: [self-hosted, macOS, ARM64, stado-precheck]. Repository access is the GitHub-side boundary: workflow-ref restrictions remain disabled because same-repository pull-request jobs execute from refs/pull/*, not the default branch.

Common queries

Fleet, queue, quota, and billing

stado overview
stado overview --json

overview resolves the configured Stado backend and enabled adapters. It does not fall back to a provider CLI, ADC, or a different storage backend.

Local agent state

stado host health <registry-target>
stado host inventory <registry-target>
stado host exec <registry-target> -- nvidia-smi

Inspect one job end-to-end

stado machine status <job_id>
stado machine logs <job_id> --cursor 0 --limit 1048576

Failure mode quick-grep

The most common failure modes — search the per-job stdout for one of these substrings to classify failures fast:

Substring	Cause
HfHubHTTPError: 429	HF Hub rate limit (free tier 1000 req / 5 min). Retry path lives in wisent.core.utils.infra_tools.infra.data.dataset_splits.get_all_docs_from_task and the cache fast-path in generate_pairs_from_task.py.
Couldn't find cache for	datasets cache miss for a config that doesn't exist on the dataset's HF repo. lm-eval task config drift.
OverflowError: int too big to convert	tokenizer model_max_length was a sentinel (1e30) handed to the rust binding's u32. Capped at 4096 in activations_collector.py.
Dataset scripts are no longer supported	datasets 4.x dropped the script loader. Pinned datasets<4.0 in the agent template.
huggingface-hub>=0.34.0,<1.0 is required	transformers 4.55.x dep-pin mismatch. Pinned huggingface-hub<1.0 in the agent template.
RuntimeError: Cannot set NUMBA_NUM_THREADS	numba init happened before wisent's env-set. Set NUMBA_NUM_THREADS=1 in the agent's env BEFORE Python starts.
does not appear to have files named	transformers shard-name miscompute on gpt_oss / 0-indexed safetensors. Fixed in transformers>=4.57.
AttributeError: ... has no attribute 'transformer'	wisent activation hook expected GPT-2 path on a model whose model_type contains gpt; gpt_oss uses Llama-style. Fixed in transformer_analysis.py.
gated repo / 401 Client Error	The scoped workload credential cannot read the requested repository. Rotate stado-huggingface/token through the stdin-only Skarbiec service path; never place the token in VM metadata or logs.
Quota 'PREEMPTIBLE_NVIDIA_*_GPUS' exceeded	Hit the regional preemptible quota. Either raise via GCP console or add zones to MACHINE_TYPE_ZONES.

Recover the Stado binary on a host

Use this break-glass path when the target's Stado binary or resolver is missing or broken, so the ordinary declaration-driven delivery cannot run. This is the single canonical procedure; do not replace any part of it with a manual copy or SSH session:

stado host recover <target> --release <exact-version>

The command uses the last valid canonical registry snapshot. Add --bundled-registry only when that authority is unavailable and you explicitly intend to use the registry bundled into the local Stado build. It then:

1. restores the registry-selected release object API before the first release baseline or catalog read and reports object_api.status plus its detail;
2. downloads the exact canonical signed Stado artifact without invoking the local resolver or a Stado binary on the target;
3. verifies the release manifest, signature, and SHA-256 before activation;
4. preserves the previous binary and installs the verified binary by atomic rename, so the active path is never half-written;
5. runs the newly installed stado resolver --help on the target;
6. atomically restores the prior binary when that probe fails, or removes the invalid install when no prior binary existed; and
7. only after a successful probe runs the existing host recovery.

The JSON answer records the object API health boundary and download, verify, backup, install, resolver, and recovery step results, plus rollback when restoration was required. Keep that one answer with the incident record.

stado host release <target> --binary stado --version <exact-version> is the routine path for a healthy fleet: it carries out the registry's managed-binary declaration. host recover --release instead bootstraps Stado itself without the resolver or remote Stado binary that may be broken. Omitting --release keeps the original recovery behavior and changes no binary.

Release / publishing

Release operations enter through Stado:

stado release catalog sync --root /path/to/registered-checkouts
stado release catalog audit
stado release submit --source /path/to/product --version <exact-version> \
  --channel candidate

Submit requires a clean committed tree but does not contact its remote. It archives the exact committed tree, publishes the create-only source object, records source and manifest identity in the Stado catalog, and creates one provider-neutral queue job pinned to a registry builder whose release_platform matches the recipe. Queue state and status/<job>/output/ remain the authoritative work and transport records.

Inspect or resume a run by repeating the same submit command. Its identity is derived from product, version, channel, source digest, and manifest digest; the durable stado://release-runs/<id>/run.json shows job IDs, output coordinates, delivery state, and failure. A terminal successful platform output is read from JobStorage and published, never rebuilt.

The release authority is configured by item name and trusted key ID:

release.signing_key_item = stado-release-signing
release.signing_key_id   = stado-release-2026-08

The Skarbiec key-pair item contains the base64 PKCS#8 value in private_key. Key bytes stay in Skarbiec. Build and delivery secrets are only checked-in item#field references and must also be permitted by agent.skarbiec.secret_fields.

Canonical publication is immutable and ordered:

release.tar.gz -> qualification.json -> release.sig -> release.json

The signed manifest is the commit marker. Required delivery jobs run only after it exists and consume its exact URI and digest. Optional mirrors do not gate canonical success. Runtime products then use the existing registry.release_control generation CAS and release-agent reconciliation. deployment.json is written only after every declared target reports the promoted version, artifact digest, and manifest digest exactly.

No Git forge credential, hosted workflow, provider repository checkout, or direct provider API is required by this chain. Existing external adapters may mirror completed releases, but their availability cannot change source, qualification, signing, desired state, or observed rollout truth. See release.md for the strict manifest and catalog contract.

Stado repository tag delivery

The Stado repository's .github/workflows/deploy.yml is a separate adapter for its versioned installer coordinates, not the signed release submit run above. A stable tag must first pass the exact-source Version check and reserve its version and platform identities.

The adapter publishes and reads back the native coordinate, then reads .wisent-release.json from that same source revision and converges every required native target with host declare-version and service converge <target> stado --apply --json. It does not choose those targets from a separate coordinator variable.

Only after all required native deliveries succeed does stable Linux publication start. Its resolver can therefore use the updated native registry reader. A Linux failure cannot prevent a compatible Mac mini from receiving the already-qualified native release. The final fleet job handles the required Linux targets rather than repeating native delivery. Candidate Linux publication retains its independent path.

The native publication, native delivery, Linux publication and Linux delivery keep separate results. One successful platform is not a complete fleet release; a refused required target or missing platform still makes the adapter fail. Its artifacts also do not supply the signed pipeline completion evidence required by release_store cleanup.

Bringing up a new local box

# Install a verified Rust release and resolved profile first.
./install-stado.sh
export STADO_CONFIG="$HOME/.stado/config.json"
export STADO_TARGET="<registry-target>"

# The provider-neutral installer validates config and preflight, then delegates
# persistent launchd/systemd ownership to Rust bootstrap.
./install.sh

# Health publication additionally requires the dedicated
# stado-host-health-beacon Skarbiec grant and non-secret Stado/Skarbiec origins
# described above. It never requires a cloud login.
stado host health "$STADO_TARGET"