On this page
Space
space is the fleet capability for understanding and reclaiming storage on a declared local target. Cleanup itself follows one built-in rule, the same on every host, and nothing about it is declared per host:
When the volume holding the host's home is at least 80% used, the janitor deletes everything the fleet put on the host. Below 80% it deletes nothing.
What it keeps is what is not the fleet's to take: ~/.ssh and everything outside a cleaner's own area (the user's data), a backup copy whose primary cannot be proven identical (it is then the only copy), the release versions the registry declares or the host runs, and the work tree of a job that is still running.
Invocations
stado space report TARGET [--json]- Reads one combined report from the declared target: the volume and the rule's verdict on it, memory readings, inventory, build-cache verdicts, the janitor lock and the janitor's last pass.
- The text report prints the rule on one line:
rule: delete everything the fleet put here at 80% used; volume 96.0% used (triggered), or(not triggered), orvolume unreadable. --jsonemits the report as a JSON object; the rule is underrule(full_percent,used_percent,triggered) and again undercoverage.rule.
stado space reclaim TARGET [--stage STAGE]... [--dry-run|--apply] [--reason TEXT] [--json]- With no
--stage, reads and selects every stage in the compiled declaration. Repeat--stageto select a subset. - Omitting both mode flags is a dry run.
--dry-runstates that choice explicitly.--applyperforms deletion and requires a non-empty--reason. - An applied fleet run appends its reason, actor, stage outcomes and freed-space readings to
.stado/audit/host-reclaim.jsonlon the target. - No stage takes a path on one look, and none judges a path by its age. A candidate is taken only when no live process names it,
lsofshows no process with it open or as its working directory, and an earlier--applyfound it the same way with the same modification time. The first apply that finds a candidate unheld records it under~/.stado/work/host-reclaim-local-evidence/settled/and keeps it, listing it in the stage'srefusedasunheld on this look; the next apply takes it if no process holds it and it is unchanged; a candidate whose ownership cannot be read is kept asprocess ownership could not be read; retained. A dry run records nothing, so it lists exactly what the next apply would take. The newest tree of every product and whatcurrentresolves to are kept regardless. registry_cleanupruns one janitor pass on the target (stado space reclaim --local --apply --reason 'fleet space reclaim', or--local --dry-runin a preview). Its row carries the janitor's own account indetail: the outcome and each cleaner's scanned, eligible and deleted counts. On a volume under 80% the pass answershealthy_noopand runs no cleaner.- The receipt carries the janitor's complete report under
janitor, in the shapespace reportpublishes undercleanup_state.report. The key is absent when no janitor ran.
- With no
stado space reclaim --local [--stage registry_cleanup] [--dry-run|--apply] [--reason TEXT]- Runs the declared janitor on this machine; preview is the default.
--applyneeds a non-empty reason. Other stages require a fleet target. The host's standing health watch runs the same rule at its declared cadence.
- Runs the declared janitor on this machine; preview is the default.
stado space file remove TARGET PATH [--json]- Removes one regular file only from a Stado-managed area on the declared target.
stado space file retire TARGET PATH --product PRODUCT [--dry-run] [--transaction ID] [--expected-sha256 SHA256] [--expected-size BYTES] [--expected-mode MODE] [--json]- Archives an obsolete executable or launchd declaration without deleting its bytes. Optional expected values are preconditions.
--dry-runreports the guarded move without performing it.
- Archives an obsolete executable or launchd declaration without deleting its bytes. Optional expected values are preconditions.
stado space relocate TARGET --namespace NAMESPACE --from-prefix PREFIX [--to-prefix PREFIX] [--store-root PATH] [--dry-run|--apply] [--limit COUNT] [--json]- Relocates object-store keys on the target holding their bytes. The default is a dry run;
--applycommits the relocation.--limit 0means no item limit.
- Relocates object-store keys on the target holding their bytes. The default is a dry run;
stado space volume mount TARGET --device NAME --mount-point PATH [--json]- Mounts one block device the host has attached (
--deviceis the/devleaf, such assdb1) at an absolute directory and writes its/etc/fstabline by UUID withnofail, marked# stado-volume, after copying the existing fstab beside itself. A device with no filesystem, an LVM member, a LUKS container or swap is refused by name: this command mounts and never formats. A device already mounted elsewhere, an fstab line that already names the UUID at another path, and a mount point under a system tree (/boot,/etc,/usr,/var, …) are refused before anything is written. A device already mounted at the requested path reportsalready_mounted. --jsonemitsstatus(mounted,already_mounted,refused), thevolumeobject (device, fstype, uuid, size, the fstab line and whether it was written, the mount output, and thedf -Pkreading of the mount point) anderror.
- Mounts one block device the host has attached (
stado space volume unmount TARGET --mount-point PATH [--json]- Unmounts the filesystem at the mount point and removes the
# stado-volumefstab linevolume mountwrote for it, after copying fstab beside itself; the data on the disk is untouched. A busy filesystem is refused withumount's own words and fstab is left as it was. An fstab line Stado did not write for that mount point is refused by name. An/etc/fstabthat exists but cannot be read is refused withawk's words before anything is unmounted, since whether another line declares the mount point is then unknown; a host without/etc/fstabdeclares nothing there. A failed backup copy is refused withcp's words, and asystemctl daemon-reloadthat fails after the line was removed is reported with its own words. A mount point neither mounted nor declared reportsnot_mountedand changes nothing. --jsonemitsstatus(unmounted,not_mounted,refused), thevolumeobject (mount_point,unmounted_now,fstab_removed) anderror.
- Unmounts the filesystem at the mount point and removes the
stado space work-root TARGET [--path PATH] [--json]- With no
--path, prints where TARGET's agent keeps the fleet's work: the declaredtargets[].work_root, or the agent's home when none is declared (~/.stado/work/jobsfor job trees,~/.stado/build-cachefor build caches). - With
--path, creates the directory on the host owned by the account the host'scom.wisent.stadounit runs as (systemctl show -p User com.wisent.stado.service, the invoking account when the unit names none), mode 0700, and refuses before writing the registry when the path lands on the root volume, on a filesystem that is not device-backed, or on a path under a system tree. Asystemctlthat cannot read the unit's account, or adfthat cannot read the path's or the root's filesystem, is refused with that command's own words rather than read as the invoking account or as no device. Only then istargets[].work_rootwritten through the canonical registry's compare-and-swap. The agent uses the new root once its unit restarts it; until then it logsthe canonical registry now declares work root …every tick. --jsonemits the generation, the owner, the device and mount point the directory landed on, its free KiB, the previous declaration, andtakes_effect.
- With no
The rule
The volume is the one holding $HOME, read with statvfs, and — when the host declares targets[].work_root on another volume — that volume too, because job trees and build caches fill it. Of the two, the one with less room before the threshold decides. A pass the agent runs knows its declared root; stado space reclaim --local knows it when STADO_WORK_ROOT is set. On macOS the home's volume is the APFS Data volume, whose usage is the container's — not the sealed system snapshot mounted at /, which can read 56% used while the Data volume is at 96%. Used percent is computed as df computes it, from the space an unprivileged writer may still take.
Below 80% a pass reads the volume, writes outcome: healthy_noop and stops: it reads no registry, takes no lock and opens no directory. At or above 80% every cleaner runs over its whole area with no age, count, byte or scan limit, and nothing it finds is kept for being young or numerous:
| Cleaner | What it takes | What stays |
|---|---|---|
agent_logs |
Every file under the harness log directories ~/.omp/logs, ~/.claude/debug, ~/.codex/log, ~/.factory/logs, ~/.kimi-code/logs |
— |
backup_twins |
Same-disk replica objects under ~/.stado/local-backup whose primary under ~/.stado/local-storage has the same size and content hash |
a replica whose primary is absent, differs, cannot be read, or is the same file |
build_caches |
Every directory carrying a build tool's own CACHEDIR.TAG anywhere under the home |
a tree whose size cannot be measured completely; the macOS privacy-protected folders listed below |
chromium_clones |
The operating system's per-launch code-signing clones in the account's temporary container | the newest clone, and any clone a live process names |
compiler_cache |
Every file of the compiler cache's store, ~/Library/Caches/kache on macOS and ~/.cache/kache on Linux |
everything, while a job is running (job_running), because a running build reads and writes it |
delivered_releases |
Release copies delivery staged under ~/.stado/releases/<product>/<version>/ (the attestation copy and retained archive of every version delivered here) |
for each product, the version ~/.stado/bin/<product>.release-version names, the version whose staged copy is byte for byte the installed ~/.stado/bin/<product> (what stado release version show attests against), and the newest version by modification time (installed_or_newest); every version of a product whose coordinate, installed binary or a staged copy exists and cannot be read (installed_release_unreadable, with the path and OS cause among the cleaner's errors) |
huggingface_cache |
Model blobs under ~/.cache/huggingface/hub |
everything, while a job is running (blocked_running_jobs) |
queue_workdirs |
Work trees under ~/.stado/work/jobs (and the declared work root) of jobs the queue reports terminal |
the tree of a running job |
job_outputs |
Payload files under status/<job>/output/ of jobs the queue lists as terminal, in ~/.stado/local-storage |
each job's own top-level receipts and logs; jobs the queue lists as live |
local_snapshots |
Every local Time Machine snapshot of the volume the home lives on (local_snapshot_volume in the report) |
com.apple.os.update-*, the operating system's recovery state; snapshots of every other volume |
object_evidence |
Product run evidence under ~/.stado/local-storage/ecosystem/probierz/runs and dispatched results under ~/.stado/local-storage/ecosystem/probierz/results, read without following symbolic links |
Stado's own release records directly under runs — release runs (release-pipeline), build records (build) and change batches (release-changes) — and native-signing inputs beneath either evidence root, reported as stado_record_or_pinned_input_kept; every symbolic link under either root, kept as a link and never walked (not_a_regular_file); everything, when a link or file stands in any component of a root's path (root_not_a_directory, also named with that component's path and the operating system's refusal among the pass's errors, so a root nothing is cleaned under is told apart from an absent one). A root, directory, entry or file it cannot read or remove is skipped and named with its path and cause among the pass's errors (root_unreadable, directory_unreadable, unreadable, deletion_refused) |
release_store |
Published release versions under ~/.stado/local-storage/ecosystem/releases |
versions the registry declares or config pins, versions this host has installed, versions an in-flight run uses, the newest complete signed release (newest_signed_release_kept), a version whose source identity or completed publication cannot be verified (source_identity_unverified, publication_completion_unverified); everything, when the registry cannot be read (registry_unreadable) |
weles_recordings |
Weles run recordings, uploaded or not, under ~/weles/recordings or the host's weles.recordings_dir |
— |
~/.ssh is in no cleaner's area. Nothing outside those areas is the fleet's, so nothing outside them is opened for deletion. The object-evidence cleaner descends from the selected home to each evidence root by opening every directory component without following links, then scans and removes entries relative to held directory descriptors; replacing a path with a link between inspection and deletion cannot redirect the pass into the user's files. Protected release-record and pinned-input directories are kept as opaque subtrees: the cleaner does not open their contents, and stado_record_or_pinned_input_kept counts each skipped subtree rather than its files.
The same pass empties every service log under ~/.stado/logs (.log, .out, .err) that this account owns, in place, so a unit still writing to one keeps its file. A log is never touched below 80%; a symlink, a hard-linked file or another account's file is left alone. Each emptied log is reported as service log emptied file=<name> bytes_before=<n>, and a failure as service log emptying failed file=<name>: <cause>.
Every error a pass meets is kept in its report, however many there are. Walks and deletions follow a tree as deep as it goes; a tree deeper than the process may hold open reports the operating system's own refusal.
A pass ends with one outcome:
outcome |
Meaning |
|---|---|
healthy_noop |
The volume is under 80%; nothing was opened. |
reclaimed_below_threshold |
The pass deleted and the volume is under 80% again. |
still_full |
The pass deleted what the fleet put there; what remains is the user's. |
no_eligible_items |
The volume is full and the fleet holds nothing there. |
blocked_running_jobs |
The volume is full and the Hugging Face cache waited for running jobs. |
partial_error |
A cleaner failed; errors names the operation and cause, and the other cleaners still ran. |
report_only |
A --dry-run pass: counted, deleted nothing. |
volume_unreadable |
statvfs failed; the rule could not be applied. |
lock_busy, lock_busy_workloads, lock_busy_unattributed |
Another holder had the lock; the holder is named. |
lock_recovery_report_only |
A retired kernel lock is still held or unreadable; nothing was scanned or deleted. |
What the agent does at the threshold
While the volume the agent writes to is at least 80% used, it claims no new work and publishes accepting_jobs: false with admission_reason: disk_pressure_active, because the jobs are what fill the disk. It keeps publishing capacity every loop with disk_used_percent and disk_pressure_active in the diagnostics. A volume it cannot read publishes disk_unreadable and refuses work until it can. stado host gates TARGET reports the reading as used_percent, headroom_gb (what may still be written before the rule deletes; negative past it), full_percent and full. stado host gates TARGET --require-disk exits 0 only when the volume is under the threshold.
Two kinds of job it still claims under pressure: the newest signed Stado release delivery pinned to it, which repairs the agent that owns this gate, and a job pinned to it that stages nothing on its disk (no repository, no packages, no pre-command, no output mirror), such as an Oko routine reading the host's own transcripts. Refusing those would free nothing and, on a host whose disk the user's own data keeps above the rule, would stop the host's upkeep for good. The agent publishes how many it admitted as disk_pressure_recovery_jobs and disk_pressure_in_place_jobs.
A full volume also writes disk-cleanup-turn.json beside the janitor lock, naming the process that asked. While that process lives no new workload takes a hold, so running jobs drain and the next pass gets the exclusive lock; a request whose process has exited is ignored. A --dry-run pass never writes it. Without it, a host that builds continuously never gives its janitor the lock.
Locks
The kernel lock, not the age of its owner record, decides whether a pass may run. A live hold returns lock_busy, lock_busy_workloads or lock_busy_unattributed, naming the recorded holder where available. Stado does not replace a live lock because it is old. A still-held retired lock prevents scanning and deletion with lock_recovery_report_only; an unreadable retired lock reports its actual error. A running workload holds the same lock in shared mode and writes one record beside it (~/.cache/wisent-compute/disk-cleanup-holds/<pid>-<id>.json). The private pass records each blocked workload in workload_holds with pid, holder and acquired_at (Unix seconds), also named in its diagnostic. This is the pass's observation, not proof the process still holds the lock when a later reader opens it.
While a standalone pass holds the lock, the agent publishes admission_reason: cleanup_in_progress and stado host gates reports it as a blocker; a lock the agent cannot probe publishes cleanup_lock_error with the error code. stado space report TARGET names the holder in cleanup_lock.holders.
stado product editor serve answers existing compiler settings without a lifetime cleanup hold. Builds, indexing and restore operations remain protected. An installation that finds a pass holding the lock exclusively waits for it: its stderr carries a czekam: lock wait naming the janitor's cleanup pass and the recorded holders, stado waits lists the open wait, and the operation takes its shared hold and runs once the pass lets go (koniec czekania). A wait that would close a wait cycle is refused naming the lock and the cycle. An installation is refused before export or compilation only while the janitor has asked running work for its turn; that refusal (… the janitor has asked running work for its turn; nothing was exported or built. It waits on: …) names recorded shared holders and their acquisition time, the janitor's turn request (process, request time and reason), and its recorded exclusive owner where available. A turn request or lock that cannot be read refuses the operation with the record's path and cause; absent records say ownership was not established, not that no process holds the lock. An already running older binary retains its original lock until that process ends; installing a new executable does not revoke kernel ownership.
The janitor's files
The janitor keeps its files in ~/.cache/wisent-compute: the run lock disk-cleanup.lock, its holder records disk-cleanup.lock.holder and disk-cleanup.lock.holder.inode.<device>.<inode>, retired lock inodes disk-cleanup.lock.retired.<name>, the state file disk-cleanup-state.json and its transaction lock disk-cleanup-state.lock, the workload holds under disk-cleanup-holds/ and the turn request disk-cleanup-turn.json. These names are shared by every Stado release: a build still running from the release a host is being moved off holds the same lock and writes its hold under the same directory while the new release's janitor runs, so the janitor sees that build and keeps what it is writing. The space reclaim engine keeps these names for that reason. The state file carries "version": 1, which every release reads exactly; a new field is added as a key rather than a new version, so an older reader still reads the file.
The janitor's promise
A janitor that runs on a period — the agent's pass every poll or the host's health watch at its declared cadence — states in its state file, under promises.<writer>, when its next pass will have written that file, and which process made the statement (pid). The promise is the pass's end plus the period, the longest pass on record for that writer on this host (longest_pass_seconds), and the longest it has been measured to start late (longest_lateness_seconds: the time between one pass's end and the next one's start beyond the period, spent writing the file, printing and, for the agent, sweeping scratch and restoring reconcilers). Lateness is measured only between two passes of one process, so a stopped service is not counted as late, and a writer's first pass on a host has none on record. A single space reclaim --local --apply pass promises nothing and leaves standing promises as they were.
stado space report TARGET shows every promise under cleanup_state.promises (writer, next_pass_by, pid, and alive: whether that process is among the host's live stado processes), the furthest as cleanup_state.next_pass_by and cleanup_state.promised_by, the live processes as cleanup_state.live_stado_pids (null when the host could not list them), and cleanup_state.prevented (whether the last pass was turned away by a workload's hold on the lock).
stado host gates TARGET counts the janitor as running while any promise holds or any process that made one is alive: a pass slower than any before it overruns its promise without the janitor having stopped, and a process being replaced is briefly gone while its promise still holds. It reports space_reclaim_stalled when the janitor is not running by that measure — including a state file from a janitor older than the promise, which states none — when no pass has ever succeeded, or when the last pass ran and did not succeed. It reports space_reclaim_lock_held while the janitor runs and its last pass was turned away by the lock. Each is a blocker on a full volume and a note otherwise.
Locations a pass never opens
On macOS the build-cache walk refuses ~/Pictures, ~/Music, ~/Movies, ~/.Trash, ~/Library/Mobile Documents, ~/Library/CloudStorage, ~/Library/Group Containers, ~/Library/Containers and ~/Library/Developer/CoreSimulator before opening them, and counts each refusal as privacy_protected in that cleaner's skipped map. Media and cloud roots may trigger privacy or file-provider reads; containers and simulator roots hold other applications' state. A tagged build cache inside a protected root is kept rather than stalling the reclaim lock or waiting on consent. stado space report prunes the same roots.
~/Documents, ~/Desktop and ~/Downloads are consent-gated as well and stay in scope, because real build trees live in them. The first open under each runs on a helper thread; the pass records consent_pending, skips that folder and walks the rest of the home. A tagged directory that contains a refused root is never deleted.
Nothing Stado builds is left in those folders when a background process cannot read them. When one refuses the janitor with Operation not permitted and Cargo has no configuration under its home ($CARGO_HOME, else ~/.cargo), the pass writes config.toml there declaring [build] build-dir = "{cargo-cache-home}/build/{workspace-path-hash}", so every checkout's intermediate build output lands under Cargo's home, which Cargo tags and this cleaner reclaims; it records cargo_build_dir_placed in the cleaner's skipped map, cargo_build_dir_declared when a configuration already declares a build directory and cargo_config_without_build_dir when one exists without it, which it leaves as it is. stado product install|update keeps every run, scratch tree and desktop .build in the checkout's build area under ~/.stado/products/builds (see Product catalog).
A build cache is measured completely before it is deleted. An unreadable entry, changed directory or different filesystem keeps that cache and records cannot measure cache <name>: with the operation and cause in errors. When macOS answers Operation not permitted for a whole folder, the error says the operating system's privacy protection refuses it, to grant Full Disk Access to the agent that runs the janitor, or to reclaim the checkouts' rebuildable trees with stado product build-trees remove from a terminal macOS lets read the folder. That command reads and removes, from the process that runs it, every tree in the workspace's checkouts that a build tool tagged or Stado declared rebuildable, and keeps any tree a build holds (see Product catalog). stado space reclaim <target> --stage tagged_build_caches --apply --reason … runs the same eviction from the operator's command, which holds that access, and skips any tree a live process names.
The walk is incremental, backgrounded and admitted by load
The home is not listed again on every pass. The janitor keeps, beside its state (~/.cache/wisent-compute/build-caches.json), the child directories of every directory it examined and the measurement of every tagged cache it found, each under the directory's device, inode and modification time. A directory adds or loses an entry only by changing its own time, so a directory whose three are what they were has the child directories it had: the pass takes them from memory and does not list it or read the kind of its files, while each child is still opened, checked for owner and device and read for its tag as before. A new tree anywhere changes the time of the directory it appears in, which is listed again. The pass counts the child directories it took from memory under listing_remembered in the cleaner's skipped map. A tagged cache whose root is what it was reports its remembered bytes (measurement_remembered, with the bytes under skipped_bytes) instead of reading every file beneath it; a file growing inside an existing subdirectory does not change the root's time, so those bytes are the last measurement's until the root changes or the cache is removed. Deletion never reads the memory: an enforcing pass removes the tree and measures the volume before and after. A completed pass rewrites the memory only when it listed or measured something new or found something gone; a pass that stopped early leaves the earlier memory. A memory that cannot be read is an error in errors, and the pass walks without it.
The walk runs in the host's background band: on macOS the thread takes PRIO_DARWIN_BG, which throttles its disk I/O and its CPU share as taskpolicy -b does for a process; on Linux it takes the lowest nice value the system declares, from which the I/O scheduler derives its priority. A system that refuses the change leaves the walk at its previous priority and records priority_not_lowered.
The census space report prints beside the verdicts (build_caches.entries, tagged_build_output) is the same walk run as a shell program on the host, with find and du placed in that band by the host's own words: /usr/sbin/taskpolicy -b on macOS, ionice -c idle nice on Linux. The program keeps as a tag only a CACHEDIR.TAG path under the scanned root; a line the walk printed about itself — the shell refusing a program, a find that did not start — is the walk's failure, printed on its error stream and never read as a directory, so a census row is always a directory under the root with its size in KiB or -. Real test: tests/space_reclaim/space/census.rs.
Every walk of these programs sends its errors to /dev/null, so a band program the host does not have would empty them all without a word. Each program therefore checks the band program first and, when the host lacks it, ends before any walk with the background band program <program> is not on this host, so no walk can run on its error stream; space report, the janitor's build-cache pass and the host reads that share the band fail with that sentence instead of reporting an empty disk.
A janitor pass walks only when the volume is at the rule's full threshold, and there the walk is the relief: it runs whatever the host's load, at background priority, and its reading clears any standing skip under scans.skipped.build_caches. Skipping it for load would leave a host that its own work keeps above the ceiling full for good, its reclaimable build caches untouched and every pass ending blocked_running_jobs. A preview (space reclaim without --apply) is admitted against the host's load ceiling: when the one-minute load average is above host.scan_load_ceiling (WC_HOST_SCAN_LOAD_CEILING), or above the host's logical core count when nothing is declared, it records load_above_ceiling and an error naming the load, the ceiling and its source, walks nothing, and the host's beacon carries the skip under scans.skipped.build_caches until a later walk is admitted. The same ceiling governs the beacon's launchd event reads and the inventory walk of space report; see Beacon.
What a deletion is worth on a Mac
macOS takes a local APFS snapshot every hour, and a snapshot pins every block the volume held when it was taken, so deleting a file frees nothing until the snapshots referencing it are gone. local_snapshots therefore runs after every other cleaner in the same pass and deletes every Time Machine snapshot of the volume the pass's home lives on, read with df -P <home> and listed with tmutil listlocalsnapshots <that volume>; the report's local_snapshot_volume names it. A home on its own volume, such as the disk image every test journey runs on, has no snapshots, so a test pass never touches the machine's own backups. A volume that cannot be read touches no snapshot and is reported as volume_unreadable with the reason. com.apple.os.update-* snapshots are counted and kept. macOS publishes no per-snapshot size, so the cleaner reports what the volume measured across it in actual_free_delta_bytes.
Declarations read
The stage vocabulary of space reclaim is compiled from stado-rs/data/fleet/space.json. The reclaim_stages array declares, in execution order:
registry_cleanup— one janitor pass of the rulebuild_scratchqueue_workdirsforeign_home_treesdelivered_treesdelivery_leftoversrebuildable_cachestagged_build_caches— everyCACHEDIR.TAGtree under the home, six levels deep, no process naming itchromium_cloneslocal_apfs_snapshots— every local Time Machine snapshotrunner_work_trees
Both space report and space reclaim read that same array. A new stage is a declaration row rather than a new command or match arm.
delivery_leftovers sweeps what every stado release local install leaves on the host: under ~/.stado/releases/<product>/<version>/<platform>/ the attestation copy and retained archive of each delivered version, and under ~/.stado/bin the backups of replaced binaries. It keeps the version the installed coordinate names, every version whose attestation copy is byte-identical to the installed binary, and the most recent delivery; the newest backup of each name survives, and <name>.previous is never taken.
Target identity, transport and platform come from the canonical registry target. The registry declares nothing about space reclaim. A stored generation with retired memory settings still parses and runs the rule; an ordinary registry write removes those settings together with retired worker-count fields. Reading alone does not rewrite the stored document.
Build-cache entries in the report preserve these verdicts: no-cache-tags when the walk found no cache tag, root-protected when the home itself is tagged, permission-denied for a directory the walk could not open (the walk goes on), and scan-failed when find failed with another error and produced no tag at all, or when the walk printed a status line without its path or size — that entry's path carries the line, and the census is then not read to its end rather than silently missing a cache. A failed walk returns its observed result without replaying the scan; there is no scan-attempt budget. Permission-denied entries mean those directories were not inspected, even when no tags were found elsewhere. The block is build_caches.scan (source, root), entries[] and error. CLI and Desktop Space retain the same entries and command receipt. The real census journey in tests/space_reclaim/space/census.rs checks a tagged cache and a directory refused by the filesystem, retaining the report and verifying that cache bytes remain unchanged.
Any permission-denied row makes tagged_build_output_read false, including when readable caches were found elsewhere. Their measured rows remain available, but do not establish completeness. Desktop Space renders that value as a warning that the census was not read to its end; the JSON and command receipt retain the refused directory.
What the report answers
inventory[].bytes preserves the host's du count in bytes; size_gb is its rounded display value. The attribution walk reaches two levels below the home on macOS and three on Linux. Every directory carrying a CACHEDIR.TAG anywhere under the home is measured beside it (tagged_build_output), and a tagged tree inside another is reported once, as the outermost. space report walks the home for tags once: the build-cache verdict's walk sizes every tagged directory it finds, judged or not, and those sizes are the census, which tagged_build_output_source names (the build_caches verdict's walk of the home); tagged_build_output_read is true when that walk ran to its end and false when it timed out, failed, found its root absent or never reached the host. A full disk read made without the verdict beside it runs its own census walk.
The coverage object contains:
rule: the threshold, the volume's used percent and whether the rule is triggered;headroom_bytes: what may still be written before the volume reaches 80% (negative past it).cleaner_scopes: each cleaner's area on this host and its measured size.covered: the stage roots for this platform, with their measured byte counts ormeasured: false.uncovered: every measured row outside the stage roots.mechanismnames the cleaner whose area contains the row; a row with no mechanism is the user's and the rule never takes it.exclusive_of_measured_childrenmarks a parent counted without its separately measured children.uncovered_rowsis their count.covered_bytes,cleaner_bytesandunswept_bytes: non-overlapping totals of the rows in stage roots, in cleaner areas outside them, and outside both.reclaimable_bytes: null. Coverage alone cannot establish recoverable capacity.verdictanddetail:
verdict |
Meaning |
|---|---|
below_threshold |
Under 80%; detail says how much more may be written before the janitor deletes. |
full |
At or past 80%; detail says how far past, how much of the measured inventory is in the fleet's areas and how much is the user's. |
unmeasured |
The volume reading is unavailable, so the rule's verdict cannot be given. |
janitor: the last recorded pass — itsoutcome, a one-linedetail, and the completereportwith per-cleaner scanned, eligible and deleted counts, exact skipped reasons, errors, writer identity and the pass's own free-space readings. Missing pass evidence is not replaced with zero counters.
Where the fleet writes: the work root
Every local host keeps the fleet's work in one directory: job trees under <work root>/jobs, the persistent Cargo build cache under <work root>/build-cache, and the free space the agent publishes and admits work against is measured on that directory's volume. Undeclared, the work root is the agent's home. targets[].work_root declares another: an absolute directory on a mounted data volume, written by stado space work-root TARGET --path PATH and shown by the same command without --path, by the registry projection Stado Desktop reads, and by Hosts → Space.
The agent reads the declaration from its own registry target on every tick; every job it starts inherits it as STADO_WORK_ROOT. A declaration that changes under a running agent with jobs is logged, not followed; once no job is running the agent replaces its own process with the same binary and arguments and starts on the new root. queue_workdirs sweeps the declared root's job trees as well as the home's.
Volumes and attached disks
The disk: line and the usage object measure the volume under the agent's home. A host can hold most of its storage elsewhere, and the report says where:
work_rootis the host's declared work root, andwork_root_mountthe mount point of the volume holding it (the longest mounted path the root sits under); both are null when nothing is declared.volumes[]lists every device-backed filesystemdf -Pkreported, in the same shape asusage. The text report prints the volume holding the work root asvolume: <free> free KiB on <device> mounted at <path> (<capacity>); the fleet's work root <root> is here, and every other one as…; the fleet does not write here.block_deviceslists every block device the Linux kernel sees, fromlsblk -b -P -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID,MODEL:readis false on a host withoutlsblk(every macOS host), and each device carries anunmountedverdict. The text report prints each asattached, not mounted: /dev/<name> <size> GiB <model> (<filesystem or no filesystem>); nothing on this host can write to it until it is mounted.
stado host gates TARGET carries the same disks as unmounted_disks and adds the note disk_attached_unmounted. It is a note, never a blocker.
Memory
space report carries the host's memory as readings only: free_kb, swap, available_bytes, available_mb, total_bytes, swap used and total, swap_used_pct, and on macOS the compressor pages, lifetime swapouts and page size. Nothing reclaims memory and nothing is declared about it; a host is never withheld from work for memory. The same three figures — memory_available_gb, memory_total_gb, memory_swap_used_pct — ride in every capacity publication and in stado host gates, where they explain a host that has gone quiet.
When the agent cannot measure the host's memory at all, its capacity publication says accepting_jobs: false with admission_reason: memory_measurement_unavailable and the operating system's own refusal as admission_detail (for example cannot read /proc/meminfo: Permission denied, /proc/meminfo carries no readable MemAvailable:, or the failed host_statistics64/sysctlbyname call on macOS). stado host gates TARGET and Stado Desktop print that detail beside the reason.
Who holds the accelerator
A GPU host's memory has a second pool, and a card can be full while no Stado job runs. The agent asks the driver which processes hold each card (nvidia-smi --query-compute-apps), maps each pid to its systemd unit through its cgroup and to the Stado job whose process tree it belongs to, and publishes the answer in its capacity document: diag.accelerator_holders (one row per process and card: device, pid, process, used_vram_gb, unit, stado_job, and unit_error when the process's /proc/<pid>/cgroup exists but cannot be read), diag.accelerator_memory_model (discrete, or unified on Apple silicon where the GPU shares the host's memory and no per-process VRAM exists), diag.vram_unattributed_gb (used memory no listed process accounts for) and diag.accelerator_holders_error when the driver could not be asked or the holder list could not be serialised.
stado host gates TARGET prints one line: accelerator: held by pid 4711 python3 80.5 GiB (unit vllm.service, not a Stado job), accelerator: held by pid 4711 python3 80.5 GiB (unit unknown (cannot read /proc/4711/cgroup: Permission denied), not a Stado job), accelerator: no process holds the accelerator through the driver, yet 94.0 GiB is in use, or accelerator: accelerator shares the host's memory; no per-process VRAM. A publication that names a memory model but carries no holder list, or one that does not decode, prints accelerator holders unknown: … or accelerator holders unreadable: <decoder reason>, never "no process holds the accelerator". Stado Desktop's host gates panel shows the same sentences. stado space report TARGET prints the same sentence as accelerators: and carries the block under accelerators in its JSON, with (publication stale) appended when the host's agent has not published within the staleness horizon. stado fleet needs raises a gpu need when the only GPU host publishes zero free VRAM with memory held by no Stado job.
Refusal sentences
stage '<stage>' is not declared; add it to stado-rs/data/fleet/space.json reclaim_stagesstado-rs/data/fleet/space.json declares no reclaim stages; add at least one to reclaim_stagesstado-rs/data/fleet/space.json reclaim stage '<stage>' declares no description; add it to reclaim_stages<target> declares no eligible space reclamation stage; add it to stado-rs/data/fleet/space.json reclaim_stagesspace reclaim --apply removes files and needs --reason <text>; the reason is appended to the target's own audit log beside the state it changed. Run without --apply to preview the declared stagesPOST /api/registry/policywith an unsupported per-target reclaim setting answers HTTP 400 naming the unsupported key;memory_reclaimanswers that it is no longer declared and the fleet's disk-full rule governs reclaim.- An unknown verb is a usage error (exit 2), for example
error: unrecognized subcommand 'watermark'.
Stado Desktop
The host inspector's Space section reads space report TARGET --json through the selected endpoint's native POST /api/operator/run API. It shows the free disk, the rule line (Deletes everything the fleet put on this host at 80% used — volume NN.N% used (triggered)), memory readings, cache verdicts, the Build output census row (the walk the census came from, read to its end or, as a warning, not read to its end: the census lists only what was seen), cleaner areas, stage roots, the rows outside them with the cleaner that takes each or user data, and the recorded janitor's counts, skipped reasons and errors. The complete command receipt remains available, including failures and incomplete inventory.
The same private report renders each Recorded cleanup hold — PID with its operation and acquisition date/time. Public Disk/beacon reports continue to omit process identities; they are not the source of private holder evidence. Products retains the complete CLI installation refusal.
The Disk screen shows the same rule line, free space, volume size and what the last pass reclaimed; Reclaim local space… asks before deleting and states what a pass at the threshold takes and what stays. Hosts shows the rule beside the claiming gates and memory as readings; Reclaim disk… runs the fleet stages of space reclaim. Registry shows each target's work root and queue eligibility; queue eligibility is the only field it writes. There is no memory screen and no cleaner editor.
The Hosts Reclaim disk… sheet reads the space reclaim TARGET --json receipt for both preview and apply. Pass diagnostics names every skipped stage and its reason, each stage's detail and refused identifiers, and the janitor's outcome, errors and every cleaner's scanned, eligible, deleted and skipped counts (with kept bytes where the cleaner measured them).
A null janitor cleaner table says the pass never reached a cleaner; a missing required host, mode, stages, skip list or stage count is an unreadable receipt, not a successful empty pass. An unknown pass mode is also refused, rather than treated as a preview. The apply remains unavailable until this host's real preview has answered and a reason is entered.
Requests retain the selected host and endpoint. A failed command keeps its returned stdout, stderr and refusal. The GUI does not launch a local CLI subprocess.
Working directories
Job work directories live under ~/.stado/work/jobs (or the declared work root, stado space work-root). They are reclaimed by the queue_workdirs cleaner: on this machine as part of the janitor pass stado space reclaim --local, which takes no other --stage (space reclaim --local supports only --stage registry_cleanup; other stages require TARGET), or on a fleet target by name with the stage selected:
stado space reclaim --local --json
stado space reclaim --local --apply --reason "disk at the rule" --json
stado space reclaim TARGET --stage queue_workdirs --apply --reason "disk at the rule" --json
The cleaner asks the queue which of the candidate job ids are retired and removes only those trees; a job the queue still carries is kept and counted under job_queued_or_running. When the queue store does not answer, the cleaner keeps every candidate and reports queue_store_unreadable under the queue_workdirs cleaner's skipped counts in the local JSON report instead of treating the silence as an empty keep-list. The fleet stage adds the process proof: a tree whose process is still present is kept and reported as such. There is no command that removes every directory under the work root regardless of what uses it.
A pass asks the queue before it takes the exclusive lock, for the work trees and job outputs on disk then: the queue answers through the object API, whose reads can stand on the registry authority, and asked under the lock they held every build on the host for as long as they stood. Under the lock the work trees are listed again and one that was not on disk when the queue was asked is kept as live; a job output is removed only when the queue named it retired. The time spent waiting on the queue is still store_wait_ms.
If a removed directory immediately returns, first identify the program recreating it.
stado service reap --host TARGET --command PROGRAM previews only the named program under the product, service and scratch roots; its --apply form sends SIGKILL only when no registry-declared unit owns the process or an ancestor.
Stado Desktop's Disk screen runs the same local stages through Reclaim local space… and shows what the pass reclaimed and kept.
Incomplete inventory is reported
The disk reader retains its filesystem, memory and janitor observations before attempting the whole-tree inventory. When that inventory fails, JSON includes inventory_incomplete with the actual command error, text output prints inventory incomplete: with the same reason, and Hosts → Space shows an Inventory incomplete warning. Missing paths were not checked: an empty coverage list in this state does not prove that the disk is covered.
A verdict whose build-cache walk ended scan-failed is reported and the command still exits 0: the rule's verdict, the inventory, the coverage and the janitor's last pass were all read before it. A host that refused the read, has no path to it, or answered any other error still fails the command.
Retiring the old recurring unit
stado service ensure stado --host TARGET --reason "unify space reclaim" --json
withdraws a loaded Stado unit whose root command no longer exists before the
unified service starts, preserving its autostart scopes to restore if startup
or the replacement resolver's serving-route check fails. The refusal retains
the original cause and the recovery outcomes; a running process alone does
not release predecessor recovery. Stderr names the unit and init-system result; JSON carries each
{unit,state,detail} under predecessors, including valid one-pass units
reported kept. A failed retirement refuses ensure. In Stado Desktop,
Services → Add service → Preconfigured Wisent services keeps the sheet open
after Run and shows the same per-unit outcomes under Service ensured.
The refusal banner preserves the CLI cause. Inspect final external state with
stado service unit show UNIT --host TARGET --json.
Real tests
The native reclaim suite, its editor and evidence cases and the launchd cutover journey are on Host space: real tests.
Source: this website