The disk-full rule

The rule

The volume is the one holding $HOME, read with statvfs, and — when the host declares targets[].work_root on another volume — that volume too, because job trees and build caches fill it. Of the two, the one with less room before the threshold decides. A pass the agent runs knows its declared root; stado space reclaim --local knows it when STADO_WORK_ROOT is set. On macOS the home's volume is the APFS Data volume, whose usage is the container's — not the sealed system snapshot mounted at /, which can read 56% used while the Data volume is at 96%. Used percent is computed as df computes it, from the space an unprivileged writer may still take.

Below 80% a pass still reads the registry and the queue, takes the lock and takes what nothing uses any more — a finished job's outputs and work tree, scratch whose owner ended, replica twins proven duplicate, release versions nothing names, run evidence, delivered copies and binary backups, browser clones and Weles recordings — and, short of the margin below, package downloads; it ends healthy_noop. Harness logs, Docker, local snapshots, the compiler cache and model blobs are only counted there and Cargo's build directories measured and kept; Xcode DerivedData, tagged build caches and source snapshots are not walked below 80%, so their rows carry no scan; service logs are not emptied. At or above 80% every cleaner runs over its whole area with no age, count, byte or scan limit, and nothing it finds is kept for being young or numerous:

Cleaner	What it takes	What stays
agent_logs	Every file under the harness log directories ~/.omp/logs, ~/.claude/debug, ~/.codex/log, ~/.factory/logs, ~/.kimi-code/logs	—
backup_twins	Same-disk replica objects under ~/.stado/local-backup whose primary under ~/.stado/local-storage has the same size and content hash	a replica whose primary is absent, differs, cannot be read, or is the same file
build_caches	Every directory carrying a build tool's own CACHEDIR.TAG anywhere under the home	a tree whose size cannot be measured completely; the macOS privacy-protected folders listed below; a cache where a running build holds its lock (held_by_build); a cache another live process has a file in open, works in or names in its arguments (held_by_process, read from lsof and ps right before it goes), because cargo test releases its lock before it runs the test binaries from the cache and a binary built there runs from it long after; every cache, when either table cannot be read (process_table_unread)
cargo_build_dirs	Each of Cargo's per-workspace build directories $CARGO_HOME/build/<xx>/<hash> (the build-dir the janitor declares for checkouts it cannot walk), tagged or not, whole	a directory where a running build holds a lock at its root or one level below (held_by_build): .cargo-lock, .cargo-artifact-lock, or the .cargo-build-lock Cargo holds in a build-dir laid out apart from its artifacts; a directory another live process has a file in open, works in or names in its arguments (held_by_process), as the test binaries cargo test runs from it after releasing the lock do; every directory, when lsof or ps cannot be read (process_table_unread); the one the running Stado executable is in (running_executable); ~/.cargo/registry and ~/.cargo/git are never taken
chromium_clones	The operating system's per-launch code-signing clones in the account's temporary container	the newest clone, and any clone a live process names; every clone, when one cannot be read while the newest is chosen (newest_unknown), because leaving it out would protect an older clone instead of the one a running browser may use
compiler_cache	Every file of a Kache store an older Stado left, ~/Library/Caches/kache on macOS and ~/.cache/kache on Linux; builds no longer write one, since kompilator keeps its own units	everything, while a job is running (job_running), because a running build of that older Stado reads and writes it
docker	Docker's images, build cache, stopped containers and volumes no container mounts, removed by the daemon itself (docker system prune --all --volumes --force); expected_bytes is what docker system df reported reclaimable, actual_free_delta_bytes what the prune says it reclaimed	every running container, the images those use and the volumes any container mounts; everything while a job runs (job_running); everything when the daemon's root directory (docker info's DockerRootDir) is not on the volume the rule judges, such as Docker Desktop's virtual machine or another disk (docker_root_not_on_judged_volume); a host without the client is docker_absent. A container meant to start again must therefore be running, as inference deployments are (--restart unless-stopped)
bin_backups	Backup copies an install leaves beside each binary in ~/.stado/bin (<binary>.release-backup-*, .bak-*, .pre-*, .<x.y.z>-backup-*), the same shapes delivery_leftovers reads	for each installed binary its newest backup by modification time (newest_backup); the binary itself, its .previous rollback target, its .release-version coordinate, every symbolic link and every name of another shape; every backup of a binary when one backup's age cannot be read (modification_time_unreadable)
delivered_releases	Release copies delivery staged under ~/.stado/releases/<product>/<version>/ (the attestation copy and retained archive of every version delivered here)	for each product, the version ~/.stado/bin/<product>.release-version names, the version whose staged copy is byte for byte the installed ~/.stado/bin/<product> (what stado release version show attests against), and the newest version by modification time (installed_or_newest); every version of a product whose coordinate, installed binary or a staged copy exists and cannot be read (installed_release_unreadable, with the path and OS cause among the cleaner's errors)
huggingface_cache	Model blobs under ~/.cache/huggingface/hub	everything, while a job is running (blocked_running_jobs)
queue_workdirs	Work trees under ~/.stado/work/jobs (and the declared work root) of jobs the queue reports terminal	the tree of a running job
registered_scratch	Each tree in the scratch register (~/.stado/space/scratch.json) whose owner process has ended, least recently changed first	a tree whose owner process still runs (owner_session_running); every tree that was never registered
job_outputs	Payload files under status/<job>/output/ of jobs the queue lists as terminal, in ~/.stado/local-storage	each job's own top-level receipts and logs; jobs the queue lists as live
local_snapshots	Every local Time Machine snapshot of the volume the home lives on (local_snapshot_volume in the report)	com.apple.os.update-*, the operating system's recovery state; snapshots of every other volume
object_evidence	Dispatched results under ~/.stado/local-storage/ecosystem/probierz/results, and under the queue's ~/.stado/local-storage/ecosystem/probierz/runs only a file of a run proven finished: a component of its path (or its name) names a run whose runs/<run id>.json manifest states at least one entry and every entry's retained outcome in a terminal prefix (completed, uploaded, failed, cancelled), and no component names a run that is not. Both roots are read without following symbolic links	Every other file under runs — the record of a run still planned, queued or running, of a run no manifest names, and every record Stado keeps there — reported as run_not_proven_finished, so neither disk pressure nor a file's age ever takes the record of a live run; a run whose manifest cannot be opened, read or parsed keeps all of its records, counted run_manifest_unreadable and named among the pass's errors as the run manifest <path> cannot be judged (<cause>), so every record of its run is kept; a pass reclaims them once that manifest is readable by the agent's account and states each of its entries settled in a terminal prefix; the run manifests themselves and Stado's release records directly under runs — release runs (release-pipeline, release-pipeline-unfinished), build records (build), change batches (release-changes), placed check bundles (quality) and queued session contexts (workload-session) — and native-signing inputs beneath either evidence root, reported as stado_record_or_pinned_input_kept; every symbolic link under either root, kept as a link and never walked (not_a_regular_file); everything, when a link or file stands in any component of a root's path (root_not_a_directory, also named with that component's path and the operating system's refusal among the pass's errors, so a root nothing is cleaned under is told apart from an absent one). A root, directory, entry or file it cannot read or remove is skipped and named with its path and cause among the pass's errors (root_unreadable, directory_unreadable, unreadable, deletion_refused). Journey: tests/space_reclaim/evidence/run_liveness.rs
release_store	Published release versions under ~/.stado/local-storage/ecosystem/releases	versions the registry declares or config pins, versions this host has installed, versions an in-flight run uses, the newest complete signed release (newest_signed_release_kept), a version whose source identity or completed publication cannot be verified (source_identity_unverified, publication_completion_unverified); everything, when the registry cannot be read (registry_unreadable)
source_snapshots	Build source snapshots under ~/.stado/local-storage/ecosystem/sources/<product>/<sha256>/source.tar.gz (one per build submit or release submit) that nothing names	each product's newest (newest_of_product), which the next build of the same tree reuses; every snapshot a release catalog entry, a queued or running job, or a release, build or quality record that is not finished names (named_by_record); everything when one of those records cannot be read (records_unreadable, the record named among the errors); pinned dependency inputs (sources/<product>/dependencies/…) are never this cleaner's — stado release catalog input retire removes them
weles_recordings	Weles run recordings, uploaded or not, under the host's weles.recordings_dir (the root the weles-recordings workload also writes into the Weles LaunchAgents); a host that declares none is skipped as recordings_dir_undeclared and no home path is swept in its place	—
xcode_derived_data	Each of Xcode's per-project build directories ~/Library/Developer/Xcode/DerivedData/<project>-<hash>, whole	a directory a running process has a file in open (lsof) or names in its arguments (held_by_process); every directory when either table cannot be read (process_table_unread); ModuleCache.noindex, archives, device support and simulators are never touched

~/.ssh is in no cleaner's area. Nothing outside those areas is the fleet's, so nothing outside them is opened for deletion. The object-evidence cleaner descends from the selected home to each evidence root by opening every directory component without following links, then scans and removes entries relative to held directory descriptors; replacing a path with a link between inspection and deletion cannot redirect the pass into the user's files. Protected release-record and pinned-input directories are kept as opaque subtrees: the cleaner does not open their contents, and stado_record_or_pinned_input_kept counts each skipped subtree rather than its files. On a host its registry entry declares shared (a person's own computer), a pass deletes only inside ~/.stado and reports the rest: Host space on a shared host.

A pass that the rule started empties the service logs under ~/.stado/logs (.log, .out, .err) that this account owns last: only once every other cleaner, the second tier included, has left the volume still at the rule, since a log is the record of why a resident service exited. Each is emptied in place, so a unit still writing to one keeps its file, and what it held is kept compressed beside it as <log>.emptied.gz (read it with gzip -dc <log>.emptied.gz), replacing the copy the last emptying kept; the copy is written beside it and renamed into place, so a pass that stops half way leaves the last complete one. A log is never touched below 80%; a symlink, a hard-linked file or another account's file is left alone. The pass's report lists the cleaner as service_logs, naming each log it emptied with the bytes that freed (what the log held, less its kept copy, plus the copy it replaced), and the pass log says service log emptied file=<name> bytes_before=<n> kept_as=<path> kept_bytes=<n>, or service log emptying failed file=<name>: <cause>. A pass beside workload holds does not run it and names it among the cleaners it waited for.

Every error a pass meets is kept in its report, however many there are. Walks and deletions follow a tree as deep as it goes; a tree deeper than the process may hold open reports the operating system's own refusal.

The second tier: package managers' caches

When the cleaners above leave the volume at the rule, or without the margin under it, package_caches takes what a package manager downloaded and fetches again, largest first, and stops as soon as the volume has the margin (room_for_the_next_build):

• Cargo's git checkouts/ (never git/db or the registry);
• npm's ~/.npm/_cacache, Bun's ~/.bun/install/cache;
• SwiftPM's, pip's and Homebrew's caches under ~/Library/Caches, and ~/.cache/pip and ~/.cache/uv;
• VS Code's update staging, ~/Library/Caches/com.microsoft.VSCode.ShipIt.

A cache stays when a running process has a file in it open, works in it or names it in its arguments (held_by_process, read from lsof and ps right before the cache goes). Cargo's checkouts also stay while any cargo or rustc runs (build_running), because a build reads them crate by crate without holding them open, and while a cargo holds its package-cache lock. Every cache stays when a process table cannot be read (process_table_unread). A cache is moved aside in one rename right after its holders are read and then removed, so a tool that starts meanwhile makes a new cache instead of reading a half-removed one; a tree left aside by an interrupted pass is removed by the next. Each removal is a line of reclaimed.jsonl. When agent output alone leaves the margin, every cache stays (agent_output_was_enough). Products' own application caches and the owner's files are never in this tier; space candidates only proposes them.

The margin: room for the next build

A pass's second tier goes on under the rule until the volume can take the largest build this host makes, so the next build does not put the volume back at the rule: the largest Cargo build directory the pass measured under Cargo's home (held ones included), or the largest one reclaimed.jsonl says a pass took, whichever is larger. A host with neither has no margin. A host under 80% but short of the margin runs what every pass runs (what nothing uses) and in addition the second tier; Cargo's build directories are measured and kept, the other caches are only counted or not walked (above), no cache but package downloads is taken and no service log is emptied, since those go only at the rule. Every report carries margin: bytes, measured_on (the directory) and source (measured_this_pass or removal_record).

A pass ends with one outcome:

outcome	Meaning
healthy_noop	The volume is under 80%; the pass took what nothing uses and, short of the margin, package downloads.
reclaimed_below_threshold	The pass deleted and the volume is under 80% again.
still_full	The pass deleted what the fleet's cleaners cover, and the volume is still at the rule: what holds it is ranked and proposed under full_disk (below).
no_eligible_items	The volume is full and no cleaner found anything to take; what holds it is ranked and proposed under full_disk.
blocked_running_jobs	The volume is full and the Hugging Face cache waited for running jobs.
partial_error	A cleaner failed; errors names the operation and cause, and the other cleaners still ran.
report_only	A --dry-run pass: counted, deleted nothing.
volume_unreadable	statvfs failed; the rule could not be applied.
lock_busy, lock_busy_workloads, lock_busy_unattributed	Another holder had the lock; the holder is named. Under lock_busy_workloads three cleaners (bin_backups, delivered_releases, and local_snapshots at the rule) may still have run beside the holds and waited_cleaners names every cleaner that did not (see Locks).
lock_recovery_report_only	A retired kernel lock is still held or unreadable; nothing was scanned or deleted.

A Docker client whose daemon does not answer (Cannot connect to the Docker daemon, as when Docker Desktop is quit) is counted as docker_daemon_not_running and is no error of the pass, like a host without the client (docker_absent): there is nothing the pass could ask it to remove, and reading it as a failure made a full volume's pass end partial_error, which hid that nothing the fleet holds was left to take. Any other refusal of docker system df stays an error.

A daemon that accepts its socket but answers nothing is given two minutes for docker system df and for docker info. Past them the query is ended, Docker's data is left alone, the cleaner counts docker_daemon_unresponsive and the pass records the error docker <query> gave no answer within 120 s, so the daemon accepts its socket but answers nothing and this pass leaves Docker's data alone (the query was ended); …, which names the restart that brings the daemon back. The pass goes on to its other cleaners and lets go of the lock. Before, the query had no bound: the pass held the exclusive cleanup lock for as long as the daemon stayed stuck, and every installation and job on the host waited on the disk janitor's cleanup pass.

A volume the cleaners leave at the rule

Every cleaner takes only its own area, so what holds a volume past the rule may lie outside all of them: an agent's output written where no cleaner looks reads as the owner's data, and it keeps the host short of room for work until someone frees it. A pass that ends still_full, no_eligible_items, partial_error or blocked_running_jobs therefore does not end there. Once it has released the lock, it ranks the home as stado space candidates does, keeps the rows no cleaner takes (action: proposal), largest first, until they add up to what the volume must free, and writes them to full-disk-proposals.json beside the janitor's state:

• state: surveyed, or survey_failed / volume_unread / record_unreadable / lock_unavailable with error naming what failed and its repair;
• used_bytes, used_percent, need_bytes (what must be freed to come under the rule) and min_bytes (the smallest item listed: 1 GiB, the default of stado space candidates, or need_bytes when that is less);
• proposals: each row's path, bytes, class, safety, producer, note and cumulative_bytes, in the shape of stado space candidates; proposed_bytes and enough (whether the proposals cover need_bytes);
• unmeasured: what the walk did not open and why (a folder macOS withholds, …);
• notices: for each proposed path, the Oko defect recorded for it (defect, what oko defects record printed) or why none was (refused).

Each proposed path is recorded once as an Oko defect of Stado, titled <host> stays at the disk-full rule and no janitor cleaner covers <path> (<size>, <producer>), naming the host's use and what it must free, and proposing the repair: a cleaner for the area if what its producer wrote there is regenerable or finished, or the owner's decision if it is his. A record Oko refused, or one made where no Oko is installed (oko defects record could not run: …), is tried again by the next ranking. Nothing proposed is deleted.

The walk reads the whole home, which takes hours on a full developer host under load, so a later pass ranks again only when the volume's used bytes moved by more than the pass's margin; until then it keeps the ranking and says why in kept_because. The agent's periodic pass does not wait for the walk: it starts it in a thread of its own at the background priority the build-cache walk uses, under full-disk-proposals.lock beside the record so one ranking runs at a time, and reports ranking_in_progress beside the last ranking; a single pass (stado space reclaim --local --apply) ranks before it returns. A pass that leaves the volume under the rule removes the record. Every pass's report, the state file every reader reads, carries the record as full_disk, so stado space report TARGET --json shows it under cleanup_state.report.full_disk. It names paths and is the operator's record; the public beacon does not carry it.

While it runs, the ranking keeps its state in the lock file it holds: the process, when it started, its phase (walking the home, recording <n> proposals in Oko, writing the record), the directories walked so far and the one it last entered, and when that state last moved, rewritten at most every 30 seconds. A pass that finds the lock held says so in ranking_in_progress, which stado space report prints as its full disk: line and Stado Desktop's Disk screen shows: a ranking started <time> (<elapsed> ago) by process <pid> holds <lock>: walking the home, <n> directories walked, last under <path>, which moved <elapsed> ago. A walk whose state stops moving is one stuck on a directory, not a slow one. A lock whose state cannot be read says why (its state cannot be read: …), and one an older Stado holds says that it records no state.

A huggingface_cache error keeps its code and says <operation> <path> in the Hugging Face cache: <cause>; <what clears it>: the path is relative to the cache root and is the root entry being examined, the repository being scanned, the file being removed or the lock barrier. A PermissionError says the janitor's account cannot read or remove that entry and to give it ownership or remove the entry yourself; a FileNotFoundError that the entry went away while the pass ran.

A cleaner whose fixed root the janitor will not walk keeps its code and says the cleaner root <path> <what it found>, so the cleaner does not walk it; <what to do>, where what it found is a symbolic link the janitor does not follow, not a directory, owned by another uid than the janitor's, on another device than the volume it walks, or resolving outside the home. A root that cannot be examined or resolved names its path and what was being done beside the operating system's answer.

One link on the way is followed, and only into the host's declared work root: a cache moved onto the fleet's volume and linked back (~/.cache/huggingface -> <work root>/hf) is on the volume the rule measures, so huggingface_cache walks it there, checks every component below the link as a directory of its own on the work root's device, and measures what a deletion freed on that volume. A link that resolves outside the work root (links to <target>, which is not beneath the declared work root <root>), a link on a host that declares no work root (is a symbolic link and this host declares no work root) and a second link on the way (is a second symbolic link on the way) are refused.

backup_twins judges every directory on its way the same way: one it will not read says the replica scan directory <path> <what it found>, so nothing under it is read; … (a symbolic link, not a directory, owned by another uid, or on another device than the replica root), counted as read_dir_failed, and a path it cannot examine or list names itself and the operation.

What the agent admits: room, not the rule

The rule decides what the janitor deletes, never whether work fits. On every tick the agent judges the room on the volume the fleet writes to: its free bytes, less the margin the janitor's latest completed pass reported (the margin), less what the jobs it runs hold. It publishes the room under disk_room in its capacity diagnostics: fits, free_bytes, work_bytes (what new work may take), margin_bytes, margin_measured, margin_source, held_bytes, held_source, and refused (each job the last claim left queued, with its job_id and reason); or {"unread": <cause>} when it could not judge one, such as the janitor's latest pass reported no margin (outcome <outcome>[: <error>]); the next pass that completes measures it. disk_pressure_active says whether the volume is at the rule; it is a note, not the admission. The room reads <w> GiB is free for new work: <f> GiB free less <m> GiB the janitor's margin (<source>) and <h> GiB held by <what holds it>.

Each queued job is judged by its own disk need, at any percentage used. The need is never assumed: it is the disk its submitter declared (stado submit --boot-disk-gb); for a release build, the scratch a completed build of the same product and platform measured, the one this host keeps under ~/.cache/wisent-compute/release-scratch/, else the fleet's newest; for any other job, the disk the newest completed job of the same command, repository and directory used on this host, kept under ~/.cache/wisent-compute/job-disk/ one file per kind and replaced by the next completion of that kind. A job of a kind never measured here is judged by the largest disk a completed job used on this host; on a host where no completed job was measured yet, it needs nothing beyond the room, which keeps the janitor's margin free. A job whose need fits what new work may take, less what the jobs the same claim admitted before it need, is claimed (past the rule its log line says loop: job <id> fits: it needs <n> GiB (<source>) and <the room>); one that does not stays queued with loop: disk_room_short: job <id> stays queued: it <reason>. A job whose need is not known stays queued past the rule and while no work fits, and below the rule is admitted on the bytes the volume takes before it. While the room cannot be judged, a known need must fit before the rule.

When no work that writes to the disk fits, the agent publishes accepting_jobs: false with admission_reason: disk_room_short, disk_room_short: true and admission_detail: no new work that writes to the disk fits: <the room>. When the room cannot be judged and the volume is at the rule, the rule decides, under the same disk_room_short reason, with admission_detail: the room for new work could not be judged (<cause>), so the janitor's 80% rule decides: <used>% used, <free> GiB free.

Capacity diagnostics retain disk_used_percent, disk_total_bytes, disk_free_bytes and disk_pressure_active. An unreadable volume publishes disk_unreadable, null measurements and its cause in disk_unread, and refuses work until it can be measured; closed admission publishes unmeasured free_vram_gb as null. The disk gate publishes measured free_disk_gb and staging_free_gb, or null with their <name>_unread causes. largest_pending_raw_dir_gb is null when no directory is pending or its complete size is unreadable; largest_pending_raw_dir_gb_unread distinguishes failure and carries its cause. An unreadable pending tree refuses work rather than admitting against a partial total; symbolic links are not traversed. A disk_gate_refused publication carries the same measured refusal as the log in admission_detail, which host gates and Desktop show beside the reason. Both surfaces also show a supplied refusal detail when its reason label is absent, rather than replacing the cause with “busy or gated”. stado host gates TARGET reports used_percent, headroom_gb (negative past the threshold), full_percent and full; --require-disk exits 0 only below the threshold. Its room: line reads room for new work: <the room>, disk_room_short: <the room>, disk_room_short: not judged by the agent (<cause>), so it refuses every ordinary job until it can judge the room (the room unread while the gates carry disk_room_short), not judged by the agent: <cause>, or not published: … when no live publication carries the room, followed by each job the agent left queued (job <id> stays queued: it <reason>); Stado Desktop's host panel shows it as Room for work beside Disk-full rule. stado host cargo refuses a compile on a host short of room as room-short, naming the volume holding its build cache.

Before it claims work the agent creates, writes, flushes, syncs and explicitly removes a probe file on that volume. A failure at any of those operations refuses work for the tick and publishes home_write_probe_ok: false with the failed operation and operating system's reason in home_write_probe_error, including a probe file on the fleet volume cannot be synced: <cause> for failed persistence or a probe file on the fleet volume cannot be removed: <cause> for failed removal; like every disk figure it publishes, the reason carries no local path. Both fleet-volume and staging probes attempt removal even after another probe operation fails, retaining both causes if removal fails too. The agent log and admission detail carry the fleet-volume refusal beside the volume's free space.

stado host gates TARGET judges disk pressure from the agent's live capacity publication, or, when that is stale, absent or states no disk_pressure_active, from the volume reading it takes itself. When neither measured the disk, the host carries the blocker disk_pressure_unresolved, the same word the agent publishes for a volume it cannot read: a disk nobody measured is never read as a disk with room. Whenever the command's own reading gave no volume, the verdict says why in local_disk_unread (JSON) and on a local disk reading: line (text): df listed no row for the fleet volume, or a field it printed is not a count of 1024-byte blocks.

Two kinds of job it still claims short of room: the newest signed Stado release delivery pinned to it, which repairs the agent that owns this gate, and a job pinned to it that stages nothing on its disk (no repository, no packages, no pre-command, no output mirror), such as an Oko routine reading the host's own transcripts. Refusing those would free nothing and, on a host whose disk the user's own data keeps full, would stop the host's upkeep for good. Of the deliveries queued, only the newest runs. The agent publishes disk_room_short_recovery_jobs (the delivery it admitted), disk_room_short_superseded_deliveries (the older ones it did not) and disk_room_short_in_place_jobs, and logs loop: disk_room_short: <detail>; admitting <n> signed Stado release delivery, <m> pinned job(s) that stage nothing on this disk, and no ordinary work, or, with neither, …; ordinary work remains blocked until there is room for new work, and neither a signed Stado release delivery nor a pinned job that stages nothing is assigned to this host.

A pass that meets running workloads' holds while the volume is past the rule and its free bytes are not above the margin the next build needs writes disk-cleanup-turn.json beside the janitor lock, naming the process that asked. While that process lives no new workload takes a hold, so running jobs drain and the next pass gets the exclusive lock; a request whose process has exited is ignored. With room, or below the rule, a pass asks no turn and withdraws a standing one, since the agent still takes each job that fits; a --dry-run pass does neither.

What the agent does past its swap ceiling

A host that declares host.swap_ceiling_pct (configuration) claims no new work while its swap use is above it, and publishes accepting_jobs: false with admission_reason: swap_ceiling_exceeded and diag.swap_ceiling: "swap <used>% of <total> MiB used, above host.swap_ceiling_pct <ceiling>"; a swap it cannot read publishes the cause there and admits work. The same reading skips the host's own disk scans and holds stado product install|update|cargo|swift before they export or compile, as a czekam: host wait naming it, until swap falls under the ceiling, since work started past the ceiling mostly pages. A host that declares no ceiling has none.

Staging off a RAM-backed /tmp

When /tmp is tmpfs or ramfs and TMPDIR names no other place, the agent selects a writable wisent-staging directory on a read-write block-device mount outside /boot, with more free space than /tmp. Both filesystems stage data in memory; ramfs has no backing store. Diagnostics name the filesystem actually observed. Mount-table fields split on space and tab only, preserving other whitespace inside paths. Kernel mount escapes are decoded once, including \043 for a hash mark in a device source. Identified SSDs rank before identified HDDs, then unidentified media; ties use measured free space. The medium comes from /sys/class/block: a partition's partition attribute selects its actual parent disk, without a device-name pattern. A missing or unreadable queue attribute is logged with its path and cause and shown as unknown-device, never as an HDD.

The writability probe creates its own temporary file exclusively and must write and remove it; either failure refuses the candidate. WISENT_STAGING_USER selects the staging account; unset, the effective UID is resolved through /etc/passwd. An empty override, an account row without a name or an unknown account is refused with its cause. A non-root agent refuses a staging account with another UID instead of silently ignoring it. The account is resolved once for the selection pass. A root agent hands staging to that account's UID and GID and reads both back; a failed read or mismatched ownership refuses that candidate. It first tries to make inaccessible parents searchable by others. Malformed mount rows, unreadable device paths, failed probes, unreadable free space and traversal failures retain the actual operation and cause in the log while other candidates remain eligible. An empty or non-UTF-8 filesystem-type answer from stat -f is an unreadable /tmp, not evidence that it is disk-backed. If the filesystem cannot be read or no candidate qualifies, staging stays where it was and the log says why.

Locks

The kernel lock, not the age of its owner record, decides whether a pass may run. A live hold returns lock_busy, lock_busy_workloads or lock_busy_unattributed, naming the recorded holder where available. Stado does not replace a live lock because it is old. A still-held retired lock prevents scanning and deletion with lock_recovery_report_only; an unreadable retired lock reports its actual error. A running workload holds the same lock in shared mode and writes one record beside it (~/.cache/wisent-compute/disk-cleanup-holds/<pid>-<id>.json). The private pass records each blocked workload in workload_holds with pid, holder and acquired_at (Unix seconds), also named in its diagnostic. This is the pass's observation, not proof the process still holds the lock when a later reader opens it. What a pass runs before it asks for the lock it runs whatever the lock's state: registered_scratch removes under its own scratch lock, while cargo_build_dirs, and at the rule xcode_derived_data and build_caches, take the run lock exclusively for each removal, so beside a hold they only measure and keep what they find (cleanup_lock_busy). Beside live workload holds a pass then runs the cleaners no running workload needs kept whole: bin_backups and delivered_releases, whose keep rules protect what a running workload uses (an install's newest backup and .previous, the installed and newest delivered release), and local_snapshots, Time Machine's local snapshots, which macOS writes and no job reads, deleted as an exclusive pass deletes them: at the rule, on a host whose whole home the pass may delete in. An enforcing pass runs those three under its own shared hold, so no exclusive pass starts meanwhile, and under disk-cleanup-unheld.lock taken exclusively, so no second pass runs them at once; when either lock refuses it, it runs none and records unheld_cleaners:BlockingIOError naming the lock that refused. A pass that meets such a pass and no workload reports lock_busy with held in shared mode by a janitor pass running bin_backups, delivered_releases, local_snapshots beside workload holds; that pass is not counted as a running job. A preview runs the three without taking either lock and deletes nothing. The removals go to reclaimed.jsonl. Every pass that found the lock held lists in waited_cleaners (private report, and public beacon, where a report written before the field existed shows null) each cleaner after the lock it did not run, the second tier package_caches among them; they run when a pass has the lock to itself. At the rule a pass that may delete asks new workloads to wait for that (disk-cleanup-turn.json, above), while workloads already holding the lock keep it until they end. stado space report adds ; with the lock held it left <cleaners> for its turn to the janitor's detail, stado space reclaim TARGET prints <cleaner> waited for the janitor's turn in its registry_cleanup line instead of counts, Stado Desktop's Disk screen marks those rows waited for its turn and says whether the three ran, and its Space section lists them as Left for the janitor's turn. Under lock_busy and lock_busy_unattributed no cleaner after the lock runs.

While a standalone pass holds the lock, the agent publishes admission_reason: cleanup_in_progress and stado host gates reports it as a blocker; a lock the agent cannot probe publishes cleanup_lock_error with the error code. stado space report TARGET names the holder in cleanup_lock.holders.

stado product editor serve answers existing compiler settings without a lifetime cleanup hold. Builds, indexing and restore operations remain protected. An installation that finds a pass holding the lock exclusively waits for it: its stderr carries a czekam: lock wait naming the janitor's cleanup pass and the recorded holders, stado waits lists the open wait, and the operation takes its shared hold and runs once the pass lets go (koniec czekania). A wait that would close a wait cycle is refused naming the lock and the cycle. An installation is refused before export or compilation only while the janitor has asked running work for its turn and running work still holds the lock; that refusal (… the janitor has asked running work for its turn; nothing was exported or built. It waits on: …) names recorded shared holders and their acquisition time, the janitor's turn request (process, request time and reason), and its recorded exclusive owner where available. The turn ends once the janitor's pass has run, so the refusal is stated as a retryable wait (exit 69, retry later); --wait waits for the turn instead. A turn request no running work holds the lock against waits on nothing — the work it waited for has ended and the janitor process that asked has not taken its turn — so the installation runs that pass itself, as --wait does, says the disk janitor asked for its turn and no running work holds the cleanup lock, so this operation runs the janitor's pass before it builds, and builds once the pass has answered the request. A turn request or lock that cannot be read refuses the operation with the record's path and cause; absent records say ownership was not established, not that no process holds the lock. An already running older binary retains its original lock until that process ends; installing a new executable does not revoke kernel ownership. A Stado operation started by one that holds the lock (an installation's quality gates and stado product kompilator stage) runs under that hold: the holder leaves STADO_JANITOR_HOLD=<pid> <holder> for the processes it starts, and a process whose running Stado ancestor that pid names takes no hold of its own, says which operation holds it, and is never refused for the turn that waits on its ancestor.

The janitor's files

The janitor keeps its files in ~/.cache/wisent-compute: the run lock disk-cleanup.lock, its holder records disk-cleanup.lock.holder and disk-cleanup.lock.holder.inode.<device>.<inode>, retired lock inodes disk-cleanup.lock.retired.<name>, the state file disk-cleanup-state.json and its transaction lock disk-cleanup-state.lock, the workload holds under disk-cleanup-holds/, the turn request disk-cleanup-turn.json and disk-cleanup-unheld.lock, which a pass beside workload holds takes while it runs the cleaners that do not wait. These names are shared by every Stado release: a build still running from the release a host is being moved off holds the same lock and writes its hold under the same directory while the new release's janitor runs, so the janitor sees that build and keeps what it is writing. The space reclaim engine keeps these names for that reason. The state file carries "version": 1, which every release reads exactly; a new field is added as a key rather than a new version, so an older reader still reads the file.

The janitor's promise

A janitor that runs on a period — the agent's pass every poll or the host's health watch at its declared cadence — states in its state file, under promises.<writer>, when its next pass will have written that file, and which process made the statement (pid). The promise is the pass's end plus the period, the longest pass on record for that writer on this host (longest_pass_seconds), and the longest it has been measured to start late (longest_lateness_seconds: the time between one pass's end and the next one's start beyond the period, spent writing the file, printing and, for the agent, sweeping scratch and restoring reconcilers). Lateness is measured only between two passes of one process, so a stopped service is not counted as late, and a writer's first pass on a host has none on record. A single space reclaim --local --apply pass promises nothing and leaves standing promises as they were.

stado space report TARGET shows every promise under cleanup_state.promises (writer, next_pass_by, pid, and alive: whether that process is among the host's live stado processes), the furthest as cleanup_state.next_pass_by and cleanup_state.promised_by, the live processes as cleanup_state.live_stado_pids (null when the host could not list them), and cleanup_state.prevented (whether the last pass was turned away by a workload's hold on the lock).

stado host gates TARGET counts the janitor as running while any promise holds or any process that made one is alive: a pass slower than any before it overruns its promise without the janitor having stopped, and a process being replaced is briefly gone while its promise still holds. It reports space_reclaim_stalled when the janitor is not running by that measure — including a state file from a janitor older than the promise, which states none — when no pass has ever succeeded, or when the last pass ran and did not succeed. It reports space_reclaim_lock_held while the janitor runs and its last pass was turned away by the lock. Each is a blocker on a full volume and a note otherwise.

Locations a pass never opens

On macOS the build-cache walk refuses ~/Pictures, ~/Music, ~/Movies, ~/.Trash, ~/Library/Mobile Documents, ~/Library/CloudStorage, ~/Library/Group Containers, ~/Library/Containers and ~/Library/Developer/CoreSimulator before opening them, and counts each refusal as privacy_protected in that cleaner's skipped map. Media and cloud roots may trigger privacy or file-provider reads; containers and simulator roots hold other applications' state. A tagged build cache inside a protected root is kept rather than stalling the reclaim lock or waiting on consent. stado space report prunes the same roots.

~/Documents, ~/Desktop and ~/Downloads are consent-gated as well and stay in scope, because real build trees live in them. The first open under each runs on a helper thread; the pass records consent_pending, skips that folder and walks the rest of the home. A tagged directory that contains a refused root is never deleted.

Nothing Stado builds is left in those folders when a background process cannot read them. The janitor writes no build tool's configuration: every Rust build is kompilator's, which keeps its output and units in the build area, never in a checkout, and a pass no longer writes a build-dir into Cargo's config.toml under its home. stado product install|update keeps every run, scratch tree and desktop .build in the checkout's build area under ~/.stado/products/builds (see Product catalog).

A build cache is measured completely before it is deleted. An unreadable entry, changed directory or different filesystem keeps that cache and records cannot measure cache <name>: with the operation and cause in errors. When macOS answers Operation not permitted for a whole folder, the error says the operating system's privacy protection refuses it, to grant Full Disk Access to the agent that runs the janitor, or to reclaim the checkouts' rebuildable trees with stado product build-trees remove from a terminal macOS lets read the folder. That command reads and removes, from the process that runs it, every tree in the workspace's checkouts that a build tool tagged or Stado declared rebuildable, and keeps any tree a build holds (see Product catalog). stado space reclaim <target> --stage tagged_build_caches --apply --reason … runs the same eviction from the operator's command, which holds that access, and skips any tree a live process names.

The walk is incremental, backgrounded and admitted by load

The home is not listed again on every pass. The janitor keeps, beside its state (~/.cache/wisent-compute/build-caches.json), the child directories of every directory it examined and the measurement of every tagged cache it found, each under the directory's device, inode and modification time. A directory adds or loses an entry only by changing its own time, so a directory whose three are what they were has the child directories it had: the pass takes them from memory and does not list it or read the kind of its files, while each child is still opened, checked for owner and device and read for its tag as before. A new tree anywhere changes the time of the directory it appears in, which is listed again. The pass counts the child directories it took from memory under listing_remembered in the cleaner's skipped map. A tagged cache whose root is what it was reports its remembered bytes (measurement_remembered, with the bytes under skipped_bytes) instead of reading every file beneath it; a file growing inside an existing subdirectory does not change the root's time, so those bytes are the last measurement's until the root changes or the cache is removed. Deletion never reads the memory: an enforcing pass removes the tree and measures the volume before and after. A completed pass rewrites the memory only when it listed or measured something new or found something gone; a pass that stopped early leaves the earlier memory. A memory that cannot be read is an error in errors, and the pass walks without it.

The walk runs in the host's background band: on macOS the thread takes PRIO_DARWIN_BG, which throttles its disk I/O and its CPU share as taskpolicy -b does for a process; on Linux it takes the lowest nice value the system declares, from which the I/O scheduler derives its priority. A system that refuses the change leaves the walk at its previous priority and records priority_not_lowered.

That band is the walk's alone. The host unit that runs stado serve declares launchd ProcessType Interactive, whether it is installed as the host unit, rendered by stado service ensure stado, or rewritten from a resident's definition; every other unit Stado deploys keeps launchd's default. A job that declares none gets launchd's light resource limits, its CPU and I/O throttled, and every process it starts inherits them, so a build an operator's session waits on compiled at scheduling priority 20 beside the session's 31. stado service unit show com.wisent.stado --host TARGET --json shows the unit file a host runs; a host takes the declaration on its next unit render.

The census space report prints beside the verdicts (build_caches.entries, tagged_build_output) is the same walk run as a shell program on the host, with find and du placed in that band by the host's own words: /usr/sbin/taskpolicy -b on macOS, ionice -c idle nice on Linux. The program keeps as a tag only a CACHEDIR.TAG path under the scanned root; a line the walk printed about itself — the shell refusing a program, a find that did not start — is the walk's failure, printed on its error stream and never read as a directory, so a census row is always a directory under the root with its size in KiB or -. Real test: tests/space_reclaim/space/census.rs.

Every walk of these programs sends its errors to /dev/null, so a band program the host does not have would empty them all without a word. Each program therefore checks the band program first and, when the host lacks it, ends before any walk with the background band program <program> is not on this host, so no walk can run on its error stream; space report, the janitor's build-cache pass and the host reads that share the band fail with that sentence instead of reporting an empty disk.

A janitor pass walks only when the volume is at the rule's full threshold, and there the walk is the relief: it runs whatever the host's load, at background priority, and its reading clears any standing skip under scans.skipped.build_caches. Skipping it for load would leave a host that its own work keeps above the ceiling full for good, its reclaimable build caches untouched and every pass ending blocked_running_jobs. A preview (space reclaim without --apply) is admitted against the host's load ceiling: when the one-minute load average is above host.scan_load_ceiling (WC_HOST_SCAN_LOAD_CEILING), or above the host's logical core count when nothing is declared, it records load_above_ceiling and an error naming the load, the ceiling and its source, walks nothing, and the host's beacon carries the skip under scans.skipped.build_caches until a later walk is admitted. The same ceiling governs the beacon's launchd event reads and the inventory walk of space report; see Beacon.

One walk runs at a time on a host, under its own lock ~/.cache/wisent-compute/build-caches.walk.lock rather than the cleanup lock, so workloads keep their holds while it reads the disk. The janitor that takes it writes its pid, writer and held_since into the lock file. A pass that finds the lock held walks nothing, counts walk_in_progress under the build_caches cleaner's skipped, and records the holder in its private report as build_caches_walk_holder: that record with alive (whether the process still runs), or unreadable saying why no record could be read (a holder whose Stado writes none leaves the file empty). A held_since hours old on a volume past the rule is a walk that is not relieving it. A lock this janitor took but could not write its record into adds an error naming the lock to the pass, and the walk still runs.

What a deletion is worth on a Mac

macOS takes a local APFS snapshot every hour, and a snapshot pins every block the volume held when it was taken, so deleting a file frees nothing until the snapshots referencing it are gone. local_snapshots therefore runs after every other cleaner in the same pass and deletes the Time Machine snapshots of the volume the pass's home lives on, read with df -P <home>, listed with tmutil listlocalsnapshots <that volume> and deleted with tmutil deletelocalsnapshots <that volume>; the report's local_snapshot_volume names it. The dated form of that command deletes the snapshots of that date on every mounted disk, so the cleaner never uses it. A home on its own volume, such as the disk image every test journey runs on, has no snapshots, so a test pass never touches the machine's own backups. A volume that cannot be read, or a listing tmutil refuses, touches no snapshot and is reported with the reason; a deletion tmutil refuses is reported with its words, its snapshots counted under deletion_refused. com.apple.os.update-* snapshots are counted and kept. macOS publishes no per-snapshot size, so the cleaner reports what the volume measured across the deletion in actual_free_delta_bytes.