Host memory

A host reclaims memory through one declaration: targets[].memory_reclaim in the canonical registry. It is the twin of targets[].disk_cleanup — the registry names the mode, the watermarks, the per-pass budget and the exact set of permitted repairs, the same two writers execute it on every tick, and a local target that declares nothing is measured against MemoryReclaimPolicy::reporting_default, which reports and repairs nothing.

It exists because on 2026-09-06 charless-mac-mini ran itself out of memory and nothing repaired it. The GitHub pre-check runner's listener died with Failed to create CoreCLR, HRESULT: 0x8007000C and exit 137, its last processed job stamped 2026-09-06T18:51:45Z; the host held about 1 GiB available of 16 GiB with 4.15 of 5.37 GB of swap in use, 181k pages in the compressor and 12.2M lifetime swapouts. Disk had a declaration, a watermark, a beacon field, two writers and a reconciler. Memory had none of the five.

"memory_reclaim": {
  "mode": "report",              // off | report | enforce; only enforce repairs
  "check_interval_seconds": 300,
  "low_free_mb": 2048,           // available memory below this is pressure
  "target_free_mb": 4096,        // a pass stops as soon as this is available
  "high_swap_used_pct": 80,      // swap at or above this is pressure on its own
  "max_repairs_per_pass": 1,
  "refuse_placement": false,     // publish accepting_jobs=false while over
  "repairs": {
    "restart_unit": { "units": ["com.wisent.stado-precheck-runner"] }
  }
}

The two watermarks

A pass is gated first on the interval and then on pressure. low_free_mb is the memory watermark: available memory below it is pressure. high_swap_used_pct is the second, and it is the one that was unambiguous during the incident — a free-memory figure alone reports a machine that has already pushed everything into the compressor and onto disk as merely busy. Either watermark makes the host over its watermark; target_free_mb is where a pass stops.

The two platforms do not mean the same thing by "available". Linux publishes MemAvailable, which the kernel computes as what a new allocation can obtain without swapping. macOS publishes no such figure, so the reader sums the page classes obtainable without evicting anonymous memory — free, speculative and purgeable — and records the compressor and swapout counters beside it as evidence.

What a pass may do

Only what the declaration names. The fleet's repair vocabulary is one compiled document, stado-rs/data/memory/repairs.json, read the way space.json's nine reclaim stages are read; the registry then names which of them a host permits.

Repair	What it does
restart_unit	Restarts a declared unit that has no live process — the state a process that could not allocate leaves behind.
reap_recovery	Runs a host-recovery program this release already ships, by name, and records what that program decided. recover-skarbiec-crypto is the one that reaps stale GnuPG daemons; keyboxd was holding 211 MB on the mini.
graphical_session	Ends a named process of the logged-in graphical session, and only when the declaration also carries allow_graphical_session.

graphical_session is the one repair that can lose a person's unsaved work, so it needs two declarations, not one: the registry must name the repair AND the repair must carry allow_graphical_session: true. Declared without the flag it reports exactly which processes it would end and how much each holds, and ends none of them. That is the rule disk_cleanup's allow_missing_upload_proof already follows.

Two further rules bound every repair. A host running queue work performs none of them and records blocked_running_jobs, because every repair here ends or restarts a process and a pass that ended one would turn a memory report into a lost job. And a repair that names no subject is refused by validation, never left inert.

Declared policies

A host is armed from a named policy, not from a typed watermark. The fleet's policies are one compiled document beside that vocabulary, stado-rs/data/memory/policies.json, and each entry carries a whole memory_reclaim declaration together with the platforms and roles it is written for:

Policy	Written for	What it does
macos-always-on-host	darwin-arm64, role always-on	Enforces 2048/4096 MiB, restarts the pre-check runner, reaps stale GnuPG daemons, ends WindowManager, Safari and Messages after ten minutes of life, and refuses placement while over its watermark.
macos-operator-workstation	darwin-arm64, role interactive	Enforces 4096/8192 MiB and reaps stale GnuPG daemons only. No unit is restarted, no session process is ended, and the host keeps accepting jobs.
linux-queue-host	linux-amd64, roles always-on, burst	Enforces 8192/16384 MiB, restarts the pre-check runner service, reaps stale GnuPG daemons, and refuses placement while over its watermark.
observe-only	both platforms, every role	Reports against real watermarks and repairs nothing. What a host gets before anybody has decided which repairs it permits.

This exists because of what arming a host looked like without it. On 2026-09-10 at 08:05:43Z the mini's pass read 799 MiB available against its own 2048 MiB low watermark, recorded pressure_active, examined its single declared repair, skipped it as unit_running, and reclaimed nothing: the declaration was report mode and the one repair it named could not fire. A policy that cannot act on the pressure it reports is the shape checks that measure nothing collects, and the fix is a reviewed declaration per host class rather than a better shell line.

Applying one is refused unless it is written for that host, and a policy that ends a graphical session needs its authorization at the call — the catalog names the processes, the operator authorizes ending them:

policy linux-queue-host is written for platforms [linux-amd64] and roles [always-on, burst], and charless-mac-mini is darwin-arm64 always-on
the policies written for it are macos-always-on-host, observe-only

policy macos-always-on-host ends the logged-in session processes WindowManager, Safari, Messages
re-run with --authorize-graphical-session to authorize ending them on charless-mac-mini, or apply a policy that ends none

Both refusals carry error_code=refused, and neither writes anything. A call that both names a policy and edits fields is refused too, because the result would be a document neither of them describes.

Reading a declaration answers whether the host is managed at all, not just what its fields say:

charless-mac-mini: enforces its watermarks and may perform graphical_session, reap_recovery, restart_unit
declared policy: macos-always-on-host

A host whose registry carries a document that is not one of the declared policies reads as written by hand, because a declaration nobody can find in git is a declaration nobody maintains. The comparison is by declaration rather than by bytes, so an empty repair list and an absent one are the same policy.

Refusing placement

refuse_placement is declared, never inferred. While a host that declares it is over its watermark, its capacity publication carries accepting_jobs: false with diag.admission_reason set to memory_pressure_active. The refusal is applied where both writers of a capacity document pass through, so the agent tick and the heartbeat republisher cannot disagree: a host that could not give a runtime its heap stops being selected for the work it cannot run.

A refusal reads the same in three places. stado space report TARGET prints memory placement: refused; the pass's own report carries placement_refusal: "memory_pressure_active"; and stado host gates TARGET shows the published accepting_jobs: false.

When a pass runs, and who runs it

Two writers run the same pass against the same policy, lock and state file: the disk-cleanup --watch unit on its own timer, and the queue agent's janitor task on every tick. Every report names which one produced it in writer, with the writer_version that produced it. The pass takes its own exclusive lock — not the disk janitor's, because a memory pass queued behind a build_caches walk of a large $HOME would arrive exactly as late as that walk. The interval is per writer, so two janitors on one host cannot gate each other out of every pass.

No autonomy mode change is required. This is a declaration the host executes, not an operator gesture.

Outcomes

The outcome vocabulary is the disk janitor's, not a parallel set: never_run, invalid_or_unavailable_policy, lock_busy, interval_noop, healthy_noop, report_only, blocked_running_jobs, reclaimed_target, reclaimed_progress, cap_reached, partial_error, no_eligible_items. caps says which budget stopped the pass. A pass that never reached its repairs emits repairs: null rather than a table of zeros, which is byte-for-byte what a pass that looked and found nothing emits.

Where it is visible

The beacon carries it. Each host's health document gained a memory block holding the last completed pass — its reading, its watermarks, its outcome and its repairs — so the fleet sees pressure without an ssh probe. The beacon publishes the product's own answer rather than running a fourth measurement of its own.

Operating it

stado space report <target>                     # memory beside disk, with the watermark verdict
stado space policies                            # every declared policy, and what each one does
stado space policies <target>                    # the verdict on this host, and the policies written for it
stado space watermark <target>                   # the declaration in force, and whether it repairs anything
stado space watermark <target> --policy macos-always-on-host \
  --authorize-graphical-session                  # arm the host from the catalog
stado space watermark <target> --policy observe-only
stado space watermark <target> --memory-mode enforce \
  --memory-low-free-mb 2048 --memory-target-free-mb 4096 \
  --memory-high-swap-used-pct 80 --memory-max-repairs-per-pass 1 \
  --memory-repair restart_unit --memory-repair-unit com.wisent.stado-precheck-runner
stado disk-cleanup --once                       # one interval-gated pass of both janitors here

--policy applies one declared policy whole; the --memory-* flags edit fields for the host whose watermarks or permitted repairs are genuinely its own. A host that declares no role matches no policy and can only be edited field by field.

space watermark with no write flag prints the declaration and says whether the host declares one at all. With a write flag it rewrites targets[].memory_reclaim through the canonical registry's own compare-and-swap, validating the whole document first, so an incoherent declaration is refused with its own sentence and never reaches the fleet:

registry.targets[0].memory_reclaim.target_free_mb: must be greater than low_free_mb
registry.targets[0].memory_reclaim.repairs: must name at least one repair when mode is 'enforce'; an enforcing pass with no declared repair reports pressure it is not permitted to act on

Stado Desktop reads the same report through the dashboard's api/cleanup.json, the declaration through api/registry.json, and the declared policies through api/memory-policies.json. The Memory screen's Declared policies section shows the same verdict this page describes, lists the policies written for the selected host, and fills the editor with the one an operator picks — with the same authorization checkbox the CLI's --authorize-graphical-session is. The editor then changes watermarks, timing, placement refusal and the repair declarations through the same validated registry write as the CLI, and the review shows the exact patch before it is sent. The repairs field accepts a JSON object; {} removes the repairs, which is refused while the resulting mode remains enforce.

Malformed JSON stays in the editor with an error and cannot be submitted. An invalid complete policy is refused by the API with its validation message; the stored registry is unchanged. Read the declaration again after a successful write to see the saved repair subjects and timing.