Storage root handoff and space reclaim

Storage root authority handoff

Inspect a durable handoff before choosing an action:

stado repair stado --step storage-root <target> --transaction <id> --phase status --json

The CLI and Desktop enter the same storage reconciliation report/outcome implementation. Desktop sends GET /api/host/storage-root-reconcile?target=<host>&transaction=<id>&phase=status for Status and POST to the same path for Run, Resume, Rollback or Finalize; neither method accepts a body. GET requires the independent storage-reconcile-read registry action and POST requires storage-reconcile-apply.

HTTP 200 means a product report exists, not that the requested phase succeeded: the response preserves exit_code, refusal and the complete report. Malformed, duplicate, empty or unknown query fields, a body, or the wrong method for a phase return 400. A missing or unauthorized action returns 401; unavailable verifier state returns 503. Failure before a product report exists also returns 503, with STORAGE_RECONCILIATION_FAILED and the actual error. Desktop captures the selected dashboard address and host when its sheet opens, shows the equivalent CLI command without launching it, and retains the HTTP status, raw API bytes, product exit, refusal and full report after errors.

Run, Resume, Rollback and Finalize first return accepted after the target-resident native worker owns the globally locked operation. Acceptance is not completion. Neither the API nor Desktop issues Status or another phase automatically; explicitly read Status for the same transaction ID after every accepted action.

Run and Resume automatically capture the queue and native-service state, acquire placement leases, pause and drain the queue, stop storage writers while retaining the transport and current runner, acquire the storage-write fence, and checkpoint both complete physical roots before constructing the additive union on fixed primary root A. Checkpoint and apply are internal stages, not operator-selectable phases.

The object API's captured loaded route at the fenced preflight pins one conflict winner for the whole transaction. If it serves A, A's bytes and metadata win every shared-path conflict and only B-only objects are imported. If it serves B, B wins shared-path conflicts and A-only objects remain in the union. The immutable checkpoint receipt records that source-selected authority; data application, post-commit correlation and the effective lifecycle snapshot must all agree with it.

Typed lifecycle handling examines records that become newly authoritative: B-only lifecycle records when A was serving, or A-only records when B was serving. This prevents an imported queued job, retained outcome, or live lock from bypassing the same classification that guards the rest of the handoff.

Before the data-commit boundary, Rollback is resumable and restores A to its exact physical checkpoint, including removal of transaction-created imports and now-empty directories, before restoring the captured route and native service state. After that boundary, Resume the same transaction to complete its recorded activation rather than starting a second transaction or reversing the winner.

Run or Resume owns the remaining core stages through declared-runtime activation, exact native-service and queue restoration, write-fence release and an activated_pending_lifecycle receipt. Ordinary lifecycle handling must then resolve the typed queued cancellations and retained outcomes. Select Finalize only after that cleanup; it validates the final typed observations and records complete, which a later explicit Status read proves.

Space reclaim

Every host follows one built-in rule, and nothing about cleanup is declared in the registry: when the volume holding the host's home is at least 80% used, the janitor deletes everything the fleet put on the host; below 80% it deletes nothing. It keeps ~/.ssh and everything outside a cleaner's area (the user's data), a backup copy whose primary cannot be proven identical, the release versions the registry declares or the host runs, and the tree of a running job. The cleaners, what each takes and what each keeps are listed in Space.

The fleet's space-reclaim rule is not a per-target setting. A stored generation with retired memory settings still runs the rule; an ordinary registry write removes those settings.

When a pass runs

Two writers run the same pass against the same lock and state file: the host's standing health watch at its declared --health-interval-seconds, and the queue agent at its own poll cadence. Every report names its writer and binary version so the last outcome is not mistaken for a host-wide verdict.

Under 80% a pass is one statvfs: it writes healthy_noop and opens nothing. At or above 80% every cleaner runs over its whole area with no age, count, byte or scan limit, local_snapshots last so the blocks the others freed are returned on a Mac. The report carries rule (full_percent, used_percent, triggered), total_bytes, free_bytes_before, free_bytes_after, used_percent_before, used_percent_after, per-cleaner scanned, eligible and deleted counts with exact skipped reasons, and errors.

An outcome is one of never_run, healthy_noop, reclaimed_below_threshold, still_full, no_eligible_items, blocked_running_jobs, partial_error, report_only, volume_unreadable, lock_busy, lock_busy_workloads, lock_busy_unattributed or lock_recovery_report_only. The kernel's live lock is never taken over because of an old owner timestamp; lock_recovery_report_only means a retired lock is still held and nothing was scanned or deleted.

While the volume is full the agent claims no new work (admission_reason: disk_pressure_active) and writes disk-cleanup-turn.json beside the lock, so running jobs drain and the next pass gets the exclusive lock.

Operating it

stado space report <target> --json
stado space reclaim <target> --stage registry_cleanup --dry-run --json
stado space reclaim --local --dry-run
stado space reclaim --local --apply --reason "reviewed cleanup"
stado space reclaim <target> --apply --reason "reviewed cleanup" --json

space reclaim --local --dry-run runs every janitor scanner and deletes nothing: it reports what a pass at the threshold would remove. An applied fleet reclamation records its reason and result on the affected host. space report retains the full recorded pass under cleanup_state.report. See Space for every field and refusal sentence.

The reclaim engine runs inside the host's Stado process: the queue agent checks it on its own poll and the host health watch follows its declared cadence. stado space report <target> names the last pass and promise. stado service list --json reports the owning unit's observed state; a silent or unowned process is not evidence that a reclaim pass worked.