What the backup store is and is not

4. What the backup store is — and is not

The configured backup backend is a disaster-recovery write mirror and a conditional read fallback, not a second write authority. Primary mutations commit first and are mirrored afterwards. Ordinary clients may retry a read that failed at the primary against the backup, but a successful absent answer from the primary remains authoritative. Every such retry is said on standard error as [storage-failover] primary <operation> failed for <path or prefix>: <the primary's error>; read from the backup instead, so a host whose primary refuses every read while the backup answers is visible from its log; the backup's own answer, a refusal included, is returned as it is. The object API server and other authority-sensitive readers use primary-only mode and return the primary error. No read path promotes the backup to writer. Cross-provider DR remains the explicit fenced migration in steps 2–3.

The coordinator's fleet-shape sweep checks, on every pass, that the resolver and the config file agree about the mirror (replica-cannot-resolve): a file that declares storage.backup.backend while the resolver answers that no replica is configured is a finding, since one half of the binary then writes to a replica the other half says does not exist; a file that declares none is a note (<primary>: no replica declared); and a config file the coordinator cannot read is a finding of its own (the config file cannot be read, so the declaration cannot be compared with the resolver's answer: …), never a quiet pass. The sweep's health check (health-green-boundaries-down) fires only when the object API's healthz states ok and degraded as booleans with at least one boundary stated closed.

For the fixed ~/.stado/local-storage (A) and ~/.stado/local-backup (B) roots on a host, use the narrower resident transaction. It requires the running object API to report which of these exact paths it actually constructed as its primary and mirror; a different root is refused rather than substituted.

stado repair stado --step storage-root --target TARGET --apply --json

The step starts the fenced, source-preserving reconciliation under one durable transaction and then reads that transaction's receipt until it settles, so one command covers the run and status phases of the protocol. The receipt reports the checkpoint, the verified object count, the writer fence and the accepted resident owner; without --apply the command reports what it would do and writes nothing.

The protocol itself has five phases — run, resume, status, rollback and finalize. run and resume address the same durable ID; resume never creates a second checkpoint or transaction identity. status only reads the receipt and the current native owner. rollback restores the exact prior route before the data-activation boundary, and finalize records completion only after the ordinary coordinator has finished typed lifecycle cleanup.

Stado Desktop drives those five phases individually from the selected host's Storage reconciliation sheet, through the authenticated host route rather than a second implementation. Each action uses the sheet's transaction ID and interprets the same durable receipt the repair step reads. Switching between the command and the sheet does not create a second workflow: use the same target and transaction ID when resuming or inspecting an operation.

Preflight identifies the real runtime

Before changing service or queue state, the running object API must report its constructed A and B roots, which root is primary, which root is its mirror, its serving PID, and the write-fence protocol it implements. The transaction also captures the object API executable and requires it to match the staged binary for the exact Stado release declared for the target. A plist path, a configured default, or a listener alone is not proof of the storage authority or runtime.

Before a lifecycle fence exists, a new run or resume selects that exact Stado release from the current canonical registry, not from an older captured target declaration. If the registry cannot be read, the attempt refuses to fence writers; it does not substitute a cached declaration. Once the lifecycle fence exists, every later phase remains pinned to the release and runtime captured in that fence.

Fence first, inventory second

The resident owner pauses and drains the queue, stops the other writers, and then acquires the reconciliation's exclusive filesystem lock against the shared lock used by ordinary object writers. It records a durable write-fence intent before treating the fence as held. The object API can continue serving reads in this interval, but object writes are refused with HTTP 503 and the name of the transaction holding the fence. The public status API exposes the same condition as storage.write_fence and reports the constructed backup as storage.backup.

Only after that fence is effective does Stado capture complete A and B inventories and compare authenticated HTTP reads byte-for-byte. An inventory taken earlier could race a queue claim, lease renewal, or mirrored write and is not preflight evidence. If B differs from A and the API's constructed authority is A, the transaction refuses a B-winning copy. This also protects A's object metadata when the body bytes happen to match.

The object API is stopped after those read-path comparisons and before the immutable physical checkpoints are made. Checkpoints preserve physical metadata as well as contents, including the recorded modes and ownership needed for exact restoration. HTTP comparison answers a different question: whether clients receive the same bytes. It deliberately ignores filesystem mode, uid, gid, and other physical metadata. A byte match is therefore not a substitute for a full physical checkpoint, and a metadata difference alone is not an HTTP-byte conflict.

The additive data step copies only the qualified ecosystem/ objects and matching metadata sidecars selected by the transaction. A-only objects remain and B is not modified. The lifecycle fence captures both the exact prior primary and the exact prior mirror; rollback never assumes that B was the sole prior route.

Activation, rollback, and fence release

Before any checkpoint exists, rollback can unwind a preparation whose queue was drained and whose placement leases were all captured. It restores the captured prior route and service state, releases any write fence, and restores the queue. Earlier preparation is refused with preparation rollback requires the recorded drained queue and complete placement leases; it has not acquired the storage write fence. An interrupted preparation rollback remains a rollback when the same transaction is resumed.

After checkpointing, rollback remains possible only before data commit. It restores the captured prior route, not an assumed B-only layout. The source roots are checked against their saved inventories; the immutable checkpoints remain evidence, and a partial additive copy in inactive A is retained rather than silently replacing the whole root.

Forward activation proves the constructed A primary and B mirror and compares the complete qualified API byte set before releasing the write fence. A checkpointed rollback proves the captured prior route and its corresponding byte set. A preparation rollback instead reports constructed-runtime-without-data-mutation: no data-copy step occurred, so this is not a claim that a full byte comparison ran. After the applicable proof, Stado releases the write fence, restores other writers, and restores the queue.

Repeating an active run or resume observes the existing owner instead of starting a competitor. An incompatible action is refused, for example native reconciliation is already executing run; cannot accept rollback; the refusal does not queue rollback behind a forward activation. If an owner was recorded as executing, status reports it that way only while the native manager still reports the same PID. Otherwise it reports interrupted, or unobserved when the manager cannot be read, while retaining the stored value as recorded_status.

Read the whole receipt

The lifecycle-fence schema is version 5. Its roots object records primary, backup, prior_primary, prior_backup, and runtime; those captured values, not conventional directory names, define activation and rollback. write_fence records status, intent, acquired_at, and released_at. A missing released_at means the fence has not been released. preflight_evidence is nullable until fenced inventories and API comparisons have actually been captured; null never means that both roots were compared or found equal.

A mutating command may return accepted as soon as the resident native owner accepts the operation. That is a launch acknowledgement, not success, activation, restoration, cleanup, or completion. Continue reading status for the same transaction ID: reconcile owner observation with recorded_status, inspect the preflight evidence and root identities, and confirm the final A byte proof and released write fence. On the forward path, only finalize after ordinary coordinator cleanup writes complete. For a rollback, require the durable restored/rolled-back outcome and released fence; an accepted rollback is no more complete than an accepted run.

On macOS, saved autostart state accepts both launchd's true/false overrides and its disabled/enabled names. Exact native-unit restoration reads back permissions, owner, group, and digest. Snapshot access may use an already available sudo -n capability when a captured root contains files the owner cannot read directly; it never asks for a password or changes the live roots' permissions.

See configuration for the storage layout and stado repair for the generated option reference.