Service reconciliation receipts

stado optimize run and scheduled autonomy ticks record service reconciliation under state/autonomy/services/. The immutable receipt is runs/<timestamp>.json; latest.json selects a receipt for stado optimize status, GET /api/service/reconciliation and Desktop's Services → Reconciliation view. The document remains schema_version 1, with its mode, created_at, summary and complete per-service outcomes.

Shared read contract

The CLI, report API and version-pinned writer decode the same native report model and validate the supported schema and RFC3339 timestamp. Missing required fields, incompatible field types or modes, invalid metadata and unreadable JSON remain errors rather than arbitrary JSON presented as a healthy report. Ordinary display reads retain configured read failover and report the failed operation, object and original cause without manufacturing an empty report.

No stored report remains API HTTP 404; an unreadable or unsupported report is HTTP 503, and CLI status returns its actual error. Desktop retains the API's sentence in The report could not be read, rather than displaying stale success underneath it. Its read-only outcome list shows native beacon/endpoint evidence and uses snapshot positions, keeping repeated rows visible without delimiter IDs or an extra array. An empty list means no per-unit outcomes were recorded, not that no service was evaluated. Layout uses platform content sizing.

Version-pinned publication

Before this service reconciliation stage acts, it reads the prior latest report and its opaque storage version together from the write authority. The shared decoder applies the same report checks as display readers; an empty conditional version additionally refuses the stage before repair. An absent report is different from a failed read.

The stage serializes its completed report once and creates the immutable run receipt without replacing an existing object. It then selects latest only by:

• Compare-and-swap against the exact version read before the stage's work.
• Create-if-absent when that prior read proved latest absent.

A concurrent change refuses publication with StorageConflict; there is no retry, read-then-unconditional replacement or fallback upload. This includes another writer creating latest after an absent read. The immutable run remains acknowledged when it was created before the conflict. A losing run can have a later timestamp and still lose that race; a later independent run begins with its own snapshot.

If a run's created_at is not later than the observed prior report, it is archived but does not replace latest or announce superseded transitions. Successful latest publication compares transition alerts against the same version-pinned prior report, not a separate unversioned read.

Ordering follows the reported start timestamps, parsed as instants rather than compared as strings. It does not establish globally synchronized host clocks, a transactional snapshot of the fleet, or rollback of host actions. A run that finishes later is not necessarily the report with the later created_at.

Versioned reads always consult the primary write authority, even when ordinary reads may fail over. A backup's opaque token cannot authorize a conditional write to primary. See Storage authority.

Listener handoff observations

A failed replacement-process inspection is unknown evidence, not a fabricated empty process. Listener standing and the repair-exclusion guard use the same strict query. Failed inspection or read-only handoff standing reports evidence_read_failed; it authorizes neither repair nor handoff/restore action, including after lease admission, and is not fed as a failed host mutation. Lease finalization still runs. Errors after a handoff mutation is chosen retain their operation cause; a failed mutation does not prove rollback or no change.

Bootout and retirement evidence

For one exact native unit, the scoped command is:

stado service bootout LABEL --host HOST --domain user --json

The positional label, registry host and selected native scope identify the request. The JSON receipt includes evidence: confirmed acknowledged operations, unconfirmed requested operations whose effect could not be established, the stated failure provenance, and independent host_mutation and host_failed facts. A pending request proves neither execution nor no change. Handoff-record acknowledgements remain distinct from verified native unit mutations; a bootout scope enters confirmed only after its required postcondition read succeeds. Earlier scopes survive later read or action errors. Malformed later fields retain validated scope acknowledgements and native refusal phases without claiming completion. See Incomplete bootout receipts.

Launchd job absence requires a readable containing domain, including after bootout. A failed or unavailable domain read is an observation refusal, not an absent job or permission to fall through to another scope. No login or domain is created to obtain that observation; earlier effects remain in the receipt.

Native failure provenance	Reconciliation classification
preparation	preparation_failed
observation	evidence_read_failed
mutation or postcondition	repair_failed
execution_unconfirmed	mutation_unconfirmed

Classification is not inferred from the diagnostic sentence. A read refusal does not itself count as a failed host mutation. An opaque mutation transport or unusable action result keeps the request unconfirmed rather than guessing whether execution failed. Known native failures remain independent of the row's primary cause. A failed row can therefore have changed: true and count both an acknowledged change and a failure without claiming completion. Earlier repair and later partial-retirement acknowledgements retain their order.

JSON refusals print their receipt and still return non-success. Their command state is refused, or unconfirmed when execution cannot be established; printing JSON does not turn either into success. Native booted_out means an action was acknowledged and its postconditions read back. absent means no exact unit was found; systemd already_correct means it was already inactive and not enabled, so no stop/disable was requested or credited as a change. Retirement retains already_correct rather than claiming that such a unit is absent; an inactive, disabled installed definition can remain when there was nothing to withdraw. The exact label is inserted after scope-template expansion, never reinterpreted as a template token.

Autostart inspection requires complete, exact-scope native evidence. Native read failures never become an empty set of enabled scopes; no-op setters require a usable matching observation. See Autostart evidence for manager states, refusals and retained lifecycle progress.

Predecessor completion and rollback

Replacement activation and API-listener takeover judge predecessor completion from typed failure and pending-effect evidence, not from a display word such as failed. An unknown prerequisite read therefore blocks starting the replacement too. Its error says that safe retirement could not be established; it does not invent a still-running predecessor.

Reversible retirement records a rollback handle only for known native changes, known native failures that may have effects, or unresolved native requests. Read-only preparation/observation refusals and matching no-ops do not trigger a compensating load of an untouched predecessor. Earlier changed units remain eligible for recovery when a later unit's prerequisite observation fails.

Recovery captures native activation intent and definition paths separately from autostart policy, bound to the observed manager and calling account. It does not start originally inactive systemd units or assume a registered launchd job needs a running PID. A failed replacement must be verified inactive before old units are reloaded. Recovery outcomes retain partial changes and refusals rather than asserting that every predecessor was given back. See Recovering retired predecessors for preparation, exact-scope restoration, Desktop presentation and limitations.

Full-scope retirement proof

Automatic predecessor retirement requires complete final observations of native inactivity and disabled autostart in every expected system/calling-account scope, bound to one observed manager and UID. A successful scoped stop or disabled policy alone is not completion.

Before action, a system definition together with an active calling-account scope refuses retirement: system-preferred bootout any cannot establish full inactivity, and the same label does not prove that broader stopping is safe. An installed inactive systemd definition still counts as native presence. Public bootout scope selection is unchanged; no broader stop is substituted.

Later drift, remaining activation or an unreadable final observation refuses completion while retaining earlier acknowledged effects. The same result feeds ensure failures and autonomy receipts, including Desktop reconciliation rows. Explicit service retire/remove retain their separate declared-unit contract; these automatic predecessor checks do not replace that workflow. The final reads are observations, not an atomic native-manager transaction.

Retirement admission

During autonomy, ordinary predecessors and idle strays use the same mutation gate as repairs and listener handoffs: live pause, action permits, host breaker and a lease for the exact old unit. Read-only kept/unknown predecessor decisions do not acquire a mutation permit. The native predecessor decision is repeated under the lease. Strays are rediscovered there and their idle, undeclared and unowned status rechecked; a newly live, declared or catalog-owned unit is kept.

Permits count admitted attempts, not successful host changes: a decision can become read-only after admission. Known no-ops and record-only changes do not reset the host breaker, and an unconfirmed request is not counted as a native failure. Partial acknowledgements and lease-release failures remain visible together. Report/emergency-paused mode still executes no retirement. These outcomes use the same CLI status, report API and Desktop reconciliation rows; no separate graphical action or alternate mutation path is introduced.

Interrupted cycles

stado optimize run can fail after earlier units already changed. Once service inventory is known, every mutation stage appends into the same report owner. A later control-read, lease-acquisition or outcome-recording error does not discard preceding rows or the current acknowledged native result. Lease-release observation is applied even when recording the native outcome also fails.

An interrupted pass adds an alert-eligible cycle_failed row for the failed product operation. Its action identifies control loading, lease acquisition, outcome recording or the actual catalog/role prerequisite; host and unit are retained when known. Empty host/unit fields indicate cycle scope, not a native identity. This row's changed: false is not proof earlier changes were absent or a failed write had no effect. Native acknowledgements remain in their own rows; failed acquisitions/writes stay unconfirmed. Retirement reuses the cycle's validated host-catalog name.

services is the known inventory count. Other tallies and rows describe progress observed before interruption, not evaluations of unvisited services. If inventory itself cannot be read, mutation stages are not entered and no empty inventory report is fabricated; the actual operation/cause is logged.

The partial report uses the same immutable-first, version-pinned publisher. An acknowledged partial publication names its actual immutable_object and latest_selected in the cycle log. It does not turn the failed command into success: the original storage error is still returned. If publication also fails, its operation/key/cause and all retained rows are logged separately, without replacing that original error. No write retry or blind latest upload is introduced. Transition alerts still require acknowledged latest selection.

CLI status, the report API and Desktop's reconciliation view expose these same partial rows. Read the cycle_failed row and timestamp before treating a readable report as a completed pass.

Managed stop preparation

Managed-service stops validate the credential requirement, unit and path and render the native prelude/body before any privileged bootout. Execution consumes those prepared inputs for the bound target and checks the existing stopped postcondition; a prepared value alone is not execution or postcondition proof. Credential input remains on stdin, with one execution buffer shared by the existing privileged jobs rather than a new copy for each job.

An explicitly declared per-login recovery label is stopped only after observing Darwin, through the calling account's native user scope, keeping a system sibling. The scope's absent/removed result is verified before reporting success. A failed UID read or invalid UID is refused with the actual exit/output cause instead of being mistaken for an empty namespace; unsupported managers are not substituted.

Launchd inspection and verification failures are not absence. Native success and the shared declared absence statuses determine the result; other exit/output is a refusal, including after bootout. An unreadable system scope does not authorize silently falling through to a user-scope action. Resident handover also requires exact native services-table membership, never a matching metadata value.

Standby pre-stop refusals

A standby's missing authoritative unit or managed-service declaration is declaration_incomplete, with the reason nothing was stopped. Target resolution, missing required credentials or invalid local stop definitions are preparation_failed, naming the host, unit and actual cause; they do not feed a failed host mutation to the breaker. Resolved or unjudged rechecks keep their existing non-mutation handling. Native stop/refused-postcondition errors retain the stop's own cause; neither the stop_standby action label nor a failed attempt proves the unit stopped.

Actions, transitions and publication failures

What a planned action label proves, how an alert's identity is read, and what a failed publication retains are on Reconciliation actions, transitions and publication failures.

Return to Missing service reconciliation for repair, ownership, lease-finalization and failure-classification behavior.