Autonomy

What does stado optimize do on its own, and how do you keep it on a leash? This page describes one autonomy cycle, the three modes that decide what a cycle may execute, and the rails that stop it. Flag-by-flag listings live in cli.

One cycle

The coordinator runs the same cycle on every scheduled autonomy tick and on every manual stado optimize run:

stado optimize run

Before anything else the cycle loads the shared control state — an emergency pause or an open circuit breaker forces the rest of the cycle into paused behavior — and refreshes the price book at state/autonomy/cost/prices.json when it is older than the policy's pricing freshness TTL. A GCP price is read from the Cloud Billing Catalog and classified by the Pricing API's product taxonomy: the purchase option comes from the SKU's usage type, the machine family of a core or memory price and the model of a GPU price come from its taxonomy categories, and a machine the catalog does not price whole is composed from its family's cheapest core, memory and GPU prices. A GPU model Stado has no accelerator id for is not priced. If the taxonomy cannot be read the GCP source is marked blocked with the Pricing API's error. Then, in order:

1. Inventory. Direct provider-API inventory, normalized into the provider-neutral resource model, enriched with prices, and published when its content changed. In report mode a cached snapshot fresher than the inventory TTL is reused instead of re-collecting.
2. Placement. Global placement of queued work across local capacity and the enabled cloud providers. This stage is skipped entirely in report mode and blocked when the inventory is incomplete. New cloud spend is bounded by the budget headroom computed from the costs forecast.
3. Resource reconciliation. Findings against the policy's resource rules, executed through the existing immutable resource-plan engine. Every executed action passes the policy's fail-closed authorization and holds a mutation-slot lease, so at most max_concurrent_mutations run at once.
4. Service reconciliation. Registry-declared services are joined against fresh host-beacon and endpoint facts. The evidence rules and the per-evidence reconciliation table are documented once, in operations under "Missing service reconciliation".
5. Advice. Rightsizing, scheduling, storage-lifecycle, network, and commitment recommendations are published. Advice is never executed by this stage; commitments in particular stay recommendations (see costs). Rightsizing needs the policy's idle.underutilized_below ratio; a policy that states none names no resource underutilized, and the rest of the policy is read and run as written. A disk whose age cannot be told — no creation time, one that is not RFC 3339, or an idle.disk_days no duration holds — is listed with why under the pass's unjudged, not left out unseen.
6. Cost reports. Allocation, forecast, and anomaly reports are persisted, past decisions are measured against realized outcomes, and the savings ledger is summarized. Read them with stado cost — see costs.
7. Lifecycle. Retention-aware cleanup of the autonomy layer's own control-plane objects, capped like every other mutation (deleted, bytes, capped are logged).

The whole layer is deliberately not a parallel control plane: it reuses the canonical queue storage, provider adapters, resource planner, and operation journal, adding coordinator stages over the same state.

Modes

The policy's mode decides what a cycle may execute. Read-only actions are always allowed; financial commitments are denied in every mode.

Mode	What executes
report (default)	Everything is observed and recorded, nothing is mutated. Placement is skipped, resource plans are not executed, and service repairs are recorded as planned with the mutation they would have made.
enforce-safe	Reversible actions on owned or adopted resources execute. New cloud placement gets zero spend headroom, so placement cannot add cloud cost. Destructive actions are denied.
enforce-owned	Reversible resource actions execute only when a resource rule allows them (service repairs need no rule), destructive actions only when an explicit rule allows them, and this is the only mode in which new cloud placement receives a nonzero budget.

With no persisted policy document, load_policy returns the default: report-only, version default-report-only.

Safety rails

• Emergency pause. stado optimize pause <REASON> stops new autonomous mutations; stado optimize resume clears it. The pause lives in shared control state, so every reconciler in the fleet honors it, and the reason and actor are recorded.
• Circuit breaker. Fed only by mutations that actually executed against a host: a failed execution increments the consecutive-failure count, a successful one resets it to zero. Reaching circuit_breaker_failures opens the breaker for circuit_breaker_cooldown_seconds, which the next cycle treats exactly like an emergency pause. Planning and read failures never feed it.
• Action caps. max_actions_per_tick and max_actions_per_provider bound each cycle; max_deleted_bytes_per_tick optionally bounds deletion volume.
• Per-service mutation leases. Before repairing a service, the reconciler takes a lease on service:<host>:<unit> with the policy's decision TTL. A second reconciler reaching the same service is recorded lease_blocked ("another reconciler owns this service mutation") instead of racing the first. Resource mutations take numbered mutation-slot leases the same way.
• Fail-closed authorization. Every non-read-only action is denied unless the policy allows it: incomplete inventory (when require_complete_inventory is set), resources that are not owned or adopted, protected production and stateful resources without an explicit rule, and actions estimated above max_single_action_usd are all refused with a recorded reason.

Ownership labels and resource kinds

Each inventory record reads who it belongs to from exactly one label (a tag on AWS and Azure). Another spelling of the same idea is not read, so a resource labelled owner instead of stado-owner has no owner.

Label	Read as
stado-owner	the owner; a resource with one and no other mark is observed
stado-workload	the workload it serves; such a resource is never idle
stado-environment	its environment; production is protected by protect_production
managed-by = stado	Stado's own resource (owned); the readers set it on resources Stado created
stado-adopted = true	brought under Stado by an operator (adopted)
stado-stateful = true	holds data, so mutations need allow_stateful_mutation

Every record also carries a resource_type and a lifecycle phase. The type is one of instance, persistent_disk, managed_disk, volume, snapshot, public_ip, machine_image, object_bucket, object_storage, container_registry, database, scale_set, managed_instance_group, load_balancer, network_interface, reservation, service_account, build, schedule, function, service, local_host, or resource for a provider type no reader names. A resource rule's resource_type must be one of them; any other word is refused when the policy is replaced. The phase is running, stopped, ended or unknown, placed by each reader from its provider's own state (the EC2 state enums, Compute Engine instance and resource statuses, Azure power and provisioning states). Schedules stop a running instance and start a stopped one; an ended resource is never classified idle; a stopped instance is idle by definition.

Reports and status

Every autonomy object is rooted under state/autonomy/ — the object gateway authorizes writes by prefix, and operations records the outage that placed it there. Service reconciliation writes state/autonomy/services/latest.json plus an immutable state/autonomy/services/runs/<timestamp>.json per run; cost reports live under state/autonomy/cost/.

stado optimize status

prints the mode, emergency-pause state and reason, circuit-breaker state with its consecutive-failure count and open-until time, the policy version, the latest inventory snapshot (resource count, completeness, snapshot id), the decision count with active leases, and the latest forecast, anomaly, savings and service-reconciliation reports. --json returns the same document machine-readably. A stored policy this Stado cannot read is refused naming its object and what could not be read in it (state/autonomy/policy.json is not an autonomy policy this Stado reads: missing field ...); the coordinator's tick logs the same sentence and holds unpinned dispatch closed until the policy reads. Each placement or resource decision is immutable and explainable:

stado optimize explain <DECISION_ID>

Policy

The policy is one versioned JSON document. Inspect it, then replace it atomically:

stado optimize policy show
stado optimize policy apply --file policy.json --expect-version <VERSION>

show prints the current version alongside the document. apply validates the document fail-closed and writes it compare-and-swapped against --expect-version; once a policy exists, --expect-version is required, so two operators cannot silently overwrite each other. There is no separate undo: rolling back is applying the previous document again, expecting the version you are replacing. Budgets inside the policy are covered in costs.