On this page
Costs
What does the fleet cost, and what stops autonomy from
spending more than you allow? Budgets live in the autonomy policy, the
forecast decides when new cloud spend is blocked, and two command groups read
the money: stado cost for attributed workload cost, stado billing for
what the providers themselves report. Flag-by-flag listings live in
cli.
Budgets in the autonomy policy
The policy's budgets block holds five optional limits. Each must be a
finite non-negative number; an absent limit does not constrain.
| Field | What it gates |
|---|---|
hourly_usd |
Hourly spend ceiling. Headroom for new cloud placement is this minus the current hourly burn. |
daily_usd |
Daily spend ceiling. Divided by 24 it also caps the hourly headroom — the tighter of the two wins. |
monthly_usd |
Monthly ceiling. Headroom is this minus the projected end-of-month total. |
max_single_action_usd |
Any single autonomous action estimated above this is denied by policy authorization. |
max_commitment_usd |
Gates the commitment recommendation only. Purchases classified as financial commitments are denied in every mode: they require an operator-approved immutable plan. |
Owned hardware is priced at the policy's top-level local_hourly_cost_usd,
the hourly cost the operator states for running a job on a host the fleet
owns. With nothing stated, an owned host has no price: the optimizer refuses
its offer with owned hardware has no declared hourly cost: set local_hourly_cost_usd in the autonomy policy, and stado cost reports its
jobs as unpriced. It is never priced as free.
Apply budget changes like any policy change — stado optimize policy apply --expect-version (autonomy).
The forecast and what budget_exceeded blocks
Every cycle builds a forecast from the attributed cost allocation, the policy
budgets, and the last billing snapshot: current hourly burn, end-of-month
projection, hourly/daily/monthly overruns, and — when the billing snapshot
carries a credit balance — the credit runway in days. budget_exceeded is
true when the hourly or daily overrun is positive, or the end-of-month
projection exceeds monthly_usd.
An exceeded budget blocks exactly one thing: new cloud placement. The
cycle logs the guard with all three overruns and gives the placement stage
zero new-cloud headroom. Existing work keeps running; reconciliation, advice,
and reporting continue. Independently of the guard, only enforce-owned mode
ever receives a nonzero new-cloud budget — report and enforce-safe place
no new cloud spend regardless of headroom.
Reading workload cost: stado cost
Per-job and per-batch cost reporting from observed wall-times:
| Command | Answers |
|---|---|
stado cost report |
$ spent per target_kind and per model, from completed jobs priced by the live quotes in prices.json; jobs no quote prices are counted as unpriced, never as free. A job that records neither a cost-tracked provider nor an instance reference is grouped under (no provider recorded) (its row's target_kind is null in --json), and one whose command carries no --model under (no --model in the command). |
stado cost estimate <BATCH_FILE> |
Projected total $ for a batch file, using the average cost of priced completed jobs. |
stado cost prices --json |
Recorded provider quotes, their observation times and source failures. |
stado cost quote --json -- JOB_ID... |
Exact quotes or explicit refusals for observed job allocations. |
stado cost allocation |
The attributed provider/owner/workload cost ledger. |
stado cost forecast |
Current burn, month-end projection, budget, credit runway. |
stado cost anomalies |
Active cost and resource anomalies. |
stado cost savings |
Predicted versus realized savings, from measured decision outcomes. |
The reports behind these are persisted by the autonomy cycle under
state/autonomy/cost/ (prices.json, allocation.json, forecast.json,
anomalies.json, savings.json), so they are readable even between cycles.
No rate is written into Stado. A cloud job is priced at the provider's live
quote for its machine and accelerator; a job on owned hardware at the autonomy
policy's declared local_hourly_cost_usd. The same book holds a job's
--max-cost-per-hour when a machine would be rented for it: a quote above the
cap, or no quote at all, keeps the job from being dispatched, and the
scheduler logs which. The local pack ranks jobs by what the cheapest live quote
for their accelerator would have charged. A claim on a host that is already
running is not judged against a cap, because it rents nothing.
Reading sourced allocation quotes
stado cost prices --json
stado cost quote --json -- JOB_ID OTHER_JOB_ID
Both commands are read-only. They neither enable autonomy nor fetch a new
provider book. prices reads the book the producer recorded; observation time
and each source's state are part of the result. A book is not an invoice or a
capacity reservation, and an absent quote is not a zero price.
quote reads canonical job state, recorded worker identity, inventory and
that price book. It does not use the requested provider, machine name or
purchase preference as proof of placement, and it does not use the legacy
estimate heuristic. local@HOST is an agent transport reference: cloud agents
use it too. Only an explicit local worker-policy record selects the declared
local hourly policy; that record does not certify a physically local host.
Before claiming work, AWS agents record their metadata account, region and instance ID; GCP agents record project, zone, name and numeric instance generation. The report retains the worker's observation time or its actual identity-read error. AWS inventory also resolves the credential's actual account through STS rather than accepting an account label from configuration. Azure native worker identity is unavailable in this revision and remains an explicit missing-identity refusal, not a requested-provider or local fallback.
A cloud allocation needs one inventory resource matching that worker's scope and generation, its actual machine and purchase model, and one complete matching USD hourly quote. Source state, resource identity and timestamps remain in the response. An inventory or quote observation older than the claimed execution is refused; a resource created after the claim is not silently treated as the original resource. A same-name replacement with a different generation cannot supply the old worker's quote.
AWS Spot collection retains the current effective rate for each machine and availability zone, including the price change before the requested instant. It does not take a median across historical or regional prices. AWS on-demand collection retains flat hourly price dimensions rather than choosing the minimum; conflicting eligible dimensions make the allocation quote ambiguous. Old books without that provenance are not accepted as those observations. For an identified Azure resource, matching uses its observed VM size, OS and documented purchase priority. GCP component-only SKU books cannot establish a complete machine quote; this interface refuses them rather than copying a machine-shape conversion table.
The report carries complete, price-book and inventory source metadata, and
one quotes entry per requested job. Each entry retains allocation, a
quote or an error. A missing job has no invented allocation. Missing,
ambiguous, incomplete, unsupported and stale observations retain their actual
refusal. Exit zero means the report was read, not that every job was priced;
a consumer must inspect complete and each entry before charging anything.
Consumers with a narrower amount representation must also refuse values they
cannot represent exactly, not round them or silently use zero.
In Stado Desktop, open Earning → Provider prices. Read prices shows the
book and source failures. Enter the exact job IDs and choose Read allocation
quotes to see the same per-job observations and refusals, including worker
origin, account/project/subscription, resource generation and identity errors.
For a cancelled execution, a scoped provider-removal observation distinguishes
the recorded generation from any replacement and retains the current resource
state. See Machine interface
for why a stopped VM or a terminal job is not removal. Both actions use the
selected native API source through /api/operator/run, retain the full
stdout/stderr receipt and exit status, and discard results from a previously
selected source. They do not invoke a local CLI fallback.
The real CLI/native-API qualification source is tests/cost/journey.rs in the
Stado repository (cargo test -p stado --test service-journeys cost::). Its live
cases require STADO_COST_QUALIFICATION_CONFIG, an authorized profile;
STADO_COST_QUOTED_JOB_ID, a running local-policy allocation whose request
named a cloud provider; and STADO_COST_QUOTED_CLOUD_JOB_ID, a running cloud
agent with a complete sourced quote. The normal producer must have recorded
the required inventory and prices, including the explicit local policy.
The cloud case checks scoped worker identity and generation despite local@
transport. Missing prerequisites are recorded as blocked, not replaced with
seeded provider responses. Reports under .build/desktop-api/ retain revision,
binary, command, API and configuration evidence.
Expansion budget and payback
stado fleet expansion plan --budget-usd 10000 --horizon-months 24 --json
compares sourced option bundles with the current fleet needs. The separate
expansion planner includes acquisition, running cost,
avoided spending, additional margin, delivery delay and payback. It saves
the original assumptions and result; unknown prices never become zero.
Its budget covers total new expenditure over the selected horizon, not just
the purchase price, and does not change autonomy permissions or buy anything.
Desktop exposes the same operation under Fleets → Plan expansion….
Reading provider billing: stado billing
Cross-cloud costs, grants, burn, and credit balances, published as the
billing snapshot at billing_health/credits.json:
stado billing show
stado billing refresh
stado billing watch --mail-days <days> --mail-organization <org>
show reads the last snapshot the coordinator published; refresh queries
the billing providers now and publishes a fresh one. watch is one watchdog
pass: it refreshes, evaluates the credit balance AND account health, and
alerts on transitions — a section with no balance figure at all
(no_credentials, error) is a finding, not a gap. Beside the APIs it reads
the provider mail Skrzynka received in the last --mail-days days, in the
mailboxes of the Skrzynka organization --mail-organization names (both
required: Stado assumes no notice period and Skrzynka assumes no
organization), and an unreachable mailbox is reported as
unavailable without failing the pass. What each notice is about (Azure, a
startup programme, credits, billing, an invoice, a quota, security) and what
it asks the operator to do is read by a model through Oko's schema-bound call
(oko gateway object), in any language: the message's categories and its
action_signals sentence are the fields of the one object that reading answers
under a strict JSON Schema, never text searched for them. A notice nobody could read
carries reading_error with the cause and counts as action required, so it
reaches the operator unread rather than as harmless. The schedule that runs it is its
cadence, for example stado schedule create --cron '*/5 * * * *' 'stado billing watch --mail-days <days> --mail-organization <org>', or cron on a machine in another cloud: a
collector that dies with its provider cannot warn you about that provider. A
pass that cannot read the previous snapshot, or cannot store the new one,
fails with that error and sends no alert, because a transition cannot be told
from a standing condition without it.
The deployment declares where its GCP billing export is and when credit
counts as exhausted; Stado builds in none of them. WC_BILLING_DATASET and
WC_BILLING_TABLE (config keys billing.dataset, billing.table) name the
BigQuery export; without them the GCP section is config_error with the BigQuery billing export is not declared: set WC_BILLING_DATASET and WC_BILLING_TABLE, and stado blast-radius skips the export probes.
WC_BILLING_NET_ALERT_USD (billing.net_alert_usd) is the net spend in USD
above which GCP credit counts as exhausted and below which an Azure balance
is low. Without it each section carries credit_depleted: null and the
reason under net_alert_detail, and watch raises one not judged warning
per provider instead of comparing the balance with an amount nobody chose.
The historical GCP project
Billing for the historical GCP project is detached on purpose and stays
detached; it is not a defect. A 403 naming billing — accountDisabled on a
GCS or provider API call — means some part of the system still depends on that
project, and the GCP dependency is the thing to remove, not the billing to
restore. The billing service principal comes from Skarbiec and nowhere else, so
no fallback can quietly recreate the cross-cloud coupling.
Source: this website