What a complete Stado test suite looks like

Written 2026-09-06 and checked against main at a2d62b4f. Every count below was read from a revision this page names, not remembered: "What the tree measures today" was re-measured 2026-09-07 at 86c71e1b, and every count after it is still the a2d62b4f reading and says so.

This page is the plan for stado-rs/tests: the rules each test obeys, what the tree measurably covers today, and the coverage that is missing. It is not a second definition of done — that lives in ~/STANDARD.md and the stage gates in product-guidelines/README.md. The rules restated here are the operator's, set 2026-08-19 in ~/AGENTS.md.

The rules every test obeys

One binary, driven from outside. A test runs CARGO_BIN_EXE_stado through std::process::Command. Not a library function, not a mock, not a re-implemented code path. A dry run, a smoke test, a canned response, a simulated provider, a schema check and a successful exit before the promised effect are not tests and never count as evidence that a feature works.

Assertions read state, not output. The document on disk, the object in the store, the file on the host, the exit code, the exact refusal sentence. Stdout is supporting evidence, never the only one.

Every command is a set of paths. Success plus its refusals — duplicate, malformed name, unknown object, occupied state — each asserted with its exact sentence, because that sentence is part of the product contract.

One test target is one command and its story. create with its refusals, assign with its refusals, delete with its refusals. The test name says what it defends.

Isolation through the environment. A tempdir plus the product's own backend-overriding variables. tests/fleet/main.rs is the reference implementation:

cmd.env("WC_STORAGE_BACKEND", "local")
   .env("WC_LOCAL_STORAGE_PATH", storage)
   // A set-but-missing STADO_CONFIG disables config-file discovery.
   .env("STADO_CONFIG", storage.join("no-such-config.json"))
   .env_remove("COMPUTE_API_KEY")
   .env_remove("COMPUTE_API_URL")
   .env_remove("WC_PROFILES_DIR");

No test touches the operator's real registry, vault or configuration. Commands that write to real operator state — invite minting, for one — are out of scope until a dedicated fixture exists.

Behaviour is probed before it is asserted. Exit codes and refusal sentences are copied from live answers, never guessed. If a command does not exist, the command is implemented first and the test second.

A real dependency is used or the run is blocked. When the feature needs a host, a provider, a browser or a persisted credential, the test uses that real component through Stado and Weles and observes the final state. When the real flow cannot run, the answer is blocked with the reason — never an easier test and never passed.

Retention. Each run records the exact source revision, the commands, the exit statuses, the process output and the supported reports, so the answer to "which revision passed this" is a fact and not a memory.

Tests are not run without an explicit instruction. Verification of a change is cargo test --no-run plus manual probes of the product commands.

The documentation a change breaks is that change's failure. A page names a file; a change in the product repository moves it; the sentence points at nothing, and nothing over there notices — the checker that would lives here. So this site publishes what it claims and that repository's gate reads it. npm run docs-inventory writes public/docs-source-paths.json — the page, line, path and repository of every claim the pages make, only ever from a corpus that passed — the site serves it at /docs-source-paths.json, and stado-rs/tests/documentation/published_paths.rs refuses a revision that does not carry a path the published pages name, printing each page and line to fix. Both sides resolve a path the same way, by segment boundary, so deploy/host_channel.rs and stado-rs/src/deploy/host_channel.rs are one claim. An inventory the check cannot read is a failure with its own sentence, never a quiet pass; STADO_DOCS_PATH_INVENTORY points it at a local file.

What the tree measures today

Re-measured 2026-09-09 from main at 0ae0ca1c, after the stand-in replacement below:

• 72 directories under stado-rs/tests. 70 of them carry a main.rs, and cargo auto-discovers tests/<dir>/main.rs, so those 70 are real test targets.
• platform-matrix/ and support/ carry no main.rs, so neither is a target.
• 57 of the 70 spawn CARGO_BIN_EXE_stado.
• Zero areas name a machine that does not exist. On 2026-09-08 five did.
• 27 cases are still #[ignore], so they run for nobody until somebody passes -- --ignored.
• 17 files are still past 300 lines, which means 17 areas cannot be edited at all until they are split — see below.
• One area, tests/scratch/, ends on a machine the fleet chose: it leases a disposable target, drives a real host command through it, and destroys it.

Seven areas were deleted the day they were written

tests/workload, tests/repair, tests/runner, tests/credentials_host, tests/space, tests/route and tests/host_release existed for one day and were removed in 3cf0670f. They were 53 checks, they all spawned the real binary, they all passed, and not one of them was a test of the capability it was named after.

Each did the same thing: a tempdir, WC_STORAGE_BACKEND=local, a seeded registry document, and assertions that a declaration parses and a refusal sentence is exact. Then it stopped. Nothing installed a runner on a host, repaired a wedged service, reclaimed a byte, opened a forward on a real target or delivered a release — which is what those seven commands do. The rule above already says it: planning, request construction and a successful exit before the promised external effect are not substitutes for the flow. I wrote them anyway and reported "53 passed" as if the capabilities were proven.

So the lesson this page was missing, and it is not the one I first wrote here: driving the real binary is the easy half. A test earns the word when the flow reaches the component the feature needs and the assertion reads the state that component was left in. Everything short of that is a green tick over an untested feature, and a green tick is worse than an empty directory, because it answers the coverage question with a number.

How each of those areas was found, and what replaced it, is Stand-ins.

The tree is not addressable by command

The 62 areas a2d62b4f carried are named after the defect each was written for — silence, truncation, workload_hold, stale_unit_image, cleanup_gate_naming. That is a good name for a regression and a useless one for the question an operator asks: is stado storage verify tested?

So the index below exists, and it is a name-level measurement, not a claim about behaviour: an area is counted for a command group when its test source names that group as a string literal and the area spawns the product binary. It answers "could this area possibly touch that command", which is the weakest useful question, and it is still enough to find the holes.

Command groups no area names at all — 20 of the 49 the binary lists: onboarding, blast-radius, resources, optimize, billing, azure, cloudflare, mail, machine, quota, profiles, schedule, cost, vast, instances, alerts, placement, database, web, stream. machine is the documented machine interface, so the surface a caller integrates against has no test of its own; billing, azure, cloudflare and vast are provider paths that spend money.

Groups some area names, with how many areas name each: host 23, service 19, storage 19, status 14, registry 8, agent 8, dashboard 7, capabilities 7, release 5, product 5, doctor 5, resolver 4, submit 4, config 3, coordinator 2, cancel 2, queue 2, identity 2, disk-cleanup 2, builds 1, fleet 1, results 1, job 1, artifact 1, egress 1, overview 1, recovery 1, bootstrap 1, credentials 1, dns 1, inference 1.

A count above one is not coverage: queue is named twice, by ci-cd and run_history, and neither is a test of the queue's own story. That is what the next section is for.

Areas to add, in build order

Each row is a command whose own story nothing tells today, even where the word appears inside another area. Ordered by what breaks the operator soonest, not by what is easiest. The names the seven capabilities carry — stado workload, repair, runner, credentials, space, route and release — belong in this list too, and the rows below that mention a host are the shape they need: every one of them ends by reading state on a machine.

That prerequisite now exists. stado scratch leases a disposable target — a throwaway local account on a host the fleet already manages, described in Scratch targets — and destroys it when the run ends, so a capability that installs a runner, restarts a unit or deletes files can be driven against a real machine without the run touching the operator's registry, vault or hosts.

A capability test on top of it has one shape, and tests/scratch/ is that shape working: ask stado scratch hosts --json which target is leasable, lease it, point WC_STORAGE_BACKEND=local and WC_LOCAL_STORAGE_PATH at the emitted storage_root, drive the capability's own commands against that disposable target, assert against the state left on the machine, and destroy the lease — with the lease's TTL and the host's own reap as the backstop for a run that dies before it can. Its eight stories cover the lifecycle, the expiry sweep and every refusal sentence, and the host is never named in the source.

runner_registration is the first row of the table below that needs it, and it now has everything it needs.

A test that cannot get a lease — no reachable host, no profile declared for the host's platform, the lease refused — reports blocked with that refusal as its reason. It does not fall back to a tempdir and a parsed declaration and call itself passed: that substitution is what produced the seven areas above.

tests/<area>/	Drives	Asserts against
host_disk	host disk --json	free kibibytes and the swap line for a host under real memory pressure; the inventory roots per platform; that a host that cannot be asked reports so rather than null
host_reclaim	host reclaim --dry-run, --apply	bytes actually freed per stage, runner_work_trees among them; the audit record on the host; the refusal when no stage is eligible
host_grant	host grant-item-read, grant-show	the grant recorded in Skarbiec, read back per item and per field; refusals for absent item, absent field, absent bearer file
host_beacon	host publish-beacon FILE [--print]	the beacon object in the store, its reported-at stamp, and host link turning stale into silent
queue_lifecycle	submit, status, results, cancel	the object moving queue/ → running/ → completed/ → uploaded/; cancelled/ written on cancel; the terminal outcome retained in run history
queue_drain	queue drain --wait, pause, resume	the paused control blob; the timeout exit being non-zero with the queue still paused; that CONFIRM_FLEET_DRAINED alone drains nothing
storage_move	storage copy, verify, stat, ls, cat between two local backends	both stores compared object for object; metadata keys folded as the copier folds them; absent and unreachable (infra_down) answered differently
registry_write	registry host add, remove, registry doctor, registry self	the registry document and its generation fence; the refusals for unknown target kind, a target with no SSH destination, a malformed weles block, a non-string host_heuristic
machine_interface	machine submit --request-file, status, logs --cursor, cancel, artifacts	the reserved request under machine_requests/<id>.json with its SHA-256; artifacts written to --output-dir; the refusal for a field outside the accepted list
release_publish	release submit, promote, quarantine list/clear, redeliver, active-binary	the published manifest read back from the channel and byte-compared; the refusals release job returned mixed or invalid output, release job omitted archive, immutable queue object differs, refusing a replacement without terminal failure
service_lifecycle	service ensure, converge, restart, list	the unit loaded on a real host and the bind held afterwards; an in-place restart leaving no window with nothing running; the reload refusal naming the program it compared
doctor	doctor	each check's own verdict, and that two checks never disagree about the same backend
desktop_parity	the Desktop screens through the Probierz CUA harness	each screen showing what the equivalent command answers, on the same state
runner_registration	stado runner install --repository, status, restart, remove, report	the registration record under the runner root on a disposable host, the scope GitHub actually accepted, the listener's launchd or systemd state, and the job slot the gate holds

The groups that spend money

billing, azure, cloudflare, vast and instances reach a provider that charges. A test that provisions there is a purchase, so it does not get written on an agent's judgement: the cost is stated to the operator first and the run stays blocked until that decision exists. What can be tested without spending is the refusal side — a missing credential, an unentitled platform, a quota already at its ceiling — and that is where these areas start.

GCP is not on that list on purpose. Billing is detached from wisent-480400 deliberately, so a test that needs a billable GCP API is not a test to enable; the dependency is the thing to remove.

What this plan is not

It is not a promise that a folder per row makes the product tested. A row earns its place only when a plausible bug fails it. A test that pins wording, plumbing, a field copy or a default asserts an implementation, and belongs deleted rather than re-pinned.