On this page
Observation
How does Stado know when anyone last actually looked at a fact — and how does it keep "nobody looked" from reading as either "it is fine" or "it is down"? An observation is the record of one look, with its author and its age.
What it is
A declaration does not decay: active written twelve days ago reads exactly
like active confirmed a second ago. That is how a service directory entry
pointing at a closed laptop stayed structurally valid for twelve days while
every consumer routed into a forward with no upstream and a worker refused
29,616 times to claim work. The model could not express the sentence "nobody
has looked since" (stado-rs/src/observations/observation.rs).
An observation is a separate kind of record, carrying four things a declaration does not:
| Field | Meaning |
|---|---|
fact |
What was checked, spelled identically by every checker (<kind>:<subject>), so two commands looking at one thing produce one row. |
vantage |
Who looked, by registry target name. Reachability has no fleet-wide answer — a loopback endpoint is true from its own host and false from everywhere else — so an observation without a vantage is not evidence of anything. |
state |
observed, unreachable, or unverified. |
at |
When, RFC 3339 UTC. The field the outage needed and did not have. |
Three states, never two. unreachable means someone looked and it was not
there — a failure. unverified means the look could not happen: host down,
helper not installed, channel refused. Collapsing those into one false is
how a fleet learns to ignore its own reports. And never is the fourth honest
answer: this fleet has no record of anyone ever checking this. Reading it as
observed is the original bug; reading it as unreachable invents an outage.
Freshness is a property of the record, not the reader. A Fresh observation
was made inside the caller's TTL and may be acted on. A Stale one still
carries the last thing anyone saw, clearly marked as history, but must never
be treated as the present. The default TTL is one hour: a laptop lid closes in
a second, and an hour is how long the fleet is willing to be wrong about it —
visibly amber within the working hour rather than silently in twelve days.
Who declares it
Nobody. An observation is precisely not a declaration; it is recorded by whatever process actually looked, stamped with its own vantage. Writers include:
stado service verify— one record per probed finding,unverifiedones included (stado-rs/src/cli/service_verify/verdicts/record.rs).stado host software <target>— persists the host's software report undersoftware:<name>@<host>with the target as the vantage, so it goes stale exactly as a reachability observation does (cli).stado service directory connect— records whether the proven route answered. A record that cannot be written must not turn a successful connect into a failure; the answer stands whether or not the host can keep notes (stado-rs/src/cli/directory/routes/connect.rs).
Who observes it
stado service directory showprints each endpoint with its freshness on one line — read alone,from operator-host: http://127.0.0.1:8080is a claim with no author and no date, the exact rendering an operator believed for twelve days (stado-rs/src/cli/directory/report/mod.rs).stado service listcarries an observed column per row.stado release statusreads software reports out of the observation file rather than opening one ssh connection per target, and a report older than the TTL, or absent entirely, makes it exit non-zero (cli).
Where it lives
~/.stado/observations.json, owner-only, written through a temporary file in
the same directory and a rename — a reader must never see half a file, and a
reader here is a routing decision (stado-rs/src/observations/store.rs).
Commands
stado service verify
stado service directory show
stado host software control-host
stado release status
Flag-by-flag detail lives in cli.
Not to be confused with
- A declaration. The registry and directory say what should be true; an observation says who last checked and what they saw. The reconciliation between the two is the subject of operations.
- A host beacon. A beacon is the host's own push about its units and disks; an observation is a look taken from a named vantage, which may be another host entirely.
unreachablevsunverified. "I looked and it is gone" and "I did not look" send an operator to two different machines, and are never collapsed.
Source: this website