Object store

Where does fleet state actually live, and who is allowed to write it? In one canonical object plane, selected by STADO_CONFIG, where every write is authorized by matching its key against a namespace's per-prefix action policies.

What it is

The object store is the single durable plane behind the fleet: queue state, capacity broadcasts, control flags, the registry document, autonomy records, and release artifacts all live under fixed stado:// prefixes. The selected STADO_CONFIG names the primary backend, replication, and release origin; coordinator and agents use the same canonical prefixes regardless of backend. There is no provider-derived storage fallback or direct client bucket path.

Writes cross an authenticated object boundary. The gateway resolves the request's namespace from configuration, checks that the namespace's policy allows this action on this key or list prefix, and then compares the bearer against the namespace's Skarbiec-held token in constant time. A key no policy covers — or an action the prefix does not allow — is refused with 401 {"error":"unauthorized or non-immutable release write"}, a sentence that names neither the namespace, the prefix, nor the grant.

That refusal shape hides the cause: a writer whose prefix no namespace policy declares is refused on every write while reads keep working, so status commands keep printing as if nothing were wrong. Every autonomy object is therefore rooted under state/ (state/autonomy/..., state/host_silence/). If you add a new prefix, root it under a namespace some policy declares.

Who declares it

Operators, in the deployment configuration: the backend, the namespaces, and each namespace's prefix/action allowlist. Writers are the coordinator, agents, the autonomy layer, and authenticated submitters — each within its namespace's policy. Reads under stado://releases/ are the one public, bearer-free surface; publication there is authenticated and create-only.

Who observes it

Everything. The scheduler reads the queue window, the reaper reads capacity and heartbeats, resolvers read registry snapshots, stado optimize status reads autonomy records. Ordinary storage clients read the configured primary first and may retry a primary error against the mirrored backup; a successful primary absent answer does not fall through. The object API server and other authority-sensitive paths instead use primary-only readers so stale replica state cannot become an authority answer. In both modes, writes commit to the primary and are then mirrored, and the backup is never promoted automatically. A readable forecast still proves nothing about the write path; stado doctor checks that the configured backend can be constructed, not that a later mutation completed.

Canonical object uploads

The CLI and graphical clients use the same authenticated object API. For a namespace declared in your deployment, upload and download use:

stado storage put stado://example/artifacts/example.bin ./example.bin
stado storage get stado://example/artifacts/example.bin ./downloaded.bin

Replace example with the configured namespace. Larger uploads use the existing chunk-and-compose protocol and SHA-256 verification. The public proxy buffers each incoming upload request before forwarding it through the Node runtime, rather than forwarding the request's live body stream. The release proxy, object proxy and chunk composition share the public-DNS origin connector, including the chunk reads, final write and chunk cleanup. It preserves the configured hostname, TLS verification, bearer and conditional headers; a missing public DNS answer is a request failure, never object absence.

An upload the transport ends is the transport's failure, never the caller's request: a connection the object API's side resets or closes while a chunk's body is still being sent (connection reset by peer, broken pipe, a stream that ended before the response) is reported as object chunk PUT <index> of <count> (<bytes> bytes at offset <offset>) for upload <id> of <uri>: <cause>, with our failure [infra_down]; retry later and the retryable exit status 69, exactly as a refused connection is. Every HTTP failure the CLI reports takes its code from its cause chain this way: an elapsed wait carries the catalogue's code for it, any other operating-system failure, or a request or body the peer ended, is infra_down; only an answer that could not be decoded stays without a code. Running the same storage put again sends every chunk again under the same upload id (the id is the object's digest); a chunk the store already holds is answered as a conflict and kept, never written twice, and the final composition is the one atomic publication of the object.

A failed request reports error: "object request failed" together with the HTTP method, object uri, failed operation, and original cause. Failure to read the incoming body returns HTTP 400. Failure while forwarding a request or writing the composed object returns HTTP 503; it does not claim that the host is unreachable merely because an upload failed. The server log retains the same details and the original error.

An HTTP 200 from a read does not prove uploads work. Keep the completed write receipt and compare the downloaded bytes with the uploaded file.

CLI connection failures

When the native HTTP transport reports that it could not establish a connection, the CLI classifies the failure as infra_down. It preserves the original cause chain and requested URL. The classification comes from the transport's error type, not from words in an exception or a command argument.

Object-store HTTP errors and registry snapshot reads carry this classification through database resolution, resolver commands and publisher host lookup. For a declared product and consumer, database resolution is a read-only way to observe that dependency:

stado database resolve "$PRODUCT" --consumer "$CONSUMER" --json

A connection failure is not an absent registry object, an HTTP authentication refusal or proof that the whole host is down. It says that the requested connection could not be established; the retained cause identifies what the transport observed. The failure uses the existing fleet failure code and retry exit policy. This classification adds neither a wait nor a retry loop.

The object API, the resolver and the local API all run in this machine's one Stado process, and a unit that exits at start is restarted by its supervisor without telling anyone. So every stado command that fails with infra_down also reads this machine's Stado unit, and when its supervisor holds it loaded with no running process and a nonzero last exit, prints after the error line:

this machine's com.wisent.stado runs no process: its last run exited <code> and its supervisor has started it <runs> time(s); its last log line: <line> (`stado service unit logs com.wisent.stado --host <this host>` reads more)

The log line is the last line of the unit's error log, or of its log when the error log is empty: the same one-line window stado service status shows for a failed unit. With --json the same sentence is the error document's host_unit field, null when the unit runs, last exited cleanly or is not loaded on this machine. A read of the unit that fails is said instead, whether this machine's Stado unit is failing could not be read: <cause>; it is never taken for a running unit. Stado Desktop shows the line in the operation's complete receipt, which carries the command's standard error.

Where it lives

In the backend the selected STADO_CONFIG names. It can be self-hosted — a Stado object API on one of your machines and its disk (stado, alias self-hosted), or MinIO, Garage, SeaweedFS or Ceph that you run — or hosted by Google Cloud Storage, Azure, Amazon S3, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean, Hetzner, Scaleway, OVHcloud, Akamai, Vultr, Oracle, IBM, Alibaba, Tencent, Huawei, Exoscale, IONOS, Contabo, UpCloud, Storj, Filebase, Tigris, Yandex or IDrive e2, or any other S3 API endpoint (s3-compatible); storage providers lists what each declares and self-host your storage walks through running your own. Stado also serves the S3 API itself over the same objects; see Stado as an S3 provider. The canonical layout (queue/, running/, status/, leases/, capacity/, control/, registry.json, state/..., and the rest) is tabulated in the architecture doc and is identical across backends. system/storage-layout.json is the versioned layout marker. A process reads one store's marker before its first use of that store, and at most once at a time: an opening that finds the read in flight waits for its answer, a failure included, instead of sending its own, so a burst of openings puts one request on the store, not one per opening. A marker found valid is not read again by that process — it is written only when absent and never replaced — and a failed read is not kept, so the next opening reads again.

Commands

stado config show
stado config validate
stado doctor
stado storage
stado artifact list

stado storage moves queue state between backends (the billing-outage migration path). Flags are in cli.

For the specialized source-preserving handoff between the fixed co-located local roots, use stado repair stado --step storage-root; its run, resume, status, rollback, and finalize boundaries are documented in disaster recovery.

Inspecting every object

stado storage ls queue/ --json lists every object under that prefix, sorted by name. There is no implicit display limit. Use --limit <count> to request a bounded result; JSON reports the requested limit, listed, total, and truncated. With no limit, limit is null. An explicit limit equal to the total is not truncated. With no prefix, stado storage ls --json reports counts across the canonical prefixes instead of object rows.

--size adds each object's size as the store's listing states it, which every backend's listing carries, so no body is downloaded for it; size_from is listing. Only an object the listing stated no size for, one removed between the listing and its stat, has its body read (size_from is body): a body that is gone is reported as vanished, and a failed read keeps size_error, never a fabricated zero. A failed prefix listing is an error, not an empty result. Malformed --limit arguments are refused before storage is opened.

In Desktop, open a declared host's Space panel, then Available operations → List objects in the selected API's configured store. Leave Maximum objects blank for all rows. The operation reads the selected API's configured object store, not the selected host's disk. Its receipt retains arguments, output, errors and exit status.

The single-object verbs have Space panel operations of their own, each running the same command with the fields its arguments take: List a namespace's objects (stado storage objects), Read whether an object is present, its size, version and metadata (stat), Print an object's body (cat), Download an object into a local file (get), Upload a local file as an object (put, with its media type and --if-absent), Remove an object (rm) and Discard the staged parts of an interrupted upload (abort-upload, with --dry-run).

Parallel object operations

storage copy, storage backup, and recovery migrate accept an optional positive --concurrency. When omitted, Stado measures the operating system's available parallelism. An unavailable measurement is refused with cannot measure available parallelism for bulk object operations and the OS cause; it does not silently select a worker count. Zero is rejected by the argument parser. Recovery resolves this value before fencing a store, stopping writers or opening a billing window, then reuses it for copy and verification.

The same measurement and error propagation apply to bulk queue reads, release reports, host software inspection, storage verify, canonical prefix counts and the opt-in storage ls --size body reads. A measurement failure is an operation failure, not an empty listing or a vanished object. Queue read batches also use the measured parallelism, rather than a separate fixed batch size. Every batch is visited unless the caller's explicit job window or scan budget has been filled; batching does not cap the result.

Desktop's Space operations also offer Copy objects between stores, Copy objects to the configured disaster-recovery store, and Compare store names, metadata and bytes. Copy and backup expose Objects in parallel; blank measures it. Mutations require the existing review confirmation. These requests and their complete failure receipts use the selected API.

Listing qualification

From the Stado checkout, run node tests/storage/listing/journey.mjs with STADO_BIN, STADO_TEST_SOURCE_REVISION (full native source hash), STADO_TEST_LIST_COUNT, STADO_TEST_LIST_LIMIT, STADO_TEST_COPY_CONCURRENCY (positive integer), and STADO_TEST_COPY_REFUSED_CONCURRENCY (zero) declared in the run's environment. Use more than 255 objects to exercise the removed byte-width cap, more than the host's measured parallelism to cross queue read batches, and a smaller positive explicit listing limit. A separate run with listing limit zero exercises an intentionally empty selection.

The journey snapshots the executable and refuses a mismatched source revision. It submits real jobs into a private local store and compares every status --json row with the persisted job, across measured read batches. It checks complete JSON and text object listings, explicit truncation, exact-boundary listing, body lengths, invalid arguments and unchanged stored bytes. It copies those objects to isolated local destinations with measured and explicit parallelism, compares every persisted body, runs storage verify, and confirms zero fan-out is refused before the destination is opened. No worker is launched. Its executable identity, commands, exit statuses and before/after digests are retained under .build/storage-listing/.

Not to be confused with

• The registry — one versioned document inside the store, not the store itself.
• The release channel — immutable objects under stado://releases/, publicly readable through https://stado.wisent.com/api/release/object; copying release objects into a second "public" store is a broken release channel.
• Skarbiec — the credential store. The object gateway reads its verifier tokens from Skarbiec; the object store itself holds no provider or product secret material.