Common queries and failure modes

Common queries

Fleet, queue, quota, and billing

stado overview
stado overview --json

overview resolves the configured Stado backend and enabled adapters. It does not fall back to a provider CLI, ADC, or a different storage backend.

--json carries the whole document. Stado Desktop's Hosts, Queue and Posture screens read the same document, so the two cannot disagree:

Field	What it says
jobs	How many jobs sit under each lifecycle prefix: queue, running, completed, uploaded, failed.
outcomes.by_model	Per model named in the job commands, its queue, running, completed and failed counts. Jobs whose command names no model are counted under model: null.
outcomes.failed	Every failed job, newest first: job_id, model, task, error, failed_at.
outcomes.completed	Every completed job, newest first: job_id, model, task, wall_seconds (null when a stamp does not read), completed_at.
outcomes.throughput	average_wall_seconds_per_completed_job over the completed jobs whose stamps read (null when none does), samples, how many that is, and unreadable_stamps: every job stamp that does not parse, with its job_id, field, value and the parser's error. Such a job is left out of the average and the newest-first order puts it last; overview prints each one, and Desktop's Posture screen names them under Average completion.
fleet.workers	Every host publishing capacity inside the staleness horizon.
fleet.targets	Every registry target. active_worker is false for a local worker that is not publishing, and null for a target that is not a local worker.
measurements_error	Why the capability measurements could not be read. When it is set, a host shown without a measurement is not proof that nothing measured it.
fleet.targets[].measurement.unreadable	Why that host's capability measurement object cannot be read as one: not JSON, no capabilities map, or a capability without a true or false value. Such a host is neither unmeasured nor measured unable; stado overview and stado registry doctor print the cause.

A missing field is a named failure in Desktop, not a zero. If the job store cannot be read, overview exits with the storage error and Desktop shows the same sentence.

Local agent state

stado host health <registry-target>
stado host inventory <registry-target>
stado host exec <registry-target> -- nvidia-smi
stado host exec <registry-target> -- docker info

docker info asks whether the host's Docker daemon answers, and which server it is, from the paths the fleet installs the Docker CLI at (/usr/bin/docker, /usr/local/bin/docker, /opt/homebrew/bin/docker). It is a fixed read: it takes no further argument and starts, stops or pulls nothing. Ask it of a builder before a Supabase schema release, whose post-build test starts its scratch database through the Supabase CLI and refuses with the runner needs the Supabase CLI and Docker without saying which of the two is missing.

stado host inventory lists every listening socket from the host's kernel socket table. When netstat cannot read that table the listener list is empty for that reason, not because nothing listens: the report says listeners: failed — the kernel socket table could not be read (<what netstat said>), so no marker above could be checked against it, and --json carries listeners_state: "failed" with netstat's words in listeners_error.

A running job whose tick fails on a storage error, such as a durable transition another writer changed, stays this agent's slot and is advanced again on the next tick; the worker does not stop over it. The agent log says loop: slot <job> could not be advanced this tick and is kept for the next: <error>, and the agent's capacity broadcast carries every such job under diag.slot_errors (job id to error), empty when every slot advanced.

A job's durable transition record whose job another writer has already moved settles itself on the next read: a prepared move whose source and destination are both gone is aborted, and a completed move whose destination the job has since left is retired, when the job stands under any lifecycle prefix. Only a job no prefix holds still fails its caller, with transition <id> lost both source and destination, and no lifecycle prefix holds job <job> or completed transition <id> has no destination, and no lifecycle prefix holds job <job>: that job's record is lost, which stado machine status <job> confirms.

Inspect one job end-to-end

stado machine status <job_id>
stado machine logs <job_id> --cursor 0 --limit 1048576

Failure mode quick-grep

The most common failure modes — search the per-job stdout for one of these substrings to classify failures fast:

Substring	Cause
HfHubHTTPError: 429	HF Hub rate limit (free tier 1000 req / 5 min). Retry path lives in wisent.core.utils.infra_tools.infra.data.dataset_splits.get_all_docs_from_task and the cache fast-path in generate_pairs_from_task.py.
Couldn't find cache for	datasets cache miss for a config that doesn't exist on the dataset's HF repo. lm-eval task config drift.
OverflowError: int too big to convert	tokenizer model_max_length was a sentinel (1e30) handed to the rust binding's u32. Capped at 4096 in activations_collector.py.
Dataset scripts are no longer supported	datasets 4.x dropped the script loader. Pinned datasets<4.0 in the agent template.
huggingface-hub>=0.34.0,<1.0 is required	transformers 4.55.x dep-pin mismatch. Pinned huggingface-hub<1.0 in the agent template.
RuntimeError: Cannot set NUMBA_NUM_THREADS	numba init happened before wisent's env-set. Set NUMBA_NUM_THREADS=1 in the agent's env BEFORE Python starts.
does not appear to have files named	transformers shard-name miscompute on gpt_oss / 0-indexed safetensors. Fixed in transformers>=4.57.
AttributeError: ... has no attribute 'transformer'	wisent activation hook expected GPT-2 path on a model whose model_type contains gpt; gpt_oss uses Llama-style. Fixed in transformer_analysis.py.
gated repo / 401 Client Error	The scoped workload credential cannot read the requested repository. Rotate stado-huggingface/token through the stdin-only Skarbiec service path; never place the token in VM metadata or logs.
Quota 'PREEMPTIBLE_NVIDIA_*_GPUS' exceeded	Hit the regional preemptible quota. Either raise via GCP console or add zones to MACHINE_TYPE_ZONES.