Providers

Stado's compute adapters, selection and live VM inventory. For operations see quota, marketplace and CLI.

Selection: WC_PROVIDERS

The compute provider list is providers in the selected STADO_CONFIG profile, overridable with the comma-separated WC_PROVIDERS environment variable. The order is the preference order. The default is empty: an unconfigured deployment has no provider rather than a hidden GCP dependency.

providers_disabled (env WC_DISABLED_PROVIDERS) is the explicit fence. A fenced provider stays visible in configuration — the profile can explain why a provisioned provider exists — but the scheduler never calls it.

The shared compute-adapter factory enforces the same fence before constructing a client or resolving its credentials. A provider outside the enabled list is refused too. This applies to VM inspection and lifecycle operations, including cancellation: selecting an existing instance never implicitly enables its provider. Storage-backend selection remains separate.

The coordinator resolves this list into tick arms and, per provider and per tick, checks running jobs, reaps dead agents, and schedules queued jobs. A provider whose constructor fails (credentials missing) is logged and skipped, so a misconfigured provider never blocks the primary one. local is in the list but has no cloud tick arm: device-local agents claim assigned jobs directly, and there is no VM lifecycle to schedule or reap.

Inspect the full catalog — capability families, variants, providers, and the active selections — with:

stado capabilities
stado capabilities --json

The catalog answers what a user can ask Stado to provide and which providers implement, partially support, expose externally, or plan each family (compute, workload execution, object storage, quota and capacity, billing, inventory, observability, and the rest). Compute is only one facet: the storage backend is selected separately by WC_STORAGE_BACKEND, not by WC_PROVIDERS.

What each provider contributes

A dispatch-capable cloud entry in WC_PROVIDERS rents agent VMs, and — per the capability catalog — may expose quota, pricing and inventory to the extent its adapter implements them. local contributes no provisioning at all: it attaches existing registered hosts, whose capacity comes from live CPU load, available RAM, disk gates, running jobs, and per-device free VRAM. box leases externally managed fixed-shape machines and reports account limits and available boxes. vast is not a dispatch provider: it is this fleet's own machine listed as a host on Vast.ai. Renting Vast.ai machines is the separate provider vast-rental, one of the GPU and cloud compute vendors.

Provider	What Stado uses it for	Where credentials live
gcp	Google Compute Engine VM dispatch and reap; live accelerator quota; machine prices, BigQuery billing export, credits, budgets; GCP asset inventory	Platform metadata identity where available, otherwise the stado-gcp Skarbiec item
azure	Azure VM dispatch and reap; configured quota reservations; machine prices, balance, usage, billing health; Stado-owned VM inventory	Azure managed identity preferred, then the stado-azure service-principal Skarbiec item
aws	Amazon EC2 VM dispatch and reap; machine-price estimation; Stado-owned EC2 inventory	stado-aws Skarbiec item (access_key_id, secret_access_key); IMDSv2 workload identity on adapter hosts without a grant; environment credential chains are disabled
box	Leasing externally managed fixed-shape boxes; account limits and available boxes	stado-box/api_key in Skarbiec
local	Executing on existing registered hosts; no machine provisioning	Registry SSH host keys in the selected credential store (stado fleet key); no OpenSSH-file fallback
vast	The stado market --provider vast adapter: listing our own idle GPU host on the Vast.ai marketplace	vast/api_key in Skarbiec
arkane	Vendor machine listing, read and release; no agent dispatch because the deploy API accepts no startup script	The Skarbiec item tagged stado:role:cloud-arkane
crusoe, cudo, hyperstack, lambda, latitude, nebius, oblivus, oracle, runpod, salad, scaleway, vast-rental, voltage-park, vultr	GPU and cloud compute vendors: agent machine dispatch and reap, Stado-owned machine inventory; see GPU and cloud compute vendors	The Skarbiec item tagged stado:role:cloud-<provider>

Cloud locators and credentials are never caller overrides: an enabled adapter receives its exact profile and provider-plugin identity, and a workload-agent grant must never contain stado-gcp, stado-azure, or stado-aws. See configuration for the credential store itself.

The reaper reads each cloud VM's own state word and places it as running, stopped or ended: on gcp the documented Compute Engine statuses, on aws the documented EC2 states (pending and running run; stopping and stopped keep their volumes; shutting-down and terminated are released). A preemptible job whose VM the provider stopped or released is put back as Spot preempted; any other job on a VM that is not running is put back as instance gone (lifecycle=<word>). On both clouds only the provider's own not-found answer reads as no VM. A state word outside the documented list, or an answer without one, is refused with the instance and the word (EC2 instance <id> returned unknown state <word>; existence cannot be decided), so the reaper stops on it instead of putting back a job whose VM may still run it.

GPU and cloud compute vendors

Fourteen GPU and cloud compute vendors dispatch agent machines like gcp, azure and aws: enable one in providers (or WC_PROVIDERS) and the coordinator dispatches, reaps and counts its agents against the reservation overlay. Arkane Cloud is the fifteenth adapter, but only lists, reads and releases vendor machines; its deploy API cannot boot an agent. The adapters share the provider-neutral VM lifecycle (providers::gpu_cloud) where their vendor API supports it.

Provider id (aliases)	Vendor	Settings (stado config set <path> <value>)	Credential fields	Agent runs
arkane (arkane-cloud)	Arkane Cloud	none	api_key	never: listed, read and released only
crusoe	Crusoe Cloud	crusoe.project_id, crusoe.location, crusoe.ssh_public_key; optional crusoe.image	access_key_id, secret_key	on the VM
cudo (cudo-compute)	Cudo Compute	cudo.project_id, cudo.data_center, cudo.image_id, cudo.vcpus, cudo.memory_gib	api_key	on the VM
hyperstack	Hyperstack	hyperstack.environment, hyperstack.image, hyperstack.key_name	api_key	on the VM
lambda (lambda-labs)	Lambda Cloud	lambda.region, lambda.ssh_key_name; optional lambda.image_family	api_key	on the instance
latitude (latitude-sh)	Latitude.sh	latitude.project, latitude.site; optional latitude.operating_system, latitude.ssh_key	api_key	on the VM
nebius	Nebius AI Cloud	nebius.project_id, nebius.subnet_id, nebius.image_family, nebius.jwt_lifetime_seconds	service_account_id, key_id, private_key	on the instance
oblivus	Oblivus Cloud	oblivus.location, oblivus.os_label, oblivus.os_name, oblivus.ssh_public_key	api_key	on the VM
oracle (oci)	Oracle Cloud Infrastructure	oracle.region, oracle.compartment_id, oracle.availability_domain, oracle.subnet_id, oracle.image_id; optional oracle.ssh_public_key	tenancy, user, fingerprint, private_key	on the instance
runpod	RunPod	runpod.image, runpod.cloud_type	api_key	inside the pod's container
salad (saladcloud)	SaladCloud	salad.organization, salad.project, salad.image, salad.vcpus, salad.memory_mb	api_key	inside the container
scaleway	Scaleway	scaleway.zone, scaleway.project_id, scaleway.image	secret_key	on the GPU Instance
vast-rental	Vast.ai	vast-rental.image	api_key	inside the instance's container
voltage-park (voltagepark)	Voltage Park	optional voltage-park.location	api_key	on the VM
vultr	Vultr	vultr.region, vultr.os_id; optional vultr.ssh_key_id	api_key	on the instance

Every setting has an environment override: the upper-case provider and key (LAMBDA_REGION, VOLTAGE_PARK_LOCATION), except Oracle's, which start with OCI_ (OCI_COMPARTMENT_ID). stado config validate refuses a profile that enables a vendor without one of its required settings, naming the path and the variable: lambda provider needs lambda.region (environment override LAMBDA_REGION).

Vast.ai is a marketplace rather than a catalogue of instance types, so vast-rental rents in two steps. For the accelerator's Vast.ai GPU name (L4, RTX 4090, RTX A6000, L40S, H100 PCIE, H100 SXM, H200) it searches the offers that are rentable, not already rented by this account, verified, on-demand, carry one such GPU and have room for the job's boot disk, and accepts the cheapest by total hourly price. An empty search is capacity, so the dispatcher moves to the next tier as it does for any vendor out of stock. The agent's startup script is the container's entrypoint (bash -c), because Vast.ai cuts the onstart field at 4,048 characters; the instance label is the agent's name and the script exports it as STADO_WORKER_NAME. An instance whose container Vast.ai reports exited, offline or unknown while it is meant to run never starts, by Vast.ai's own documentation, and is read as failed, so the reaper releases it. Use a Vast.ai API key of its own under cloud-vast-rental: it spends the account's credit, while the selling side reads the host's key.

stado config set vast-rental.image <image with bash and the agent's runtime>
stado credentials item put --host <vault host> --role cloud-vast-rental --type api-key   # {"api_key": "…"} on standard input
stado config set providers '["local", "vast-rental"]'
stado config validate
stado instances list --provider vast-rental

Credentials

A vendor's account credential is one Skarbiec item tagged stado:role:cloud-<provider> (cloud-lambda, cloud-voltage-park, …), holding the fields in the table. Store it on the vault host with the JSON object on standard input, never on the command line:

stado credentials item put --host <vault host> --role cloud-lambda --type api-key

The coordinator reads the item through its own grant; a workload grant may not carry it, and stado config validate refuses agent.skarbiec.roles or agent.skarbiec.secret_fields that name any cloud-<provider> role. A missing item is refused at the first call that needs it, naming the role and this command: Lambda Cloud: no live Skarbiec item tagged stado:role:cloud-lambda carries a non-empty field api_key; store the Lambda Cloud credential with stado credentials item put --host <vault host> --role cloud-lambda --type api-key and the JSON object on standard input. A vendor that refuses the credential is reported with the role and the vendor's own answer: Lambda Cloud: list instances: the credential in Skarbiec role cloud-lambda was refused (HTTP 401): ….

How many machines run

No vendor exposes a quota Stado can read, so the reservation overlay is the ceiling: config/quotas.json in the queue store declares, per vendor and accelerator, how many machines may run at once. A vendor with no section dispatches nothing, and stado doctor says so: lambda: has no quota API, and config/quotas.json in the queue store declares no lambda section, so no lambda machine is ever dispatched.

The operator declares total from its capacity agreement and reserved from capacity it has committed elsewhere under the chosen vendor and accelerator keys. Stado chooses neither count.

Each vendor declares the instance types it offers and the accelerator each carries; a job's VRAM need selects the smallest offer whose accelerator memory covers it, and a vendor that reports no capacity for a type moves the job to the next larger offer. Lambda, Vultr, Voltage Park and Crusoe report capacity before a launch (Vultr's region availability, Voltage Park's instant presets, Crusoe's capacities, Lambda's insufficient-capacity code); Scaleway's out_of_stock refusal is capacity too. Machines are rented on demand only; a preemptible request is refused.

Provider	Instance type → accelerator
crusoe	l40s-48gb.1x → nvidia-l40s, a100-80gb.1x → nvidia-a100-80gb, h100-80gb-sxm-ib.8x → nvidia-h100-80gb, h200-141gb-sxm-ib.8x → nvidia-h200-141gb
cudo	epyc-milan-rtx-a6000 → nvidia-rtx-a6000-48gb, epyc-milan-a100-pcie → nvidia-tesla-a100, sapphire-rapids-h100 → nvidia-h100-80gb
hyperstack	n3-RTX-A6000x1 → nvidia-rtx-a6000-48gb, n3-L40x1 → nvidia-l40-48gb, n3-A100x1 → nvidia-a100-80gb, n3-H100x1 → nvidia-h100-80gb, n3-H100-SXM5x8 → nvidia-h100-80gb
lambda	gpu_1x_a10 → nvidia-a10, gpu_1x_rtx6000 → nvidia-rtx6000-24gb, gpu_1x_a100_sxm4 → nvidia-tesla-a100, gpu_1x_a100 → nvidia-tesla-a100, gpu_1x_a6000 → nvidia-rtx-a6000-48gb, gpu_1x_h100_pcie → nvidia-h100-80gb, gpu_1x_h100_sxm5 → nvidia-h100-80gb, gpu_8x_a100_80gb_sxm4 → nvidia-a100-80gb, gpu_8x_v100 → nvidia-tesla-v100-16gb
latitude	g3.l40s.medium → nvidia-l40s, g3.h100.large → nvidia-h100-80gb
nebius	gpu-l40s-a/1gpu-8vcpu-32gb → nvidia-l40s, gpu-h100-sxm/1gpu-16vcpu-200gb → nvidia-h100-80gb, gpu-h200-sxm/1gpu-16vcpu-200gb → nvidia-h200-141gb
oblivus	RTX_4090_x1 → nvidia-rtx-4090-24gb, RTX_A6000_x1 → nvidia-rtx-a6000-48gb, L40_x1 → nvidia-l40-48gb, A100_PCIE_80GB_x1 → nvidia-a100-80gb, H100_PCIE_80GB_x1 → nvidia-h100-80gb
oracle	VM.GPU.A10.1 → nvidia-a10, VM.GPU3.1 → nvidia-tesla-v100-16gb, VM.GPU.L40S.1 → nvidia-l40s, BM.GPU.A100-v2.8 → nvidia-a100-80gb, BM.GPU.H100.8 → nvidia-h100-80gb
runpod	NVIDIA L4 → nvidia-l4, NVIDIA GeForce RTX 4090 → nvidia-rtx-4090-24gb, NVIDIA RTX A6000 → nvidia-rtx-a6000-48gb, NVIDIA L40S → nvidia-l40s, NVIDIA A100 80GB PCIe → nvidia-a100-80gb, NVIDIA A100-SXM4-80GB → nvidia-a100-80gb, NVIDIA H100 80GB HBM3 → nvidia-h100-80gb, NVIDIA H100 PCIe → nvidia-h100-80gb, AMD Instinct MI300X OAM → amd-mi300x-192gb
salad	RTX 3090 (24 GB) → nvidia-rtx-3090-24gb, RTX 4090 (24 GB) → nvidia-rtx-4090-24gb, RTX 5090 (32 GB) → nvidia-rtx-5090-32gb
scaleway	L4-1-24G → nvidia-l4, L40S-1-48G → nvidia-l40s, H100-1-80G → nvidia-h100-80gb
voltage-park	h100-sxm5-80gb → nvidia-h100-80gb, h200-sxm5-141gb → nvidia-h200-141gb
vultr	vcg-a40-8c-40g-16vram → nvidia-a40-16gb, vcg-a40-12c-60g-24vram → nvidia-a40-24gb, vcg-l40s-16c-180g-48vram → nvidia-l40s

When two offers carry accelerators of the same memory, the one listed first is the one a job is sized onto; the others are still counted when they run.

The agent on a rented machine

Every machine boots the same provider-neutral startup script the other clouds use, delivered as the vendor's cloud-init, user data, start script or container command. On a virtual or bare-metal machine the script first sets the hostname to the instance name (wisent-agent-<accel>-<tick>-<n>). A container cannot set its hostname, so on RunPod and SaladCloud the script exports STADO_WORKER_NAME instead — the instance name on SaladCloud, the pod id from RUNPOD_POD_ID on RunPod — and the agent publishes capacity under that name. The reaper matches the same name, and only machines whose name starts with the fleet's instance prefix are treated as agents; anything else the credential can see is left alone.

Arkane Cloud's deploy API takes no startup script, so no agent can boot there. Its machines are listed, read and released, it has no sizing ladder, the coordinator gives it no dispatch arm, and a launch is refused: Arkane Cloud: the vendor's deploy API takes no startup script, so Stado cannot boot an agent on a machine it rents there; arkane machines are listed, read and released but never dispatched.

Azure operator sign-in and role repair

Read Azure operator sign-in and role repair.

Quota

Read quota diagnostics and operations for live capacity, requests, refusals and Desktop confirmation.

Live instances

stado instances list shows live agent VMs across the configured providers and flags ones no queue job or lease still references. Its operational inventory path can publish expired capacity records. For a read-only census that makes no capacity publication, use stado resources show --provider <provider> --json; its compute row names the vendor's enumeration state and error instead of reporting an inaccessible fleet as empty.

stado resources show --provider gcp --json
stado instances list --provider gcp --json

--provider cannot expand the enabled set. A fenced selection returns PROVIDER_DISABLED; a provider absent from the enabled list returns PROVIDER_NOT_ENABLED. These refusals happen before queue or credential reads. The command exits 1. With --json, its error envelope contains status: "error", error_code: "refused", retryable: false, and a message naming the provider and the governing configuration field.

The resource inventory uses the same compute-adapter boundary. Its refused provider row stays blocked, with unknown instance and orphan counts and the reason attached; it is not evidence of an empty fleet. Cancellation retains the refusal instead of reporting a remote instance deleted.

Reaping runs on each coordinator tick, including autonomy.

Offering idle GPUs on a marketplace

The Vast.ai host marketplace lists this fleet's idle GPU. Buying machines is the separate vast-rental compute provider.

The provider-neutral rule

Every command above enters through Stado. Provider diagnostics belong inside the corresponding adapter and are unavailable unless that provider is explicitly enabled in the selected profile. Nothing falls back to a cloud CLI, provider SDK, direct bucket URL, ambient credential, or a different backend: stado overview resolves only the configured Stado backend and enabled adapters, and no bootstrap, health, recovery, or release path invokes gcloud, gsutil, or az. The operational consequences are in operations.