Local inference

The optional top-level inference section is the single desired-state and routing catalog for Stado-managed vLLM. gateway_target is the registered host running Brama; deployments run on registered local GPU targets and expose their OpenAI-compatible endpoint only on a Tailscale IPv4 address.

{
  "inference": {
    "gateway_target": "my-brama-host",
    "deployments": [],
    "routes": {}
  }
}

The lifecycle is deliberately two-step and generation-fenced:

Create the shared bearer without exposing it in argv, output, or a file:

stado inference init-credential

Bootstrap the gateway snapshot before Brama's first managed start:

stado inference route set example-client/chat/primary \
  --to openai/gpt-4.1-mini \
  --expected absent --gateway gateway-host

stado inference plan chat-primary \
  --host gpu-host \
  --image 'vllm/vllm-openai@sha256:<image-digest>' \
  --cache-dir /srv/stado/inference/chat-primary \
  --model 'example/model' \
  --gpu-mode yieldable \
  --max-model-len 16384 \
  --revision '<model-revision>'
stado inference apply <plan-id>
stado inference doctor chat-primary
stado inference verify chat-primary
stado inference route set example-client/chat/primary \
  --to chat-primary \
  --fallback openai/gpt-4.1-mini \
  --expected openai/gpt-4.1-mini \
  --gateway gateway-host

The immutable model revision and container image digest prevent silent model or runtime replacement.

kv_cache_memory_gb sets a fixed vLLM KV-cache allocation in GiB. Omitting it preserves the pinned image's own memory policy. A smaller cache releases VRAM without changing model weights or output quality, but lowers the number of tokens and long requests that can remain active concurrently.

gpu_mode defaults to exclusive, which keeps the GPU reserved for inference. yieldable makes the local Stado agent the lifecycle owner: it pauses the inference container when an eligible GPU job is queued, advertises the released capacity, and resumes inference only after queued and active GPU work has cleared. Eligibility includes provider, accelerator, host pin, centralized assignment, and capacity constraints; there is no timeout-based eviction. The ordered provider fallback remains available while the local container is yielded. route set therefore accepts a non-ready yieldable deployment only as the primary of a route with at least one ordered fallback. It still requires an exclusive primary and every local fallback to be ready, so an unavailable deployment cannot be published as the route's only destination.

If plan or apply reports an unmanaged GPU workload, inspect it through the same target-scoped host channel instead of opening an ad hoc SSH session:

stado inference blockers --host gpu-host
stado inference release --host gpu-host \
  --identity <PID:START_TICKS>

blockers reports the executable, owner, VRAM use, cgroup, and an identity made from both PID and /proc start ticks. release refuses a stale identity, sends TERM, and waits for exit. Add --force only to escalate that same verified process to KILL; it never accepts a bare PID.

A cancelled or failed pre-commit plan can leave a runtime or root-owned model cache without a registry deployment. Clean that exact saved plan through the managed channel:

stado inference plan-logs <plan-id>
stado inference abort <plan-id> --purge-cache

plan-logs reads the not-yet-committed container logs through the same managed host channel.

abort never changes the registry. It stops only the runtime described by the immutable local plan, removes its cache through the pinned container runtime, and consumes the plan after successful cleanup. Set --cache-dir during plan when the target's home filesystem is not the intended model volume.

plan inventories the host, requires Docker, NVIDIA tooling, and a live Tailscale address, then saves an immutable plan bound to the current registry digest. apply rechecks that precondition, installs the digest-pinned vLLM container under Docker's unless-stopped supervisor, waits for an authenticated readiness probe, and only then commits the deployment. A failed runtime, readiness check, or registry compare-and-swap restores the prior runtime.

The deployment and Brama use one centrally stored credential item: provider:local-openai, containing a non-empty token field. stado inference init-credential generates and stores it without printing the token, and refuses to overwrite an existing item. Deliberate rotation can use stado credentials put provider:local-openai with JSON on standard input, but must be coordinated with runtime replacement; never place the token in argv or registry data. Route changes require --expected, probe the destination first, stage an owner-only route snapshot on the gateway, compare-and-swap the registry, and then atomically commit the snapshot. Brama reloads that file per request, so cutover needs no backend restart. Ordered --fallback destinations are attempted when the primary provider fails; An external provider fallback therefore preserves the same model contract while local inference is unavailable. rollback reinstalls the recorded prior deployment; retire refuses while any primary or fallback route still selects the deployment and retains model cache unless --purge-cache is explicit.