Stado as an S3 provider

Stado serves the S3 API itself, at /s3 on the object API listener, over the same objects /api/object serves. Any S3 client — the AWS CLI, an AWS SDK, rclone, mc, another Stado's s3-compatible backend — reads and writes a namespace's objects there, and an object written through either door is read through the other.

Buckets, keys and secrets

A bucket is a namespace object_api.namespaces declares. Buckets are not created through the S3 API: CreateBucket answers 409 BucketAlreadyOwnedByYou for the signer's own namespace.
The access key id is the namespace, and the secret access key is the namespace's bearer: the token of the Skarbiec item the namespace is bound to, the same value Authorization: Bearer presents on /api/object. The S3 endpoint adds no credential of its own, and rotating the item rotates both doors at the next request.
A key is the object key: s3://<namespace>/<key> is stado://<namespace>/<key>, stored at ecosystem/<namespace>/<key>.
Every request is authorized by the namespace's prefix policy: get for GetObject and a copy's source, stat for HeadObject, list for listings, put for PutObject, CopyObject and multipart uploads, delete for deletes and aborted uploads.
Addressing is by path, <origin>/s3/<bucket>/<key>, and the region is whatever the client signs for; Stado's own signers name stado.

stado storage s3-endpoint <namespace> prints what a client needs and first proves it with one signed HeadBucket:

stado storage s3-endpoint example --json

{
  "namespace": "example",
  "endpoint": "https://stado.example.test/s3",
  "bucket": "example",
  "access_key_id": "example",
  "region": "stado",
  "addressing": "path",
  "secret_access_key": {
    "source": "the object API bearer this client presents: STADO_API_TOKEN, STADO_API_TOKEN_FILE or storage.stado.token_file",
    "item": "example-object-api",
    "field": "token"
  },
  "check": { "request": "HEAD /s3/example", "status": 200 }
}

item is null on a client whose configuration does not declare the namespace. A refused check exits non-zero with the S3 endpoint <endpoint> refused a signed HeadBucket of <namespace> with HTTP <status>: <code> <message>; a release namespace is refused with <namespace> is a release namespace, published through stado storage put and read from its public URL; it has no S3 bucket.

With those values any S3 client works:

export AWS_ACCESS_KEY_ID=example
export AWS_SECRET_ACCESS_KEY="$(cat "$STADO_API_TOKEN_FILE")"
aws --endpoint-url https://stado.example.test/s3 --region stado \
  s3 cp ./report.pdf s3://example/reports/report.pdf

Operations

Served
ListBuckets	The one namespace the signer holds.
HeadBucket, GetBucketLocation	The signer's namespace; the location is the region it signed for.
GetBucketVersioning	The empty configuration of a bucket whose versioning was never enabled: Stado keeps no object versions.
ListObjects, ListObjectsV2	Prefix, delimiter, `max-keys` (at most 1000), continuation token, `start-after`, `marker`, `encoding-type=url`. A prefix no single `list` grant covers, such as the whole bucket, lists the union of every granted prefix under it. Entries carry key, size and last modification, no ETag. Staged multipart parts are never listed.
GetObject, HeadObject	A single byte range, `If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`, and the `response-*` header overrides.
PutObject	Unconditional; create-only with `If-None-Match: `; compare-and-swap with `If-Match: "<etag>"` for a text object. `x-amz-meta-` metadata. The bytes are stored first and the media type and metadata right after, as `/api/object` writes them, so a read between the two, or after a crash between them, sees the object without them.
CopyObject	Within one namespace, with `COPY` or `REPLACE` metadata and the `x-amz-copy-source-if-*` preconditions.
DeleteObject, DeleteObjects	Each key on its own grant; refusals reported beside the deletions that held.
CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload	Parts 1 to 10000, staged where `stado storage put` stages its chunks (`<key>.__stado_upload/<upload-id>/<part number>`, beside the upload's `manifest`), so `stado storage abort-upload` discards an upload a client abandoned. Each part keeps the SHA-256 its ETag is.
UploadPartCopy	A part taken on the server from an object of the same namespace the signer may `get`, whole or the one range `x-amz-copy-source-range: bytes=<first>-<last>` names (a range past the source's last byte is `InvalidRange`), with the `x-amz-copy-source-if-*` preconditions. This is how `aws s3 cp s3://<bucket>/<a> s3://<bucket>/<b>` copies an object above its multipart threshold.
ListMultipartUploads	The uploads in progress under a prefix the namespace may `list`, by key and upload id, with `key-marker`, `upload-id-marker`, `max-uploads` (at most 1000) and `encoding-type=url`; a delimiter is not served. The object API's own chunked uploads have no manifest and are not listed.
ListParts	The parts of one upload the signer may `put`, by part number, each with its size, last modification and ETag, with `part-number-marker` and `max-parts` (at most 1000).

Any other operation answers 501 NotImplemented naming what was asked: versions (versionId), turning versioning on (PutBucketVersioning), reading one part of a completed upload, and every other sub-resource (?acl, ?policy, ?lifecycle, …), on a bucket PUT as on any other request; only a PUT with no sub-resource is a CreateBucket.

An object's ETag is the store's own compare-and-swap version for an object whose bytes are text, so If-Match compares exactly what the store compares, and the SHA-256 of the bytes for a binary object. Metadata written over an existing object is merged with what it held: a write that omits an earlier x-amz-meta-* field does not remove it. Release-governed objects (stado://releases, stado://sources, the release catalog) are not served: release-governed objects are published with stado storage put and read from /api/release/object, not through the S3 endpoint.

Signatures and integrity

Requests are signed with AWS Signature Version 4, in the Authorization header or as a presigned URL. A header-signed request must be within fifteen minutes of the server's clock (RequestTimeTooSkewed), and a presigned URL lives from one second to seven days. A body is accepted as a stated SHA-256, UNSIGNED-PAYLOAD, or aws-chunked with each chunk signature and the trailer signature verified. Every checksum a client states — x-amz-checksum-crc32, -crc32c, -crc64nvme, -sha1, -sha256, in a header or a trailer, and Content-MD5 — is recomputed, and a body that does not match is refused with BadDigest before anything is stored. A client that sends Expect: 100-continue is answered before it sends its body.

The refusals a client meets, each with a sentence naming what did not hold: AccessDenied (unsigned request, a grant that does not cover the key, a bucket that is not the signer's namespace), InvalidAccessKeyId (a namespace nobody declares; the caller is not yet authenticated, so the served namespaces are not listed), SignatureDoesNotMatch (the secret is not the namespace's current bearer), XAmzContentSHA256Mismatch, BadDigest, NoSuchKey, NoSuchUpload, InvalidPart, InvalidPartOrder, PreconditionFailed, and ServiceUnavailable when the namespace's secret cannot be read from the vault, naming the read that failed. As on Amazon S3, InvalidAccessKeyId and SignatureDoesNotMatch differ, so a caller without the secret can tell a declared namespace from one nobody declares; it learns nothing of the namespace's objects or its secret. A namespace whose name must stay unknown is not served on an endpoint reachable by those who should not learn it.

The endpoint answers wherever the object API does. A TLS reverse proxy in front of it must pass the Host header through unchanged, because the signature covers it.

Presigned URLs

stado storage url prints a presigned S3 URL when asked for one: a link that needs no bearer and stops working when it expires.

stado storage url stado://example/reports/report.pdf --presign get --expires 3600
stado storage url stado://example/uploads/incoming.bin --presign put --expires 600 --json

--presign get lets the holder download the object, --presign put lets the holder upload its bytes (curl -T file "<url>"); the namespace's policy still decides whether that action is allowed on that key. --expires is required with --presign and is a whole number of seconds from 1 to 604800; a presigned URL lives between 1 and 604800 seconds; <n> was asked refuses any other, and a release object is refused with release objects are read from their public URL without a signature; run stado storage url without --presign. The URL is signed with the bearer this client presents for the object API, so a client without one is refused before anything is signed.

Copying and moving on the server

stado storage cp and stado storage mv copy one object to another key of its namespace through the endpoint's CopyObject, so the bytes never travel to the machine that asked:

stado storage cp stado://example/reports/report.pdf stado://example/archive/2024.pdf
stado storage mv stado://example/inbox/scan.pdf stado://example/reports/scan.pdf --json

{
  "source": "stado://example/inbox/scan.pdf",
  "target": "stado://example/reports/scan.pdf",
  "copied": true,
  "removed": "stado://example/inbox/scan.pdf"
}

The copy is signed with the bearer this client presents for the object API, as the namespace's secret, so the namespace's policy decides it: get on the source and put on the target. The target keeps the source's media type and metadata. mv removes the source through the object API, as stado storage rm does, only after the endpoint answered the copy, and needs delete on the source; a copy that fails leaves the source in place, and a removal that fails after the copy is refused with <source> was copied to <target>, and removing <source> failed; the object is at both keys: <cause>. removed is null for cp.

Refused before anything is asked: two namespaces (<source> and <target> are in different namespaces; an object is copied within its own namespace, whose secret signs the copy), a release object (release objects are published with stado storage put and never copied or moved; cp and mv take product objects), and the same object twice (<source> is both the source and the target). A copy the endpoint refuses exits non-zero with the S3 endpoint <url> refused a CopyObject of <source> to <target> with HTTP <status>: <code> <message>, for example NoSuchKey for an absent source or AccessDenied for a key the grant does not cover; nothing is removed.

Stado Desktop offers all four under the Space panel's available operations: Print an object's URL, or a presigned S3 URL that expires, Describe and check a namespace's S3 endpoint, Copy an object to another key of its namespace, on the server and Move an object to another key of its namespace, on the server.

The real journey is tests/storage/s3_endpoint in the Stado repository (cargo test --test storage-s3-endpoint): in an isolated deployment with its own vault, the AWS CLI puts, lists, reads, copies, uploads in parts and deletes objects whose bytes are checked in the served store; a wrong secret, an unknown access key id and another bucket are refused; stado storage s3-endpoint reports the endpoint after its signed HeadBucket; presigned PUT and GET URLs work with no bearer while the same URL moved to another key is refused; stado storage cp lands the bytes at the target and keeps the source, mv removes it, and a copy to another namespace or from an absent key is refused with the target left as it was.

Source: this website