On this page

The submit queue

stado change lands agents' commits on a repository's main through one queue per repository. An agent commits on any main, hands the commit to the queue, and the queue writes it again on the current main, runs the product's declared gates on the result across the fleet, classifies its risk, has it reviewed before or after landing, pushes it, and records every defect it repairs.

stado change submit /path/to/checkout --commit FULL_SHA --defect DEFECT_ID [--defect DEFECT_ID]... [--json]
stado change run [/path/to/checkout] [--host HOST]... [--integrated DIR] [--json]
stado change status ID [--json]
stado change list [/path/to/checkout] [--state queued|running|landed|refused|repaired|cancelled] [--json]
stado change cancel ID --reason TEXT [--json]
stado change check /path/to/checkout --parent COMMIT --tree TREE [--defect DEFECT_ID]... [--host HOST]... [--json]

The checkout is the repository's one canonical checkout on the machine that runs the queue. The queue never reads or writes its working tree or index: every Git step works on commits and trees in its object store, so other sessions' uncommitted edits in the same checkout are never part of a candidate.

Submit

submit resolves --commit in the checkout, and fetches it from origin by its id when the checkout does not hold it, so a commit an agent pushed to a branch of origin on another host can be submitted from here. It records a queued candidate whose id begins with the submission time; a repository's candidates run oldest first. Each --defect is an Oko defect the commit repairs: the first scopes the harsh review, and every one is recorded repaired when the change lands. The candidate also records the agent session that submitted it (tama sessions self; none when the operator submits it himself), because its gates are that session's work. A session that cannot be read refuses the submission: the agent session submitting <commit> could not be read (<cause>), so whose work its gates are is not known.

A commit made on a main the queue has moved past (its parent is neither on origin's main nor under a candidate still queued or running) carries, under it, the original commits whose rewritten copies already landed. submit writes the commit's own change alone on the tip and queues that commit: stado change submit: <commit> was made on a main that carries <n> commit(s) <owner/name>'s main (<tip>) does not hold, the nearest <commit>; its own change is written alone on <tip> as <written>, and that commit is queued. A commit whose own change main already holds (an empty commit, or a change that landed under another id) is on main only when every commit under it is, by its change (git cherry); when one of them carries a change main does not hold, the commit is queued as it is, so the pass writes that whole change on the tip: stado change submit: <commit> changes nothing of its own on <owner/name>'s main (<tip>), but <k> of the <n> commit(s) it was made on carry a change main does not hold, the nearest <commit>; it is queued as it is, so the queue writes that whole change on <tip>.

A commit a harsh review left (see Review) sits on the review's own commits, none of them on main, and usually changes what they added, so its own change does not merge on the tip alone. When main has changed none of the files that chain changes since it was made, nothing under it landed under another id, and the commit is queued as it is, so the pass writes the whole change on the tip: stado change submit: <commit> was made on <n> commit(s) <owner/name>'s main (<tip>) does not hold, the nearest <commit>; main has changed none of the <k> file(s) they and it change since <base>, so it is queued as it is and the queue writes that whole change on <tip>. A chain whose files main did change is refused as stale-main, below.

Before a candidate takes its place, its commit is kept by a ref of its own, refs/stado/candidates/<commit>, in the checkout and on origin. A candidate is usually a commit made from a temporary index, which no branch names; without the ref the checkout's next git gc removed it, and with it the only copy of the work, while the queue still named it. A push origin refuses leaves the commit kept in the checkout only, said on stderr (stado change submit: <commit> is kept by <ref> in this checkout only; <why>; every pass tries origin again), and every pass keeps the commits of waiting candidates not yet kept on origin (those submitted earlier, review follow-ups, refused pushes). A pass whose checkout lacks a candidate's commit fetches it from origin's ref. The landing removes both refs; a refused, repaired or cancelled candidate keeps them, so its author can still read and submit what it holds. stado change status says kept by <ref> in its checkout and on origin (or in its checkout only: <why>), the record carries kept, and the pre-push check does not judge a push of a ref under refs/stado/candidates/.

Refusals:

  • a commit neither the checkout nor origin holds: <checkout> holds no commit <rev>, and origin did not give it (<git's answer>); push it to a branch of origin or commit it in this checkout, then submit it again;
  • a commit main already holds — the commit itself, or, for a commit whose own change main already holds, every commit under it by its change: <commit> is already on <owner/name>'s main (<tip>); there is nothing to land;
  • a commit already queued or running: <commit> is already <state> as candidate <id> of <owner/name>; \stado change status ` follows it`;
  • a commit whose change a queued or running candidate of the same repository already carries (stado.change.carried-elsewhere), the same change from their merge bases with main or a file both add, unless one is stacked on the other: submitting <commit> because waiting candidate <id> of <owner/name> already carries it (the same change | both add <files>), so whichever landed second would meet the first's copy on main after its whole review; the fix is to stack the change on that candidate's commit, withdraw one of the two, or keep only what that candidate does not carry;
  • a commit that adds, from its merge base with main, a path main already holds with other content (stado.change.added-on-main): submitting <commit> because it adds <paths>, which main (<tip>) already holds with other content, so the two additions would meet as a conflict only when it lands; the fix is to change main's copy instead (write the commit on the tip and edit what it holds) or to give the file a name main does not hold, and for a changelog part stado product changelog --entry writes the change's own entry file. A path main holds with the same content merges as one and is queued;
  • a checkout with no origin: <checkout> has no origin remote, and the submit queue lands on origin's main: …;
  • a commit whose keeping ref the checkout cannot write (stado.change.commit-unkept): keeping commit <commit> by refs/stado/candidates/<commit> in <checkout>: git refused the ref that keeps the candidate's commit, and a commit no ref names is removed by the checkout's next garbage collection, taking the candidate's work with it; observed: <git's answer>, fixed by repairing what git names (a file or directory in the way under .git/refs/stado, the checkout's access or space) and submitting again;
  • a product whose record declares no submit policy, or a commit whose .wisent-release.json the queue cannot plan;
  • a repository no catalogued product claims: no catalogued product in <catalog> claims the repository <owner/name> as its owner_repository or a surface's repository; the submit queue classifies a change by the product its repository belongs to: add <owner/name> to that product's surfaces;
  • a manifest naming another product: .wisent-release.json of <commit> names the product "<name>", but the catalog <catalog> maps the repository <owner/name> to the product <product>; set its "product" field to one of "<product>", "<repository name>"[, "<release product>"…] (the product, the repository's own name, or a release product an installation of it from this repository declares) (a pass meeting such a candidate refuses it with the same words);
  • a change the risk rules (see Risk) already call high over its own commits and declared gates that names no --defect: <commit> is high risk (<reasons>), so it is reviewed before it lands, and that review is scoped to the defect it repairs; submit it with --defect ID;
  • a commit whose author already holds an open entry in Oko's harsh-review queue, unless it is stacked on the candidate that carries that entry (stado.change.author-held): submitting <commit> because Oko's harsh-review queue keeps one open entry per author and <author> holds <entry>, so the pass could not enter this candidate's author before its review and the candidate would leave the queue unreviewed; the fix is to finish that change first (its reviewer records a round, or oko review harsh end ends the entry), or to build this commit on top of the candidate that carries it. Before, such a candidate was refused at its pass, after its gates had run, for its author not being entered: over three hundred candidates were refused that way.

Every verdict the queue gives a candidate is read by Oko each period (oko sessions verdicts pass): a refusal, a repair or a withdrawal goes to the session that submitted it, or to the session that took its work over when it was ended, with the reason; every verdict of an integrator's candidate wakes that integrator; and a refusal shape met twice becomes a defect for Oko's process engineer.

Check before review

check gates a staged tree as a pass gates a candidate, with no candidate queued: the tree is written as a commit no branch holds on --parent, and the product's declared gates run on it on the fleet (the same plan, source check and test gates as Run, with Oko's admission asked for the --defect purposes), so no person and no agent compiles it. It exits 0 with the verdict, passed or the gates' refusal; a gate that could not run at all is the command's error, with the cause a pass would name (stado.change.run, such as no-gate-host). Oko asks it of every harsh-review entry before the entry is handed to a reviewer (oko review harsh check), and an entry the gates refuse goes back to its author, so a reviewer never spends a round on a tree that does not compile. tests/change/check.rs stages a tree whose gates no host of their platform can run and shows it refused with no-gate-host and no candidate queued.

Run

run takes the repository's lock (one pass per repository at a time; a second is refused with a pass of <owner/name>'s submit queue is already running; …) and carries every queued candidate to a verdict, and every running one whose pass is gone (see Who runs the queue), which it says it resumes: <id> was left running by a pass that is gone; this pass resumes it. Without a checkout it does this for every repository of this machine's queue that has a candidate waiting for a pass, each from the checkout its oldest waiting candidate was submitted from, all at once: each repository's pass holds only that repository's lock, so one whose candidates wait on long reviews does not hold another's queue behind it. A repository whose pass already runs is named and left to it before any pass starts, and one whose pass stops does not stop the others: the pass carried <n> candidate(s) to a verdict, and the queue of <m> repository(ies) did not move: <owner/name>: <cause>; ….

  1. Rebase. origin's main is fetched. The candidate's own commits — those past its merge base with main — are merged one by one onto the tip with git merge-tree --write-tree and committed with git commit-tree, each keeping its author, author date and message. Each merge reads the merge attributes of the tree it merges onto (git -c attr.tree=<tip> merge-tree …; the queue has no working tree, so Git would read none otherwise): a repository whose .gitattributes declares CHANGELOG.md merge=union, as Stado's does, keeps both entries when two candidates each add one at the same place, instead of refusing every candidate queued behind the first to land. A Git older than 2.42 ignores attr.tree and merges without the attributes. A commit whose change main already holds is left out: one whose merge changes nothing, and one whose patch main already carries under another id (git cherry <tip> <commit> <base> marks it -), which is how a candidate's change lands through a candidate stacked on it, written again on main; a candidate all of whose commits are left out reads as already on main. A commit Git cannot merge refuses the candidate: <commit> of the candidate does not merge onto main <tip>: Git cannot merge <files>; rebase the change onto the current main and submit the new commit. Merge commits are refused; submit linear commits.
  2. Plan. The rebased commit's .wisent-release.json declares its gates; its product is the one the catalog maps the repository (origin's owner/name) to, as that product's owner_repository or a surface's repository (see Risk). A commit with no manifest, or whose manifest's product is none of the names the catalog accepts for that repository (see Risk), is refused with the submit refusal's words; a catalog that cannot be read or a repository no product claims holds the candidate on a declaration (see the waits table). Every gate runs on the host exactly as the manifest declares its argv — no gate is turned into a Cargo run, and none is cut into shards per test target — in the delivered tree's root, after the same prelude as stado host kompilator TARGET gate: kompilator compiled from the sources Stado delivers and first on PATH, the source contract WISENT_SOURCE_COMMIT and WISENT_SOURCE_SHA256, the host's installed Stado (~/.stado/bin/stado) as STADO_BIN and first on PATH, as a release worker's job runs the worker's own, and the history records a release build's archive carries beside the commit's files (.wisent-provenance/baseline.json and the published diff) written into the delivered tree, so a gate that calls Stado back, such as an application's version gate stado release gate app check, reads what it reads in a release build.
  3. Classify. See Risk below.
  4. Gates. The source check runs here, compiling nothing and running no declared gate: it reads the locks, the module declarations, the changelog links and the reach, as stado quality check --pre-push reads a commit's source. Then every test gate of every declared platform runs as stado host kompilator on a host of that platform, each host one shard at a time and all hosts at once; no format or lint gate runs. Before any of them, Oko's admission is asked once. A pass that runs under no agent session — the resident pass — asks first with no session, oko admit test --product P --purpose DEFECT…, the run the operator's consents held to the fleet's daily build (test[:<product>]@fleet-budget) admit, and only when those do not, as the submitting session's work (--session <submitter>). A pass started inside an agent session asks as the submitter only, never as the fleet's daily build. When the rulings rule tests out, no test gate runs and the pass goes on with what ran: each is kept on the candidate as a shard with exit -1, host none and left_out holding Oko's words, which quote the ruling that decided it by id and words — for the resident pass test of <product> is not run: the operator's rulings do not allow it now (<Oko's reason>); the resident pass asked with no session, as the fleet's daily build, and no consent of the operator held to the fleet's build budget covers test of <product>: a ruling of his bound to \test:@fleet-budget` (or `test@fleet-budget`) admits it, and `oko rulings constraints` lists what stands now, followed, for a candidate a session submitted whose own ask is refused too, by ; nor as the work of its submitter, session : test of is not run: the operator's rulings do not allow it now (<Oko's reason>); for a pass inside a session …; this run was started inside an agent session, so its gates were asked as 's work and never as the fleet's daily build, which only a run under no agent session asks for: the resident pass of the submit queue (the host routine running `stado change run`). The pass then prints the risk again from the gates that ran (stado change run: is risk from the gates that ran: ; see *Risk*), and the review refusal and the review-round wait end ; no lint or test gate ran on it (), so the reviewer judges whether it compiles by reading it. A candidate that lands with a gate left out landed without it: its integration line says gates: not-run. An Oko that gives no decision leaves the candidate queued: test of is not run: Oko gave no admission decision, …; gate () for was not run`. A machine with no Oko installed has no rulings, and its gates run. A shard that already passed on the same tree, platform, gate and argv is answered by the action cache without running. The first refused shard stops the hosts taking new shards and refuses the candidate with every refused shard's argv, host, exit and report.
  5. Review a high-risk candidate before landing (see Review).
  6. Land. The rebased commit is recorded first as the candidate's landing, then pushed: git push origin <rebased>:refs/heads/main, never forced, through the checkout's own push hooks. The pre-push gate reads that record and lets that exact commit through without running its gates again; any other push to main runs them. A main that moved meanwhile takes the candidate back to step 1, whose rebase clears the landing, so what lands is always the tree the gates checked. A refusal by the remote or a hook refuses the candidate with its words.
  7. Record every defect: one integration line {id, commit, gates, criterion, evidence, why_not_run} kept on the candidate and, with --integrated DIR, appended to DIR/<product>.jsonl; and oko defects repaired <defect> --revision <owner/name>@<landed>. A register write Oko refuses is kept under unrecorded; the landing stands.
  8. Review a low-risk candidate after it landed.

A refused candidate is a verdict: the pass goes on with the next ones and exits 0. A pass that cannot reach a verdict — origin, a fleet host or Oko's admission does not answer — puts every candidate of the train back to queued with the pass stopped before a verdict: <cause> and exits non-zero: <id> stays queued; the pass of <owner/name> stopped: <cause>.

One test run for the whole queue: the train

A pass does not test its repository's candidates one at a time. It takes every waiting candidate in the queue's order and writes them onto main one on top of the next: the first on the tip, each next one on the commit written for the one before it (steps 1–3 for each, the risk classified over the candidate's own change only). Then it runs the gates (step 4) once, on the newest commit, which holds them all. That run is the train. Each candidate's record keeps it as train: {gated, tree, candidates}, the commit the gates ran on, its tree and the ids it carried. status prints it as tested in a train of <n> candidate(s) at <gated>: <ids>, and the pass says stado change run: a train of <n> candidate(s) of <owner/name> on <tip>: <ids>.

A candidate leaves the train before the gates, and the ones behind it are written on the one before it, when:

  • its rebase onto main or its plan is refused;
  • Oko gives no admission decision for its own defects and submitter (it is held, as in step 4);
  • it is a candidate of Stado itself and its review before the gates does not admit it;
  • it does not write onto the candidates before it: Git cannot merge it onto them, or one of them already carries its change. It is not refused for that, because those candidates may still be refused, and then it writes onto main as it is. It waits for the next train, which judges it against what they came to: stado change run: <id> does not write onto the candidates before it in the train (<why>); it waits for the next train.

Oko's admission for the train's gates is asked once with every candidate's defects. The resident pass asks first as the fleet's daily build, then as each submitter's work. A pass inside a session asks as each submitter's work. A test ruled out for any of them is left out for the whole train, as in step 4.

A newer candidate supersedes the running gates, the way a newer release submission supersedes an older one's builds. While the gates of a train run, the pass reads the queue every 15 seconds, and once more as they end, because the source check holds the pass's thread while it exports and reads the tree. When a candidate of the repository was queued meanwhile, or a candidate of the train was asked to stop (stado change cancel):

  1. The gates are stopped at once: each host's gate process ends with the SSH session that runs it. A shard that already passed stays in the action cache, so the next run does not repeat it.
  2. The pass writes the train again from the current queue and tests its newest commit: stado change run: candidate <id> was queued while the gates of <commit> ran over <n> candidate(s); they are stopped, and the newest version is tested instead.
  3. A stopped candidate is withdrawn, and the train is written again without it.

Only the gates of a whole train are superseded. The gates that look for a refused candidate (below) run to their end.

A cancel ends a harsh review that is running. While Tama reviews a candidate before landing (or a candidate of Stado itself before its gates), the pass reads that candidate's cancel request every 15 seconds. Once one stands, the review's tama process is ended and the candidate is withdrawn with who asked, when and why, as a cancel met between steps is; the pass goes on with the rest of the queue instead of waiting out a review nobody wants.

When the train's gates pass, each high-risk candidate in it is reviewed before landing, in order (step 5). One the review does not admit leaves the train. The candidates behind it are written and tested again without it, because the tree the gates passed held its change. Otherwise the newest commit is pushed once (step 6), which lands the whole train. Each candidate is recorded landed as its own rewritten commit (step 7), and a low-risk one is reviewed after (step 8). A main that moved before the push writes the whole train again.

When the train's gates refuse, the pass looks for the candidate that broke it by halves. It runs the gates on the commit of the train's first half:

  • that half passes: it lands as a train of its own, and the candidates behind it wait for the pass's next train;
  • that half refuses: it is halved again.

A train of one candidate whose gates refuse refuses that candidate with them, unless the main it was written on is refused by the same gates without it. Before refusing, the pass runs the gates over that main alone (<id> alone was refused; main <commit> is tested without it); a main that landed through the queue passed them already, so the action cache answers it. When main refuses too, the flaw is its own (a push past the queue, or a manifest that no longer reads), and no candidate is refused for it: the pass stops with stado.change.run.main-refused, naming main's refusal and the candidate's, and every candidate stays queued. Submit the commit that repairs main and move it ahead (below): a train whose first half holds it lands that half. A train of n candidates costs at most about log₂ n gate runs beyond the first, and the action cache answers every tree that already passed.

A candidate held for its review round does not wait for the next pass once a round is recorded: before a pass ends it reads the rounds again over each candidate it held for review, and takes again those that now have one that counts for the tree they land (<id> has a review round that counts for the tree it lands, recorded after this pass held it; this pass takes it again).

A tree the harsh review before landing admitted is not reviewed again. The admission is kept on the candidate before the merged program is built, so a pass stopped after it (its build killed, its host restarted) leaves it, and the pass that resumes the candidate goes on to the gates with that admission as its review (an earlier run admitted tree <tree>: <answer>). Before, the resumed pass handed the same tree to a fresh reviewer, and an edit of one word refused what was already admitted.

A gate or build that fails on its host names where its evidence is (a service's stderr, a journey's report), and that place is inside the run directory the run removes as it ends. So before the removal, stado host kompilator HOST build|gate copies back every absolute path under the run directory that the run's stdout or stderr names, keeping its place in the run directory, into evidence-<run> beside the run's export: <crate>/target/host-kompilator/ for a build, the checkout's build area under host-gate/ for a gate. The receipt lists each under evidence, as {"state": "kept", "named", "at"} or, for a path that was gone or could not be copied, {"state": "unkept", "named", "cause"}; without --json each is one evidence: <named> is kept at <at> or evidence: <named> was not kept: <cause> line on stderr. A run that succeeded keeps nothing. The journey is tests/host_gates/evidence/journey.mjs.

The queue's order

A repository's candidates are carried oldest first, save those moved ahead, which go before every other in the order they were moved:

stado change prioritize ID --reason TEXT [--json]

The reason, the agent session that moved it (the operator when none) and when are kept beside the candidate in <owner>/<name>/<id>.priority, which every read of the candidate takes, so a pass writing its record cannot undo it; list and status print moved ahead by session <id> at <time>: <reason> under its line, --json carries priority, and list lists in the queue's order. Moving one ahead again re-stamps it, which sends it behind the others moved ahead before. Refusals:

  • an empty reason: --reason is empty; say why this candidate goes before the ones submitted before it, so whoever reads the queue sees why its order changed;
  • a candidate at a verdict: candidate <id> is <state>, so no pass takes it again and its place in the queue decides nothing; prioritize a queued candidate (\stado change list --state queued`)`;
  • a candidate stacked on another that still waits and was not moved ahead: candidate <id> is stacked on candidate <id> (<commit>), which still waits and was not moved ahead, so ahead of it the pass would land that candidate's commits as this one's; move it ahead first (\stado change prioritize --reason …`), then this one`.

Withdrawing a candidate

stado change cancel ID --reason TEXT [--json]

A queued candidate is cancelled at once; a running one is asked for, and its pass stops it before its next step. The request, <owner>/<name>/<id>.cancel, holds the reason, the agent session that cancelled it (none for the operator's own), the session that submitted it and when, and the candidate's record keeps them as cancellation ({"reason", "by", "submitted_by", "at", "recorded"}) once it is cancelled; list and status print cancelled by session <id> at <time>: <reason> under its line (the operator when no session ran it), followed by (submitted by session <id>) or (submitted by the operator) when someone other than the submitter withdrew it. A request that names nobody still withdraws a running candidate when its pass meets it, so no pass is ever held at a request it cannot read: the empty file an earlier release wrote (a release that kept no record of who asked or why wrote its cancel request), a file that does not hold a request (its cancel request <path> does not hold one (<parse error>), so who asked and why are not known), and one that cannot be read (its cancel request <path> cannot be read (<cause>), so who asked and why are not known). Its record then has recorded: false and the file's time (the time the pass read it when even that cannot be read), printed as cancelled by an unrecorded caller at <time>: <reason>. A candidate whose record was cancelled by an earlier release, before cancellation existed, prints no note.

Refusals:

  • no --reason: the following required arguments were not provided: --reason <REASON>;
  • an empty reason: --reason is empty; say why this candidate is withdrawn, so its author and whoever reads the queue see why it stopped;
  • a candidate at a verdict: candidate <id> is <state>, so there is nothing to cancel, followed by who cancelled it and why when it is cancelled. The state is read again under the request's lock before the request is placed, so a cancel that waited for that lock while the candidate landed is refused this way too;
  • a candidate cancelled while its pass pushes it (after the pass last read its request) is not refused, but the cancel comes too late: the candidate lands, and once the push landed the pass retires <id>.cancel under the request's lock, keeps who asked, when and why as the landed record's cancellation, and says on stderr, in status and in list cancel asked by <who> at <time>: <reason>; it came after the candidate could be withdrawn: it landed. A request the pass cannot retire is named among the record's unrecorded lines (the cancel request placed while it was pushed was not retired: <cause>);
  • with no session readable: the agent session cancelling candidate <id> could not be read (<cause>), so who withdrew it is not known;
  • a running candidate already asked to stop: candidate <id> already has a cancel request followed by cancel asked by <who> at <time>: <reason>; its pass withdraws it before its next step — the first recorded request stands, so a second cancel never erases who asked first. A request that names nobody (an earlier release's empty file, or one that cannot be read or parsed) does not stand: the cancel replaces it with its own, so its session and reason are kept. A queued candidate whose recorded request is already there (a pass put it back before it read the request) is not refused: it is withdrawn at once with that first request. A cancel of a running candidate answers with its request (cancel asked by …; its pass withdraws it before its next step under its running line, cancellation in --json); nothing claims it cancelled until its pass withdraws it, which may first refuse or land it, and its record stays the pass's;
  • a directory where the request belongs (a request that cannot be read, which a pass still takes as one): the cancel request <path> is a directory, so no request can be placed there; move that directory aside, then cancel again; the candidate stays as it was;
  • a request that cannot be written: writing the cancel request <path>: <cause>; repair what the error names (the queue directory's access or space), then cancel again;
  • a cancel lock that cannot be opened or taken: cannot lock <owner>/<name>/<id>.cancel.lock to place the cancel request: <cause>; repair what the error names (the queue directory's access or space), then cancel again;
  • a wait for that lock that would close a wait cycle: the wait cycle refusal (a wait cycle: process <pid> (<command>) waits on …).

A cancel standing behind another cancel of the same candidate waits for its lock as every Stado wait does (waiting): it says so on stderr (czekam: the cancel request lock of candidate <id>; gdzie: <the lock's full path>; rodzaj: blokada; …, then koniec czekania: …) and stado waits lists it while it lasts.

The hosts

Without --host the hosts come from their capacity publications, as a placed quality check chooses them: a host that is admitted or waiting takes shards, a refused one does not, and every verdict is printed (stado change run: placement: <host> admitted). With --host the named registry targets are the pool, as stado host kompilator HOST is the operator's own placement. A declared platform no host serves ends the pass: no fleet host of <platforms> can run the candidate's compiling gates: <every host's verdict>; name hosts with --host, or bring a host of that platform back to publishing capacity. A host that cannot run its shard (unreachable, its build volume full) is left out of the pass and its shard goes to the platform's other hosts; shards no host is left to run end the pass unchecked.

When the fleet store that keeps the capacity publications cannot be read (its object API answers 503, the vault behind it hangs), the pass — and the pre-push gate, which places its lint gates the same way — does not stop: this host runs the gates of its own platform first and every other host the registry names runs those of its platform unjudged, each said with the store's error (stado change run: placement: <host> (<platform>) is this host and runs the gates of its platform first, without the fleet's placement, since the hosts' capacity publications could not be read: <error>). A repair of the fleet store therefore lands through the queue that would otherwise wait on it. A registry that names no such host ends the pass with the store's error and and the registry names no host this machine can reach in the fleet's place; name hosts with --host, or let the fleet store answer again.

The action cache

A passing shard is kept under ~/.stado/change-queue/cache/<key>.json, the key being the SHA-256 of the tree, platform, gate and argv. A refusal is never cached, so a flaky test is run again rather than remembered.

Risk

A change's product is the one the catalog maps its repository (origin's owner/name) to, as that product's owner_repository or a surface's repository, never the name its .wisent-release.json states: a surface repository, such as the one holding Oko's landing site or its desktop app, is classified, gated and recorded as its product (oko). The manifest's product must name that product, the repository itself, or the release_product an installation of the product from this repository declares (a renamed product's installations keep its former id, as the worker installation from Weles's own repository declares weles-worker). A product declares how its changes are classified in its record of the product catalog:

submit:
  criticality: standard            # or critical
  review_paths:
  - .wisent-release.json
  - .github/

No size of a change decides anything: a declaration names no number of lines, and one that does (max_changed_lines, low_risk_max_changed_lines) is refused as --submit: <key> is not a field; the fields are criticality, review_paths. The catalog's records may still carry both, which the queue reads and ignores, until the Stado release that ignores them is installed: the release before it refuses every submit of a record without them.

A change is high risk for every reason that holds: the product is critical; it changes a path equal to a review_paths entry or below one ending in /; or no lint or test gate compiles or runs it. A gate the rulings leave out runs nothing: when one was, the risk is classified again once the gates end, from the gates that ran or the action cache answered, so a change whose every compiling gate was left out is high risk and reviewed before it lands, its reason reading <gate> (<kind>) for <platform> left out by the operator's rulings; no lint or test gate ran on it, only the source check read it. Otherwise it is low risk. The candidate keeps the level, every reason, the files and the lines changed. A product with no submit declaration is refused: <product>'s record in <catalog> declares no submit policy, so the queue cannot say whether a change of it is reviewed before or after it lands; …. The catalog refuses a review path that leaves the repository.

Review

A review is first the rounds that count for the tree the pass lands, recorded with stado change review by sessions that are not the candidate's author. A round binds the candidate's own change, not only that tree: the change is the candidate's diff from its merge base with main, made with options the queue pins rather than the host's or the repository's Git configuration (three context lines around each hunk, no rename detection, full blob ids, binary patches, no text conversion, every submodule change, a/ and b/ prefixes, the Myers algorithm), and its identity is git patch-id --verbatim of those bytes, so whitespace and line endings count. A round over the tree the pass lands always counts; a round over the candidate's own submitted commit, or over a tree an earlier pass wrote it as, counts for it when both have the same identity. So a main that moved under the candidate without touching those lines asks for no new round, and one that changed a line in or around a hunk, a re-indented line too (Git still merges it), asks for a round over the new tree. A landing that absorbed part of the change asks for one only when the tree the pass lands differs from the one the round read: on an unmoved main the upper candidate of a stack lands on its round over the same tree once the lower one lands, while on a main that also moved its round read a change that no longer holds the lower's hunks. A rebase Git cannot merge is refused (see Rebase). status, the review-round wait and the refusal that holds a candidate say, for each round over another tree that does not count, its commit and why: the round over commit <commit> (tree <tree>) read another change: <where>, where each place is a hunk of the landed tree the round did not read (<file>:<first>-<last>), a hunk the round read that the landed change no longer holds (the change no longer holds <file>:<first>-<last>, numbered as the round read it; a hunk whose context changed is named both ways), <file> (its mode or binary content), the change now holds <file> or the change no longer holds <file>, or, when the identities differ where this listing shows nothing, the bytes differ where this listing does not show them; or that it could not be compared, naming the Git read that failed. A verdict names a counted round over another tree by its session, time and tree (session <s> at <time> over tree <tree>, the same change) and leads the rounds that count for tree <tree> by … found nothing or the rounds that count for tree <tree>: …:

stado change review ID --findings FILE --improvements FILE [--json]

The reviewer is the agent session the command runs in, as its harness names it (tama sessions self, else the transcript PI_SESSION_FILE names), never a name typed. Each file holds a JSON array: every finding ([] when none), and what the change suggests would further improve the codebase ([] when nothing). The round is over the commit the next pass lands — the candidate as the last pass wrote it on main, else its own commit — and keeps that commit's tree; it is kept beside the candidate at <owner>/<name>/<id>.rounds/, never in the record a pass writes, so it can be recorded while a pass carries the candidate. At the review before landing, each reviewer's newest round that counts for the tree (over it, or over the candidate's own commit or an earlier tree with the same change identity) counts: any finding refuses the candidate naming it, and rounds that all found nothing admit the tree, so one reviewer never clears another's findings. A round over a change a moved main altered decides nothing.

Refusals:

  • a command no session runs: no agent session runs this command (neither \tama sessions self` nor the harness's PI_SESSION_FILE names one), so the round would be nobody's; a round is recorded from the reviewing agent session's own shell`;
  • the session that submitted the candidate (operator when none did): session <id> submitted candidate <id>, so it cannot review it: a round counts only from a reviewer that is not the author; have another session read \git -C show ` and record its round`;
  • a session name that is not a plain id: reviewer session "<name>" is not a session id (letters, digits, '.', '_' and '-', not starting with '.'), so no round is kept under it;
  • a file that is not a JSON array: --findings <file> holds <an object|a string|…>, not a JSON array; write a JSON array of every finding, [] when none;
  • a candidate at a verdict: candidate <id> is <state>, so no review of it can decide anything; ….

With no round that counts for the tree, the review is Tama's harsh review, one adapter of it. A machine without Tama runs no review itself: the candidate stays queued, waits on: a review round over commit <commit> by a session that is not its author (<author>); unblock: a reviewer reads \git -C show ` and, from its own session's shell, records `stado change review --findings FILE --improvements FILE`, then `stado change run `; when rounds over another tree do not count, the unblock first names each and where its change differs, in parentheses (unblock: (the round over commit (tree ) read another change: ); a reviewer reads …`).

A candidate whose gates the rulings left out is not handed to Tama either, since Tama reads only the candidate's own message, which cannot say that no gate ran it: it waits for a recorded round in the same way, and its refusal and its wait end ; no lint or test gate ran on it (<each left-out gate>), so the reviewer judges whether it compiles by reading it.

Tama's harsh review runs tama reviews harsh run --defect <first defect> over the tree staged in an index of its own in the checkout's Git directory: a reviewer session that is not the author attacks every changed line and stages its repairs. Its verdict is admitted when its rounds admit the tree and it repaired nothing, else changes-requested. The reviewer's repairs are committed on the reviewed commit, with the message as the reviewer left it, and queued once as a new candidate carrying the same defects, so they go back through the gates. When the reviewer's own rounds admit the tree it left (Tama exits 0), its newest round over that tree in the review repository's Tama register (tama-harsh-review/<parent>.jsonl), one that found nothing, is kept beside that new candidate as a round of the same reviewer session over the repairs' commit: the new candidate's review counts it, also over the same change written again on a newer main, and lands it without asking the reviewer again; the repaired candidate's record ends carrying the round of reviewer session <id> that admits it. A register that cannot be read, or a round that cannot be kept, is said in the answer and leaves the new candidate to a review of its own. A follow-up's own repairs are queued the same way when such a round admits them; with none they are not queued again: the refusal names their commit (… and no round of its reviewer admits it, so it is not queued: read it (\git show `) and submit what stands with `stado change submit``), so the reviews of one change end. A reviewer that cannot be run, or repairs that cannot be committed, are no verdict: the candidate stays queued and the pass ends with the cause.

  • High risk: reviewed before the push. Before the review the candidate is entered in Oko's harsh-review queue (oko review harsh enqueue) as staged by the session that submitted it (the operator when none did), so Tama judges the rounds against that author and never against the session running the pass. Oko queues an author's change beside the author's open entries, so an entry an earlier pass left waiting does not keep the next candidate out. When Oko is not installed or does not run, the review runs anyway and its record begins with why; when Oko refuses the entry, no review runs, and the review ends with no round (no-round), answering no review ran: the candidate's author was not entered in Oko's harsh-review queue, so Tama would judge the tree as written by the session running the pass: <Oko's words>. A review whose rounds admit the tree and which repaired nothing lands it. A review that found something and repaired nothing refuses it: the harsh review before landing did not admit it (changes-requested: <what was found>). A review whose reviewer staged repairs, rewrote only the message, or ended with no round covering what it left takes the candidate off the head of the queue as repaired: its record names the reviewer's commit and tree (repairs), and that commit — the reviewer's tree and message on the reviewed commit, or the reviewed commit itself when nothing was staged — is queued at the tail as its follow-up with the same defects, and with the reviewer's round when its rounds admit what it left, so the next pass never runs the same review at the head again and the candidates behind it move. A message-only repair keeps the reviewed tree, so the follow-up's gates are answered by the action cache. A follow-up whose own review ends that way is refused, naming the commit for its author to read and submit, unless a round of its reviewer admits what the review left: then that commit is queued carrying the round, as above. A candidate with no --defect cannot be reviewed and is refused.
  • What a refused or repaired candidate says: a candidate the review before landing did not admit carries the reason the harsh review before landing did not admit it (<verdict>: <what was found>). What was found is the reviewers' recorded rounds over the reviewed tree — who read it and each finding — and the reviewer's last line only when no round recorded one, because a reviewer session may close every answer with a block its own contract appends, and that block's last line says nothing of the change.
  • Low risk: pushed, then reviewed. The repairs of that retroactive review are its follow-up candidate; a review that fails after landing is recorded as not run and never puts a landed candidate back in the queue.

Read and withdraw

status prints a candidate whole: its rebase, risk and reasons, every shard with its host, exit, time and whether the cache answered it, each gate the rulings left out as gate <name> (<kind>) for <platform>: not run, left out: <Oko's words>, every review, its landing, refusal and unrecorded writes. list prints one line per candidate, ending, for one whose gates the rulings left out, ; gates left out: <name> (<kind>) for <platform>, …, see \stado change status `. cancelcancels a queued candidate at once; for a running one it writes//.cancelbeside the record, which the pass reads before each step (before each rebase and before the push), so a running candidate's record has one writer, its pass. Cancelling a landed, refused or cancelled candidate is refused:candidate is , so there is nothing to cancel. An id that names no candidate: no candidate of the submit queue at

has the id ; `stado change list` names every candidate`; an ambiguous prefix names the candidates it matches.

A shard's report is everything its gate printed, standard output then error output, and a review's answer is everything the reviewer said, its findings and its verdict, with the wait lines (czekam:, koniec czekania:, blad czekania:) left out. Neither is cut to its last lines: the line that says why a gate refused, a compiler error above its warnings or a failing test above the summary, is kept wherever it stands, and a refused pass's refusal carries each refused shard's whole report. status prints each shard's exit and time and each review's verdict, not their text, except that a left-out shard's line carries Oko's words for the ruling that left it out; the whole text is read with stado change status <id> --json, as shards[].report and reviews[].answer, or in the candidate's own record at <owner>/<name>/<id>.json under the queue's directory.

Where a candidate's time went

Every candidate's record keeps periods: each step of its way to main, in the order it went through them, with started_at and ended_at (null while the step lasts). The steps are queued (from its submission, and again each time a pass puts it back), written onto main (its rebase, plan, risk and Oko's admission of its gates), review before its gates (a candidate of Stado itself), gates (its train's gates on the hosts; each gate's own time stays on its shard), review before landing (a high-risk candidate), push to main (with the push turn it waits for), recording the landing (its defects repaired in Oko, its refs released) and review after landing (a low-risk candidate). A pass enters each step where it starts it, which ends the step before; saving a candidate at its verdict ends its last step. status prints each as time <step> <how long> (<from> to <to>), or time <step> <how long> so far (since <from>) for the step it is in, and list adds, for a candidate without a verdict, in step "<step>" for <how long>, <how long> since it was submitted. A record written before periods were kept has none and prints no time.

What a candidate waits on

Every candidate without a verdict carries, under its line in list and status (and as waits in --json), what it waits on and the command that unblocks it. A pass that stops before a verdict records the step as waits_on:

waits on Meaning Unblock
the next pass of the queue queued, no pass has taken it stado change run <checkout>, or the resident pass
a pass running, and the pass that carried it (its pass.pid) is gone: the repository's pass lock is free stado change run <checkout>, which resumes it
the remote origin did not answer a fetch or a push make git -C <checkout> fetch origin main answer, then stado change run <checkout>
Oko's admission of its gates Oko gave no admission decision for a test gate (its register unreadable, a purpose that names no open defect, the submitter's role or delegation unreadable), so whether the rulings allow it was never read make the pass's own asks answer, oko admit test --product <product> --purpose <each defect> --json and, for a candidate a session submitted, oko admit test --product <product> --purpose <each defect> --session <submitter> --json, then stado change run <checkout> from no agent session
a host for its gates no host of its platforms could run them stado change run <checkout> --host HOST, or bring a host of that platform back to publishing capacity
a declaration the catalog could not be read, no catalogued product claims its repository, or its product's record has no submit policy (a catalog that does not validate, a repository missing from every product's surfaces, a record without a submit policy): this machine's declaration, not the change, so the candidate is not refused, and the pass carries the candidates behind it add its repository to its product's surfaces, declare a policy (stado product registry set <product> --submit '{…}') or repair the catalog by a commit to it, then stado change run <checkout>; the first pass after it reads judges the candidate
a review round its review before landing has no round over the tree it lands or over an earlier tree of the same change: none was recorded with stado change review (or one was, over a change a moved main altered), and Tama is not installed, a gate the rulings left out keeps the candidate from Tama (the wait then ends ; no lint or test gate ran on it (<each left-out gate>), so the reviewer judges whether it compiles by reading it), or its reviewer was stopped before tama reviews harsh record; such a hold needs no defect a session that is not the author records stado change review <id> --findings FILE --improvements FILE from its own shell, then stado change run <checkout>
the next pass of the queue, which lands it / which refuses it held for its review, and rounds have been recorded since over the tree it lands or an earlier tree of the same change: status judges them as the pass will, naming each finding stado change run <checkout>; for a refusal, the author repairs what the rounds found and submits the repair

A candidate a pass is carrying now reads carried by a pass (pid <pid>, since <time>); \stado change status ` follows it. Whether its pass still runs is read from the kernel, not from the lock: the process holding the pid the candidate records must have been born no later than that pass wrote itself on it, else that pass is gone, whichever pass holds the lock. A candidate whose pass is gone while another pass holds the repository's lock reads waits on: a pass; the pass that carried it (pid ) is gone (killed, or its machine restarted), and another pass of / holds its lock now: that pass resumes it in the queue's order (`stado change prioritize` moves only a queued candidate), and `stado change run ` is refused while it runs; with the lock free it reads … and holds no lock of /; unblock: `stado change run `, which resumes it, or the resident pass of the queue`.

Who runs the queue

The Stado CLI installation declares the queue's resident pass as a host routine of every machine it installs on: stado schedule create --id <install id> --converge --host-store --cron '*/10 * * * *' '"$STADO_BIN" change run'. The host's own stado serve --worker fires it from ~/.stado/host-store every ten minutes (overlap skipped, so a pass that runs longer is never doubled), outside any agent session and whether or not the fleet store answers (host-local routines). Its output is in ~/.stado/host-store/logs/<job-id>.log.

A pass run inside an agent session's own tool call ends with that session: its process, the harsh reviewer it started and the candidate it carried are killed together, and a reviewer killed before tama reviews harsh record records no round. That is why a session submits and the resident pass runs. A pass records itself on the candidate it carries (pass: {pid, started_at}), and a running candidate whose recorded pass no longer runs (the kernel holds no process of that pid born before started_at) is one a pass that is gone left behind: the pass that holds the lock next resumes it in the queue's order.

The queue is kept under ~/.stado/change-queue/ (STADO_CHANGE_QUEUE moves it): one JSON document per candidate at <owner>/<name>/<id>.json, its cancel request beside it with the lock two cancels of it take in turn (<id>.cancel.lock, so a second cancel always finds the first's request), and the pass lock at <owner>/<name>.lock. A cancel that cannot take that lock says cannot lock <path> to place the cancel request: <cause>; repair what the error names (the queue directory's access or space), then cancel again.

A harsh review runs in a bare review repository the pass makes under the checkout's Git directory and removes when the review ends. Tama keeps each reviewer run's brief, events and errors in it, so when a review records no round or fails, those runs are first moved beside the candidate's record, to <owner>/<name>/<id>.reviews/<when>/, and the review's answer names that folder instead of the removed repository (stado change run says the reviewer runs of <id>'s review are kept at <folder>). When they cannot be moved, the review repository is left in place and the pass says why and where it stands.

The shared checkout's work

Writing a commit of the whole shared tree (stado change commit), carrying every session's uncommitted work onto main (stado change carry), bringing a side tree's work into the checkout (stado change gather) and bringing back a checkout whose main moved without its working tree (stado change sync, --follow), with each refusal: the shared checkout's work.

In Stado Desktop

The Queue screen carries the same nine operations below the job operations: Queue a commit to land on its repository's main, Run one pass of a repository's submit queue: every waiting candidate as one train, its gates once on the newest version on the fleet, review, land, Read one submit queue candidate: its rebase, risk, every gate shard, reviews and landing, List the submit queue's candidates in the queue's order: those moved ahead first, then oldest first, Withdraw a queued or running submit queue candidate, with the reason, Move a queued submit queue candidate ahead of the queue, with the reason, Bring a checkout whose main moved without its working tree back to HEAD, keeping every path somebody wrote, and move its own main to origin's (its Give each stale path HEAD's content switch is --apply, its Then move the checkout's main to origin's switch is --follow, and its carried commits, one per line, are --carried), Write a commit of every change the shared working tree holds onto the repository's main (chosen files only when that carry overlaps main) (its chosen files, one per line, are --put; blank commits the whole tree) and Gate a staged tree by its product's declared gates before its review, with no candidate queued. Each runs the same stado change command on the selected Stado API host and shows its answer or refusal; Read one submit queue candidate shows the train its gates ran in. The withdraw operation's Why it is withdrawn field and the move-ahead operation's Why it goes before the candidates submitted before it field are required: each is refused without one, as the CLI is.

Source: this website