What a placement move does on its hosts

What a move does on its hosts

A move needs its source to answer: it stops the source's units, copies its states and fences its store over the source's channel. When the source does not answer on any SSH path (switched off, asleep, off the network), each of those steps stands until its caller gives up, so the service stays placed on a host nobody reaches. Two commands take from such a host instead, each run on the host that takes, each refused while the source answers on any path (then it is moved), and each recorded in placement history:

• stado placement role take SERVICE --reason WHY [--check] [--json] takes a role of the host Stado process. This host must hold the registry authority (stado placement move --authority); its own object API serves the role, and the registry from its last-known-good copy is written there. What the silent store held and this one does not is named as not_recovered.
• stado placement take SERVICE... --reason WHY [--check] [--json] takes a placement profile. Each SSH path of the source is probed once under the channel's own connect bound; one that answers refuses it (<source>, where <profile> is placed, answers over <path>; move it with its states instead: stado placement move <services> --to-host <here>). SSH closed is not the units stopped, so each probe URL the profile declares for the source is then asked at every network name the source is known by (each SSH path's host and each declared host name, with the URL's own port and path), within the refresh interval this host's resolver declares; an HTTP answer of any status refuses it, since a copy started here would serve beside the source and the two would write diverging state (<source> still serves <service> at <url>, although no SSH path of it answers; … stop it on <source>, or move the profile once <source> answers over SSH). A resolver that declares no refresh interval refuses it too, naming that. This host must be one of the profile's hosts (placement profile "<profile>" does not support destination "<here>"; prepare it with stado placement standby …) and must hold every required state, since nothing can copy it from the source (<here> holds none of $HOME/<path>, which <profile> requires, and <source> does not answer to give it; put it on <here> first, or move the profile once <source> answers); an optional state it lacks is listed as not_recovered. For every state it holds, the receipt gives the age of this host's copy, the newest modification time of the file or of anything in the tree (state_age: {"<state>": {"modified", "seconds_ago"}}, <state>: <here>'s copy was last written <time> (<n>s ago); what <source> took after that is not in it in text): every write the source took after it is lost. A routing unit on the source refuses it, since nothing can retire it there; routes on other hosts are applied as a move applies them. The profile's units then start here in its start order, every probe must answer 2xx, and one registry commit places the profile here, the same change a move commits. The source's units are not stopped: the receipt names them under unretired_source (still on <source>: <unit>; retire it there before <source> serves again). A profile carrying the fleet vault is refused, since its owner key cannot be rotated without the source. --check reads the source's paths and services and this host's states, and starts or writes nothing (state: would_take); --json prints {"state", "profile", "from_host", "to_host", "reason", "state_age", "not_recovered", "unretired_source", "registry_generation"}. Real journey: tests/placement/take_profile.rs.

A destination that already holds its own file or tree at a state path keeps it. The move writes the source's state over that path behind a backup, and once the move commits the backup is renamed to <path>.replaced-by-placement-<transaction> beside the moved state instead of being deleted, because what the destination held there was never part of the source. The receipt lists each kept path under kept_replaced_state (kept what <host> held there before the move: <path> in text). Every backup is settled at once and listed in the profile's order. A backup that could not be settled is listed under unsettled_state with its cause and stays under its .pre-stado-placement-<transaction> name; the move itself is committed either way. A path the destination did not hold leaves nothing behind, and a rollback before the commit restores the destination's own file in place as before.

Every step of a move is its own wait on stderr, opened before the step starts and closed with how long it took: czekam: placement move <transaction> of <profile>: <step>; gdzie: <host>; rodzaj: host; …, then koniec czekania: …; trwalo: <seconds>s or blad czekania: …; przyczyna: <cause>. The steps are claim the transaction and commit the placement (on the registry), preflight (on both hosts), copy <state> while it serves, fence <unit>, read what changed in <state> and retire <unit> (on the source), and write <state>, apply routes and start <unit> (on the destination); each health probe is its own wait on its unit answering 2xx. The preflight and every copy <state> while it serves run before claim the transaction: an open transaction refuses the profile's services to every consumer, so the first copy is made while they still resolve and answer. The claim is then laid on the registry as it stands after the copy, and refused, with nothing claimed or fenced, when the profile is no longer placed on the source. After the fence each state is read again into the same copy, compared by content (rsync --checksum), so rsync sends only what changed in between and the services are refused for that difference, not for the whole copy. Every copy of a profile runs at once on the source, then every write <state> at once on the destination, each its own wait, so a profile with several stores stands its longest transfer rather than their sum; when one of them fails the others still end before the rollback starts, and the failure reported is the first one in the profile's order. The copies this machine staged are removed once the states are installed, when the move rolls back and when it fails before its claim. A move that stands names the step it stands on, stado waits lists it however the move's stderr was kept, and a later move's wait on the same step opens with the time it is due, measured from the longest one that ended.

The receipt lists every step the move ran with its host and time: --json carries steps: [{step, host, seconds, succeeded}], including each probe as probe <service>, and the text prints one line per step, <step> on <host>: <seconds>s, with (failed) on the step that ended without its answer. A move that fails names the same lines in its error after steps the move ran:, so the step that held a long move is read from its own receipt. The preflight asks the source and the destination about each unit at once, so a slow source no longer delays the destination's answer; when both refuse, the source's refusal is the one reported. The running source's health probes, the existence of every state file on the source and every routing unit are each asked at once too, and the first refusal in the profile's order is the one reported; a state read that fails refuses the move whether or not that state is required. Before the source is fenced the preflight also measures what the states take on the source (du -sk, at once for every state) and what the destination's volume under each state root has free (df -Pk of the nearest existing directory of the state's path), and refuses when the states written under one root need more: <destination>: the volume the move writes <states> to has <free> KiB free, and they take <need> KiB on <source>; free space there, or declare a larger work root, before moving. Every state, a file as much as a directory, is copied through the machine running the move with rsync -a over the host's channel, never inside a host script's text, so its size is also checked against the free space under that machine's staging directory (its work root's placement, or ~/.stado/placement): <host>: <states> is copied through <dir> on this machine, which has <free> KiB free, and it takes <need> KiB on <source>; …; a machine the registry does not name is not measured, which is said on stderr. A state is pushed to <path>.placement-<transaction>.stage beside its path on the destination and renamed over it, after the destination's previous file or tree was renamed to the backup the rollback restores; a file state is installed owner-only (mode 600). A state that is not the kind the profile declares (a directory where a file is declared, or the reverse) is refused naming its path. A file state under the home is a file of that account: before it is pushed, each mention of the source account's home in it, followed by /, a quote or the end of a line, is rewritten in the copy to the destination account's home, and the move prints <destination>: <n> mention(s) of <source home> in <path> now name <destination home> on stderr. Copied verbatim, a Weles worker.env named STADO_BIN=<source home>/.stado/bin/stado on the destination, and the moved service exited on start. A directory state, a work-rooted state and a file that is not UTF-8 are carried byte for byte. The move's progress lines are printed on stderr; with --json, stdout carries only its report. A reading that returns no figure, or one that is not a number, refuses the move naming the host.

A profile moved by a Stado that did not yet rewrite homes runs with its file states still naming the source account's home. They are rewritten in place, on the host the profile runs on, with:

stado placement rehome <profile> --host <host> --from-home <source home> [--json]

Each home-rooted file state of the profile is read, every mention of <source home> is rewritten as a move rewrites it, and the file is installed through the same stage and rename, owner-only; the file it replaces is kept beside it as <path>.replaced-by-placement-<id>. One line per state reads <state> <outcome> <mentions> <kept path or ->, the outcome being rewritten, unchanged (no mention), absent (the host holds none) or carried_as_is (a directory or work-rooted state, which is never rewritten). Refused: a profile the registry does not hold, a host the profile does not run on (<profile> runs on <placed>, not <host>), a --from-home that is not an absolute path, and a --from-home equal to the host's own home. Restart the profile's services afterwards so they read the rewritten files.