How a move's health probe ends
How a move's health probe ends
A health probe ends on facts, never on a count of attempts or a clock. A
running source's probe URL must answer 2xx once; any other answer refuses the
move before the source is fenced, as <host>: <url> answered HTTP <status> or
… answered no HTTP answer (curl exit <code>: <curl's error>). After the
destination's units start, every probe URL is read at once, each again for
as long as its unit's process runs (launchd state = running, systemd
active) until it answers 2xx from that process: a 2xx or 4xx counts only once
the socket table shows the unit's process listening on the URL's port, since
another program on that port answered 2xx for a unit that had exited and the
move committed a service that never ran. The move goes on once every probe
has ended, and when more than one refuses, the first in the profile's order
is the one reported. Every answer that differs from the one before is printed
as <host>: unit <unit> runs; <url> answered …; reading it again (… does not listen on that port yet; … while another program or nothing answers). A 4xx
answer from the unit is the service refusing the request itself, so the move
is rolled back at once with <host>: <url> answered HTTP <status> while unit <unit> runs; the probe path or the service refuses the request, so the move does not wait for it. Once the unit no longer runs the move is rolled back
with <host>: unit <unit> is no longer running; <url> last answered …, which
adds , from another program listening on port <port>: the unit's own log says why it exited, and the port the directory gives it is taken when that
last answer was 2xx or 4xx.
Between reads the destination's socket table is asked which TCP ports the unit's
process (launchd pid, systemd MainPID) listens on: lsof on Darwin, ss
on Linux. A process that listens on ports none of which is the probe URL's is
serving elsewhere, and the move is rolled back with <host>: unit <unit> listens on TCP [<ports>], not on port <port> of <url>, which answered …; the probe URL or the service's port is wrong. A process that listens on no port
yet is still starting, and one listening on the URL's port that answers 5xx is
the service saying it is not ready; both are read again, a new process id is
printed as <host>: unit <unit> restarted: process … is now …, and any line
the socket table's reader writes that is not a socket is printed as
<host>: socket table of unit <unit>: <line>. The wait has no limit; it
stands in the register of open waits, so stado waits names it for as long
as it lasts. Releasing the placement claim goes through the registry's
commit, which reapplies it only after a lost race and answers every other
failure as it is.
Source: this website