Repository navigation
Conversation
…ing it A Colima VM that Lima's VZ driver has put into an error state is still listed as running, with its host agent alive and its Docker socket dead. The role's start found the profile running and did nothing, so the runtime tasks failed and nothing in the role could recover the host. When the runtime does not answer and `colima list` shows the profile as running, the role now ends it with `colima stop --force` and starts it again from the same disk. Nothing in the VM's disk, containers or volumes is removed. A profile that is listed as stopped, one whose runtime answers and a run under --check are left as before.
…opping it A single failed docker info is no longer enough to stop a VM. A profile that Colima lists as running while its runtime is silent is stopped only when docker info keeps failing on repeated bounded tries, a command in the guest fails, the VM's host agent is older than a boot grace period and the Docker context points at that profile's own socket. The stop is colima stop first and colima stop --force only if that fails. The stop has its own switch, github_runner_cluster_runtime_force_stop_wedged, separate from autostart. Switched off, or when a condition does not hold, the run fails with the signals it saw and the commands to recover by hand. Under --check every read is made and nothing is stopped.
…'s age live A timer set with alarm before exec does not end a Go program, which ignores SIGALRM, so a runtime that accepts a connection and never replies held the run, and a clean stop could block for ever. The probes, the guest command and the clean stop now run under files/run_bounded.pl, which gives the command its own process group, sends TERM at the deadline and KILL after a grace period, and reports a timeout distinctly. The first probe of the runtime is bounded the same way. The age of the VM's host agent now comes from the live process the pid file names, and only when that process is Lima's host agent. A pid file that outlived its process, as one from before a reboot does, or one that names another program, leaves the age unknown, which never counts as old.
… correctly A context whose endpoint is not a Colima socket is refused earlier, in the filter that reads the home from it, so the stop's own comparison is reached only when the home variable names another home than the context's socket. The test now sets exactly that, and the test of the variable winning over the context no longer asserts that the context is not read, since the comparison reads it.
… a started VM answer The report of a VM whose host agent is young now says "for less than" the grace period, matching the other wording of the age. The test stand-in for a restarted VM clears the hang modes of the stopped one, as a real start does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Colima VM in Lima's VZ error state is still listed as running with its host agent alive, so a start does nothing and the Docker socket stays dead. The role now stops such a profile (colima stop, then colima stop --force if that fails) and starts it again from the same disk, but only when docker info keeps failing on repeated bounded tries, a command in the guest fails, the VM's host agent process has been running for longer than a boot grace period and the Docker context points at that profile's own socket. Every probe, the guest command and the clean stop run under files/run_bounded.pl, which owns the deadline (TERM to the process group, then KILL after a grace), because Go programs ignore SIGALRM. The agent's age is read from the live process the pid file names, never from the file's date. The stop has its own switch, github_runner_cluster_runtime_force_stop_wedged, separate from autostart. When the switch is off or a condition does not hold the run fails with the signals it saw and the commands to recover by hand. Under --check every read is made and nothing is stopped. No disk, container or volume is removed; a forced stop ends the guest without a graceful shutdown. Tests cover the confirmed case, a clean stop that fails, probes and stops that never reply and ignore SIGALRM and TERM, a daemon that answers on a later try, a guest that answers, a booting VM, a stale or foreign pid file, a context pointing at another socket, the switch, check mode and a second run, and the supervisor script itself.