Repository navigation
CKS: refuse or fail fast when etcd nodes cannot get etcd from the Kubernetes ISO - #14706
Draft
andrijapanicsb wants to merge 3 commits into
Draft
andrijapanicsb wants to merge 3 commits into
andrijapanicsb wants to merge 3 commits into
Conversation
A cluster with dedicated etcd nodes was accepted even when the ISO of the chosen Kubernetes version did not include the etcd binaries. The etcd nodes then never came up and the cluster sat in Starting until the one hour start timeout. The secondary storage VM can now be asked whether an ISO contains a given file. The result for etcd/etcd-linux-amd64.tar.gz is saved as the ISO detail cks.etcd.included, so each ISO is checked only once. A background task checks the Kubernetes ISOs that have no result yet, which also covers ISOs registered before an upgrade. If there is still no result when a cluster with etcd nodes is requested, the check runs at that point. createKubernetesCluster with etcd nodes is rejected with a clear message when the ISO has no etcd. When the ISO cannot be checked, for example because the storage VM is down or the ISO is direct download, the request is allowed as before. listKubernetesSupportedVersions returns supportsetcd, and the UI shows a warning when etcd nodes are asked for with a version that lacks etcd. Fixes apache#14703
When etcd could not be set up on a dedicated etcd node, nothing noticed. The cluster stayed in Starting until the one hour start timeout and no reason was given. The etcd node setup script now stops when the etcd archive is missing or broken, or when the Kubernetes ISO never shows up, and writes the reason to /home/cloud/etcd-setup-failed. etcd is only started when the setup worked. After the ISO is attached, the management server checks over SSH that etcd is running on every etcd node. etcd has to be seen running on two checks in a row, so a service that keeps restarting does not pass. If a node reports a failure, or etcd is not running within cloud.kubernetes.etcd.node.setup.timeout (default 600 seconds), the cluster goes to Error with the reason. Related to apache#14703
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #14706 +/- ##
=============================================
- Coverage 20.77% 3.86% -16.92%
=============================================
Files 6053 494 -5559
Lines 548230 42875 -505355
Branches 73237 8136 -65101
=============================================
- Hits 113903 1657 -112246
+ Misses 420988 40986 -380002
+ Partials 13339 232 -13107
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
When a cluster with etcd nodes is requested and the ISO has not been checked yet, read the copy of the ISO in the zone the cluster is deployed into. Before this, any copy that was ready could be read, also one in another zone. If the copy in that zone cannot be checked, the request is allowed as before and the check during cluster start catches a failed etcd setup. The background task is not tied to a zone and still reads any copy that is ready. Related to apache#14703
Contributor
Author
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Failed. Packaging results:
No public repository was promoted. Status: Package building failed. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In short: a Kubernetes cluster with dedicated etcd nodes is now refused straight away when the Kubernetes ISO has no etcd in it. If something still goes wrong with etcd on a node, the cluster fails in minutes with the reason, not after an hour with none.
Fixes: #14703
Why it matters
An operator registers a Kubernetes version from an ISO built without the optional etcd version (the ISO build script only adds etcd when asked). A user then creates a cluster with one or more etcd nodes. Today the request is accepted, the etcd nodes boot, find no etcd on the ISO and quietly do nothing. The cluster sits in Starting for the full one hour start timeout and then fails with no hint about the cause.
What changes for the user
How it works
Two parts, one commit each.
1. Refuse up front. The secondary storage VM (the system VM that serves templates and ISOs) can now be asked whether an ISO contains a given file. The answer for the etcd archive is saved with the ISO, so each ISO is looked at only once. A background task looks at Kubernetes ISOs that have no answer yet. If there is still no answer when a cluster with etcd nodes is requested, the check runs right then. When the check cannot run, the request is allowed as before; this never blocks a cluster that would have worked.
2. Fail fast. The etcd node setup script stops when the etcd archive is missing or broken and writes down why. During cluster start, the management server checks over SSH that etcd is running on every etcd node, within a bounded wait. If not, the cluster goes to Error with the reason.
Details
Part 1
CheckFileInIsoCommand/CheckFileInIsoAnswer.NfsSecondaryStorageResourcelists the ISO withisoinfo -R -f -i <iso>(no mount) and looks for the path.cks.etcd.included=true/falseforetcd/etcd-linux-amd64.tar.gz. No detail means "not checked yet".KubernetesVersionManagerImplruns a background check every 5 minutes for Ready Kubernetes ISOs without the detail; it stops trying an ISO after 3 failed background checks (the check at request time still runs).createKubernetesClusterwithetcdnodes> 0:truegoes ahead,falsethrowsInvalidParameterValueException, unknown runs the check once and acts on it, and a check that cannot run lets the request through.listKubernetesSupportedVersionsreturnssupportsetcdonce the ISO has been checked. The create cluster form shows a warning when etcd nodes are set and the selected version hassupportsetcd= false.Part 2
etcd-node.yml: on a missing or broken archive, or when the ISO never shows up, the script writes the reason to/home/cloud/etcd-setup-failedand exits non-zero.systemctl enable --now etcdonly runs when the setup worked.KubernetesClusterStartWorker.verifyEtcdRunningOnEtcdNodesruns right after the ISO is attached. etcd has to be seen active on two checks in a row, so a service that keeps restarting does not pass.cloud.kubernetes.etcd.node.setup.timeout, default 600 seconds. The wait never goes past the cluster start timeout.CreateFailed).Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
Screenshots (if appropriate):
None yet.
How Has This Been Tested?
Draft: unit tests only so far. A live run is planned on an environment built from this PR's packages.
testExecuteQuerySnapshotZoneCopyCommandNot verified yet:
How did you try to break this feature and the system with this change?
In unit tests: ISO missing on storage, non-NFS store,
isoinfofailing, direct download ISO, no storage VM endpoint, an answer from an older agent that does not know the command, etcd that flips between running and stopped, an etcd node that cannot be reached over SSH, and a setup timeout longer than the cluster start timeout. In all cases where the ISO cannot be checked the request is let through, so an upgrade cannot start refusing clusters that worked before.