[v1.15] Prevent Cilium agents from incorrectly restarting an etcd watch against a different clustermesh-apiserver instance. #32005
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ upstream commit df3c02f ] [ backporter's notes: dropped the session affinity changes, and backported only the introduction of the unique cluster id which, together with the interceptors backported as part of the next commit, prevents Cilium agents from incorrectly restarting an etcd watch against a different clustermesh-apiserver instance. ] This commit makes changes to the helm templates for clustermesh-apiserver to support deploying multiple replicas. - Use a unique cluster id for etcd: Each replica of the clustermesh-apiserver deploys its own discrete etcd cluster. Utilize the K8s downward API to provide the Pod UUID to the etcd cluster as an initial cluster token, so that each instance has a unique cluster ID. This is necessary to distinguish connections to multiple clustermesh-apiserver Pods using the same K8s Service. - Use session affinity for the clustermesh-apiserver Service Session affinity ensures that connections from a client are passed to the same service backend each time. This will allow a Cilium Agent or KVStoreMesh instance to maintain a connection to the same backend for both long-living, streaming connections, such as watches on the kv store, and short, single-response connections, such as checking the status of a cluster. However, this can be unreliable if the l3/l4 loadbalancer used does not also implement sticky sessions to direct connections from a particular client to the same cluster node. Signed-off-by: Tim Horner <timothy.horner@isovalent.com> Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
[ upstream commit 174e721 ] [ backporter's notes: backported a stripped down version of the upstream commit including the introduction of the interceptors only, as fixing a bug occurring in a single clustermesh-apiserver configuration as well (during rollouts), by preventing Cilium agents from incorrectly restarting an etcd watch against a different clustermesh-apiserver instance. ] In a configuration where there are mutliple replicas of the clustermesh-apiserver, each Pod runs its own etcd instance with a unique cluster ID. This commit adds a `clusterLock` type, which is a wrapper around a uint64 that can only be set once. `clusterLock` is used to create gRPC unary and stream interceptors that are provided to the etcd client to intercept and validate the cluster ID in the header of all responses from the etcd server. If the client receives a response from a different cluster, the connection is terminated and restarted. This is designed to prevent accepting responses from another cluster and potentially missing events or retaining invalid data. Since the addition of the interceptors allows quick detection of a failover event, we no longer need to rely on endpoint status checks to determine if the connection is healthy. Additionally, since service session affinity can be unreliable, the status checks could trigger a false failover event and cause a connection restart. To allow creating etcd clients for ClusterMesh that do not perform endpoint status checks, the option NoEndpointStatusChecks was added to ExtraOptions. Signed-off-by: Tim Horner <timothy.horner@isovalent.com> Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
giorio94
added
kind/bug
maintainer-s-little-helper
bot
added
backport/1.15
|
/test-backport-1.15 |
aanm
approved these changes
Apr 16, 2024
aanm
added
the
release-note/bug
|
@giorio94 can we remove the section after "Once this PR is merged, a GitHub action will update the labels of these PRs:" and change the labels manually? Otherwise the changelog tool will not pick the release note from this PR. |
ccfish2
reviewed
Apr 16, 2024
julianwiedmann
added
the
ready-to-merge
julianwiedmann
changed the title
maintainer-s-little-helper
bot
removed
the
ready-to-merge
github-merge-queue bot
pushed a commit
to microsoft/retina
that referenced
this pull request
May 15, 2024
Bumps [github.com/cilium/cilium](https://github.com/cilium/cilium) from 1.15.4 to 1.15.5. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/cilium/cilium/blob/1.15.5/CHANGELOG.md">github.com/cilium/cilium's changelog</a>.</em></p> <blockquote> <h2>v1.15.5</h2> <h2>Summary of Changes</h2> <p><strong>Minor Changes:</strong></p> <ul> <li><code>cilium/cilium#32413</code><a href="https://github.com/sayboras"><code>@sayboras</code></a>)</li> <li>labels: Add controller-uid into default ignore list (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31964">#31964</a>, <a href="https://github.com/sayboras"><code>@sayboras</code></a>)</li> </ul> <p><strong>Bugfixes:</strong></p> <ul> <li>Agent: add kubeconfigPath to initContainers (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32008">#32008</a>, <a href="https://github.com/darox"><code>@darox</code></a>)</li> <li>Avoids drops with "No mapping for NAT masquerade" for ICMP messages by local service backends. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32155">#32155</a>, <a href="https://github.com/julianwiedmann"><code>@julianwiedmann</code></a>)</li> <li>cilium-cni: Reserve ports that can conflict with transparent DNS proxy (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32418">#32418</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32128">#32128</a>, <a href="https://github.com/gandro"><code>@gandro</code></a>)</li> <li>cni: Use correct route MTU when ENI, Azure or Alibaba Cloud IPAM is enabled (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32244">#32244</a>, <a href="https://github.com/learnitall"><code>@learnitall</code></a>)</li> <li>dnsproxy: Fix bug where DNS request timed out too soon (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31999">#31999</a>, <a href="https://github.com/gandro"><code>@gandro</code></a>)</li> <li>Envoy upstream connections are now unique for each downstream connection when using the original source address of a source pod. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32312">#32312</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32270">#32270</a>, <a href="https://github.com/jrajahalme"><code>@jrajahalme</code></a>)</li> <li>envoy: pass idle timeout configuration option to cilium configmap (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32203">#32203</a>, <a href="https://github.com/mhofstetter"><code>@mhofstetter</code></a>)</li> <li>Fix failing service connections, when the service requests are transported via cilium's overlay network. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32116">#32116</a>, <a href="https://github.com/julianwiedmann"><code>@julianwiedmann</code></a>)</li> <li>Fix issue causing clustermesh-apiserver/kvstoremesh to not start when run with a non-root user (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/31879">#31879</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31539">#31539</a>, <a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>Fix service connection to terminating backend, when the service has no more backends available. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32092">#32092</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31840">#31840</a>, <a href="https://github.com/julianwiedmann"><code>@julianwiedmann</code></a>)</li> <li>Fix various bugs related to restart of StatefulSet pods that may result in connectivity issues (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32432">#32432</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31605">#31605</a>, <a href="https://github.com/christarazi"><code>@christarazi</code></a>)</li> <li>Fixes a bug where Cilium in chained mode removed the <code>agent-not-ready</code> taint too early if the primary network is slow in deploying. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32168">#32168</a>, <a href="https://github.com/squeed"><code>@squeed</code></a>)</li> <li>Fixes an (unlikely) bug where HostFirewall policies may miss updates to a node's labels. (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/30548">#30548</a>, <a href="https://github.com/squeed"><code>@squeed</code></a>)</li> <li>fqdn: fix memory leak in transparent mode when there was a moderately high number of parallel DNS requests (>100). (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31959">#31959</a>, <a href="https://github.com/marseel"><code>@marseel</code></a>)</li> <li>Ingress/Gateway API: merge Envoy listeners for HTTP(S) and TLS passthrough (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32178">#32178</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31646">#31646</a>, <a href="https://github.com/mhofstetter"><code>@mhofstetter</code></a>)</li> <li>ipam: retry netlink.LinkList call when setting up ENI devices (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32099">#32099</a>, <a href="https://github.com/jasonaliyetti"><code>@jasonaliyetti</code></a>)</li> <li>loader: sanitize bpffs directory strings for netdevs (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32090">#32090</a>, <a href="https://github.com/rgo3"><code>@rgo3</code></a>)</li> <li><code>cilium/cilium#32005</code><a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>tables: Sort node addresses also by public vs private IP (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/30579">#30579</a>, <a href="https://github.com/joamaki"><code>@joamaki</code></a>)</li> </ul> <p><strong>CI Changes:</strong></p> <ul> <li>alibabacloud/eni: avoid racing node mgr in test (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/31967">#31967</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31877">#31877</a>, <a href="https://github.com/bimmlerd"><code>@bimmlerd</code></a>)</li> <li>ci: Filter supported versions of AKS (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32303">#32303</a>, <a href="https://github.com/marseel"><code>@marseel</code></a>)</li> <li>ci: Increase timeout for images for l4lb test (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32201">#32201</a>, <a href="https://github.com/marseel"><code>@marseel</code></a>)</li> <li>ci: Set hubble.relay.retryTimeout=5s (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32066">#32066</a>, <a href="https://github.com/chancez"><code>@chancez</code></a>)</li> <li>enable kube cache mutation detector (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32069">#32069</a>, <a href="https://github.com/aanm"><code>@aanm</code></a>)</li> <li>gha: bump post-upgrade timeout in clustermesh upgrade/downgrade tests (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32347">#32347</a>, <a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>gha: configure fully-qualified DNS names as external targets (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31510">#31510</a>, <a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>gha: drop double installation of Cilium CLI in conformance-eks (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32042">#32042</a>, <a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>Miscellaneous improvements to the clustermesh upgrade/downgrade test (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/31958">#31958</a>, <a href="https://github.com/giorio94"><code>@giorio94</code></a>)</li> <li>route: dedicated net ns for each subtest of runListRules (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/29916">#29916</a>, <a href="https://github.com/mhofstetter"><code>@mhofstetter</code></a>)</li> <li>test: De-flake xds server_e2e_test (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32103">#32103</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32004">#32004</a>, <a href="https://github.com/jrajahalme"><code>@jrajahalme</code></a>)</li> <li>workflows: Fix CI jobs for push events on private forks (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32085">#32085</a>, <a href="https://github.com/pchaigno"><code>@pchaigno</code></a>)</li> </ul> <p><strong>Misc Changes:</strong></p> <ul> <li>bpf: host: simplify MARK_MAGIC_PROXY_EGRESS_EPID handling (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32384">#32384</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/29803">#29803</a>, <a href="https://github.com/julianwiedmann"><code>@julianwiedmann</code></a>)</li> <li>build(deps): bump pydantic from 2.3.0 to 2.4.0 in /Documentation (Backport PR <a href="https://redirect.github.com/cilium/cilium/issues/32230">#32230</a>, Upstream PR <a href="https://redirect.github.com/cilium/cilium/issues/32176">#32176</a>, <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot])</li> <li><code>cilium/cilium#31954</code><a href="https://github.com/renovate"><code>@renovate</code></a>[bot])</li> <li><code>cilium/cilium#32107</code><a href="https://github.com/renovate"><code>@renovate</code></a>[bot])</li> <li><code>cilium/cilium#32366</code><a href="https://github.com/renovate"><code>@renovate</code></a>[bot])</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/cilium/cilium/commit/8c7e442ccd48b9011a10f34a128ec98751d9a80e"><code>8c7e442</code></a> Prepare for release v1.15.5</li> <li><a href="https://github.com/cilium/cilium/commit/05e5d8ff6f37aa2d854f6a3565310428081c523b"><code>05e5d8f</code></a> images: update cilium-{runtime,builder}</li> <li><a href="https://github.com/cilium/cilium/commit/67e80e3205cb4fff3028947d262b58cb62c533d2"><code>67e80e3</code></a> k8s/watchers: Fix pod IP modified check</li> <li><a href="https://github.com/cilium/cilium/commit/6cc13936114175122ac2d2000aec17583cc96546"><code>6cc1393</code></a> daemon,endpoint,cni: Pass pod UID through CNI ADD</li> <li><a href="https://github.com/cilium/cilium/commit/c5ae085561e01ed0614408a274f3a182ddcfaa5c"><code>c5ae085</code></a> watchers: Detect Pod UID changes</li> <li><a href="https://github.com/cilium/cilium/commit/6750bb330d22ae9eff7c780be518864d34aa7370"><code>6750bb3</code></a> daemon, endpoint: Consolidate K8s metadata into struct</li> <li><a href="https://github.com/cilium/cilium/commit/b09552599b61ab595b887bed3dbb3a3031b15a26"><code>b095525</code></a> k8s/watcher: Remove outdated comments about K8sEventHandover</li> <li><a href="https://github.com/cilium/cilium/commit/fa03bba448d1a5082c0587bd4c1507df2145f5d7"><code>fa03bba</code></a> cilium-cni: Reserve ports that can conflict with transparent DNS proxy</li> <li><a href="https://github.com/cilium/cilium/commit/a29a4ab80100a442289605d00a5e8aa5cf927f35"><code>a29a4ab</code></a> wireguard: Export ListenPort constant</li> <li><a href="https://github.com/cilium/cilium/commit/eb7b7448429c3fe852e899adaa8c719d9b2ef600"><code>eb7b744</code></a> wireguard: Remove unused constants</li> <li>Additional commits viewable in <a href="https://github.com/cilium/cilium/compare/1.15.4...1.15.5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evan Baker <rbtr@users.noreply.github.com>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a custom and stripped down backport of #31677, including only the configuration of a unique etcd Cluster ID and the introduction of the etcd interceptors to validate the etcd Cluster ID itself. The goal being to fix a bug occurring in case the agents incorrectly restart etcd watches against a different clustermesh-apiserver instance (e.g., following a clustermesh-apiserver rollout), without fully restarting the etcd connection.
In that case, the retrieved data would be incoherent in terms of revisions, as pulled out from a completely separate etcd instance, and would possibly cause connectivity disruption. While in most cases the watchers normally just hang as the expected revision is too high compared to that of the new instance, and the watchdog eventually restarts the connection from scratch, we have been recently witnessing cases in which they actually resumed, leading to flakiness (#30964 (comment)). The unique etcd cluster id, complemented by the interceptors, would prevent this from happening, triggering a restart of the etcd connection upon detecting a change of the cluster id.
Please note that the changes backported here are not sufficient to safely allow to increase the number of the clustermesh-apiserver replicas. That part has been left out on purpose as not qualifying as a bug fix, and hence not matching the backporting criteria.