v1.9: CI: K8sServicesTest Checks local redirect policy LRP connectivity: cilium pre-flight checks failed: JoinEP: Failed to load program: Failed to replace Qdisc for lxcXXXXXXXXXXXX: Link not found

Seen in v1.9 backports PR https://github.com/cilium/cilium/pull/16910.
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.19/213/testReport/junit/Suite-k8s-1/18/K8sServicesTest_Checks_local_redirect_policy_LRP_connectivity/
3047b711_K8sServicesTest_Checks_local_redirect_policy_LRP_connectivity.zip
Test: Suite-k8s-1.18.K8sServicesTest Checks local redirect policy LRP connectivity
Summary
The core failure seems to be that there are pods that are removed while Cilium is down, then Cilium attempts to restore the endpoint after restart and the restore fails because the link for the pod is no longer available. We have seen similar behaviour in the past and were able to mitigate it, so it's possible that something has changed in the endpoint store / restore logic that brought this back.
Here are some sample logs:
2021-07-19T04:44:27.009240146Z level=warning msg="JoinEP: Failed to load program" containerID=5360c3e7f9 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=160 error="Failed to replace Qdisc for lxc314ae629618e: Link not found" file-path=160_next/bpf_lxc.o identity=41050 ipv4=<IP_ADDRESS> ipv6="fd00::2c" k8sPodName=default/echo-748bf97b8f-7zrnj subsys=datapath-loader veth=lxc314ae629618e
2021-07-19T04:44:27.009267275Z level=error msg="Error while rewriting endpoint BPF program" containerID=5360c3e7f9 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=160 error="Failed to replace Qdisc for lxc314ae629618e: Link not found" identity=41050 ipv4=<IP_ADDRESS> ipv6="fd00::2c" k8sPodName=default/echo-748bf97b8f-7zrnj subsys=endpoint
2021-07-19T04:44:27.013782637Z level=warning msg="generating BPF for endpoint failed, keeping stale directory." containerID=5360c3e7f9 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=160 file-path=160_next_fail identity=41050 ipv4=<IP_ADDRESS> ipv6="fd00::2c" k8sPodName=default/echo-748bf97b8f-7zrnj subsys=endpoint

Given that we have seen similar behaviour on other tests in the past, it's possible that this is unrelated to the content of the LRP tests.
Stacktrace
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:461
cilium pre-flight checks failed
Expected
    <*errors.errorString | 0xc000713740>: {
        s: "Cilium validation failed: 4m0s timeout expired: Last polled error: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0 \nStdout:\n \t KVStore:                Ok   Disabled\n\t Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]\n\t Kubernetes APIs:        [\"cilium/v2::CiliumClusterwideNetworkPolicy\", \"cilium/v2::CiliumEndpoint\", \"cilium/v2::CiliumLocalRedirectPolicy\", \"cilium/v2::CiliumNetworkPolicy\", \"cilium/v2::CiliumNode\", \"core/v1::Namespace\", \"core/v1::Node\", \"core/v1::Pods\", \"core/v1::Service\", \"discovery/v1beta1::EndpointSlice\", \"networking.k8s.io/v1::NetworkPolicy\"]\n\t KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]\n\t Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)\n\t NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory\n\t Cilium health daemon:   Ok   \n\t IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120\n\t BandwidthManager:       Disabled\n\t Host Routing:           Legacy\n\t Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8\n\t Controller Status:      26/27 healthy\n\t   Name                                          Last success   Last error   Count   Message\n\t   cilium-health-ep                              29s ago        never        0       no error                       \n\t   dns-garbage-collector-job                     13s ago        never        0       no error                       \n\t   endpoint-160-regeneration-recovery            never          10s ago      16      regeneration recovery failed   \n\t   endpoint-2455-regeneration-recovery           never          never        0       no error                       \n\t   endpoint-3206-regeneration-recovery           never          never        0       no error                       \n\t   endpoint-3506-regeneration-recovery           never          never        0       no error                       \n\t   k8s-heartbeat                                 13s ago        never        0       no error                       \n\t   mark-k8s-node-as-available                    4m30s ago      never        0       no error                       \n\t   metricsmap-bpf-prom-sync                      3s ago         never        0       no error                       \n\t   neighbor-table-refresh                        4m30s ago      never        0       no error                       \n\t   resolve-identity-2455                         4m29s ago      never        0       no error                       \n\t   restoring-ep-identity (160)                   4m31s ago      never        0       no error                       \n\t   restoring-ep-identity (3206)                  4m31s ago      never        0       no error                       \n\t   restoring-ep-identity (3506)                  4m31s ago      never        0       no error                       \n\t   sync-endpoints-and-host-ips                   31s ago        never        0       no error                       \n\t   sync-lb-maps-with-k8s-services                4m31s ago      never        0       no error                       \n\t   sync-policymap-2455                           10s ago        never        0       no error                       \n\t   sync-policymap-3206                           1m0s ago       never        0       no error                       \n\t   sync-policymap-3506                           11s ago        never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (160)              10s ago        4m1s ago     0       no error                       \n\t   sync-to-k8s-ciliumendpoint (2455)             9s ago         never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (3206)             1s ago         never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (3506)             1s ago         never        0       no error                       \n\t   template-dir-watcher                          never          never        0       no error                       \n\t   update-k8s-node-annotations                   5m11s ago      never        0       no error                       \n\t   waiting-initial-global-identities-ep (160)    4m31s ago      never        0       no error                       \n\t   waiting-initial-global-identities-ep (3506)   4m31s ago      never        0       no error                       \n\t Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000\n\t Hubble:           Ok              Current/Max Flows: 1994/4096 (48.68%), Flows/s: 7.40   Metrics: Disabled\n\t Cluster health:   2/2 reachable   (2021-07-19T04:48:42Z)\n\t \nStderr:\n \t \n",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.18-kernel-4.19/src/github.com/cilium/cilium/test/k8sT/Services.go:466

Standard Output
Click to show
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 1
No errors/warnings found in logs
⚠  Found "JoinEP: " in logs 34 times
Number of "context deadline exceeded" in logs: 2
⚠  Number of "level=error" in logs: 70
⚠  Number of "level=warning" in logs: 112
Number of "Cilium API handler panicked" in logs: 0
⚠  Number of "Goroutine took lock for more than" in logs: 16
Top 5 errors/warnings:
Error while rewriting endpoint BPF program
endpoint regeneration failed
Cannot update CEP
Regeneration of endpoint failed
JoinEP: Failed to load program
Cilium pods: [cilium-m6s2p cilium-v2nvw]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod                        Ingress   Egress
coredns-7964865f77-pctpv
⚠  Cilium agent 'cilium-m6s2p': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 27 Failed 1
Failed controllers:
 controller endpoint-160-regeneration-recovery failure 'regeneration recovery failed'
⚠  Cilium agent 'cilium-v2nvw': Status: Ok  Health: Ok Nodes "" ContinerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 22 Failed 1
Failed controllers:
 controller endpoint-160-regeneration-recovery failure 'regeneration recovery failed'
controller endpoint-326-regeneration-recovery failure 'regeneration recovery failed'

Standard Error
04:43:12 STEP: Running BeforeAll block for EntireTestsuite K8sServicesTest Checks local redirect policy
04:43:12 STEP: Installing Cilium
04:43:22 STEP: Waiting for Cilium to become ready
04:43:22 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:23 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:25 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:26 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:27 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:29 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:30 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:31 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:32 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:34 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:35 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:36 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:38 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:39 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:40 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:41 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:43 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:44 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:45 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:47 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:48 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:49 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:51 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:52 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:53 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:54 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:56 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:57 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:58 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:43:59 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:01 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:02 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:03 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:05 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:06 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:07 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:08 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:10 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:11 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:12 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:14 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:15 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:16 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:18 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:19 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:20 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:22 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:23 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:24 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:26 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:27 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:28 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:29 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:31 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:32 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:33 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:35 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:36 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:37 STEP: Cilium DaemonSet not ready yet: only 0 of 2 desired pods are ready
04:44:38 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
04:44:40 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
04:44:41 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
04:44:42 STEP: Cilium DaemonSet not ready yet: only 1 of 2 desired pods are ready
04:44:44 STEP: Number of ready Cilium pods: 2
04:44:44 STEP: Validating if Kubernetes DNS is deployed
04:44:44 STEP: Checking if deployment is ready
04:44:45 STEP: Checking if kube-dns service is plumbed correctly
04:44:45 STEP: Checking if DNS can resolve
04:44:45 STEP: Checking if pods have identity
04:44:49 STEP: Kubernetes DNS is up and operational
04:44:49 STEP: Validating Cilium Installation
04:44:49 STEP: Performing Cilium status preflight check
04:44:49 STEP: Performing Cilium health check
04:44:49 STEP: Performing Cilium controllers preflight check
04:44:57 STEP: Performing Cilium service preflight check
04:44:57 STEP: Performing K8s service preflight check
04:44:57 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              36s ago        never        0       no error
	   dns-garbage-collector-job                     20s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          6s ago       4       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 20s ago        never        0       no error
	   mark-k8s-node-as-available                    37s ago        never        0       no error
	   metricsmap-bpf-prom-sync                      5s ago         never        0       no error
	   neighbor-table-refresh                        37s ago        never        0       no error
	   resolve-identity-2455                         36s ago        never        0       no error
	   restoring-ep-identity (160)                   38s ago        never        0       no error
	   restoring-ep-identity (3206)                  38s ago        never        0       no error
	   restoring-ep-identity (3506)                  38s ago        never        0       no error
	   sync-endpoints-and-host-ips                   38s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                38s ago        never        0       no error
	   sync-policymap-2455                           18s ago        never        0       no error
	   sync-policymap-3206                           7s ago         never        0       no error
	   sync-policymap-3506                           18s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              7s ago         8s ago       0       no error
	   sync-to-k8s-ciliumendpoint (2455)             6s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             8s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             8s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m19s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    38s ago        never        0       no error
	   waiting-initial-global-identities-ep (3506)   38s ago        never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 168/4096 (4.10%), Flows/s: 4.91   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:44:51Z)

Stderr:

04:44:57 STEP: Performing Cilium status preflight check
04:44:57 STEP: Performing Cilium controllers preflight check
04:44:57 STEP: Performing Cilium health check
04:45:04 STEP: Performing Cilium service preflight check
04:45:04 STEP: Performing K8s service preflight check
04:45:04 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              44s ago        never        0       no error
	   dns-garbage-collector-job                     28s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          5s ago       5       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 28s ago        never        0       no error
	   mark-k8s-node-as-available                    45s ago        never        0       no error
	   metricsmap-bpf-prom-sync                      3s ago         never        0       no error
	   neighbor-table-refresh                        45s ago        never        0       no error
	   resolve-identity-2455                         44s ago        never        0       no error
	   restoring-ep-identity (160)                   46s ago        never        0       no error
	   restoring-ep-identity (3206)                  46s ago        never        0       no error
	   restoring-ep-identity (3506)                  46s ago        never        0       no error
	   sync-endpoints-and-host-ips                   46s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                46s ago        never        0       no error
	   sync-policymap-2455                           25s ago        never        0       no error
	   sync-policymap-3206                           15s ago        never        0       no error
	   sync-policymap-3506                           26s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              5s ago         16s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             6s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             6s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m26s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    46s ago        never        0       no error
	   waiting-initial-global-identities-ep (3506)   46s ago        never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 217/4096 (5.30%), Flows/s: 4.91   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:44:59Z)

Stderr:

04:45:04 STEP: Performing Cilium controllers preflight check
04:45:04 STEP: Performing Cilium status preflight check
04:45:04 STEP: Performing Cilium health check
04:45:11 STEP: Performing Cilium service preflight check
04:45:11 STEP: Performing K8s service preflight check
04:45:11 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              52s ago        never        0       no error
	   dns-garbage-collector-job                     35s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          13s ago      5       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 6s ago         never        0       no error
	   mark-k8s-node-as-available                    52s ago        never        0       no error
	   metricsmap-bpf-prom-sync                      5s ago         never        0       no error
	   neighbor-table-refresh                        52s ago        never        0       no error
	   resolve-identity-2455                         52s ago        never        0       no error
	   restoring-ep-identity (160)                   53s ago        never        0       no error
	   restoring-ep-identity (3206)                  53s ago        never        0       no error
	   restoring-ep-identity (3506)                  53s ago        never        0       no error
	   sync-endpoints-and-host-ips                   54s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                53s ago        never        0       no error
	   sync-policymap-2455                           33s ago        never        0       no error
	   sync-policymap-3206                           23s ago        never        0       no error
	   sync-policymap-3506                           33s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              12s ago        23s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             12s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             3s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             3s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m34s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    53s ago        never        0       no error
	   waiting-initial-global-identities-ep (3506)   53s ago        never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 243/4096 (5.93%), Flows/s: 4.94   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:07Z)

Stderr:

04:45:11 STEP: Performing Cilium status preflight check
04:45:11 STEP: Performing Cilium health check
04:45:11 STEP: Performing Cilium controllers preflight check
04:45:19 STEP: Performing Cilium service preflight check
04:45:19 STEP: Performing K8s service preflight check
04:45:19 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              1m0s ago       never        0       no error
	   dns-garbage-collector-job                     43s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          11s ago      6       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 13s ago        never        0       no error
	   mark-k8s-node-as-available                    1m0s ago       never        0       no error
	   metricsmap-bpf-prom-sync                      3s ago         never        0       no error
	   neighbor-table-refresh                        1m0s ago       never        0       no error
	   resolve-identity-2455                         1m0s ago       never        0       no error
	   restoring-ep-identity (160)                   1m1s ago       never        0       no error
	   restoring-ep-identity (3206)                  1m1s ago       never        0       no error
	   restoring-ep-identity (3506)                  1m1s ago       never        0       no error
	   sync-endpoints-and-host-ips                   1s ago         never        0       no error
	   sync-lb-maps-with-k8s-services                1m1s ago       never        0       no error
	   sync-policymap-2455                           41s ago        never        0       no error
	   sync-policymap-3206                           31s ago        never        0       no error
	   sync-policymap-3506                           41s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              10s ago        31s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             10s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             1s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             1s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m42s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    1m1s ago       never        0       no error
	   waiting-initial-global-identities-ep (3506)   1m1s ago       never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 293/4096 (7.15%), Flows/s: 4.95   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:14Z)

Stderr:

04:45:19 STEP: Performing Cilium controllers preflight check
04:45:19 STEP: Performing Cilium status preflight check
04:45:19 STEP: Performing Cilium health check
04:45:27 STEP: Performing Cilium service preflight check
04:45:27 STEP: Performing K8s service preflight check
04:45:27 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              7s ago         never        0       no error
	   dns-garbage-collector-job                     51s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          6s ago       7       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 21s ago        never        0       no error
	   mark-k8s-node-as-available                    1m7s ago       never        0       no error
	   metricsmap-bpf-prom-sync                      6s ago         never        0       no error
	   neighbor-table-refresh                        1m7s ago       never        0       no error
	   resolve-identity-2455                         1m7s ago       never        0       no error
	   restoring-ep-identity (160)                   1m8s ago       never        0       no error
	   restoring-ep-identity (3206)                  1m8s ago       never        0       no error
	   restoring-ep-identity (3506)                  1m8s ago       never        0       no error
	   sync-endpoints-and-host-ips                   9s ago         never        0       no error
	   sync-lb-maps-with-k8s-services                1m8s ago       never        0       no error
	   sync-policymap-2455                           48s ago        never        0       no error
	   sync-policymap-3206                           38s ago        never        0       no error
	   sync-policymap-3506                           48s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              7s ago         38s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             7s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             8s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             8s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m49s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    1m8s ago       never        0       no error
	   waiting-initial-global-identities-ep (3506)   1m8s ago       never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 319/4096 (7.79%), Flows/s: 4.97   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:22Z)

Stderr:

04:45:27 STEP: Performing Cilium controllers preflight check
04:45:27 STEP: Performing Cilium health check
04:45:27 STEP: Performing Cilium status preflight check
04:45:30 STEP: Performing Cilium service preflight check
04:45:30 STEP: Performing K8s service preflight check
04:45:31 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              13s ago        never        0       no error
	   dns-garbage-collector-job                     57s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          12s ago      7       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 27s ago        never        0       no error
	   mark-k8s-node-as-available                    1m14s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      2s ago         never        0       no error
	   neighbor-table-refresh                        1m14s ago      never        0       no error
	   resolve-identity-2455                         1m13s ago      never        0       no error
	   restoring-ep-identity (160)                   1m15s ago      never        0       no error
	   restoring-ep-identity (3206)                  1m15s ago      never        0       no error
	   restoring-ep-identity (3506)                  1m15s ago      never        0       no error
	   sync-endpoints-and-host-ips                   15s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                1m15s ago      never        0       no error
	   sync-policymap-2455                           54s ago        never        0       no error
	   sync-policymap-3206                           44s ago        never        0       no error
	   sync-policymap-3506                           55s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              4s ago         45s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             3s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             5s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             5s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m55s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    1m15s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   1m15s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 368/4096 (8.98%), Flows/s: 4.96   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:22Z)

Stderr:

04:45:31 STEP: Performing Cilium controllers preflight check
04:45:31 STEP: Performing Cilium health check
04:45:31 STEP: Performing Cilium status preflight check
04:45:32 STEP: Performing Cilium service preflight check
04:45:32 STEP: Performing K8s service preflight check
04:45:33 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              16s ago        never        0       no error
	   dns-garbage-collector-job                     1m0s ago       never        0       no error
	   endpoint-160-regeneration-recovery            never          15s ago      7       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 30s ago        never        0       no error
	   mark-k8s-node-as-available                    1m17s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      5s ago         never        0       no error
	   neighbor-table-refresh                        1m17s ago      never        0       no error
	   resolve-identity-2455                         1m16s ago      never        0       no error
	   restoring-ep-identity (160)                   1m18s ago      never        0       no error
	   restoring-ep-identity (3206)                  1m18s ago      never        0       no error
	   restoring-ep-identity (3506)                  1m18s ago      never        0       no error
	   sync-endpoints-and-host-ips                   18s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                1m18s ago      never        0       no error
	   sync-policymap-2455                           57s ago        never        0       no error
	   sync-policymap-3206                           47s ago        never        0       no error
	   sync-policymap-3506                           57s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              6s ago         47s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             6s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             8s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             7s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   1m58s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    1m18s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   1m18s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 368/4096 (8.98%), Flows/s: 4.96   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:30Z)

Stderr:

04:45:33 STEP: Performing Cilium controllers preflight check
04:45:33 STEP: Performing Cilium status preflight check
04:45:33 STEP: Performing Cilium health check
04:45:36 STEP: Performing Cilium service preflight check
04:45:36 STEP: Performing K8s service preflight check
04:45:37 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              19s ago        never        0       no error
	   dns-garbage-collector-job                     2s ago         never        0       no error
	   endpoint-160-regeneration-recovery            never          4s ago       8       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 2s ago         never        0       no error
	   mark-k8s-node-as-available                    1m19s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      2s ago         never        0       no error
	   neighbor-table-refresh                        1m19s ago      never        0       no error
	   resolve-identity-2455                         1m19s ago      never        0       no error
	   restoring-ep-identity (160)                   1m20s ago      never        0       no error
	   restoring-ep-identity (3206)                  1m20s ago      never        0       no error
	   restoring-ep-identity (3506)                  1m20s ago      never        0       no error
	   sync-endpoints-and-host-ips                   20s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                1m20s ago      never        0       no error
	   sync-policymap-2455                           1m0s ago       never        0       no error
	   sync-policymap-3206                           50s ago        never        0       no error
	   sync-policymap-3506                           0s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              9s ago         50s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             9s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             0s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             0s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   2m1s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    1m20s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   1m20s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 409/4096 (9.99%), Flows/s: 5.16   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:45:32Z)

Stderr:

04:45:37 STEP: Performing Cilium controllers preflight check
04:45:37 STEP: Performing Cilium health check
04:45:37 STEP: Performing Cilium status preflight check
04:45:40 STEP: Performing Cilium service preflight check
04:45:40 STEP: Performing K8s service preflight check
04:45:41 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              22s ago        never        0       no error
	   dns-garbage-collector-job                     6s ago         never        0       no error
	   endpoint-160-regeneration-recovery            never          7s ago       8       regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 6s ago         never        0       no error
	   mark-k8s-node-as-available                    1m23s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      6s ago         never        0       no error
	   neighbor-table-refresh                        1m23s ago      never        0       no error
	   resolve-identity-2455                         1m22s ago      never        0       no error
	   restoring-ep-identity (160)                   1m24s ago      never        0       no error
	   restoring-ep-identity (3206)                  1m24s ago      never        0       no error
	   restoring-ep-identity (3506)                  1m24s ago      never        0       no error
	   sync-endpoints-and-host-ips                   24s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                1m24s ago      never        0       no error
	   sync-policymap-2455                           1m3s ago       never        0       no error
	   sync-policymap-3206                           53s ago        never        0       no error
	   sync-policymap-3506                           4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              13s ago        54s ago      0       no error
	   sync-to-k8s-ciliumendpoint (2455)             12s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             4s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   2m4s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    1m24s ago      never        0       no error
...[truncated 285429 chars]...
        0       no error
	   mark-k8s-node-as-available                    4m12s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      5s ago         never        0       no error
	   neighbor-table-refresh                        4m12s ago      never        0       no error
	   resolve-identity-2455                         4m11s ago      never        0       no error
	   restoring-ep-identity (160)                   4m13s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m13s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m13s ago      never        0       no error
	   sync-endpoints-and-host-ips                   13s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m13s ago      never        0       no error
	   sync-policymap-2455                           53s ago        never        0       no error
	   sync-policymap-3206                           43s ago        never        0       no error
	   sync-policymap-3506                           53s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              12s ago        3m43s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             11s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             3s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             3s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   4m54s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    4m13s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m13s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1832/4096 (44.73%), Flows/s: 7.35   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:24Z)

Stderr:

04:48:29 STEP: Performing Cilium controllers preflight check
04:48:29 STEP: Performing Cilium health check
04:48:29 STEP: Performing Cilium status preflight check
04:48:31 STEP: Performing Cilium service preflight check
04:48:31 STEP: Performing K8s service preflight check
04:48:32 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              15s ago        never        0       no error
	   dns-garbage-collector-job                     58s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          26s ago      15      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 28s ago        never        0       no error
	   mark-k8s-node-as-available                    4m15s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      3s ago         never        0       no error
	   neighbor-table-refresh                        4m15s ago      never        0       no error
	   resolve-identity-2455                         4m15s ago      never        0       no error
	   restoring-ep-identity (160)                   4m16s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m16s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m16s ago      never        0       no error
	   sync-endpoints-and-host-ips                   16s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m16s ago      never        0       no error
	   sync-policymap-2455                           56s ago        never        0       no error
	   sync-policymap-3206                           46s ago        never        0       no error
	   sync-policymap-3506                           56s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              5s ago         3m46s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             5s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             6s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             6s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   4m57s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    4m16s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m16s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1868/4096 (45.61%), Flows/s: 7.35   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:28Z)

Stderr:

04:48:32 STEP: Performing Cilium controllers preflight check
04:48:32 STEP: Performing Cilium status preflight check
04:48:32 STEP: Performing Cilium health check
04:48:34 STEP: Performing Cilium service preflight check
04:48:34 STEP: Performing K8s service preflight check
04:48:35 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              18s ago        never        0       no error
	   dns-garbage-collector-job                     1m1s ago       never        0       no error
	   endpoint-160-regeneration-recovery            never          29s ago      15      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 31s ago        never        0       no error
	   mark-k8s-node-as-available                    4m18s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      6s ago         never        0       no error
	   neighbor-table-refresh                        4m18s ago      never        0       no error
	   resolve-identity-2455                         4m18s ago      never        0       no error
	   restoring-ep-identity (160)                   4m19s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m19s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m19s ago      never        0       no error
	   sync-endpoints-and-host-ips                   19s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m19s ago      never        0       no error
	   sync-policymap-2455                           59s ago        never        0       no error
	   sync-policymap-3206                           49s ago        never        0       no error
	   sync-policymap-3506                           59s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              8s ago         3m49s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             8s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             9s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             9s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m0s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    4m19s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m19s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1868/4096 (45.61%), Flows/s: 7.35   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:31Z)

Stderr:

04:48:35 STEP: Performing Cilium controllers preflight check
04:48:35 STEP: Performing Cilium health check
04:48:35 STEP: Performing Cilium status preflight check
04:48:37 STEP: Performing Cilium service preflight check
04:48:37 STEP: Performing K8s service preflight check
04:48:38 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              20s ago        never        0       no error
	   dns-garbage-collector-job                     4s ago         never        0       no error
	   endpoint-160-regeneration-recovery            never          31s ago      15      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 4s ago         never        0       no error
	   mark-k8s-node-as-available                    4m21s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      4s ago         never        0       no error
	   neighbor-table-refresh                        4m21s ago      never        0       no error
	   resolve-identity-2455                         4m20s ago      never        0       no error
	   restoring-ep-identity (160)                   4m22s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m22s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m22s ago      never        0       no error
	   sync-endpoints-and-host-ips                   22s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m22s ago      never        0       no error
	   sync-policymap-2455                           1m1s ago       never        0       no error
	   sync-policymap-3206                           51s ago        never        0       no error
	   sync-policymap-3506                           2s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              11s ago        3m52s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             10s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             2s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             2s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m2s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    4m22s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m22s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1907/4096 (46.56%), Flows/s: 7.35   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:34Z)

Stderr:

04:48:38 STEP: Performing Cilium controllers preflight check
04:48:38 STEP: Performing Cilium health check
04:48:38 STEP: Performing Cilium status preflight check
04:48:39 STEP: Performing Cilium service preflight check
04:48:39 STEP: Performing K8s service preflight check
04:48:41 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              23s ago        never        0       no error
	   dns-garbage-collector-job                     6s ago         never        0       no error
	   endpoint-160-regeneration-recovery            never          34s ago      15      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 7s ago         never        0       no error
	   mark-k8s-node-as-available                    4m23s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      6s ago         never        0       no error
	   neighbor-table-refresh                        4m23s ago      never        0       no error
	   resolve-identity-2455                         4m23s ago      never        0       no error
	   restoring-ep-identity (160)                   4m24s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m24s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m24s ago      never        0       no error
	   sync-endpoints-and-host-ips                   25s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m24s ago      never        0       no error
	   sync-policymap-2455                           1m4s ago       never        0       no error
	   sync-policymap-3206                           54s ago        never        0       no error
	   sync-policymap-3506                           4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              13s ago        3m54s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             13s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             4s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m5s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    4m24s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m24s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1907/4096 (46.56%), Flows/s: 7.35   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:37Z)

Stderr:

04:48:41 STEP: Performing Cilium controllers preflight check
04:48:41 STEP: Performing Cilium health check
04:48:41 STEP: Performing Cilium status preflight check
04:48:43 STEP: Performing Cilium service preflight check
04:48:43 STEP: Performing K8s service preflight check
04:48:44 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              26s ago        never        0       no error
	   dns-garbage-collector-job                     9s ago         never        0       no error
	   endpoint-160-regeneration-recovery            never          7s ago       16      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 9s ago         never        0       no error
	   mark-k8s-node-as-available                    4m26s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      4s ago         never        0       no error
	   neighbor-table-refresh                        4m26s ago      never        0       no error
	   resolve-identity-2455                         4m26s ago      never        0       no error
	   restoring-ep-identity (160)                   4m27s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m27s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m27s ago      never        0       no error
	   sync-endpoints-and-host-ips                   27s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m27s ago      never        0       no error
	   sync-policymap-2455                           7s ago         never        0       no error
	   sync-policymap-3206                           57s ago        never        0       no error
	   sync-policymap-3506                           7s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              6s ago         3m57s ago    0       no error
	   sync-to-k8s-ciliumendpoint (2455)             6s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             7s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             7s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m8s ago       never        0       no error
	   waiting-initial-global-identities-ep (160)    4m27s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m27s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1948/4096 (47.56%), Flows/s: 7.37   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:39Z)

Stderr:

04:48:44 STEP: Performing Cilium controllers preflight check
04:48:44 STEP: Performing Cilium status preflight check
04:48:44 STEP: Performing Cilium health check
04:48:46 STEP: Performing Cilium service preflight check
04:48:46 STEP: Performing K8s service preflight check
04:48:47 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              29s ago        never        0       no error
	   dns-garbage-collector-job                     13s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          10s ago      16      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 13s ago        never        0       no error
	   mark-k8s-node-as-available                    4m30s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      3s ago         never        0       no error
	   neighbor-table-refresh                        4m30s ago      never        0       no error
	   resolve-identity-2455                         4m29s ago      never        0       no error
	   restoring-ep-identity (160)                   4m31s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m31s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m31s ago      never        0       no error
	   sync-endpoints-and-host-ips                   31s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m31s ago      never        0       no error
	   sync-policymap-2455                           10s ago        never        0       no error
	   sync-policymap-3206                           1m0s ago       never        0       no error
	   sync-policymap-3506                           11s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              10s ago        4m1s ago     0       no error
	   sync-to-k8s-ciliumendpoint (2455)             9s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             1s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             1s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m11s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    4m31s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m31s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1994/4096 (48.68%), Flows/s: 7.40   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:42Z)

Stderr:

04:48:47 STEP: Performing Cilium controllers preflight check
04:48:47 STEP: Performing Cilium status preflight check
04:48:47 STEP: Performing Cilium health check
04:48:48 STEP: Performing Cilium service preflight check
04:48:48 STEP: Performing K8s service preflight check
FAIL: cilium pre-flight checks failed
Expected
    <*errors.errorString | 0xc000713740>: {
        s: "Cilium validation failed: 4m0s timeout expired: Last polled error: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0 \nStdout:\n \t KVStore:                Ok   Disabled\n\t Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]\n\t Kubernetes APIs:        [\"cilium/v2::CiliumClusterwideNetworkPolicy\", \"cilium/v2::CiliumEndpoint\", \"cilium/v2::CiliumLocalRedirectPolicy\", \"cilium/v2::CiliumNetworkPolicy\", \"cilium/v2::CiliumNode\", \"core/v1::Namespace\", \"core/v1::Node\", \"core/v1::Pods\", \"core/v1::Service\", \"discovery/v1beta1::EndpointSlice\", \"networking.k8s.io/v1::NetworkPolicy\"]\n\t KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]\n\t Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)\n\t NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory\n\t Cilium health daemon:   Ok   \n\t IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120\n\t BandwidthManager:       Disabled\n\t Host Routing:           Legacy\n\t Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8\n\t Controller Status:      26/27 healthy\n\t   Name                                          Last success   Last error   Count   Message\n\t   cilium-health-ep                              29s ago        never        0       no error                       \n\t   dns-garbage-collector-job                     13s ago        never        0       no error                       \n\t   endpoint-160-regeneration-recovery            never          10s ago      16      regeneration recovery failed   \n\t   endpoint-2455-regeneration-recovery           never          never        0       no error                       \n\t   endpoint-3206-regeneration-recovery           never          never        0       no error                       \n\t   endpoint-3506-regeneration-recovery           never          never        0       no error                       \n\t   k8s-heartbeat                                 13s ago        never        0       no error                       \n\t   mark-k8s-node-as-available                    4m30s ago      never        0       no error                       \n\t   metricsmap-bpf-prom-sync                      3s ago         never        0       no error                       \n\t   neighbor-table-refresh                        4m30s ago      never        0       no error                       \n\t   resolve-identity-2455                         4m29s ago      never        0       no error                       \n\t   restoring-ep-identity (160)                   4m31s ago      never        0       no error                       \n\t   restoring-ep-identity (3206)                  4m31s ago      never        0       no error                       \n\t   restoring-ep-identity (3506)                  4m31s ago      never        0       no error                       \n\t   sync-endpoints-and-host-ips                   31s ago        never        0       no error                       \n\t   sync-lb-maps-with-k8s-services                4m31s ago      never        0       no error                       \n\t   sync-policymap-2455                           10s ago        never        0       no error                       \n\t   sync-policymap-3206                           1m0s ago       never        0       no error                       \n\t   sync-policymap-3506                           11s ago        never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (160)              10s ago        4m1s ago     0       no error                       \n\t   sync-to-k8s-ciliumendpoint (2455)             9s ago         never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (3206)             1s ago         never        0       no error                       \n\t   sync-to-k8s-ciliumendpoint (3506)             1s ago         never        0       no error                       \n\t   template-dir-watcher                          never          never        0       no error                       \n\t   update-k8s-node-annotations                   5m11s ago      never        0       no error                       \n\t   waiting-initial-global-identities-ep (160)    4m31s ago      never        0       no error                       \n\t   waiting-initial-global-identities-ep (3506)   4m31s ago      never        0       no error                       \n\t Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000\n\t Hubble:           Ok              Current/Max Flows: 1994/4096 (48.68%), Flows/s: 7.40   Metrics: Disabled\n\t Cluster health:   2/2 reachable   (2021-07-19T04:48:42Z)\n\t \nStderr:\n \t \n",
    }
to be nil
04:48:49 STEP: Running JustAfterEach block for EntireTestsuite K8sServicesTest
04:48:50 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-m6s2p': controller endpoint-160-regeneration-recovery is failing: Exitcode: 0
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.18 (v1.18.20) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Strict   [enp0s8 (Direct Routing), enp0s3]
	 Cilium:                 Ok   1.9.8 (v1.9.8-770f20068)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 Cilium health daemon:   Ok
	 IPAM:                   IPv4: 4/255 allocated from <IP_ADDRESS>/24, IPv6: 4/255 allocated from fd00::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           BPF   [enp0s8, enp0s3]   <IP_ADDRESS>/8
	 Controller Status:      26/27 healthy
	   Name                                          Last success   Last error   Count   Message
	   cilium-health-ep                              32s ago        never        0       no error
	   dns-garbage-collector-job                     16s ago        never        0       no error
	   endpoint-160-regeneration-recovery            never          13s ago      16      regeneration recovery failed
	   endpoint-2455-regeneration-recovery           never          never        0       no error
	   endpoint-3206-regeneration-recovery           never          never        0       no error
	   endpoint-3506-regeneration-recovery           never          never        0       no error
	   k8s-heartbeat                                 16s ago        never        0       no error
	   mark-k8s-node-as-available                    4m33s ago      never        0       no error
	   metricsmap-bpf-prom-sync                      6s ago         never        0       no error
	   neighbor-table-refresh                        4m33s ago      never        0       no error
	   resolve-identity-2455                         4m32s ago      never        0       no error
	   restoring-ep-identity (160)                   4m34s ago      never        0       no error
	   restoring-ep-identity (3206)                  4m34s ago      never        0       no error
	   restoring-ep-identity (3506)                  4m34s ago      never        0       no error
	   sync-endpoints-and-host-ips                   34s ago        never        0       no error
	   sync-lb-maps-with-k8s-services                4m34s ago      never        0       no error
	   sync-policymap-2455                           13s ago        never        0       no error
	   sync-policymap-3206                           1m3s ago       never        0       no error
	   sync-policymap-3506                           13s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (160)              12s ago        4m3s ago     0       no error
	   sync-to-k8s-ciliumendpoint (2455)             12s ago        never        0       no error
	   sync-to-k8s-ciliumendpoint (3206)             4s ago         never        0       no error
	   sync-to-k8s-ciliumendpoint (3506)             3s ago         never        0       no error
	   template-dir-watcher                          never          never        0       no error
	   update-k8s-node-annotations                   5m14s ago      never        0       no error
	   waiting-initial-global-identities-ep (160)    4m34s ago      never        0       no error
	   waiting-initial-global-identities-ep (3506)   4m34s ago      never        0       no error
	 Proxy Status:     OK, ip <IP_ADDRESS>, 0 redirects active on ports 10000-20000
	 Hubble:           Ok              Current/Max Flows: 1994/4096 (48.68%), Flows/s: 7.40   Metrics: Disabled
	 Cluster health:   2/2 reachable   (2021-07-19T04:48:46Z)

Stderr:

FAIL: Found 1 k8s-app=cilium logs matching list of errors that must be investigated:
JoinEP:
===================== TEST FAILED =====================
04:48:50 STEP: Running AfterFailed block for EntireTestsuite K8sServicesTest
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
 	 NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-54dbdc987-ltbr7            0/1     Running   0          53m     <IP_ADDRESS>      k8s1   <none>           <none>
	 cilium-monitoring   prometheus-6ff848df8b-jjsbc        1/1     Running   0          53m     <IP_ADDRESS>      k8s1   <none>           <none>
	 kube-system         cilium-m6s2p                       1/1     Running   0          5m32s   <IP_ADDRESS>   k8s2   <none>           <none>
	 kube-system         cilium-operator-6bbb776b6d-npb5f   1/1     Running   0          5m31s   <IP_ADDRESS>   k8s2   <none>           <none>
	 kube-system         cilium-operator-6bbb776b6d-tqs5j   1/1     Running   0          5m31s   <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         cilium-v2nvw                       1/1     Running   0          5m32s   <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         coredns-7964865f77-pctpv           1/1     Running   0          8m32s   <IP_ADDRESS>      k8s2   <none>           <none>
	 kube-system         etcd-k8s1                          1/1     Running   0          63m     <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1                1/1     Running   0          63m     <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1       1/1     Running   0          63m     <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1                1/1     Running   0          63m     <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         log-gatherer-csddr                 1/1     Running   0          53m     <IP_ADDRESS>   k8s1   <none>           <none>
	 kube-system         log-gatherer-zvnwl                 1/1     Running   0          53m     <IP_ADDRESS>   k8s2   <none>           <none>
	 kube-system         registry-adder-8xn69               1/1     Running   0          54m     <IP_ADDRESS>   k8s2   <none>           <none>
	 kube-system         registry-adder-w2qjq               1/1     Running   0          54m     <IP_ADDRESS>   k8s1   <none>           <none>

Stderr:

Fetching command output from pods [cilium-m6s2p cilium-v2nvw]
cmd: kubectl exec -n kube-system cilium-m6s2p -- cilium service list
Exitcode: 0
Stdout:
 	 ID   Frontend             Service Type   Backend
	 1    <IP_ADDRESS>:3000   ClusterIP
	 2    <IP_ADDRESS>:9090   ClusterIP      1 => <IP_ADDRESS>:9090
	 3    <IP_ADDRESS>:443        ClusterIP      1 => <IP_ADDRESS>:6443
	 4    <IP_ADDRESS>:53        ClusterIP      1 => <IP_ADDRESS>:53
	 5    <IP_ADDRESS>:9153      ClusterIP      1 => <IP_ADDRESS>:9153

Stderr:

cmd: kubectl exec -n kube-system cilium-m6s2p -- cilium endpoint list
Exitcode: 0
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6       IPv4         STATUS
	            ENFORCEMENT        ENFORCEMENT
	 160        Disabled           Disabled          41050      k8s:io.cilium.k8s.policy.cluster=default          fd00::2c   <IP_ADDRESS>   not-ready
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
	                                                            k8s:io.kubernetes.pod.namespace=default
	                                                            k8s:name=echo
	 2455       Disabled           Disabled          4          reserved:health                                   fd00::3e   <IP_ADDRESS>    ready
	 3206       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                ready
	                                                            reserved:host
	 3506       Disabled           Disabled          21111      k8s:io.cilium.k8s.policy.cluster=default          fd00::b8   <IP_ADDRESS>   ready
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
	                                                            k8s:io.kubernetes.pod.namespace=kube-system
	                                                            k8s:k8s-app=kube-dns

Stderr:

cmd: kubectl exec -n kube-system cilium-v2nvw -- cilium service list
Exitcode: 0
Stdout:
 	 ID   Frontend             Service Type   Backend
	 1    <IP_ADDRESS>:3000   ClusterIP
	 2    <IP_ADDRESS>:53        ClusterIP      1 => <IP_ADDRESS>:53
	 3    <IP_ADDRESS>:9153      ClusterIP      1 => <IP_ADDRESS>:9153
	 4    <IP_ADDRESS>:9090   ClusterIP      1 => <IP_ADDRESS>:9090
	 5    <IP_ADDRESS>:443        ClusterIP      1 => <IP_ADDRESS>:6443

Stderr:

cmd: kubectl exec -n kube-system cilium-v2nvw -- cilium endpoint list
Exitcode: 0
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6        IPv4        STATUS
	            ENFORCEMENT        ENFORCEMENT
	 326        Disabled           Disabled          41050      k8s:io.cilium.k8s.policy.cluster=default          fd00::1e5   <IP_ADDRESS>   not-ready
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default
	                                                            k8s:io.kubernetes.pod.namespace=default
	                                                            k8s:name=echo
	 1024       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                ready
	                                                            k8s:node-role.kubernetes.io/master
	                                                            reserved:host
	 3715       Disabled           Disabled          4          reserved:health                                   fd00::11f   <IP_ADDRESS>   ready

Stderr:

===================== Exiting AfterFailed =====================
04:49:08 STEP: Running AfterEach for block EntireTestsuite K8sServicesTest
04:49:09 STEP: Running AfterEach for block EntireTestsuite

This is the first time I've seen a failure like this on v1.9 in a very long time. This is likely newly introduced to the v1.9 tree, or alternatively exceedingly rare.
@jrajahalme could this be related to https://github.com/cilium/cilium/commit/01e76877eeee4ed70e76c459a589ef1579728346 ?
I did a quick look over the CI insights dashboard and it seems like we rearranged the tests around June 21st so it's difficult to tell whether we simply started running the test more often around then, or whether the refactoring of the tests increased the likelihood of the test failing, or whether there may have been a change introduced around that time that increased the failure rate. There are other failures, but I don't know whether they were caused by the same issue as reported above or whether they were something different.

I have seen a very similar error on the master branch: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/1140/testReport/junit/Suite-k8s-1/16/K8sPolicyTest_Basic_Test_Validate_to_entities_policies_Validate_toEntities_All/
Different test suite, likely confirms the suspicion that it is unrelated to LRP.

Managed to reproduce it locally:
level=warning msg="JoinEP: Failed to load program" containerID=22bc4cf900 datapathPolicyRevision=0 desired
PolicyRevision=1 endpointID=175 error="Failed to replace Qdisc for lxcc93256387b59: Link not found" file-p
ath=175_next/bpf_lxc.o identity=1124 ipv4=<IP_ADDRESS> ipv6="fd02::e8" k8sPodName=kube-system/coredns-5fc58db
489-45xr5 subsys=datapath-loader veth=lxcc93256387b59

The 175 endpoint was CoreDNS which was recreated while cilium-agent was initializing. The new CoreDNS pod got a new identity, and the endpoint-175-regeneration-recovery controller was failing for some time until the 175 endpoint was removed which made the test to fail because of one of these:

JoinEP was found in the logs
The controller removal took longer than waiting for cilium to become healthy.

cilium-agent-with-175.log

Do you think it's possible for us to control CoreDNS pod redeployment wrt the restart? Is this triggered by the test framework being careless about the timing between restarting DNS pods and restarting the agent?

Yeah, I was thinking about proposing to wait with restarts until all cilium-agents are ready. As you said, avoiding to fail in the CI might hide real bugs.

Hit also in K8sPolicyTestExtended Validate toEntities KubeAPIServer Allows connection to KubeAPIServer

Hit in https://github.com/cilium/cilium/pull/19007.
dbdabccf_K8sPolicyTestExtended_Validate_toEntities_KubeAPIServer_Allows_connection_to_KubeAPIServer.zip

Same JoinEP due to old core-dns pod being deleted by Cilium/CI job while the EP being regenerated: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-5.4/1177/testReport/junit/Suite-k8s-1/21/K8sDatapathConfig_Host_firewall_With_VXLAN_and_endpoint_routes/

@jrajahalme thanks for the data point. I see two similarities between our PRs:

both fail when replacing the qdisc. In my patch, that happens after loading the prog. In iproute2-based Cilium, it happens before calling ip, which both loads and attaches the prog
the eventConsumeOffQueueWaitTime span is 1m+ in both our PRs

Luckily, with my patch, the time spent in the verifier is now also visible. This is on a run of K8sDatapathConfig_AutoDirectNodeRoutes_Check_connectivity_with_automatic_direct_nodes_routes.
(pprof) top20
Showing nodes accounting for 23.18s, 98.93% of 23.43s total
Dropped 180 nodes (cum <= 0.12s)
      flat  flat%   sum%        cum   cum%
    23.18s 98.93% 98.93%     23.19s 98.98%  syscall.Syscall
         0     0% 98.93%     23.18s 98.93%  github.com/cilium/cilium/pkg/bpf.LoadCollection
         0     0% 98.93%     23.24s 99.19%  github.com/cilium/cilium/pkg/datapath/loader.(*Loader).CompileOrLoad
         0     0% 98.93%     23.23s 99.15%  github.com/cilium/cilium/pkg/datapath/loader.(*Loader).ReloadDatapath
         0     0% 98.93%     23.23s 99.15%  github.com/cilium/cilium/pkg/datapath/loader.(*Loader).reloadDatapath
         0     0% 98.93%     23.23s 99.15%  github.com/cilium/cilium/pkg/datapath/loader.replaceDatapath
         0     0% 98.93%     23.24s 99.19%  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).realizeBPFState
         0     0% 98.93%     23.25s 99.23%  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerate
         0     0% 98.93%     23.24s 99.19%  github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regenerateBPF
         0     0% 98.93%     23.25s 99.23%  github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle
         0     0% 98.93%     23.25s 99.23%  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run
         0     0% 98.93%     23.25s 99.23%  github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run.func1
         0     0% 98.93%     23.16s 98.85%  github.com/cilium/ebpf.(*collectionLoader).loadProgram
         0     0% 98.93%     23.17s 98.89%  github.com/cilium/ebpf.NewCollectionWithOptions
         0     0% 98.93%     23.16s 98.85%  github.com/cilium/ebpf.newProgramWithOptions
         0     0% 98.93%     23.17s 98.89%  github.com/cilium/ebpf/internal/sys.BPF
         0     0% 98.93%     23.16s 98.85%  github.com/cilium/ebpf/internal/sys.ProgLoad
         0     0% 98.93%     23.17s 98.89%  github.com/cilium/ebpf/internal/unix.Syscall (inline)

Looking at the call graph, syscall.Syscall is indeed at the end of the chain, so it's indeed the one called from sys.ProgLoad. bpf.LoadCollection is the high-level function responsible for all bpf syscalls, so it does look to me like we're stuck in the verifier. With iproute2, this issue might be less or completely invisible in cpu profiles if Command.Execute() is never sampled during the profile.
This is a profile from Jarno's PR, can't infer much from this:
(pprof) top5
Showing nodes accounting for 110ms, 47.83% of 230ms total
Showing top 5 nodes out of 127
      flat  flat%   sum%        cum   cum%
      40ms 17.39% 17.39%       40ms 17.39%  runtime.futex
      20ms  8.70% 26.09%       20ms  8.70%  runtime.epollwait
      20ms  8.70% 34.78%      100ms 43.48%  runtime.findrunnable
      20ms  8.70% 43.48%       30ms 13.04%  syscall.Syscall
      10ms  4.35% 47.83%       10ms  4.35%  encoding/json.structEncoder.encode

This was addressed by #19159.