CI: kubernetes-e2e (ipv4): ReplicaSet should serve a basic image on each replica with a public image failed

CI failure
Link: https://github.com/cilium/cilium/actions/runs/5116398094/jobs/9198525758?pr=25743
Sysdump: cilium-sysdumps.zip
Kind Logs: kind-logs.zip
2023-05-30T00:59:16.0791494Z • [FAILED] [17.190 seconds]
2023-05-30T00:59:16.0797004Z [sig-apps] ReplicaSet [It] should serve a basic image on each replica with a public image  [Conformance]
2023-05-30T00:59:16.0800210Z test/e2e/apps/replica_set.go:111
2023-05-30T00:59:16.0800429Z
2023-05-30T00:59:16.0806465Z   Timeline >>
2023-05-30T00:59:16.0807247Z   STEP: Creating a kubernetes client @ 05/30/23 00:58:58.801
2023-05-30T00:59:16.0818090Z   May 30 00:58:58.801: INFO: >>> kubeConfig: /home/runner/work/cilium/cilium/_artifacts/kubeconfig.conf
2023-05-30T00:59:16.0820605Z   STEP: Building a namespace api object, basename replicaset @ 05/30/23 00:58:58.805
2023-05-30T00:59:16.0829388Z   STEP: Waiting for a default service account to be provisioned in namespace @ 05/30/23 00:58:58.928
2023-05-30T00:59:16.0838817Z   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 05/30/23 00:58:58.947
2023-05-30T00:59:16.0846998Z   May 30 00:58:58.971: INFO: Creating ReplicaSet my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd
2023-05-30T00:59:16.0857832Z   May 30 00:58:59.026: INFO: Pod name my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd: Found 0 pods out of 1
2023-05-30T00:59:16.0863733Z   May 30 00:59:04.096: INFO: Pod name my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd: Found 1 pods out of 1
2023-05-30T00:59:16.0870275Z   May 30 00:59:04.096: INFO: Ensuring a pod for ReplicaSet "my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd" is running
2023-05-30T00:59:16.0877855Z   May 30 00:59:08.236: INFO: Pod "my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f" is running (conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-05-30 00:58:59 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-05-30 00:58:59 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-05-30 00:58:59 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2023-05-30 00:58:59 +0000 UTC Reason: Message:}])
2023-05-30T00:59:16.0882662Z   May 30 00:59:08.236: INFO: Trying to dial the pod
2023-05-30T00:59:16.0887003Z   STEP: trying to dial each unique pod @ 05/30/23 00:59:08.236
2023-05-30T00:59:16.0887364Z   May 30 00:59:08.320: INFO: Failed inside E2E framework:
2023-05-30T00:59:16.0892486Z       k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodsResponding({0x7f57c48cc958, 0xc004695980}, {0x72b3810?, 0xc001cbcea0}, {0xc00517edc0, 0xf}, {0xc0046c8040, 0x36}, 0x1, 0x1bf08eb000, ...)
2023-05-30T00:59:16.0893096Z       	test/e2e/framework/pod/wait.go:625 +0x3ef
2023-05-30T00:59:16.0898368Z       k8s.io/kubernetes/test/e2e/apps.testReplicaSetServeImageOrFail({0x7f57c48cc958, 0xc004695980}, 0xc0009383c0, {0x6a9bbda, 0x5}, {0xc0004d9f20, 0x2c})
2023-05-30T00:59:16.0898892Z       	test/e2e/apps/replica_set.go:228 +0xac5
2023-05-30T00:59:16.0907491Z       k8s.io/kubernetes/test/e2e/apps.glob..func9.1({0x7f57c48cc958?, 0xc004695980?})
2023-05-30T00:59:16.0910492Z       	test/e2e/apps/replica_set.go:112 +0x45
2023-05-30T00:59:16.0911186Z   [FAILED] in [It] - test/e2e/apps/replica_set.go:228 @ 05/30/23 00:59:08.32
2023-05-30T00:59:16.0915813Z   May 30 00:59:08.320: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
2023-05-30T00:59:16.0920156Z   STEP: dump namespace information after failure @ 05/30/23 00:59:08.453
2023-05-30T00:59:16.0922680Z   STEP: Collecting events from namespace "replicaset-9420". @ 05/30/23 00:59:08.454
2023-05-30T00:59:16.0926967Z   STEP: Found 5 events. @ 05/30/23 00:59:08.472
2023-05-30T00:59:16.0927919Z   May 30 00:59:08.472: INFO: At 2023-05-30 00:58:59 +0000 UTC - event for my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd: {replicaset-controller } SuccessfulCreate: Created pod: my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f
2023-05-30T00:59:16.0931518Z   May 30 00:59:08.472: INFO: At 2023-05-30 00:58:59 +0000 UTC - event for my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f: {default-scheduler } Scheduled: Successfully assigned replicaset-9420/my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f to cilium-testing-worker2
2023-05-30T00:59:16.0935890Z   May 30 00:59:08.472: INFO: At 2023-05-30 00:59:04 +0000 UTC - event for my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f: {kubelet cilium-testing-worker2} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
2023-05-30T00:59:16.0937203Z   May 30 00:59:08.472: INFO: At 2023-05-30 00:59:04 +0000 UTC - event for my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f: {kubelet cilium-testing-worker2} Created: Created container my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd
2023-05-30T00:59:16.0941096Z   May 30 00:59:08.472: INFO: At 2023-05-30 00:59:07 +0000 UTC - event for my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f: {kubelet cilium-testing-worker2} Started: Started container my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd
2023-05-30T00:59:16.0941770Z   May 30 00:59:08.500: INFO: POD                                                           NODE                    PHASE    GRACE  CONDITIONS
2023-05-30T00:59:16.0948669Z   May 30 00:59:08.500: INFO: my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f  cilium-testing-worker2  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-05-30 00:58:59 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-05-30 00:59:08 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-05-30 00:59:08 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-05-30 00:58:59 +0000 UTC  }]
2023-05-30T00:59:16.0949694Z   May 30 00:59:08.500: INFO:
2023-05-30T00:59:16.0952871Z   May 30 00:59:08.610: INFO:
2023-05-30T00:59:16.0953442Z   Logging node info for node cilium-testing-control-plane
2023-05-30T00:59:16.0987200Z   May 30 00:59:08.648: INFO: Node Info: &Node{ObjectMeta:{cilium-testing-control-plane    91cd365f-daff-4ad0-8e7a-afd68eab878b 12009 0 2023-05-30 00:50:00 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:cilium-testing-control-plane kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers:] map[kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-05-30 00:50:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2023-05-30 00:50:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-05-30 00:52:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"<IP_ADDRESS>/24\"":{}}}} } {cilium-operator-generic Update v1 2023-05-30 00:52:52 +0000 UTC FieldsV1 {"f:spec":{"f:taints":{}}} } {cilium-operator-generic Update v1 2023-05-30 00:52:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-05-30 00:58:14 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:<IP_ADDRESS>/24,DoNotUseExternalID:,ProviderID:kind://docker/cilium-testing/cilium-testing-control-plane,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[<IP_ADDRESS>/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-05-30 00:52:52 +0000 UTC,LastTransitionTime:2023-05-30 00:52:52 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-05-30 00:58:14 +0000 UTC,LastTransitionTime:2023-05-30 00:49:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-05-30 00:58:14 +0000 UTC,LastTransitionTime:2023-05-30 00:49:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-05-30 00:58:14 +0000 UTC,LastTransitionTime:2023-05-30 00:49:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-05-30 00:58:14 +0000 UTC,LastTransitionTime:2023-05-30 00:52:46 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:<IP_ADDRESS>,},NodeAddress{Type:Hostname,Address:cilium-testing-control-plane,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:75a172cf48be43bc8beebf9388195e2f,SystemUUID:5d692b2f-d751-4abf-b5f1-1df855081e49,BootID:9d37c8de-b8bf-4b4e-9eb2-4133cdd22a24,KernelVersion:5.15.0-1037-azure,OSImage:Debian GNU/Linux 11 (bullseye),ContainerRuntimeVersion:containerd://1.6.21,KubeletVersion:v1.27.1,KubeProxyVersion:v1.27.1,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium-ci@sha256:44d6ae625ae8072db9c5156aca3f5feb65a8f3652a8f33682d597178fabc707b quay.io/cilium/cilium-ci:ae50feb36532dd37a3e78e0239501b01e41a6b9a],SizeBytes:186501338,},ContainerImage{Names:[registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:e47e5ddd7105c32de2fee0049d89ebb3253262694f678b92c70678f4e7b1d1c1 registry.k8s.io/kube-apiserver:v1.27.1],SizeBytes:83436031,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:c2a0c5ea531ff4e7f4c4e44fe5969bbe8e77dbdb6171961c81e07454648c6526 registry.k8s.io/kube-controller-manager:v1.27.1],SizeBytes:74407565,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:a9d9b674a55b137b1f10bbcd7f0e43b7568d20c8df5bf969d1bb8a37f3d365db registry.k8s.io/kube-proxy:v1.27.1],SizeBytes:72702973,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:8b1ab2b02004fc7ae90a8c562dfdf13d63361770d15f426feb8f79779557277f registry.k8s.io/kube-scheduler:v1.27.1],SizeBytes:59781261,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20230511-dc714da8],SizeBytes:27731571,},ContainerImage{Names:[docker.io/kindest/local-path-provisioner:v20230511-dc714da8],SizeBytes:19351145,},ContainerImage{Names:[registry.k8s.io/coredns/coredns:v1.10.1],SizeBytes:16190758,},ContainerImage{Names:[docker.io/kindest/local-path-helper:v20230510-486859a6],SizeBytes:3052318,},ContainerImage{Names:[registry.k8s.io/pause:3.7],SizeBytes:311278,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
2023-05-30T00:59:16.1009956Z   May 30 00:59:08.648: INFO:
2023-05-30T00:59:16.1014662Z   Logging kubelet events for node cilium-testing-control-plane
2023-05-30T00:59:16.1017615Z   May 30 00:59:08.700: INFO:
2023-05-30T00:59:16.1023990Z   Logging pods the kubelet thinks is on node cilium-testing-control-plane
2023-05-30T00:59:16.1028337Z   May 30 00:59:08.814: INFO: kube-apiserver-cilium-testing-control-plane started at 2023-05-30 00:50:03 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1032336Z   May 30 00:59:08.814: INFO: 	Container kube-apiserver ready: true, restart count 0
2023-05-30T00:59:16.1033255Z   May 30 00:59:08.814: INFO: kube-controller-manager-cilium-testing-control-plane started at 2023-05-30 00:50:03 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1034158Z   May 30 00:59:08.814: INFO: 	Container kube-controller-manager ready: true, restart count 0
2023-05-30T00:59:16.1040887Z   May 30 00:59:08.814: INFO: kube-scheduler-cilium-testing-control-plane started at 2023-05-30 00:50:03 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1047760Z   May 30 00:59:08.814: INFO: 	Container kube-scheduler ready: true, restart count 0
2023-05-30T00:59:16.1051287Z   May 30 00:59:08.814: INFO: etcd-cilium-testing-control-plane started at 2023-05-30 00:50:03 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1052061Z   May 30 00:59:08.814: INFO: 	Container etcd ready: true, restart count 0
2023-05-30T00:59:16.1056574Z   May 30 00:59:08.814: INFO: cilium-j7qbf started at 2023-05-30 00:52:17 +0000 UTC (6+1 container statuses recorded)
2023-05-30T00:59:16.1057102Z   May 30 00:59:08.814: INFO: 	Init container config ready: true, restart count 0
2023-05-30T00:59:16.1061369Z   May 30 00:59:08.814: INFO: 	Init container mount-cgroup ready: true, restart count 0
2023-05-30T00:59:16.1062174Z   May 30 00:59:08.814: INFO: 	Init container apply-sysctl-overwrites ready: true, restart count 0
2023-05-30T00:59:16.1062731Z   May 30 00:59:08.814: INFO: 	Init container mount-bpf-fs ready: true, restart count 0
2023-05-30T00:59:16.1066649Z   May 30 00:59:08.814: INFO: 	Init container clean-cilium-state ready: true, restart count 0
2023-05-30T00:59:16.1067173Z   May 30 00:59:08.814: INFO: 	Init container install-cni-binaries ready: true, restart count 0
2023-05-30T00:59:16.1072967Z   May 30 00:59:08.814: INFO: 	Container cilium-agent ready: true, restart count 0
2023-05-30T00:59:16.1073406Z   May 30 00:59:09.061: INFO:
2023-05-30T00:59:16.1077926Z   Latency metrics for node cilium-testing-control-plane
2023-05-30T00:59:16.1078297Z   May 30 00:59:09.061: INFO:
2023-05-30T00:59:16.1078780Z   Logging node info for node cilium-testing-worker
2023-05-30T00:59:16.1098555Z   May 30 00:59:09.103: INFO: Node Info: &Node{ObjectMeta:{cilium-testing-worker    7a8730bc-0e38-42f2-8ccf-429777d4f5f4 6426 0 2023-05-30 00:50:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:cilium-testing-worker kubernetes.io/os:linux] map[kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-05-30 00:50:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2023-05-30 00:50:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kube-controller-manager Update v1 2023-05-30 00:52:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"<IP_ADDRESS>/24\"":{}}}} } {cilium-operator-generic Update v1 2023-05-30 00:52:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-05-30 00:55:57 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:<IP_ADDRESS>/24,DoNotUseExternalID:,ProviderID:kind://docker/cilium-testing/cilium-testing-worker,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[<IP_ADDRESS>/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-05-30 00:52:52 +0000 UTC,LastTransitionTime:2023-05-30 00:52:52 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-05-30 00:55:57 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-05-30 00:55:57 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-05-30 00:55:57 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-05-30 00:55:57 +0000 UTC,LastTransitionTime:2023-05-30 00:52:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:<IP_ADDRESS>,},NodeAddress{Type:Hostname,Address:cilium-testing-worker,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:265b2ba0de6b43ed8bdfa8bf19c64fd5,SystemUUID:953e86a3-7494-46b5-a757-649eef0a5222,BootID:9d37c8de-b8bf-4b4e-9eb2-4133cdd22a24,KernelVersion:5.15.0-1037-azure,OSImage:Debian GNU/Linux 11 (bullseye),ContainerRuntimeVersion:containerd://1.6.21,KubeletVersion:v1.27.1,KubeProxyVersion:v1.27.1,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium-ci@sha256:44d6ae625ae8072db9c5156aca3f5feb65a8f3652a8f33682d597178fabc707b quay.io/cilium/cilium-ci:ae50feb36532dd37a3e78e0239501b01e41a6b9a],SizeBytes:186501338,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:e47e5ddd7105c32de2fee0049d89ebb3253262694f678b92c70678f4e7b1d1c1 registry.k8s.io/kube-apiserver:v1.27.1],SizeBytes:83436031,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:c2a0c5ea531ff4e7f4c4e44fe5969bbe8e77dbdb6171961c81e07454648c6526 registry.k8s.io/kube-controller-manager:v1.27.1],SizeBytes:74407565,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:a9d9b674a55b137b1f10bbcd7f0e43b7568d20c8df5bf969d1bb8a37f3d365db registry.k8s.io/kube-proxy:v1.27.1],SizeBytes:72702973,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:8b1ab2b02004fc7ae90a8c562dfdf13d63361770d15f426feb8f79779557277f registry.k8s.io/kube-scheduler:v1.27.1],SizeBytes:59781261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20230511-dc714da8],SizeBytes:27731571,},ContainerImage{Names:[quay.io/cilium/operator-generic-ci@sha256:803a8f2a60f195de59a3cd56703adc3bdcea92dade3b5b37a3b3efb2ba9fc513 quay.io/cilium/operator-generic-ci:ae50feb36532dd37a3e78e0239501b01e41a6b9a],SizeBytes:23118400,},ContainerImage{Names:[docker.io/kindest/local-path-provisioner:v20230511-dc714da8],SizeBytes:19351145,},ContainerImage{Names:[registry.k8s.io/coredns/coredns:v1.10.1],SizeBytes:16190758,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[docker.io/kindest/local-path-helper:v20230510-486859a6],SizeBytes:3052318,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause:3.7],SizeBytes:311278,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
2023-05-30T00:59:16.1132326Z   May 30 00:59:09.103: INFO:
2023-05-30T00:59:16.1132887Z   Logging kubelet events for node cilium-testing-worker
2023-05-30T00:59:16.1133241Z   May 30 00:59:09.216: INFO:
2023-05-30T00:59:16.1137288Z   Logging pods the kubelet thinks is on node cilium-testing-worker
2023-05-30T00:59:16.1137960Z   May 30 00:59:09.273: INFO: pod-secrets-07d24698-9f0c-4cf5-a515-da9b579dc9ac started at 2023-05-30 00:57:53 +0000 UTC (0+3 container statuses recorded)
2023-05-30T00:59:16.1142060Z   May 30 00:59:09.273: INFO: 	Container creates-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1142732Z   May 30 00:59:09.273: INFO: 	Container dels-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1146343Z   May 30 00:59:09.273: INFO: 	Container upds-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1147072Z   May 30 00:59:09.273: INFO: externalname-service-sn9w4 started at 2023-05-30 00:58:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1147670Z   May 30 00:59:09.273: INFO: 	Container externalname-service ready: true, restart count 0
2023-05-30T00:59:16.1152775Z   May 30 00:59:09.273: INFO: e2e-configmap-dns-server-69ee586e-0b34-473b-bffc-4aaac8d5f4c2 started at 2023-05-30 00:58:54 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1153422Z   May 30 00:59:09.273: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1154035Z   May 30 00:59:09.273: INFO: webserver-pod started at 2023-05-30 00:56:26 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1158289Z   May 30 00:59:09.273: INFO: 	Container agnhost ready: false, restart count 0
2023-05-30T00:59:16.1165635Z   May 30 00:59:09.273: INFO: cilium-rdjbg started at 2023-05-30 00:52:18 +0000 UTC (6+1 container statuses recorded)
2023-05-30T00:59:16.1166129Z   May 30 00:59:09.273: INFO: 	Init container config ready: true, restart count 0
2023-05-30T00:59:16.1168561Z   May 30 00:59:09.273: INFO: 	Init container mount-cgroup ready: true, restart count 0
2023-05-30T00:59:16.1173117Z   May 30 00:59:09.273: INFO: 	Init container apply-sysctl-overwrites ready: true, restart count 0
2023-05-30T00:59:16.1173682Z   May 30 00:59:09.273: INFO: 	Init container mount-bpf-fs ready: true, restart count 0
2023-05-30T00:59:16.1176895Z   May 30 00:59:09.273: INFO: 	Init container clean-cilium-state ready: true, restart count 0
2023-05-30T00:59:16.1177496Z   May 30 00:59:09.273: INFO: 	Init container install-cni-binaries ready: true, restart count 0
2023-05-30T00:59:16.1180418Z   May 30 00:59:09.273: INFO: 	Container cilium-agent ready: true, restart count 0
2023-05-30T00:59:16.1181438Z   May 30 00:59:09.273: INFO: liveness-ae9bc4f3-087b-4aa8-84ad-aeb3f69d5327 started at 2023-05-30 00:58:21 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1182179Z   May 30 00:59:09.273: INFO: 	Container agnhost-container ready: true, restart count 1
2023-05-30T00:59:16.1185826Z   May 30 00:59:09.273: INFO: ss2-0 started at 2023-05-30 00:58:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1186266Z   May 30 00:59:09.273: INFO: 	Container webserver ready: true, restart count 0
2023-05-30T00:59:16.1189646Z   May 30 00:59:09.273: INFO: ss2-0 started at 2023-05-30 00:58:19 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1190316Z   May 30 00:59:09.273: INFO: 	Container webserver ready: true, restart count 0
2023-05-30T00:59:16.1194119Z   May 30 00:59:09.273: INFO: ss-0 started at 2023-05-30 00:58:20 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1194581Z   May 30 00:59:09.273: INFO: 	Container webserver ready: false, restart count 0
2023-05-30T00:59:16.1198417Z   May 30 00:59:09.273: INFO: var-expansion-048f0808-ee75-4dd4-a661-587f87e58081 started at 2023-05-30 00:59:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1199082Z   May 30 00:59:09.273: INFO: 	Container dapi-container ready: false, restart count 0
2023-05-30T00:59:16.1203470Z   May 30 00:59:09.273: INFO: affinity-clusterip-timeout-j77gp started at 2023-05-30 00:59:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1204200Z   May 30 00:59:09.273: INFO: 	Container affinity-clusterip-timeout ready: false, restart count 0
2023-05-30T00:59:16.1204958Z   May 30 00:59:09.273: INFO: pod-projected-configmaps-2df510fe-9588-467c-b9fe-b00e82fd2899 started at 2023-05-30 00:57:45 +0000 UTC (0+3 container statuses recorded)
2023-05-30T00:59:16.1212308Z   May 30 00:59:09.273: INFO: 	Container createcm-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1212876Z   May 30 00:59:09.273: INFO: 	Container delcm-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1216227Z   May 30 00:59:09.273: INFO: 	Container updcm-volume-test ready: true, restart count 0
2023-05-30T00:59:16.1216878Z   May 30 00:59:09.273: INFO: e2e-dns-utils started at 2023-05-30 00:59:00 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1217442Z   May 30 00:59:09.273: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1221666Z   May 30 00:59:09.273: INFO: affinity-clusterip-transition-2b5tn started at 2023-05-30 00:58:58 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1222269Z   May 30 00:59:09.273: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
2023-05-30T00:59:16.1227070Z   May 30 00:59:09.273: INFO: cilium-operator-78fbd7759-v42wf started at 2023-05-30 00:52:17 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1227781Z   May 30 00:59:09.273: INFO: 	Container cilium-operator ready: true, restart count 0
2023-05-30T00:59:16.1230989Z   May 30 00:59:09.273: INFO: concurrent-28090139-v2c94 started at 2023-05-30 00:59:00 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1231445Z   May 30 00:59:09.273: INFO: 	Container c ready: true, restart count 0
2023-05-30T00:59:16.1232119Z   May 30 00:59:09.273: INFO: affinity-clusterip-timeout-cfzw6 started at 2023-05-30 00:59:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1238864Z   May 30 00:59:09.273: INFO: 	Container affinity-clusterip-timeout ready: false, restart count 0
2023-05-30T00:59:16.1239630Z   May 30 00:59:09.273: INFO: pod-configmaps-1af4ced9-a6a4-4eba-9df6-61d4b11adba7 started at 2023-05-30 00:57:05 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1243139Z   May 30 00:59:09.273: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1243887Z   May 30 00:59:09.273: INFO: httpd-deployment-5cd84d4f9-tr5cc started at 2023-05-30 00:59:08 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1248044Z   May 30 00:59:09.273: INFO: 	Container httpd ready: false, restart count 0
2023-05-30T00:59:16.1248970Z   May 30 00:59:09.273: INFO: ss-0 started at 2023-05-30 00:59:03 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1249418Z   May 30 00:59:09.273: INFO: 	Container webserver ready: false, restart count 0
2023-05-30T00:59:16.1253980Z   May 30 00:59:09.273: INFO: pod-csi-inline-volumes started at 2023-05-30 00:57:47 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1254595Z   May 30 00:59:09.273: INFO: 	Container pod-csi-inline-volumes ready: false, restart count 0
2023-05-30T00:59:16.1257971Z   May 30 00:59:09.273: INFO: ss2-2 started at 2023-05-30 00:58:22 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1258687Z   May 30 00:59:09.273: INFO: 	Container webserver ready: false, restart count 0
2023-05-30T00:59:16.1261681Z   May 30 00:59:09.273: INFO: webserver-pod started at 2023-05-30 00:58:24 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1264908Z   May 30 00:59:09.273: INFO: 	Container agnhost ready: true, restart count 0
2023-05-30T00:59:16.1265356Z   May 30 00:59:10.880: INFO:
2023-05-30T00:59:16.1268996Z   Latency metrics for node cilium-testing-worker
2023-05-30T00:59:16.1269361Z   May 30 00:59:10.880: INFO:
2023-05-30T00:59:16.1269758Z   Logging node info for node cilium-testing-worker2
2023-05-30T00:59:16.1294598Z   May 30 00:59:10.998: INFO: Node Info: &Node{ObjectMeta:{cilium-testing-worker2    8c2c5a61-02b0-4800-9311-ce38679d7a27 13943 0 2023-05-30 00:50:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:cilium-testing-worker2 kubernetes.io/os:linux] map[kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-05-30 00:50:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}}} } {kubeadm Update v1 2023-05-30 00:50:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}} } {kube-controller-manager Update v1 2023-05-30 00:52:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"<IP_ADDRESS>/24\"":{}}}} } {cilium-operator-generic Update v1 2023-05-30 00:52:49 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-05-30 00:59:04 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:<IP_ADDRESS>/24,DoNotUseExternalID:,ProviderID:kind://docker/cilium-testing/cilium-testing-worker2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[<IP_ADDRESS>/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{89297309696 0} {<nil>} 87204404Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7281266688 0} {<nil>} 7110612Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-05-30 00:52:49 +0000 UTC,LastTransitionTime:2023-05-30 00:52:49 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-05-30 00:59:04 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-05-30 00:59:04 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-05-30 00:59:04 +0000 UTC,LastTransitionTime:2023-05-30 00:50:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-05-30 00:59:04 +0000 UTC,LastTransitionTime:2023-05-30 00:52:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:<IP_ADDRESS>,},NodeAddress{Type:Hostname,Address:cilium-testing-worker2,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:4bf1135f85e8409a8e1aed9e87c4005b,SystemUUID:925c6a5c-a2b1-44e5-953e-3fe83b45e86a,BootID:9d37c8de-b8bf-4b4e-9eb2-4133cdd22a24,KernelVersion:5.15.0-1037-azure,OSImage:Debian GNU/Linux 11 (bullseye),ContainerRuntimeVersion:containerd://1.6.21,KubeletVersion:v1.27.1,KubeProxyVersion:v1.27.1,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium-ci@sha256:44d6ae625ae8072db9c5156aca3f5feb65a8f3652a8f33682d597178fabc707b quay.io/cilium/cilium-ci:ae50feb36532dd37a3e78e0239501b01e41a6b9a],SizeBytes:186501338,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:e47e5ddd7105c32de2fee0049d89ebb3253262694f678b92c70678f4e7b1d1c1 registry.k8s.io/kube-apiserver:v1.27.1],SizeBytes:83436031,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:c2a0c5ea531ff4e7f4c4e44fe5969bbe8e77dbdb6171961c81e07454648c6526 registry.k8s.io/kube-controller-manager:v1.27.1],SizeBytes:74407565,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:a9d9b674a55b137b1f10bbcd7f0e43b7568d20c8df5bf969d1bb8a37f3d365db registry.k8s.io/kube-proxy:v1.27.1],SizeBytes:72702973,},ContainerImage{Names:[docker.io/library/import-2023-05-12@sha256:8b1ab2b02004fc7ae90a8c562dfdf13d63361770d15f426feb8f79779557277f registry.k8s.io/kube-scheduler:v1.27.1],SizeBytes:59781261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20230511-dc714da8],SizeBytes:27731571,},ContainerImage{Names:[docker.io/kindest/local-path-provisioner:v20230511-dc714da8],SizeBytes:19351145,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/coredns/coredns:v1.10.1],SizeBytes:16190758,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[docker.io/kindest/local-path-helper:v20230510-486859a6],SizeBytes:3052318,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause:3.7],SizeBytes:311278,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
2023-05-30T00:59:16.1317969Z   May 30 00:59:10.999: INFO:
2023-05-30T00:59:16.1321440Z   Logging kubelet events for node cilium-testing-worker2
2023-05-30T00:59:16.1321881Z   May 30 00:59:11.137: INFO:
2023-05-30T00:59:16.1329128Z   Logging pods the kubelet thinks is on node cilium-testing-worker2
2023-05-30T00:59:16.1330048Z   May 30 00:59:11.192: INFO: affinity-clusterip-transition-6xz97 started at 2023-05-30 00:58:58 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1330678Z   May 30 00:59:11.192: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
2023-05-30T00:59:16.1336521Z   May 30 00:59:11.192: INFO: affinity-clusterip-transition-597tg started at 2023-05-30 00:58:58 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1337171Z   May 30 00:59:11.192: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
2023-05-30T00:59:16.1345073Z   May 30 00:59:11.192: INFO: test-dns-nameservers started at 2023-05-30 00:59:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1345844Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: false, restart count 0
2023-05-30T00:59:16.1346585Z   May 30 00:59:11.192: INFO: execpod-affinityblqm9 started at 2023-05-30 00:59:07 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1352358Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: false, restart count 0
2023-05-30T00:59:16.1352966Z   May 30 00:59:11.192: INFO: ss2-1 started at 2023-05-30 00:58:40 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1357569Z   May 30 00:59:11.192: INFO: 	Container webserver ready: false, restart count 0
2023-05-30T00:59:16.1362046Z   May 30 00:59:11.192: INFO: coredns-5d78c9869d-bsvpd started at 2023-05-30 00:52:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1362584Z   May 30 00:59:11.192: INFO: 	Container coredns ready: true, restart count 0
2023-05-30T00:59:16.1367458Z   May 30 00:59:11.192: INFO: my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f started at 2023-05-30 00:58:59 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1368288Z   May 30 00:59:11.192: INFO: 	Container my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd ready: true, restart count 0
2023-05-30T00:59:16.1369057Z   May 30 00:59:11.192: INFO: var-expansion-e351013b-0660-477f-b118-46538895d807 started at 2023-05-30 00:59:05 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1373579Z   May 30 00:59:11.192: INFO: 	Container dapi-container ready: false, restart count 0
2023-05-30T00:59:16.1374294Z   May 30 00:59:11.192: INFO: liveness-68469f6a-90ba-47ed-b152-84eaa36eed10 started at 2023-05-30 00:58:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1378840Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1379568Z   May 30 00:59:11.192: INFO: externalname-service-jwndg started at 2023-05-30 00:58:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1384335Z   May 30 00:59:11.192: INFO: 	Container externalname-service ready: true, restart count 0
2023-05-30T00:59:16.1385041Z   May 30 00:59:11.192: INFO: kube-proxy-mode-detector started at 2023-05-30 00:58:52 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1385624Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1390850Z   May 30 00:59:11.192: INFO: test-rs-tgn76 started at 2023-05-30 00:59:09 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1391582Z   May 30 00:59:11.192: INFO: 	Container httpd ready: false, restart count 0
2023-05-30T00:59:16.1397026Z   May 30 00:59:11.192: INFO: ss2-1 started at 2023-05-30 00:57:17 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1397613Z   May 30 00:59:11.192: INFO: 	Container webserver ready: true, restart count 0
2023-05-30T00:59:16.1400506Z   May 30 00:59:11.192: INFO: pause-pod-1 started at 2023-05-30 00:58:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1401126Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1401675Z   May 30 00:59:11.192: INFO: execpodjzg55 started at 2023-05-30 00:59:01 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1406212Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1406809Z   May 30 00:59:11.192: INFO: cilium-kq4kt started at 2023-05-30 00:52:18 +0000 UTC (6+1 container statuses recorded)
2023-05-30T00:59:16.1410152Z   May 30 00:59:11.192: INFO: 	Init container config ready: true, restart count 0
2023-05-30T00:59:16.1410732Z   May 30 00:59:11.192: INFO: 	Init container mount-cgroup ready: true, restart count 0
2023-05-30T00:59:16.1413976Z   May 30 00:59:11.192: INFO: 	Init container apply-sysctl-overwrites ready: true, restart count 0
2023-05-30T00:59:16.1414646Z   May 30 00:59:11.192: INFO: 	Init container mount-bpf-fs ready: true, restart count 0
2023-05-30T00:59:16.1415159Z   May 30 00:59:11.192: INFO: 	Init container clean-cilium-state ready: true, restart count 0
2023-05-30T00:59:16.1417976Z   May 30 00:59:11.192: INFO: 	Init container install-cni-binaries ready: true, restart count 0
2023-05-30T00:59:16.1418510Z   May 30 00:59:11.192: INFO: 	Container cilium-agent ready: true, restart count 0
2023-05-30T00:59:16.1423403Z   May 30 00:59:11.192: INFO: local-path-provisioner-6bc4bddd6b-v7krz started at 2023-05-30 00:52:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1424024Z   May 30 00:59:11.192: INFO: 	Container local-path-provisioner ready: true, restart count 0
2023-05-30T00:59:16.1427582Z   May 30 00:59:11.192: INFO: pod-4ba99a0e-b377-4deb-b237-f17306396d89 started at 2023-05-30 00:59:07 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1428279Z   May 30 00:59:11.192: INFO: 	Container test-container ready: false, restart count 0
2023-05-30T00:59:16.1433533Z   May 30 00:59:11.192: INFO: forbid-28090134-tm9lt started at 2023-05-30 00:54:00 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1435778Z   May 30 00:59:11.192: INFO: 	Container c ready: true, restart count 0
2023-05-30T00:59:16.1443597Z   May 30 00:59:11.192: INFO: netserver-1 started at 2023-05-30 00:59:09 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1449114Z   May 30 00:59:11.192: INFO: 	Container webserver ready: false, restart count 0
2023-05-30T00:59:16.1449877Z   May 30 00:59:11.192: INFO: affinity-clusterip-timeout-4glzc started at 2023-05-30 00:59:02 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1460240Z   May 30 00:59:11.192: INFO: 	Container affinity-clusterip-timeout ready: false, restart count 0
2023-05-30T00:59:16.1461210Z   May 30 00:59:11.192: INFO: pod-exec-websocket-8f595541-38aa-431c-bd7c-23b6278858fe started at 2023-05-30 00:58:52 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1465044Z   May 30 00:59:11.192: INFO: 	Container main ready: true, restart count 0
2023-05-30T00:59:16.1465933Z   May 30 00:59:11.192: INFO: ss2-2 started at 2023-05-30 00:58:48 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1476130Z   May 30 00:59:11.192: INFO: 	Container webserver ready: true, restart count 0
2023-05-30T00:59:16.1476970Z   May 30 00:59:11.192: INFO: pod-4db9e0ad-149b-4290-9ed3-0dcc619843e0 started at 2023-05-30 00:59:00 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1481832Z   May 30 00:59:11.192: INFO: 	Container test-container ready: true, restart count 0
2023-05-30T00:59:16.1482792Z   May 30 00:59:11.192: INFO: pause-pod-1 started at 2023-05-30 00:57:44 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1483354Z   May 30 00:59:11.192: INFO: 	Container agnhost-container ready: true, restart count 0
2023-05-30T00:59:16.1492509Z   May 30 00:59:11.192: INFO: coredns-5d78c9869d-9t64c started at 2023-05-30 00:52:49 +0000 UTC (0+1 container statuses recorded)
2023-05-30T00:59:16.1492940Z   May 30 00:59:11.192: INFO: 	Container coredns ready: true, restart count 0
2023-05-30T00:59:16.1495778Z   May 30 00:59:15.985: INFO:
2023-05-30T00:59:16.1496213Z   Latency metrics for node cilium-testing-worker2
2023-05-30T00:59:16.1500578Z   STEP: Destroying namespace "replicaset-9420" for this suite. @ 05/30/23 00:59:15.985
2023-05-30T00:59:16.1501191Z   << Timeline
2023-05-30T00:59:16.1501368Z
2023-05-30T00:59:16.1501661Z   [FAILED] checking pod responses: Told to stop trying after 0.083s.
2023-05-30T00:59:16.1507645Z   Unexpected final error while getting []pod.response: Controller my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd: failed to Get from replica pod my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f:
2023-05-30T00:59:16.1508309Z       <*errors.StatusError | 0xc0041b8f00>:
2023-05-30T00:59:16.1512007Z       the server is currently unable to handle the request (get pods my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f)
2023-05-30T00:59:16.1512493Z       {
2023-05-30T00:59:16.1512775Z           ErrStatus:
2023-05-30T00:59:16.1513017Z               code: 503
2023-05-30T00:59:16.1519102Z               details:
2023-05-30T00:59:16.1528049Z                 causes:
2023-05-30T00:59:16.1528438Z                 - message: unknown
2023-05-30T00:59:16.1533864Z                   reason: UnexpectedServerResponse
2023-05-30T00:59:16.1534274Z                 kind: pods
2023-05-30T00:59:16.1534804Z                 name: my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f
2023-05-30T00:59:16.1538808Z               message: the server is currently unable to handle the request (get pods my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd-vdx7f)
2023-05-30T00:59:16.1539301Z               metadata: {}
2023-05-30T00:59:16.1542095Z               reason: ServiceUnavailable
2023-05-30T00:59:16.1542370Z               status: Failure,
2023-05-30T00:59:16.1545023Z       }
2023-05-30T00:59:16.1547480Z   Pod status:
2023-05-30T00:59:16.1549410Z       <v1.PodStatus>:
2023-05-30T00:59:16.1552402Z           conditions:
2023-05-30T00:59:16.1552847Z           - lastProbeTime: null
2023-05-30T00:59:16.1553224Z             lastTransitionTime: "2023-05-30T00:58:59Z"
2023-05-30T00:59:16.1559822Z             status: "True"
2023-05-30T00:59:16.1560137Z             type: Initialized
2023-05-30T00:59:16.1563056Z           - lastProbeTime: null
2023-05-30T00:59:16.1563550Z             lastTransitionTime: "2023-05-30T00:58:59Z"
2023-05-30T00:59:16.1567914Z             message: 'containers with unready status: [my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd]'
2023-05-30T00:59:16.1568420Z             reason: ContainersNotReady
2023-05-30T00:59:16.1568728Z             status: "False"
2023-05-30T00:59:16.1571490Z             type: Ready
2023-05-30T00:59:16.1571854Z           - lastProbeTime: null
2023-05-30T00:59:16.1574788Z             lastTransitionTime: "2023-05-30T00:58:59Z"
2023-05-30T00:59:16.1575456Z             message: 'containers with unready status: [my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd]'
2023-05-30T00:59:16.1578850Z             reason: ContainersNotReady
2023-05-30T00:59:16.1579219Z             status: "False"
2023-05-30T00:59:16.1579519Z             type: ContainersReady
2023-05-30T00:59:16.1583714Z           - lastProbeTime: null
2023-05-30T00:59:16.1584099Z             lastTransitionTime: "2023-05-30T00:58:59Z"
2023-05-30T00:59:16.1587185Z             status: "True"
2023-05-30T00:59:16.1587549Z             type: PodScheduled
2023-05-30T00:59:16.1587813Z           containerStatuses:
2023-05-30T00:59:16.1591858Z           - image: registry.k8s.io/e2e-test-images/agnhost:2.43
2023-05-30T00:59:16.1592208Z             imageID: ""
2023-05-30T00:59:16.1595834Z             lastState: {}
2023-05-30T00:59:16.1596409Z             name: my-hostname-basic-844c69c5-d5ca-4dbb-8fce-5500c01348cd
2023-05-30T00:59:16.1599881Z             ready: false
2023-05-30T00:59:16.1600143Z             restartCount: 0
2023-05-30T00:59:16.1600428Z             started: false
2023-05-30T00:59:16.1604061Z             state:
2023-05-30T00:59:16.1604305Z               waiting:
2023-05-30T00:59:16.1612497Z                 reason: ContainerCreating
2023-05-30T00:59:16.1612922Z           hostIP: <IP_ADDRESS>
2023-05-30T00:59:16.1613170Z           phase: Pending
2023-05-30T00:59:16.1618342Z           qosClass: BestEffort
2023-05-30T00:59:16.1618819Z           startTime: "2023-05-30T00:58:59Z"
2023-05-30T00:59:16.1622934Z   In [It] at: test/e2e/apps/replica_set.go:228 @ 05/30/23 00:59:08.32

Hit the same failure on https://github.com/cilium/cilium/pull/26070: https://github.com/cilium/cilium/actions/runs/5252682413/jobs/9489036227?pr=26070
Logs: logs_1604977.zip
Sysdump: cilium-sysdumps.zip

Hit on https://github.com/cilium/cilium/pull/27825: https://github.com/cilium/cilium/actions/runs/6024954270/job/16344693626?pr=27825

Hit on https://github.com/cilium/cilium/actions/runs/6124686393/job/16627443044?pr=28021

Hit on https://github.com/cilium/cilium/actions/runs/6628718180/job/18006436680?pr=28763


docs: small improvment in backup

Just a small addition to help user during backup configuration.

@nunix do we need to mark somehow docs related PRs before merging so they do not get lost within the migration?


Query null values

Describe the solution you'd like
I would like to query a table for null values in a specific column

Hey any movement on this? Started using Azure storage tables and this is one of the most useful potential features for me to have access to.


Optimized Privilege Evaluation

Description
This implements the optimized privilege evaluation as described in #3870.
This introduces de-normalized data structures that are optimized for the checks that need to be done during privilege evaluation. Additionally, certain objects  (like DLS queries) are prepared ahead of time, as early as possible in order to minimize the overhead during actual privilege evaluation.
This is a big change set - in order to facilitate the review, I have split it into three major commits:

Optimized action privilege evaluation
Optimized DLS/FLS/FM privilege evaluation
Removal of unused code

The code is extensively commented - I hope that will help during review.

Category: Enhancement
Why these changes are required?

Performance tests indicate that the OpenSearch security layer adds a noticeable overhead to the indexing throughput of an OpenSearch cluster. The overhead may vary depending on the number of indices, the use of aliases, the number of roles and the size of the user object. The goal of these changes is to improve privilege evaluation performance and to make it less dependent on the number of indices, etc.

What is the old behavior before changes and new behavior after changes?

No significant behavioral changes in the "happy case", when privileges are present.
The undocumented config option config.dynamic.multi_rolespan_enabled is no longer evaluated. The code now behaves like it is always set to true  - that is the former default. See #4495 for details.
Some slight changes are present in error cases:

More detailed error messages for missing privileges, showing a index/action matrix of missing privileges
Errors in the role configuration might be reported (as error log messages) more early, directly after the configuration was applied
The DLS/FLS implementation now defaults to a "deny by default" implementation. This is not relevant for normal cases. This will be only relevant if index requests pass through privileges evaluator even though there are no roles which grant privileges to the requested indices. Note: This would only happen in case of a bug in the code. In the previous versions, the DLS/FLS implementation would grant full access to the indices. Now, the DLS/FLS implementation acts as a second barrier, denying access to the indices.

Issues Resolved

#3870

Testing

Each new component is accompanied by its own unit test.
The high-level functionality is validated by the existing integration tests
Mixed cluster behavior is verfied by SecurityBackwardsCompatibilityIT (extended in #4817 )

Check List

[x] New functionality includes testing
[x] New functionality has been documented
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Please have a look at the approaches. I would be very interested in your opinions.
As mentioned above, the implementation is not complete yet. The code contains a couple of TODO comments to indicate what work needs to be done.
I would also like to discuss whether a couple of things would be really necessary or whether there might be a chance to simplify the implementation by abolishing them.
These are:

At the moment, only RoleV7 and ActionGroupV7 configurations are supported by the class ActionPrivileges. I am wondering wether there are any plans to get rid of the *V6 configurations at some point in time. The figure V6 still stems from ODFE support for Elasticsearch 6. Are there really still OpenSearch users on this configuration? Additionally, the use of two impl classes per config type makes many unsafe casts of the generic SecurityDynamicConfiguration class necessary. If there would be only one impl class per config type, the APIs could be designed much safer.
In config.yml, the semantics of roles.yml can be changed by setting multi_rolespan_enabled to false. The OpenSearch docs do not mention this flag. In my perception, there is no real use of this setting except maintaining backwards compatiblity. However, for OpenSearch, the default of was always true since its inception. Are there really users having it set to false?

I have also started to work on the micro benchmarks as discussed in #3903. The generally accepted standard for micro benchmarks in Java is the JMH framework. However, this is licensed as GPL v2 with classpath exception: https://github.com/openjdk/jmh/blob/master/LICENSE Is the inclusion of a dependency with such a license acceptable in OpenSearch?

I have also started to work on the micro benchmarks as discussed in #3903. The generally accepted standard for micro benchmarks in Java is the JMH framework. However, this is licensed as GPL v2 with classpath exception: https://github.com/openjdk/jmh/blob/master/LICENSE Is the inclusion of a dependency with such a license acceptable in OpenSearch?

@cwperks Can you look into this?

@cwperks Can you look into this?

We're looking into this and will get back with an answer.

However, this is licensed as GPL v2 with classpath exception: https://github.com/openjdk/jmh/blob/master/LICENSE Is the inclusion of a dependency with such a license acceptable in OpenSearch?

The feedback we received was that the code can be used only for internal operations. Since JMH usage will be part of Open-source security, my understanding is that this is not approved.

@cwperks @peternied @DarshitChanpura
Just FYI:
I worked a bit on the micro benchmarking part of this issue. As JMH was out due to its license, I reviewed other frameworks. It is remarkable that in most cases the descriptions of the frameworks will say "rather use JMH instead of this framework".
Anyway, I tried out https://github.com/noconnor/JUnitPerf because the idea of using JUnit infrastructure seemed to be nice. The big downside of JUnitPerf is that it does not work well together with parameterized JUnit tests.
See here for an example:
https://github.com/opensearch-project/security/blob/004df3bbdc69514f0c95acd2a1653a01e71758b9/src/performanceTest/java/org/opensearch/security/privileges/PrivilegesEvaluatorPerformanceTest.java
The high number of very similar methods is caused by the lack of parameter support - in the end we need to test quite a few different dimensions (like number of indices, number of roles, etc), on the same operation.
As I was really keen on getting some broader result, I went on the "roll your own" path and quick threw together some naive micro benchmarking code. So, this is just a temporary thing, thus very messy, but it gives me some numbers. See here:
https://github.com/opensearch-project/security/blob/004df3bbdc69514f0c95acd2a1653a01e71758b9/src/performanceTest/java/org/opensearch/security/privileges/PrivilegesEvaluatorPeformanceTest2.java
So, I let run some tests and here are some preliminary results.
Micro benchmark test results
Disclaimer
Generally, the real world meaningfulness of micro benchmarks is limited. On a full real cluster, this can look totally different due to:

Proportion of effects to other time consuming operations
Effects caused by garbage collection or thread synchronization
Different hardware which can sustain constant CPU load much better than the consumer system I used to run the benchmarks

On the other hand, micro benchmarks make some tests so much easier. A Metadata instance with 100000 indices can be mocked within a few seconds, creating so many indices on a real cluster would take much, much longer.
Full cluster benchmarks are also coming up, but these are still in the works.
Scope
The micro benchmarks were applied to the following code:
https://github.com/opensearch-project/security/blob/004df3bbdc69514f0c95acd2a1653a01e71758b9/src/performanceTest/java/org/opensearch/security/privileges/PrivilegesEvaluatorPeformanceTest2.java#L501-L512
For comparison, we also applied the micro benchmarks to the following code on the old code base:
https://github.com/nibix/security/blob/300d138578ef853071d649d647335d8430320f14/src/performanceTest/java/org/opensearch/security/privileges/PrivilegesEvaluatorPeformanceTest2.java#L502-L510
Due to refactorings, the code looks different. However, the what happens under the hood is effectively the same.
Additionally some further code changes were necessary to make PrivilegeEvaluator independent from dependencies like ClusterService in order to make it really unit testable/benchmarkable. I first tried to use Mockito to mock ClusterService instances but had to learn that the performance characteristics of Mockito are so bad that it is unsuitable for micro benchmarking.
As we only look at the evaluate() method, DLS/FLS evaluation is disabled for this scope.
Tested dimensions
Action requests
We tested privilege evaluation with three different actions:

indices:data/write/bulk[s] with BulkShardRequest

with 10 bulk items
with 1000 bulk items

indices:data/write/bulk with BulkRequest
indices:data/read/search with SearchRequest

with an index pattern that matches 2% of all indices (randomized)
with an index pattern that matches 20% of all indices (randomized)

Number of indices on cluster
We tested with these indices:

10 indices:index_a0,  index_a1, index_b0, index_b1, index_c0, index_c1, ... , index_e0, index_e1
30 indices: index_a0, ..., index_a5, ... , index_e0, ... index_e5
100 indices: index_a0, ..., index_a19, ... , index_e0, ... index_e19
300 indices
1000 indices
3000 indices
10000 indices
30000 indices
100000 indices

Different user configurations

A user with full privileges (using * for index_permissions and cluster_permssions)
A user with a single role giving CRUD permissions on index_a* and index_b*
A user with 20 roles giving CRUD permissions individually on index_a0, index_a1, ...
A user with 40 roles in total: 20 roles giving READ permissions individually on index_a0, index_a1, ... and 20 more roles giving WRITE permissions on the same indices
A user with a single role which uses a regex index pattern with a user attribute. This is interesting because it makes certain pre-computations impossible and requires to re-evaluate the index patterns for each request.

Results
The raw result data can be found here: https://docs.google.com/spreadsheets/d/1Hd6pZFICTeplXIun3UpEplANAwQDE0jXbqLnnJz61AI/edit?usp=sharing
In the shards below, dashed lines indicate the performance of the old privilege evaluation code on a particular combination of test dimensions. Solid lines with the same color indicate the performance of the new code with the same test dimensions. The x-axis represents the number of indices on the cluster, the y-axis represents the throughput in operations per second.
bulk[s], BulkShardRequest
The performance of BulkShardRequests is the most interesting factor on clusters doing heavy ingestion. A single bulk requests will be broken down into the individual indices and shards, resulting in quite a few  BulkShardRequests for which the privilege evaluation needs to be done in parallel, thus performance characteristics here have a high impact.
The privilege evaluation for the top level BulkRequest is less interesting because it is just an index-independent cluster privilege, which is easy to evaluate. Still, we will also review this below.
Requests with 10 items

Requests with 1000 items

Observation
The performance of the old code degrades with the increasing number of indices. Starting with 30000 indices, we have a method call latency which is > 10 ms. This is where users on ingestion heavy clusters often start to experience performance issues and the method calls start to show up in the hot thread dumps.
In contrast, the throughput of the new code stays constant, independent of the number of indices. It can be seen that the number of roles still has quite an effect on the throughput. But here we walk about time differences below 0.1 ms, which should not be significant in a real world cluster.
bulk, BulkRequest
The top level bulk action is a cluster action, so it does not require considering the indices on a cluster.

Observation
As expected, performance is independent of number of indices, both on the new implementation and on the old implementation. However, the new implementation improves throughput by a factor between 2 and 3.
search, SearchRequest
Search operations become interesting when there are monitoring/alerting solutions issuing search requests on broad index patterns in short time intervals.
Search with search patterns that match 2% of the indices

Search with search patterns that match 20% of the indices

Observation
Both the old and new code degrade with the growing number of indices. Profiling shows that this is mostly not due to privilege evaluation, but due to the index pattern expression resolution.
However, the new code retains method call latencies below 20 ms even on clusters with 100000 indices. The old code however, takes up to 5 seconds for a single method call on clusters with 100000 indices.

@nibix thank you for the update. The results of bulk indexing in a cluster with a large number of indices is great to see and I like how the test cases isolate PrivilegeEvaluation for performance testing. Is there any work remaining before labeling this PR ready for review?

@cwperks

Is there any work remaining before labeling this PR ready for review?

Yes. Actually, I was also about to ping you regarding this.

The heap used by the de-normalized data structure needs to be capped (see https://github.com/opensearch-project/security/issues/3870#issuecomment-2222509679 )
We need to come to a conclusion on the breaking changes (see https://github.com/opensearch-project/security/issues/4493 and https://github.com/opensearch-project/security/issues/4495 ). At the moment, this only supports RoleV7. Also, multi_rolespan_enabled: false is unsupported. Of course it would be nice to have the benefits already available for OpenSearch 2. But that would mean either additional effort or the acceptance of some breaking changes. I have sketched a way to retire RoleV6 without breaking changes in https://github.com/opensearch-project/security/pull/4546 . However, I am at the moment not sure how a solution could look like that supports multi_rolespan_enabled: false in the new code.
We are also working on actual full cluster benchmarks, which give you concrete figures on the actual throughput improvements.

Note: This leaves the DLS/FLS implementation unchanged at the moment. Thus, we will still have a part of the problematic performance characteristics. However, I would still get this merged and then add DLS/FLS support in a further PR to keep things (relatively) small.

Benchmark results
After having shared the micro benchmarks before, I can now share results of benchmarks performed on an actual cluster. This brings the following benefits:

It puts the performance gains into perspective of the performance characteristics of the whole software. This way it gets apparent whether the performance gains actually have significance or if they just "vanish" behind other dominating performance characteristics.
We can also test the performance gains for DLS. With micro benchmarks, this is not possible as DLS is not implemented as an isolated piece of code, but is performed distributed over the cluster during a search operation.

Disclaimer
The usual benchmarking disclaimer: Of course, these figures can only represent very specific scenarios. Other scenarios can look quite different, as there are so many variables which can be different. Yet, especially the "slow parts" of the benchmark results can give one an impression where real performance issues are.
Test environment
We ran the tests on Kubernetes. The host machine for the K8S nodes was a server with an AMD EPYC 7401P CPU. We ran a OpenSearch cluster with 3 nodes, each node had 64 GB of RAM, 32 GB of Java Heap, and a cpu: 8 limit configured in K8S. The version of OpenSearch was a recent snapshot of main.
Tested dimensions
Operations
We benchmarked the following operations:

Ingestion with the REST _bulk API into a single index with 8 parallel clients

with 10 bulk items
with 1000 bulk items

Search requests with the REST _search API with 16 parallel clients

on exactly one index
on an index expression matching 2% of the indices on the cluster
on an index expression matching 20% of the indices on the cluster

Number of indices on cluster
We tested with these indices:

10 indices: index_a0, index_a1, index_b0, index_b1, index_c0, index_c1, ... , index_e0, index_e1
30 indices: index_a0, ..., index_a5, ... , index_e0, ... index_e5
100 indices: index_a0, ..., index_a19, ... , index_e0, ... index_e19
300 indices
1000 indices
3000 indices
10000 indices

User configurations
We tested with differently configured users to find out about the effects of complexity of roles, DLS rules, etc. The users were:

A user identified by the super admin certificate. Using this certificate, most of the privilege evaluation code is by-passed. This gives us a rough "upper limit" for the possible performance.
A user with full privileges (using * for index_permissions and cluster_permssions)
A user with a single role giving CRUD permissions on index_a* and index_b*
A user with 20 roles giving CRUD permissions individually on index_a0, index_a1, ...
A user with 40 roles in total: 20 roles giving READ permissions individually on index_a0, index_a1, ... and 20 more roles giving WRITE permissions on the same indices
A user with a single role giving CRUD permissions on index_a* and index_b* and additionally a simple DLS query

Test results
A commented summary of  the results follows in the upcoming sections.
The raw test results can be also reviewed at https://docs.google.com/spreadsheets/d/16VRr9B2bPTyyS_T-IZobUG3C0uJxnXRxlbmsdH_MeCg/edit?usp=sharing
Indexing throughput
Bulk size 10

In this and the following charts, the dashed lines represent OpenSearch with the standard security plugin. The solid lines represent OpenSearch with the security plugin with the optimized privilege evaluation code.
The blue line represents requests authenticated by the super admin certificate, which by-passes most of privilege evaluation. Thus, the blue line forms a kind of "hull curve", it can be seen as a rough theoretical maximum from a security plugin POV
The green lines represent users with full privileges. Yellow lines represent users with limited privileges - the darker the yellow, the more complex the role configuration.
The benchmark results for the standard security plugin show a clear performance decline starting at about 300 indices on the cluster. This is caused by the privilege evaluation code resolving the index patterns in the roles configuration against all cluster indices for each request.  At 1000 indices, we only get roughly 60% of the throughput that was observed for 10 indices.
Additionally, it can be seen that growing role complexity has a clear effect on throughput. More complex roles show a significant lower throughput.
On the other hand, the optimized security plugin shows a mostly linear throughput, which is independent of the number of indices. There is a slight decline starting from 3000 indices. The reasons for this are unknown so far. Yet, these values are still well above the standard plugin.
Bulk size 1000

Larger bulk sizes shift the place where the gap between standard and optimized plugin performance opens a bit to the right. Here we see a strong performance decline in the standard security plugin starting from 1000 indices.
The optimized security plugin exhibits better performance in all cases, though. Additionally, the performance decline that was visible for the bulk: 10 case is not really visible here any more.
The charts show a low throughput for the full privileges user both for the standard plugin and for the optimized plugin in the indices: 10 case. However, I would guess that this is some artifact of the used benchmarking process and is not a real performance characteristic.
Search throughput
Search on single index

The search throughput graphs introduce one further user: Users symbolized by the purple line are users which do have a document level access restriction implemented with a DLS query in one role.
Like for the bulk indexing, the standard plugin shows a declining performance with increasing number of indices on the clusters. Also here, the complexity of the roles configuration has a very significant effect on the throughput. Especially the user with DLS exhibts heavy performance degradations. With 300 indices, the DLS user shows less than half the throughput of the full privileges user of the standard plugin.
On the other hand, the optimized plugin shows mostly constant performance characteristics, independent of the number of indices. Also the DLS user does not show any significant degradation of performance.
Search on 2% of indices

When an index pattern comes into play, both the standard plugin and the optimized plugin show a performance degradation with a growing number of indices. This is due to the necessary resolution of the index pattern against all indices on the cluster.
The blue line - the super admin user - shows that there is quite a gap (about 20%, growing with higher number of indices) between the theoretically possible throughput and also the optimized plugin. This is likely due to the code in https://github.com/nibix/security/blob/main/src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java which still leaves some room for improvement.
Still, the optimized plugin delivers a clearly higher throughput in all cases.
Especially the DLS performance has strongly improved. For 1000 indices, we see a throughput of 1035 operations per second on the optimized plugin. The standard plugin just manages  53 operations per second with a service time of 300 ms per request. With 10000 indices, DLS gets virtually unusable on the standard plugin with just 0.6 operations per second and a service time of 16 seconds. The optimized plugin still delivers 99 operations per second.
Search on 20% of indices

With a search operation on 20% of indices, the index resolution performance gets so dominant that real performance gains are no longer visible - except for DLS, which still shows very strong improvements.
The blue line - the admin cert user - shows that there is still room for improvement, though.

@nibix The results are very promising from the benchmarks. Should the index patten resolver be optimized as follow-up?

Side note: This is not the end of the story - there's still significant potential for performance improvements. I will file issues about that soon.

Note: The patch coverage is only 84.5% which is a bit low for my taste: https://github.com/opensearch-project/security/pull/4380#issuecomment-2197145658
However, this low figure is mostly due to indentation changes in DLS/FLS classes due to newly necessary exception handlers. Thus, the low coverage of these classes leaks into this patch.
I have a couple of ideas on how to improve coverage for these classes, but I guess that this is outside of the scope of this PR.

@nibix Can you rebase with the latest from main? I don't have any further comments on this PR. IMO I'd like to get this PR merged to main in the near future for a couple of reasons: 1) A 3.0.0 snapshot can be produced where we can perform additional testing and 2) allow other PRs to be rebased on top of this PR.
An additional review can/will be done for this change when backported to 2.x.

@cwperks I just pushed some final fixed and a rebased branch

For, I have also resolved this comment about setting the correct release version of this feature: https://github.com/opensearch-project/security/pull/4380#pullrequestreview-2374642450
However, we have to keep in mind to adapt this version in the backport and in the main branch as soon as the release version is known.

@reta @willyborankin would love to have your inputs on this as well on this major enhancement by @nibix

FYI: I believe that I just stumbled over one issue related to date math. I will try to fix it over the course of the day.

@reta @willyborankin would love to have your inputs on this as well on this major enhancement by @nibix

Apologies, struggling to keep up with reviews, I think you have sufficient amount of eyes already, @peternied I think only your approval is pending

Please don't wait on my approval, I'm thick of another release.  It looks like we've got 3 maintainers approvals, IMO that seems like more than enough of a reason to merge.  If there is a specific question or thread that you'd like me to follow up on I can take a targeted look.
What do you think @nibix?

@peternied

What do you think @nibix?

For me personally, that's fine.

@nibix Can you open a backport to 2.x? I'd add the backport label, but I'm fairly confident that this will require a manual backport ;)

Backport in #4898


More specific error messages & allow spaces between tags

Fixes #194, fixes #180, fixes #169, fixes #144, fixes #66

If command word is invalid it will throw unknown command error with the wrong command, even if type is wrong too

If command type is invalid it will throw unknown type instead

allow spaces between tag names (maybe need to mention in UG)

Codecov Report

Merging #200 (e3322cd) into master (713d7ed) will decrease coverage by 0.18%.
The diff coverage is 63.15%.

@@             Coverage Diff              @@
##             master     #200      +/-   ##
============================================
- Coverage     71.69%   71.51%   -0.19%
- Complexity      742      744       +2
============================================
  Files           124      124
  Lines          2420     2436      +16
  Branches        303      306       +3
============================================
+ Hits           1735     1742       +7
- Misses          599      604       +5
- Partials         86       90       +4

Impacted Files
Coverage Δ

src/main/java/fridgy/commons/core/Messages.java
0.00% <ø> (ø)

...main/java/fridgy/logic/commands/DeleteCommand.java
100.00% <ø> (ø)

...c/main/java/fridgy/logic/commands/EditCommand.java
95.77% <ø> (ø)

...c/main/java/fridgy/logic/commands/ListCommand.java
100.00% <ø> (ø)

...dgy/logic/commands/recipe/DeleteRecipeCommand.java
100.00% <ø> (ø)

...rc/main/java/fridgy/model/ingredient/Quantity.java
85.71% <ø> (ø)

src/main/java/fridgy/model/tag/Tag.java
76.92% <ø> (ø)

src/main/java/fridgy/ui/IngredientListPanel.java
0.00% <0.00%> (ø)

src/main/java/fridgy/ui/RecipeListPanel.java
0.00% <0.00%> (ø)

...rc/main/java/fridgy/logic/parser/FridgyParser.java
76.19% <70.58%> (-19.97%)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 713d7ed...e3322cd. Read the comment docs.


Upload failure in posix backend when filesystem is cifs.

The bug
Cannot upload files to versitygw when using the posix backend on cifs mounts.
To Reproduce
Setup:
$ versitygw posix /mnt/cifs_mount
No IAM service configured, enabling single account mode
 ┌───────────────────────────────────────────────────┐
 │                     versitygw                     │
 │                   Fiber v2.51.0                   │
 │               http://<IP_ADDRESS>:7070               │
 │       (bound on host <IP_ADDRESS> and port 7070)       │
 │                                                   │
 │ Handlers ............ 25  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............. 12345 │
 └───────────────────────────────────────────────────┘

Client:
$ aws --endpoint=http://<IP_ADDRESS>:7070 s3 mb s3://bucket
make_bucket: bucket
$ aws --endpoint=http://<IP_ADDRESS>:7070 s3 cp testfile s3://bucket/testfile
upload failed: ./upload.txt to s3://signage/upload.txt An error occurred (ExistingObjectIsDirectory) when calling the PutObject operation: Existing Object is a directory.
$ ls /mnt/cifs_mount/bucket/
drwxr-xr-x 2 root root 0 Jan  1 00:00 .
drwxr-xr-x 2 root root 0 Jan  1 00:00 ..
drwxr-xr-x 2 root root 0 Jan  1 00:00 .sgwtmp

Server output:
00:00:00 | 200 |   19.209157ms |       <IP_ADDRESS> | PUT     | /bucket
00:00:01 | 409 |  111.251961ms |       <IP_ADDRESS> | PUT     | /bucket/testfile

Expected behavior
Client:
$ aws --endpoint=http://<IP_ADDRESS>:7070 s3 cp testfile s3://bucket/testfile
upload: ./testfile to s3://bucket/testfile
$ ls /mnt/cifs_mount/bucket/
drwxr-xr-x 3 root root 4.0K Jan  5 00:00 .
drwxr-xr-x 3 root root 4.0K Jan  2 00:00 ..
drwx------ 2 root root 4.0K Jan  2 00:00 .sgwtmp
-rw------- 1 root root    8 Feb  2 00:00 testfile

Server output:
00:00:00 | 200 |   19.209157ms |       <IP_ADDRESS> | PUT     | /bucket
00:00:01 | 200 |  111.251961ms |       <IP_ADDRESS> | PUT     | /bucket/testfile

Server Version
Version  : 0.11
Build    : 8b31d6d93c76ef5d2d563047518c1e91b28d38c0
BuildTime: 2023-12-30T05:41:01Z
Additional context
Debian 11.7
cifs-utils 2:6.11-3.1+deb11u1
awscli     1.19.1-1
CIFS share from Windows Server 2016

@rhamon I just tested against a redhat samba server mounted with cifs on a redhat client, and this looks to work ok. I can try to setup the debian system to reproduce, but I wanted to check the cifs mount options to see if maybe my tests are different somehow. Can you check to see if there are any options set from mount | grep cifs_mount that might be relevant to these tests?
The error "ExistingObjectIsDirectory" generally means that you are trying to upload an object, but there is already a directory with this name that exists server side. I'm not seeing this in the ls output, so likely is some gateway bug but I wanted to double check that /mnt/cifs_mount/bucket/testfile is not showing up as a directory at any time during this test.

Closing this as we are not able to reproduce.  Can open again if we have more specifics about whats needed to reproduce this.

I haven't tested with a Samba share on Redhat but I would assume Samba is different in features to a Microsoft Windows Server.
My test was done mouting a MS Windows Server 2016 shared folder to a Debian 11 server running Versity 0.11.
I can test the latest version of Versity on a Redhat type OS instead of Debian but the share has to come from a MS Windows Server for my use case.

The mount looked like this during the initial report:
//Win2k16/share/ on /mnt/cifs_mount type cifs (rw,relatime,vers=3.0,cache=strict,uid=0,noforceuid,gid=0,noforcegid,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=0)

I'm also not able to reproduce the initial issue with the latest version of versitygw.
Thanks and we can keep this closed.
Cheers


Add Black Screen Fix

I've had someone do this on the Discord, and realized that it was pretty common. Some people don't have file extensions enabled.
This just teaches them how to.

Lgtm!


update-to-head.sh works with openshift-knative/client

https://issues.redhat.com/browse/SRVCLI-358
These changes are required for the nightly sync job in jenkins to work on release-next branch.
Similar changes were done for Serving and Eventing. link

/hold

/unhold

/lgtm


fix the backslash to forward slash

@wely : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@malopMSFT, you can only sign off in the main PR conversation, not in review comments.

#sign-off


Get-UnitySetupInstance can't find some of installed unity versions on mac

I have installed all versions of unity in my mac via UnitySetup powershell module but it cant find some of them when using Get-UnitySetupInstance. I am using unity setup module version of 5.3.134.
All unity versions are located in same directory which is default unity hub install location. For example unity version 2019.4.0f1 is installed but it isn't listed when calling Get-UnitySetupInstance.

I'm experiencing this with 2020.1.0f1 installed via this package.

In my case, it's the inability of the system to determine the version number that appears to be the problem. Is that what's happening to you?

Not sure if this is a seperate bug but might be related. On windows after I installed a 2019 version I started getting the following after Get-UnitySetupInstance and Start-UnityEditor
'Unable to find version for C:\Program Files\Unity\Hub\Editor\2019.4.14f1'

Not sure if this is a seperate bug but might be related. On windows after I installed a 2019 version I started getting the following after Get-UnitySetupInstance and Start-UnityEditor
'Unable to find version for C:\Program Files\Unity\Hub\Editor\2019.4.14f1'

Not sure if this is a seperate bug but might be related. On windows after I installed a 2019 version I started getting the following after Get-UnitySetupInstance and Start-UnityEditor
'Unable to find version for C:\Program Files\Unity\Hub\Editor\2019.4.14f1'

I was able to resolve this by updating UnitySetup, duh.

Not sure if this is a seperate bug but might be related. On windows after I installed a 2019 version I started getting the following after Get-UnitySetupInstance and Start-UnityEditor
'Unable to find version for C:\Program Files\Unity\Hub\Editor\2019.4.14f1'

I was able to resolve this by updating UnitySetup, duh.

woops, did not mean to close the whole issue

woops, did not mean to close the whole issue

I got this on a new build-server with 2021.3.3f1. The old build-server (also installed from these scripts) does not have this issue. Both servers have the same version of the scripts (5.4.130). Any info I can provide you to help out?