Can't create VMs in openshift CI using GCP

/kind bug
What happened:
All HCO GCP tests are failing. These tests are running in openshift-ci on gcp cluster. The tests fail to create VMs.
The errors in virt-handler look like this:
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"testvmi-svqq5","namespace":"kubevirt-test-default1","pos":"vm.go:1435","reason":"failed to configure vmi network: failed to detect isolation for launcher pod: dial unix //pods/acbcc68e-eb17-4c1d-8f30-0537e7cd0cae/volumes/kubernetes.io~empty-dir/sockets/launcher-sock: connect: connection refused","timestamp":"2021-03-29T08:45:54.782956Z","uid":"9d4a231a-dc50-4d86-9668-9cf1ee189bd9"}

There is no direct access to the cluster, and there are no virt-launcher logs.
There are many examples of these failures because all GCP tests are failing. Here is one: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1242/pull-ci-kubevirt-hyperconverged-cluster-operator-master-hco-e2e-image-index-gcp/1381618709780500480

on the journal on the node we see in a loop:
Apr 12 15:57:35.069093 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:35.069048940Z" level=info msg="Creating container: kubevirt-test-default/virt-launcher-testvmi-vdhcrlxpj7j5-7998g/compute" id=a5eb73f9-d28f-4ec7-8cb3-6d0c18746f67 name=/runtime.v1alpha2.RuntimeService/CreateContainer
Apr 12 15:57:35.108143 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: Started crio-conmon-a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.scope.
Apr 12 15:57:35.140354 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: Started libcontainer container a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.
Apr 12 15:57:35.236023 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:35.235958687Z" level=info msg="Created container a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e: kubevirt-test-default/virt-launcher-testvmi-vdhcrlxpj7j5-7998g/compute" id=a5eb73f9-d28f-4ec7-8cb3-6d0c18746f67 name=/runtime.v1alpha2.RuntimeService/CreateContainer
Apr 12 15:57:35.236631 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:35.236591150Z" level=info msg="Starting container: a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e" id=421fd635-6a54-44a0-ac8d-04f876f3e8e3 name=/runtime.v1alpha2.RuntimeService/StartContainer
Apr 12 15:57:35.252004 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:35.251939504Z" level=info msg="Started container a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e: kubevirt-test-default/virt-launcher-testvmi-vdhcrlxpj7j5-7998g/compute" id=421fd635-6a54-44a0-ac8d-04f876f3e8e3 name=/runtime.v1alpha2.RuntimeService/StartContainer
Apr 12 15:57:35.355594 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:35.354227    1509 kubelet.go:1958] SyncLoop (PLEG): "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)", event: &pleg.PodLifecycleEvent{ID:"4cda71d8-0f37-4d49-8f6d-7081bcc21347", Type:"ContainerStarted", Data:"a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e"}
Apr 12 15:57:36.442169 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Apr 12 15:57:36.455323 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 1(k6t-eth0-nic) entered blocking state
Apr 12 15:57:36.455497 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 1(k6t-eth0-nic) entered disabled state
Apr 12 15:57:36.455536 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: device k6t-eth0-nic entered promiscuous mode
Apr 12 15:57:36.473980 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 1(k6t-eth0-nic) entered blocking state
Apr 12 15:57:36.474161 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 1(k6t-eth0-nic) entered forwarding state
Apr 12 15:57:36.474204 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_UP): k6t-eth0: link is not ready
Apr 12 15:57:36.509224 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 2(tap0) entered blocking state
Apr 12 15:57:36.510318 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 2(tap0) entered disabled state
Apr 12 15:57:36.510354 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: device tap0 entered promiscuous mode
Apr 12 15:57:36.520560 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_UP): tap0: link is not ready
Apr 12 15:57:36.520679 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 2(tap0) entered blocking state
Apr 12 15:57:36.528006 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 2(tap0) entered forwarding state
Apr 12 15:57:36.624771 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.scope: Succeeded.
Apr 12 15:57:36.625476 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.scope: Consumed 486ms CPU time
Apr 12 15:57:36.636819 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-conmon-a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.scope: Succeeded.
Apr 12 15:57:36.637546 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-conmon-a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e.scope: Consumed 50ms CPU time
Apr 12 15:57:37.363382 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.363318    1509 kubelet.go:1958] SyncLoop (PLEG): "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)", event: &pleg.PodLifecycleEvent{ID:"4cda71d8-0f37-4d49-8f6d-7081bcc21347", Type:"ContainerDied", Data:"a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e"}
Apr 12 15:57:37.364009 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.363953702Z" level=info msg="Stopping pod sandbox: 2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1" id=8aa68499-5b5a-402d-9666-cc9aa40838ee name=/runtime.v1alpha2.RuntimeService/StopPodSandbox
Apr 12 15:57:37.364489 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.364412339Z" level=warning msg="Unknown CNI cache file /var/lib/cni/results/openshift-sdn-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1-eth0 kind \"\""
Apr 12 15:57:37.364980 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.364927340Z" level=info msg="Got pod network &{Name:virt-launcher-testvmi-vdhcrlxpj7j5-7998g Namespace:kubevirt-test-default ID:2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1 NetNS:/var/run/netns/e0b8bc65-30dc-4d17-8b4a-6f7cea1887a2 Networks:[{Name:multus-cni-network Ifname:eth0}] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
Apr 12 15:57:37.365198 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.365159882Z" level=info msg="About to del CNI network multus-cni-network (type=multus)"
Apr 12 15:57:37.392221 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.392120    1509 reflector.go:225] Stopping reflector *v1.Secret (0s) from object-"kubevirt-test-default"/"default-dockercfg-c469z"
Apr 12 15:57:37.436099 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.436060    1509 kubelet_pods.go:1270] Killing unwanted pod "virt-launcher-testvmi-vdhcrlxpj7j5-7998g"
Apr 12 15:57:37.436932 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.436887    1509 kuberuntime_container.go:650] Killing container "cri-o://a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e" with a 30 second grace period
Apr 12 15:57:37.437229 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.437112637Z" level=info msg="Stopping container: a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e (timeout: 30s)" id=9f6e2972-e267-44f3-9d8f-262dc1837483 name=/runtime.v1alpha2.RuntimeService/StopContainer
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447107    1509 reconciler.go:196] operationExecutor.UnmountVolume started for volume "container-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-container-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347")
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447216    1509 reconciler.go:196] operationExecutor.UnmountVolume started for volume "hotplug-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-hotplug-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347")
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447253    1509 reconciler.go:196] operationExecutor.UnmountVolume started for volume "ephemeral-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-ephemeral-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347")
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447282    1509 reconciler.go:196] operationExecutor.UnmountVolume started for volume "sockets" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-sockets") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347")
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447334    1509 reconciler.go:196] operationExecutor.UnmountVolume started for volume "libvirt-runtime" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-libvirt-runtime") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347")
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:37.447552    1509 empty_dir.go:520] Warning: Failed to clear quota on /var/lib/kubelet/pods/4cda71d8-0f37-4d49-8f6d-7081bcc21347/volumes/kubernetes.io~empty-dir/hotplug-disks: clearQuota called, but quotas disabled
Apr 12 15:57:37.448044 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:37.447552    1509 empty_dir.go:520] Warning: Failed to clear quota on /var/lib/kubelet/pods/4cda71d8-0f37-4d49-8f6d-7081bcc21347/volumes/kubernetes.io~empty-dir/container-disks: clearQuota called, but quotas disabled
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447614    1509 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-hotplug-disks" (OuterVolumeSpecName: "hotplug-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347"). InnerVolumeSpecName "hotplug-disks". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447642    1509 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-container-disks" (OuterVolumeSpecName: "container-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347"). InnerVolumeSpecName "container-disks". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:37.447667    1509 empty_dir.go:520] Warning: Failed to clear quota on /var/lib/kubelet/pods/4cda71d8-0f37-4d49-8f6d-7081bcc21347/volumes/kubernetes.io~empty-dir/libvirt-runtime: clearQuota called, but quotas disabled
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:37.447785    1509 empty_dir.go:520] Warning: Failed to clear quota on /var/lib/kubelet/pods/4cda71d8-0f37-4d49-8f6d-7081bcc21347/volumes/kubernetes.io~empty-dir/ephemeral-disks: clearQuota called, but quotas disabled
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:37.447803    1509 empty_dir.go:520] Warning: Failed to clear quota on /var/lib/kubelet/pods/4cda71d8-0f37-4d49-8f6d-7081bcc21347/volumes/kubernetes.io~empty-dir/sockets: clearQuota called, but quotas disabled
Apr 12 15:57:37.448424 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.447990    1509 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-sockets" (OuterVolumeSpecName: "sockets") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347"). InnerVolumeSpecName "sockets". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Apr 12 15:57:37.448609 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.448055    1509 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-ephemeral-disks" (OuterVolumeSpecName: "ephemeral-disks") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347"). InnerVolumeSpecName "ephemeral-disks". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Apr 12 15:57:37.448609 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.448355    1509 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-libvirt-runtime" (OuterVolumeSpecName: "libvirt-runtime") pod "4cda71d8-0f37-4d49-8f6d-7081bcc21347" (UID: "4cda71d8-0f37-4d49-8f6d-7081bcc21347"). InnerVolumeSpecName "libvirt-runtime". PluginName "kubernetes.io/empty-dir", VolumeGidValue ""
Apr 12 15:57:37.508215 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: device veth16160e94 left promiscuous mode
Apr 12 15:57:37.521665 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: 2021-04-12T15:57:37Z [verbose] Del: kubevirt-test-default:virt-launcher-testvmi-vdhcrlxpj7j5-7998g:4cda71d8-0f37-4d49-8f6d-7081bcc21347:openshift-sdn:eth0 {"cniVersion":"0.3.1","name":"openshift-sdn","type":"openshift-sdn"}
Apr 12 15:57:37.532446 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): k6t-eth0: link becomes ready
Apr 12 15:57:37.538677 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.536356623Z" level=info msg="Stopped container a64e0abfaef3771cf00ea3e2ff021f026ff62190df56b3752aaae1ee89bca49e: kubevirt-test-default/virt-launcher-testvmi-vdhcrlxpj7j5-7998g/compute" id=9f6e2972-e267-44f3-9d8f-262dc1837483 name=/runtime.v1alpha2.RuntimeService/StopContainer
Apr 12 15:57:37.538677 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:37.537217944Z" level=info msg="Stopping pod sandbox: 2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1" id=344aafa3-8c1e-416a-b2cb-d7a23b46b975 name=/runtime.v1alpha2.RuntimeService/StopPodSandbox
Apr 12 15:57:37.539196 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: k6t-eth0: port 2(tap0) entered disabled state
Apr 12 15:57:37.548796 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.548748    1509 reconciler.go:319] Volume detached for volume "libvirt-runtime" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-libvirt-runtime") on node "ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk" DevicePath ""
Apr 12 15:57:37.548796 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.548798    1509 reconciler.go:319] Volume detached for volume "container-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-container-disks") on node "ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk" DevicePath ""
Apr 12 15:57:37.548796 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.548812    1509 reconciler.go:319] Volume detached for volume "hotplug-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-hotplug-disks") on node "ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk" DevicePath ""
Apr 12 15:57:37.549055 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.548826    1509 reconciler.go:319] Volume detached for volume "ephemeral-disks" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-ephemeral-disks") on node "ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk" DevicePath ""
Apr 12 15:57:37.549055 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:37.548840    1509 reconciler.go:319] Volume detached for volume "sockets" (UniqueName: "kubernetes.io/empty-dir/4cda71d8-0f37-4d49-8f6d-7081bcc21347-sockets") on node "ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk" DevicePath ""
Apr 12 15:57:37.551814 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1.scope: Succeeded.
Apr 12 15:57:37.552633 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1.scope: Consumed 313ms CPU time
Apr 12 15:57:37.563307 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-conmon-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1.scope: Succeeded.
Apr 12 15:57:37.564038 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: crio-conmon-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1.scope: Consumed 47ms CPU time
Apr 12 15:57:37.600912 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: run-runc-b537d5e3ddbc7be55adf9d4644715f2f24ea696ceb773f8c48cc465d2af03858-runc.o3nfHU.mount: Succeeded.
Apr 12 15:57:37.713099 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: run-containers-storage-overlay\x2dcontainers-2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1-userdata-shm.mount: Succeeded.
Apr 12 15:57:37.719290 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: var-lib-containers-storage-overlay-f344b4babdf169bb2c9e81ba53f777b5b6be10cb45da12c7d5d7daf557304326-merged.mount: Succeeded.
Apr 12 15:57:38.377049 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 15:57:38.376943    1509 kubelet.go:1958] SyncLoop (PLEG): "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)", event: &pleg.PodLifecycleEvent{ID:"4cda71d8-0f37-4d49-8f6d-7081bcc21347", Type:"ContainerDied", Data:"2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1"}
Apr 12 15:57:38.377580 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: W0412 15:57:38.377112    1509 pod_container_deletor.go:79] Container "2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1" not found in pod's containers
Apr 12 15:57:39.703921 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:39.703858900Z" level=info msg="Stopped pod sandbox: 2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1" id=8aa68499-5b5a-402d-9666-cc9aa40838ee name=/runtime.v1alpha2.RuntimeService/StopPodSandbox
Apr 12 15:57:39.704614 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 15:57:39.704051658Z" level=info msg="Stopped pod sandbox (already stopped): 2da9c1906a58a682c161361e794c744bebec5def2dbfec6cd0fa58e08915adb1" id=344aafa3-8c1e-416a-b2cb-d7a23b46b975 name=/runtime.v1alpha2.RuntimeService/StopPodSandbox
Apr 12 15:57:41.442597 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: Removed slice libcontainer container kubepods-burstable-pod4cda71d8_0f37_4d49_8f6d_7081bcc21347.slice.
Apr 12 15:57:41.442653 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: kubepods-burstable-pod4cda71d8_0f37_4d49_8f6d_7081bcc21347.slice: Consumed 898ms CPU time
Apr 12 16:00:59.516276 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk bash[1294]: Processing route for NIC 0/42:01:0a:00:80:04 for <IP_ADDRESS>
Apr 12 16:00:59.518797 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk bash[1294]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Apr 12 16:00:59.520355 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk bash[1294]: ensuring rule for <IP_ADDRESS>
Apr 12 16:00:59.522333 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk bash[1294]: done applying vip rules
Apr 12 16:02:34.270829 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 16:02:34.270702410Z" level=info msg="Checking image status: registry.build01.ci.openshift.org/ci-op-ji3f5bsp/stable@sha256:fe086828bcc352853f5c4312bf8ad11ea34488b6ea732fd2961a82724a2223a6" id=7f0b4058-109c-4c93-8e9f-523e2292ae51 name=/runtime.v1alpha2.ImageService/ImageStatus
Apr 12 16:02:34.275151 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 16:02:34.272421359Z" level=info msg="Image status: &{0xc0003a19d0 map[]}" id=7f0b4058-109c-4c93-8e9f-523e2292ae51 name=/runtime.v1alpha2.ImageService/ImageStatus
Apr 12 16:03:14.226678 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.226625    1509 kubelet.go:1936] SyncLoop (DELETE, "api"): "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)"
Apr 12 16:03:14.239298 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.239249    1509 kubelet.go:1930] SyncLoop (REMOVE, "api"): "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)"
Apr 12 16:03:14.239298 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.239305    1509 kubelet.go:2120] Failed to delete pod "virt-launcher-testvmi-vdhcrlxpj7j5-7998g_kubevirt-test-default(4cda71d8-0f37-4d49-8f6d-7081bcc21347)", err: pod not found
Apr 12 16:03:14.415391 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.415340    1509 kubelet.go:1920] SyncLoop (ADD, "api"): "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp_kubevirt-test-default(94893f9a-73fe-4ac8-80f7-88fb73a58ed4)"
Apr 12 16:03:14.415587 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.415454    1509 topology_manager.go:187] [topologymanager] Topology Admit Handler
Apr 12 16:03:14.415587 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.415529    1509 reflector.go:219] Starting reflector *v1.Secret (0s) from object-"kubevirt-test-default"/"default-dockercfg-c469z"
Apr 12 16:03:14.427955 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd[1]: Created slice libcontainer container kubepods-burstable-pod94893f9a_73fe_4ac8_80f7_88fb73a58ed4.slice.
Apr 12 16:03:14.493947 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.493780    1509 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "sockets" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-sockets") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.493947 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.493845    1509 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "container-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-container-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.493947 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.493935    1509 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "libvirt-runtime" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-libvirt-runtime") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.494317 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.493983    1509 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "hotplug-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-hotplug-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.494317 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.494042    1509 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ephemeral-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-ephemeral-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.594763 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.594690    1509 reconciler.go:269] operationExecutor.MountVolume started for volume "container-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-container-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.594763 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.594762    1509 reconciler.go:269] operationExecutor.MountVolume started for volume "libvirt-runtime" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-libvirt-runtime") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.594763 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.594793    1509 reconciler.go:269] operationExecutor.MountVolume started for volume "hotplug-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-hotplug-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595074 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.594930    1509 reconciler.go:269] operationExecutor.MountVolume started for volume "ephemeral-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-ephemeral-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595074 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.594997    1509 reconciler.go:269] operationExecutor.MountVolume started for volume "sockets" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-sockets") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595668 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.595536    1509 operation_generator.go:672] MountVolume.SetUp succeeded for volume "container-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-container-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595668 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.595588    1509 operation_generator.go:672] MountVolume.SetUp succeeded for volume "hotplug-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-hotplug-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595668 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.595630    1509 operation_generator.go:672] MountVolume.SetUp succeeded for volume "sockets" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-sockets") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595834 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.595684    1509 operation_generator.go:672] MountVolume.SetUp succeeded for volume "libvirt-runtime" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-libvirt-runtime") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.595834 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.595813    1509 operation_generator.go:672] MountVolume.SetUp succeeded for volume "ephemeral-disks" (UniqueName: "kubernetes.io/empty-dir/94893f9a-73fe-4ac8-80f7-88fb73a58ed4-ephemeral-disks") pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp" (UID: "94893f9a-73fe-4ac8-80f7-88fb73a58ed4")
Apr 12 16:03:14.739482 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk hyperkube[1509]: I0412 16:03:14.739392    1509 kuberuntime_manager.go:443] No sandbox for pod "virt-launcher-testvmi-c4l5qhxjptnz-j8cvp_kubevirt-test-default(94893f9a-73fe-4ac8-80f7-88fb73a58ed4)" can be found. Need to start a new one
Apr 12 16:03:14.740299 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 16:03:14.740238414Z" level=info msg="Running pod sandbox: kubevirt-test-default/virt-launcher-testvmi-c4l5qhxjptnz-j8cvp/POD" id=3fd6b172-4cfa-47fe-8b9f-64fca439d2da name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Apr 12 16:03:14.772981 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 16:03:14.772838105Z" level=info msg="Got pod network &{Name:virt-launcher-testvmi-c4l5qhxjptnz-j8cvp Namespace:kubevirt-test-default ID:6a46db75ffa480374a9d8fcfdc588ce72740801089769b4a383149ea4d1f0bee NetNS:/var/run/netns/2c79f897-e787-4ee1-adf6-7ec9577834b2 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
Apr 12 16:03:14.772981 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk crio[1478]: time="2021-04-12 16:03:14.772890310Z" level=info msg="About to add CNI network multus-cni-network (type=multus)"
Apr 12 16:03:14.811148 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Apr 12 16:03:14.822550 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk NetworkManager[1289]: <info>  [1618243394.8225] manager: (veth12398ed5): new Veth device (/org/freedesktop/NetworkManager/Devices/56)
Apr 12 16:03:14.823773 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk NetworkManager[1289]: <info>  [1618243394.8237] device (veth12398ed5): carrier: link connected
Apr 12 16:03:14.829155 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_UP): veth12398ed5: link is not ready
Apr 12 16:03:14.838264 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth12398ed5: link becomes ready
Apr 12 16:03:14.830963 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd-udevd[30230]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 12 16:03:14.849173 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Apr 12 16:03:14.854462 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk systemd-udevd[30230]: Could not generate persistent MAC address for veth12398ed5: No such file or directory
Apr 12 16:03:14.862175 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk kernel: device veth12398ed5 entered promiscuous mode
Apr 12 16:03:14.862321 ci-op-ji3f5bsp-88247-j6w2b-worker-d-ddfdk NetworkManager[1289]: <info>  [1618243394.8622] manager: (veth12398ed5): new Open vSwitch Port device (/org/freedesktop/NetworkManager/Devices/57)

@nunnatsa we won't be able to fix this without necessary logs. Is there a way to get the list of KubeVirt pods running in the namespace (to confirm that the deployment is healthy)?

yes. For example:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1263/pull-ci-kubevirt-hyperconverged-cluster-operator-master-hco-e2e-image-index-gcp/1383754513193635840/artifacts/hco-e2e-image-index-gcp/gather-extra/artifacts/pods/
And since it fails all the time, there are many logs.

@phoracek looks like all the KV pods are in place. According to the virt-handler  log there are many errors of this kind: failed to configure vmi network: failed to detect isolation for launcher pod: dial unix //pods/99014787-915e-40cf-8d37-f99168395ef0/volumes/kubernetes.io~empty-dir/sockets/launcher-sock: connect: connection refused

@enp0s3 but virt-launcher logs are empty, that makes me thing it never started, and due to that we failed talking to it through the socket. Does that sound reasonable?

And there are other logs where handler fails with this error while launcher is not listed in logs at all, e.g. https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1242/pull-ci-kubevirt-hyperconverged-cluster-operator-master-hco-e2e-image-index-gcp/1381618709780500480/artifacts/hco-e2e-image-index-gcp/gather-extra/artifacts/pods/

This is still happening

@enp0s3 , @phoracek - This is still happening: 100% of the HCO tests on gcp are failing.

@nunnatsa could you arrange a debugging session Igor mentions in https://github.com/kubevirt/kubevirt/issues/5447#issuecomment-836314516? I don't see any other way to get the issue identified and fixed

Even if we'll have long sleep, we don't access to openshift-ci clusters

We are hitting the same issue for AWS as well. It breaks our pipeline.
https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directory/rehearse-20950-pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-aws

@tiraboschi can you extract the auditlog? Access denied could be due to selinux.

@tiraboschi can you extract the auditlog? Access denied could be due to selinux.

But it is probably not that, because I think we would get a Permission denied then, and not a connection refused. :/

@tiraboschi can you extract the auditlog? Access denied could be due to selinux.

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1472/pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-gcp/1425686614738210816/artifacts/hco-e2e-image-index-gcp/gather-audit-logs/artifacts/audit-logs.tar

oh, the SELinux audit logs? sorry, checking

Here we have kubevirt-must-gather logs, if the info that we need to debug this are not there, we should think about improving kubevirt-must-gather:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1329/pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-image-index-gcp/1425837152389828608/artifacts/hco-e2e-image-index-gcp/e2e-test/artifacts/kubevirt-must-gather/

ok, now we added SELinux audit logs tokubevirt-must-gather, but we don't see anything strange there

Looking at an HCO GCP test cluster, which has 3 masters and 3 workers, I noticed that for 7.5 minutes after deployment and into the test suite, the virt-handler daemonset only deployed 1 virt-handler pod.
That might be a red herring, and an acceptable solution in production, but during those 7.5 minutes a lot of tests that require multiple nodes can fail, most likely on timeouts.
Here's the output of oc describe daemonset virt-handler -n kubevirt-hyperconverged:

When there was only 1 virt-handler
```
Name:           virt-handler
Selector:       kubevirt.io=virt-handler
Node-Selector:  kubernetes.io/os=linux,node.kubernetes.io/hco-test-node-type=workloads
Labels:         app.kubernetes.io/component=kubevirt
                app.kubernetes.io/managed-by=kubevirt-operator
                app.kubernetes.io/version=v0.45.0
                kubevirt.io=virt-handler
Annotations:    deprecated.daemonset.template.generation: 1
                kubevirt.io/customizer-identifier: bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f
                kubevirt.io/generation: 3
                kubevirt.io/install-strategy-identifier: 0aff4b2c5a0f883660519903ee2ac71e75b87e1a
                kubevirt.io/install-strategy-registry: quay.io/kubevirt
                kubevirt.io/install-strategy-version: v0.45.0
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=kubevirt
                    app.kubernetes.io/managed-by=kubevirt-operator
                    app.kubernetes.io/version=v0.45.0
                    kubevirt.io=virt-handler
                    prometheus.kubevirt.io=
  Annotations:      kubevirt.io/install-strategy-identifier: 0aff4b2c5a0f883660519903ee2ac71e75b87e1a
                    kubevirt.io/install-strategy-registry: quay.io/kubevirt
                    kubevirt.io/install-strategy-version: v0.45.0
                    scheduler.alpha.kubernetes.io/critical-pod:
  Service Account:  kubevirt-handler
  Init Containers:
   virt-launcher:
    Image:      quay.io/kubevirt/virt-launcher@sha256:58106a76841a41bafb38006745020c1be0ae2fd0dc92052f654be408347eb13d
    Port:
    Host Port:
    Command:
      /bin/sh
      -c
    Args:
      node-labeller.sh
    Environment:
    Mounts:
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
  Containers:
   virt-handler:
    Image:      quay.io/kubevirt/virt-handler@sha256:6ed571ff38b6831fef7311bef5dd7aba84bf34ca43fc0d3a1e3d4227135098e7
    Port:       8443/TCP
    Host Port:  0/TCP
    Command:
      virt-handler
      --port
      8443
      --hostname-override
      $(NODE_NAME)
      --pod-ip-address
      $(MY_POD_IP)
      --max-metric-requests
      3
      --console-server-port
      8186
      --graceful-shutdown-seconds
      315
      -v
      2
    Requests:
      cpu:      10m
      memory:   230Mi
    Liveness:   http-get https://:8443/healthz delay=15s timeout=10s period=45s #success=1 #failure=3
    Readiness:  http-get https://:8443/healthz delay=15s timeout=10s period=20s #success=1 #failure=3
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      MY_POD_IP:   (v1:status.podIP)
    Mounts:
      /etc/virt-handler/clientcertificates from kubevirt-virt-handler-certs (ro)
      /etc/virt-handler/servercertificates from kubevirt-virt-handler-server-certs (ro)
      /pods from kubelet-pods-shortened (rw)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/lib/kubelet/pods from kubelet-pods (rw)
      /var/lib/kubevirt from virt-lib-dir (rw)
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-libvirt-runtimes from libvirt-runtimes (rw)
      /var/run/kubevirt-private from virt-private-dir (rw)
  Volumes:
   kubevirt-virt-handler-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-certs
    Optional:    true
   kubevirt-virt-handler-server-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-server-certs
    Optional:    true
   libvirt-runtimes:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-libvirt-runtimes
    HostPathType:
   virt-share-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt
    HostPathType:
   virt-lib-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubevirt
    HostPathType:
   virt-private-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-private
    HostPathType:
   device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:
   kubelet-pods-shortened:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
   kubelet-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
   node-labeller:
    Type:               HostPath (bare host directory volume)
    Path:               /var/lib/kubevirt-node-labeller
    HostPathType:
  Priority Class Name:  kubevirt-cluster-critical
Events:
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  5m23s  daemonset-controller  Created pod: virt-handler-vgdr9
```

After all 3 started
```
Name:           virt-handler
Selector:       kubevirt.io=virt-handler
Node-Selector:  kubernetes.io/os=linux
Labels:         app.kubernetes.io/component=kubevirt
                app.kubernetes.io/managed-by=kubevirt-operator
                app.kubernetes.io/version=v0.45.0
                kubevirt.io=virt-handler
Annotations:    deprecated.daemonset.template.generation: 3
                kubevirt.io/customizer-identifier: bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f
                kubevirt.io/generation: 6
                kubevirt.io/install-strategy-identifier: 0aff4b2c5a0f883660519903ee2ac71e75b87e1a
                kubevirt.io/install-strategy-registry: quay.io/kubevirt
                kubevirt.io/install-strategy-version: v0.45.0
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 2 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=kubevirt
                    app.kubernetes.io/managed-by=kubevirt-operator
                    app.kubernetes.io/version=v0.45.0
                    kubevirt.io=virt-handler
                    prometheus.kubevirt.io=
  Annotations:      kubevirt.io/install-strategy-identifier: 0aff4b2c5a0f883660519903ee2ac71e75b87e1a
                    kubevirt.io/install-strategy-registry: quay.io/kubevirt
                    kubevirt.io/install-strategy-version: v0.45.0
                    scheduler.alpha.kubernetes.io/critical-pod:
  Service Account:  kubevirt-handler
  Init Containers:
   virt-launcher:
    Image:      quay.io/kubevirt/virt-launcher@sha256:58106a76841a41bafb38006745020c1be0ae2fd0dc92052f654be408347eb13d
    Port:
    Host Port:
    Command:
      /bin/sh
      -c
    Args:
      node-labeller.sh
    Environment:
    Mounts:
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
  Containers:
   virt-handler:
    Image:      quay.io/kubevirt/virt-handler@sha256:6ed571ff38b6831fef7311bef5dd7aba84bf34ca43fc0d3a1e3d4227135098e7
    Port:       8443/TCP
    Host Port:  0/TCP
    Command:
      virt-handler
      --port
      8443
      --hostname-override
      $(NODE_NAME)
      --pod-ip-address
      $(MY_POD_IP)
      --max-metric-requests
      3
      --console-server-port
      8186
      --graceful-shutdown-seconds
      315
      -v
      2
    Requests:
      cpu:      10m
      memory:   230Mi
    Liveness:   http-get https://:8443/healthz delay=15s timeout=10s period=45s #success=1 #failure=3
    Readiness:  http-get https://:8443/healthz delay=15s timeout=10s period=20s #success=1 #failure=3
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      MY_POD_IP:   (v1:status.podIP)
    Mounts:
      /etc/virt-handler/clientcertificates from kubevirt-virt-handler-certs (ro)
      /etc/virt-handler/servercertificates from kubevirt-virt-handler-server-certs (ro)
      /pods from kubelet-pods-shortened (rw)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/lib/kubelet/pods from kubelet-pods (rw)
      /var/lib/kubevirt from virt-lib-dir (rw)
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-libvirt-runtimes from libvirt-runtimes (rw)
      /var/run/kubevirt-private from virt-private-dir (rw)
  Volumes:
   kubevirt-virt-handler-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-certs
    Optional:    true
   kubevirt-virt-handler-server-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-server-certs
    Optional:    true
   libvirt-runtimes:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-libvirt-runtimes
    HostPathType:
   virt-share-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt
    HostPathType:
   virt-lib-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubevirt
    HostPathType:
   virt-private-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-private
    HostPathType:
   device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:
   kubelet-pods-shortened:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
   kubelet-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
   node-labeller:
    Type:               HostPath (bare host directory volume)
    Path:               /var/lib/kubevirt-node-labeller
    HostPathType:
  Priority Class Name:  kubevirt-cluster-critical
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  9m7s  daemonset-controller  Created pod: virt-handler-vgdr9
  Normal  SuccessfulDelete  96s   daemonset-controller  Deleted pod: virt-handler-vgdr9
  Normal  SuccessfulCreate  94s   daemonset-controller  Created pod: virt-handler-vk2fj
  Normal  SuccessfulCreate  27s   daemonset-controller  Created pod: virt-handler-gzf6v
  Normal  SuccessfulCreate  26s   daemonset-controller  Created pod: virt-handler-9r2jb
```

Even if that's not the whole issue, I'd suggest waiting until 3 virt-handlers are running to start the test suite.

The reason for that we have only virt-handler pod is that we are labeling the nodes with infra/workload labels and then use nodePlacement configuration to assign VMIs into workload nodes. This is a part of that test so having one virt-handler pod is expected.
I  did re-trigger the job and connect to the cluster while tests are being running. Then access the virt-launcher logs. Here is the real error we hit:
{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:74","timestamp":"2021-09-24T13:39:26.707967Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:77","timestamp":"2021-09-24T13:39:26.708066Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:492","timestamp":"2021-09-24T13:39:26.713035Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:500","timestamp":"2021-09-24T13:39:26.716224Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 7.0.0, package: 12.fc32 (Unknown, 2021-04-06-14:25:02, )","subcomponent":"libvirt","thread":"41","timestamp":"2021-09-24T13:39:26.747000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: testvmi-2sd77","subcomponent":"libvirt","thread":"41","timestamp":"2021-09-24T13:39:26.747000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Child process (dmidecode -q -t 0,1,2,3,4,11,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2767","subcomponent":"libvirt","thread":"41","timestamp":"2021-09-24T13:39:26.747000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:508","timestamp":"2021-09-24T13:39:27.219631Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:496","timestamp":"2021-09-24T13:39:27.224113Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:80","timestamp":"2021-09-24T13:39:27.224665Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Executing PreStartHook on VMI pod environment","name":"testvmi-2sd77","namespace":"kubevirt-test-default","pos":"manager.go:428","timestamp":"2021-09-24T13:39:27.724514Z","uid":"7743beb5-bcf4-4a68-b6af-6840b3923545"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Starting PreCloudInitIso hook","name":"testvmi-2sd77","namespace":"kubevirt-test-default","pos":"manager.go:437","timestamp":"2021-09-24T13:39:27.724598Z","uid":"7743beb5-bcf4-4a68-b6af-6840b3923545"}
{"component":"virt-launcher","level":"info","msg":"Found nameservers in /etc/resolv.conf: \ufffd\u001e\u0000\n","pos":"network.go:273","timestamp":"2021-09-24T13:39:27.725146Z"}
{"component":"virt-launcher","level":"info","msg":"Found search domains in /etc/resolv.conf: kubevirt-test-default.svc.cluster.local svc.cluster.local cluster.local c.openshift-gce-devel-ci.internal google.internal c.openshift-gce-devel-","pos":"network.go:274","timestamp":"2021-09-24T13:39:27.725197Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Domain XML generated. Base64 dump PGRvbWFpbiB0eXBlPSJrdm0iIHhtbG5zOnFlbXU9Imh0dHA6Ly9saWJ2aXJ0Lm9yZy9zY2hlbWFzL2RvbWFpbi9xZW11LzEuMCI+Cgk8bmFtZT5rdWJldmlydC10ZXN0LWRlZmF1bHRfdGVzdHZtaS0yc2Q3NzwvbmFtZT4KCTxtZW1vcnkgdW5pdD0iYiI+MTM0MjE3NzI4PC9tZW1vcnk+Cgk8b3M+CgkJPHR5cGUgYXJjaD0ieDg2XzY0IiBtYWNoaW5lPSJxMzUiPmh2bTwvdHlwZT4KCQk8c21iaW9zIG1vZGU9InN5c2luZm8iPjwvc21iaW9zPgoJPC9vcz4KCTxzeXNpbmZvIHR5cGU9InNtYmlvcyI+CgkJPHN5c3RlbT4KCQkJPGVudHJ5IG5hbWU9InV1aWQiPmJiZDg5MjgyLWJkNDQtNDkyMi1hODk3LThlYWQxNjAxYjU4YzwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJtYW51ZmFjdHVyZXIiPkt1YmVWaXJ0PC9lbnRyeT4KCQkJPGVudHJ5IG5hbWU9ImZhbWlseSI+S3ViZVZpcnQ8L2VudHJ5PgoJCQk8ZW50cnkgbmFtZT0icHJvZHVjdCI+Tm9uZTwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJza3UiPjwvZW50cnk+CgkJCTxlbnRyeSBuYW1lPSJ2ZXJzaW9uIj48L2VudHJ5PgoJCTwvc3lzdGVtPgoJCTxiaW9zPjwvYmlvcz4KCQk8YmFzZUJvYXJkPjwvYmFzZUJvYXJkPgoJCTxjaGFzc2lzPjwvY2hhc3Npcz4KCTwvc3lzaW5mbz4KCTxkZXZpY2VzPgoJCTxpbnRlcmZhY2UgdHlwZT0iZXRoZXJuZXQiPgoJCQk8c291cmNlPjwvc291cmNlPgoJCQk8dGFyZ2V0IGRldj0idGFwMCIgbWFuYWdlZD0ibm8iPjwvdGFyZ2V0PgoJCQk8bW9kZWwgdHlwZT0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPjwvbW9kZWw+CgkJCTxtdHUgc2l6ZT0iMTQxMCI+PC9tdHU+CgkJCTxhbGlhcyBuYW1lPSJ1YS1kZWZhdWx0Ij48L2FsaWFzPgoJCQk8cm9tIGVuYWJsZWQ9Im5vIj48L3JvbT4KCQk8L2ludGVyZmFjZT4KCQk8Y2hhbm5lbCB0eXBlPSJ1bml4Ij4KCQkJPHRhcmdldCBuYW1lPSJvcmcucWVtdS5ndWVzdF9hZ2VudC4wIiB0eXBlPSJ2aXJ0aW8iPjwvdGFyZ2V0PgoJCTwvY2hhbm5lbD4KCQk8Y29udHJvbGxlciB0eXBlPSJ1c2IiIGluZGV4PSIwIiBtb2RlbD0ibm9uZSI+PC9jb250cm9sbGVyPgoJCTxjb250cm9sbGVyIHR5cGU9InNjc2kiIGluZGV4PSIwIiBtb2RlbD0idmlydGlvLW5vbi10cmFuc2l0aW9uYWwiPjwvY29udHJvbGxlcj4KCQk8Y29udHJvbGxlciB0eXBlPSJ2aXJ0aW8tc2VyaWFsIiBpbmRleD0iMCIgbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj48L2NvbnRyb2xsZXI+CgkJPHZpZGVvPgoJCQk8bW9kZWwgdHlwZT0idmdhIiBoZWFkcz0iMSIgdnJhbT0iMTYzODQiPjwvbW9kZWw+CgkJPC92aWRlbz4KCQk8Z3JhcGhpY3MgdHlwZT0idm5jIj4KCQkJPGxpc3RlbiB0eXBlPSJzb2NrZXQiIHNvY2tldD0iL3Zhci9ydW4va3ViZXZpcnQtcHJpdmF0ZS83NzQzYmViNS1iY2Y0LTRhNjgtYjZhZi02ODQwYjM5MjM1NDUvdmlydC12bmMiPjwvbGlzdGVuPgoJCTwvZ3JhcGhpY3M+CgkJPG1lbWJhbGxvb24gbW9kZWw9InZpcnRpby1ub24tdHJhbnNpdGlvbmFsIj4KCQkJPHN0YXRzIHBlcmlvZD0iMTAiPjwvc3RhdHM+CgkJPC9tZW1iYWxsb29uPgoJCTxzZXJpYWwgdHlwZT0idW5peCI+CgkJCTx0YXJnZXQgcG9ydD0iMCI+PC90YXJnZXQ+CgkJCTxzb3VyY2UgbW9kZT0iYmluZCIgcGF0aD0iL3Zhci9ydW4va3ViZXZpcnQtcHJpdmF0ZS83NzQzYmViNS1iY2Y0LTRhNjgtYjZhZi02ODQwYjM5MjM1NDUvdmlydC1zZXJpYWwwIj48L3NvdXJjZT4KCQk8L3NlcmlhbD4KCQk8Y29uc29sZSB0eXBlPSJwdHkiPgoJCQk8dGFyZ2V0IHR5cGU9InNlcmlhbCIgcG9ydD0iMCI+PC90YXJnZXQ+CgkJPC9jb25zb2xlPgoJPC9kZXZpY2VzPgoJPG1ldGFkYXRhPgoJCTxrdWJldmlydCB4bWxucz0iaHR0cDovL2t1YmV2aXJ0LmlvIj4KCQkJPHVpZD43NzQzYmViNS1iY2Y0LTRhNjgtYjZhZi02ODQwYjM5MjM1NDU8L3VpZD4KCQkJPGdyYWNlcGVyaW9kPgoJCQkJPGRlbGV0aW9uR3JhY2VQZXJpb2RTZWNvbmRzPjA8L2RlbGV0aW9uR3JhY2VQZXJpb2RTZWNvbmRzPgoJCQk8L2dyYWNlcGVyaW9kPgoJCTwva3ViZXZpcnQ+Cgk8L21ldGFkYXRhPgoJPGZlYXR1cmVzPgoJCTxhY3BpPjwvYWNwaT4KCTwvZmVhdHVyZXM+Cgk8Y3B1IG1vZGU9Imhvc3QtbW9kZWwiPgoJCTx0b3BvbG9neSBzb2NrZXRzPSIxIiBjb3Jlcz0iMSIgdGhyZWFkcz0iMSI+PC90b3BvbG9neT4KCTwvY3B1PgoJPHZjcHUgcGxhY2VtZW50PSJzdGF0aWMiPjE8L3ZjcHU+CjwvZG9tYWluPg==","name":"testvmi-2sd77","namespace":"kubevirt-test-default","pos":"libvirt_helper.go:126","timestamp":"2021-09-24T13:39:27.726065Z","uid":"7743beb5-bcf4-4a68-b6af-6840b3923545"}
{"component":"virt-launcher","level":"info","msg":"Starting SingleClientDHCPv6Server","pos":"serverv6.go:44","timestamp":"2021-09-24T13:39:27.726352Z"}
{"component":"virt-launcher","level":"info","msg":"Starting SingleClientDHCPServer","pos":"server.go:63","timestamp":"2021-09-24T13:39:27.726872Z"}
{"component":"virt-launcher","level":"error","msg":"failed to run DHCP: Search domain is not valid: 'c.openshift-gce-devel-'","pos":"common.go:362","timestamp":"2021-09-24T13:39:27.727113Z"}
panic: Search domain is not valid: 'c.openshift-gce-devel-'

goroutine 46 [running]:
kubevirt.io/kubevirt/pkg/network/driver.(*NetworkUtilsHandler).StartDHCP.func1(0xc000361cc0, 0xc001b65ab8, 0x8, 0xc001b74180, 0x1, 0x1, 0xc001b7e300, 0x6, 0x8, 0x0, ...)
	pkg/network/driver/common.go:363 +0x36b
created by kubevirt.io/kubevirt/pkg/network/driver.(*NetworkUtilsHandler).StartDHCP
	pkg/network/driver/common.go:348 +0x2f0
{"component":"virt-launcher","level":"info","msg":"Reaped pid 14 with status 512","pos":"virt-launcher.go:549","timestamp":"2021-09-24T13:39:27.735146Z"}
{"component":"virt-launcher","level":"error","msg":"dirty virt-launcher shutdown: exit-code 2","pos":"virt-launcher.go:567","timestamp":"2021-09-24T13:39:27.735524Z"}
{"component":"virt-launcher","level":"error","msg":"error when checking for istio-proxy presence","pos":"virt-launcher.go:657","reason":"Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused","timestamp":"2021-09-24T13:39:33.864055Z"}
{"component":"virt-launcher","level":"error","msg":"error when checking for istio-proxy presence","pos":"virt-launcher.go:657","reason":"Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused","timestamp":"2021-09-24T13:39:36.935980Z"}
{"component":"virt-launcher","level":"error","msg":"error when checking for istio-proxy presence","pos":"virt-launcher.go:657","reason":"Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused","timestamp":"2021-09-24T13:39:40.008037Z"}
{"component":"virt-launcher","level":"error","msg":"error when checking for istio-proxy presence","pos":"virt-launcher.go:657","reason":"Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused","timestamp":"2021-09-24T13:39:43.080006Z"}

Thanks @erkanerol, it seems that it fails on https://github.com/kubevirt/kubevirt/blob/962aabb5f4816066c99b18993513abdca96fcf49/pkg/network/dhcp/server/server.go#L274, since "c.openshift-gce-devel-" does not match our expectation for a DNS domain. It should not be difficult to fix it, we'll try to address it the next week.

Confirmed that this is a cluster-wide network issue by opening a debug shell to a worker node:
sh-4.4# chroot /host
sh-4.4# cat /etc/resolv.conf
# Generated by NetworkManager
search c.openshift-gce-devel-ci.internal google.internal c.openshift-gce-devel-
nameserver <IP_ADDRESS>
sh-4.4#

c.openshift-gce-devel- does not appear to be all that valid, at least dig does not like it:
$ dig c.openshift-gce-devel-
dig: 'c.openshift-gce-devel-' is not a legal IDN name (string start/ends with forbidden hyphen), use +noidnin

$ dig c.openshift-gce-devel- +noidnin
dig: 'c.openshift-gce-devel-.' is not a legal IDNA2008 name (string start/ends with forbidden hyphen), use +noidnout

$ dig c.openshift-gce-devel- +noidnin +noidnout

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8 <<>> c.openshift-gce-devel- +noidnin +noidnout
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 37996
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;c.openshift-gce-devel-.		IN	A

;; Query time: 0 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Fri Sep 24 17:10:47 IDT 2021
;; MSG SIZE  rcvd: 51

Do we have an understanding of whoever is configuring that particular DNS name ?

@phoracek @maiqueb Hi. Thanks for your comments. I don't know so much about this field so sorry for my dumb question: How can we proceed from this point? Should I contact with OpenShift CI team?

I think so - imo, this seems to be a configuration issue. Plus a bug on some component that allowed that domain name to be set. I unfortunately do not know which.
I've found this unit test on our code that seems to prevent domains starting with - , which, are not Internationalized Domain Names.
From the DNS RFC, section 2.3.1 I see that (quoting below):
The labels must follow the rules for ARPANET host names.  They must
start with a letter, end with a letter or digit, and have as interior
characters only letters, digits, and hyphen.  There are also some
restrictions on the length.  Labels must be 63 characters or less.

I created a bug against Network Manager of Red Hat Enterprise Linux.  https://bugzilla.redhat.com/show_bug.cgi?id=2008521

@erkanerol Network Manager usually just writes whatever the DHCP server said to /etc/resolv.conf. IMO, the only way this is its fault is if it truncates the search list to 72 characters.
To me it looks a lot more like a GCP+OCP cluster misconfiguration.
I could be interesting to run a dhcp client in verbose mode to see the IP of the DHCP server. That IP should point you directly to the culprit!

I think NetworkManager just reports what the DHCP is providing. It's systemd-resolved which populates /etc/resolv.conf.

Hi, not sure its related but it might,
When debugging other problem, I saw that fedora which has cloud-init NoCloud config, cause kubevirt to set cloud-init's metadata local-hostname, which in turn cause the VM to disregard what DHCP configures,
removing the marked line, or removing the cloud-init "fixed" the problem, but its under research still (in order to know the consequences)
func readCloudInitNoCloudMetaData(name, hostname, namespace string) *NoCloudMetadata {
	return &NoCloudMetadata{
		InstanceID:    fmt.Sprintf("%s.%s", name, namespace),
		LocalHostname: hostname, // Remove this line
	}
}

/remove-lifecycle stale

bug https://bugzilla.redhat.com/show_bug.cgi?id=2008521 is now fixed and we are finally able to start kubevirt VMs also in openshift CI using GCP.
See https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/1789/pull-ci-kubevirt-hyperconverged-cluster-operator-main-hco-e2e-kv-smoke-gcp/1496023014616076288 for an example
So now we can close this

I AM REALLY GLAD TO SEE THIS! CONGRATS!

/close
The issue seems to be resolved.


mafCompare Sample Size Error

Describe the issue
When using mafCompare on two groups where one has a small sample size, an error is returned which seems to relate, as far as I can tell with the use of getAnywhere() to look at the code, to a sample size issue. However, I have been unable to figure out what is happening exactly. I think perhaps I end up with a negative sample size?
Command
Please post your commands and the output (errors or any unexpected output)
Group1 <- read.maf(file.choose())

Group2 <- read.maf(file.choose())

Group1.vs.Group2 <- mafCompare(m1 = Group1, m2 = Group2, m1Name = "Group1", m2Name = "Group2", minMut = 5)
sink(file = "Group1.vs.Group2.txt")
print(Group1.vs.Group2)
sink(file = NULL)

sessionInfo()

Session info
Run sessionInfo() and post the output below
R version 4.0.3 (2020-10-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] maftools_2.4.12

loaded via a namespace (and not attached):
 [1] bit_4.0.4          compiler_4.0.3     Matrix_1.2-18      tools_4.0.3        RColorBrewer_1.1-2 survival_3.2-11    yaml_2.2.1
 [8] bit64_4.0.5        splines_4.0.3      grid_4.0.3         data.table_1.14.0  lattice_0.20-41

Hi,
I am sorry for the tardy reply. Is it possible to share some reproducible data?

Apologies, just had to check in with my supervisors. I can provide the files that I used for the analysis, if there is a non-public way that I can get them to you?

Sure, I understand. You can mail me at<EMAIL_ADDRESS>Feel free to remove any sensitive information and change sample names if you would like.

The problem is now successfully resolved.
Thanks again :)


how can i add my yaml file

Documentation link

Describe the problem

Suggested fix

I have my YAML file for API doc can you give me a way how can I add this YAML file like you add swagger pet store API

Hi @nitinmcsam, please refer to the docs and the examples included in the demo and template:
https://docusaurus-openapi.tryingpan.dev/


WIP: Requre testing

Requre tests: test_status.

recheck

recheck

/packit copr-build

It contains some useful but very outdated stuff... I will try to rebase that some day..;)


How to cache a json object into a map as value?

Hi, I have JSON file
{
    "type":"design",
    "guid":"86a71c67-dd7b-4bdd-9104-fca9ab12f953",
    "children":[
        {
            "type":"folder",
            "guid":"2c2af70d-8b05-4453-a0e1-8c233b0f7de1",
            "children":[
                {
                    "type":"resource",
                    "guid":"b92fda27-53a8-43aa-a240-fce65762462f"
                }
            ]
        },
        {
            "type":"resource",
            "guid":"b18aa915-d268-48b7-83c9-75ee35ab6e37"
        },
        {
            "type":"resource",
            "guid":"98f8626e-964d-4ca1-b3b6-45eff9c600c8"
        }
    ]
}

It is generated by CJSON by the following code
bool BubbleWriter::AppendNode(const UTF8String& parentId)
{
        std::map<UTF8String, cJSON*>::iterator it = m_content.find(parentId);
        if (it == m_content.end())
            return false;

        cJSON* parent = it->second;
        cJSON* pChildren = cJSON_GetObjectItem(parent, "children");
        if (NULL == pChildren)
        {
            pChildren = cJSON_CreateArray();
            cJSON_AddItemToObject(parent, "children", pChildren);
        }

        cJSON* pChild = cJSON_CreateObject();
        cJSON_AddItemToArray(pChildren, pChild);
        ...
 }

m_content is a map to cache the GUID and its corresponding json object, the function AppendNode() could append an object into a json array but the function does not know how deep is the current object in the json file.
I would like to replace CJSON to json.hpp, but I don't know how to cache a json object into a map as value by json.hpp. Any ideas?

You can just use std::map<std::string, nlohmann::json>.

The title of my question might not accurate enough, actually I would like to know how to cache a json object's pointer into a map, like std::map<std::string, nlohmann::json *>
Because I want to directly append child object under the cached objects, so that the original json object could be updated accordingly.
bool BubbleWriter::AppendNode(const UTF8String& parentId)
{
    std::map<UTF8String, json*>::iterator it = m_content.find(parentId);
    if (it == m_content.end())
        return false;

    json * parent = it->second;

    json pChild;
    pChild["type"] = "resource";
    pChild["guid"] = guid.Value();
    (*parent)["chirldren"].push_back(pChild);

    m_content.insert(make_pair(guid, **???**));
    ...
}

The code above is my changing based on previous CJSON logic, but I don't know how to write in the ??? place.
I tried &(*parent)["children"].back(), but the cached value would be changed if I push_back another object.
I also noticed the API get() / get_ptr() might meet my requirements, but I find the warning: Writing data to the pointee of the result yields an undefined state.

You can cache a pointer to a json object in the map without a problem.  The problem that you actually have is caching a pointer to an element of a vector whose size may change during the lifetime of the map.  That is something that you can't do unless you've reserved enough space in the vector ahead of time such that it can never reallocate.  What you could do is instead of caching a pointer to the object, cache a pointer to the parent and an index within the vector.  So instead of std::map<std::string, nlohmann::json *>, you have std::map<std::string, std::pair<nlohmann::json *, size_t>>.

@gregmarr Thank you so much! It works well to me.


Only set url value when it's not empty

Only a small nit.
Currently docker start API always contains a trailing checkpoint= even
if value is empty, this is unnecessary, we can set it only when it's not
empty.
Signed-off-by: Zhang Wei<EMAIL_ADDRESS>

One question / nit, other than that LGTM :+1:

ideally, it wouldn't send any parameter that is "default" (think there's still some other parameters that are sent even if no value is set by the user), so SGTM

Updated,  I didn't touch some very old APIs in case to break anything.
PTAL.  :smile:

Updated

LGTM do we need to change something in docker/docker when re-vendoring there?

@runcom yes a small thing where ContainerStart is called :angel:. Given the fact that checkpointID is optional we should have make the option struct first time but meh :stuck_out_tongue_closed_eyes:.
Still LGTM :+1:

Re-vendoring is enough, we haven't put this API

LGTM

I'll do a vendoring to make it work in docker/docker :smile:

@WeiZhang555 I oprned a PR on docker/docker for it yesterday (Bump engine-api ...) :stuck_out_tongue_closed_eyes:


[LLM Runtime] Sync inference scripts

Type of Change: feature & infra
API not changed
Description

moving cpp_graph_inference.sh under graph directory so it can be shared with internal CI system
Update  cpp_graph_inference.sh

Expected Behavior & Potential Risk
N/A
How has this PR been tested?
CI base on GHA & internal tests
Dependency Change?
N/A

For reviews: you can check the first commit solely for script moving and the second commit solely for its content changes.


JSON file doesn't show Alias Tokens

When I reference Tokens (Alias tokens) and try to export them, I only see raw values in the JSON file.
Does the plugin supports different types of tokens? (ex: Global/Alias/...)

Thanks for the report! This is on the same topic as #154 — currently the export itself transforms aliases, but it's quite understandable users want their aliases for style dictionary. What we need to do to get there is modify the export function to translate any aliases into style dictionary readable aliases — which are written inside curly brackets {token.alias}. Figma Tokens allows users to also write aliases with a $ sign prepended $token.alias.

@six7 I see this has been merged, but hasn't been released yet? If so when are you planning on releasing, keen to get the aliases working with SD? Thanks!

Hey @gmlnchv - this has been released quite a while ago. The plugin now retains aliases on SD Export. However, we'd still need an option for users to turn off this behavior, as SD can't use math right now.
Note: You can use {token.alias} style aliases in Figma Tokens as well, if you write them like that SD can handle them (except for math)


Allow exceptions to be disabled, enable "tie" support

This does a few things:

Removes all exception-throwing simdjson methods when SIMDJSON_EXCEPTIONS=0
Defaults SIMDJSON_EXCEPTIONS=0 when -fno-exceptions is supplied
Adds SIMDJSON_EXCEPTIONS=ON/OFF option to cmake
Run CI tests with SIMDJSON_EXCEPTIONS=OFF as well as with -fno-exceptions
Add error code versions of all basictests
Makes tie(value, error) = ... work everywhere (this was necessary to make the tests nice)
Uses common simdjson_result class everywhere for error pattern

Without this, we don't compile with -fno-exceptions, which is used in 50% of C++ projects. Even with exceptions allowed, some users may wish to turn this off, because invalid JSON may not actually be an exceptional situation for them. For those users, this gives them confidence they are explicitly handling all errors as they come up, and ensures they won't get a nightmare abort on a bad simdjson document somewhere deep in their code.
Fixes #521.

Worth noting: when simdjson is compiled with -fno-exceptions, twitter.json parses at exactly the same speed. We don't throw any exceptions internally, so it's not surprising, but still good to validate.

OK, now that it actually compiles with -fno-exceptions, I think this is baked. as long as this passes the tests I think this is baked.

@DBJDBJ will likely be pleased to see we compile with -fno-exceptions now. There were a few existing places that had to be excised.

Note: drone/push passed checkperf, but the drone/pr failed it a few times. This patch doesn't touch anything inside parsing and shouldn't have an effect, but it's worth a double check.

@jkeiser Do we document this somehow? I am happy not adding any specific documentation, so this is a genuine question.

It's not there, I just found it when I was writing examples. I think the right place for this documentation is an "error explainer." simdjson's error handling is rich enough to give it its own topic.

Note that the whole tie thing still makes me a bit uneasy; but I haven't figured out any better way to handle errors in C++.
We're also not following @DBJDBJ's recommendation to make the value an optional<T>: we instead rely on the user doing if (error) and not using the doc. It makes it easier to work with, but it does give them the chance to shoot themselves in the foot if they so desire. Continuing the grand C++ tradition :)


Introduce atom::internal as first argument for private behaviors

:notebook_with_decorative_cover: Description

Adds atom::internal as first argument for private behaviors to make them visible at the call site.
Adds atom::intel as a nice-to-have for the matcher plugin—snuck this in here because touching the atoms header is expensive and I don't want to do it twice.

:memo: Checklist

[x] All user-facing changes have changelog entries.
[x] The changes are reflected on docs.tenzir.com/vast, if necessary.
[x] The PR description contains instructions for the reviewer, if necessary.

:dart: Review Instructions
File-by-file.

The commit message of the second commit should be reworded if possible, since it doesn't make sense outside the context of this PR.

Valid point, I reworded it.

The change looks good to me assuming that you can fix the CI issues without any major additional work.

Unrelated GitHub failure I think.


Add base and abstracts icon.

Hi all!
Today I was implementing the 7:1 architecture in my project and I noticed that there were folder icons for components, pages, vendors, layout and such, but not for base and abstracts.
Can those be added as well?

+1 for the base folder icon (People may also use it when they are using interface, since it's a way for .net programmers)
@PKief May I do a proposal or it's a no op?

We've already added an icon for base. Not sure what is related to "abstract". With "abstract" anything can be related. That's why I think we should keep it with the default folder icon to not confuse someone.


New version: NirSoft.NirCmd version 2.87

Pull request has been created with komac v2.6.0 :rocket:

Resolves #174470

Microsoft Reviewers: Open in CodeFlow

@wingetbot run

Works

@wingetbot run

Works

@wingetbot run

Works

@wingetbot run

@wingetbot run

Works

@wingetbot run

Works

@wingetbot run

Works

If you try to run the executable without any parameters does it return "0" or an error code?

Both nircmd and nircmdc return a 0 exit code:

OK, we'll have to dig in to figure out why validation is failing then. Most often it's caused by a non-zero return code because the portable application is expecting some input.

I looked into the logs and it reported a blocking detection. I'll have to see if I can figure out what triggered that result.

@wingetbot run

@wingetbot run

Works


F# 4.1 struct records cannot be serialised

Description
Trying to serialise a simple record type introduced in F# 4.1, which is marked as a struct fails.
Repro code
open MBrace.FsPickler
open System.IO

[<Struct>]
type Broken =
    { Value: int }

[<EntryPoint>]
let main _ =
    let binary = FsPickler.CreateBinarySerializer()
    let broken = { Value = 1 }
    let ms = new MemoryStream()
    binary.Serialize(ms, broken)
    0

Expected behavior
Ideally it should work, or provide a clearer error.
Actual behavior
From this example, an exception is thrown:
MBrace.FsPickler.FsPicklerException occurred
  HResult=0x80131500
  Message=Error serializing object of type 'Program+Broken'.
  Source=FsPickler
  StackTrace:
   at MBrace.FsPickler.RootSerialization.writeRootObject[T](IPicklerResolver resolver, ReflectionCache reflectionCache, IPickleFormatWriter formatter, FSharpOption`1 streamingContext, FSharpOption`1 sifter, Boolean isHash, Boolean disableSubtypes, Pickler`1 pickler, T value)
   at MBrace.FsPickler.FsPicklerSerializer.Serialize[T](Stream stream, T value, FSharpOption`1 pickler, FSharpOption`1 streamingContext, FSharpOption`1 encoding, FSharpOption`1 leaveOpen)
   at Program.main(String[] _arg1) in Program.fs:line 14
Inner Exception 1:
NullReferenceException: Object reference not set to an instance of an object.

I observed a lot of VERY obscure errors while attempting to serialise fairly complex F# 4.1 types in a larger project. These were non-determinstic, and included:
System.Runtime.InteropServices.SEHException: 'External component has thrown an exception.'
and
Managed Debugging Assistant 'FatalExecutionEngineError' : 'The runtime has encountered a fatal error. The address of the error was at 0x246d6a66, on thread 0x8d78. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.'
and
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Finally, under a mono server which actually helped point me in the right direction, I received the following error in the larger project:
MBrace.FsPickler.PicklerGenerationException : Error generating pickler for type 'Test.MyStructRecordType'.
---- System.InvalidProgramException : Invalid IL code in (wrapper dynamic-method) MBrace.FsPickler.Emit/DynamicMethod/Marker:recordSerializer (MBrace.FsPickler.Pickler[],MBrace.FsPickler.WriteState,Test.MyStructRecordType): IL_0013: call      0x00000005

Related information

Operating system - Windows 10 & Linux
F# <IP_ADDRESS> (F# 4.1)
net452
FsPickler 3.2.0
Runtimes: .NET Framework & Mono on Linux

Workarounds:
I'm going to try writing struct records by hand for now, as we did prior to 4.1.

Reporting that the workaround of writing structs by hand in a pre-F# 4.1 does allow working around the issue, but isn't an elegant solution at all.

@varon this library has not been updated to take struct records into account. I pushed commit https://github.com/mbraceproject/FsPickler/commit/8b419014c64af6984a564197e8495624edd67aa7 so that the library reports proper error messages on struct record/union serialization.

Thanks for the super fast response, @eiriktsarpalis!
Is this still being actively maintained? Is there any timeframe on an update? What would be involved in getting this up and running?

@varon A group of us co-maintain the mbrace repos, so we can help guide you if you would like to contribute a PR (it would be great to have you on board).
It looks like it's mostly a matter of adjusting the codegen to take structnes into account.  That can be quite painful though - you have to take extra addresses quite often when accessing fields and so on

@dsyme - Thanks for the incredibly welcoming gesture.
I can shuffle some time around to get a PR for this together. It seems like a great opportunity to get some advanced mentorship on what seems to be a fairly clean but challenging codebase. I'll be happy to give it a bash if you can judge it's doable in a reasonable amount of time (for someone unfamiliar with the library internals).
I can probably figure out what to do, but a rough outline of the process would be great.

From glancing at the code

First get things working with the define EMIT_IL off (assuming that non-struct tests pass ok with that).  It could be that everything just works

Next look at the EMIT_IL cases. will need to look more closely at that

@dsyme - Thanks, I'll take a look.

I'll need to add a reference to System.ValueTuple for the struct tuples to work.
We may also need to upgrade F# 4.1 for that.
How would you like this to be done? NuGet? F# version selector?

I've attempted this in what seems to be a reasonable way: https://github.com/varon/FsPickler/commit/405a883ec1a413f7023bbbf86c568a671f4e6473
If it helps, you can view the PR in progress here: https://github.com/varon/FsPickler/commits/fsharp41-compat

@varon Great job so far, looking forward to the PR :)

I've seen that several PR have been merged to solve this bug. But it's not published on nuget yet ?
Is there still work on this issue ?

Fixed in FsPickler 4.0


Job Cleanup in Testing

Summary
Currently, there are places where jobs are not being fully cleaned up after tests are done executing.
Details and comments
Example of the GitHub Actions Push Test working here.
Fixes #1010

Condenses/unifies test suite job creation to TestSuite._sim_job(..)
Adds synchronization wherever possible (via  job.wait_for_final_state())
For the 2 tests that check concurrent jobs, an environment variable (LIMIT_CONCURRENT_JOBS) was introduced into the GitHub main.yml for Push/PR tests only. This variable would be used only in CI builds that are testing multiple python versions concurrently.

A few general comments:

_sim_job is really not the best name as some of these tests use real backends, not simulators
We should only hit IBMQBackendJobLimitError in test_basic_server_paths, because that's the only test that uses an open account.
Do we really need to wait_for_final_state()? Since these tests run on production, we want to take as little time on the queue/device as possible, to minimize impact to real user jobs. Canceling the jobs early instead of waiting for them to complete helps with that, unless the tests need completed jobs.

@jyu00 Updates:

_sim_job() is now renamed to _run_job()
wait_for_final_state() was replaced with cancel_job()

Do you really need to add all the additional cancel()? IBMQTestCase in theory would cancel all jobs when the test case ends (if it doesn't, then perhaps that code needs to be updated).

@jyu00 the idea is to cancel any jobs in loops (s.t there would be 2+ ongoing before the test ends) or those that occur before an independent chunk of logic. Any other cancels are unnecessary.


Fix sample ranges for F# tour

They were off, as #6562 pointed out

Thanks for commenting back. I've approved this, and you can :shipit: when ready.

Thanks!


Getting a TypeError instead of evaluation error

Getting the wrong exception type when expression uses all
Expect to get an CELEvalError but instead of a TypeError
import celpy
from celpy import CELEvalError

env = celpy.Environment()

# Press the green button in the gutter to run the script.
if __name__ == '__main__':
    all_positive = "data.all(x, x>0)"
    breaks = {'data_': [1, 2, 3] }

    ast = env.compile(all_positive)
    prgm = env.program(ast)

    try:
        prgm.evaluate(breaks)
    except CELEvalError as ex:
        print('Got an evaluation error')
    except TypeError as ex:
        print('Got a type error')

Happy to try and fix with some guidance

I'm curious why your context has data_ where your expression uses data? I see an evaluation error using:
    all_positive = "data.all(x, x > 0)"
    breaks = {'data': [1, 2, '3']}

I'm not sure what the baseline Golang CEL implementation does.
I am confident it does not expose a Python exception; it seems more likely that it returns a CEL error code.


minor: [BLUEP-570] script for crawling events in azure

https://k6scope.atlassian.net/browse/BLUEP-570

The resource for event crawl and data crawl should be set up as part of a single terrform module.


Error on drop database and table

Which version are you using?
2.3.7
Describe the bug
Dropping tables always return next error:

core.js:9110 ERROR Error: Uncaught (in promise): TypeError: Cannot set property 'count' of undefined

Dropping databases always return next error:

Error: TypeError: Cannot read property 'pkCol' of undefined

Tables are not dropped really in IndexedDB storage. Wherever, when query 'show tables' is submmited, this tables don't are returned
Example
nSQL().dropDatabase("my_db").then(() => {     // never }).catch(() => {     // ALWAYS!!  })
    nSQL('users')
      .query('drop')
      .exec()
      .then(() => {
    // never
}).catch(() => {
    // ALWAYS!!
})`

I've seen something similar.
Which adapter (or value of mode in db config) are you using?

I'm using 'PERM' mode.
Just like in:
https://nanosql.io/databases.html#creating-a-database

Hi! Did you get any far with this? I'm using 'TEMP' and seeing it as well.

To be more specific, seeing it when dropping a table:
await nSQL(this.tableName).query('drop').exec();

No problems using 'drop' in my application.
result = await nSQL( this.dbTableName )).query( 'drop' ).exec()
You could try rebuilding indexes.
const result = await nSQL( this.dbTableName ).query( 'rebuild indexes' ).exec()


Please help with performance of partial view with model and without model.

Help us make content visible

Tell us what search terms you used and how you searched docs.
Tell us what docs you found that didn't address your concern.

Describe the new topic

Explain why this topic is needed.
Suggest a location in the Table of Contents.
Write an abstract. In one short paragraph, describe what this topic will cover.
Create an outline for the new topic. We'll help review the outline and approve it before anyone writes a topic.

Hello! Thanks for contacting us. This sounds like a general question about using ASP.NET Core. While we try to look at and respond to all issues, for questions like this we recommend posting to a community support group like Stack Overflow with the asp.net-core tag.


Link to Exciting Code

@deyulu @FCMeng can one of you put the right link to the Exciting code in the README? Or just link me to it here. Thanks!

@matthewcarbone can you update with this link: https://exciting-code.org/ ?

Done!


[FEATURE] error message for is_active=False user

Does anyone know how to extend the APIVIew to return more accurate messages for is_active=False users?

from rest_framework.decorators import api_view
from rest_framework.response import Response

@api_view()
def hello_world(request):
    if request.user.is_active:
        return Response({"message": "Hello, world!"})

   return Response({"message": "Not hello, world!"})

?

This library for some reason decides that if a user isn't active it just won't ever send an email out and it does this by monkey patching the user model to expose a new function
I don't know if this is the best or most elegant fix, but you can monkey patch it back out. A env setting to allow inactive users to have password resets would be so much better.
# Override for django-rest-passwordreset
from django.conf import settings
from django.contrib.auth import get_user_model

def override_password_reset():
    # add eligible_for_reset to the user class
    UserModel = get_user_model()
    UserModel.add_to_class("eligible_for_reset", eligible_for_reset)

def eligible_for_reset(self):
    if getattr(settings, 'DJANGO_REST_MULTITOKENAUTH_REQUIRE_USABLE_PASSWORD', True):
        # if we require a usable password then return the result of has_usable_password()
        return self.has_usable_password()
    else:
        # otherwise return True because we dont care about the result of has_usable_password()
        return True

override_password_reset()


[beta] Map button redirects to /map instead of opening sidecar for non React routes

Also Map button doesn't do anything when I'm on /map.

Yes, those are both expected behaviors. We can't support both the iframe and the react map component. Also, while on map there is no reason to also open the map drawer.

@bugron can this be closed as resolved?

Closed. But in my opinion behavior of the Map button should be the same on every page (except /map where there is no need to open the sidecar).

Umm yeah, I understand, but its technically very costly to do that on the non-react pages. On those pages if we did have a sidecar, clicking the links would mean a redirect request to the server and not the react routes.
While on react pages we can simply render the corresponding challenge after hitting the react routes within the page.
Hence the behavior.
Maybe if in future we have all the pages reactified, we could do this easily. Hope this helps?

Sure, I understand. Hopefully, in future this behavior will be changed.


Upload SBOM directly to OCI registry

What would you like to be added:
cosign supports attaching SBOMs to OCI registries[^1] (also has a spec for it [^2]), so we (w/@dentrax) thought that it would be nice to have the same one for Syft, here is the design that we thought.
$ syft packages <image> -o oci://<repository>/<image>

Why is this needed:
Additional context:
[^1]: https://github.com/sigstore/cosign/blob/1f67ea73113f76c62429aa605d75105d77c370fa/cmd/cosign/cli/attach.go#L57
[^2]: https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md
cc: @dlorenc @luhring

yeah, SGTM, we (w/@dentrax) can make it happen if you all agree on this attach command. We can start working on it ASAP.
Let me summarize the steps that we are going to take:

Create a new command called attach.
Take a container image URL as an argument.
Generate an SBOM file for an image, and upload it to the OCI registry (maybe using cosign under the hood)

is that all we need @luhring, WDYT?

@developer-guy I think so! If we think of any more considerations, we'll put them here 😃
Thanks for taking this on. 🙏
Reach out at any point if it's not obvious to how use the existing patterns or Syft API for what you're doing and we'll jump in to take a look ❤

My two cents: a similar approach would make sense for Syft following the cosign attach sbom ... process:
syft attach <image>

I think making attach as a separated command would be making logic a bit hard to implement since packagesExec() function in packages.go file requires an image as an input argument. (We expect an SBOM to be generated before pass into cosign/attach command.)
The better UX IMHO would be passing a --attach flag in packages command. By this design, it'd be easier to call cosign/attach (if bool flag passed) just after bus.Publish in packagesExecWorker() func.
By the way, a new attach command could be useful in case we want to pass a pre-generated SBOM file. So we would not be necessary to generate SBOM in the attach command.
cc: @developer-guy

(We expect an SBOM to be generated before pass into cosign/attach command.)

Absolutely. On the implementation side, we can generate an SBOM in multiple execution paths. So we could make sure the attach command has an SBOM created before we invoke cosign's "attach" functionality, which should be all we need to do.
Another reason why attach makes sense as a new command is that it has its own distinct configuration needs, such as "write" access to registries (Syft's other commands don't require this), whether or not to override an existing attachment, etc.
I think the attest and attach functions should probably look the same on the command line, meaning they should either both be subcommands or they should both be flags, or else it might be confusing to users. IMHO, choosing to implement these as subcommands is more inline with the direction Syft is heading (see #516). cc: @wagoodman and @spiffcs for fact-checking me on this! 😄

By the way, a new attach command could be useful in case we want to pass a pre-generated SBOM file. So we would not be necessary to generate SBOM in the attach command.

I'm not sure I understand this part. If an SBOM has already been generated, why would users want to do the "attach" with syft instead of just using cosign attach?

This was discussed in the Anchore community WG meeting today (notes available at https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit ) -
These were the options that were proposed -

We could implement this in syft

it could potentially leverage the multi-output syntax being introduced by https://github.com/anchore/syft/pull/732 to have something like -o json=registry://my/image:latest
syft could potentially import cosign as a library using the logic at https://github.com/sigstore/cosign/blob/f19f4f754de43e8151b941ed2307374dae55381d/cmd/cosign/cli/attach/sbom.go#L34-L61

We could continue to let cosign handle SBOM/attachment

we could explicitly document how the two tools interact in the syft docs (or even in the cosign docs?) so as to let users know that they can work with each other
Since cosign appects sbom via stdin, this would allow syft to directly upload an sbom to the registry via syft packages <image> -o json | cosign attach sbom <image> --sbom - --type syft. This also follows the unix philosophy of one tool doing one thing well, while being able to be chained with other tools

In the WG meeting, people were leaning towards the latter + clear documentation to note that syft is compatible with cosign unless there are some other advantages to natively supporting cosign in syft.

The cosign behavior is documented here as a "spec", with the intention of being interoperable!
https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md
Either approach sounds great to me.

Added https://github.com/sigstore/cosign/pull/1278 to update the spec as well

Just catching up — if we're going with option 2 from the comment above, does that mean there's no work in Syft code for this any more, and it's just documentation? (of the flow for syft ... | cosign attach sbom ...?)

@luhring that's correct :) https://github.com/sigstore/cosign/pull/1137 and https://github.com/sigstore/cosign/pull/1278 handle the syft support and documentation on the cosign side. We just need to do the same at https://github.com/anchore/syft#adding-an-sbom-to-an-image-as-an-attestation to also mention sbom attachments

Perfect, thanks @samj1912!

Hi folk, as @samj1912 said, syft is now capable of uploading SBOM in form of in-toto attestation format to the registries, also, there is another option like "cosign attach", but AFAIK, there is still no direct support for uploading SBOMs to the OCI registries. Maybe, we could add another output option like "oci" to support this, the UX would be:
# will upload SBOM in a json format by using "application/vnd.syft+json" mediaType
$ syft packages alpine:latest -o json --upload/export oci://devopps/my-sbom-img

@dentrax @luhring

kindly ping

From discussion:

using oci could mean a number of different things, so we should tread carefully with naming, for example does this mean OCI Reference Types or OCI Attachments?


Cannot read property 'query' of undefined

Hi!
I have this error:
Cannot read property 'query' of undefined
When i nagivate to /dashboard
the app.jsx is this
import React from 'react'
import { render } from 'react-dom'
import { Router, Route, Link } from 'react-router'
import { createHistory } from 'history'

const App = React.createClass({
  getInitialState() {
    return {
      loggedIn: false
    }
  },

  render() {
    return (

            {this.state.loggedIn ? (
              <Link to="/logout">Log out</Link>
            ) : (
              <Link to="/login">Sign in</Link>
            )}

<Link to="/dashboard">Dashboard</Link> (authenticated)

        {this.props.children ||
You are {!this.state.loggedIn && 'not'} logged in.
}

    )
  }
})

const Dashboard = React.createClass({
  render() {

    return (

Dashboard

You made it!

    )
  }
})

const Login = React.createClass({

  contextTypes: {
    router: React.PropTypes.object.isRequired
  },

  getInitialState() {
    return {
      error: false
    }
  },

  render() {
    return (

Login Form goes here!

    )
  }
})

function requireAuth(nextState, replace) {
  if (true) {
    replace({
      pathname: '/login',
      state: { nextPathname: nextState.location.pathname },
      query: '',
    })
  }
}

var history = createHistory()

render((
  <Router history={history}>
    <Route path="/" component={App}>
      <Route path="login" component={Login} />
      <Route path="dashboard" component={Dashboard} onEnter={requireAuth} />
    </Route>
  </Router>
), document.getElementById('main'))

ReactDOM v0.14.6
React (with addons) v0.14.6
react-router: 2.0.0-rc5 (kept from umd directory under node_modules/react-router)

Any suggestions?
Thanks

Thanks for your question!
We want to make sure that the GitHub issue tracker remains the best place to track bug reports and feature requests that affect the development of React Router.
Questions like yours deserve a purpose-built Q&A forum. Could you post this question to Stack Overflow with the tag #react-router? https://stackoverflow.com/questions/ask?tags=react-router
We also have an active and helpful React Router community on Reactiflux, which is a great place to get fast help with React Router and with the rest of the React ecosystem. You can join at https://discord.gg/0ZcbPKXt5bYaNQ46.

This is an actual issue with the auth example.
Here's the stackoverflow q/a


Avoid download go packages repeatedly when "make build".

We already have a "docker-in" target which can leverage go cache to build
binaries. This patch use "docker-in" target in "make build" stage to speed
the build process. And "Dockerfile.build.ubuntu" can considered to be removed
beacause it is unused after this change.
Fixes #67

/test-e2e

Thanks @ruicao93 for the PR, could you fix typos in your commit messages (docker-bin, beacause) and remove the trailing period in the title?

Thanks @ruicao93 for the PR, could you fix typos in your commit messages (docker-in, beacause) and remove the trailing period in the title?

OK. I will update soon.

If I "fix" the go.mod first by running go mod tidy with go version 1.12 (it seems that the current go.mod was generated with go 1.13), then run make I no longer run into the permission issue (because the go.mod file is no longer overwritten by the docker image). However, the binary files under bin/ are owned by root, which is not optimal.

I don't recommend to generate files owned by root in workspace. It'll break Jenkins.

I'm adding a tidy checker which ensure the go.mod and go.sum are cleaned. After applying this, the go.mod and go.sum shall not be changed in e2e.

If I "fix" the go.mod first by running go mod tidy with go version 1.12 (it seems that the current go.mod was generated with go 1.13), then run make I no longer run into the permission issue (because the go.mod file is no longer overwritten by the docker image). However, the binary files under bin/ are owned by root, which is not optimal.

Yes,  it's still a issue.

@weiqiangt Yes but even if we ignore the binary files under bin/ (which are now owned by root), consider the following case:
I make modifications to the Go source code, adding some new dependencies.
To test my changes, I run make.
Because of the new dependencies, go.mod / go.sum need to be updated. This is going to happen inside the docker, thus creating permission issues.

Instead of this change I suggest keeping the Dockerfile.build.ubuntu as it is and using go mod download to avoid having to re-download dependencies every time something changes in the directory:
diff --git a/build/images/Dockerfile.build.ubuntu b/build/images/Dockerfile.build.ubuntu
index 21fe278..1807da2 100644
--- a/build/images/Dockerfile.build.ubuntu
+++ b/build/images/Dockerfile.build.ubuntu
@@ -9,10 +9,14 @@ RUN mkdir -p /opt/cni/bin && \

 FROM golang:1.12 as antrea-build

-COPY . /antrea
-
 WORKDIR /antrea

+COPY go.mod /antrea/go.mod
+
+RUN go mod download
+
+COPY . /antrea
+
 RUN make bin

This will make all these permission issues moot and the build will be just as fast as with your proposal.

Instead of this change I suggest keeping the Dockerfile.build.ubuntu as it is and using go mod download to avoid having to re-download dependencies every time something changes in the directory:
diff --git a/build/images/Dockerfile.build.ubuntu b/build/images/Dockerfile.build.ubuntu
index 21fe278..1807da2 100644
--- a/build/images/Dockerfile.build.ubuntu
+++ b/build/images/Dockerfile.build.ubuntu
@@ -9,10 +9,14 @@ RUN mkdir -p /opt/cni/bin && \

 FROM golang:1.12 as antrea-build

-COPY . /antrea
-
 WORKDIR /antrea

+COPY go.mod /antrea/go.mod
+
+RUN go mod download
+
+COPY . /antrea
+
 RUN make bin

This will make all these permission issues moot and the build will be just as fast as with your proposal.

Thanks @antoninbas , let me have a try locally.

/test-e2e

Thanks Antonin. It's really a good way to COPY go.mod and then the update operation will only be executed if there's update happens on file.

/test-e2e

/test-e2e

/test-e2e

@ruicao93 since you are a VMWare employee, I believe you need to join the VMware Github organization: https://github.vmware.com/home. After that you should not have any CLA issues any more.