CI: error doing L3+L4 policymap updates

Build link: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2033/execution/node/58/log/
Relevant output:
05:28:47 Summarizing 30 Failures:
05:28:47
05:28:47 [Fail] RuntimeValidatedLB Services Policies [It] tests with conntrack disabled
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/lb.go:384
05:28:47
05:28:47 [Fail] RuntimeValidatedLB Services Policies [It] tests with service performing L4 port mapping
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/lb.go:354
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers Cilium monitor verbose mode
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/monitor.go:54
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers Cilium monitor event types
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers cilium monitor check --from
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers cilium monitor check --to
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers cilium monitor check --related-to
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers multiple monitors
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedMonitorTest [BeforeEach] With Sample Containers checks container ids match monitor output
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedKVStoreTest [It] Consul KVStore
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/kvstore.go:86
05:28:47
05:28:47 [Fail] RuntimeValidatedKVStoreTest [It] Etcd KVStore
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/kvstore.go:106
05:28:47
05:28:47 [Fail] RuntimeValidatedKafka [BeforeEach] Kafka Policy Ingress
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/kafka.go:116
05:28:47
05:28:47 [Fail] RuntimeValidatedKafka [BeforeEach] Kafka Policy Role Ingress
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/kafka.go:116
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] L3/L4 Checks
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:421
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] L4Policy Checks
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] L7 Checks
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] Checks CIDR Policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] Extended HTTP Methods tests
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicies [BeforeEach] Tests Egress To World
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:224
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Default [It] Default values
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:101
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Default [It] Default to Always without policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:117
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Default [It] Default to Always with policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:160
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Default [It] Default to Never without policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:176
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Always [It] Container creation
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:238
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Always [It] Always to Never with policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:253
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Always [It] Always to Never without policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:288
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Never [It] Container creation
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:327
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Never [It] Never to default with policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:343
05:28:47
05:28:47 [Fail] RuntimeValidatedPolicyEnforcement Policy Enforcement Never [It] Never to default without policy
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:380
05:28:47
05:28:47 [Fail] RuntimeValidatedConntrackTest [It] Conntrack-related configuration options for endpoints
05:28:47 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/connectivity.go:662

Assigning to myself for triage.

Actually, the first failure was in an earlier test:
04:30:36 time="2018-04-10T04:30:27Z" level=error msg="policy wait failed: \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              \rWaiting for endpoints to run policy revision 17: 6/7              Error: \n1 endpoints have failed regeneration after 1m0.511777069s\n\nError: \n1 endpoints have failed regeneration after 1m0.511777069s\n\n"
04:30:36 time="2018-04-10T04:30:27Z" level=info msg="Deleting all policy in agent"
04:30:36 time="2018-04-10T04:30:27Z" level=debug msg="running command: sudo cilium policy delete --all -o json | jq '.revision'"
04:30:36 time="2018-04-10T04:30:27Z" level=debug msg="running command: sudo cilium policy wait 18"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo journalctl -au cilium --since '376.445478814 seconds ago'"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo cilium endpoint list"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo cilium service list"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo cilium endpoint list"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo cilium bpf ct list global"
04:31:34 time="2018-04-10T04:31:28Z" level=debug msg="running command: sudo cilium bpf tunnel list"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: cilium kvstore get cilium --recursive"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: cilium service list -o json"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: cilium config"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: sudo cilium bpf lb list"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: cilium endpoint list -o json"
04:31:34 time="2018-04-10T04:31:29Z" level=debug msg="running command: cilium policy get"
04:31:34 time="2018-04-10T04:31:30Z" level=debug msg="running command: cilium status --all-controllers"
04:31:34 time="2018-04-10T04:31:30Z" level=debug msg="running command: sudo cilium-bugtool -t /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo journalctl -au cilium --no-pager"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo journalctl -au cilium-docker --no-pager"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo bash -c \"gops memstats $(pgrep cilium-agent)\""
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo bash -c \"gops stack $(pgrep cilium-agent)\""
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo bash -c \"gops stats $(pgrep cilium-agent)\""
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo docker logs cilium-consul"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo cp -r /var/run/cilium /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib"
04:31:39 time="2018-04-10T04:31:38Z" level=error msg="cannot gather files for cmd 'sudo cp -r /var/run/cilium /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib': cp: cannot create special file '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/xds.sock': Operation not permitted\ncp: cannot create special file '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/access_log.sock': Operation not permitted\ncp: cannot create fifo '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/events.sock': Operation not permitted\ncp: cannot create special file '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/monitor.sock': Operation not permitted\ncp: cannot create special file '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/cilium.sock': Operation not permitted\ncp: cannot create special file '/home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/lib/health.sock': Operation not permitted\n" test=RuntimeLB
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: sudo cp -r /var/lib/cilium /home/vagrant/go/src/github.com/cilium/cilium/test/test_results/RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled/run"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: docker ps -aq"
04:31:39 time="2018-04-10T04:31:38Z" level=debug msg="running command: docker logs c27ea932e132"
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 19d0df11838e"
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 6407d79c7a39"
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 18d19d503c69"
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs "
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs c7e1ebef1b91"
04:31:39 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 06c291ae317e"
04:31:40 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs c7896ae1e917"
04:31:40 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 3ea39a2b7373"
04:31:40 time="2018-04-10T04:31:39Z" level=debug msg="running command: docker logs 7065c7967961"
04:31:40 time="2018-04-10T04:31:39Z" level=debug msg="running command: sudo journalctl -au cilium | grep -qi -B 5 -A 5 deadlock"
04:31:40 time="2018-04-10T04:31:40Z" level=debug msg="running command: sudo ip addr del fd02:1:1:1:1:1:1:1/128 dev cilium_host"
04:31:40 time="2018-04-10T04:31:40Z" level=info msg="output of \"sudo ip addr del fd02:1:1:1:1:1:1:1/128 dev cilium_host\": RTNETLINK answers: Cannot assign requested address\nRTNETLINK answers: Cannot assign requested address\n"
04:31:40 time="2018-04-10T04:31:40Z" level=debug msg="running command: sudo ip link del dev lbtest1"
04:31:40 time="2018-04-10T04:31:40Z" level=info msg="output of \"sudo ip link del dev lbtest1\": Cannot find device \"lbtest1\"\nCannot find device \"lbtest1\"\n"
04:31:40
04:31:40 • Failure [388.304 seconds]
04:31:40 RuntimeValidatedLB
04:31:40 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:238
04:31:40   Services Policies
04:31:40   /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:238
04:31:40     tests with conntrack disabled [It]
04:31:40     /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:270
04:31:40
04:31:40     Expected
04:31:40         <*errors.errorString | 0xc42053e2b0>: {
04:31:40             s: "Timeout reached: could not import policy",
04:31:40         }
04:31:40     to be nil
04:31:40
04:31:40     /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/lb.go:384

Logs:
RuntimeValidatedLBServicesPoliciestestswithconntrackdisabled.zip

Timestamp where test failed:
04:30:36 time="2018-04-10T04:30:27Z" level=info msg="Deleting all policy in agent"

When searching the logs for the offending endpoint ID, I see the following error:
...
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="egress verdict: denied" endpointID=29381 labels="[reserved:cluster]" policyID=cluster
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="Updating policy BPF map: denying Identity" identity=host policymap=/sys/fs/bpf/tc/globals/cilium_policy_29381 trafficDirection=Ingress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=warning msg="Update of policy map failed" error="Unable to delete element: invalid argument" identity=host policymap=/sys/fs/bpf/tc/globals/cilium_policy_29381 trafficDirection=Ingress
...

I'm unsure exactly what is the 'invalid argument' here. But, this explains why the endpoint wait timed out. Luckily, it appears that we try to regenerate endpoints every five minutes if necessary, so the system was able to recover.

This is related to https://github.com/cilium/cilium/issues/3161 . I see #3161 as the logical fix to this, as I don't really know what caused map update to fail. @joestringer any ideas?
Reducing priority on this ticket as it's been triaged and we have a ticket which is tracking the fix (controller for map updates).

I'll take a look at the next test which failed to see what happened. However, there were so many failures here that it would be quite a novel on here to triage all of them :)

Archive is 145 MB :( and GitHub won't allow logs larger than 10 MB. Will attach logs from first test failure, which was:
04:37:53 • Failure [373.173 seconds]
04:37:53 RuntimeValidatedLB
04:37:53 /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:238
04:37:53   Services Policies
04:37:53   /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:238
04:37:53     tests with service performing L4 port mapping [It]
04:37:53     /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:270
04:37:53
04:37:53     Expected
04:37:53         <bool>: false
04:37:53     to be true
04:37:53
04:37:53     /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/lb.go:354
04:37:53 ------------------------------
04:37:53 SS
04:37:53 ------------------------------

Logs from first test failure: RuntimeValidatedLBServicesPoliciestestswithserviceperformingL4portmapping.zip

Console output from when test failed:
04:36:30 time="2018-04-10T04:36:30Z" level=info msg="'6' containers are in a 'ready' state of a total of '7' containers." functionName=WaitEndpointsReady status="map[ready:6 not-ready:1]" test=RuntimeLB
04:36:35 time="2018-04-10T04:36:35Z" level=debug msg="running command: cilium endpoint list -o jsonpath='{range [*]}{<EMAIL_ADDRESS>04:36:35 time="2018-04-10T04:36:35Z" level=info msg="'6' containers are in a 'ready' state of a total of '7' containers." functionName=WaitEndpointsReady status="map[ready:6 not-ready:1]" test=RuntimeLB
04:36:40 time="2018-04-10T04:36:40Z" level=warning msg="Endpoints are not ready after timeout" error="Timeout reached: Endpoints are not ready after timeout" functionName=WaitEndpointsReady test=RuntimeLB

Same thing as before, same endpoint is in not-ready state:
29381      Disabled           Disabled          22759      container:id.app2             f00d::a0f:0:0:72c5   <IP_ADDRESS>   not-ready

I don't think triaging the rest of the tests is a useful exercise at this time.

From the first test failure, I checked for logs related to endpoint 29381:
$ grep "29381" cilium.log | grep -v "Evaluating egress context\|Evaluating ingress context\|Evaluating context for destination PolicyID\|Evaluating context for source PolicyID\|Filtering CT map entry"
...
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=46898 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=20707 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=48640 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=2 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=1 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=4 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=3 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=22759 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=17388 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=46144 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=57673 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=31813 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=46898 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=20707 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=48640 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=2 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=1 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=22759 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=17388 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=46144 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=57673 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=31813 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=4 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
Apr 10 04:25:04 runtime cilium-agent[15189]: level=debug msg="deletion of old L4 policy failed" PolicyID.L4=3 containerID=c7e1ebef1b endpointID=29381 error="Unable to delete element: invalid argument" policyRevision=14 trafficDirection=Egress
...

Not much to go off of here; more logs are needed (#3666 tracks that).

We discussed on slack, from @borkmann :

For the hashtab delete I see two occasions where EINVAL can happen: i) if the fd is valid, but not a fd pointing to a BPF map, and ii) if the union bpf_attr that was passed to bpf syscall had non-zero bytes that came after the key member (meaning either 'value' or 'flags' member were non-zero).

In Cilium golang code we are sometimes setting old fds to 0, if I recall correctly, meaning 0 is still a valid fd, but not a BPF fd, seems plausible this could be the case somewhere

So potentially a 'use-after-close' ?

Given the log message above,

Timestamp where test failed:
04:30:36 time="2018-04-10T04:30:27Z" level=info msg="Deleting all policy in agent"

It seems like it is likely that policy is deleted, and we close the policymap for an endpoint, then there is some policy recomputation and it attempts to delete policies from the previous map (which now has fd 0).
Not sure why it gets set to not-ready state though - is it just that if a policymap update like this fails, we set the endpoint as not-ready in case there was a legitimate failure to apply policy?

@ianvernon OK, well in this case I think it would be safe to consider deletes on a map with fd=0 to be successful, because the map has been closed. (Though this does rely on the rest of the regeneration succeeding, otherwise the BPF program running on the device isn't replaced so even if userspace closed the map, the existing BPF program could still be referring to it..)

Similar issue was seen in https://github.com/cilium/cilium/issues/4052 .

Another build:  https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/792/testReport/junit/k8s-1/7/K8sValidatedKafkaPolicyTest_KafkaPolicies/

oh, I think I know what this is. The not-ready status is a synthesized state. It's driven by this line (https://github.com/cilium/cilium/blob/master/pkg/endpoint/endpoint.go#L772)
	if currentState == models.EndpointStateReady && e.Status.CurrentStatus() != OK {
		currentState = models.EndpointStateNotReady
	}

Since we failed earlier in this endpoint's lifetime, we are returning not OK from CurrentStatus (https://github.com/cilium/cilium/blob/master/pkg/endpoint/endpoint.go#L1192) and we follow a scheme where we errors trump OKs. I suspect I need to change that logic somewhat, or we missed a call to set the status in a way that returns OK.
I'll track this down.

IIRC @jrajahalme has some context as to why we synthesize not-ready state this way. I think the real issue though is that we still don't know why we see logs indicating BPF map updates fail.
The only actionable thing I can think of to do here is to work on https://github.com/cilium/cilium/issues/3666 n so we can attempt to debug this better when we get hit by it.

Another similar case: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2438/testReport/junit/(root)/runtime/RuntimeValidatedConntrackTest_Conntrack_related_configuration_options_for_endpoints/
Failure output:
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:269
Expected
    <*errors.errorString | 0xc420422120>: {
        s: "Timeout reached: could not import policy",
    }
to be nil
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/connectivity.go:628

Relevant build logs:
time="2018-05-13T20:45:31Z" level=error msg="policy wait failed: \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              \rWaiting for endpoints to run policy revision 89: 6/7              Error: \n1 endpoints have failed regeneration after 1m0.586900002s\n\n"

Logs:
RuntimeValidatedConntrackTestConntrack-relatedconfigurationoptionsforendpoints.zip

Another similar case: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2446/testReport/junit/(root)/runtime/RuntimeValidatedPolicies_L4Policy_Checks/
Failure output:
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:312
Expected
    <*errors.errorString | 0xc4202da4b0>: {
        s: "Timeout reached: could not import policy",
    }
to be nil
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/runtime/Policies.go:402

Relevant build logs:
time="2018-05-14T12:44:07Z" level=debug msg="running command: sudo cilium policy wait 24"
time="2018-05-14T12:45:08Z" level=error msg="policy wait failed: \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              \rWaiting for endpoints to run policy revision 24: 6/7              Error: \n1 endpoints have failed regeneration after 1m0.691022128s\n\n"

Logs:
RuntimeValidatedPoliciesL4PolicyChecks.zip

https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3409
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:338
Expected
    <*errors.errorString | 0xc420220800>: {
        s: "Timeout reached: could not import policy",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/runtime/lb.go:412

test_results_Cilium-PR-Ginkgo-Tests-Validated_3409.zip

@joestringer I'm seen this error quite a lot. In master:
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2536/testReport/junit/(root)/runtime/RuntimeValidatedLB_Services_Policies_tests_with_service_performing_L4_port_mapping/
The issue looks like that the endpoint is not regenerated correctly:
May 21 23:06:49 runtime cilium-agent[11388]: level=info msg="Regeneration of BPF program has completed" buildDuration=10.000827132s containerID=06ce38bb5f endpointID=38939 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:981b" k8sPodName=":" policyRevision=20
May 21 23:06:49 runtime cilium-agent[11388]: level=warning msg="Generating BPF for endpoint failed, keeping stale directory." containerID=06ce38bb5f endpointID=38939 file-path=38939_next_fail ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:981b" k8sPodName=":" policyRevision=20
May 21 23:06:49 runtime cilium-agent[11388]: level=debug msg="Completed endpoint regeneration with no pending regeneration requests" code=OK containerID=06ce38bb5f endpointID=38939 endpointState=ready ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:981b" k8sPodName=":" policyRevision=20 type=0
May 21 23:06:49 runtime cilium-agent[11388]: level=warning msg="Regeneration of endpoint program failed" containerID=06ce38bb5f endpointID=38939 error="Error while configuring proxy redirects: proxy state changes failed: context deadline exceeded" ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:981b" k8sPodName=":" policyRevision=19

Policy
[
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd1": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "reserved:host": ""
            }
          },
          {
            "matchLabels": {
              "any:id.app1": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              },
              {
                "port": "8080",
                "protocol": "TCP"
              }
            ],
            "rules": {
              "http": [
                {
                  "path": "/public",
                  "method": "GET"
                }
              ]
            }
          }
        ]
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd2": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "reserved:host": ""
            }
          },
          {
            "matchLabels": {
              "any:id.app2": ""
            }
          }
        ]
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.app2": ""
      }
    },
    "egress": [
      {
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              },
              {
                "port": "8080",
                "protocol": "TCP"
              }
            ],
            "rules": {
              "http": [
                {
                  "path": "/public",
                  "method": "GET"
                }
              ]
            }
          }
        ]
      }
    ]
  }
]

Endpoint list
cmd: "sudo cilium endpoint list" exitCode: 0
 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
173        Disabled           Disabled          15268      container:id.app3             f00d::a0f:0:0:ad     <IP_ADDRESS>   ready
13949      Enabled            Disabled          15761      container:id.httpd1           f00d::a0f:0:0:367d   <IP_ADDRESS>    ready
                                                           container:id.service1
29381      Enabled            Disabled          43141      container:id.httpd2           f00d::a0f:0:0:72c5   <IP_ADDRESS>   ready
                                                           container:id.service1
29898      Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:74ca   <IP_ADDRESS>    ready
38939      Disabled           Enabled           18814      container:id.app2             f00d::a0f:0:0:981b   <IP_ADDRESS>   not-ready
48896      Disabled           Disabled          4121       container:id.httpd3           f00d::a0f:0:0:bf00   <IP_ADDRESS>     ready
                                                           container:id.service1
56326      Disabled           Disabled          35620      container:id.app1             f00d::a0f:0:0:dc06   <IP_ADDRESS>    ready

@eloycoto This looks like an independent issue, I don't see either of the following messages in the original report on this issue:

Generating BPF for endpoint failed
Error while configuring proxy redirects: proxy state changes failed: context deadline exceeded

Please file a separate issue so we can track each failure independently. There are likely different root causes for this "Failed to import policy" message.

Hmm, that endpoint might have been deleted? The various errors seem related to that.
Endpoint Log 39811
Timestamp              Status    State                   Message
2018-05-28T17:25:13Z   Failure   ready                   Error regenerating endpoint: unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument]

Controller Status:   38/39 healthy
  Name                                 Last success   Last error   Count   Message
...
  sync-policymap-39811                 never          8s ago       11      synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]

BPF CT List 39811
Error: Unable to open /sys/fs/bpf/tc/globals/cilium_ct6_39811: Unable to get object /sys/fs/bpf/tc/globals/cilium_ct6_39811: no such file or directory

Finally, the cilium log shows it failing to build at 17:25, which is before the policy apply that failed in the test (while waiting for the ready state). Does anyone know what the invalid element mean?
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Starting regenerate..." containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="No change in CIDR policy" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="ingress and egress policy enforcement not enabled" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Removed update controller" name=sync-policymap-39811 subsys=controller uuid=10e21e97-629c-11e8-8587-08002740ae4f
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Updated controller" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Done regenerating" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" optsChanged=true policyChanged=false policyRevision=17 policyRevision.next=18
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="TriggerPolicyUpdatesLocked: changed: true" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Triggering endpoint regeneration due to policy updates" code=OK containerID=59d85a1c85 endpointID=39811 endpointState=waiting-to-regenerate ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17 type=0
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Dequeued endpoint from build queue" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Regenerating endpoint..." containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=info msg="Regenerating BPF program" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17 startTime="2018-05-28 17:25:13.553793142 +0000 UTC m=+115.114045597"
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Regenerating Endpoint BPF: endpoint policy updated & changes were needed" code=OK containerID=59d85a1c85 endpointID=39811 endpointState=regenerating ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17 type=0
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Starting regenerate..." containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="skipping policy recalculation" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyChanged=true policyRevision=17 policyRevision.next=18 policyRevision.repo=18
May 28 17:25:13 runtime cilium-agent[7236]: level=info msg="Regeneration of BPF program has completed" buildDuration="283.032µs" containerID=59d85a1c85 endpointID=39811 ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=warning msg="Generating BPF for endpoint failed, keeping stale directory." containerID=59d85a1c85 endpointID=39811 file-path=39811_next_fail ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Completed endpoint regeneration with no pending regeneration requests" code=OK containerID=59d85a1c85 endpointID=39811 endpointState=ready ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17 type=0
May 28 17:25:13 runtime cilium-agent[7236]: level=warning msg="Regeneration of endpoint program failed" containerID=59d85a1c85 endpointID=39811 error="unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument]" ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Error regenerating endpoint: unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument]" code=Failure containerID=59d85a1c85 endpointID=39811 endpointState=ready ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:9b83" k8sPodName=":" policyRevision=17 type=200
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Shutting down controller for stopForUpdate; skipping stop function" name=sync-policymap-39811 subsys=controller uuid=10e21e97-629c-11e8-8587-08002740ae4f
May 28 17:25:13 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=1 error="synchronizing desired PolicyMap state failed: [Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument Unable to update element: invalid argument]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:14 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=2 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:16 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=3 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:19 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=4 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:21 runtime cilium-agent[7236]: level=debug msg="Refreshing labels of endpoint" containerID=59d85a1c85 endpointID=39811 identityLabels="container:id.httpd1,container:id.service1" infoLabels=
May 28 17:25:23 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=5 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:28 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=6 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:34 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=7 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:41 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=8 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:49 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=9 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:25:51 runtime cilium-agent[7236]: level=debug msg="Refreshing labels of endpoint" containerID=59d85a1c85 endpointID=39811 identityLabels="container:id.httpd1,container:id.service1" infoLabels=
May 28 17:25:58 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=10 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:26:08 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=11 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:26:19 runtime cilium-agent[7236]: level=debug msg="Controller run failed" consecutiveErrors=12 error="synchronizing desired PolicyMap state failed: [Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor Unable to update element: bad file descriptor]" name=sync-policymap-39811 subsys=controller uuid=14a45d21-629c-11e8-8587-08002740ae4f
May 28 17:26:21 runtime cilium-agent[7236]: level=debug msg="GET /endpoint/{id} request" endpointID=39811
May 28 17:26:21 runtime cilium-agent[7236]: level=debug msg="Refreshing labels of endpoint" containerID=59d85a1c85 endpointID=39811 identityLabels="container:id.httpd1,container:id.service1" infoLabels=
May 28 17:26:21 runtime cilium-agent[7236]: level=debug msg="GET /endpoint/{id}/log request" endpointID=39811
May 28 17:26:21 runtime cilium-agent[7236]: level=debug msg="GET /endpoint/{id}/log request" endpointID=39811

Oh, I guess if we fail on the first ever regeneration we wouldn't create a state directory, but only the _new (or whatever) equivalent....

Seen again in build: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/3768/execution/node/63/log/?consoleFull

@ashwinp Could you explain how you linked that failure to this issue? Looking through build 3768's logs I only see errors bringing up cilium to begin with, I'm not even sure any tests executed at all.
test_results_Cilium-PR-Ginkgo-Tests-Validated_3768.zip

@ianvernon @raybejjani another failure:
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2710/
afd3a30c_RuntimeValidatedPolicies_L3-L4_Checks.zip
Endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
2345       Disabled           Disabled          13336      container:id.app3             f00d::a0f:0:0:929    <IP_ADDRESS>   not-ready
9412       Disabled           Disabled          59810      container:id.httpd2           f00d::a0f:0:0:24c4   <IP_ADDRESS>     ready
                                                           container:id.service1
21116      Disabled           Disabled          5981       container:id.app2             f00d::a0f:0:0:527c   <IP_ADDRESS>     ready
21750      Disabled           Disabled          41844      container:id.app1             f00d::a0f:0:0:54f6   <IP_ADDRESS>   ready
37895      Disabled           Disabled          44346      container:id.httpd1           f00d::a0f:0:0:9407   <IP_ADDRESS>    ready
                                                           container:id.service1
42118      Disabled           Disabled          41878      container:id.httpd3           f00d::a0f:0:0:a486   <IP_ADDRESS>   ready
                                                           container:id.service1
47371      Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:b90b   <IP_ADDRESS>      ready

BPF Policy Get 2345
DIRECTION   LABELS (source:key[=value])   PORT/PROTO   BYTES   PACKETS
Ingress     container:id.httpd2           ANY          0       0
            container:id.service1
Ingress     36629                         ANY          0       0
Ingress     container:id.app1             ANY          0       0
Egress      container:id.httpd2           8000/TCP     0       0
            container:id.service1
Ingress     reserved:world                ANY          0       0
Ingress     1571                          ANY          0       0
Ingress     10708                         ANY          0       0
Ingress     container:id.app2             ANY          0       0
Ingress     reserved:cluster              ANY          0       0
Ingress     16574                         ANY          0       0
Egress      container:id.httpd2           8080/UDP     0       0
            container:id.service1
Ingress     reserved:health               ANY          0       0
Ingress     28940                         ANY          0       0
Ingress     reserved:host                 ANY          0       0
Egress      container:id.httpd2           80/TCP       1099    12
            container:id.service1
Ingress     12333                         ANY          0       0
Ingress     container:id.app3             ANY          0       0
Ingress     933                           ANY          0       0
Ingress     1437                          ANY          0       0
Egress      container:id.httpd2           8080/TCP     0       0
            container:id.service1
Ingress     57874                         ANY          0       0
Ingress     reserved:init                 ANY          0       0
Ingress     container:id.httpd1           ANY          0       0
            container:id.service1
Ingress     container:id.httpd3           ANY          0       0
            container:id.service1

BPF CT List 2345
Error: Unable to open /sys/fs/bpf/tc/globals/cilium_ct6_2345: Unable to get object /sys/fs/bpf/tc/globals/cilium_ct6_2345: no such file or directory

Endpoint Get 2345
[
  {
    "id": 2345,
    "spec": {
      "label-configuration": {
        "user": []
      },
      "options": {
        "Conntrack": "Enabled",
        "ConntrackAccounting": "Enabled",
        "ConntrackLocal": "Disabled",
        "Debug": "Enabled",
        "DebugLB": "Disabled",
        "DropNotification": "Enabled",
        "EgressPolicy": "Disabled",
        "IngressPolicy": "Disabled",
        "NAT46": "Disabled",
        "TraceNotification": "Enabled"
      }
    },
    "status": {
      "controllers": [
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "resolve-identity-2345",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-06-05T10:58:31.589Z",
            "success-count": 1
          },
          "uuid": "627e1989-68af-11e8-a637-080027aa233b"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv4-identity-mapping (2345)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-06-05T10:58:31.593Z",
            "success-count": 1
          },
          "uuid": "627f7749-68af-11e8-a637-080027aa233b"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "5m0s"
          },
          "name": "sync-IPv6-identity-mapping (2345)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-06-05T10:58:31.592Z",
            "success-count": 1
          },
          "uuid": "627f82ca-68af-11e8-a637-080027aa233b"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "1m0s"
          },
          "name": "sync-identity-to-k8s-pod (2345)",
          "status": {
            "last-failure-timestamp": "0001-01-01T00:00:00.000Z",
            "last-success-timestamp": "2018-06-05T11:01:31.593Z",
            "success-count": 4
          },
          "uuid": "627f68d5-68af-11e8-a637-080027aa233b"
        },
        {
          "configuration": {
            "error-retry": true,
            "interval": "1m0s"
          },
          "name": "sync-policymap-2345",
          "status": {
            "consecutive-failure-count": 11,
            "failure-count": 11,
            "last-failure-msg": "synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]",
            "last-failure-timestamp": "2018-06-05T11:02:05.951Z",
            "last-success-timestamp": "0001-01-01T00:00:00.000Z"
          },
          "uuid": "c176c3ef-68af-11e8-a637-080027aa233b"
        }
      ],
      "external-identifiers": {
        "container-id": "99962c09444dc0385be8e52fa55e1ef9a90636412cfaf5888c909bd721c0908d",
        "container-name": "app3",
        "docker-endpoint-id": "a5324de6ab62f74a7ef4839ae10d13c983e4bda0b14dfca67f40c7199b58de82",
        "docker-network-id": "4ce6f9e08a07c410add30b8131c07a4c1bcf6e453dc010e8e0ab4003e88c10e4",
        "pod-name": ":"
      },
      "health": {
        "bpf": "Warning",
        "connected": true,
        "overallHealth": "Warning",
        "policy": "Warning"
      },
      "identity": {
        "id": 13336,
        "labels": [
          "container:id.app3"
        ],
        "labelsSHA256": "2186c66cd5d8fb631adf86d9ca4ce027fcb02f3de2e5de8868f83250f203240e"
      },
      "labels": {
        "derived": [],
        "disabled": [],
        "realized": {
          "user": []
        },
        "security-relevant": [
          "container:id.app3"
        ]
      },
      "log": [
        {
          "code": "Failure",
          "message": "Error regenerating endpoint: unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument]",
          "state": "ready",
          "timestamp": "2018-06-05T11:01:10Z"
        }
      ],
      "networking": {
        "addressing": [
          {
            "ipv4": "<IP_ADDRESS>",
            "ipv6": "f00d::a0f:0:0:929"
          }
        ],
        "host-mac": "46:bd:5f:73:df:4d",
        "interface-index": 107,
        "interface-name": "lxca5324",
        "mac": "5e:30:c9:6b:57:e9"
      },
      "policy": {
        "proxy-statistics": [],
        "realized": {
          "allowed-egress-identities": [],
          "allowed-ingress-identities": [
            5981,
            5,
            4,
            13336,
            59810,
            2,
            3,
            12333,
            41878,
            1437,
            36629,
            41844,
            1571,
            57874,
            10708,
            1,
            28940,
            16574,
            44346,
            933
          ],
          "build": 87,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 13336,
          "l4": {
            "egress": [
              {
                "derived-from-rules": [
                  []
                ],
                "rule": {
            "port": 8000,
            "protocol": "TCP"
          }
              },
              {
                "derived-from-rules": [
                  []
                ],
                "rule": {
            "port": 80,
            "protocol": "TCP"
          }
              },
              {
                "derived-from-rules": [
                  []
                ],
                "rule": {
            "port": 8080,
            "protocol": "TCP"
          }
              },
              {
                "derived-from-rules": [
                  []
                ],
                "rule": {
            "port": 8080,
            "protocol": "UDP"
          }
              }
            ],
            "ingress": []
          },
          "policy-enabled": "none",
          "policy-revision": 86
        },
        "spec": {
          "allowed-egress-identities": [
            3,
            1,
            4,
            59810,
            41878,
            5981,
            41844,
            2,
            13336,
            44346,
            5
          ],
          "allowed-ingress-identities": [
            2,
            5,
            1,
            4,
            13336,
            5981,
            59810,
            41844,
            44346,
            41878,
            3
          ],
          "build": 87,
          "cidr-policy": {
            "egress": [],
            "ingress": []
          },
          "id": 13336,
          "l4": {
            "egress": [],
            "ingress": []
          },
          "policy-enabled": "none",
          "policy-revision": 87
        }
      },
      "realized": {
        "label-configuration": {
          "user": []
        },
        "options": {
          "Conntrack": "Enabled",
          "ConntrackAccounting": "Enabled",
          "ConntrackLocal": "Disabled",
          "Debug": "Enabled",
          "DebugLB": "Disabled",
          "DropNotification": "Enabled",
          "EgressPolicy": "Disabled",
          "IngressPolicy": "Disabled",
          "NAT46": "Disabled",
          "TraceNotification": "Enabled"
        }
      },
      "state": "not-ready"
    }
  }
]

Endpoint Health 2345
Overall Health:   Warning
BPF Health:       Warning
Policy Health:    Warning
Connected:        yes

Endpoint Log 2345
Timestamp              Status    State                   Message
2018-06-05T11:01:10Z   Failure   ready                   Error regenerating endpoint: unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument]
2018-06-05T11:01:10Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:01:10Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:01:10Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:54Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T11:00:54Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:00:53Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:00:53Z   OK        waiting-to-regenerate   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T11:00:53Z   Warning   waiting-to-regenerate   Skipped invalid state transition to ready due to: Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:00:52Z   Warning   waiting-to-regenerate   Skipped invalid state transition to waiting-to-regenerate due to: Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   Warning   waiting-to-regenerate   Skipped invalid state transition to waiting-to-regenerate due to: Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   Warning   waiting-to-regenerate   Skipped invalid state transition to waiting-to-regenerate due to: Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   Warning   waiting-to-regenerate   Skipped invalid state transition to waiting-to-regenerate due to: Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:00:52Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:52Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T11:00:52Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:00:52Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:00:52Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:51Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T11:00:51Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:00:51Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:00:51Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T11:00:51Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T11:00:51Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T11:00:51Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T11:00:51Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T10:58:51Z   OK        ready                   Successfully regenerated endpoint program due to endpoint policy updated & changes were needed
2018-06-05T10:58:51Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T10:58:47Z   OK        regenerating            Regenerating Endpoint BPF: endpoint policy updated & changes were needed
2018-06-05T10:58:47Z   OK        waiting-to-regenerate   Triggering endpoint regeneration due to policy updates
2018-06-05T10:58:34Z   OK        ready                   Successfully regenerated endpoint program due to updated security labels
2018-06-05T10:58:34Z   OK        ready                   Completed endpoint regeneration with no pending regeneration requests
2018-06-05T10:58:33Z   OK        regenerating            Endpoint policy update skipped because no changes were needed
2018-06-05T10:58:31Z   OK        regenerating            Regenerating Endpoint BPF: updated security labels
2018-06-05T10:58:31Z   Warning   waiting-to-regenerate   Skipped invalid state transition to waiting-to-regenerate due to: Triggering regeneration due to new identity
2018-06-05T10:58:31Z   OK        waiting-to-regenerate   Triggering regeneration due to new identity
2018-06-05T10:58:31Z   OK        ready                   Set identity for this endpoint

This was hit in a recent build: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/2710/
Logs:
test_results-6.zip
Tons of errors in the logs with updating maps, which now contain the file descriptors of the maps:
Jun 05 10:52:20 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:20 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:22 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:22 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:23 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=55436 error="unable to delete element 55436 from map /sys/fs/bpf/tc/globals/cilium_policy_55436: [Unable to delete key f00d::a0f:0:0:d88c in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory Unable to delete key <IP_ADDRESS> in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory]"
Jun 05 10:52:23 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=55436 error="unable to close policymap /sys/fs/bpf/tc/globals/cilium_policy: bad file descriptor"
Jun 05 10:52:23 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:24 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:26 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:26 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:27 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:27 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:30 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:30 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:30 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:32 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:52:32 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:06 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:07 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:07 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:09 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:09 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:10 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:10 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:12 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:12 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:13 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:13 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:16 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:16 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:33 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:53:33 runtime cilium-agent[12708]: level=warning msg="Unable to remove proxymap entry after closing connection" error="Unable to delete element from map with file descriptor 9: no such file or directory"
Jun 05 10:55:45 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=62304 error="unable to delete element 62304 from map /sys/fs/bpf/tc/globals/cilium_policy_62304: [Unable to delete key f00d::a0f:0:0:f360 in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory Unable to delete key <IP_ADDRESS> in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory]"
Jun 05 10:55:45 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=62304 error="unable to close policymap /sys/fs/bpf/tc/globals/cilium_policy: bad file descriptor"
Jun 05 10:56:59 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=57775 error="unable to delete element 57775 from map /sys/fs/bpf/tc/globals/cilium_policy_57775: [Unable to delete key f00d::a0f:0:0:e1af in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory Unable to delete key <IP_ADDRESS> in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory]"
Jun 05 10:56:59 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=57775 error="unable to close policymap /sys/fs/bpf/tc/globals/cilium_policy: bad file descriptor"
Jun 05 10:58:22 runtime cilium-agent[12708]: level=warning msg="Ignoring error while deleting endpoint" endpointID=19257 error="unable to delete element 19257 from map /sys/fs/bpf/tc/globals/cilium_policy_19257: [Unable to delete key f00d::a0f:0:0:4b39 in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory Unable to delete key <IP_ADDRESS> in endpoint BPF map: Unable to delete element from map with file descriptor 6: no such file or directory]"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=warning msg="Regeneration of endpoint program failed" containerID=99962c0944 endpointID=2345 error="unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument]" ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:929" k8sPodName=":" policyRevision=86
Jun 05 11:01:10 runtime cilium-agent[12708]: level=debug msg="Error regenerating endpoint: unable to regenerate policy because PolicyMap synchronization failed: synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument]" code=Failure containerID=99962c0944 endpointID=2345 endpointState=ready ipv4=<IP_ADDRESS> ipv6="f00d::a0f:0:0:929" k8sPodName=":" policyRevision=86 type=200
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: invalid argument"
Jun 05 11:01:10 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=1 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument Unable to update element for map with file descriptor 21: invalid argument]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:11 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=2 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:13 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=3 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:16 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=4 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:20 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=5 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:25 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=6 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:31 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=7 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=44346) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=1) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=3) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41878) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5981) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=4) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=59810) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=2) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=41844) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:38 runtime cilium-agent[12708]: level=debug msg="Controller run failed" consecutiveErrors=8 error="synchronizing desired PolicyMap state failed: [Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor Unable to update element for map with file descriptor 21: bad file descriptor]" name=sync-policymap-2345 subsys=controller uuid=c176c3ef-68af-11e8-a637-080027aa233b
Jun 05 11:01:46 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=5) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"
Jun 05 11:01:46 runtime cilium-agent[12708]: level=error msg="failed to add key {%!s(uint32=13336) %!s(uint16=0) %!s(uint8=0) %!s(uint8=1)}: Unable to update element for map with file descriptor 21: bad file descriptor"