Workflow-run confirmation modal

Probably through a replacement of the open modal

This will be closed by f1767dc3


cilium-etcd-operator on eks.1, kube 1.11, health check for peer could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout

I provisioned cilium-etcd-operator yesterday in three different stacks following the repo README

Run fine on a self-built Kubespray stack based on EC2 instances,
Not ok on a Terraform newly created EKS stack following https://github.com/terraform-providers/terraform-provider-aws/tree/master/examples/eks-getting-started, here're logs

$ kubectl -n kube-system logs cilium-etcd-ksshl8n6sr
2018-12-19 03:30:04.505962 I | etcdmain: etcd Version: 3.3.10
2018-12-19 03:30:04.506981 I | etcdmain: Git SHA: 27fc7e2
2018-12-19 03:30:04.506988 I | etcdmain: Go Version: go1.10.4
2018-12-19 03:30:04.506994 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-19 03:30:04.507000 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-19 03:30:04.507061 I | embed: peerTLS: cert = /etc/etcdtls/member/peer-tls/peer.crt, key = /etc/etcdtls/member/peer-tls/peer.key, ca = , trusted-ca = /etc/etcdtls/member/peer-tls/peer-ca.crt, client-cert-auth = true, crl-file =
2018-12-19 03:30:04.507966 I | embed: listening for peers on https://<IP_ADDRESS>:2380
2018-12-19 03:30:04.508010 I | embed: listening for client requests on <IP_ADDRESS>:2379
2018-12-19 03:30:04.529947 W | etcdserver: could not get cluster response from https://cilium-etcd-h4cnnxtln8.cilium-etcd.kube-system.svc:2380: Get https://cilium-etcd-h4cnnxtln8.cilium-etcd.kube-system.svc:2380/members: EOF
2018-12-19 03:30:04.531075 C | etcdmain: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given urls

Not ok on a CloudFormation newly created EKS stack following https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html, here're logs

$ kubectl logs -n kube-system cilium-etcd-227hpxxjtg
2018-12-19 11:43:30.110922 I | etcdmain: etcd Version: 3.3.10
2018-12-19 11:43:30.110994 I | etcdmain: Git SHA: 27fc7e2
2018-12-19 11:43:30.111004 I | etcdmain: Go Version: go1.10.4
2018-12-19 11:43:30.111009 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-19 11:43:30.111015 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-19 11:43:30.111115 I | embed: peerTLS: cert = /etc/etcdtls/member/peer-tls/peer.crt, key = /etc/etcdtls/member/peer-tls/peer.key, ca = , trusted-ca = /etc/etcdtls/member/peer-tls/peer-ca.crt, client-cert-auth = true, crl-file =
2018-12-19 11:43:30.112731 I | embed: listening for peers on https://<IP_ADDRESS>:2380
2018-12-19 11:43:30.112781 I | embed: listening for client requests on <IP_ADDRESS>:2379
2018-12-19 11:43:30.130722 W | etcdserver: could not get cluster response from https://cilium-etcd-mxmqzcllq9.cilium-etcd.kube-system.svc:2380: Get https://cilium-etcd-mxmqzcllq9.cilium-etcd.kube-system.svc:2380/members: EOF
2018-12-19 11:43:30.131583 C | etcdmain: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given urls

Here are some more logs.

$ kubectl exec -it  -n kube-system cilium-etcd-7mjlsb8qj7 -- sh
/ # netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:46446         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      1 <IP_ADDRESS>:44440        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      1 <IP_ADDRESS>:44428        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      0 <IP_ADDRESS>:46068         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      1 <IP_ADDRESS>:44430        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      1 <IP_ADDRESS>:44426        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      0 <IP_ADDRESS>:46448         <IP_ADDRESS>:2379          ESTABLISHED 374/etcdctl
tcp        0      1 <IP_ADDRESS>:44438        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      0 <IP_ADDRESS>:46072         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      0 <IP_ADDRESS>:46404         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      0 <IP_ADDRESS>:46326         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      1 <IP_ADDRESS>:44436        <IP_ADDRESS>:2380         SYN_SENT    1/etcd
tcp        0      0 <IP_ADDRESS>:46232         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      0 <IP_ADDRESS>:46160         <IP_ADDRESS>:2379          TIME_WAIT   -
tcp        0      0 :::2379                 :::*                    LISTEN      1/etcd
tcp        0      0 :::2380                 :::*                    LISTEN      1/etcd
tcp        0      0 ::ffff:<IP_ADDRESS>:2379   ::ffff:<IP_ADDRESS>:46448  ESTABLISHED 1/etcd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
/ # etcdctl  --endpoints=https://localhost:2379 --cert-file=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key-file=/etc/etcdtls/operator/etcd-tls/etcd-client.key --ca-file=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt  cluster-health
member 23d0bb99aaba843 is unhealthy: got unhealthy result from https://cilium-etcd-7mjlsb8qj7.cilium-etcd.kube-system.svc:2379
member 8f9554046bd37e81 is unreachable: no available published client urls
cluster is unavailable
/ # wget https://localhost:2380
Connecting to localhost:2380 (<IP_ADDRESS>:2380)
ssl_client: localhost: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer
/ # wget https://localhost:2379
Connecting to localhost:2379 (<IP_ADDRESS>:2379)
ssl_client: localhost: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

$ kubectl exec -it cilium-etcd-vpdz7s2v7m -n kube-system -- sh
/ # /bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=
/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-
tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt e
ndpoint health
/ #
/ #
/ # /bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=
/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-
tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt e
ndpoint health
/ #

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS      RESTARTS   AGE
kube-system   aws-node-97cnj                   1/1     Running     0          13h
kube-system   aws-node-qb9hs                   1/1     Running     0          13h
kube-system   aws-node-sxbcn                   1/1     Running     0          13h
kube-system   cilium-etcd-gsc2pwb4kz           0/1     Completed   0          13h
kube-system   cilium-etcd-tvhvwvtss9           0/1     Error       0          13h
kube-system   coredns-6797b744bf-9xblw         1/1     Running     0          13h
kube-system   coredns-6797b744bf-sk585         1/1     Running     0          13h
kube-system   etcd-operator-759954d8db-pqx76   1/1     Running     0          13h
kube-system   kube-proxy-65b45                 1/1     Running     0          15h
kube-system   kube-proxy-7v6ss                 1/1     Running     0          15h
kube-system   kube-proxy-vmmvx                 1/1     Running     0          15h
kube-system   tiller-deploy-57f988f854-94png   1/1     Running     0          14h

$ kubectl logs cilium-etcd-gsc2pwb4kz  -n kube-system
2018-12-19 13:21:07.954321 I | etcdmain: etcd Version: 3.3.9
2018-12-19 13:21:07.954381 I | etcdmain: Git SHA: fca8add78
2018-12-19 13:21:07.954389 I | etcdmain: Go Version: go1.10.3
2018-12-19 13:21:07.954396 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-19 13:21:07.954401 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-19 13:21:07.954441 I | embed: peerTLS: cert = /etc/etcdtls/member/peer-tls/peer.crt, key = /etc/etcdtls/member/peer-tls/peer.key, ca = , trusted-ca = /etc/etcdtls/member/peer-tls/peer-ca.crt, client-cert-auth = true, crl-file =
2018-12-19 13:21:07.955095 I | embed: listening for peers on https://<IP_ADDRESS>:2380
2018-12-19 13:21:07.955138 I | embed: listening for client requests on <IP_ADDRESS>:2379
2018-12-19 13:21:07.961468 W | pkg/netutil: failed resolving host cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380 (lookup cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host); retrying in 1s
2018-12-19 13:21:08.963232 I | pkg/netutil: resolving cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380 to <IP_ADDRESS>:2380
2018-12-19 13:21:08.967573 W | pkg/netutil: failed resolving host cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380 (lookup cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host); retrying in 1s
2018-12-19 13:21:09.971499 W | pkg/netutil: failed resolving host cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380 (lookup cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host); retrying in 1s
2018-12-19 13:21:10.973445 I | pkg/netutil: resolving cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380 to <IP_ADDRESS>:2380
2018-12-19 13:21:10.973600 I | etcdserver: name = cilium-etcd-gsc2pwb4kz
2018-12-19 13:21:10.973614 I | etcdserver: data dir = /var/etcd/data
2018-12-19 13:21:10.973624 I | etcdserver: member dir = /var/etcd/data/member
2018-12-19 13:21:10.973650 I | etcdserver: heartbeat = 100ms
2018-12-19 13:21:10.973660 I | etcdserver: election = 1000ms
2018-12-19 13:21:10.973686 I | etcdserver: snapshot count = 100000
2018-12-19 13:21:10.973707 I | etcdserver: advertise client URLs = https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2379
2018-12-19 13:21:10.973734 I | etcdserver: initial advertise peer URLs = https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380
2018-12-19 13:21:10.973759 I | etcdserver: initial cluster = cilium-etcd-gsc2pwb4kz=https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380
2018-12-19 13:21:10.976183 I | etcdserver: starting member 906aa7ee78e8c45a in cluster e141811bd8f2dec9
2018-12-19 13:21:10.976255 I | raft: 906aa7ee78e8c45a became follower at term 0
2018-12-19 13:21:10.976276 I | raft: newRaft 906aa7ee78e8c45a [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018-12-19 13:21:10.976335 I | raft: 906aa7ee78e8c45a became follower at term 1
2018-12-19 13:21:10.982503 W | auth: simple token is not cryptographically signed
2018-12-19 13:21:10.986069 I | etcdserver: starting server... [version: 3.3.9, cluster version: to_be_decided]
2018-12-19 13:21:10.986673 I | etcdserver: 906aa7ee78e8c45a as single-node; fast-forwarding 9 ticks (election ticks 10)
2018-12-19 13:21:10.987269 I | etcdserver/membership: added member 906aa7ee78e8c45a [https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380] to cluster e141811bd8f2dec9
2018-12-19 13:21:10.990097 I | embed: ClientTLS: cert = /etc/etcdtls/member/server-tls/server.crt, key = /etc/etcdtls/member/server-tls/server.key, ca = , trusted-ca = /etc/etcdtls/member/server-tls/server-ca.crt, client-cert-auth = true, crl-file =
2018-12-19 13:21:11.776741 I | raft: 906aa7ee78e8c45a is starting a new election at term 1
2018-12-19 13:21:11.776778 I | raft: 906aa7ee78e8c45a became candidate at term 2
2018-12-19 13:21:11.776809 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 2
2018-12-19 13:21:11.776828 I | raft: 906aa7ee78e8c45a became leader at term 2
2018-12-19 13:21:11.776839 I | raft: raft.node: 906aa7ee78e8c45a elected leader 906aa7ee78e8c45a at term 2
2018-12-19 13:21:11.777142 I | etcdserver: published {Name:cilium-etcd-gsc2pwb4kz ClientURLs:[https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2379]} to cluster e141811bd8f2dec9
2018-12-19 13:21:11.777373 I | etcdserver: setting up the initial cluster version to 3.3
2018-12-19 13:21:11.777455 I | embed: ready to serve client requests
2018-12-19 13:21:11.777925 N | etcdserver/membership: set the initial cluster version to 3.3
2018-12-19 13:21:11.778102 I | etcdserver/api: enabled capabilities for version 3.3
2018-12-19 13:21:11.779946 I | embed: serving client requests on [::]:2379
2018-12-19 13:21:11.795963 I | embed: rejected connection from "<IP_ADDRESS>:42044" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2018/12/19 13:21:11 Failed to dial <IP_ADDRESS>:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
2018-12-19 13:21:20.142690 I | etcdserver/membership: added member e878b2893b78414d [https://cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc:2380] to cluster e141811bd8f2dec9
2018-12-19 13:21:20.142722 I | rafthttp: starting peer e878b2893b78414d...
2018-12-19 13:21:20.142784 I | rafthttp: started HTTP pipelining with peer e878b2893b78414d
2018-12-19 13:21:20.148223 I | rafthttp: started streaming with peer e878b2893b78414d (writer)
2018-12-19 13:21:20.148266 I | rafthttp: started streaming with peer e878b2893b78414d (writer)
2018-12-19 13:21:20.153644 I | rafthttp: started peer e878b2893b78414d
2018-12-19 13:21:20.153687 I | rafthttp: added peer e878b2893b78414d
2018-12-19 13:21:20.154213 I | rafthttp: started streaming with peer e878b2893b78414d (stream MsgApp v2 reader)
2018-12-19 13:21:20.155528 I | rafthttp: started streaming with peer e878b2893b78414d (stream Message reader)
2018-12-19 13:21:21.776705 W | raft: 906aa7ee78e8c45a stepped down to follower since quorum is not active
2018-12-19 13:21:21.776738 I | raft: 906aa7ee78e8c45a became follower at term 2
2018-12-19 13:21:21.776748 I | raft: raft.node: 906aa7ee78e8c45a lost leader 906aa7ee78e8c45a at term 2
2018-12-19 13:21:23.076744 I | raft: 906aa7ee78e8c45a is starting a new election at term 2
2018-12-19 13:21:23.076782 I | raft: 906aa7ee78e8c45a became candidate at term 3
2018-12-19 13:21:23.076797 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 3
2018-12-19 13:21:23.076877 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 3
2018-12-19 13:21:24.476697 I | raft: 906aa7ee78e8c45a is starting a new election at term 3
2018-12-19 13:21:24.476732 I | raft: 906aa7ee78e8c45a became candidate at term 4
2018-12-19 13:21:24.476745 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 4
2018-12-19 13:21:24.476808 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 4
2018-12-19 13:21:25.155854 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:25.199604 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988274913s) to execute
2018-12-19 13:21:26.376690 I | raft: 906aa7ee78e8c45a is starting a new election at term 4
2018-12-19 13:21:26.376726 I | raft: 906aa7ee78e8c45a became candidate at term 5
2018-12-19 13:21:26.376739 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 5
2018-12-19 13:21:26.376811 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 5
2018-12-19 13:21:27.212507 W | etcdserver: timed out waiting for read index response
2018-12-19 13:21:27.576715 I | raft: 906aa7ee78e8c45a is starting a new election at term 5
2018-12-19 13:21:27.576746 I | raft: 906aa7ee78e8c45a became candidate at term 6
2018-12-19 13:21:27.576758 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 6
2018-12-19 13:21:27.576769 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 6
2018-12-19 13:21:28.976701 I | raft: 906aa7ee78e8c45a is starting a new election at term 6
2018-12-19 13:21:28.976733 I | raft: 906aa7ee78e8c45a became candidate at term 7
2018-12-19 13:21:28.976746 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 7
2018-12-19 13:21:28.976826 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 7
2018-12-19 13:21:30.076731 I | raft: 906aa7ee78e8c45a is starting a new election at term 7
2018-12-19 13:21:30.076768 I | raft: 906aa7ee78e8c45a became candidate at term 8
2018-12-19 13:21:30.076781 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 8
2018-12-19 13:21:30.076800 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 8
2018-12-19 13:21:30.156068 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:30.249634 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988871693s) to execute
2018-12-19 13:21:30.250253 I | embed: rejected connection from "<IP_ADDRESS>:42146" (error "EOF", ServerName "")
2018-12-19 13:21:31.976705 I | raft: 906aa7ee78e8c45a is starting a new election at term 8
2018-12-19 13:21:31.976742 I | raft: 906aa7ee78e8c45a became candidate at term 9
2018-12-19 13:21:31.976757 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 9
2018-12-19 13:21:31.976769 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 9
2018-12-19 13:21:33.476673 I | raft: 906aa7ee78e8c45a is starting a new election at term 9
2018-12-19 13:21:33.476844 I | raft: 906aa7ee78e8c45a became candidate at term 10
2018-12-19 13:21:33.476867 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 10
2018-12-19 13:21:33.476880 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 10
2018-12-19 13:21:34.212718 W | etcdserver: timed out waiting for read index response
2018-12-19 13:21:34.876685 I | raft: 906aa7ee78e8c45a is starting a new election at term 10
2018-12-19 13:21:34.876720 I | raft: 906aa7ee78e8c45a became candidate at term 11
2018-12-19 13:21:34.876732 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 11
2018-12-19 13:21:34.876743 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 11
2018-12-19 13:21:35.156282 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:35.285618 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988782086s) to execute
2018-12-19 13:21:36.676734 I | raft: 906aa7ee78e8c45a is starting a new election at term 11
2018-12-19 13:21:36.676768 I | raft: 906aa7ee78e8c45a became candidate at term 12
2018-12-19 13:21:36.676783 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 12
2018-12-19 13:21:36.676890 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 12
2018-12-19 13:21:37.976703 I | raft: 906aa7ee78e8c45a is starting a new election at term 12
2018-12-19 13:21:37.976732 I | raft: 906aa7ee78e8c45a became candidate at term 13
2018-12-19 13:21:37.976745 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 13
2018-12-19 13:21:37.976760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 13
2018-12-19 13:21:39.276692 I | raft: 906aa7ee78e8c45a is starting a new election at term 13
2018-12-19 13:21:39.276726 I | raft: 906aa7ee78e8c45a became candidate at term 14
2018-12-19 13:21:39.276738 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 14
2018-12-19 13:21:39.276749 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 14
2018-12-19 13:21:40.156558 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:40.276713 I | raft: 906aa7ee78e8c45a is starting a new election at term 14
2018-12-19 13:21:40.276739 I | raft: 906aa7ee78e8c45a became candidate at term 15
2018-12-19 13:21:40.276751 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 15
2018-12-19 13:21:40.276888 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 15
2018-12-19 13:21:40.326534 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988732434s) to execute
2018-12-19 13:21:41.212959 W | etcdserver: timed out waiting for read index response
2018-12-19 13:21:42.176644 I | raft: 906aa7ee78e8c45a is starting a new election at term 15
2018-12-19 13:21:42.176677 I | raft: 906aa7ee78e8c45a became candidate at term 16
2018-12-19 13:21:42.176689 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 16
2018-12-19 13:21:42.176703 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 16
2018-12-19 13:21:43.876688 I | raft: 906aa7ee78e8c45a is starting a new election at term 16
2018-12-19 13:21:43.876722 I | raft: 906aa7ee78e8c45a became candidate at term 17
2018-12-19 13:21:43.876734 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 17
2018-12-19 13:21:43.876745 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 17
2018-12-19 13:21:45.156748 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:45.364608 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.991580094s) to execute
2018-12-19 13:21:45.368496 I | embed: rejected connection from "<IP_ADDRESS>:42220" (error "EOF", ServerName "")
2018-12-19 13:21:45.377544 I | raft: 906aa7ee78e8c45a is starting a new election at term 17
2018-12-19 13:21:45.377572 I | raft: 906aa7ee78e8c45a became candidate at term 18
2018-12-19 13:21:45.377585 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 18
2018-12-19 13:21:45.377606 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 18
2018-12-19 13:21:46.476704 I | raft: 906aa7ee78e8c45a is starting a new election at term 18
2018-12-19 13:21:46.476734 I | raft: 906aa7ee78e8c45a became candidate at term 19
2018-12-19 13:21:46.476748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 19
2018-12-19 13:21:46.476760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 19
2018-12-19 13:21:47.676702 I | raft: 906aa7ee78e8c45a is starting a new election at term 19
2018-12-19 13:21:47.676732 I | raft: 906aa7ee78e8c45a became candidate at term 20
2018-12-19 13:21:47.676745 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 20
2018-12-19 13:21:47.676756 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 20
2018-12-19 13:21:48.213159 W | etcdserver: timed out waiting for read index response
2018-12-19 13:21:48.976710 I | raft: 906aa7ee78e8c45a is starting a new election at term 20
2018-12-19 13:21:48.976739 I | raft: 906aa7ee78e8c45a became candidate at term 21
2018-12-19 13:21:48.976752 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 21
2018-12-19 13:21:48.976764 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 21
2018-12-19 13:21:50.157168 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp: lookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc on <IP_ADDRESS>:53: no such host
2018-12-19 13:21:50.276694 I | raft: 906aa7ee78e8c45a is starting a new election at term 21
2018-12-19 13:21:50.276718 I | raft: 906aa7ee78e8c45a became candidate at term 22
2018-12-19 13:21:50.276751 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 22
2018-12-19 13:21:50.276800 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 22
2018-12-19 13:21:50.441505 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.982443429s) to execute
2018-12-19 13:21:51.376685 I | raft: 906aa7ee78e8c45a is starting a new election at term 22
2018-12-19 13:21:51.376719 I | raft: 906aa7ee78e8c45a became candidate at term 23
2018-12-19 13:21:51.376732 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 23
2018-12-19 13:21:51.376819 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 23
2018-12-19 13:21:52.376708 I | raft: 906aa7ee78e8c45a is starting a new election at term 23
2018-12-19 13:21:52.376738 I | raft: 906aa7ee78e8c45a became candidate at term 24
2018-12-19 13:21:52.376750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 24
2018-12-19 13:21:52.376801 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 24
2018-12-19 13:21:53.876709 I | raft: 906aa7ee78e8c45a is starting a new election at term 24
2018-12-19 13:21:53.876744 I | raft: 906aa7ee78e8c45a became candidate at term 25
2018-12-19 13:21:53.876757 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 25
2018-12-19 13:21:53.876769 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 25
2018-12-19 13:21:54.150094 I | embed: rejected connection from "<IP_ADDRESS>:34042" (error "tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"*.cilium-etcd.kube-system.svc\" \"*.cilium-etcd.kube-system.svc.cluster.local\"]", ServerName "cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc", IPAddresses [], DNSNames ["*.cilium-etcd.kube-system.svc" "*.cilium-etcd.kube-system.svc.cluster.local"])
2018-12-19 13:21:54.876700 I | raft: 906aa7ee78e8c45a is starting a new election at term 25
2018-12-19 13:21:54.876731 I | raft: 906aa7ee78e8c45a became candidate at term 26
2018-12-19 13:21:54.876743 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 26
2018-12-19 13:21:54.876754 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 26
2018-12-19 13:21:55.157375 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
2018-12-19 13:21:55.213354 W | etcdserver: timed out waiting for read index response
2018-12-19 13:21:55.478966 I | embed: rejected connection from "<IP_ADDRESS>:42490" (error "EOF", ServerName "")
2018-12-19 13:21:55.479061 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.98808638s) to execute
2018-12-19 13:21:56.077448 I | raft: 906aa7ee78e8c45a is starting a new election at term 26
2018-12-19 13:21:56.077489 I | raft: 906aa7ee78e8c45a became candidate at term 27
2018-12-19 13:21:56.077505 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 27
2018-12-19 13:21:56.077536 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 27
2018-12-19 13:21:57.376695 I | raft: 906aa7ee78e8c45a is starting a new election at term 27
2018-12-19 13:21:57.376733 I | raft: 906aa7ee78e8c45a became candidate at term 28
2018-12-19 13:21:57.376748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 28
2018-12-19 13:21:57.376804 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 28
2018-12-19 13:21:58.576691 I | raft: 906aa7ee78e8c45a is starting a new election at term 28
2018-12-19 13:21:58.576724 I | raft: 906aa7ee78e8c45a became candidate at term 29
2018-12-19 13:21:58.576739 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 29
2018-12-19 13:21:58.576753 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 29
2018-12-19 13:22:00.157609 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:00.276712 I | raft: 906aa7ee78e8c45a is starting a new election at term 29
2018-12-19 13:22:00.276736 I | raft: 906aa7ee78e8c45a became candidate at term 30
2018-12-19 13:22:00.276766 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 30
2018-12-19 13:22:00.276841 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 30
2018-12-19 13:22:00.514538 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988916324s) to execute
2018-12-19 13:22:01.776694 I | raft: 906aa7ee78e8c45a is starting a new election at term 30
2018-12-19 13:22:01.776733 I | raft: 906aa7ee78e8c45a became candidate at term 31
2018-12-19 13:22:01.776771 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 31
2018-12-19 13:22:01.776810 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 31
2018-12-19 13:22:02.213557 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:02.876703 I | raft: 906aa7ee78e8c45a is starting a new election at term 31
2018-12-19 13:22:02.876734 I | raft: 906aa7ee78e8c45a became candidate at term 32
2018-12-19 13:22:02.876747 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 32
2018-12-19 13:22:02.876759 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 32
2018-12-19 13:22:04.676706 I | raft: 906aa7ee78e8c45a is starting a new election at term 32
2018-12-19 13:22:04.676740 I | raft: 906aa7ee78e8c45a became candidate at term 33
2018-12-19 13:22:04.676755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 33
2018-12-19 13:22:04.676767 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 33
2018-12-19 13:22:05.157814 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:05.548803 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.984037681s) to execute
2018-12-19 13:22:05.549898 I | embed: rejected connection from "<IP_ADDRESS>:42582" (error "EOF", ServerName "")
2018-12-19 13:22:05.876688 I | raft: 906aa7ee78e8c45a is starting a new election at term 33
2018-12-19 13:22:05.876722 I | raft: 906aa7ee78e8c45a became candidate at term 34
2018-12-19 13:22:05.876736 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 34
2018-12-19 13:22:05.876747 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 34
2018-12-19 13:22:07.476716 I | raft: 906aa7ee78e8c45a is starting a new election at term 34
2018-12-19 13:22:07.476749 I | raft: 906aa7ee78e8c45a became candidate at term 35
2018-12-19 13:22:07.476764 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 35
2018-12-19 13:22:07.476776 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 35
2018-12-19 13:22:07.883047 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988004564s) to execute
2018-12-19 13:22:07.888270 I | embed: rejected connection from "<IP_ADDRESS>:42600" (error "EOF", ServerName "localhost")
2018-12-19 13:22:09.213806 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:09.376740 I | raft: 906aa7ee78e8c45a is starting a new election at term 35
2018-12-19 13:22:09.376783 I | raft: 906aa7ee78e8c45a became candidate at term 36
2018-12-19 13:22:09.376798 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 36
2018-12-19 13:22:09.376810 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 36
2018-12-19 13:22:10.158101 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:10.585354 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.989028918s) to execute
2018-12-19 13:22:10.591441 I | embed: rejected connection from "<IP_ADDRESS>:42618" (error "read tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:42618: read: connection reset by peer", ServerName "localhost")
2018-12-19 13:22:10.976700 I | raft: 906aa7ee78e8c45a is starting a new election at term 36
2018-12-19 13:22:10.976731 I | raft: 906aa7ee78e8c45a became candidate at term 37
2018-12-19 13:22:10.976743 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 37
2018-12-19 13:22:10.976812 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 37
2018-12-19 13:22:11.976714 I | raft: 906aa7ee78e8c45a is starting a new election at term 37
2018-12-19 13:22:11.976753 I | raft: 906aa7ee78e8c45a became candidate at term 38
2018-12-19 13:22:11.976768 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 38
2018-12-19 13:22:11.976785 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 38
2018-12-19 13:22:13.676731 I | raft: 906aa7ee78e8c45a is starting a new election at term 38
2018-12-19 13:22:13.676768 I | raft: 906aa7ee78e8c45a became candidate at term 39
2018-12-19 13:22:13.676784 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 39
2018-12-19 13:22:13.676795 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 39
2018-12-19 13:22:14.776704 I | raft: 906aa7ee78e8c45a is starting a new election at term 39
2018-12-19 13:22:14.776738 I | raft: 906aa7ee78e8c45a became candidate at term 40
2018-12-19 13:22:14.776762 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 40
2018-12-19 13:22:14.776777 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 40
2018-12-19 13:22:15.158315 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:15.626430 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988288462s) to execute
2018-12-19 13:22:15.627659 I | embed: rejected connection from "<IP_ADDRESS>:42670" (error "EOF", ServerName "")
2018-12-19 13:22:16.214012 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:16.476717 I | raft: 906aa7ee78e8c45a is starting a new election at term 40
2018-12-19 13:22:16.476750 I | raft: 906aa7ee78e8c45a became candidate at term 41
2018-12-19 13:22:16.476764 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 41
2018-12-19 13:22:16.476825 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 41
2018-12-19 13:22:17.976697 I | raft: 906aa7ee78e8c45a is starting a new election at term 41
2018-12-19 13:22:17.976733 I | raft: 906aa7ee78e8c45a became candidate at term 42
2018-12-19 13:22:17.976748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 42
2018-12-19 13:22:17.976760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 42
2018-12-19 13:22:18.976707 I | raft: 906aa7ee78e8c45a is starting a new election at term 42
2018-12-19 13:22:18.976738 I | raft: 906aa7ee78e8c45a became candidate at term 43
2018-12-19 13:22:18.976749 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 43
2018-12-19 13:22:18.976760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 43
2018-12-19 13:22:20.158514 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:20.663524 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.989171457s) to execute
2018-12-19 13:22:20.676695 I | raft: 906aa7ee78e8c45a is starting a new election at term 43
2018-12-19 13:22:20.676719 I | raft: 906aa7ee78e8c45a became candidate at term 44
2018-12-19 13:22:20.676732 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 44
2018-12-19 13:22:20.676744 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 44
2018-12-19 13:22:22.076710 I | raft: 906aa7ee78e8c45a is starting a new election at term 44
2018-12-19 13:22:22.076744 I | raft: 906aa7ee78e8c45a became candidate at term 45
2018-12-19 13:22:22.076760 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 45
2018-12-19 13:22:22.076772 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 45
2018-12-19 13:22:23.214213 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:23.776712 I | raft: 906aa7ee78e8c45a is starting a new election at term 45
2018-12-19 13:22:23.776746 I | raft: 906aa7ee78e8c45a became candidate at term 46
2018-12-19 13:22:23.776761 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 46
2018-12-19 13:22:23.776773 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 46
2018-12-19 13:22:24.776708 I | raft: 906aa7ee78e8c45a is starting a new election at term 46
2018-12-19 13:22:24.776741 I | raft: 906aa7ee78e8c45a became candidate at term 47
2018-12-19 13:22:24.776755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 47
2018-12-19 13:22:24.776840 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 47
2018-12-19 13:22:25.158711 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:25.703908 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988498753s) to execute
2018-12-19 13:22:25.705153 I | embed: rejected connection from "<IP_ADDRESS>:42774" (error "EOF", ServerName "")
2018-12-19 13:22:26.176594 I | raft: 906aa7ee78e8c45a is starting a new election at term 47
2018-12-19 13:22:26.176633 I | raft: 906aa7ee78e8c45a became candidate at term 48
2018-12-19 13:22:26.176648 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 48
2018-12-19 13:22:26.176659 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 48
2018-12-19 13:22:27.176692 I | raft: 906aa7ee78e8c45a is starting a new election at term 48
2018-12-19 13:22:27.176728 I | raft: 906aa7ee78e8c45a became candidate at term 49
2018-12-19 13:22:27.176743 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 49
2018-12-19 13:22:27.176756 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 49
2018-12-19 13:22:28.176695 I | raft: 906aa7ee78e8c45a is starting a new election at term 49
2018-12-19 13:22:28.176729 I | raft: 906aa7ee78e8c45a became candidate at term 50
2018-12-19 13:22:28.176743 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 50
2018-12-19 13:22:28.176754 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 50
2018-12-19 13:22:29.276709 I | raft: 906aa7ee78e8c45a is starting a new election at term 50
2018-12-19 13:22:29.276744 I | raft: 906aa7ee78e8c45a became candidate at term 51
2018-12-19 13:22:29.276759 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 51
2018-12-19 13:22:29.276771 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 51
2018-12-19 13:22:30.158907 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:30.214418 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:30.742490 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988806669s) to execute
2018-12-19 13:22:30.743715 I | embed: rejected connection from "<IP_ADDRESS>:42810" (error "EOF", ServerName "")
2018-12-19 13:22:30.876690 I | raft: 906aa7ee78e8c45a is starting a new election at term 51
2018-12-19 13:22:30.876721 I | raft: 906aa7ee78e8c45a became candidate at term 52
2018-12-19 13:22:30.876734 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 52
2018-12-19 13:22:30.876745 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 52
2018-12-19 13:22:32.276705 I | raft: 906aa7ee78e8c45a is starting a new election at term 52
2018-12-19 13:22:32.276741 I | raft: 906aa7ee78e8c45a became candidate at term 53
2018-12-19 13:22:32.276755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 53
2018-12-19 13:22:32.276809 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 53
2018-12-19 13:22:33.876726 I | raft: 906aa7ee78e8c45a is starting a new election at term 53
2018-12-19 13:22:33.876758 I | raft: 906aa7ee78e8c45a became candidate at term 54
2018-12-19 13:22:33.876772 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 54
2018-12-19 13:22:33.876792 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 54
2018-12-19 13:22:35.159108 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:35.376699 I | raft: 906aa7ee78e8c45a is starting a new election at term 54
2018-12-19 13:22:35.376735 I | raft: 906aa7ee78e8c45a became candidate at term 55
2018-12-19 13:22:35.376750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 55
2018-12-19 13:22:35.376762 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 55
2018-12-19 13:22:35.779801 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988837588s) to execute
2018-12-19 13:22:36.476697 I | raft: 906aa7ee78e8c45a is starting a new election at term 55
2018-12-19 13:22:36.476734 I | raft: 906aa7ee78e8c45a became candidate at term 56
2018-12-19 13:22:36.476750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 56
2018-12-19 13:22:36.476762 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 56
2018-12-19 13:22:37.214630 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:37.776689 I | raft: 906aa7ee78e8c45a is starting a new election at term 56
2018-12-19 13:22:37.776725 I | raft: 906aa7ee78e8c45a became candidate at term 57
2018-12-19 13:22:37.776740 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 57
2018-12-19 13:22:37.776752 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 57
2018-12-19 13:22:38.776692 I | raft: 906aa7ee78e8c45a is starting a new election at term 57
2018-12-19 13:22:38.776733 I | raft: 906aa7ee78e8c45a became candidate at term 58
2018-12-19 13:22:38.776748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 58
2018-12-19 13:22:38.776761 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 58
2018-12-19 13:22:40.159306 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:40.176705 I | raft: 906aa7ee78e8c45a is starting a new election at term 58
2018-12-19 13:22:40.176735 I | raft: 906aa7ee78e8c45a became candidate at term 59
2018-12-19 13:22:40.176749 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 59
2018-12-19 13:22:40.176796 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 59
2018-12-19 13:22:40.819646 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.989427149s) to execute
2018-12-19 13:22:40.825814 I | embed: rejected connection from "<IP_ADDRESS>:42900" (error "EOF", ServerName "")
2018-12-19 13:22:41.176698 I | raft: 906aa7ee78e8c45a is starting a new election at term 59
2018-12-19 13:22:41.176735 I | raft: 906aa7ee78e8c45a became candidate at term 60
2018-12-19 13:22:41.176750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 60
2018-12-19 13:22:41.176763 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 60
2018-12-19 13:22:42.676696 I | raft: 906aa7ee78e8c45a is starting a new election at term 60
2018-12-19 13:22:42.676732 I | raft: 906aa7ee78e8c45a became candidate at term 61
2018-12-19 13:22:42.676748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 61
2018-12-19 13:22:42.676760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 61
2018-12-19 13:22:44.076736 I | raft: 906aa7ee78e8c45a is starting a new election at term 61
2018-12-19 13:22:44.076772 I | raft: 906aa7ee78e8c45a became candidate at term 62
2018-12-19 13:22:44.076786 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 62
2018-12-19 13:22:44.076799 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 62
2018-12-19 13:22:44.214842 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:45.159505 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:45.676707 I | raft: 906aa7ee78e8c45a is starting a new election at term 62
2018-12-19 13:22:45.676741 I | raft: 906aa7ee78e8c45a became candidate at term 63
2018-12-19 13:22:45.676756 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 63
2018-12-19 13:22:45.676768 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 63
2018-12-19 13:22:45.864111 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988729038s) to execute
2018-12-19 13:22:47.376703 I | raft: 906aa7ee78e8c45a is starting a new election at term 63
2018-12-19 13:22:47.376738 I | raft: 906aa7ee78e8c45a became candidate at term 64
2018-12-19 13:22:47.376780 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 64
2018-12-19 13:22:47.376797 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 64
2018-12-19 13:22:48.876693 I | raft: 906aa7ee78e8c45a is starting a new election at term 64
2018-12-19 13:22:48.876727 I | raft: 906aa7ee78e8c45a became candidate at term 65
2018-12-19 13:22:48.876741 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 65
2018-12-19 13:22:48.876753 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 65
2018-12-19 13:22:50.159705 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:50.676700 I | raft: 906aa7ee78e8c45a is starting a new election at term 65
2018-12-19 13:22:50.676734 I | raft: 906aa7ee78e8c45a became candidate at term 66
2018-12-19 13:22:50.676749 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 66
2018-12-19 13:22:50.676836 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 66
2018-12-19 13:22:50.900992 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988886142s) to execute
2018-12-19 13:22:51.215062 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:52.376694 I | raft: 906aa7ee78e8c45a is starting a new election at term 66
2018-12-19 13:22:52.376728 I | raft: 906aa7ee78e8c45a became candidate at term 67
2018-12-19 13:22:52.376742 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 67
2018-12-19 13:22:52.376754 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 67
2018-12-19 13:22:53.376694 I | raft: 906aa7ee78e8c45a is starting a new election at term 67
2018-12-19 13:22:53.376728 I | raft: 906aa7ee78e8c45a became candidate at term 68
2018-12-19 13:22:53.376744 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 68
2018-12-19 13:22:53.376756 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 68
2018-12-19 13:22:54.476615 I | raft: 906aa7ee78e8c45a is starting a new election at term 68
2018-12-19 13:22:54.476650 I | raft: 906aa7ee78e8c45a became candidate at term 69
2018-12-19 13:22:54.476665 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 69
2018-12-19 13:22:54.476679 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 69
2018-12-19 13:22:55.159915 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:22:55.576719 I | raft: 906aa7ee78e8c45a is starting a new election at term 69
2018-12-19 13:22:55.576755 I | raft: 906aa7ee78e8c45a became candidate at term 70
2018-12-19 13:22:55.576771 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 70
2018-12-19 13:22:55.576783 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 70
2018-12-19 13:22:55.941286 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.987821599s) to execute
2018-12-19 13:22:56.876694 I | raft: 906aa7ee78e8c45a is starting a new election at term 70
2018-12-19 13:22:56.876721 I | raft: 906aa7ee78e8c45a became candidate at term 71
2018-12-19 13:22:56.876736 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 71
2018-12-19 13:22:56.876748 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 71
2018-12-19 13:22:58.215271 W | etcdserver: timed out waiting for read index response
2018-12-19 13:22:58.276690 I | raft: 906aa7ee78e8c45a is starting a new election at term 71
2018-12-19 13:22:58.276720 I | raft: 906aa7ee78e8c45a became candidate at term 72
2018-12-19 13:22:58.276745 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 72
2018-12-19 13:22:58.276760 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 72
2018-12-19 13:22:59.476690 I | raft: 906aa7ee78e8c45a is starting a new election at term 72
2018-12-19 13:22:59.476724 I | raft: 906aa7ee78e8c45a became candidate at term 73
2018-12-19 13:22:59.476740 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 73
2018-12-19 13:22:59.476752 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 73
2018-12-19 13:23:00.160115 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:00.776717 I | raft: 906aa7ee78e8c45a is starting a new election at term 73
2018-12-19 13:23:00.776754 I | raft: 906aa7ee78e8c45a became candidate at term 74
2018-12-19 13:23:00.776770 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 74
2018-12-19 13:23:00.776856 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 74
2018-12-19 13:23:00.980292 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.989941694s) to execute
2018-12-19 13:23:02.276718 I | raft: 906aa7ee78e8c45a is starting a new election at term 74
2018-12-19 13:23:02.276752 I | raft: 906aa7ee78e8c45a became candidate at term 75
2018-12-19 13:23:02.276790 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 75
2018-12-19 13:23:02.276830 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 75
2018-12-19 13:23:03.576707 I | raft: 906aa7ee78e8c45a is starting a new election at term 75
2018-12-19 13:23:03.576740 I | raft: 906aa7ee78e8c45a became candidate at term 76
2018-12-19 13:23:03.576755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 76
2018-12-19 13:23:03.576807 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 76
2018-12-19 13:23:05.160315 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:05.215485 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:05.376704 I | raft: 906aa7ee78e8c45a is starting a new election at term 76
2018-12-19 13:23:05.376730 I | raft: 906aa7ee78e8c45a became candidate at term 77
2018-12-19 13:23:05.376763 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 77
2018-12-19 13:23:05.376790 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 77
2018-12-19 13:23:06.013427 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.987357383s) to execute
2018-12-19 13:23:06.020278 I | embed: rejected connection from "<IP_ADDRESS>:43130" (error "EOF", ServerName "localhost")
2018-12-19 13:23:06.376702 I | raft: 906aa7ee78e8c45a is starting a new election at term 77
2018-12-19 13:23:06.376745 I | raft: 906aa7ee78e8c45a became candidate at term 78
2018-12-19 13:23:06.376759 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 78
2018-12-19 13:23:06.376770 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 78
2018-12-19 13:23:07.376693 I | raft: 906aa7ee78e8c45a is starting a new election at term 78
2018-12-19 13:23:07.376730 I | raft: 906aa7ee78e8c45a became candidate at term 79
2018-12-19 13:23:07.376773 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 79
2018-12-19 13:23:07.376790 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 79
2018-12-19 13:23:07.881031 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988592376s) to execute
2018-12-19 13:23:09.076691 I | raft: 906aa7ee78e8c45a is starting a new election at term 79
2018-12-19 13:23:09.076723 I | raft: 906aa7ee78e8c45a became candidate at term 80
2018-12-19 13:23:09.076737 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 80
2018-12-19 13:23:09.076748 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 80
2018-12-19 13:23:10.160493 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:10.576704 I | raft: 906aa7ee78e8c45a is starting a new election at term 80
2018-12-19 13:23:10.576745 I | raft: 906aa7ee78e8c45a became candidate at term 81
2018-12-19 13:23:10.576789 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 81
2018-12-19 13:23:10.576812 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 81
2018-12-19 13:23:11.056522 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988482818s) to execute
2018-12-19 13:23:12.176710 I | raft: 906aa7ee78e8c45a is starting a new election at term 81
2018-12-19 13:23:12.176818 I | raft: 906aa7ee78e8c45a became candidate at term 82
2018-12-19 13:23:12.176946 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 82
2018-12-19 13:23:12.178796 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 82
2018-12-19 13:23:12.215681 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:13.876720 I | raft: 906aa7ee78e8c45a is starting a new election at term 82
2018-12-19 13:23:13.876754 I | raft: 906aa7ee78e8c45a became candidate at term 83
2018-12-19 13:23:13.876769 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 83
2018-12-19 13:23:13.876781 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 83
2018-12-19 13:23:14.876714 I | raft: 906aa7ee78e8c45a is starting a new election at term 83
2018-12-19 13:23:14.876747 I | raft: 906aa7ee78e8c45a became candidate at term 84
2018-12-19 13:23:14.876759 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 84
2018-12-19 13:23:14.876770 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 84
2018-12-19 13:23:15.160700 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:16.093161 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.984580541s) to execute
2018-12-19 13:23:16.676663 I | raft: 906aa7ee78e8c45a is starting a new election at term 84
2018-12-19 13:23:16.676708 I | raft: 906aa7ee78e8c45a became candidate at term 85
2018-12-19 13:23:16.676724 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 85
2018-12-19 13:23:16.676739 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 85
2018-12-19 13:23:18.576715 I | raft: 906aa7ee78e8c45a is starting a new election at term 85
2018-12-19 13:23:18.576749 I | raft: 906aa7ee78e8c45a became candidate at term 86
2018-12-19 13:23:18.576789 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 86
2018-12-19 13:23:18.576821 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 86
2018-12-19 13:23:19.215826 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:19.676703 I | raft: 906aa7ee78e8c45a is starting a new election at term 86
2018-12-19 13:23:19.676741 I | raft: 906aa7ee78e8c45a became candidate at term 87
2018-12-19 13:23:19.676755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 87
2018-12-19 13:23:19.676768 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 87
2018-12-19 13:23:20.160902 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:20.876692 I | raft: 906aa7ee78e8c45a is starting a new election at term 87
2018-12-19 13:23:20.876721 I | raft: 906aa7ee78e8c45a became candidate at term 88
2018-12-19 13:23:20.876734 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 88
2018-12-19 13:23:20.876793 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 88
2018-12-19 13:23:21.136371 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.989212378s) to execute
2018-12-19 13:23:22.376690 I | raft: 906aa7ee78e8c45a is starting a new election at term 88
2018-12-19 13:23:22.376728 I | raft: 906aa7ee78e8c45a became candidate at term 89
2018-12-19 13:23:22.376743 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 89
2018-12-19 13:23:22.376756 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 89
2018-12-19 13:23:23.476708 I | raft: 906aa7ee78e8c45a is starting a new election at term 89
2018-12-19 13:23:23.476740 I | raft: 906aa7ee78e8c45a became candidate at term 90
2018-12-19 13:23:23.476755 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 90
2018-12-19 13:23:23.476766 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 90
2018-12-19 13:23:24.476640 I | raft: 906aa7ee78e8c45a is starting a new election at term 90
2018-12-19 13:23:24.476672 I | raft: 906aa7ee78e8c45a became candidate at term 91
2018-12-19 13:23:24.476685 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 91
2018-12-19 13:23:24.476701 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 91
2018-12-19 13:23:25.161102 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:26.172371 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.989228736s) to execute
2018-12-19 13:23:26.215967 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:26.276868 I | raft: 906aa7ee78e8c45a is starting a new election at term 91
2018-12-19 13:23:26.276895 I | raft: 906aa7ee78e8c45a became candidate at term 92
2018-12-19 13:23:26.276909 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 92
2018-12-19 13:23:26.276922 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 92
2018-12-19 13:23:27.776693 I | raft: 906aa7ee78e8c45a is starting a new election at term 92
2018-12-19 13:23:27.776731 I | raft: 906aa7ee78e8c45a became candidate at term 93
2018-12-19 13:23:27.776746 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 93
2018-12-19 13:23:27.776759 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 93
2018-12-19 13:23:29.676699 I | raft: 906aa7ee78e8c45a is starting a new election at term 93
2018-12-19 13:23:29.676735 I | raft: 906aa7ee78e8c45a became candidate at term 94
2018-12-19 13:23:29.676773 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 94
2018-12-19 13:23:29.676791 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 94
2018-12-19 13:23:30.161306 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:30.976696 I | raft: 906aa7ee78e8c45a is starting a new election at term 94
2018-12-19 13:23:30.976729 I | raft: 906aa7ee78e8c45a became candidate at term 95
2018-12-19 13:23:30.976742 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 95
2018-12-19 13:23:30.976763 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 95
2018-12-19 13:23:31.208283 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.98841629s) to execute
2018-12-19 13:23:32.077329 I | raft: 906aa7ee78e8c45a is starting a new election at term 95
2018-12-19 13:23:32.077807 I | raft: 906aa7ee78e8c45a became candidate at term 96
2018-12-19 13:23:32.077822 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 96
2018-12-19 13:23:32.077831 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 96
2018-12-19 13:23:33.216200 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:33.976629 I | raft: 906aa7ee78e8c45a is starting a new election at term 96
2018-12-19 13:23:33.976667 I | raft: 906aa7ee78e8c45a became candidate at term 97
2018-12-19 13:23:33.976683 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 97
2018-12-19 13:23:33.976696 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 97
2018-12-19 13:23:35.161502 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:35.276698 I | raft: 906aa7ee78e8c45a is starting a new election at term 97
2018-12-19 13:23:35.276721 I | raft: 906aa7ee78e8c45a became candidate at term 98
2018-12-19 13:23:35.276734 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 98
2018-12-19 13:23:35.276755 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 98
2018-12-19 13:23:36.249330 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.98843485s) to execute
2018-12-19 13:23:36.976700 I | raft: 906aa7ee78e8c45a is starting a new election at term 98
2018-12-19 13:23:36.976733 I | raft: 906aa7ee78e8c45a became candidate at term 99
2018-12-19 13:23:36.976749 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 99
2018-12-19 13:23:36.976770 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 99
2018-12-19 13:23:38.676698 I | raft: 906aa7ee78e8c45a is starting a new election at term 99
2018-12-19 13:23:38.676734 I | raft: 906aa7ee78e8c45a became candidate at term 100
2018-12-19 13:23:38.676750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 100
2018-12-19 13:23:38.676763 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 100
2018-12-19 13:23:39.876705 I | raft: 906aa7ee78e8c45a is starting a new election at term 100
2018-12-19 13:23:39.876747 I | raft: 906aa7ee78e8c45a became candidate at term 101
2018-12-19 13:23:39.876830 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 101
2018-12-19 13:23:39.876850 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 101
2018-12-19 13:23:40.161699 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:40.216341 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:41.176706 I | raft: 906aa7ee78e8c45a is starting a new election at term 101
2018-12-19 13:23:41.176741 I | raft: 906aa7ee78e8c45a became candidate at term 102
2018-12-19 13:23:41.176756 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 102
2018-12-19 13:23:41.176767 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 102
2018-12-19 13:23:41.285811 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988533115s) to execute
2018-12-19 13:23:42.176691 I | raft: 906aa7ee78e8c45a is starting a new election at term 102
2018-12-19 13:23:42.176725 I | raft: 906aa7ee78e8c45a became candidate at term 103
2018-12-19 13:23:42.176739 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 103
2018-12-19 13:23:42.176752 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 103
2018-12-19 13:23:43.776685 I | raft: 906aa7ee78e8c45a is starting a new election at term 103
2018-12-19 13:23:43.776724 I | raft: 906aa7ee78e8c45a became candidate at term 104
2018-12-19 13:23:43.776739 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 104
2018-12-19 13:23:43.776799 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 104
2018-12-19 13:23:45.161894 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:45.276728 I | raft: 906aa7ee78e8c45a is starting a new election at term 104
2018-12-19 13:23:45.276759 I | raft: 906aa7ee78e8c45a became candidate at term 105
2018-12-19 13:23:45.276794 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 105
2018-12-19 13:23:45.276828 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 105
2018-12-19 13:23:46.321093 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988267232s) to execute
2018-12-19 13:23:47.076700 I | raft: 906aa7ee78e8c45a is starting a new election at term 105
2018-12-19 13:23:47.076742 I | raft: 906aa7ee78e8c45a became candidate at term 106
2018-12-19 13:23:47.076783 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 106
2018-12-19 13:23:47.076802 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 106
2018-12-19 13:23:47.216547 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:48.176692 I | raft: 906aa7ee78e8c45a is starting a new election at term 106
2018-12-19 13:23:48.176727 I | raft: 906aa7ee78e8c45a became candidate at term 107
2018-12-19 13:23:48.176741 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 107
2018-12-19 13:23:48.176753 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 107
2018-12-19 13:23:49.676700 I | raft: 906aa7ee78e8c45a is starting a new election at term 107
2018-12-19 13:23:49.676734 I | raft: 906aa7ee78e8c45a became candidate at term 108
2018-12-19 13:23:49.676750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 108
2018-12-19 13:23:49.676762 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 108
2018-12-19 13:23:50.162093 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:51.076702 I | raft: 906aa7ee78e8c45a is starting a new election at term 108
2018-12-19 13:23:51.076735 I | raft: 906aa7ee78e8c45a became candidate at term 109
2018-12-19 13:23:51.076750 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 109
2018-12-19 13:23:51.076762 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 109
2018-12-19 13:23:51.357193 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988480967s) to execute
2018-12-19 13:23:51.358233 I | embed: rejected connection from "<IP_ADDRESS>:43532" (error "EOF", ServerName "")
2018-12-19 13:23:52.076700 I | raft: 906aa7ee78e8c45a is starting a new election at term 109
2018-12-19 13:23:52.076733 I | raft: 906aa7ee78e8c45a became candidate at term 110
2018-12-19 13:23:52.076748 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 110
2018-12-19 13:23:52.076762 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 110
2018-12-19 13:23:53.076651 I | raft: 906aa7ee78e8c45a is starting a new election at term 110
2018-12-19 13:23:53.076682 I | raft: 906aa7ee78e8c45a became candidate at term 111
2018-12-19 13:23:53.076695 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 111
2018-12-19 13:23:53.076711 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 111
2018-12-19 13:23:54.216770 W | etcdserver: timed out waiting for read index response
2018-12-19 13:23:54.376721 I | raft: 906aa7ee78e8c45a is starting a new election at term 111
2018-12-19 13:23:54.376758 I | raft: 906aa7ee78e8c45a became candidate at term 112
2018-12-19 13:23:54.376774 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 112
2018-12-19 13:23:54.376837 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 112
2018-12-19 13:23:55.162289 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:23:55.876706 I | raft: 906aa7ee78e8c45a is starting a new election at term 112
2018-12-19 13:23:55.876739 I | raft: 906aa7ee78e8c45a became candidate at term 113
2018-12-19 13:23:55.876753 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 113
2018-12-19 13:23:55.876764 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 113
2018-12-19 13:23:56.393744 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988852626s) to execute
2018-12-19 13:23:57.076720 I | raft: 906aa7ee78e8c45a is starting a new election at term 113
2018-12-19 13:23:57.076753 I | raft: 906aa7ee78e8c45a became candidate at term 114
2018-12-19 13:23:57.076768 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 114
2018-12-19 13:23:57.076781 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 114
2018-12-19 13:23:58.276724 I | raft: 906aa7ee78e8c45a is starting a new election at term 114
2018-12-19 13:23:58.276765 I | raft: 906aa7ee78e8c45a became candidate at term 115
2018-12-19 13:23:58.276781 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 115
2018-12-19 13:23:58.276793 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 115
2018-12-19 13:23:59.876714 I | raft: 906aa7ee78e8c45a is starting a new election at term 115
2018-12-19 13:23:59.876747 I | raft: 906aa7ee78e8c45a became candidate at term 116
2018-12-19 13:23:59.876762 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 116
2018-12-19 13:23:59.876929 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 116
2018-12-19 13:24:00.162478 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:24:01.076711 I | raft: 906aa7ee78e8c45a is starting a new election at term 116
2018-12-19 13:24:01.076748 I | raft: 906aa7ee78e8c45a became candidate at term 117
2018-12-19 13:24:01.076762 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 117
2018-12-19 13:24:01.076774 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 117
2018-12-19 13:24:01.217002 W | etcdserver: timed out waiting for read index response
2018-12-19 13:24:01.431336 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context deadline exceeded" took too long (4.988603125s) to execute
2018-12-19 13:24:02.376718 I | raft: 906aa7ee78e8c45a is starting a new election at term 117
2018-12-19 13:24:02.376754 I | raft: 906aa7ee78e8c45a became candidate at term 118
2018-12-19 13:24:02.376772 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 118
2018-12-19 13:24:02.376784 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 118
2018-12-19 13:24:03.876698 I | raft: 906aa7ee78e8c45a is starting a new election at term 118
2018-12-19 13:24:03.876734 I | raft: 906aa7ee78e8c45a became candidate at term 119
2018-12-19 13:24:03.876777 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 119
2018-12-19 13:24:03.876840 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 119
2018-12-19 13:24:05.076712 I | raft: 906aa7ee78e8c45a is starting a new election at term 119
2018-12-19 13:24:05.076746 I | raft: 906aa7ee78e8c45a became candidate at term 120
2018-12-19 13:24:05.076761 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 120
2018-12-19 13:24:05.076773 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 120
2018-12-19 13:24:05.162671 W | rafthttp: health check for peer e878b2893b78414d could not connect: dial tcp <IP_ADDRESS>:2380: i/o timeout
2018-12-19 13:24:06.467969 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.987104079s) to execute
2018-12-19 13:24:06.478450 I | embed: rejected connection from "<IP_ADDRESS>:43668" (error "EOF", ServerName "localhost")
2018-12-19 13:24:06.877003 I | raft: 906aa7ee78e8c45a is starting a new election at term 120
2018-12-19 13:24:06.877031 I | raft: 906aa7ee78e8c45a became candidate at term 121
2018-12-19 13:24:06.877044 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 121
2018-12-19 13:24:06.877055 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 121
2018-12-19 13:24:07.882605 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (4.988472481s) to execute
2018-12-19 13:24:07.883555 I | embed: rejected connection from "<IP_ADDRESS>:43682" (error "EOF", ServerName "")
2018-12-19 13:24:08.186905 N | pkg/osutil: received terminated signal, shutting down...
2018-12-19 13:24:08.187046 W | etcdserver: read-only range request "key:\"foo\" " with result "error:context canceled" took too long (1.668816166s) to execute
WARNING: 2018/12/19 13:24:08 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-12-19 13:24:08.217216 W | etcdserver: timed out waiting for read index response
2018-12-19 13:24:08.576714 I | raft: 906aa7ee78e8c45a is starting a new election at term 121
2018-12-19 13:24:08.576746 I | raft: 906aa7ee78e8c45a became candidate at term 122
2018-12-19 13:24:08.576763 I | raft: 906aa7ee78e8c45a received MsgVoteResp from 906aa7ee78e8c45a at term 122
2018-12-19 13:24:08.576831 I | raft: 906aa7ee78e8c45a [logterm: 2, index: 5] sent MsgVote request to e878b2893b78414d at term 122
2018-12-19 13:24:08.687335 I | etcdserver: skipped leadership transfer for stopping non-leader member
2018-12-19 13:24:08.687427 I | rafthttp: stopping peer e878b2893b78414d...
2018-12-19 13:24:08.687450 I | rafthttp: stopped streaming with peer e878b2893b78414d (writer)
2018-12-19 13:24:08.687462 I | rafthttp: stopped streaming with peer e878b2893b78414d (writer)
2018-12-19 13:24:08.688653 I | rafthttp: stopped HTTP pipelining with peer e878b2893b78414d
2018-12-19 13:24:08.688682 I | rafthttp: stopped streaming with peer e878b2893b78414d (stream MsgApp v2 reader)
2018-12-19 13:24:08.688767 I | rafthttp: stopped streaming with peer e878b2893b78414d (stream Message reader)
2018-12-19 13:24:08.688780 I | rafthttp: stopped peer e878b2893b78414d

$ kubectl describe pod cilium-etcd-gsc2pwb4kz  -n kube-system
Name:               cilium-etcd-gsc2pwb4kz
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-190-2-104.ec2.internal/<IP_ADDRESS>
Start Time:         Wed, 19 Dec 2018 13:20:59 +0000
Labels:             app=etcd
                    etcd_cluster=cilium-etcd
                    etcd_node=cilium-etcd-gsc2pwb4kz
                    io.cilium/app=etcd-operator
Annotations:        etcd.version: 3.3.9
Status:             Succeeded
IP:                 <IP_ADDRESS>
Controlled By:      EtcdCluster/cilium-etcd
Init Containers:
  check-dns:
    Container ID:  docker://cd1d3f861c892814d0043341968e878716ec2eb5fee23d50cb12b188959e1238
    Image:         busybox:1.28.0-glibc
    Image ID:      docker-pullable://busybox@sha256:0b55a30394294ab23b9afd58fab94e61a923f5834fba7ddbae7f8e0c11ba85e6
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c

                            while ( ! nslookup cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc )
                            do
                              sleep 2
                            done
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Dec 2018 13:21:00 +0000
      Finished:     Wed, 19 Dec 2018 13:21:04 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  etcd:
    Container ID:  docker://87336ca4dde23016cc5f6ac608880377f45cd84237f68b4d4044ad26e5c06b08
    Image:         quay.io/coreos/etcd:v3.3.9
    Image ID:      docker-pullable://quay.io/coreos/etcd@sha256:ea49a3d44a50a50770bff84eab87bac2542c7171254c4d84c609b8c66aefc211
    Ports:         2380/TCP, 2379/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/local/bin/etcd
      --data-dir=/var/etcd/data
      --name=cilium-etcd-gsc2pwb4kz
      --initial-advertise-peer-urls=https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380
      --listen-peer-urls=https://<IP_ADDRESS>:2380
      --listen-client-urls=https://<IP_ADDRESS>:2379
      --advertise-client-urls=https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2379
      --initial-cluster=cilium-etcd-gsc2pwb4kz=https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380
      --initial-cluster-state=new
      --peer-client-cert-auth=true
      --peer-trusted-ca-file=/etc/etcdtls/member/peer-tls/peer-ca.crt
      --peer-cert-file=/etc/etcdtls/member/peer-tls/peer.crt
      --peer-key-file=/etc/etcdtls/member/peer-tls/peer.key
      --client-cert-auth=true
      --trusted-ca-file=/etc/etcdtls/member/server-tls/server-ca.crt
      --cert-file=/etc/etcdtls/member/server-tls/server.crt
      --key-file=/etc/etcdtls/member/server-tls/server.key
      --initial-cluster-token=b4f22cc3-eaac-4b4d-b40e-4258ba738f52
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Dec 2018 13:21:07 +0000
      Finished:     Wed, 19 Dec 2018 13:24:08 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       exec [/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt get foo] delay=10s timeout=10s period=60s #success=1 #failure=3
    Readiness:      exec [/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt get foo] delay=1s timeout=5s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/etcdtls/member/peer-tls from member-peer-tls (rw)
      /etc/etcdtls/member/server-tls from member-server-tls (rw)
      /etc/etcdtls/operator/etcd-tls from etcd-client-tls (rw)
      /var/etcd from etcd-data (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  member-peer-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-peer-tls
    Optional:    false
  member-server-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-server-tls
    Optional:    false
  etcd-client-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-client-tls
    Optional:    false
  etcd-data:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

$ kubectl logs cilium-etcd-tvhvwvtss9  -n kube-system
2018-12-19 13:21:54.119350 I | etcdmain: etcd Version: 3.3.9
2018-12-19 13:21:54.119487 I | etcdmain: Git SHA: fca8add78
2018-12-19 13:21:54.119530 I | etcdmain: Go Version: go1.10.3
2018-12-19 13:21:54.119558 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-19 13:21:54.119591 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-19 13:21:54.119697 I | embed: peerTLS: cert = /etc/etcdtls/member/peer-tls/peer.crt, key = /etc/etcdtls/member/peer-tls/peer.key, ca = , trusted-ca = /etc/etcdtls/member/peer-tls/peer-ca.crt, client-cert-auth = true, crl-file =
2018-12-19 13:21:54.124321 I | embed: listening for peers on https://<IP_ADDRESS>:2380
2018-12-19 13:21:54.124369 I | embed: listening for client requests on <IP_ADDRESS>:2379
2018-12-19 13:21:54.137202 I | embed: rejected connection from "<IP_ADDRESS>:34192" (error "tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"*.cilium-etcd.kube-system.svc\" \"*.cilium-etcd.kube-system.svc.cluster.local\"]", ServerName "cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc", IPAddresses [], DNSNames ["*.cilium-etcd.kube-system.svc" "*.cilium-etcd.kube-system.svc.cluster.local"])
2018-12-19 13:21:54.142536 I | embed: rejected connection from "<IP_ADDRESS>:34194" (error "tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"*.cilium-etcd.kube-system.svc\" \"*.cilium-etcd.kube-system.svc.cluster.local\"]", ServerName "cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc", IPAddresses [], DNSNames ["*.cilium-etcd.kube-system.svc" "*.cilium-etcd.kube-system.svc.cluster.local"])
2018-12-19 13:21:54.150561 W | etcdserver: could not get cluster response from https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380: Get https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380/members: EOF
2018-12-19 13:21:54.151398 C | etcdmain: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given urls

$ kubectl describe pod cilium-etcd-tvhvwvtss9  -n kube-system
Name:               cilium-etcd-tvhvwvtss9
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-190-0-176.ec2.internal/<IP_ADDRESS>
Start Time:         Wed, 19 Dec 2018 13:21:20 +0000
Labels:             app=etcd
                    etcd_cluster=cilium-etcd
                    etcd_node=cilium-etcd-tvhvwvtss9
                    io.cilium/app=etcd-operator
Annotations:        etcd.version: 3.3.9
Status:             Failed
IP:                 <IP_ADDRESS>
Controlled By:      EtcdCluster/cilium-etcd
Init Containers:
  check-dns:
    Container ID:  docker://eca6b8a07d7fa80e5316df32a0ca834682e09154b9ee4f74a0aa4b9dd157442c
    Image:         busybox:1.28.0-glibc
    Image ID:      docker-pullable://busybox@sha256:0b55a30394294ab23b9afd58fab94e61a923f5834fba7ddbae7f8e0c11ba85e6
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c

                            while ( ! nslookup cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc )
                            do
                              sleep 2
                            done
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Dec 2018 13:21:20 +0000
      Finished:     Wed, 19 Dec 2018 13:21:51 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  etcd:
    Container ID:  docker://253700d808338775bcaed08b1770a22e23506cdc1add37119557b567c20f1457
    Image:         quay.io/coreos/etcd:v3.3.9
    Image ID:      docker-pullable://quay.io/coreos/etcd@sha256:ea49a3d44a50a50770bff84eab87bac2542c7171254c4d84c609b8c66aefc211
    Ports:         2380/TCP, 2379/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/local/bin/etcd
      --data-dir=/var/etcd/data
      --name=cilium-etcd-tvhvwvtss9
      --initial-advertise-peer-urls=https://cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc:2380
      --listen-peer-urls=https://<IP_ADDRESS>:2380
      --listen-client-urls=https://<IP_ADDRESS>:2379
      --advertise-client-urls=https://cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc:2379
      --initial-cluster=cilium-etcd-tvhvwvtss9=https://cilium-etcd-tvhvwvtss9.cilium-etcd.kube-system.svc:2380,cilium-etcd-gsc2pwb4kz=https://cilium-etcd-gsc2pwb4kz.cilium-etcd.kube-system.svc:2380
      --initial-cluster-state=existing
      --peer-client-cert-auth=true
      --peer-trusted-ca-file=/etc/etcdtls/member/peer-tls/peer-ca.crt
      --peer-cert-file=/etc/etcdtls/member/peer-tls/peer.crt
      --peer-key-file=/etc/etcdtls/member/peer-tls/peer.key
      --client-cert-auth=true
      --trusted-ca-file=/etc/etcdtls/member/server-tls/server-ca.crt
      --cert-file=/etc/etcdtls/member/server-tls/server.crt
      --key-file=/etc/etcdtls/member/server-tls/server.key
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 19 Dec 2018 13:21:54 +0000
      Finished:     Wed, 19 Dec 2018 13:21:54 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       exec [/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt get foo] delay=10s timeout=10s period=60s #success=1 #failure=3
    Readiness:      exec [/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt get foo] delay=1s timeout=5s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/etcdtls/member/peer-tls from member-peer-tls (rw)
      /etc/etcdtls/member/server-tls from member-server-tls (rw)
      /etc/etcdtls/operator/etcd-tls from etcd-client-tls (rw)
      /var/etcd from etcd-data (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  member-peer-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-peer-tls
    Optional:    false
  member-server-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-server-tls
    Optional:    false
  etcd-client-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cilium-etcd-client-tls
    Optional:    false
  etcd-data:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

My workstation, an EC2 instance in us-east-1 region.
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-eks-6bad6d", GitCommit:"6bad6d9c768dc0864dab48a11653aa53b5a47043", GitTreeState:"clean", BuildDate:"2018-12-06T23:13:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Succeeded on "Kubernetes 1.10" with cilium-etcd-operator" using CloudFormation following guide https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html
$ aws eks create-cluster --name eks --role-arn arn:aws:iam::**************:role/eks --resources-vpc-config subnetIds=subnet-01c******************,subnet-085******************,subnet-032******************,subnet-096******************,securityGroupIds=sg-032****************** --kubernetes-version 1.10

{
    "cluster": {
        "status": "CREATING",
        "name": "eks",
        "certificateAuthority": {},
        "roleArn": "arn:aws:iam::**************:role/eks",
        "resourcesVpcConfig": {
            "subnetIds": [
                "subnet-01c******************",
                "subnet-085******************",
                "subnet-032******************",
                "subnet-096******************"
            ],
            "vpcId": "vpc-0ee******************",
            "securityGroupIds": [
                "sg-032******************"
            ]
        },
        "version": "1.10",
        "arn": "arn:aws:eks:us-east-1:*************:cluster/eks",
        "platformVersion": "eks.3",
        "createdAt":<PHONE_NUMBER>.597
    }
}

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   aws-node-2hbm6                   1/1     Running   0          2h
kube-system   aws-node-fprzx                   1/1     Running   0          2h
kube-system   aws-node-mw5lj                   1/1     Running   0          2h
kube-system   cilium-etcd-4g2nzdf92f           1/1     Running   0          2h
kube-system   cilium-etcd-4m959f6clq           1/1     Running   0          2h
kube-system   cilium-etcd-pcxczbcvxj           1/1     Running   0          2h
kube-system   cilium-g2s2q                     1/1     Running   0          2h
kube-system   cilium-lz42j                     1/1     Running   0          2h
kube-system   cilium-v2rql                     1/1     Running   0          2h
kube-system   etcd-operator-656f9697f5-kg4nh   1/1     Running   0          2h
kube-system   kube-dns-64b69465b4-zf442        3/3     Running   0          2h
kube-system   kube-proxy-ddzz2                 1/1     Running   0          2h
kube-system   kube-proxy-zswqs                 1/1     Running   0          2h
kube-system   kube-proxy-zxqsr                 1/1     Running   0          2h

$ kubectl -n kube-system exec -ti cilium-etcd-4g2nzdf92f -- etcdctl  --endpoints=https://localhost:2379 --cert-file=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key-file=/etc/etcdtls/operator/etcd-tls/etcd-client.key --ca-file=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt  cluster-health
member 6cb8ea7d62fb6e2b is healthy: got healthy result from https://cilium-etcd-4g2nzdf92f.cilium-etcd.kube-system.svc:2379
member c2ab07ce9aa5edb2 is healthy: got healthy result from https://cilium-etcd-pcxczbcvxj.cilium-etcd.kube-system.svc:2379
member c625997e1144d312 is healthy: got healthy result from https://cilium-etcd-4m959f6clq.cilium-etcd.kube-system.svc:2379
cluster is healthy

$ kubectl -n kube-system exec -ti cilium-lz42j cilium-health status
Probe time:   2018-12-20T12:45:28Z
Nodes:
  ip-10-190-2-101.ec2.internal (localhost):
    Host connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=442.871µs
      HTTP via L3:   OK, RTT=428.906µs
    Endpoint connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=419.381µs
      HTTP via L3:   OK, RTT=553.833µs
  ip-10-190-0-46.ec2.internal:
    Host connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=707.443µs
      HTTP via L3:   OK, RTT=1.171419ms
    Endpoint connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=747.636µs
      HTTP via L3:   OK, RTT=1.188967ms
  ip-10-190-0-66.ec2.internal:
    Host connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=718.199µs
      HTTP via L3:   OK, RTT=1.215955ms
    Endpoint connectivity to <IP_ADDRESS>:
      ICMP:          OK, RTT=718.3µs
      HTTP via L3:   OK, RTT=1.928104ms

@brant4test Thanks for the excellent bug report. The issue is being fixed in https://github.com/cilium/cilium/pull/6487. It's caused by a missing well-known identity neede for k8s >=1.11 on EKS


Handle KeyboardInterrupt of Batch.run_jobs() gracefully

Difficulty: ★★☆☆☆

Background
The only way to interrupt the Batch.run_jobs() loop when it is running on the main process is the interrupt it:

Python shell ctrl+c
Jupyter i+i

It is possible to use Python to intercept the keyboard interrupt in order to handle this gracefully.
https://stackoverflow.com/a/4205386/5739514
Problem
Despite the fact that the AIQC backend knows how to handle an interrupted queue of jobs, the interrupt spits out a whole page of traceback. If I was new to AIQC, I feel like this would cause me to be worried about the integrity of my data.

Solution
Intercept the command and print a message to the user like "Queue was gracefully interrupted."

🎉 Solved by @amoh-godwin ~ Thank you!

Actually, it's not killing the loop, it's just acknowledging the KeyboardInterrupt and printing the message

had to put the entire for loop inside the try statement. Previously it was inside the for loop acting on a single iteration

@aiqc Thank you so much for accepting my commit and fixing the errors with it. Just learnt something new.


[RESTEASY-2182] Remove executable file permission where not needed

https://issues.jboss.org/browse/RESTEASY-2182
This removes executable bit from all files that are not meant to be executable but were (probably by mistake) marked as such.

@jbliznak I'm sorry, but this now has conflicts. Can you please rebase against master and solve them? Thanks

@asoldano rebased for the time being


Testing, please ignore. (sccqxr)

This is a bug bounty test.. Please DO NOT APPROVE THIS! (sccqxr)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.sccqxr chkay seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Want to Re-edit Contributor section and new About us section in Home Page

Is your feature request related to a problem? Please describe.

Want to Re-edit Contributor section and new About us section in Home Page

Describe the solution you'd like

Want to Re-edit Contributor section and new About us section in Home Page

Do you want to work on this issue - [Yes/No]

only these two sections?

I just want Flow like

Welcome page as we have
About Us page (Kurakoo Main Purpose)(Height 100vh)
Contributor Section(Kurakoo Collaborators and Contributors)(Height 60vh)
Footer

I also want to add Mentor Profile Card section but for this I will raise new issue and take it forward

@vedant-jain03 #252 work on this issue