etcd cluster no leader failure

We repeatedly receive below error on our k8s cluster on etcd
E0429 15:54:53.235735       1 status.go:62] apiserver received an error that is not an metav1.Status: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:35:17.626027       1 status.go:62] apiserver received an error that is not an metav1.Status: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
The controller-manager and scheduler repeatedly report this error:
command kubectl logs kube-controller-manager-k8s-master-3688e46e-0 -n kube-system
2018-04-29 15:54:55.424209 I | proto: duplicate proto type registered: google.protobuf.Any
2018-04-29 15:54:55.424350 I | proto: duplicate proto type registered: google.protobuf.Duration
2018-04-29 15:54:55.424383 I | proto: duplicate proto type registered: google.protobuf.Timestamp
I0429 15:54:55.478452       1 controllermanager.go:108] Version: v1.7.7
I0429 15:54:55.482362       1 leaderelection.go:179] attempting to acquire leader lease...
E0429 16:00:54.906678       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 16:21:09.870318       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 16:54:15.669323       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 17:27:01.326286       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 17:45:59.433418       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:35:18.510143       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:39:06.037933       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:44:28.138011       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:44:41.994036       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 19:51:28.825744       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 21:04:53.971803       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 21:07:39.440264       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 22:07:37.679411       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0429 22:09:39.748550       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
command kubectl logs kube-scheduler-k8s-master-3688e46e-0   -n kube-system
2018-04-05 14:31:32.349208 I | proto: duplicate proto type registered: google.protobuf.Any
2018-04-05 14:31:32.349264 I | proto: duplicate proto type registered: google.protobuf.Duration
2018-04-05 14:31:32.349278 I | proto: duplicate proto type registered: google.protobuf.Timestamp
I0405 14:31:32.399235       1 factory.go:340] Creating scheduler from algorithm provider 'DefaultProvider'
I0405 14:31:32.399249       1 factory.go:401] Creating scheduler with fit predicates 'map[NoVolumeNodeConflict:{} NoVolumeZoneConflict:{} MaxEBSVolumeCount:{} MaxAzureDiskVolumeCount:{} NoDiskConflict:{} PodToleratesNodeTaints:{} MaxGCEPDVolumeCount:{} MatchInterPodAffinity:{} GeneralPredicates:{} CheckNodeMemoryPressure:{} CheckNodeDiskPressure:{}]' and priority functions 'map[TaintTolerationPriority:{} SelectorSpreadPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{}]
I0405 14:31:33.100049       1 controller_utils.go:994] Waiting for caches to sync for scheduler controller
I0405 14:31:33.200162       1 controller_utils.go:1001] Caches are synced for scheduler controller
I0405 14:31:33.200220       1 leaderelection.go:179] attempting to acquire leader lease...
E0405 14:44:37.628519       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 15:14:18.516524       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 15:18:14.961635       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 17:16:49.757218       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 17:34:39.107423       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 19:03:40.206397       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 19:26:16.043213       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 19:27:49.275685       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 21:52:20.397328       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0405 23:24:05.957381       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0406 00:11:17.061948       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0406 02:04:41.839332       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader
E0406 09:43:22.850567       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0
Here is  details etcd configuration:
root@k8s-master-3688E46E-0:/home/cineplanetadmin# etcd --version
etcd Version: 2.2.5
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.6.2
Go OS/Arch: linux/amd64
root@k8s-master-3688E46E-0:/home/cineplanetadmin# sudo systemctl cat etcd
/lib/systemd/system/etcd.service
[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target
[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=always
#RestartSec=10s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd2.service
#########################
root@k8s-master-3688E46E-1:/home/cineplanetadmin# etcd --version
etcd Version: 2.2.5
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.6.2
Go OS/Arch: linux/amd64
root@k8s-master-3688E46E-1:/home/cineplanetadmin# sudo systemctl cat etcd
/lib/systemd/system/etcd.service
[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target
[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=always
#RestartSec=10s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd2.service
####################
root@k8s-master-3688E46E-2:/home/cineplanetadmin# etcd --version
etcd Version: 2.2.5
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.6.2
Go OS/Arch: linux/amd64
root@k8s-master-3688E46E-2:/home/cineplanetadmin# sudo systemctl cat etcd
/lib/systemd/system/etcd.service
[Unit]
Description=etcd - highly-available key value store
Documentation=https://github.com/coreos/etcd
Documentation=man:etcd
After=network.target
Wants=network-online.target
[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=always
#RestartSec=10s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd2.service
command service etcd status
● etcd.service - etcd - highly-available key value store
Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2017-12-18 05:05:48 UTC; 4 months 10 days ago
Docs: https://github.com/coreos/etcd
man:etcd
Main PID: 1224 (etcd)
Tasks: 13
Memory: 442.0M
CPU: 3d 4h 15min 14.923s
CGroup: /system.slice/etcd.service
└─1224 /usr/bin/etcd --name k8s-master-3688E46E-0 --initial-advertise-peer-urls http://<IP_ADDRESS>:2380 --listen-peer-urls http://<IP_ADDRESS>:2380 --advertise-client-urls http://<IP_ADDRESS>:2379 --listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:2379 --initial-cluster-token k8s-etcd-cluster --initial-cluster k8s-master-3688E46E-0=http://<IP_ADDRESS>:2380,k8s-master-3688E46E-1=http://<IP_ADDRESS>:2380,k8s-master-3688E46E-2=http://<IP_ADDRESS>:2380 --data-dir /var/lib/etcddisk --initial-cluster-state new
Apr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12307, index: 153932328, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12307, index: 153932323] at term 12308
Apr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12308
Apr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153932324 [existing term: 12307, conflicting term: 12308]
Apr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153932324
Apr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12308
Apr 29 22:09:39 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153936812, lastsnap: 153926811)
Apr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153936812
Apr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153931812
Apr 29 22:15:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003011-00000000092c2057.snap successfully
Any ideas?

has no leader literally means the cluster has lost the leader. Does any of your server logs warn about leader elections?

@gyuho Not aware of that, but the service deployed will become very slow and want to check whether the etcd cluster will cause the issue or not

the service deployed will become very slow

If there's no leader, etcd service not only slows down, it will be unavailable.

We need etcd server side logging to see what might have caused such errors.

@gyuho You mean journalctl -u etcd?

Yes.

@gyuho
etcd log on master 0:Apr 29 09:09:37 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153286737, lastsnap: 153276736)
Apr 29 09:09:37 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153286737
Apr 29 09:09:37 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153281737
Apr 29 09:09:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fc8-00000000092234fc.snap successfully
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12244
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12245
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12245
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12244, index: 153295042] sent vote request to 91d25258d70ad868 at term 12245
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12244, index: 153295042] sent vote request to 2a0965170762fd09 at term 12245
Apr 29 09:19:44 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12245
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12245
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12244]
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12245
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 09:19:45 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12245
Apr 29 09:19:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12245] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12246]
Apr 29 09:19:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12246
Apr 29 09:19:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12244, index: 153295042, vote: 0] voted for 2a0965170762fd09 [logterm: 12244, index: 153295053] at term 12246
Apr 29 09:19:46 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12246
Apr 29 09:21:50 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153296738, lastsnap: 153286737)
Apr 29 09:21:50 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153296738
Apr 29 09:21:50 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153291738
Apr 29 09:21:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fc8-0000000009225c0d.snap successfully
Apr 29 09:33:56 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153306739, lastsnap: 153296738)
Apr 29 09:33:56 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153306739
Apr 29 09:33:56 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153301739
Apr 29 09:33:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fcc-000000000922831e.snap successfully
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12246] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12247]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12247
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12246, index: 153308914, vote: 0] voted for 91d25258d70ad868 [logterm: 12246, index: 153308914] at term 12247
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12247
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12247
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12247] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12246]
Apr 29 09:36:39 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 09:36:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 09:36:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 09:46:03 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153316740, lastsnap: 153306739)
Apr 29 09:46:03 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153316740
Apr 29 09:46:03 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153311740
Apr 29 09:46:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd1-000000000922aa2f.snap successfully
Apr 29 09:57:58 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153326741, lastsnap: 153316740)
Apr 29 09:57:58 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153326741
Apr 29 09:57:58 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153321741
Apr 29 09:58:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd2-000000000922d140.snap successfully
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12247
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12247, index: 153330073] sent vote request to 2a0965170762fd09 at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12247, index: 153330073] sent vote request to 91d25258d70ad868 at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12247, index: 153330073, vote: 134af6e5d3e35861] rejected vote from 2a0965170762fd09 [logterm: 12247, index: 153330073] at term 2fd8
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12248
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12247]
Apr 29 10:02:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12248
Apr 29 10:02:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 10:02:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12248
Apr 29 10:02:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12248] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12249]
Apr 29 10:02:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12249
Apr 29 10:02:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12247, index: 153330073, vote: 0] voted for 2a0965170762fd09 [logterm: 12247, index: 153330073] at term 12249
Apr 29 10:02:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12247, index: 153330073, vote: 2a0965170762fd09] rejected vote from 91d25258d70ad868 [logterm: 12247, index: 153330079] at term 12249
Apr 29 10:02:06 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12249
Apr 29 10:02:11 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:02:11 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12249
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12249, index: 153335869] sent vote request to 2a0965170762fd09 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12249, index: 153335869] sent vote request to 91d25258d70ad868 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12249, index: 153335869, vote: 134af6e5d3e35861] rejected vote from 91d25258d70ad868 [logterm: 12249, index: 153335869] at term 2fda
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12250] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12249]
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12250
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 10:09:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12250
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12250
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12249, index: 153335869] sent vote request to 2a0965170762fd09 at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12249, index: 153335869] sent vote request to 91d25258d70ad868 at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 91d25258d70ad868 at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 1 vote rejections
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12251
Apr 29 10:09:05 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12251
Apr 29 10:09:11 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:10:09 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153336742, lastsnap: 153326741)
Apr 29 10:10:09 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153336742
Apr 29 10:10:09 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153331742
Apr 29 10:10:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd4-000000000922f851.snap successfully
Apr 29 10:17:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12251] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12252]
Apr 29 10:17:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12252
Apr 29 10:17:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12251, index: 153343045, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12251, index: 153343038] at term 12252
Apr 29 10:17:50 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12252
Apr 29 10:17:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12252] ignored a MsgAppResp message with lower term from 2a0965170762fd09 [term: 12251]
Apr 29 10:17:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12252] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12253]
Apr 29 10:17:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12253
Apr 29 10:17:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12251, index: 153343045, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12251, index: 153343038] at term 12253
Apr 29 10:17:53 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12253] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12254]
Apr 29 10:17:53 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12254
Apr 29 10:17:53 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12251, index: 153343045, vote: 0] voted for 2a0965170762fd09 [logterm: 12251, index: 153343045] at term 12254
Apr 29 10:17:53 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12254
Apr 29 10:22:15 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153346743, lastsnap: 153336742)
Apr 29 10:22:15 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153346743
Apr 29 10:22:15 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153341743
Apr 29 10:22:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd6-0000000009231f62.snap successfully
Apr 29 10:27:02 k8s-master-3688E46E-0 etcd[1224]: segmented wal file /var/lib/etcddisk/member/wal/00000000000004c3-000000000923f1e7.wal is created
Apr 29 10:27:24 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/wal/00000000000004be-00000000091a755e.wal successfully
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12254
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12254, index: 153355393] sent vote request to 2a0965170762fd09 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12254, index: 153355393] sent vote request to 91d25258d70ad868 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12254, index: 153355393, vote: 134af6e5d3e35861] rejected vote from 91d25258d70ad868 [logterm: 12254, index: 153355393] at term 2fdf
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12255] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12254]
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12255
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 10:32:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12255
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12255
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12254, index: 153355393] sent vote request to 2a0965170762fd09 at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12254, index: 153355393] sent vote request to 91d25258d70ad868 at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 91d25258d70ad868 at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 1 vote rejections
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12256
Apr 29 10:32:39 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12256
Apr 29 10:32:43 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:32:43 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:32:43 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:32:43 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 10:34:17 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153356744, lastsnap: 153346743)
Apr 29 10:34:17 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153356744
Apr 29 10:34:17 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153351744
Apr 29 10:34:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd6-0000000009234673.snap successfully
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12256] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12257]
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12257
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12256, index: 153359464, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12256, index: 153359453] at term 12257
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12257
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153359454 [existing term: 12256, conflicting term: 12257]
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153359454
Apr 29 10:37:34 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12257
Apr 29 10:37:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:37:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:37:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:37:40 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12257] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12258]
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12258
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12257, index: 153364883, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12257, index: 153364879] at term 12258
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12258
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12258] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12257]
Apr 29 10:44:04 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12258] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12257]
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12258
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12257, index: 153364883] sent vote request to 2a0965170762fd09 at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12257, index: 153364883] sent vote request to 91d25258d70ad868 at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12259
Apr 29 10:44:05 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12259
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12259] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12260]
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12260
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12259, index: 153365066, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12259, index: 153365060] at term 12260
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12260
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153365061 [existing term: 12259, conflicting term: 12260]
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153365061
Apr 29 10:44:15 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12260
Apr 29 10:44:22 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:44:22 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:44:22 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 10:46:22 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153366750, lastsnap: 153356744)
Apr 29 10:46:23 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153366750
Apr 29 10:46:23 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153361750
Apr 29 10:46:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd7-0000000009236d84.snap successfully
Apr 29 10:58:13 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153376751, lastsnap: 153366750)
Apr 29 10:58:13 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153376751
Apr 29 10:58:13 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153371751
Apr 29 10:58:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fd7-0000000009239495.snap successfully
Apr 29 11:10:17 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153386752, lastsnap: 153376751)
Apr 29 11:10:17 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153386752
Apr 29 11:10:17 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153381752
Apr 29 11:10:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fdb-000000000923bba6.snap successfully
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12260] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12261]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12261
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12260, index: 153391150, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12260, index: 153391149] at term 12261
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12261
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:35 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12260]
Apr 29 11:15:36 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12261] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12262]
Apr 29 11:15:36 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12262
Apr 29 11:15:36 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12260, index: 153391150, vote: 0] voted for 91d25258d70ad868 [logterm: 12260, index: 153391157] at term 12262
Apr 29 11:15:36 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12262
Apr 29 11:22:19 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153396753, lastsnap: 153386752)
Apr 29 11:22:19 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153396753
Apr 29 11:22:19 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153391753
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12262
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12262, index: 153396832] sent vote request to 2a0965170762fd09 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12262, index: 153396832] sent vote request to 91d25258d70ad868 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12262]
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12263
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 11:22:25 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12263
Apr 29 11:22:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fde-000000000923e2b7.snap successfully
Apr 29 11:22:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12263] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12264]
Apr 29 11:22:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12264
Apr 29 11:22:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12262, index: 153396832, vote: 0] voted for 2a0965170762fd09 [logterm: 12262, index: 153396847] at term 12264
Apr 29 11:22:26 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12264
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12264] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12265]
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12265
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12264, index: 153400110, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12264, index: 153400109] at term 12265
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12265
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12264]
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12264]
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12264]
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12264]
Apr 29 11:26:26 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12264]
Apr 29 11:26:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12265] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12266]
Apr 29 11:26:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12266
Apr 29 11:26:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12264, index: 153400110, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12264, index: 153400109] at term 12266
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12266
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12264, index: 153400110] sent vote request to 91d25258d70ad868 at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12264, index: 153400110] sent vote request to 2a0965170762fd09 at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 91d25258d70ad868 at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12267
Apr 29 11:26:28 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12267
Apr 29 11:26:32 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 11:26:32 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12267] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12268]
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12268
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12267, index: 153403609, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12267, index: 153403606] at term 12268
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12268
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153403607 [existing term: 12267, conflicting term: 12268]
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153403607
Apr 29 11:30:44 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12268
Apr 29 11:30:51 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 11:34:31 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153406756, lastsnap: 153396753)
Apr 29 11:34:31 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153406756
Apr 29 11:34:31 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153401756
Apr 29 11:34:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fe0-00000000092409c8.snap successfully
Apr 29 11:46:41 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153416757, lastsnap: 153406756)
Apr 29 11:46:41 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153416757
Apr 29 11:46:41 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153411757
Apr 29 11:46:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fe4-00000000092430de.snap successfully
Apr 29 11:58:48 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153426758, lastsnap: 153416757)
Apr 29 11:58:48 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153426758
Apr 29 11:58:48 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153421758
Apr 29 11:58:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fe4-00000000092457ef.snap successfully
Apr 29 12:10:58 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153436759, lastsnap: 153426758)
Apr 29 12:10:58 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153436759
Apr 29 12:10:58 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153431759
Apr 29 12:11:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fe4-0000000009247f00.snap successfully
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12268] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12269]
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12269
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12268, index: 153443145, vote: 0] voted for 91d25258d70ad868 [logterm: 12268, index: 153443145] at term 12269
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12269
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12269
Apr 29 12:18:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12268]
Apr 29 12:18:46 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 12:18:47 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 12:18:47 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 12:23:04 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153446760, lastsnap: 153436759)
Apr 29 12:23:05 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153446760
Apr 29 12:23:05 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153441760
Apr 29 12:23:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fe6-000000000924a611.snap successfully
Apr 29 12:35:09 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153456761, lastsnap: 153446760)
Apr 29 12:35:09 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153456761
Apr 29 12:35:09 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153451761
Apr 29 12:35:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fec-000000000924cd24.snap successfully
Apr 29 12:46:56 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153466762, lastsnap: 153456761)
Apr 29 12:46:56 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153466762
Apr 29 12:46:56 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153461762
Apr 29 12:46:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fec-000000000924f435.snap successfully
Apr 29 12:53:06 k8s-master-3688E46E-0 etcd[1224]: segmented wal file /var/lib/etcddisk/member/wal/00000000000004c4-000000000925cbbc.wal is created
Apr 29 12:53:24 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/wal/00000000000004bf-00000000091c666a.wal successfully
Apr 29 12:58:54 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153476763, lastsnap: 153466762)
Apr 29 12:58:54 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153476763
Apr 29 12:58:54 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153471763
Apr 29 12:58:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fec-0000000009251b46.snap successfully
Apr 29 13:10:58 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153486764, lastsnap: 153476763)
Apr 29 13:10:58 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153486764
Apr 29 13:10:58 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153481764
Apr 29 13:11:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fec-0000000009254257.snap successfully
Apr 29 13:23:00 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153496767, lastsnap: 153486764)
Apr 29 13:23:00 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153496767
Apr 29 13:23:00 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153491767
Apr 29 13:23:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-0000000009256968.snap successfully
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12269] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12270]
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12270
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12269, index: 153500495, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12269, index: 153500494] at term 12270
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12270
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12269]
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12269]
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12269]
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12269]
Apr 29 13:27:27 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12269]
Apr 29 13:27:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12270] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12271]
Apr 29 13:27:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12271
Apr 29 13:27:28 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12269, index: 153500495, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12269, index: 153500494] at term 12271
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12271
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12269, index: 153500495] sent vote request to 2a0965170762fd09 at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12269, index: 153500495] sent vote request to 91d25258d70ad868 at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12269, index: 153500495, vote: 134af6e5d3e35861] rejected vote from 91d25258d70ad868 [logterm: 12269, index: 153500498] at term 2ff0
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 1 vote rejections
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12272
Apr 29 13:27:29 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12272
Apr 29 13:27:33 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 13:35:06 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153506768, lastsnap: 153496767)
Apr 29 13:35:06 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153506768
Apr 29 13:35:06 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153501768
Apr 29 13:35:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-0000000009259079.snap successfully
Apr 29 13:47:10 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153516769, lastsnap: 153506768)
Apr 29 13:47:10 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153516769
Apr 29 13:47:10 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153511769
Apr 29 13:47:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-000000000925b78a.snap successfully
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12272] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12273]
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12273
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12272, index: 153526718, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12272, index: 153526713] at term 12273
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12273
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153526714 [existing term: 12272, conflicting term: 12273]
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153526714
Apr 29 13:59:15 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12273
Apr 29 13:59:18 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153526770, lastsnap: 153516769)
Apr 29 13:59:18 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153526770
Apr 29 13:59:18 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153521770
Apr 29 13:59:21 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 13:59:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-000000000925de9b.snap successfully
Apr 29 14:11:23 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153536771, lastsnap: 153526770)
Apr 29 14:11:23 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153536771
Apr 29 14:11:23 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153531771
Apr 29 14:11:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-00000000092605ac.snap successfully
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12273
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12274
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12274
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12273, index: 153539851] sent vote request to 2a0965170762fd09 at term 12274
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12273, index: 153539851] sent vote request to 91d25258d70ad868 at term 12274
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12274
Apr 29 14:15:08 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12273]
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12274
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12274
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 14:15:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12274
Apr 29 14:15:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12274] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12275]
Apr 29 14:15:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12275
Apr 29 14:15:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12273, index: 153539851, vote: 0] voted for 91d25258d70ad868 [logterm: 12273, index: 153539864] at term 12275
Apr 29 14:15:10 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12275
Apr 29 14:23:26 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153546772, lastsnap: 153536771)
Apr 29 14:23:26 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153546772
Apr 29 14:23:26 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153541772
Apr 29 14:23:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fed-0000000009262cbf.snap successfully
Apr 29 14:35:33 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153556773, lastsnap: 153546772)
Apr 29 14:35:33 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153556773
Apr 29 14:35:33 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153551773
Apr 29 14:35:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff0-00000000092653d0.snap successfully
Apr 29 14:47:35 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153566774, lastsnap: 153556773)
Apr 29 14:47:35 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153566774
Apr 29 14:47:35 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153561774
Apr 29 14:47:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff0-0000000009267ae1.snap successfully
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12275] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12276]
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12276
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12275, index: 153571066, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12275, index: 153571059] at term 12276
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12276
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12276] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12275]
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12276] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12275]
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12276] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12275]
Apr 29 14:52:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12276] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12275]
Apr 29 14:52:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12276] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12277]
Apr 29 14:52:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12277
Apr 29 14:52:51 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12275, index: 153571066, vote: 0] voted for 91d25258d70ad868 [logterm: 12275, index: 153571068] at term 12277
Apr 29 14:52:51 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12277
Apr 29 14:59:40 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153576775, lastsnap: 153566774)
Apr 29 14:59:40 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153576775
Apr 29 14:59:40 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153571775
Apr 29 14:59:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff1-000000000926a1f2.snap successfully
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12277
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12278
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12278
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12277, index: 153580089] sent vote request to 2a0965170762fd09 at term 12278
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12277, index: 153580089] sent vote request to 91d25258d70ad868 at term 12278
Apr 29 15:03:39 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12278
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12277]
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12278
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12278
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 15:03:40 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12278
Apr 29 15:03:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12278] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12279]
Apr 29 15:03:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12279
Apr 29 15:03:41 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12277, index: 153580089, vote: 0] voted for 91d25258d70ad868 [logterm: 12277, index: 153580095] at term 12279
Apr 29 15:03:41 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12279
Apr 29 15:11:50 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153586776, lastsnap: 153576775)
Apr 29 15:11:50 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153586776
Apr 29 15:11:50 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153581776
Apr 29 15:11:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff1-000000000926c903.snap successfully
Apr 29 15:17:38 k8s-master-3688E46E-0 etcd[1224]: segmented wal file /var/lib/etcddisk/member/wal/00000000000004c5-0000000009279f7a.wal is created
Apr 29 15:17:54 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/wal/00000000000004c0-00000000091e5bf8.wal successfully
Apr 29 15:23:52 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153596777, lastsnap: 153586776)
Apr 29 15:23:52 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153596777
Apr 29 15:23:52 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153591777
Apr 29 15:23:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff3-000000000926f014.snap successfully
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12279
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12279, index: 153606425] sent vote request to 2a0965170762fd09 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12279, index: 153606425] sent vote request to 91d25258d70ad868 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12279, index: 153606425, vote: 134af6e5d3e35861] rejected vote from 2a0965170762fd09 [logterm: 12279, index: 153606425] at term 2ff8
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12280
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:29 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12279]
Apr 29 15:35:30 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12280] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12281]
Apr 29 15:35:30 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12281
Apr 29 15:35:30 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12279, index: 153606425, vote: 0] voted for 91d25258d70ad868 [logterm: 12279, index: 153606432] at term 12281
Apr 29 15:35:30 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12281
Apr 29 15:35:58 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153606778, lastsnap: 153596777)
Apr 29 15:35:58 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153606778
Apr 29 15:35:58 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153601778
Apr 29 15:36:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff3-0000000009271725.snap successfully
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12281
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12282
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12282
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12281, index: 153613545] sent vote request to 2a0965170762fd09 at term 12282
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12281, index: 153613545] sent vote request to 91d25258d70ad868 at term 12282
Apr 29 15:44:06 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12282
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12282
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12282
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12282
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12281]
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12281]
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12281]
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12281]
Apr 29 15:44:07 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12281]
Apr 29 15:44:13 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 15:48:02 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153616779, lastsnap: 153606778)
Apr 29 15:48:02 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153616779
Apr 29 15:48:02 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153611779
Apr 29 15:48:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff3-0000000009273e36.snap successfully
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12282] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12283
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12282, index: 153622258, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12282, index: 153622253] at term 12283
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12283
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153622254 [existing term: 12282, conflicting term: 12283]
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153622254
Apr 29 15:54:38 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12283
Apr 29 15:54:44 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 15:54:44 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 15:54:45 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12283
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 2a0965170762fd09 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 91d25258d70ad868 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12284] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12283]
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12284
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 15:54:46 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12284
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12284
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 2a0965170762fd09 at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 91d25258d70ad868 at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307, vote: 134af6e5d3e35861] rejected vote from 91d25258d70ad868 [logterm: 12283, index: 153622308] at term 2ffd
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12285
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 15:54:47 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12285
Apr 29 15:54:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12285
Apr 29 15:54:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12286
Apr 29 15:54:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12286
Apr 29 15:54:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 91d25258d70ad868 at term 12286
Apr 29 15:54:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307] sent vote request to 2a0965170762fd09 at term 12286
Apr 29 15:54:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12286
Apr 29 15:54:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 15:54:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12286
Apr 29 15:54:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 15:54:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12286
Apr 29 15:54:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12286] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12287]
Apr 29 15:54:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12287
Apr 29 15:54:50 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12283, index: 153622307, vote: 0] voted for 91d25258d70ad868 [logterm: 12283, index: 153622308] at term 12287
Apr 29 15:54:50 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12287
Apr 29 15:54:52 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 15:54:53 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 15:54:53 k8s-master-3688E46E-0 etcd[1224]: got unexpected response error (etcdserver: request timed out)
Apr 29 16:00:03 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153626780, lastsnap: 153616779)
Apr 29 16:00:04 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153626780
Apr 29 16:00:04 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153621780
Apr 29 16:00:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff5-0000000009276547.snap successfully
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12287
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12287, index: 153627362] sent vote request to 2a0965170762fd09 at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12287, index: 153627362] sent vote request to 91d25258d70ad868 at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12288
Apr 29 16:00:48 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12288
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:49 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12287]
Apr 29 16:00:54 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 16:00:55 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 16:12:10 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153636781, lastsnap: 153626780)
Apr 29 16:12:10 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153636781
Apr 29 16:12:10 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153631781
Apr 29 16:12:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff7-0000000009278c58.snap successfully
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12288] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12289]
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12289
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12288, index: 153644065, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12288, index: 153644056] at term 12289
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12289
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153644057 [existing term: 12288, conflicting term: 12289]
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153644057
Apr 29 16:21:03 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12289
Apr 29 16:21:09 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 16:21:10 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 16:24:18 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153646782, lastsnap: 153636781)
Apr 29 16:24:18 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153646782
Apr 29 16:24:18 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153641782
Apr 29 16:24:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff7-000000000927b369.snap successfully
Apr 29 16:36:25 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153656783, lastsnap: 153646782)
Apr 29 16:36:25 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153656783
Apr 29 16:36:25 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153651783
Apr 29 16:36:26 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ff9-000000000927da7a.snap successfully
Apr 29 16:39:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12289] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12290]
Apr 29 16:39:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12290
Apr 29 16:39:38 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12289, index: 153659417, vote: 0] voted for 91d25258d70ad868 [logterm: 12289, index: 153659417] at term 12290
Apr 29 16:39:38 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12290
Apr 29 16:39:38 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12290
Apr 29 16:39:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12290] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12289]
Apr 29 16:39:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12290] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12289]
Apr 29 16:39:39 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12290] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12289]
Apr 29 16:48:28 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153666784, lastsnap: 153656783)
Apr 29 16:48:28 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153666784
Apr 29 16:48:28 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153661784
Apr 29 16:48:56 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002ffa-000000000928018b.snap successfully
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12290
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12290, index: 153671470] sent vote request to 2a0965170762fd09 at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12290, index: 153671470] sent vote request to 91d25258d70ad868 at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12291
Apr 29 16:54:09 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12291
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:10 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12290]
Apr 29 16:54:15 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:00:37 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153676785, lastsnap: 153666784)
Apr 29 17:00:37 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153676785
Apr 29 17:00:37 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153671785
Apr 29 17:00:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000002fff-000000000928289c.snap successfully
Apr 29 17:12:44 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153686786, lastsnap: 153676785)
Apr 29 17:12:44 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153686786
Apr 29 17:12:44 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153681786
Apr 29 17:12:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003000-0000000009284fad.snap successfully
Apr 29 17:24:57 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153696787, lastsnap: 153686786)
Apr 29 17:24:57 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153696787
Apr 29 17:24:57 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153691787
Apr 29 17:25:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003001-00000000092876be.snap successfully
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12291] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12292]
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12292
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12291, index: 153698388, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12291, index: 153698381] at term 12292
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12292
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153698382 [existing term: 12291, conflicting term: 12292]
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153698382
Apr 29 17:26:55 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12292
Apr 29 17:27:01 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:27:01 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:27:01 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:37:05 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153706788, lastsnap: 153696787)
Apr 29 17:37:05 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153706788
Apr 29 17:37:05 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153701788
Apr 29 17:37:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003001-0000000009289dcf.snap successfully
Apr 29 17:42:59 k8s-master-3688E46E-0 etcd[1224]: segmented wal file /var/lib/etcddisk/member/wal/00000000000004c6-0000000009297425.wal is created
Apr 29 17:43:24 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/wal/00000000000004c1-0000000009203f3b.wal successfully
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12292
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12292, index: 153713987] sent vote request to 2a0965170762fd09 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12292, index: 153713987] sent vote request to 91d25258d70ad868 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 91d25258d70ad868 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12293
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:52 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12292]
Apr 29 17:45:58 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:45:59 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:45:59 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 17:49:10 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153716789, lastsnap: 153706788)
Apr 29 17:49:11 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153716789
Apr 29 17:49:11 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153711789
Apr 29 17:49:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003002-000000000928c4e0.snap successfully
Apr 29 18:01:18 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153726790, lastsnap: 153716789)
Apr 29 18:01:19 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153726790
Apr 29 18:01:19 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153721790
Apr 29 18:01:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003003-000000000928ebf1.snap successfully
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12293] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12294]
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12294
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12293, index: 153730056, vote: 0] rejected vote from 91d25258d70ad868 [logterm: 12293, index: 153730049] at term 12294
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12294
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153730050 [existing term: 12293, conflicting term: 12294]
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153730050
Apr 29 18:05:16 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12294
Apr 29 18:05:22 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:05:22 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12294
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12294, index: 153735472] sent vote request to 2a0965170762fd09 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12294, index: 153735472] sent vote request to 91d25258d70ad868 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12294, index: 153735472, vote: 134af6e5d3e35861] rejected vote from 2a0965170762fd09 [logterm: 12294, index: 153735472] at term 3007
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 2a0965170762fd09 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 1 vote rejections
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgHeartbeat message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12294]
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote rejection from 91d25258d70ad868 at term 12295
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 1 votes and 2 vote rejections
Apr 29 18:11:56 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12295
Apr 29 18:11:57 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12295] received a MsgVote message with higher term from 91d25258d70ad868 [term: 12296]
Apr 29 18:11:57 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12296
Apr 29 18:11:57 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12294, index: 153735472, vote: 0] voted for 91d25258d70ad868 [logterm: 12294, index: 153735477] at term 12296
Apr 29 18:11:57 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 91d25258d70ad868 at term 12296
Apr 29 18:13:21 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153736791, lastsnap: 153726790)
Apr 29 18:13:21 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153736791
Apr 29 18:13:21 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153731791
Apr 29 18:13:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003003-0000000009291302.snap successfully
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12296
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12296, index: 153736987] sent vote request to 2a0965170762fd09 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12296, index: 153736987] sent vote request to 91d25258d70ad868 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 91d25258d70ad868 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 2a0965170762fd09 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12297
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12297] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12296]
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12297] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12296]
Apr 29 18:13:37 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12297] ignored a MsgApp message with lower term from 91d25258d70ad868 [term: 12296]
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12297] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12298]
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12298
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12297, index: 153746584, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12297, index: 153746578] at term 12298
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12298
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153746579 [existing term: 12297, conflicting term: 12298]
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153746579
Apr 29 18:25:18 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12298
Apr 29 18:25:23 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:25:24 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:25:24 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:25:25 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 18:25:34 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153746792, lastsnap: 153736791)
Apr 29 18:25:34 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153746792
Apr 29 18:25:34 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153741792
Apr 29 18:25:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003003-0000000009293a13.snap successfully
Apr 29 18:37:34 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153756794, lastsnap: 153746792)
Apr 29 18:37:34 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153756794
Apr 29 18:37:34 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153751794
Apr 29 18:37:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003004-0000000009296124.snap successfully
Apr 29 18:49:41 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153766795, lastsnap: 153756794)
Apr 29 18:49:41 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153766795
Apr 29 18:49:41 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153761795
Apr 29 18:49:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003005-0000000009298835.snap successfully
Apr 29 19:01:38 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153776796, lastsnap: 153766795)
Apr 29 19:01:39 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153776796
Apr 29 19:01:39 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153771796
Apr 29 19:01:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003005-000000000929af46.snap successfully
Apr 29 19:13:43 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153786797, lastsnap: 153776796)
Apr 29 19:13:43 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153786797
Apr 29 19:13:43 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153781797
Apr 29 19:13:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003008-000000000929d657.snap successfully
Apr 29 19:25:52 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153796798, lastsnap: 153786797)
Apr 29 19:25:52 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153796798
Apr 29 19:25:52 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153791798
Apr 29 19:25:57 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/000000000000300a-000000000929fd68.snap successfully
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12298
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12298, index: 153804557] sent vote request to 2a0965170762fd09 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12298, index: 153804557] sent vote request to 91d25258d70ad868 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 2a0965170762fd09 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 91d25258d70ad868 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [q:2] has received 2 votes and 0 vote rejections
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became leader at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 134af6e5d3e35861 at term 12299
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgHeartbeat message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:11 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] ignored a MsgApp message with lower term from 2a0965170762fd09 [term: 12298]
Apr 29 19:35:17 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:35:17 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:35:18 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:37:59 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153806799, lastsnap: 153796798)
Apr 29 19:37:59 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153806799
Apr 29 19:37:59 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153801799
Apr 29 19:38:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/000000000000300a-00000000092a247a.snap successfully
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [term: 12299] received a MsgVote message with higher term from 2a0965170762fd09 [term: 12300]
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became follower at term 12300
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12299, index: 153807611, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12299, index: 153807603] at term 12300
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12300
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153807604 [existing term: 12299, conflicting term: 12300]
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153807604
Apr 29 19:38:59 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12300
Apr 29 19:39:05 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:39:05 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:39:06 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure
Apr 29 19:44:21 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 is starting a new election at term 12300
Apr 29 19:44:21 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 became candidate at term 12301
Apr 29 19:44:21 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 12301
Apr 29 19:44:21 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12300, index: 153812065] sent vote request to 2a0965170762fd09 at term 12301
Apr 29 19:44:21 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12300, index: 153812065] sent vote request to 91d25258d70ad868 at term 12301