After upgrade rancher server crashes and restarts
E0913 21:18:08.421833      32 leaderelection.go:306] error retrieving resource lock kube-system/kube-scheduler: Get https://localhost:6444/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E0913 21:18:19.941823      32 server.go:252] lost master
F0913 21:18:19.941871      32 controllermanager.go:248] leaderelection lost
goroutine 1455 [running]:
github.com/rancher/k3s/vendor/k8s.io/klog.stacks(0xc000655800, 0xc0093082d0, 0x4c, 0xf0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:828 +0xb1
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).output(0x6082c40, 0xc000000003, 0xc0026541c0, 0x5d916fd, 0x14, 0xf8, 0x0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:779 +0x2d9
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).printf(0x6082c40, 0x3, 0x34479ed, 0x13, 0x0, 0x0, 0x0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:678 +0x14e
github.com/rancher/k3s/vendor/k8s.io/klog.Fatalf(...)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:1209
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2()
        /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:248 +0x5c
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1(0xc002c457a0)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175 +0x40
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc002c457a0, 0x3c4d4a0, 0xc006368200)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:184 +0x10f
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie(0x3c4d4e0, 0xc00005e048, 0x3c74d80, 0xc002d9da40, 0x37e11d600, 0x2540be400, 0x77359400, 0xc002cadda0, 0x3569778, 0x0, ...)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:196 +0x96
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run(0xc002c12800, 0xc00005a8a0, 0xc002aa2f70, 0xc000ba1818)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:240 +0x958
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.NewControllerManagerCommand.func1(0xc002a5a280, 0xc000ee3340, 0x0, 0xb)
        /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:101 +0x27c
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).execute(0xc002a5a280, 0xc002a6a700, 0xb, 0x10, 0xc002a5a280, 0xc002a6a700)
        /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:760 +0x2ae
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc002a5a280, 0x34801d1, 0x22, 0xc002836fa0)
        /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:846 +0x2ec
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:794
github.com/rancher/k3s/pkg/daemons/control.controllerManager.func1(0xc002a6a700, 0xb, 0x10, 0xc002a5a280)
        /go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:121 +0xb2
created by github.com/rancher/k3s/pkg/daemons/control.controllerManager
        /go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:119 +0x5cc
lost lease
2019/09/13 21:18:20 [ERROR] NodeController gke-c-7vl2z-default-0-1abc5201-5pc6 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-bvws9/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController soumyarkecont1 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h6qz4/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [WARNING] Catalog-manager error deleting old template content: Delete https://localhost:6443/apis/management.cattle.io/v3/templatecontents?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController gke-c-7vl2z-default-0-1abc5201-x1r0 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-bvws9/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController soumyarkeetcd2 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h6qz4/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController <IP_ADDRESS> [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-s2jkf/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController soumyarkeetcd1 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h6qz4/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController soumyarkeetcd3 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h6qz4/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [ERROR] NodeController ip-172-31-9-124 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-hcq8v/apps/rancher-logging?timeout=30s: EOF
2019/09/13 21:18:20 [FATAL] k3s exited with: exit status 1

The fix for this issue is assumed to correct the underlying cause of a massive number of writes to etcd resulting in the container failing. These writes are thought to come from nodesyncer.go which had a bug which could cause nodesyncer to loop extremely quickly and constantly update the v3.Node object. These high number of writes would then overwhelm etcd.
For testing:
This needs to be tested multiple times since this bug is transient and hard to pin down. Please reopen if you think you see this issue again.

Tested with an upgrade from v2.2.8 to v2.3.0-rc4.

            "Restarting": false,
        "RestartCount": 0,

Tested with an upgrade from v2.2.8 to v2.3.0-rc5
Did not observe the restarts for this upgrade.
root@jiaqi-rancher-228-to-master:~# docker ps
CONTAINER ID        IMAGE                        COMMAND             CREATED             STATUS              PORTS                                      NAMES
b9c233b81d34        rancher/rancher:v2.3.0-rc6   "entrypoint.sh"     18 hours ago        Up 18 hours         <IP_ADDRESS>:80->80/tcp, <IP_ADDRESS>:443->443/tcp   practical_swirles

root@jiaqi-rancher-228-to-master:~# docker inspect practical_swirles | grep Restart
            "Restarting": false,
        "RestartCount": 0,
            "RestartPolicy": {

Observed another rancher server crash on a fresh install master-head setup.
Testing around cluster templates and cluster creation was being done on this setup.
2019-09-20 03:12:58.911899 W | etcdserver: failed to revoke 694d6d4ba6a35df6 ("lease not found")
2019-09-20 03:12:58.911916 W | etcdserver: failed to revoke 694d6d4ba6a35df6 ("lease not found")
2019-09-20 03:12:58.911924 W | etcdserver: failed to revoke 694d6d4ba6a35df6 ("lease not found")
2019-09-20 03:12:58.911935 W | etcdserver: failed to revoke 694d6d4ba6a35df6 ("lease not found")
2019-09-20 03:12:58.911943 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:539" took too long (5.123156416s) to execute
2019-09-20 03:12:58.911979 W | etcdserver: failed to revoke 694d6d4ba6a35df6 ("lease not found")
2019-09-20 03:12:58.912000 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:8" took too long (106.32649ms) to execute
2019-09-20 03:12:58.912038 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "range_response_count:1 size:502" took too long (3.314348232s) to execute
2019-09-20 03:12:58.912102 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/pipelines\" range_end:\"/registry/project.cattle.io/pipelinet\" count_only:true " with result "range_response_count:0 size:8" took too long (3.628606197s) to execute
2019-09-20 03:12:58.912157 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projectalertrules\" range_end:\"/registry/management.cattle.io/projectalertrulet\" count_only:true " with result "range_response_count:0 size:8" took too long (2.750353589s) to execute
2019-09-20 03:12:58.912175 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:557" took too long (5.121077355s) to execute
2019-09-20 03:12:58.912201 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/users\" range_end:\"/registry/management.cattle.io/usert\" count_only:true " with result "range_response_count:0 size:8" took too long (2.004262015s) to execute
2019-09-20 03:13:07.419387 W | wal: sync duration of 1.129146615s, expected less than 1s
2019-09-20 03:13:07.435472 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:539" took too long (487.794767ms) to execute
2019-09-20 03:13:07.435517 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "range_response_count:1 size:502" took too long (503.322334ms) to execute
2019-09-20 03:13:07.435594 W | etcdserver: read-only range request "key:\"/registry/storageclasses\" range_end:\"/registry/storageclasset\" count_only:true " with result "range_response_count:0 size:6" took too long (431.609566ms) to execute
2019-09-20 03:13:07.435633 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:471" took too long (498.268047ms) to execute
2019-09-20 03:13:07.435685 W | etcdserver: read-only range request "key:\"/registry/daemonsets\" range_end:\"/registry/daemonsett\" count_only:true " with result "range_response_count:0 size:6" took too long (1.052408639s) to execute
2019-09-20 03:13:07.435707 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:557" took too long (499.824953ms) to execute
2019-09-20 03:13:17.135034 W | wal: sync duration of 3.67657708s, expected less than 1s
2019-09-20 03:13:17.217466 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:471" took too long (3.710234769s) to execute

.......

.......

2019-09-20 03:14:13.733042 W | wal: sync duration of 5.814387212s, expected less than 1s
2019-09-20 03:14:13.785608 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:539" took too long (5.814221551s) to execute
2019-09-20 03:14:13.785664 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:471" took too long (5.81433041s) to execute
2019-09-20 03:14:13.785716 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/userattributes\" range_end:\"/registry/management.cattle.io/userattributet\" count_only:true " with result "range_response_count:0 size:8" took too long (4.096431058s) to execute
2019-09-20 03:14:13.785744 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/podsecuritypolicytemplateprojectbindings\" range_end:\"/registry/management.cattle.io/podsecuritypolicytemplateprojectbindingt\" count_only:true " with result "range_response_count:0 size:6" took too long (4.884425799s) to execute
2019-09-20 03:14:13.785869 W | etcdserver: read-only range request "key:\"/registry/clusterroles\" range_end:\"/registry/clusterrolet\" count_only:true " with result "range_response_count:0 size:8" took too long (5.383193622s) to execute
2019-09-20 03:14:13.785893 W | etcdserver: read-only range request "key:\"/registry/daemonsets\" range_end:\"/registry/daemonsett\" count_only:true " with result "range_response_count:0 size:6" took too long (1.710012856s) to execute
2019-09-20 03:14:13.785920 W | etcdserver: read-only range request "key:\"/registry/statefulsets\" range_end:\"/registry/statefulsett\" count_only:true " with result "range_response_count:0 size:6" took too long (2.973881249s) to execute
2019-09-20 03:14:13.785957 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "range_response_count:1 size:502" took too long (5.842221373s) to execute
2019-09-20 03:14:13.785971 W | etcdserver: read-only range request "key:\"/registry/jobs/\" range_end:\"/registry/jobs0\" limit:500 " with result "range_response_count:0 size:6" took too long (4.367447592s) to execute
2019-09-20 03:14:13.786007 W | etcdserver: read-only range request "key:\"/registry/cronjobs\" range_end:\"/registry/cronjobt\" count_only:true " with result "range_response_count:0 size:6" took too long (1.081683436s) to execute
2019-09-20 03:14:13.786089 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:557" took too long (3.863742527s) to execute
2019-09-20 03:14:13.786108 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:272" took too long (2.666555305s) to execute
2019-09-20 03:14:26.809866 W | etcdserver: timed out waiting for read index response (local node might have slow network)
2019-09-20 03:14:26.810003 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "error:etcdserver: request timed out" took too long (7.000327896s) to execute
2019-09-20 03:14:29.808217 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "error:context canceled" took too long (2.995875266s) to execute
E0920 03:14:29.807733      30 server.go:252] lost master
lost lease
2019-09-20 03:14:29.845172 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/pipelineexecutions\" range_end:\"/registry/project.cattle.io/pipelineexecutiont\" count_only:true " with result "error:context canceled" took too long (5.031526908s) to execute
2019/09/20 03:14:29 [FATAL] k3s exited with: exit status 1

We are keeping this open to see if it occurs again. No immediate action necessary from dev

Faced a similar issue on 2.3-head - commit id: e7ac6ffe6

3 - 4 clusters deployed in this server.
A cluster with k8s v 1.16.2 was seen stuck in Provisioning state, which recovered after about 2 hours.
After this the server was seen to crash after about 1 hour of the cluster recovering ^^^
(^^^ Some of the actions performed in this rancher server setup)
Logs:

2019/10/17 21:40:29 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2019/10/17 21:40:29 [INFO] kontainerdriver googlekubernetesengine stopped
2019/10/17 21:40:29 [INFO] kontainerdriver azurekubernetesservice stopped
2019/10/17 21:40:29 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2019/10/17 21:40:29 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2019-10-17 21:40:58.733995 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:471" took too long (218.304125ms) to execute
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/10/17 21:41:01 [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
2019-10-17 21:41:01.322151 W | wal: sync duration of 2.585010956s, expected less than 1s
2019-10-17 21:41:01.322894 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/cattle-controllers\" " with result "range_response_count:1 size:502" took too long (1.231981436s) to execute
2019-10-17 21:41:01.323088 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:539" took too long (1.752097129s) to execute
2019-10-17 21:41:01.323609 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:471" took too long (584.282393ms) to execute
2019-10-17 21:41:01.323927 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:557" took too long (1.026294263s) to execute
2019-10-17 21:41:01.324084 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/rkek8ssystemimages\" range_end:\"/registry/management.cattle.io/rkek8ssystemimaget\" count_only:true " with result "range_response_count:0 size:8" took too long (445.381306ms) to execute
2019-10-17 21:41:01.324137 W | etcdserver: read-only range request "key:\"/registry/jobs/\" range_end:\"/registry/jobs0\" limit:500 " with result "range_response_count:0 size:6" took too long (776.821515ms) to execute
2019-10-17 21:41:08.197147 W | wal: sync duration of 6.865558678s, expected less than 1s
E1017 21:41:08.553419      26 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}
E1017 21:41:08.553444      26 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}
E1017 21:41:08.553456      26 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"}
2019-10-17 21:41:08.554072 W | etcdserver: read-only range request "key:\"/registry/cronjobs/\" range_end:\"/registry/cronjobs0\" limit:500 " with result "range_response_count:0 size:6" took too long (7.221649131s) to execute
2019-10-17 21:41:08.554289 W | etcdserver: request "header:<ID:7587841731874606353 > txn:<compare:<target:MOD key:\"/registry/configmaps/kube-system/cattle-controllers\" mod_revision:55739 > success:<request_put:<key:\"/registry/configmaps/kube-system/cattle-controllers\" value_size:425 >> failure:<request_range:<key:\"/registry/configmaps/kube-system/cattle-controllers\" > >>" with result "size:18" took too long (356.309446ms) to execute
2019-10-17 21:41:08.554639 W | etcdserver: failed to revoke 694d6dda60454ce6 ("lease not found")
2019-10-17 21:41:08.554694 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations\" range_end:\"/registry/validatingwebhookconfigurationt\" count_only:true " with result "range_response_count:0 size:6" took too long (3.006648593s) to execute
2019-10-17 21:41:08.554767 W | etcdserver: read-only range request "key:\"/registry/networkpolicies\" range_end:\"/registry/networkpoliciet\" count_only:true " with result "range_response_count:0 size:6" took too long (4.281187255s) to execute
2019-10-17 21:41:08.554828 W | etcdserver: read-only range request "key:\"/registry/daemonsets\" range_end:\"/registry/daemonsett\" count_only:true " with result "range_response_count:0 size:6" took too long (4.455683201s) to execute
2019-10-17 21:41:08.554912 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:8" took too long (5.168889478s) to execute
2019-10-17 21:41:08.554979 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:539" took too long (5.220400513s) to execute
2019-10-17 21:41:08.555036 W | etcdserver: read-only range request "key:\"/registry/pods\" range_end:\"/registry/podt\" count_only:true " with result "range_response_count:0 size:6" took too long (5.699668475s) to execute
2019-10-17 21:41:08.555093 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:272" took too long (6.129680609s) to execute
2019-10-17 21:41:08.555171 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:8" took too long (6.192150813s) to execute
2019-10-17 21:41:08.555245 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projectroletemplatebindings\" range_end:\"/registry/management.cattle.io/projectroletemplatebindingt\" count_only:true " with result "range_response_count:0 size:8" took too long (6.708086984s) to execute
2019-10-17 21:41:08.555308 W | etcdserver: read-only range request "key:\"/registry/project.cattle.io/apps/p-4bpx7/rancher-logging\" " with result "range_response_count:0 size:6" took too long (292.35791ms) to execute
E1017 21:41:08.560660       7 leaderelection.go:365] Failed to update lock: etcdserver: request timed out
E1017 21:41:08.562301      26 leaderelection.go:341] Failed to update lock: etcdserver: request timed out
E1017 21:41:08.563529      26 leaderelection.go:341] Failed to update lock: etcdserver: request timed out
E1017 21:41:10.296084      26 event.go:247] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' '2232176663e3_65f3fbc7-f0f4-11e9-8f3c-0242ac110002 stopped leading'
F1017 21:41:10.296771      26 controllermanager.go:248] leaderelection lost
goroutine 2176 [running]:
github.com/rancher/k3s/vendor/k8s.io/klog.stacks(0xc000535600, 0xc003391180, 0x4c, 0xde)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:828 +0xb1
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).output(0x6082c40, 0xc000000003, 0xc01869bea0, 0x5d916fd, 0x14, 0xf8, 0x0)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:779 +0x2d9
github.com/rancher/k3s/vendor/k8s.io/klog.(*loggingT).printf(0x6082c40, 0x3, 0x34479ed, 0x13, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:678 +0x14e
github.com/rancher/k3s/vendor/k8s.io/klog.Fatalf(...)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/klog.go:1209
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2()
	/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:248 +0x5c
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1(0xc002f2e900)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175 +0x40
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc002f2e900, 0x3c4d4a0, 0xc005639040)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:184 +0x10f
github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie(0x3c4d4e0, 0xc0000c0018, 0x3c74d80, 0xc002f9aa00, 0x37e11d600, 0x2540be400, 0x77359400, 0xc002f67be0, 0x3569778, 0x0, ...)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:196 +0x96
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.Run(0xc002a358c8, 0xc00005a2a0, 0xc002fbc370, 0xc002a34250)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:240 +0x958
github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.NewControllerManagerCommand.func1(0xc002c8c280, 0xc0017f1d90, 0x0, 0xb)
	/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:101 +0x27c
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).execute(0xc002c8c280, 0xc002ac3100, 0xb, 0x10, 0xc002c8c280, 0xc002ac3100)
	/go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:760 +0x2ae
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc002c8c280, 0x34801d1, 0x22, 0xc002ad67a0)
	/go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:846 +0x2ec
github.com/rancher/k3s/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/rancher/k3s/vendor/github.com/spf13/cobra/command.go:794
github.com/rancher/k3s/pkg/daemons/control.controllerManager.func1(0xc002ac3100, 0xb, 0x10, 0xc002c8c280)
	/go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:121 +0xb2
created by github.com/rancher/k3s/pkg/daemons/control.controllerManager
	/go/src/github.com/rancher/k3s/pkg/daemons/control/server.go:119 +0x5cc
2019/10/17 21:41:10 [FATAL] k3s exited with: exit status 255

I see the same issue in Rancher: v2.3-head e7ac6ffe6 ,
the Rancher's log shows the leaderelection lost and  k3s exited with: exit status 255, then Rancher restarts

Hello. I'm facing the same issue.
I deploy rancher in Nomad, v2.3.0 and I was facing that problem, so yesterday I change to v2.3.1, to try resolving this problem, but with no success.
At this moment, Rancher continues to restart, here you can find my nomad events:
Oct 23, '19 09:13:03 +0100 | Restarting | Task restarting in 28.012941746s
Oct 23, '19 09:13:03 +0100 | Terminated | Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
Oct 23, '19 09:01:39 +0100 | Started | Task started by client
Oct 23, '19 09:01:28 +0100 | Driver | Downloading image
Oct 23, '19 09:00:59 +0100 | Restarting | Task restarting in 29.072467767s
Oct 23, '19 09:00:58 +0100 | Terminated | Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"

And this is my Rancher logs:
2019/10/23 09:31:17 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:31:17 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:35:29 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:35:29 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:35:29 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:35:29 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:35:29 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:35:31 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:35:31 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:35:31 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:35:33 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:35:33 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:35:33 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:35:33 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:36:37 [ERROR] NodeController ip-192-168-79-73.eu-west-1.compute.internal [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-2rk7f/apps/rancher-logging?timeout=30s: EOF
2019/10/23 09:36:37 [ERROR] NodeController ip-192-168-24-213.eu-west-1.compute.internal [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-2rk7f/apps/rancher-logging?timeout=30s: EOF
2019/10/23 09:36:37 [ERROR] NodeController do-consul-b8rn [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h7wkq/apps/rancher-logging?timeout=30s: EOF
2019/10/23 09:36:37 [ERROR] NodeController ip-192-168-55-127.eu-west-1.compute.internal [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-2rk7f/apps/rancher-logging?timeout=30s: EOF
2019/10/23 09:36:37 [ERROR] NodeController do-consul-b8r3 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h7wkq/apps/rancher-logging?timeout=30s: EOF
2019/10/23 09:36:37 [FATAL] k3s exited with: exit status 1
2019/10/23 09:37:15 [INFO] Rancher version v2.3.1 is starting
2019/10/23 09:37:15 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false KubeConfig: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false NoCACerts:false ListenConfig:<nil> AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}
2019/10/23 09:37:15 [INFO] Listening on /tmp/log.sock
2019/10/23 09:37:15 [INFO] Running etcd --data-dir=management-state/etcd
2019/10/23 09:38:16 [INFO] Waiting for k3s to start
2019/10/23 09:38:17 [INFO] Waiting for k3s to start
2019/10/23 09:38:18 [INFO] Waiting for k3s to start
2019/10/23 09:38:19 [INFO] Waiting for k3s to start
2019/10/23 09:38:20 [INFO] Waiting for k3s to start
2019/10/23 09:38:21 [INFO] Running in single server mode, will not peer connections
2019/10/23 09:38:24 [INFO] Starting API controllers
2019/10/23 09:38:28 [INFO] Listening on :443
2019/10/23 09:38:28 [INFO] Listening on :80
2019/10/23 09:38:28 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:28 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:28 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:28 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:28 [ERROR] ClusterController c-2c9n5 failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:28 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-2c9n5 failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-8gnjc failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-brwcn failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [ERROR] ClusterController c-krrwb failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [ERROR] ClusterController c-8gnjc failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-brwcn failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-2c9n5 failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-krrwb failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:29 [ERROR] ClusterController c-8gnjc failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:29 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:30 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:30 [ERROR] ClusterController c-brwcn failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:30 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:30 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:30 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:30 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:30 [ERROR] ClusterController c-krrwb failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:31 [ERROR] ClusterController c-2c9n5 failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:31 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:32 [ERROR] ClusterController c-8gnjc failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:32 [ERROR] ClusterController c-brwcn failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:32 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:32 [ERROR] ClusterController c-krrwb failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:34 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:34 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:34 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:34 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:35 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:35 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:35 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:35 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:35 [ERROR] ClusterController c-2c9n5 failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:36 [ERROR] ClusterController c-8gnjc failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:36 [ERROR] ClusterController c-brwcn failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-krrwb
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-2c9n5
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-8gnjc
2019/10/23 09:38:36 [INFO] Stopping cluster agent for c-brwcn
2019/10/23 09:38:36 [ERROR] ClusterController c-krrwb failed to start user controllers for cluster c-krrwb: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-2c9n5: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-8gnjc: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-brwcn: failed to contact server: Get https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system?timeout=30s: waiting for cluster agent to connect
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-8gnjc:machine-ww52m]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-2c9n5:machine-6ltgx]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-8gnjc:machine-88j9s]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-8gnjc]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-brwcn]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-krrwb:machine-b5p74]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-2c9n5:machine-cs8d9]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-krrwb:machine-4xk55]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-brwcn:machine-gsbt7]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-8gnjc:machine-drhkz]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-2c9n5]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-krrwb]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-brwcn:machine-58qdz]
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-brwcn:machine-psr9b]
2019/10/23 09:38:39 [INFO] Registering project network policy
2019/10/23 09:38:39 [INFO] Registering CIS controller
2019/10/23 09:38:39 [INFO] registering podsecuritypolicy cluster handler for cluster c-2c9n5
2019/10/23 09:38:39 [INFO] registering podsecuritypolicy project handler for cluster c-2c9n5
2019/10/23 09:38:39 [INFO] registering podsecuritypolicy namespace handler for cluster c-2c9n5
2019/10/23 09:38:39 [INFO] registering podsecuritypolicy serviceaccount handler for cluster c-2c9n5
2019/10/23 09:38:39 [INFO] registering podsecuritypolicy template handler for cluster c-2c9n5
2019/10/23 09:38:39 [INFO] Handling backend connection request [c-krrwb:machine-l78j4]
2019/10/23 09:38:45 [INFO] Registering monitoring for cluster "c-2c9n5"
2019/10/23 09:38:45 [INFO] Registering istio for cluster "c-2c9n5"
2019/10/23 09:38:45 [INFO] Starting cluster controllers for c-2c9n5
2019/10/23 09:38:46 [INFO] Starting cluster agent for c-2c9n5 [owner=true]
2019/10/23 09:38:46 [INFO] Registering project network policy
2019/10/23 09:38:46 [INFO] Registering CIS controller
2019/10/23 09:38:46 [INFO] registering podsecuritypolicy cluster handler for cluster c-8gnjc
2019/10/23 09:38:46 [INFO] registering podsecuritypolicy project handler for cluster c-8gnjc
2019/10/23 09:38:46 [INFO] registering podsecuritypolicy namespace handler for cluster c-8gnjc
2019/10/23 09:38:46 [INFO] registering podsecuritypolicy serviceaccount handler for cluster c-8gnjc
2019/10/23 09:38:46 [INFO] registering podsecuritypolicy template handler for cluster c-8gnjc
2019/10/23 09:38:51 [INFO] Registering monitoring for cluster "c-8gnjc"
2019/10/23 09:38:51 [INFO] Registering istio for cluster "c-8gnjc"
2019/10/23 09:38:51 [INFO] Starting cluster controllers for c-8gnjc
2019/10/23 09:38:51 [INFO] Starting cluster agent for c-8gnjc [owner=true]
2019/10/23 09:38:51 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:51 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:38:51 [ERROR] error getting app project-monitoring app.project.cattle.io "p-mxbfk/project-monitoring" not found
2019/10/23 09:38:51 [ERROR] error getting app monitoring-operator app.project.cattle.io "p-874g5/monitoring-operator" not found
2019/10/23 09:38:51 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:51 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:38:51 [ERROR] error getting app project-monitoring app.project.cattle.io "p-mxbfk/project-monitoring" not found
2019/10/23 09:38:51 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:51 [INFO] Registering project network policy
2019/10/23 09:38:51 [INFO] Registering CIS controller
2019/10/23 09:38:51 [INFO] registering podsecuritypolicy cluster handler for cluster c-brwcn
2019/10/23 09:38:51 [INFO] registering podsecuritypolicy project handler for cluster c-brwcn
2019/10/23 09:38:51 [INFO] registering podsecuritypolicy namespace handler for cluster c-brwcn
2019/10/23 09:38:51 [INFO] registering podsecuritypolicy serviceaccount handler for cluster c-brwcn
2019/10/23 09:38:51 [INFO] registering podsecuritypolicy template handler for cluster c-brwcn
2019/10/23 09:38:52 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:52 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:38:52 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:52 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:38:52 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:53 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:53 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:38:53 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:53 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:38:53 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:53 [INFO] Registering monitoring for cluster "c-brwcn"
2019/10/23 09:38:53 [INFO] Registering istio for cluster "c-brwcn"
2019/10/23 09:38:53 [INFO] Starting cluster controllers for c-brwcn
2019/10/23 09:38:53 [INFO] Starting cluster agent for c-brwcn [owner=true]
2019/10/23 09:38:53 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:38:53 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:53 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:53 [INFO] Registering project network policy
2019/10/23 09:38:53 [INFO] Registering CIS controller
2019/10/23 09:38:53 [INFO] registering podsecuritypolicy cluster handler for cluster c-krrwb
2019/10/23 09:38:53 [INFO] registering podsecuritypolicy project handler for cluster c-krrwb
2019/10/23 09:38:53 [INFO] registering podsecuritypolicy namespace handler for cluster c-krrwb
2019/10/23 09:38:53 [INFO] registering podsecuritypolicy serviceaccount handler for cluster c-krrwb
2019/10/23 09:38:53 [INFO] registering podsecuritypolicy template handler for cluster c-krrwb
2019/10/23 09:38:54 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:38:54 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:54 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:55 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:55 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:38:55 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:55 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:38:55 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:55 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:55 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:38:55 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:57 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:57 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:38:57 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:38:58 [INFO] Registering monitoring for cluster "c-krrwb"
2019/10/23 09:38:58 [INFO] Registering istio for cluster "c-krrwb"
2019/10/23 09:38:58 [INFO] Starting cluster controllers for c-krrwb
2019/10/23 09:38:58 [INFO] Starting cluster agent for c-krrwb [owner=true]
2019/10/23 09:38:58 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:58 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:58 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:58 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:38:59 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:59 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:38:59 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:59 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:38:59 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:38:59 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:59 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:59 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:38:59 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:00 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:00 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:00 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:00 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:01 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:39:01 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:01 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:02 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:02 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:02 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:02 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:06 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:06 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:06 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:06 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:07 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:07 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:39:07 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:07 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:39:07 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:09 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:39:09 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:09 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:14 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:14 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:14 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:14 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:18 [INFO] Starting catalog controller
2019/10/23 09:39:18 [INFO] Starting project-level catalog controller
2019/10/23 09:39:18 [INFO] Starting cluster-level catalog controller
2019/10/23 09:39:20 [INFO] Starting management controllers
2019/10/23 09:39:20 [INFO] update kontainerdriver rancherkubernetesengine
2019/10/23 09:39:20 [INFO] update kontainerdriver tencentkubernetesengine
2019/10/23 09:39:20 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2019/10/23 09:39:20 [INFO] update kontainerdriver amazonelasticcontainerservice
2019/10/23 09:39:20 [INFO] update kontainerdriver azurekubernetesservice
2019/10/23 09:39:20 [INFO] update kontainerdriver baiducloudcontainerengine
2019/10/23 09:39:20 [INFO] update kontainerdriver googlekubernetesengine
2019/10/23 09:39:20 [INFO] update kontainerdriver huaweicontainercloudengine
2019/10/23 09:39:20 [ERROR] UserAttributeController user-ccw7l [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
2019/10/23 09:39:21 [INFO] Reconciling GlobalRoles
2019/10/23 09:39:21 [ERROR] UserAttributeController user-ccw7l [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
2019/10/23 09:39:21 [INFO] Reconciling RoleTemplates
2019/10/23 09:39:22 [ERROR] UserAttributeController user-ccw7l [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized
2019/10/23 09:39:23 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:39:23 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:23 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:23 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:39:23 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:23 [INFO] adding kontainer driver rancherKubernetesEngine
2019/10/23 09:39:24 [INFO] adding kontainer driver googleKubernetesEngine
2019/10/23 09:39:24 [INFO] adding kontainer driver azureKubernetesService
2019/10/23 09:39:24 [INFO] adding kontainer driver amazonElasticContainerService
2019/10/23 09:39:24 [INFO] adding kontainer driver baiducloudcontainerengine
2019/10/23 09:39:24 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2019/10/23 09:39:24 [INFO] adding kontainer driver tencentkubernetesengine
2019/10/23 09:39:24 [INFO] adding kontainer driver huaweicontainercloudengine
2019/10/23 09:39:24 [INFO] Rancher startup complete
2019/10/23 09:39:25 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:25 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:39:25 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:26 [INFO] kontainerdriver amazonelasticcontainerservice listening on address <IP_ADDRESS>:46301
2019/10/23 09:39:26 [INFO] kontainerdriver googlekubernetesengine listening on address <IP_ADDRESS>:37595
2019/10/23 09:39:26 [INFO] kontainerdriver azurekubernetesservice listening on address <IP_ADDRESS>:42239
2019/10/23 09:39:26 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2019/10/23 09:39:26 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2019/10/23 09:39:26 [INFO] kontainerdriver azurekubernetesservice stopped
2019/10/23 09:39:26 [INFO] kontainerdriver googlekubernetesengine stopped
2019/10/23 09:39:26 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2019/10/23 09:39:26 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2019/10/23 09:39:30 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:30 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:30 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:39:30 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:39:55 [ERROR] NamespaceController cattle-prometheus [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-874g5" not found
2019/10/23 09:39:55 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:55 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:55 [ERROR] NamespaceController cattle-prometheus-p-mxbfk [resourceQuotaSyncController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-95lcj/p-mxbfk" not found
2019/10/23 09:39:55 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-cdlfb/p-9mqm8" not found
2019/10/23 09:39:57 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-ds7zb" not found
2019/10/23 09:39:57 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:39:57 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-j5qmv/p-2vslc" not found
2019/10/23 09:40:02 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:40:02 [ERROR] NamespaceController default [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-t6q55" not found
2019/10/23 09:40:02 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/23 09:40:02 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found

At the moment, this restarts are happening with 10 minutes of interval between them:

At the moment, my rancher have 4 cluster (3 DigitalOcean + 1 AWS), and no extra features enable.
I hope this can help you to debug this problem.

I'm seeing a lot of [ERROR] watcher error unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body) in my logs after upgrade to 2.3.1 from 2.2.9.

Some more logs for debug, with v2.3.2
2019/10/29 15:08:59 [INFO] Creating user for principal system://p-j8sn6
2019/10/29 15:08:59 [INFO] Creating user for principal system://p-m8pfl
2019/10/29 15:09:00 [INFO] Creating globalRoleBindings for u-ajomim7epn
2019/10/29 15:09:00 [INFO] Creating globalRoleBindings for u-wxcgiimtwt
2019/10/29 15:09:01 [INFO] Creating token for user u-uerxhbpxff
2019/10/29 15:09:01 [INFO] Created machine for node [ip-192-168-24-213.eu-west-1.compute.internal]
2019/10/29 15:09:01 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-zjvh7 for subject u-uerxhbpxff
2019/10/29 15:09:01 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-uerxhbpxff with role cluster-owner in namespace
2019/10/29 15:09:01 [INFO] Creating roleBinding User u-uerxhbpxff Role cluster-owner
2019/10/29 15:09:01 [ERROR] CatalogController system-library [system-image-upgrade-catalog-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:01 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-uerxhbpxff with role cluster-owner in namespace
2019/10/29 15:09:02 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-58pp6
2019/10/29 15:09:02 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-58pp6 for user u-uerxhbpxff with role cattle-globalrole-user
2019/10/29 15:09:02 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-uerxhbpxff with role cluster-owner in namespace
2019/10/29 15:09:02 [INFO] Creating clusterRole for roleTemplate Project Member (project-member).
2019/10/29 15:09:02 [INFO] Creating clusterRole for roleTemplate Project Member (project-member).
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role project-member
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-ajomim7epn Role edit
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-ajomim7epn Role project-member
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role edit
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-m8pfl-projectmember
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-j8sn6-projectmember
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-m8pfl for subject u-wxcgiimtwt
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-j8sn6 for subject u-ajomim7epn
2019/10/29 15:09:02 [INFO] Creating clusterRole project-member-promoted for project access to global resource.
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role project-member
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role edit
2019/10/29 15:09:02 [INFO] Updating clusterRole project-member-promoted for project access to global resource.
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role edit
2019/10/29 15:09:02 [INFO] Updating clusterRole project-member-promoted for project access to global resource.
2019/10/29 15:09:02 [INFO] Creating roleBinding User u-wxcgiimtwt Role project-member
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-ajomim7epn role p-j8sn6-namespaces-edit.
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-zjvh7 for subject u-ajomim7epn
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-zjvh7 for subject u-wxcgiimtwt
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-wxcgiimtwt role p-m8pfl-namespaces-edit.
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-ajomim7epn role project-member-promoted.
2019/10/29 15:09:02 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:02 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-wr5pb
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating role project-member in namespace c-zjvh7
2019/10/29 15:09:02 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-wr5pb for user u-ajomim7epn with role cattle-globalrole-user
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-wxcgiimtwt role project-member-promoted.
2019/10/29 15:09:02 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-6xw8g
2019/10/29 15:09:02 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-6xw8g for user u-wxcgiimtwt with role cattle-globalrole-user
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-ajomim7epn role create-ns.
2019/10/29 15:09:02 [ERROR] ProjectController c-zjvh7/p-m8pfl [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating role project-member in namespace c-zjvh7
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-wxcgiimtwt with role project-member in namespace
2019/10/29 15:09:02 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-wxcgiimtwt role create-ns.
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating role project-member in namespace p-m8pfl
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating role edit in namespace p-m8pfl
2019/10/29 15:09:02 [INFO] Updating role project-member in p-m8pfl because of rules difference with roleTemplate Project Member (project-member).
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-wxcgiimtwt with role project-member in namespace
2019/10/29 15:09:02 [ERROR] ProjectRoleTemplateBindingController p-j8sn6/u-ajomim7epn-member [mgmt-auth-prtb-controller] failed with : couldn't create role project-member: roles.rbac.authorization.k8s.io "project-member" already exists
2019/10/29 15:09:02 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-ajomim7epn with role project-member in namespace
2019/10/29 15:09:03 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-wxcgiimtwt with role edit in namespace
2019/10/29 15:09:03 [INFO] [mgmt-auth-prtb-controller] Creating role project-member in namespace p-j8sn6
2019/10/29 15:09:03 [INFO] Handling backend connection request [c-zjvh7:machine-bjqck]
2019/10/29 15:09:03 [INFO] [mgmt-auth-prtb-controller] Creating role edit in namespace p-j8sn6
2019/10/29 15:09:03 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-ajomim7epn with role project-member in namespace
2019/10/29 15:09:03 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-ajomim7epn with role edit in namespace
2019/10/29 15:09:03 [INFO] Updating role project-member in p-j8sn6 because of rules difference with roleTemplate Project Member (project-member).
2019/10/29 15:09:03 [INFO] Handling backend connection request [c-zjvh7:machine-ts2b4]
2019/10/29 15:09:04 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:04 [ERROR] ProjectController c-zjvh7/p-m8pfl [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:05 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2019/10/29 15:09:05 [ERROR] CatalogController system-library [system-image-upgrade-catalog-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:05 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:06 [ERROR] ProjectController c-zjvh7/p-m8pfl [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:07 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2019/10/29 15:09:07 [ERROR] NamespaceController default [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/default/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController kube-public [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-public/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController cattle-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController kube-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] ProjectAlertGroupController p-m8pfl/projectalert-workload-alert [project-alert-group-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController default [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/default/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] ProjectAlertRuleController p-m8pfl/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController default [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/default/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController cattle-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController kube-public [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-public/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController kube-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] ProjectController c-zjvh7/p-m8pfl [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:07 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:07 [ERROR] ProjectAlertGroupController p-j8sn6/projectalert-workload-alert [project-alert-group-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:07 [ERROR] NamespaceController default [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/default/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/memory-close-to-resource-limited [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2019/10/29 15:09:08 [ERROR] ProjectAlertGroupController p-j8sn6/projectalert-workload-alert [project-alert-group-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertGroupController p-m8pfl/projectalert-workload-alert [project-alert-group-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/memory-close-to-resource-limited [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-m8pfl/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/memory-close-to-resource-limited [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertGroupController p-j8sn6/projectalert-workload-alert [project-alert-group-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/less-than-half-workload-available [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectAlertRuleController p-j8sn6/memory-close-to-resource-limited [project-alert-rule-deployer] failed with : clean up alertmanager failed, Delete https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-prometheus/secrets/alertmanager-cluster-alerting?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:08 [ERROR] NamespaceController cattle-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/cattle-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] NamespaceController kube-public [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-public/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:08 [ERROR] NamespaceController kube-system [secretsController] failed with : Post https://<IP_ADDRESS>:443/api/v1/namespaces/kube-system/secrets?timeout=30s: waiting for cluster agent to connect
2019/10/29 15:09:09 [ERROR] ProjectController c-zjvh7/p-m8pfl [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:09 [INFO] Handling backend connection request [c-zjvh7]
2019/10/29 15:09:09 [ERROR] ProjectController c-zjvh7/p-j8sn6 [system-image-upgrade-controller] failed with : upgrade cluster c-zjvh7 system service logging failed: cluster c-zjvh7 not ready
2019/10/29 15:09:09 [INFO] Updating workload [kube-system/aws-node] with public endpoints [[{"nodeName":"c-zjvh7:machine-ts2b4","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-zltwz","allNodes":false},{"nodeName":"c-zjvh7:machine-lztn4","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-hpcgw","allNodes":false},{"nodeName":"c-zjvh7:machine-bjqck","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-qnk69","allNodes":false}]]
2019/10/29 15:09:09 [INFO] Updating workload [kube-system/aws-node] with public endpoints [[{"nodeName":"c-zjvh7:machine-ts2b4","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-zltwz","allNodes":false},{"nodeName":"c-zjvh7:machine-lztn4","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-hpcgw","allNodes":false},{"nodeName":"c-zjvh7:machine-bjqck","addresses":["<IP_ADDRESS>"],"port":61678,"protocol":"TCP","podName":"kube-system:aws-node-qnk69","allNodes":false}]]
2019/10/29 15:09:10 [INFO] Handling backend connection request [c-zjvh7:machine-ts2b4]
2019/10/29 15:09:11 [INFO] Handling backend connection request [c-zjvh7:machine-lztn4]
2019/10/29 15:09:22 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2019/10/29 15:09:22 [INFO] Handling backend connection request [c-zjvh7:machine-bjqck]
2019/10/29 15:09:34 [INFO] Handling backend connection request [c-zjvh7:machine-lztn4]
2019/10/29 15:10:07 [ERROR] NamespaceController kube-public [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/29 15:10:07 [ERROR] NamespaceController cattle-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/29 15:10:07 [ERROR] NamespaceController kube-system [resourceQuotaSyncController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found, [resourceQuotaUsedLimitController] failed with : project.management.cattle.io "c-pdmg9/p-qtpw2" not found
2019/10/29 15:11:05 [INFO] [mgmt-project-rbac-create] Creating namespace p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-ccw7l for project p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-7w9ld-projectowner
2019/10/29 15:11:05 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-7w9ld
2019/10/29 15:11:05 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-ccw7l role p-7w9ld-namespaces-edit.
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-7w9ld for subject user-ccw7l
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding clusterrolebinding-kz5xz for cluster membership in cluster c-zjvh7 for subject user-ccw7l
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-ccw7l with role project-owner in namespace
2019/10/29 15:11:05 [INFO] Updating clusterRoleBinding clusterrolebinding-htztq for project access to global resource for subject user-ccw7l role project-owner-promoted.
2019/10/29 15:11:05 [INFO] Updating clusterRoleBinding clusterrolebinding-drkcr for project access to global resource for subject user-ccw7l role project-owner-promoted.
2019/10/29 15:11:05 [INFO] Updating clusterRoleBinding clusterrolebinding-pq5r5 for project access to global resource for subject user-ccw7l role create-ns.
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-7w9ld
2019/10/29 15:11:05 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-ccw7l with role project-owner in namespace
2019/10/29 15:11:05 [INFO] [mgmt-project-rbac-create] Updating project p-7w9ld
2019/10/29 15:11:06 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-ccw7l with role admin in namespace
2019/10/29 15:11:06 [INFO] Updating role project-owner in p-7w9ld because of rules difference with roleTemplate Project Owner (project-owner).
2019/10/29 15:11:06 [INFO] [mgmt-project-rbac-create] Updating project p-7w9ld
2019/10/29 15:11:06 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-7w9ld
2019/10/29 15:11:06 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-7w9ld
2019/10/29 15:11:06 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-ccw7l with role cluster-owner in namespace
2019/10/29 15:11:06 [ERROR] ClusterRoleTemplateBindingController c-zjvh7/u-uerxhbpxff-admin [mgmt-auth-crtb-controller] failed with : couldn't create role cluster-owner: roles.rbac.authorization.k8s.io "cluster-owner" already exists
2019/10/29 15:11:06 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-uerxhbpxff with role cluster-owner in namespace
2019/10/29 15:11:06 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-uerxhbpxff with role cluster-owner in namespace
2019/10/29 15:11:06 [INFO] Creating user for principal system://p-7w9ld
2019/10/29 15:11:06 [INFO] Creating globalRoleBindings for u-hwopxqanca
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-7w9ld-projectmember
2019/10/29 15:11:07 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-pzt7c
2019/10/29 15:11:07 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-pzt7c for user u-hwopxqanca with role cattle-globalrole-user
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-7w9ld for subject u-hwopxqanca
2019/10/29 15:11:07 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-hwopxqanca role p-7w9ld-namespaces-edit.
2019/10/29 15:11:07 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-hwopxqanca role project-member-promoted.
2019/10/29 15:11:07 [INFO] Creating clusterRoleBinding for project access to global resource for subject u-hwopxqanca role create-ns.
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-zjvh7 for subject u-hwopxqanca
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-hwopxqanca with role project-member in namespace
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating role project-member in namespace p-7w9ld
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating role edit in namespace p-7w9ld
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-hwopxqanca with role project-member in namespace
2019/10/29 15:11:07 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject u-hwopxqanca with role edit in namespace
2019/10/29 15:11:07 [INFO] Updating role project-member in p-7w9ld because of rules difference with roleTemplate Project Member (project-member).
2019/10/29 15:11:16 [ERROR] ProjectController c-zjvh7/p-7w9ld [project-precan-alert-controller] failed with : etcdserver: request timed out
2019/10/29 15:11:20 [ERROR] CatalogController system-library [catalog] failed with : Update failed: fatal: Unable to create '/var/lib/rancher/management-state/catalog-cache/346ec2a1b7d48302a17ec6186e4713b5028d004fceacff4dc976fb20f9617766/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
: exit status 128
2019/10/29 15:11:20 [ERROR] NodeController k8s-repo-rpg7 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-x962h/apps/rancher-logging?timeout=30s: EOF
2019/10/29 15:11:20 [ERROR] NodeController ip-192-168-79-73.eu-west-1.compute.internal [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-m8pfl/apps/rancher-logging?timeout=30s: EOF
2019/10/29 15:11:20 [ERROR] Unknown error: Post https://localhost:6443/apis/management.cattle.io/v3/namespaces/c-zjvh7/projects?timeout=30s: EOF
2019/10/29 15:11:20 [ERROR] NodeController do-consul-b8r8 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-h7wkq/apps/rancher-logging?timeout=30s: EOF
2019/10/29 15:11:20 [ERROR] NodeController ip-192-168-24-213.eu-west-1.compute.internal [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-m8pfl/apps/rancher-logging?timeout=30s: EOF
2019/10/29 15:11:20 [FATAL] k3s exited with: exit status 255
2019/10/29 15:11:47 [INFO] Rancher version v2.3.2 is starting
2019/10/29 15:11:47 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false KubeConfig: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false NoCACerts:false ListenConfig:<nil> AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}
2019/10/29 15:11:47 [INFO] Listening on /tmp/log.sock
2019/10/29 15:11:47 [INFO] Running etcd --data-dir=management-state/etcd

2.3.3 same problem :
[FATAL] k3s exited with: exit status 255

Can everyone who has reported this issue please share more details about your setup. Specifically interested in:

Are you running Rancher in HA on stand-alone mode?
What are the specs of the machine(s) that Rancher is running on?

CPU
Memory
If in single-node mode, IOPs of the disk where docker volumes live (this will be where Rancher writes it data)

My generally theory here is that slow disk writes cause the etcd heartbeat check to fail which then causes k3s crash.

Maybe related? https://github.com/rancher/rancher/issues/19585

I was with this same problem and I found out that some workloads were being deployed unlimitedly, because they were misconfigured(The user didn't put health check). I stopped that workload and then my rancher came back work perfectly.

@cjellick We're running Rancher in stand-alone mode (v2.3.3 and have been experiencing it since a few versions already) and I do feel like it has become more frequent after adding more clusters.
We're currently running 6 clusters on a machine with:

4 cores on an Intel Xeon CPU
16 GB of RAM
320 GB SSD, unaware of the IOPS unfortunately

According to the system requirements listed on https://rancher.com/docs/rancher/v2.x/en/installation/requirements/#cpu-and-memory that should be more than plenty
I've gone ahead and did an iostat -y 5 to give more insight into the ongoing IO though:
root@rancher-prod1:~# iostat -y 5
Linux 4.4.0-164-generic (rancher-prod1) 	12/12/2019 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.67    0.00    0.96    0.35    0.05   96.97

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              22.00         0.00        96.00          0        480

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.81    0.00    1.12    0.30    0.00   97.77

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              20.40         0.00        96.00          0        480

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.74    0.00    1.06    0.30    0.05   94.84

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              23.60         0.00       102.40          0        512

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.91    0.00    0.86    0.56    0.05   97.61

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              19.80         0.00        91.20          0        456

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.27    0.00    1.01    0.51    0.00   97.21

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              28.40         0.00       606.40          0       3032

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.86    0.00    1.12    0.51    0.10   97.41

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              17.20         0.00        83.20          0        416

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.37    0.00    1.16    0.61    0.00   96.86

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda              22.40         0.00       120.00          0        600

And a screenshot of iotop -o after the container had been running for about 20 minutes again

Alongside of Rancher 2.x there's also still a Rancher 1.6 instance running on it (in case you're wondering about some of the IO shown)

My generally theory here is that slow disk writes cause the etcd heartbeat check to fail which then causes k3s crash.
@cjellick we think that too. Happens to us still with 2.3.5. We have two single rancher instances running on the same host (single deploy) and both crash at the very same time.
Still: Disk slowliness is very much bad but still shouldn't kill the etcd imho.

My generally theory here is that slow disk writes cause the etcd heartbeat check to fail which then causes k3s crash.
@cjellick we think that too. Happens to us still with 2.3.5. We have two different single rancher instances running on the same host (both single deploy) and both crash at the very same time.
Still: Disk slowliness is very much bad but still shouldn't kill the etcd imho.

How do you fix it?

My generally theory here is that slow disk writes cause the etcd heartbeat check to fail which then causes k3s crash.
@cjellick we think that too. Happens to us still with 2.3.5. We have two different single rancher instances running on the same host (both single deploy) and both crash at the very same time.
Still: Disk slowliness is very much bad but still shouldn't kill the etcd imho.

How do you fix it?

Actually we putting a second disk and have the volumes of disk intensive docker containers mounted on that "external" disk for now.

I see the same thing happening on a test setup with v2.4.0-rc9 (also with 2.3.x)
This is on a hosted Openstack environment where there is very limited control over disk speeds.
etcdserver throws a LOT of 'read-only range request "key:xyz" took too long (xxx.xxxms) to execute'
It's on a dedicated instance with 4 cores (2physical), 8GB and SSD storage, there is a k8s test cluster with prometheus/grafana and isto running but I have seen it happen on a fresh install with 0 clusters.

My tail of the log:
I0511 12:35:36.128915      34 trace.go:116] Trace[1037310414]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b/leader-election,client:<IP_ADDRESS> (started: 2020-05-11 12:35:29.437846003 +0000 UTC m=+36.403486853) (total time: 6.691055274s):
Trace[1037310414]: [6.691026476s] [6.691012877s] About to write a response
I0511 12:35:36.129013      34 trace.go:116] Trace[2099288295]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b/kube-controller-manager,client:<IP_ADDRESS> (started: 2020-05-11 12:35:31.516761957 +0000 UTC m=+38.482402807) (total time: 4.612223713s):
Trace[2099288295]: [4.612201115s] [4.612107122s] Object stored in database
I0511 12:35:37.083779      34 trace.go:116] Trace[74892424]: "GuaranteedUpdate etcd3" type:*core.Endpoints (started: 2020-05-11 12:35:31.516830852 +0000 UTC m=+38.482471702) (total time: 5.566937776s):
Trace[74892424]: [5.566924977s] [5.566789388s] Transaction committed
I0511 12:35:37.083836      34 trace.go:116] Trace[320186803]: "Update" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b/leader-election,client:<IP_ADDRESS> (started: 2020-05-11 12:35:31.516761857 +0000 UTC m=+38.482402707) (total time: 5.567062766s):
Trace[320186803]: [5.567030569s] [5.566991572s] Object stored in database
I0511 12:35:36.129107      34 trace.go:116] Trace[636646487]: "Create" url:/api/v1/namespaces/kube-system/events,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b/scheduler,client:<IP_ADDRESS> (started: 2020-05-11 12:35:31.516990239 +0000 UTC m=+38.482631089) (total time: 4.612106922s):
Trace[636646487]: [4.612088523s] [4.526331888s] Object stored in database
I0511 12:35:36.712126      34 trace.go:116] Trace[252503938]: "Get" url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b,client:<IP_ADDRESS> (started: 2020-05-11 12:35:30.613639507 +0000 UTC m=+37.579280357) (total time: 6.098442907s):
Trace[252503938]: [6.098349915s] [6.098342516s] About to write a response
I0511 12:35:37.084024      34 trace.go:116] Trace[1411088259]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b,client:<IP_ADDRESS> (started: 2020-05-11 12:35:30.461470766 +0000 UTC m=+37.427111616) (total time: 6.622538142s):
Trace[1411088259]: [6.622517944s] [6.622512644s] About to write a response
I0511 12:35:37.084777      34 trace.go:116] Trace[1596588953]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager,user-agent:k3s/v1.17.2+k3s1 (linux/amd64) kubernetes/cdab19b/leader-election,client:<IP_ADDRESS> (started: 2020-05-11 12:35:29.952073038 +0000 UTC m=+36.917713988) (total time: 7.132686509s):
Trace[1596588953]: [7.132663611s] [7.132659811s] About to write a response
I0511 12:35:37.087400      34 trace.go:116] Trace[1397387700]: "Get" url:/api/v1/namespaces/kube-system/configmaps/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,client:<IP_ADDRESS> (started: 2020-05-11 12:35:36.128837082 +0000 UTC m=+43.094477932) (total time: 958.550152ms):
Trace[1397387700]: [958.524754ms] [958.511455ms] About to write a response
2020/05/11 12:35:39 [FATAL] k3s exited with: exit status 255

I had issues in the past due to too low resources assigned to the Rancher instance, so I increased them and yet, same error.
Here my set up:

Hyper-V, Gen2
6 physical cores
12GB memory
Debian Bullseye
rancher:latest as of 2020/05/11 which is equivalent to rancher:v2.4.3-linux-amd64
created empty cluster
enabled all Helm repos

After this setup I get the crashes.
Before each crash happens, one cannot connect to the Rancher UI, at all. I can curl and ncat to the Rancher UI port on the Rancher node itself, if using <IP_ADDRESS>. But I cannot connect with the static IP assigned to the VM. Neither from within the Rancher node itself, nor from outside (as would be usually the case).
I have experience with Rancher inside VirtualBox VMs in the times of Rancher 2.2.x and 2.3.x and didn't have any issues, except the aforementioned resource issues.

It appears over time this issue is getting worse over time until the point where Rancher is just crashing continiously. Is there any update on a fix for this?