[v2.2.4] Upgrade fails for setups with clusters having OpenStack CloudProvider with a LoadBalancer config: cannot unmarshal number into Go value of type string

What kind of request is this (question/bug/enhancement/feature request): Bug
Steps to reproduce (least amount of steps as possible):
Upgrade from v2.2.3 to v2.2.4
Result:
Rancher Server fails to start with the following error mesages:
E0606 07:39:20.296926       8 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v3.Cluster: json: cannot unmarshal number into Go value of type string

Other details that may be helpful:
After a Downgrade to v2.2.3 Rancher is running fine again
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.2.4
Installation option (single install/HA): Single Installation

Ran into same issue myself trying to update.

@ptrunk do you also have your own certs as @DJAyth mentioned in their setup?
I am not able to reproduce the issue. I am using self-signed.

@davidnuzik yes i use Traefik in front of Rancher with a letsencrypt certificate. So my (single) rancher is running with --no-cacerts.

Thanks @ptrunk I or someone on my team will investigate further.

@ptrunk can you share the output of kubectl get cluster --all-namespaces -o json

@ptrunk @DJAyth Do you have openstack lb configured?

Yes as I matter of fact I do. The rancher setup sits behind an Openstack LB, in the root cluster that gets created when the Rancher setup is knows nothing of Openstack to my knowledge.
Also my SSL setup terminates at Rancher itself on the nginx-ingress, which is different than @ptrunk I believe.

@DJAyth our theory is that the bug can be caused by https://github.com/rancher/types/commit/17ad82ec47246e37d734cf9e87ee6590ea60315d - a fix for https://github.com/rancher/rancher/issues/14577. If your user cluster objects have OpenStack with LB provider configured, and monitorDelay/monitorTimeout parameters set, they'd have to be changed from int to string. We are working on reproducing the issue and figuring out steps to mitigate; will update the bug shortly.

@mrajashree Yes my cluster which is managed by Rancher is using the openstack cloud provider with the openstack lb
@alena1108
root@18d92dabdd1a:/var/lib/rancher# kubectl get cluster --all-namespaces -o json
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "management.cattle.io/v3",
            "kind": "Cluster",
            "metadata": {
                "annotations": {
                    "authz.management.cattle.io/creator-role-bindings": "{\"created\":[\"cluster-owner\"],\"required\":[\"cluster-owner\"]}",
                    "field.cattle.io/creatorId": "user-bkxx5",
                    "field.cattle.io/overwriteAppAnswers": "{\"answers\":{\"exporter-node.enabled\":\"true\",\"exporter-node.ports.metrics.port\":\"9796\",\"exporter-kubelets.https\":\"true\",\"exporter-node.resources.limits.cpu\":\"200m\",\"exporter-node.resources.limits.memory\":\"200Mi\",\"prometheus.retention\":\"24h\",\"grafana.persistence.enabled\":\"true\",\"prometheus.persistence.enabled\":\"true\",\"prometheus.persistence.storageClass\":\"default\",\"grafana.persistence.storageClass\":\"default\",\"grafana.persistence.size\":\"1Gi\",\"prometheus.persistence.size\":\"10Gi\",\"prometheus.resources.core.requests.cpu\":\"750m\",\"prometheus.resources.core.limits.cpu\":\"1000m\",\"prometheus.resources.core.requests.memory\":\"750Mi\",\"prometheus.resources.core.limits.memory\":\"1000Mi\"}}",
                    "lifecycle.cattle.io/create.cluster-agent-controller-cleanup": "true",
                    "lifecycle.cattle.io/create.cluster-provisioner-controller": "true",
                    "lifecycle.cattle.io/create.cluster-scoped-gc": "true",
                    "lifecycle.cattle.io/create.mgmt-cluster-rbac-remove": "true",
                    "networking.management.cattle.io/enable-network-policy": "false",
                    "provisioner.cattle.io/ke-driver-update": "updated"
                },
                "creationTimestamp": "2019-05-16T13:48:41Z",
                "finalizers": [
                    "controller.cattle.io/cluster-agent-controller-cleanup",
                    "controller.cattle.io/cluster-scoped-gc",
                    "controller.cattle.io/cluster-provisioner-controller",
                    "controller.cattle.io/mgmt-cluster-rbac-remove"
                ],
                "generation": 1,
                "labels": {
                    "cattle.io/creator": "norman"
                },
                "name": "c-jp966",
                "resourceVersion": "2054723",
                "selfLink": "/apis/management.cattle.io/v3/clusters/c-jp966",
                "uid": "504dbedc-77e1-11e9-a9ac-02420a000140"
            },
            "spec": {
                "description": "Dev Cluster",
                "desiredAgentImage": "",
                "desiredAuthImage": "",
                "displayName": "dev",
                "dockerRootDir": "/var/lib/docker",
                "enableClusterAlerting": false,
                "enableClusterMonitoring": true,
                "enableNetworkPolicy": false,
                "internal": false,
                "localClusterAuthEndpoint": {
                    "enabled": false
                },
                "rancherKubernetesEngineConfig": {
                    "addonJobTimeout": 30,
                    "authentication": {
                        "strategy": "x509"
                    },
                    "authorization": {},
                    "bastionHost": {},
                    "cloudProvider": {
                        "name": "openstack",
                        "openstackCloudProvider": {
                            "blockStorage": {
                                "bs-version": "",
                                "ignore-volume-az": false,
                                "trust-device-path": false
                            },
                            "global": {
                                "auth-url": "https://keystone.url",
                                "ca-file": "",
                                "domain-id": "xxx",
                                "domain-name": "",
                                "password": "xxx",
                                "region": "",
                                "tenant-id": "xxx",
                                "tenant-name": "",
                                "trust-id": "",
                                "user-id": "",
                                "username": "xxx"
                            },
                            "loadBalancer": {
                                "create-monitor": false,
                                "floating-network-id": "",
                                "lb-method": "",
                                "lb-provider": "",
                                "lb-version": "",
                                "manage-security-groups": false,
                                "monitor-delay": 0,
                                "monitor-max-retries": 0,
                                "monitor-timeout": 0,
                                "subnet-id": "64cf703f-9504-4b32-baa3-741905e8b43a",
                                "use-octavia": true
                            },
                            "metadata": {
                                "request-timeout": 0,
                                "search-order": ""
                            },
                            "route": {
                                "router-id": ""
                            }
                        }
                    },
                    "dns": {
                        "provider": "coredns",
                        "upstreamnameservers": [
                            "<IP_ADDRESS>",
                            "<IP_ADDRESS>"
                        ]
                    },
                    "ignoreDockerVersion": true,
                    "ingress": {
                        "provider": "nginx"
                    },
                    "kubernetesVersion": "v1.14.1-rancher1-1",
                    "monitoring": {
                        "provider": "metrics-server"
                    },
                    "network": {
                        "plugin": "canal"
                    },
                    "restore": {},
                    "services": {
                        "etcd": {
                            "backupConfig": {
                                "enabled": true,
                                "intervalHours": 12,
                                "retention": 6,
                                "s3BackupConfig": null
                            },
                            "creation": "12h",
                            "extraArgs": {
                                "election-timeout": "5000",
                                "heartbeat-interval": "500"
                            },
                            "retention": "72h",
                            "snapshot": false
                        },
                        "kubeApi": {
                            "serviceNodePortRange": "30000-32767"
                        },
                        "kubeController": {},
                        "kubelet": {},
                        "kubeproxy": {},
                        "scheduler": {}
                    },
                    "sshAgentAuth": false,
                    "systemImages": {}
                }
            },
            "status": {
                "agentImage": "rancher/rancher-agent:v2.2.3",
                "allocatable": {
                    "cpu": "8",
                    "memory": "16121976Ki",
                    "pods": "220"
                },
                "apiEndpoint": "https://<IP_ADDRESS>:6443",
                "appliedEnableNetworkPolicy": false,
                "appliedPodSecurityPolicyTemplateId": "",
                "appliedSpec": {
                    "description": "Dev Cluster",
                    "desiredAgentImage": "",
                    "desiredAuthImage": "",
                    "displayName": "dev",
                    "dockerRootDir": "/var/lib/docker",
                    "enableClusterAlerting": false,
                    "enableClusterMonitoring": true,
                    "enableNetworkPolicy": false,
                    "internal": false,
                    "localClusterAuthEndpoint": {
                        "enabled": false
                    },
                    "rancherKubernetesEngineConfig": {
                        "addonJobTimeout": 30,
                        "authentication": {
                            "strategy": "x509"
                        },
                        "authorization": {},
                        "bastionHost": {},
                        "cloudProvider": {
                            "name": "openstack",
                            "openstackCloudProvider": {
                                "blockStorage": {
                                    "bs-version": "",
                                    "ignore-volume-az": false,
                                    "trust-device-path": false
                                },
                                "global": {
                                    "auth-url": "https://keystone.url",
                                    "ca-file": "",
                                    "domain-id": "xxx",
                                    "domain-name": "",
                                    "password": "xxx",
                                    "region": "",
                                    "tenant-id": "xxx",
                                    "tenant-name": "",
                                    "trust-id": "",
                                    "user-id": "",
                                    "username": "xxx"
                                },
                                "loadBalancer": {
                                    "create-monitor": false,
                                    "floating-network-id": "",
                                    "lb-method": "",
                                    "lb-provider": "",
                                    "lb-version": "",
                                    "manage-security-groups": false,
                                    "monitor-delay": 0,
                                    "monitor-max-retries": 0,
                                    "monitor-timeout": 0,
                                    "subnet-id": "64cf703f-9504-4b32-baa3-741905e8b43a",
                                    "use-octavia": true
                                },
                                "metadata": {
                                    "request-timeout": 0,
                                    "search-order": ""
                                },
                                "route": {
                                    "router-id": ""
                                }
                            }
                        },
                        "dns": {
                            "provider": "coredns",
                            "upstreamnameservers": [
                                "<IP_ADDRESS>",
                                "<IP_ADDRESS>"
                            ]
                        },
                        "ignoreDockerVersion": true,
                        "ingress": {
                            "provider": "nginx"
                        },
                        "kubernetesVersion": "v1.14.1-rancher1-1",
                        "monitoring": {
                            "provider": "metrics-server"
                        },
                        "network": {
                            "plugin": "canal"
                        },
                        "nodes": [
                            {
                                "address": "<IP_ADDRESS>",
                                "hostnameOverride": "dkw02",
                                "nodeName": "c-jp966:m-0f8b84fb7c0c",
                                "port": "22",
                                "role": [
                                    "worker"
                                ],
                                "user": "root"
                            },
                            {
                                "address": "<IP_ADDRESS>",
                                "hostnameOverride": "dkm01",
                                "nodeName": "c-jp966:m-449da24cb51f",
                                "port": "22",
                                "role": [
                                    "etcd",
                                    "controlplane"
                                ],
                                "user": "root"
                            },
                            {
                                "address": "<IP_ADDRESS>",
                                "hostnameOverride": "dkw01",
                                "nodeName": "c-jp966:m-9be6efc52982",
                                "port": "22",
                                "role": [
                                    "worker"
                                ],
                                "user": "root"
                            }
                        ],
                        "restore": {},
                        "services": {
                            "etcd": {
                                "backupConfig": {
                                    "enabled": true,
                                    "intervalHours": 12,
                                    "retention": 6,
                                    "s3BackupConfig": null
                                },
                                "creation": "12h",
                                "extraArgs": {
                                    "election-timeout": "5000",
                                    "heartbeat-interval": "500"
                                },
                                "retention": "72h",
                                "snapshot": false
                            },
                            "kubeApi": {
                                "serviceNodePortRange": "30000-32767"
                            },
                            "kubeController": {},
                            "kubelet": {},
                            "kubeproxy": {},
                            "scheduler": {}
                        },
                        "sshAgentAuth": false,
                        "systemImages": {
                            "alpine": "rancher/rke-tools:v0.1.28",
                            "calicoCni": "rancher/calico-cni:v3.4.0",
                            "calicoCtl": "rancher/calico-ctl:v2.0.0",
                            "calicoNode": "rancher/calico-node:v3.4.0",
                            "canalCni": "rancher/calico-cni:v3.4.0",
                            "canalFlannel": "rancher/coreos-flannel:v0.10.0",
                            "canalNode": "rancher/calico-node:v3.4.0",
                            "certDownloader": "rancher/rke-tools:v0.1.28",
                            "coredns": "coredns/coredns:1.3.1",
                            "corednsAutoscaler": "rancher/cluster-proportional-autoscaler:1.3.0",
                            "dnsmasq": "rancher/k8s-dns-dnsmasq-nanny:1.15.0",
                            "etcd": "rancher/coreos-etcd:v3.3.10-rancher1",
                            "flannel": "rancher/coreos-flannel:v0.10.0-rancher1",
                            "flannelCni": "rancher/flannel-cni:v0.3.0-rancher1",
                            "ingress": "rancher/nginx-ingress-controller:0.21.0-rancher3",
                            "ingressBackend": "rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1",
                            "kubedns": "rancher/k8s-dns-kube-dns:1.15.0",
                            "kubednsAutoscaler": "rancher/cluster-proportional-autoscaler:1.3.0",
                            "kubednsSidecar": "rancher/k8s-dns-sidecar:1.15.0",
                            "kubernetes": "rancher/hyperkube:v1.14.1-rancher1",
                            "kubernetesServicesSidecar": "rancher/rke-tools:v0.1.28",
                            "metricsServer": "rancher/metrics-server:v0.3.1",
                            "nginxProxy": "rancher/rke-tools:v0.1.28",
                            "podInfraContainer": "rancher/pause:3.1",
                            "weaveCni": "weaveworks/weave-npc:2.5.0",
                            "weaveNode": "weaveworks/weave-kube:2.5.0"
                        }
                    }
                },
                "authImage": "",
                "caCert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN3akNDQWFxZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFTTVJBd0RnWURWUVFERXdkcmRXSmwKTFdOaE1CNFhEVEU1TURVeE5qRXpORGcxTWxvWERUSTVNRFV4TXpFek5EZzFNbG93RWpFUU1BNEdBMVVFQXhNSAphM1ZpWlMxallUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU95c1BRdE9RcUhnCldhSGZJalo3b2NUSFZ2NzFoVGFmUmVkelhRS212WkJhaGM5N2lVYXdWT3dQN1NOQmFqUlNiRlhFSmpIanZ0cUUKdkk5THcyZFE0YUh3dmpqTzY4REVaSVlob0ozS2Nxb3VjYUZBQnd1ZDVCNGFRRi83dThkbTFRZlpBM1BPNmhhawo1UTRUMkE2eDNkZFB6eXlVaTBjUGoxUk1zTW15Z3gyUklnakRjZUZmVGY5Z2JtTG5Ec2Y1bVZGdjE4b1JpMmdjCjZrbk1tYm1lZnAwbGovc29mNTAxVHo1ckYrRUpZZUVJUHRGQm1ZRUg2cDkySjVzK2FhQWRYS29rKzVkaCswQlMKa3gzc2FKZWJud3RMd25kcUVMdmd1K3NRS051LzM1akluKzFQVzZ4dll2RGRiMHpudms1cTVhZkZJWmNDZnhaawpVZzV5bzFYWlg5RUNBd0VBQWFNak1DRXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CCkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBR1dhTFdaRVd3b2pHMGw1KzMwUmNlUWlPZ0d0MStvcUx3ZHoKazlGUkdvZW5Gc0VpNWY0Q1RVWVVodkg5Zm1mTDR3eHBTTkNhY1Z3alY3a3NJTitHK2ErTzM4anhWT0dyKzdpQgoydmd6VnU0bFR4ZTk0djliZVk2Y0JhN0l2MWdWdFk2NTBySkwxMlFMS29OYUgrT3ltVmptMVIxNWxTNmhYOEQ0Cms1NWZSQk1vaCtrR0tmbXBDSkxsUzVtbEo4VVgwVzVtWThWNndSM2JLWmtwVEJQcm9walZGeWRHTEEvQXF0RWYKWjVuU2J6US9UazRXQS9BMjErZUFtZDNHcTdZQkIyRkZFV2pDREcyYk1IVStVVm9xallyN1BKL3QyaGhaU3M4MAp1azdHbTdqTXBWRy8zOWtpWWpvV3dqRHdoWnQ2eVc5eTdsNXA5Tzk3N09zZCs4OVFub289Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
                "capabilities": {
                    "ingressCapabilities": [
                        {
                            "ingressProvider": "Nginx"
                        }
                    ],
                    "loadBalancerCapabilities": {
                        "enabled": false
                    },
                    "nodePortRange": "30000-32767"
                },
                "capacity": {
                    "cpu": "8",
                    "memory": "16326776Ki",
                    "pods": "220"
                },
                "componentStatuses": [
                    {
                        "conditions": [
                            {
                                "message": "ok",
                                "status": "True",
                                "type": "Healthy"
                            }
                        ],
                        "name": "controller-manager"
                    },
                    {
                        "conditions": [
                            {
                                "message": "{\"health\":\"true\"}",
                                "status": "True",
                                "type": "Healthy"
                            }
                        ],
                        "name": "etcd-0"
                    },
                    {
                        "conditions": [
                            {
                                "message": "ok",
                                "status": "True",
                                "type": "Healthy"
                            }
                        ],
                        "name": "scheduler"
                    }
                ],
                "conditions": [
                    {
                        "status": "True",
                        "type": "Pending"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:50:47Z",
                        "status": "True",
                        "type": "Provisioned"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:51:02Z",
                        "status": "True",
                        "type": "Waiting"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "BackingNamespaceCreated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "DefaultProjectCreated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "SystemProjectCreated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "InitialRolesPopulated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "CreatorMadeOwner"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "NoDiskPressure"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:48:41Z",
                        "status": "True",
                        "type": "NoMemoryPressure"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:50:47Z",
                        "status": "True",
                        "type": "GlobalAdminsSynced"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:50:48Z",
                        "status": "False",
                        "type": "AlertingEnabled"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:50:49Z",
                        "status": "True",
                        "type": "SystemAccountCreated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:50:49Z",
                        "status": "True",
                        "type": "AgentDeployed"
                    },
                    {
                        "lastUpdateTime": "2019-05-17T22:40:45Z",
                        "status": "True",
                        "type": "PrometheusOperatorDeployed"
                    },
                    {
                        "lastUpdateTime": "2019-05-20T15:58:07Z",
                        "status": "True",
                        "type": "Updated"
                    },
                    {
                        "lastUpdateTime": "2019-05-16T13:51:00Z",
                        "status": "True",
                        "type": "ServiceAccountMigrated"
                    },
                    {
                        "lastUpdateTime": "2019-06-06T08:17:40Z",
                        "status": "True",
                        "type": "Ready"
                    },
                    {
                        "lastUpdateTime": "2019-05-31T15:56:21Z",
                        "status": "True",
                        "type": "MonitoringEnabled"
                    }
                ],
                "driver": "rancherKubernetesEngine",
                "limits": {
                    "cpu": "2760m",
                    "memory": "2565Mi",
                    "pods": "0"
                },
                "monitoringStatus": {
                    "conditions": [
                        {
                            "lastUpdateTime": "2019-05-17T22:41:21Z",
                            "status": "True",
                            "type": "MetricExpressionDeployed"
                        }
                    ],
                    "grafanaEndpoint": "/k8s/clusters/c-jp966/api/v1/namespaces/cattle-prometheus/services/http:access-grafana:80/proxy/"
                },
                "requested": {
                    "cpu": "2590m",
                    "memory": "1817Mi",
                    "pods": "25"
                },
                "serviceAccountToken": "xxx",
                "version": {
                    "buildDate": "2019-04-08T17:02:58Z",
                    "compiler": "gc",
                    "gitCommit": "b7394102d6ef778017f2ca4046abbaa23b88c290",
                    "gitTreeState": "clean",
                    "gitVersion": "v1.14.1",
                    "goVersion": "go1.12.1",
                    "major": "1",
                    "minor": "14",
                    "platform": "linux/amd64"
                }
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

@alena1108 Thank you very much! You are absolutely right. Changing the values of monitor-delay and monitor-timeout manually  by changing the cluster specs (kubectl edit cluster c-xxx) has fixed the problem (at least for me).

Steps to mitigate:
* run kubectl on a rancher management plane, find the affected cluster
* kubectl edit clustername --namespace clusternamespace
* Find the param in the clusterSpec and ClusterStatus, quote the param and add a unit to it. Example: if it was 30, change it to "30s", if it was 0, change it to ""
* Click save

With the fix causing the issue, the assumption was made that the MonitorDelay and MonitorTimeout would never be set on user clusters as with them set the kubelet never starts. But Rancher api framework defaults the int value to 0 if not set.

I'll give that a shot tomorrow morning. Editing the cluster via kubectl will work, but a problem still exists if you try to modify the cluster in the Rancher UI. I can set it to "" with kubectl, but if I edit the cluster in the UI it interprets those as 0 still. But if you save it in the UI they get set back to 0.
As mentioned will test this out tomorrow.

@DJAyth

but a problem still exists if you try to modify the cluster in the Rancher UI. I can set it to "" with kubectl, but if I edit the cluster in the UI it interprets those as 0 still.

Rancher 2.2.4 UI will not set it to 0 anymore once you upgrade your existing clusters via kubectl prior to upgrade. On every subsequent cluster upgrade in 2.2.4, or for newly created clusters, the value will be defaulted to ""

Perfect. I'll give this a shot. Many thanks all.

Hi,
we have the same Problem with the upgrade to 2.2.4 and enabled Openstack LBAAs in 2 Clusters. I tried the workaround, edited the clusters and changed the values for monitor-delay & monitor-timeout.
After that i did an helm upgrade rancher rancher-latest/rancher --reuse-values
But the upgrade fails again:
kubectl --kubeconfig=rancherserver/kube_config_rancher-cluster.yml describe pod -n cattle-system rancher-b95ffcd98-zggn8
Name:               rancher-b95ffcd98-zggn8
Namespace:          cattle-system
Priority:           0
PriorityClassName:  <none>
Node:               rancherserver2.XXXXX/<IP_ADDRESS>
Start Time:         Thu, 06 Jun 2019 21:26:29 +0200
Labels:             app=rancher
                    pod-template-hash=b95ffcd98
                    release=rancher
Annotations:        cni.projectcalico.org/podIP: <IP_ADDRESS>/32
Status:             Running
IP:                 <IP_ADDRESS>
Controlled By:      ReplicaSet/rancher-b95ffcd98
Containers:
  rancher:
    Container ID:  docker://9612e698932db42a29468fccc37f987541b00c7a9023ef5c293cfc89bd8197b0
    Image:         rancher/rancher:v2.2.4
    Image ID:      docker-pullable://rancher/rancher@sha256:b22d5e239b8750edb639d0fe181c528fccacea67fb23412dff3c5f66e57e44a2
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=auto
    State:          Running
      Started:      Thu, 06 Jun 2019 21:28:33 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Jun 2019 21:26:31 +0200
      Finished:     Thu, 06 Jun 2019 21:28:32 +0200
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-m7mqf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  rancher-token-m7mqf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-m7mqf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                             Message
  ----     ------     ----                   ----                                             -------
  Normal   Scheduled  4m33s                  default-scheduler                                Successfully assigned cattle-system/rancher-b95ffcd98-zggn8 to rancherserver2.XXXXX
  Normal   Pulled     2m30s (x2 over 4m32s)  kubelet, rancherserver2.XXXXX  Container image "rancher/rancher:v2.2.4" already present on machine
  Normal   Killing    2m30s                  kubelet, rancherserver2.XXXXX  Killing container with id docker://rancher:Container failed liveness probe.. Container will be killed and recreated.

Failed to list *v3.Cluster: json: cannot unmarshal number into Go value of type string

@Redbierbaron what did you edit the values of monitor-delay & monitor-timeout to?

@mrajashree i did the same steps:

In 2.2.3, those values were 0
I upgraded to 2.2.4, upgrade fails - got: Failed to list *v3.Cluster: json: cannot unmarshal number into Go value of type string
I edited clusters to make them ""
pods did not start with error above

Log output from rancher 2.2.4 container:
2019/06/07 06:13:58 [INFO] Adding peer wss://<IP_ADDRESS>/v3/connect, <IP_ADDRESS> 2019/06/07 06:13:58 [INFO] Adding peer wss://<IP_ADDRESS>/v3/connect, <IP_ADDRESS> 2019/06/07 06:13:58 [INFO] Adding peer wss://<IP_ADDRESS>/v3/connect, <IP_ADDRESS> 2019/06/07 06:13:58 [INFO] Listening on :443 2019/06/07 06:13:58 [INFO] Listening on :80 I0607 06:13:58.194517      10 http.go:110] HTTP2 has been explicitly disabled 2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:13:58 [INFO] Stopping cluster agent for c-7cbb2 I0607 06:13:58.693483      10 http.go:110] HTTP2 has been explicitly disabled 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:13:59 [INFO] Starting cluster controllers for local I0607 06:13:59.359300      10 leaderelection.go:187] attempting to acquire leader lease  kube-system/cattle-controllers... 2019/06/07 06:13:59 [INFO] Starting cluster agent for local [owner=false] 2019/06/07 06:13:59 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:13:59 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:13:59 [ERROR] Failed syncing peers [{<IP_ADDRESS> [<IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>] false false}]: failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:13:59 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:13:59 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:00 [ERROR] ClusterController local failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:00 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:00 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:00 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:00 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:00 [ERROR] ClusterController local failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:01 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:02 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:02 [ERROR] ClusterController local failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:02 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:02 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:02 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:02 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:02 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:02 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:03 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:03 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:03 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:04 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:04 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:04 [ERROR] ClusterController local failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:06 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:06 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:06 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:06 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:06 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:06 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:08 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:08 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:08 [ERROR] ClusterController local failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:08 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:08 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:08 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:13 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:13 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:13 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake 2019/06/07 06:14:14 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:14 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:14 [ERROR] ClusterController c-7cbb2 failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:14 [INFO] Stopping cluster agent for c-7cbb2 2019/06/07 06:14:15 [INFO] Handling backend connection request [<IP_ADDRESS>] 2019/06/07 06:14:15 [INFO] Handling backend connection request [<IP_ADDRESS>] 2019/06/07 06:14:15 [INFO] Handling backend connection request [<IP_ADDRESS>] 2019/06/07 06:14:15 [INFO] Removing peer <IP_ADDRESS> 2019/06/07 06:14:15 [INFO] Stopping cluster agent for c-c8g4g 2019/06/07 06:14:15 [ERROR] ClusterController c-c8g4g failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect, failed to start user controllers for cluster c-c8g4g: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:15 [INFO] Starting cluster controllers for c-7cbb2 2019/06/07 06:14:15 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF 2019/06/07 06:14:15 [INFO] Starting cluster agent for c-7cbb2 [owner=false] 2019/06/07 06:14:15 [INFO] Starting cluster controllers for c-c8g4g 2019/06/07 06:14:15 [INFO] Starting cluster agent for c-c8g4g [owner=false] E0607 06:14:25.173896      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.174632      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.174706      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.174853      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.175074      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.175233      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.175298      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.175366      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.178985      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179154      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179203      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179236      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179298      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179347      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:25.179551      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=13896140&timeout=30m0s&timeoutSeconds=420: waiting for cluster agent to connect E0607 06:14:25.179708      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/watch/namespaces?resourceVersion=13896167&timeout=30m0s&timeoutSeconds=489: waiting for cluster agent to connect E0607 06:14:25.179833      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/rolebindings?resourceVersion=13896141&timeout=30m0s&timeoutSeconds=391: waiting for cluster agent to connect E0607 06:14:25.179990      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/watch/services?resourceVersion=13967675&timeout=30m0s&timeoutSeconds=335: waiting for cluster agent to connect E0607 06:14:25.180059      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/watch/endpoints?resourceVersion=14103482&timeout=30m0s&timeoutSeconds=431: waiting for cluster agent to connect E0607 06:14:25.180392      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=13898514&timeout=30m0s&timeoutSeconds=439: waiting for cluster agent to connect E0607 06:14:25.180720      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=11190268&timeout=30m0s&timeoutSeconds=524: waiting for cluster agent to connect 2019/06/07 06:14:25 [INFO] Handling backend connection request [c-c8g4g:m-frrh5] 2019/06/07 06:14:25 [INFO] Handling backend connection request [c-c8g4g:m-6z5r7] E0607 06:14:25.792229      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/watch/endpoints?resourceVersion=10020221&timeout=30m0s&timeoutSeconds=314: tunnel disconnect E0607 06:14:26.180090      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.181013      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.182038      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.184885      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.186042      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.187371      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:26.188219      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:26 [ERROR] Failed syncing peers [{<IP_ADDRESS> [<IP_ADDRESS> <IP_ADDRESS>] true false}]: failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: waiting for cluster agent to connect E0607 06:14:26.980279      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/watch/namespaces?resourceVersion=8232163&timeout=30m0s&timeoutSeconds=513: tunnel disconnect E0607 06:14:26.980651      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=7152695&timeout=30m0s&timeoutSeconds=325: tunnel disconnect E0607 06:14:26.980885      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=9652537&timeout=30m0s&timeoutSeconds=583: tunnel disconnect E0607 06:14:26.980964      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=478963&timeout=30m0s&timeoutSeconds=484: tunnel disconnect E0607 06:14:27.180609      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.181530      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.182508      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.187077      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.188270      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.188641      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:27.189620      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.181354      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.182074      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.183638      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.187594      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.188606      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.189748      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:28.191001      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect 2019/06/07 06:14:35 [INFO] Handling backend connection request [c-7cbb2:m-flcgl] 2019/06/07 06:14:35 [INFO] Handling backend connection request [local] 2019/06/07 06:14:36 [INFO] Handling backend connection request [c-7cbb2:m-t7bb9] 2019/06/07 06:14:38 [ERROR] ClusterController c-7cbb2 [user-controllers-controller] failed with : failed to start user controllers for cluster c-7cbb2: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: tunnel disconnect E0607 06:14:38.505823      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.505891      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.505953      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.506027      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.506079      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.506174      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect E0607 06:14:38.506265      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: tunnel disconnect 2019/06/07 06:14:42 [INFO] Handling backend connection request [c-c8g4g:m-wg8fx] 2019/06/07 06:14:45 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:53314: i/o timeout 2019/06/07 06:14:46 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:47922: i/o timeout 2019/06/07 06:14:47 [INFO] Adding peer wss://<IP_ADDRESS>/v3/connect, <IP_ADDRESS> 2019/06/07 06:14:48 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF 2019/06/07 06:14:48 [ERROR] Failed to serve peer connection <IP_ADDRESS>: websocket: close 1006 unexpected EOF 2019/06/07 06:14:48 [INFO] Handling backend connection request [<IP_ADDRESS>] 2019/06/07 06:14:48 [INFO] Removing peer <IP_ADDRESS> 2019/06/07 06:14:52 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:53344: i/o timeout 2019/06/07 06:14:58 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:14:58.195928      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.196023      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.205583      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=9652537&timeout=30m0s&timeoutSeconds=555: tunnel disconnect 2019/06/07 06:14:58 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF 2019/06/07 06:14:58 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:14:58.208680      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.209320      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.209366      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.209392      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:14:58.209866      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.ClusterRole ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received E0607 06:14:58.519873      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.520117      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.520230      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.520150      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.520183      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:14:58.520516      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.RoleBinding ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received W0607 06:14:58.520570      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Service ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received W0607 06:14:58.520635      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Endpoints ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received E0607 06:14:58.521904      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.522233      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.522385      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.523129      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/watch/services?resourceVersion=13967675&timeout=30m0s&timeoutSeconds=571: waiting for cluster agent to connect E0607 06:14:58.523337      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/watch/namespaces?resourceVersion=13896167&timeout=30m0s&timeoutSeconds=503: waiting for cluster agent to connect E0607 06:14:58.526138      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.526534      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.527062      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=13896140&timeout=30m0s&timeoutSeconds=407: waiting for cluster agent to connect E0607 06:14:58.526590      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.527442      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/rolebindings?resourceVersion=13896141&timeout=30m0s&timeoutSeconds=512: waiting for cluster agent to connect E0607 06:14:58.526633      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:14:58.527903      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/watch/endpoints?resourceVersion=14103594&timeout=30m0s&timeoutSeconds=339: waiting for cluster agent to connect E0607 06:14:58.528008      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=12424649&timeout=30m0s&timeoutSeconds=394: waiting for cluster agent to connect E0607 06:14:58.529388      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=13898514&timeout=30m0s&timeoutSeconds=480: waiting for cluster agent to connect E0607 06:14:59.523617      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.524723      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.527991      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.528519      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.529667      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.533158      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:14:59.534011      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.524110      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.525285      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.528585      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.529406      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.530517      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.533672      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:00.534630      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.524610      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/services?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.525750      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/namespaces?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.528951      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.530376      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.531547      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Endpoints: Get https://<IP_ADDRESS>:6443/api/v1/endpoints?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.534060      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect E0607 06:15:01.535079      10 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0&timeout=30s: waiting for cluster agent to connect 2019/06/07 06:15:02 [INFO] Handling backend connection request [c-7cbb2:m-jxc89] 2019/06/07 06:15:03 [INFO] Handling backend connection request [c-c8g4g:m-jhfl2] 2019/06/07 06:15:08 [INFO] Handling backend connection request [c-7cbb2:m-ktxln] 2019/06/07 06:15:08 [INFO] Handling backend connection request [c-c8g4g:m-6z5r7] 2019/06/07 06:15:08 [INFO] Handling backend connection request [local] 2019/06/07 06:15:08 [INFO] Handling backend connection request [c-7cbb2:m-jxc89] 2019/06/07 06:15:13 [INFO] Handling backend connection request [c-7cbb2:m-t7bb9] 2019/06/07 06:15:13 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:53798: i/o timeout 2019/06/07 06:15:16 [INFO] Handling backend connection request [c-7cbb2:m-4tlrs] 2019/06/07 06:15:18 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:47712: i/o timeout 2019/06/07 06:15:18 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:47718: i/o timeout 2019/06/07 06:15:26 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:15:26.082295      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082345      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082448      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082471      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082780      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082802      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:26.082831      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:26.087924      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Role ended with: too old resource version: 11190268 (12424649) 2019/06/07 06:15:26 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:53898: i/o timeout W0607 06:15:31.028242      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Role ended with: too old resource version: 477283 (478963) 2019/06/07 06:15:34 [INFO] Handling backend connection request [c-c8g4g:m-sswhn] E0607 06:15:37.079819      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect 2019/06/07 06:15:40 [INFO] Handling backend connection request [c-c8g4g:m-wg8fx] 2019/06/07 06:15:41 [INFO] Handling backend connection request [c-c8g4g:m-6z5r7] 2019/06/07 06:15:46 [INFO] Adding peer wss://<IP_ADDRESS>/v3/connect, <IP_ADDRESS> 2019/06/07 06:15:47 [ERROR] Failed to serve peer connection <IP_ADDRESS>: websocket: close 1006 unexpected EOF 2019/06/07 06:15:47 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:15:47.078038      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.078047      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.078131      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.078545      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.078609      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.079481      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:47.079575      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect 2019/06/07 06:15:47 [INFO] Removing peer <IP_ADDRESS> 2019/06/07 06:15:49 [INFO] Handling backend connection request [<IP_ADDRESS>] 2019/06/07 06:15:50 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:53968: i/o timeout 2019/06/07 06:15:55 [INFO] Handling backend connection request [c-7cbb2:m-4tlrs] E0607 06:15:56.372190      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372436      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372474      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372454      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372519      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372944      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.372972      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.393681      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=478963&timeout=30m0s&timeoutSeconds=476: tunnel disconnect 2019/06/07 06:15:56 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF 2019/06/07 06:15:56 [INFO] error in remotedialer server [400]: read tcp <IP_ADDRESS>:80-><IP_ADDRESS>:48140: read: connection reset by peer E0607 06:15:56.403368      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.403475      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.404067      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.404323      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.405687      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://<IP_ADDRESS>:6443/api/v1/watch/namespaces?resourceVersion=8232163&timeout=30m0s&timeoutSeconds=376: tunnel disconnect E0607 06:15:56.405843      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/watch/services?resourceVersion=9657754&timeout=30m0s&timeoutSeconds=568: tunnel disconnect E0607 06:15:56.406710      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.RoleBinding: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/rolebindings?resourceVersion=8193185&timeout=30m0s&timeoutSeconds=581: tunnel disconnect E0607 06:15:56.406866      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://<IP_ADDRESS>:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=7152695&timeout=30m0s&timeoutSeconds=437: tunnel disconnect 2019/06/07 06:15:56 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF 2019/06/07 06:15:56 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:15:56.924283      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:56.924341      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Endpoints ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received 2019/06/07 06:15:56 [INFO] error in remotedialer server [400]: websocket: close 1006 unexpected EOF E0607 06:15:56.924657      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:56.924677      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.ClusterRoleBinding ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received E0607 06:15:56.924715      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.924732      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:56.924736      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.RoleBinding ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received E0607 06:15:56.929528      10 reflector.go:251] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Service: Get https://<IP_ADDRESS>:6443/api/v1/watch/services?resourceVersion=13967675&timeout=30m0s&timeoutSeconds=495: failed to find Session for client c-7cbb2:m-z7nq8 E0607 06:15:56.929605      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect E0607 06:15:56.929685      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:56.929781      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Role ended with: very short watch: github.com/rancher/norman/controller/generic_controller.go:175: Unexpected watch close - watch lasted less than a second and no items received E0607 06:15:56.929941      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect 2019/06/07 06:15:57 [INFO] Handling backend connection request [local:machine-wzbv6] 2019/06/07 06:15:57 [INFO] Handling backend connection request [c-7cbb2] 2019/06/07 06:15:57 [INFO] Handling backend connection request [local:machine-4hpdd] E0607 06:15:59.735036      10 streamwatcher.go:109] Unable to decode an event from the watch stream: tunnel disconnect W0607 06:15:59.740022      10 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: watch of *v1.Role ended with: too old resource version: 477283 (478963)

after 10 Minutes of waiting, hundreds of error and restarts of the rancher-containers, the upgrade finishs.... the errors in the log correspond an issue https://github.com/rancher/rancher/issues/20265 (apparently nobody of the rancher team interested in....)

@Redbierbaron looks like a different issue, as the logs don't show cannot unmarshal number into Go value of type string anymore, and instead you see:
2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake
2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake
2019/06/07 06:13:58 [ERROR] Failed to connect to peer wss://<IP_ADDRESS>/v3/connect [local ID=<IP_ADDRESS>]: websocket: bad handshake
2019/06/07 06:13:58 [INFO] Stopping cluster agent for c-7cbb2

So yeah, if this could just work that would be great. ;)
This morning I edited the cluster via kubectl changing the values for the OpenStack LB to double quotes instead of the 0 it was. However, I still see the exact same behaviour, update fails, same error out of the pod. However, to top it off this time, now my second cluster is missing from the Rancher UI! It still shows with kubectl so it's there, but the UI doesn't see it at all.

Went back through the entire cluster and reset the values back to what they were and the cluster is back in the UI now.

@DJAyth can you try the steps mentioned by @kinarashah - she specifically tested it for HA use case:
In 2.2.3, those values were 0
I upgraded to 2.2.4, upgrade fails
I edited clusters to make them ""
pods start running and I can access the login page

If you change the value while being on 2.2.3, the 2.2.3 UI will fail to load as it doesn't understand string. I've updated my initial comment to reflect this info. Please let us know how if that works for you

This happens on OpenStack even when not using the load balancer. Those fields were still populated even while I was not specifying the loadBalancer block at all when creating the cluster.
The relevant subsection of "kubectl get cluster --all-namespaces -o json" looks like:
                       "loadBalancer": {
                            "create-monitor": false,
                            "floating-network-id": "",
                            "lb-method": "",
                            "lb-provider": "",
                            "lb-version": "",
                            "manage-security-groups": false,
                            "monitor-delay": 0,
                            "monitor-max-retries": 0,
                            "monitor-timeout": 0,
                            "subnet-id": "",
                            "use-octavia": false
                        },

Also found the workaround steps somewhat confusing. In that I was trying to find and edit those settings (monitor-delay and monitor-timeout) on the "local" cluster, cause that is the one to which I am re-deploying rancher server. And obviously not finding it cause "local" cluster is "imported" and won't show you these values. But in fact, it was the subordinate Custom cluster that I needed to edit. And the values for that one were affecting the inability for rancher container to start in the Rancher main "local" cluster. Just FYI for anyone struggling.

We had the same issue trying to update from 2..2.2 to 2.24, followed the instructions and had rancher running with 2.2.4 finally.
BUT, with every change of a client cluster (editing the cluster.yaml), the cluster is seen unhealthy by rancher.

This cluster is currently Updating.
[etcd] Failed to bring up Etcd Plane: [etcd] Etcd Cluster is not healthy

Enabled DEBUG shows the following error on the rancher application container:
 [DEBUG] Failed to get /health for host [<IP_ADDRESS>]: Get https://<IP_ADDRESS>:2379/health: remote error: tls: bad certificate
etcd in client clusters are with 3 healthy member
That looks like known issues with rke (e.g. https://github.com/rancher/rke/issues/1244), where a missing rke cluster state file let rke create new certificates not recognizing that there is a cluster.
Never thought something like this can happen with rancher, with having rke state files as config map in every client cluster and also under /var/lib/rancher/management-state/rke in one application container.
Now it seems that all our client clusters are unusable furthermore, either already in faulty state or will be with next change in cluster.yaml.
Anybody with same experiences? Any suggestion how to solve this?

@galal-hussein has this been seen before? after upgrade to 2.2.4
 I verified that the configmap full-cluster-state still contains original certificates, but the state files under /var/lib/rancher/management-state/rke in the rancher application container contains new certificates geenrated after update.

@galal-hussein, @kinarashah
I opened a separate issue for the above problem: https://github.com/rancher/rancher/issues/20909
Some more analysis showed that rancher master is constantly create new certificates for the faulty client cluster, but with this certificates the (unchanged, but working) etcds of client cluster can not be reached.

@surco1971 the issue you are seeing, is unrelated to the original issue described in the first comment. Can you file it as a separate bug?

@alena1108 , that is what I did - see above comment - https://github.com/rancher/rancher/issues/20909. And I mentioned it here, because the origin of all is to try to update to 2.2.4. So it is a kind of related, at least for users, which potentially can at the end having faulty client clusters.

I am seeing the same issue. My steps:

Try to upgrade to 2.2.4
See that theres the error about unmarshalling number
edit the clusters in kubectl and change the monitor- stuff from 0 to ""
The error persists
Figure I will try to use helm to apply the "same" upgrade

???
I got nothing. Help?

@tibers ideally, there's no need to do step 5.
Could you confirm step 3 was applied correctly by doing kubectl get cluster --all-namespaces -o json and checking values of monitor-delay & monitor-timeout?

Since we have a mix of openstack clusters and all sorts of other fun stuff in rancher, what I did was:
for h in `kubectl get clusters | grep -v NAME | cut -f 1 -d ' '`
do
kubectl edit cluster $h
done

Which edits EVERYTHING, and then while I was in vi I changed monitor-delay and monitor-timeout to just monitor-delay: "" and monitor-timeout: "".  Then trying to get 2.2.4 installed still throws the error. Is there possibly another value which has to be set to ""?

@tibers you can set it to any acceptable value for delay and timeout based on your use case, just needs to be in string format

I went back and changed both those to "1s" and have the same problem:
reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v3.Cluster: json: cannot unmarshal number into Go value of type string

I saw the problem in client v2.2.0:

cannot unmarshal string into Go struct field LoadBalancerOpenstackOpts.monitor-delay of type int64

But downloading latest rc for client v2.3. instead solved it for me.

Rancher upgrade from 2.2.3 to 2.2.9 went well with the workaround (manually editing those cluster values from 0 to "" with kubectl).
However now that I try to upgrade k8s from v1.13.5-rancher1-2 to v1.14.9-rancher1-1 in one of those clusters I'm getting:
[ERROR] ClusterController c-xxxxx [certificate-expiration] failed with : json: cannot unmarshal number into Go struct field LoadBalancerOpenstackOpts.monitor-delay of type string
I'm not really sure where could rancher be getting the int-type value since the values for monitor-delay and monitor-timeout are strings in the cluster configuration.
I tried including both parameters as string in the yaml file but I get the same results.
As a result the cluster is permanently stuck in a Updating state showing CA Certificate or Key is empty
Extra info: our clusters dont use Openstack LoadBalancer and they were provisioned with rancher older than 2.2.0
Has anyone tried this? @kinarashah do you maybe have an idea?
Thanks in advance for any help!

@pameladelgado could you post your cluster yaml?

@kinarashah sure
kind: Cluster
metadata:
  annotations:
    authz.management.cattle.io/creator-role-bindings: '{"created":["cluster-owner"],"required":["cluster-owner"]}'
    field.cattle.io/creatorId: u-zixjny6zsp
    lifecycle.cattle.io/create.cluster-agent-controller-cleanup: "true"
    lifecycle.cattle.io/create.cluster-provisioner-controller: "true"
    lifecycle.cattle.io/create.cluster-scoped-gc: "true"
    lifecycle.cattle.io/create.mgmt-cluster-rbac-remove: "true"
    provisioner.cattle.io/ke-driver-update: updated
  creationTimestamp: 2018-11-29T15:20:43Z
  finalizers:
  - controller.cattle.io/cluster-agent-controller-cleanup
  - controller.cattle.io/cluster-provisioner-controller
  - controller.cattle.io/mgmt-cluster-rbac-remove
  - controller.cattle.io/cluster-scoped-gc
  generation: 2593
  labels:
    cattle.io/creator: norman
  name: c-xxxxx
  resourceVersion: "201622471"
  selfLink: /apis/management.cattle.io/v3/clusters/c-xxxxx
  uid: 56790e48-f3ea-11e8-8170-fa163e548b12
spec:
  description: DNS address xxx Git repo xxx
  desiredAgentImage: ""
  desiredAuthImage: ""
  displayName: xxxx
  dockerRootDir: /var/lib/docker
  enableClusterAlerting: true
  enableClusterMonitoring: false
  enableNetworkPolicy: false
  internal: false
  localClusterAuthEndpoint:
    enabled: false
  rancherKubernetesEngineConfig:
    addonJobTimeout: 30
    authentication:
      sans:
      - 86.xxx.xx.xxx
      - my.dns.xxx
      strategy: x509
    authorization: {}
    bastionHost: {}
    cloudProvider:
      name: openstack
      openstackCloudProvider:
        blockStorage:
          bs-version: v3
          ignore-volume-az: true
          trust-device-path: false
        global:
          auth-url: https://keystone.url
          ca-file: ""
          domain-id: ""
          domain-name: xxxx
          password: xxxx
          region: xxxx
          tenant-id: xxxx
          tenant-name: xxxx
          trust-id: ""
          user-id: ""
          username: xxxx
        loadBalancer:
          create-monitor: false
          floating-network-id: ""
          lb-method: ""
          lb-provider: ""
          lb-version: ""
          manage-security-groups: false
          monitor-delay: ""
          monitor-max-retries: 0
          monitor-timeout: ""
          subnet-id: ""
          use-octavia: false
        metadata:
          request-timeout: 0
          search-order: ""
        route:
          router-id: xxxx
    ignoreDockerVersion: true
    ingress:
      provider: none
    kubernetesVersion: v1.13.5-rancher1-2
    monitoring:
      provider: none
    network:
      plugin: weave
      weaveNetworkProvider:
        password: xxxx
    restore: {}
    services:
      etcd:
        backupConfig:
          enabled: false
          intervalHours: 12
          retention: 6
          s3BackupConfig: null
        creation: 12h
        extraArgs:
          election-timeout: "5000"
          heartbeat-interval: "500"
        retention: 72h
        snapshot: false
      kubeApi:
        serviceNodePortRange: 30000-32767
      kubeController: {}
      kubelet:
        extraArgs:
          eviction-hard: memory.available<500Mi,nodefs.available<1Gi,imagefs.available<1Gi
          system-reserved: cpu=100m,memory=1.5Gi,ephemeral-storage=4Gi
      kubeproxy: {}
      scheduler: {}
    sshAgentAuth: false
    systemImages: {}
status:
  agentImage: rancher/rancher-agent:v2.2.9
  allocatable:
    cpu: 47600m
    memory: 189477024Ki
    pods: "440"
  apiEndpoint: https://<IP_ADDRESS>:6443
  appliedEnableNetworkPolicy: false
  appliedPodSecurityPolicyTemplateId: ""
  appliedSpec:
    description: DNS address xxxx Git repo xxxx
    desiredAgentImage: ""
    desiredAuthImage: ""
    displayName: xxxx
    dockerRootDir: /var/lib/docker
    enableClusterAlerting: true
    enableClusterMonitoring: false
    enableNetworkPolicy: false
    internal: false
    localClusterAuthEndpoint:
      enabled: false
    rancherKubernetesEngineConfig:
      addonJobTimeout: 30
      authentication:
        sans:
        - 86.xxx.xx.xxx
        - my.dns.xxx
        strategy: x509
      authorization: {}
      bastionHost: {}
      cloudProvider:
        name: openstack
        openstackCloudProvider:
          blockStorage:
            bs-version: v3
            ignore-volume-az: true
            trust-device-path: false
          global:
            auth-url: https://keystone.url
            ca-file: ""
            domain-id: ""
            domain-name: xxxx
            password: xxxx
            region: xxxx
            tenant-id: xxxx
            tenant-name: xxxx
            trust-id: ""
            user-id: ""
            username: xxxx
          loadBalancer:
            create-monitor: false
            floating-network-id: ""
            lb-method: ""
            lb-provider: ""
            lb-version: ""
            manage-security-groups: false
            monitor-delay: ""
            monitor-max-retries: 0
            monitor-timeout: ""
            subnet-id: ""
            use-octavia: false
          metadata:
            request-timeout: 0
            search-order: ""
          route:
            router-id: xxxx
      ignoreDockerVersion: true
      ingress:
        provider: none
      kubernetesVersion: v1.13.5-rancher1-2
      monitoring:
        provider: none
      network:
        plugin: weave
      nodes:
      - address: <IP_ADDRESS>
        hostnameOverride: node-2
        nodeName: c-xxxxx:m-02278373fb98
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: master
        nodeName: c-xxxxx:m-07bd7c8f6906
        port: "22"
        role:
        - etcd
        - controlplane
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: ngpu-2
        nodeName: c-xxxxx:m-157f12514e12
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: ngpu-1
        nodeName: c-xxxxx:m-27c807c18309
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: test
        nodeName: c-xxxxx:m-2851d83353d0
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: node-1
        nodeName: c-xxxxx:m-2dce1d568bb5
        port: "22"
        role:
        - worker
        user: root
      restore: {}
      services:
        etcd:
          backupConfig:
            enabled: false
            intervalHours: 12
            retention: 6
            s3BackupConfig: null
          creation: 12h
          extraArgs:
            election-timeout: "5000"
            heartbeat-interval: "500"
          retention: 72h
          snapshot: false
        kubeApi:
          serviceNodePortRange: 30000-32767
        kubeController: {}
        kubelet:
          extraArgs:
            eviction-hard: memory.available<500Mi,nodefs.available<1Gi,imagefs.available<1Gi
            system-reserved: cpu=100m,memory=1.5Gi,ephemeral-storage=4Gi
        kubeproxy: {}
        scheduler: {}
      sshAgentAuth: false
      systemImages:
        alpine: rancher/rke-tools:v0.1.27
        calicoCni: rancher/calico-cni:v3.4.0
        calicoCtl: rancher/calico-ctl:v2.0.0
        calicoNode: rancher/calico-node:v3.4.0
        canalCni: rancher/calico-cni:v3.4.0
        canalFlannel: rancher/coreos-flannel:v0.10.0
        canalNode: rancher/calico-node:v3.4.0
        certDownloader: rancher/rke-tools:v0.1.27
        coredns: coredns/coredns:1.2.6
        corednsAutoscaler: rancher/cluster-proportional-autoscaler:1.0.0
        dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
        etcd: rancher/coreos-etcd:v3.2.24-rancher1
        flannel: rancher/coreos-flannel:v0.10.0-rancher1
        flannelCni: rancher/flannel-cni:v0.3.0-rancher1
        ingress: rancher/nginx-ingress-controller:0.21.0-rancher3
        ingressBackend: rancher/nginx-ingress-controller-defaultbackend:1.4-rancher1
        kubedns: rancher/k8s-dns-kube-dns:1.15.0
        kubednsAutoscaler: rancher/cluster-proportional-autoscaler:1.0.0
        kubednsSidecar: rancher/k8s-dns-sidecar:1.15.0
        kubernetes: rancher/hyperkube:v1.13.5-rancher1
        kubernetesServicesSidecar: rancher/rke-tools:v0.1.27
        metricsServer: rancher/metrics-server:v0.3.1
        nginxProxy: rancher/rke-tools:v0.1.27
        podInfraContainer: rancher/pause:3.1
        weaveCni: weaveworks/weave-npc:2.5.0
        weaveNode: weaveworks/weave-kube:2.5.0
  authImage: ""
  caCert: xxxx
  capabilities:
    ingressCapabilities:
    - ingressProvider: Nginx
    loadBalancerCapabilities: {}
    nodePortRange: 30000-32767
  capacity:
    cpu: "48"
    memory: 197816480Ki
    pods: "440"
  componentStatuses:
  - conditions:
    - message: ok
      status: "True"
      type: Healthy
    name: controller-manager
  - conditions:
    - message: '{"health": "true"}'
      status: "True"
      type: Healthy
    name: etcd-0
  - conditions:
    - message: ok
      status: "True"
      type: Healthy
    name: scheduler
  conditions:
  - lastUpdateTime: 2018-11-29T15:20:43Z
    status: "True"
    type: BackingNamespaceCreated
  - lastUpdateTime: 2018-11-29T15:20:43Z
    status: "True"
    type: DefaultProjectCreated
  - lastUpdateTime: 2018-11-29T15:20:43Z
    status: "True"
    type: SystemProjectCreated
  - lastUpdateTime: 2018-11-29T15:20:43Z
    status: "True"
    type: InitialRolesPopulated
  - lastUpdateTime: 2018-11-29T15:20:43Z
    status: "True"
    type: Pending
  - lastUpdateTime: 2018-11-29T15:26:29Z
    status: "True"
    type: Provisioned
  - lastUpdateTime: 2018-11-29T15:26:45Z
    status: "True"
    type: Waiting
  - lastUpdateTime: 2018-11-29T15:20:44Z
    status: "True"
    type: NoDiskPressure
  - lastUpdateTime: 2018-11-29T15:20:44Z
    status: "True"
    type: NoMemoryPressure
  - lastUpdateTime: 2018-11-29T15:20:44Z
    status: "True"
    type: CreatorMadeOwner
  - lastUpdateTime: 2018-11-29T15:26:30Z
    status: "True"
    type: GlobalAdminsSynced
  - lastUpdateTime: 2020-01-10T13:20:37Z
    message: CA Certificate or Key is empty
    reason: Error
    status: "False"
    type: Updated
  - lastUpdateTime: 2018-11-29T15:26:37Z
    status: "True"
    type: SystemAccountCreated
  - lastUpdateTime: 2018-11-29T15:26:38Z
    status: "True"
    type: AgentDeployed
  - lastUpdateTime: 2020-01-10T15:10:18Z
    status: "True"
    type: Ready
  - lastUpdateTime: 2020-01-10T19:18:04Z
    status: "True"
    type: PrometheusOperatorDeployed
  - lastUpdateTime: 2019-05-20T20:04:39Z
    status: "True"
    type: ServiceAccountMigrated
  - lastUpdateTime: 2019-07-04T14:20:20Z
    status: "True"
    type: AlertingEnabled
  driver: rancherKubernetesEngine
  failedSpec:
    description: DNS address xxxx Git repo xxxx
    desiredAgentImage: ""
    desiredAuthImage: ""
    displayName: xxxx
    dockerRootDir: /var/lib/docker
    enableClusterAlerting: true
    enableClusterMonitoring: false
    enableNetworkPolicy: false
    internal: false
    localClusterAuthEndpoint:
      enabled: false
    rancherKubernetesEngineConfig:
      addonJobTimeout: 30
      authentication:
        sans:
        - 86.xxx.xx.xxx
        - my.dns.xxx
        strategy: x509
      authorization: {}
      bastionHost: {}
      cloudProvider:
        name: openstack
        openstackCloudProvider:
          blockStorage:
            bs-version: v3
            ignore-volume-az: true
            trust-device-path: false
          global:
            auth-url: https://keystone.url
            ca-file: ""
            domain-id: ""
            domain-name: xxxx
            password: xxxx
            region: xxxx
            tenant-id: xxxx
            tenant-name: xxxx
            trust-id: ""
            user-id: ""
            username: xxxx
          loadBalancer:
            create-monitor: false
            floating-network-id: ""
            lb-method: ""
            lb-provider: ""
            lb-version: ""
            manage-security-groups: false
            monitor-delay: ""
            monitor-max-retries: 0
            monitor-timeout: ""
            subnet-id: ""
            use-octavia: false
          metadata:
            request-timeout: 0
            search-order: ""
          route:
            router-id: xxxx
      ignoreDockerVersion: true
      ingress:
        provider: none
      kubernetesVersion: v1.13.5-rancher1-2
      monitoring:
        provider: none
      network:
        plugin: weave
        weaveNetworkProvider:
          password: xxxx
      nodes:
      - address: <IP_ADDRESS>
        hostnameOverride: node-2
        nodeName: c-xxxxx:m-02278373fb98
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: master
        nodeName: c-xxxxx:m-07bd7c8f6906
        port: "22"
        role:
        - etcd
        - controlplane
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: ngpu-2
        nodeName: c-xxxxx:m-157f12514e12
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: ngpu-1
        nodeName: c-xxxxx:m-27c807c18309
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: test
        nodeName: c-xxxxx:m-2851d83353d0
        port: "22"
        role:
        - worker
        user: root
      - address: <IP_ADDRESS>
        hostnameOverride: node-1
        nodeName: c-xxxxx:m-2dce1d568bb5
        port: "22"
        role:
        - worker
        user: root
      restore: {}
      services:
        etcd:
          backupConfig:
            enabled: false
            intervalHours: 12
            retention: 6
            s3BackupConfig: null
          creation: 12h
          extraArgs:
            election-timeout: "5000"
            heartbeat-interval: "500"
          retention: 72h
          snapshot: false
        kubeApi:
          serviceNodePortRange: 30000-32767
        kubeController: {}
        kubelet:
          extraArgs:
            eviction-hard: memory.available<500Mi,nodefs.available<1Gi,imagefs.available<1Gi
            system-reserved: cpu=100m,memory=1.5Gi,ephemeral-storage=4Gi
        kubeproxy: {}
        scheduler: {}
      sshAgentAuth: false
      systemImages:
        alpine: rancher/rke-tools:v0.1.27
        calicoCni: rancher/calico-cni:v3.4.0
        calicoCtl: rancher/calico-ctl:v2.0.0
        calicoNode: rancher/calico-node:v3.4.0
        canalCni: rancher/calico-cni:v3.4.0
        canalFlannel: rancher/coreos-flannel:v0.10.0
        canalNode: rancher/calico-node:v3.4.0
        certDownloader: rancher/rke-tools:v0.1.27
        coredns: coredns/coredns:1.2.6
        corednsAutoscaler: rancher/cluster-proportional-autoscaler:1.0.0
        dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
        etcd: rancher/coreos-etcd:v3.2.24-rancher1
        flannel: rancher/coreos-flannel:v0.10.0-rancher1
        flannelCni: rancher/flannel-cni:v0.3.0-rancher1
        ingress: rancher/nginx-ingress-controller:0.21.0-rancher3
        ingressBackend: rancher/nginx-ingress-controller-defaultbackend:1.4-rancher1
        kubedns: rancher/k8s-dns-kube-dns:1.15.0
        kubednsAutoscaler: rancher/cluster-proportional-autoscaler:1.0.0
        kubednsSidecar: rancher/k8s-dns-sidecar:1.15.0
        kubernetes: rancher/hyperkube:v1.13.5-rancher1
        kubernetesServicesSidecar: rancher/rke-tools:v0.1.27
        metricsServer: rancher/metrics-server:v0.3.1
        nginxProxy: rancher/rke-tools:v0.1.27
        podInfraContainer: rancher/pause:3.1
        weaveCni: weaveworks/weave-npc:2.5.0
        weaveNode: weaveworks/weave-kube:2.5.0
  limits:
    cpu: 5300m
    memory: 24005Mi
    pods: "0"
  requested:
    cpu: 12320m
    memory: 35221Mi
    pods: "61"
  serviceAccountToken: xxxx
  version:
    buildDate: 2019-03-25T15:19:22Z
    compiler: gc
    gitCommit: 2166946f41b36dea2c4626f90a77706f426cdea2
    gitTreeState: clean
    gitVersion: v1.13.5
    goVersion: go1.11.5
    major: "1"
    minor: "13"
    platform: linux/amd64

I can also create a separate issue.

For anyone else seeing the same issue as @pameladelgado, we fixed it using:  https://github.com/rancher/rancher/issues/20909#issuecomment-521352246

cert rotate helped to fix this issue
rke cert rotate --config cluster.yml