cattle-agents: "Failed to connect to proxy" error="websocket: bad handshake" in Rancher HA Air Gap Install (RKE Addon) with Certs signed by Private CA

What kind of request is this (question/bug/enhancement/feature request):
bug??
I cannot get the Rancher HA configuration for an Air Gap environment to work and I am using the RKE Add-on Install... I have tried the helm chart install method with no success.
I am using rancher/rancher:v2.0.8
After running "rke up --config rancher-cluster.yaml", the rancher servers appear to operate fine. I can access the Rancher UI and set the server-url. Once set, the cattle-agents deploy and show logging errors like so:
time="2019-03-19T17:12:41Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token hstztrvjzqx4nsg4nldscxvnn6ncx5w6xcqtmzrg4l25rxw5d8fwsx"
time="2019-03-19T17:12:41Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-19T17:12:41Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-19T17:12:41Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"

Steps to reproduce (least amount of steps as possible):

Provision 3 AWS t2.large ec2-instances with one in AZ-A and the others in AZ-B (ami: RHEL7)

Air Gap Environment!

Provision AWS Network Load Balancer based on config described here, with a minor tweak:

https://rancher.com/docs/rancher/v2.x/en/installation/ha/rke-add-on/layer-4-lb/nlb/
minor tweak = set the NLB to "internal"... NOT "internet-facing" (due to Air Gap Install)
Side Note: NLB Config Document should be updated for this use-case for Air Gap environment:
- for air gap env --> the NLB will be internal... NOT internet-facing
- AWS ELB's are innately configured to reject loop-back requests. This means when the cattle-agent is deployed on the same host as the rancher server, if the agent send the request to the AWS ELB and the request gets routed back to the same host, the request will just hang.
- AWS identifies a solution to this by changing Target Type from "Instance" to "IP"
- doc can be viewed: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout

Generate/grab certs to be used for Rancher (signed certs with private CA)
Pull all the needed docker images into private registry (as per Rancher Docs)
Initialize kubernetes cluster with rancher deployed to it using rancher-cluster.yaml and rke

rke version: v0.1.16
rancher/rancher:v2.0.8
kubernetes version: v1.11.2-rancher1-1
take rancher-cluster.yaml and replace the following in the config file:
- address for all 3 nodes
- internal_address for all 3 nodes
- ssh_key_path for all 3 nodes
- url for private_registry
- replace "hostname-for-private-registry:5000" everywhere in file with the correct url for private_registry
- insert tls.crt in base64 encoding
- insert tls.key in base64 encoding
- insert cacerts.pem in base64 encoding
- replace "my-rancher-cluster.com" with FQDN for AWS NLB which gives access to cattle server
- replace the string

See rancher-cluster.yaml below:
nodes:
- address: ip-XX-XX-1-XX.us-west-1.compute.internal
  internal_address: XX.XX.1.XX
  user: rke-user
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /path/to/ssh_key.pem
- address: ip-XX-XX-2-YY.us-west-1.compute.internal
  internal_address: XX.XX.2.YY
  user: rke-user
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /path/to/ssh_key.pem
- address: ip-XX-XX-1-ZZ.us-west-1.compute.internal
  internal_address: XX.XX.1.ZZ
  user: rke-user
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /path/to/ssh_key.pem

private_registries:
- url: hostname-for-private-registry:5000
  is_default: true

cluster_name: my-cluster

network:
    plugin: calico

authorization:
    mode: rbac
addon_job_timeout: 30
authentication:
  strategy: "x509"

ignore_docker_version: true

ingress:
  provider: "nginx"
  extra_args:
    default-ssl-certificate: "cattle-system/cattle-keys-ingress"

kubernetes_version: "v1.11.2-rancher1-1"
monitoring:
  provider: "metrics-server"

network:
  plugin: "calico"

services:
  etcd:
    extra_args:
      election-timeout: "5000"
      heartbeat-interval: "500"
    snapshot: false
  kube-api:
    pod_security_policy: false
  kubelet:
    fail_swap_on: false

ssh_agent_auth: false

addons: |-
  ---
  kind: Namespace
  apiVersion: v1
  metadata:
    name: cattle-system
  ---
  kind: ServiceAccount
  apiVersion: v1
  metadata:
    name: cattle-admin
    namespace: cattle-system
  ---
  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: cattle-crb
    namespace: cattle-system
  subjects:
  - kind: ServiceAccount
    name: cattle-admin
    namespace: cattle-system
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: cattle-keys-ingress
    namespace: cattle-system
  type: Opaque
  data:
    tls.crt: # INSERT TLS.CRT IN BASE64 ENCODING
    tls.key: # INSERT TLS.KEY IN BASE64 ENCODING
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: cattle-keys-server
    namespace: cattle-system
  type: Opaque
  data:
    cacerts.pem: # INSERT CA CERT IN BASE64 ENCODING
  ---
  apiVersion: v1
  kind: Service
  metadata:
    namespace: cattle-system
    name: cattle-service
    labels:
      app: cattle
  spec:
    ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
    selector:
      app: cattle
  ---
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    namespace: cattle-system
    name: cattle-ingress-http
    annotations:
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"   # Max time in seconds for ws to remain shell window open
      nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"   # Max time in seconds for ws to remain shell window open
  spec:
    rules:
    - host: my-rancher-cluster.com  # FQDN to access cattle server
      http:
        paths:
        - backend:
            serviceName: cattle-service
            servicePort: 80
    tls:
    - secretName: cattle-keys-ingress
      hosts:
      - my-rancher-cluster.com      # FQDN to access cattle server
  ---
  kind: Deployment
  apiVersion: extensions/v1beta1
  metadata:
    namespace: cattle-system
    name: cattle
  spec:
    replicas: 3
    template:
      metadata:
        labels:
          app: cattle
      spec:
        serviceAccountName: cattle-admin
        containers:
        # Rancher install via RKE addons is only supported up to v2.0.8
        - image: hostname-for-private-registry:5000/rancher/rancher:v2.0.8
          imagePullPolicy: Always
          name: cattle-server
          env:
  #       - name: HTTP_PROXY
  #         value: "http://your_proxy_address:port"
  #       - name: HTTPS_PROXY
  #         value: "http://your_proxy_address:port"
  #       - name: NO_PROXY
  #         value: "localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/16,your_network_ranges_that_dont_need_proxy_to_access"
          - name: CATTLE_SYSTEM_DEFAULT_REGISTRY
            value: "hostname-for-private-registry:5000"
          livenessProbe:
            httpGet:
              path: /ping
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 60
          readinessProbe:
            httpGet:
              path: /ping
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 10
          ports:
          - containerPort: 80
            protocol: TCP
          - containerPort: 443
            protocol: TCP
          volumeMounts:
          - mountPath: /etc/rancher/ssl
            name: cattle-keys-volume
            readOnly: true
        volumes:
        - name: cattle-keys-volume
          secret:
            defaultMode: 420
            secretName: cattle-keys-server

system_images:
  etcd: hostname-for-private-registry:5000/rancher/coreos-etcd:v3.2.18
  alpine: hostname-for-private-registry:5000/rancher/rke-tools:v0.1.13
  nginx_proxy: hostname-for-private-registry:5000/rancher/rke-tools:v0.1.13
  cert_downloader: hostname-for-private-registry:5000/rancher/rke-tools:v0.1.13
  kubernetes_services_sidecar: hostname-for-private-registry:5000/rancher/rke-tools:v0.1.13
  kubedns: hostname-for-private-registry:5000/rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: hostname-for-private-registry:5000/rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: hostname-for-private-registry:5000/rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: hostname-for-private-registry:5000/rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: hostname-for-private-registry:5000/rancher/hyperkube:v1.11.2-rancher1
  calico_node: hostname-for-private-registry:5000/rancher/calico-node:v3.1.1
  calico_cni: hostname-for-private-registry:5000/rancher/calico-cni:v3.1.1
  calico_controllers: ""
  calico_ctl: hostname-for-private-registry:5000/rancher/calico-ctl:v2.0.0
  pod_infra_container: hostname-for-private-registry:5000/rancher/pause-amd64:3.1
  ingress: hostname-for-private-registry:5000/rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: hostname-for-private-registry:5000/rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: hostname-for-private-registry:5000/rancher/metrics-server-amd64:v0.2.1

Log into Rancher UI and set server-url to the one that is auto-filled.
Wait for the cattle-agents to get deployed and check their logs --> their logs will show the errors I am trying to remove.
Try adding a new cluster with any other host and observe the cattle-agent fail to register to rancher due to similar errors shown from cattle-agents from local cluster.

Result:
cattle-cluster-agent log:
INFO: Environment: CATTLE_ADDRESS=XX.42.2.3 CATTLE_CA_CHECKSUM=<PROVIDED CA CHECKSUM> CATTLE_CLUSTER=true CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-6498dddf78-6vwlp CATTLE_SERVER=https://my-rancher-cluster.com CATTLE_SERVICE_PORT=tcp://XX.43.65.75:80 CATTLE_SERVICE_PORT_443_TCP=tcp://XX.43.65.75:443 CATTLE_SERVICE_PORT_443_TCP_ADDR=XX.43.65.75 CATTLE_SERVICE_PORT_443_TCP_PORT=443 CATTLE_SERVICE_PORT_443_TCP_PROTO=tcp CATTLE_SERVICE_PORT_80_TCP=tcp://XX.43.65.75:80 CATTLE_SERVICE_PORT_80_TCP_ADDR=XX.43.65.75 CATTLE_SERVICE_PORT_80_TCP_PORT=80 CATTLE_SERVICE_PORT_80_TCP_PROTO=tcp CATTLE_SERVICE_SERVICE_HOST=XX.43.65.75 CATTLE_SERVICE_SERVICE_PORT=80 CATTLE_SERVICE_SERVICE_PORT_HTTP=80 CATTLE_SERVICE_SERVICE_PORT_HTTPS=443
INFO: Using resolv.conf: nameserver XX.43.0.10 search cattle-system.svc.cluster.local svc.cluster.local cluster.local example01.private.internal.dns example02.private.internal.dns options ndots:5
INFO: https://my-rancher-cluster.com/ping is accessible
INFO: Value from https://my-rancher-cluster.com/v3/settings/cacerts is an x509 certificate
time="2019-03-20T15:17:54Z" level=info msg="Rancher agent version v2.0.8 is starting"
time="2019-03-20T15:17:54Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect/register with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:17:54Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect/register"
time="2019-03-20T15:17:54Z" level=info msg="Listening on /tmp/log.sock"
time="2019-03-20T15:17:54Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:17:54Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:04Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect/register with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:04Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect/register"
time="2019-03-20T15:18:04Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:04Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:14Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect/register with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:14Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect/register"
time="2019-03-20T15:18:14Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:14Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:24Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect/register with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:24Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect/register"
time="2019-03-20T15:18:24Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:24Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"

cattle-node-agents log:
INFO: Environment: CATTLE_ADDRESS=XX.XX.1.ZZ CATTLE_AGENT_CONNECT=true CATTLE_CA_CHECKSUM=<PROVIDED CA CHECKSUM> CATTLE_CLUSTER=false CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=ip-XX-XX-1-ZZ.us-west-1.compute.internal CATTLE_SERVER=https://my-rancher-cluster.com CATTLE_SERVICE_PORT=tcp://XX.43.65.75:80 CATTLE_SERVICE_PORT_443_TCP=tcp://XX.43.65.75:443 CATTLE_SERVICE_PORT_443_TCP_ADDR=XX.43.65.75 CATTLE_SERVICE_PORT_443_TCP_PORT=443 CATTLE_SERVICE_PORT_443_TCP_PROTO=tcp CATTLE_SERVICE_PORT_80_TCP=tcp://XX.43.65.75:80 CATTLE_SERVICE_PORT_80_TCP_ADDR=XX.43.65.75 CATTLE_SERVICE_PORT_80_TCP_PORT=80 CATTLE_SERVICE_PORT_80_TCP_PROTO=tcp CATTLE_SERVICE_SERVICE_HOST=XX.43.65.75 CATTLE_SERVICE_SERVICE_PORT=80 CATTLE_SERVICE_SERVICE_PORT_HTTP=80 CATTLE_SERVICE_SERVICE_PORT_HTTPS=443
INFO: Using resolv.conf: nameserver XX.XX.40.5 nameserver XX.XX.40.40 nameserver XX.XX.0.2 search example01.private.internal.dns example02.private.internal.dns
INFO: https://my-rancher-cluster.com/ping is accessible
INFO: Value from https://my-rancher-cluster.com/v3/settings/cacerts is an x509 certificate
time="2019-03-20T15:17:53Z" level=info msg="Rancher agent version v2.0.8 is starting"
time="2019-03-20T15:17:53Z" level=info msg="Option controlPlane=false"
time="2019-03-20T15:17:53Z" level=info msg="Option worker=false"
time="2019-03-20T15:17:53Z" level=info msg="Option requestedHostname=ip-XX-XX-1-ZZ.us-west-1.compute.internal"
time="2019-03-20T15:17:53Z" level=info msg="Option customConfig=map[label:map[] address:XX.XX.1.ZZ internalAddress: roles:[]]"
time="2019-03-20T15:17:53Z" level=info msg="Option etcd=false"
time="2019-03-20T15:17:53Z" level=info msg="Listening on /tmp/log.sock"
time="2019-03-20T15:17:53Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:17:53Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:17:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:17:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:03Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:03Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:13Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:13Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:23Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:23Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:23Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:23Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:33Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:33Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:33Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:33Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:43Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:43Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:43Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:43Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:53Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:18:53Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:18:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:18:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:03Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:03Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:13Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:13Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:23Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:23Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:23Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:23Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:33Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:33Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:33Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:33Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:43Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:43Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:43Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:43Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:53Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:19:53Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:19:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:19:53Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:20:03Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:20:03Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:20:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:20:03Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:20:13Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:20:13Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:20:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:20:13Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
time="2019-03-20T15:20:23Z" level=info msg="Connecting to wss://my-rancher-cluster.com/v3/connect with token mqz5pxj75t2ztbr8kw77m6mh6vjvckhclrv6c845bdlxmjxzk9ksrt"
time="2019-03-20T15:20:23Z" level=info msg="Connecting to proxy" url="wss://my-rancher-cluster.com/v3/connect"
time="2019-03-20T15:20:23Z" level=info msg="Starting plan monitor"

ingress-controllers log:
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.16.2-rancher1
  Build:      d36d4cc
  Repository: git@github.com:rancher/ingress-nginx.git
-------------------------------------------------------------------------------
nginx version: nginx/1.13.12
W0320 15:01:45.674210       5 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0320 15:01:45.674496       5 main.go:183] Creating API client for https://XX.43.0.1:443
I0320 15:01:45.698165       5 main.go:227] Running in Kubernetes cluster version v1.11 (v1.11.2) - git (clean) commit bb9ffb1654d4a729bb4cec18ff088eacc153c239 - platform linux/amd64
I0320 15:01:45.703223       5 main.go:100] Validated ingress-nginx/default-http-backend as the default backend.
I0320 15:01:46.036765       5 nginx.go:250] Starting NGINX Ingress controller
I0320 15:01:46.045631       5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"0bd931d1-4b21-11e9-a434-0227f1bf9692", APIVersion:"v1", ResourceVersion:"589", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0320 15:01:46.045674       5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"0bd6d578-4b21-11e9-a434-0227f1bf9692", APIVersion:"v1", ResourceVersion:"586", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0320 15:01:46.046725       5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"0bd43581-4b21-11e9-a434-0227f1bf9692", APIVersion:"v1", ResourceVersion:"584", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0320 15:01:46.047100       5 backend_ssl.go:69] adding secret cattle-system/cattle-keys-ingress to the local store
I0320 15:01:47.139926       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"cattle-ingress-http", UID:"0e79c53d-4b21-11e9-a7ab-0667cd301f52", APIVersion:"extensions", ResourceVersion:"676", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress cattle-system/cattle-ingress-http
I0320 15:01:47.237352       5 nginx.go:271] Starting NGINX process
I0320 15:01:47.240534       5 leaderelection.go:175] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
W0320 15:01:47.241421       5 controller.go:349] Service "ingress-nginx/default-http-backend" does not have any active Endpoint
W0320 15:01:47.241457       5 controller.go:769] Service "cattle-system/cattle-service" does not have any active Endpoint.
I0320 15:01:47.241510       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:01:47.257987       5 leaderelection.go:184] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0320 15:01:47.258493       5 status.go:197] new leader elected: nginx-ingress-controller-ftv4x
I0320 15:01:47.446474       5 controller.go:179] Backend successfully reloaded.
I0320 15:01:47.474399       5 backend_ssl.go:181] updating local copy of ssl certificate cattle-system/cattle-keys-ingress with missing intermediate CA certs
XX.XX.2.AA - [XX.XX.2.AA] - - [20/Mar/2019:15:01:48 +0000] "GET /v3/connect HTTP/1.1" 503 214 "-" "-" 4839 0.000 [] - - - - 5c6eafce35000df5d55e07f76df8801a
XX.XX.2.AA - [XX.XX.2.AA] - - [20/Mar/2019:15:01:50 +0000] "GET /v3/connect HTTP/1.1" 503 214 "-" "-" 4811 0.000 [] - - - - d5f2f67b27233e3ada66e4720cfbdda7
W0320 15:01:50.574902       5 controller.go:349] Service "ingress-nginx/default-http-backend" does not have any active Endpoint
W0320 15:01:50.575399       5 controller.go:769] Service "cattle-system/cattle-service" does not have any active Endpoint.
I0320 15:01:50.575494       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:01:50.667390       5 controller.go:179] Backend successfully reloaded.

I0320 15:02:13.401600       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:02:13.542864       5 controller.go:179] Backend successfully reloaded.
I0320 15:02:16.535376       5 backend_ssl.go:61] updating secret cattle-system/cattle-keys-ingress in the local store
I0320 15:02:16.535394       5 store.go:375] secret cattle-system/cattle-keys-ingress was updated and it is used in ingress annotations. Parsing...
I0320 15:02:16.566053       5 store.go:375] secret cattle-system/cattle-keys-ingress was updated and it is used in ingress annotations. Parsing...
I0320 15:02:16.581691       5 store.go:375] secret cattle-system/cattle-keys-ingress was updated and it is used in ingress annotations. Parsing...
W0320 15:02:16.734789       5 controller.go:769] Service "cattle-system/cattle-service" does not have any active Endpoint.
I0320 15:02:16.734865       5 controller.go:169] Configuration changes detected, backend reload required.

W0320 15:02:20.068161       5 controller.go:769] Service "cattle-system/cattle-service" does not have any active Endpoint.
I0320 15:02:36.783209       5 controller.go:169] Configuration changes detected, backend reload required.

I0320 15:02:40.116754       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:02:40.209497       5 controller.go:179] Backend successfully reloaded.
I0320 15:02:43.450053       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:02:43.538979       5 controller.go:179] Backend successfully reloaded.
I0320 15:02:47.286710       5 status.go:362] updating Ingress cattle-system/cattle-ingress-http status to [{ ip-XX-XX-1-XX.us-west-1.compute.internal} { ip-XX-XX-1-ZZ.us-west-1.compute.internal} { ip-XX-XX-2-YY.us-west-1.compute.internal}]
I0320 15:02:47.294058       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"cattle-ingress-http", UID:"0e79c53d-4b21-11e9-a7ab-0667cd301f52", APIVersion:"extensions", ResourceVersion:"3260", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/cattle-ingress-http
I0320 15:02:47.305233       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cattle-system", Name:"cattle-ingress-http", UID:"0e79c53d-4b21-11e9-a7ab-0667cd301f52", APIVersion:"extensions", ResourceVersion:"3263", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress cattle-system/cattle-ingress-http
I0320 15:02:47.475125       5 backend_ssl.go:181] updating local copy of ssl certificate cattle-system/cattle-keys-ingress with missing intermediate CA certs
I0320 15:02:50.627501       5 controller.go:169] Configuration changes detected, backend reload required.
I0320 15:02:50.714519       5 controller.go:179] Backend successfully reloaded.

I0320 15:02:13.313986       5 leaderelection.go:184] successfully acquired lease kube-system/cattle-controllers
2019/03/20 15:02:13 [INFO] Starting catalog controller
2019/03/20 15:02:13 [INFO] Starting management controllers
W0320 15:02:13.524804       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.GlobalRoleBinding ended with: too old resource version: 1055 (1059)
W0320 15:02:13.544784       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.GlobalRole ended with: too old resource version: 1056 (1059)
2019/03/20 15:02:14 [INFO] Reconciling GlobalRoles
2019/03/20 15:02:14 [INFO] Creating user-base
2019/03/20 15:02:14 [INFO] Creating clusters-create
2019/03/20 15:02:14 [INFO] Creating catalogs-manage
2019/03/20 15:02:14 [INFO] Creating roles-manage
2019/03/20 15:02:14 [INFO] Creating authn-manage
2019/03/20 15:02:14 [INFO] Creating settings-manage
2019/03/20 15:02:14 [INFO] Creating admin
2019/03/20 15:02:14 [INFO] Creating user
2019/03/20 15:02:14 [INFO] Creating nodedrivers-manage
2019/03/20 15:02:14 [INFO] Creating catalogs-use
2019/03/20 15:02:14 [INFO] Creating users-manage
2019/03/20 15:02:14 [INFO] Creating podsecuritypolicytemplates-manage
2019/03/20 15:02:14 [INFO] Creating resourcequotatemplates-manage
2019/03/20 15:02:14 [INFO] Reconciling RoleTemplates
2019/03/20 15:02:14 [INFO] Creating projectroletemplatebindings-manage
2019/03/20 15:02:14 [INFO] Creating edit
2019/03/20 15:02:14 [INFO] Creating cluster-owner
2019/03/20 15:02:14 [INFO] Creating configmaps-view
2019/03/20 15:02:14 [INFO] Creating serviceaccounts-view
2019/03/20 15:02:14 [INFO] Creating projectroletemplatebindings-view
2019/03/20 15:02:14 [INFO] Creating cluster-admin
2019/03/20 15:02:14 [INFO] Creating secrets-manage
2019/03/20 15:02:14 [INFO] Creating persistentvolumeclaims-manage
2019/03/20 15:02:14 [INFO] Creating serviceaccounts-manage
2019/03/20 15:02:14 [INFO] Creating workloads-manage
2019/03/20 15:02:14 [INFO] Creating secrets-view
2019/03/20 15:02:14 [INFO] Creating configmaps-manage
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-catalogs-use for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-nodedrivers-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-settings-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-user for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-resourcequotatemplates-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-user-base for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-users-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] Creating cluster-member
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-clusters-create for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-authn-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-catalogs-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-podsecuritypolicytemplates-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-admin for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-roles-manage for corresponding GlobalRole
2019/03/20 15:02:14 [INFO] Creating projects-create
2019/03/20 15:02:14 [INFO] Creating storage-manage
2019/03/20 15:02:14 [INFO] Creating clusterroletemplatebindings-manage
2019/03/20 15:02:14 [INFO] Creating project-owner
2019/03/20 15:02:14 [INFO] Creating project-member
2019/03/20 15:02:14 [INFO] Creating create-ns
2019/03/20 15:02:14 [INFO] Creating services-view
2019/03/20 15:02:14 [INFO] Creating persistentvolumeclaims-view
2019/03/20 15:02:14 [INFO] Creating view
2019/03/20 15:02:14 [INFO] Creating nodes-view
2019/03/20 15:02:14 [INFO] Creating clusterroletemplatebindings-view
2019/03/20 15:02:14 [INFO] Creating read-only
2019/03/20 15:02:14 [INFO] Creating services-manage
2019/03/20 15:02:14 [INFO] Creating admin
2019/03/20 15:02:14 [INFO] Creating nodes-manage
2019/03/20 15:02:14 [INFO] Creating workloads-view
2019/03/20 15:02:14 [INFO] Creating ingress-manage
2019/03/20 15:02:14 [INFO] Creating projects-view
2019/03/20 15:02:14 [INFO] Creating ingress-view
2019/03/20 15:02:14 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding globalrolebinding-7z5p9
2019/03/20 15:02:14 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding globalrolebinding-7z5p9 for user user-j8fq7 with role cattle-globalrole-admin
2019/03/20 15:02:14 [INFO] [mgmt-cluster-rbac-delete] Creating namespace local
2019/03/20 15:02:14 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster local
2019/03/20 15:02:14 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster local
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Creating namespace p-bx2c2
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Creating namespace p-wlhj8
2019/03/20 15:02:14 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-j8fq7 for project p-bx2c2
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Creating creator clusterRoleTemplateBinding for user user-j8fq7 for cluster local
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-j8fq7 for project p-wlhj8
2019/03/20 15:02:14 [INFO] [mgmt-auth-crtb-controller] Creating clusterRole local-clusterowner
2019/03/20 15:02:14 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-bx2c2
2019/03/20 15:02:14 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-wlhj8
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Updating project p-wlhj8
2019/03/20 15:02:14 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-bx2c2-projectowner
2019/03/20 15:02:14 [INFO] [mgmt-project-rbac-create] Updating project p-bx2c2
2019/03/20 15:02:14 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on cluster
2019/03/20 15:02:14 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2019/03/20 15:02:14 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole p-wlhj8-projectowner
2019/03/20 15:02:15 [INFO] [mgmt-project-rbac-create] Updating project p-bx2c2
2019/03/20 15:02:15 [INFO] [mgmt-project-rbac-create] Updating project p-wlhj8
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster local for subject user-j8fq7
2019/03/20 15:02:15 [INFO] [mgmt-project-rbac-create] Updating project p-bx2c2
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace local
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-wlhj8 for subject user-j8fq7
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole local-clustermember
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-bx2c2 for subject user-j8fq7
2019/03/20 15:02:15 [INFO] Starting cluster agent forlocal
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating clusterRole local-clustermember
2019/03/20 15:02:15 [INFO] Creating node driver amazonec2
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-j8fq7 with role cluster-owner in namespace
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-bx2c2
2019/03/20 15:02:15 [INFO] Creating node driver azure
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster local for subject user-j8fq7
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace local
2019/03/20 15:02:15 [INFO] Creating node driver digitalocean
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-j8fq7 with role cluster-owner in namespace
2019/03/20 15:02:15 [INFO] Creating node driver exoscale
2019/03/20 15:02:15 [ERROR] ProjectRoleTemplateBindingController p-bx2c2/creator-project-owner [mgmt-auth-prtb-controller] failed with : clusterroles.rbac.authorization.k8s.io "local-clustermember" already exists
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding clusterrolebinding-k2wm8 for cluster membership in cluster local for subject user-j8fq7
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-wlhj8
2019/03/20 15:02:15 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2019/03/20 15:02:15 [INFO] Creating node driver openstack
2019/03/20 15:02:15 [INFO] Registering project network policy
2019/03/20 15:02:15 [INFO] Registering namespaceHandler for adding labels
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace local
2019/03/20 15:02:15 [INFO] Creating node driver otc
2019/03/20 15:02:15 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-j8fq7 with role cluster-owner in namespace
2019/03/20 15:02:15 [INFO] registering podsecuritypolicy cluster handler for cluster local
2019/03/20 15:02:15 [INFO] registering podsecuritypolicy project handler for cluster local
2019/03/20 15:02:15 [INFO] registering podsecuritypolicy namespace handler for cluster local
2019/03/20 15:02:15 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2019/03/20 15:02:15 [INFO] registering podsecuritypolicy template handler for cluster local
2019/03/20 15:02:15 [INFO] Creating node driver packet
2019/03/20 15:02:15 [INFO] Creating node driver rackspace
2019/03/20 15:02:15 [INFO] Starting cluster controllers for local
2019/03/20 15:02:15 [INFO] Creating node driver softlayer
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role project-owner in namespace
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-wlhj8
2019/03/20 15:02:15 [INFO] Creating node driver aliyunecs
2019/03/20 15:02:15 [ERROR] ProjectRoleTemplateBindingController p-bx2c2/creator-project-owner [mgmt-auth-prtb-controller] failed with : couldn't create role project-owner: roles.rbac.authorization.k8s.io "project-owner" already exists
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role project-owner in namespace
E0320 15:02:15.508117       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field Project.status of type v3.ProjectStatus
W0320 15:02:15.508143       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.Project ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-bx2c2
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-wlhj8
E0320 15:02:15.537315       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field PipelineExecution.status of type v3.PipelineExecutionStatus
W0320 15:02:15.537338       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.PipelineExecution ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
E0320 15:02:15.537825       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field ClusterLogging.status of type v3.ClusterLoggingStatus
W0320 15:02:15.537885       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.ClusterLogging ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
W0320 15:02:15.538285       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.RoleTemplate ended with: too old resource version: 1161 (1274)
E0320 15:02:15.539249       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field App.status of type v3.AppStatus
W0320 15:02:15.539268       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.App ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
2019/03/20 15:02:15 [INFO] Creating node driver vmwarevsphere
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-bx2c2
2019/03/20 15:02:15 [INFO] Rancher startup complete
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role project-owner in namespace
E0320 15:02:15.573035       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field SourceCodeRepository.status of type v3.SourceCodeRepositoryStatus
W0320 15:02:15.573059       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.SourceCodeRepository ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role admin in namespace
E0320 15:02:15.597360       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field ProjectNetworkPolicy.status of type v3.ProjectNetworkPolicyStatus
W0320 15:02:15.597384       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.ProjectNetworkPolicy ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role project-owner in namespace
2019/03/20 15:02:15 [INFO] Creating clusterRole for roleTemplate Project Owner (project-owner).
2019/03/20 15:02:15 [INFO] Creating clusterRole for roleTemplate Cluster Owner (cluster-owner).
2019/03/20 15:02:15 [INFO] Creating clusterRole project-owner-promoted for project access to global resource.
2019/03/20 15:02:15 [INFO] Creating clusterRole project-owner-promoted for project access to global resource.
2019/03/20 15:02:15 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-j8fq7 with role admin in namespace
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role cluster-owner
2019/03/20 15:02:15 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-bx2c2 to namespace=default
2019/03/20 15:02:15 [INFO] Updating clusterRole project-owner-promoted for project access to global resource.
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role cluster-owner
2019/03/20 15:02:15 [ERROR] ProjectRoleTemplateBindingController p-bx2c2/creator-project-owner [cluster-prtb-sync] failed with : couldn't create role project-owner-promoted: clusterroles.rbac.authorization.k8s.io "project-owner-promoted" already exists
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role project-owner
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role project-owner
2019/03/20 15:02:15 [INFO] Deleting roleBinding clusterrolebinding-8b2dv
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role project-owner
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role admin
2019/03/20 15:02:15 [INFO] Creating clusterRole for roleTemplate Create Namespaces (create-ns).
2019/03/20 15:02:15 [INFO] Creating roleBinding User user-j8fq7 Role admin
2019/03/20 15:02:16 [INFO] Created machine for node [ip-XX-XX-1-XX.us-west-1.compute.internal]
2019/03/20 15:02:16 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-j8fq7 role p-wlhj8-namespaces-edit.
2019/03/20 15:02:16 [INFO] Creating roleBinding User user-j8fq7 Role admin
2019/03/20 15:02:16 [INFO] Creating user for principal system://local
2019/03/20 15:02:16 [INFO] uploading amazonec2Config to node schema
2019/03/20 15:02:16 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wlhj8 to namespace=ingress-nginx
2019/03/20 15:02:16 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-j8fq7 role p-bx2c2-namespaces-edit.
2019/03/20 15:02:16 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-j8fq7 role project-owner-promoted.
2019/03/20 15:02:16 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-j8fq7 role project-owner-promoted.
2019/03/20 15:02:16 [INFO] Created machine for node [ip-XX-XX-2-YY.us-west-1.compute.internal]
2019/03/20 15:02:16 [INFO] uploading amazonec2Config to node schema
2019/03/20 15:02:16 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-j8fq7 role create-ns.
2019/03/20 15:02:16 [INFO] uploading azureConfig to node schema
2019/03/20 15:02:16 [INFO] Creating roleBinding User user-j8fq7 Role admin
2019/03/20 15:02:16 [INFO] Updating clusterRoleBinding clusterrolebinding-dtbtx for project access to global resource for subject user-j8fq7 role project-owner-promoted.
2019/03/20 15:02:16 [ERROR] error updating ns p-bx2c2 status: Operation cannot be fulfilled on namespaces "p-bx2c2": the object has been modified; please apply your changes to the latest version and try again
2019/03/20 15:02:16 [INFO] uploading azureConfig to node schema
2019/03/20 15:02:16 [INFO] Created machine for node [ip-XX-XX-1-ZZ.us-west-1.compute.internal]
2019/03/20 15:02:16 [INFO] Creating roleBinding User user-j8fq7 Role project-owner
2019/03/20 15:02:16 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wlhj8 to namespace=kube-public
2019/03/20 15:02:16 [INFO] Creating globalRoleBindings for u-b4qkhsnliz
2019/03/20 15:02:16 [INFO] uploading vmwarevsphereConfig to node schema
2019/03/20 15:02:16 [INFO] uploading vmwarevsphereConfig to node schema
2019/03/20 15:02:16 [ERROR] UserController u-b4qkhsnliz [mgmt-auth-users-controller] failed with : clusterrolebindings.rbac.authorization.k8s.io "grb-u-b4qkhsnliz-view" already exists
2019/03/20 15:02:16 [INFO] uploading digitaloceanConfig to node schema
2019/03/20 15:02:16 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-tsjxj
2019/03/20 15:02:16 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-tsjxj for user u-b4qkhsnliz with role cattle-globalrole-user
2019/03/20 15:02:16 [INFO] uploading digitaloceanConfig to node schema
2019/03/20 15:02:16 [INFO] Creating token for user u-b4qkhsnliz
2019/03/20 15:02:16 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster local for subject u-b4qkhsnliz
2019/03/20 15:02:16 [INFO] Creating roleBinding User u-b4qkhsnliz Role cluster-owner
2019/03/20 15:02:16 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace
2019/03/20 15:02:16 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace
2019/03/20 15:02:16 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wlhj8 to namespace=kube-system
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [INFO] Creating roleBinding User user-j8fq7 Role project-owner
2019/03/20 15:02:16 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wlhj8 to namespace=cattle-system
2019/03/20 15:02:16 [INFO] Creating roleBinding User user-j8fq7 Role admin
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [INFO] Updating clusterRoleBinding clusterrolebinding-rfpq7 for project access to global resource for subject user-j8fq7 role project-owner-promoted.
2019/03/20 15:02:16 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:16 [INFO] Updating clusterRoleBinding clusterrolebinding-52xk2 for project access to global resource for subject user-j8fq7 role create-ns.
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:20 [ERROR] CatalogController library [catalog] failed with : Get https://git.rancher.io/charts/index.yaml: dial tcp: lookup git.rancher.io on XX.43.0.10:53: server misbehaving
2019/03/20 15:02:21 [INFO] Updating catalog library
2019/03/20 15:02:30 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:02:45 [INFO] Catalog sync done. 30 templates created, 0 templates updated, 0 templates deleted
2019/03/20 15:02:45 [ERROR] CatalogController library [catalog] failed with : failed to sync templates. Resetting commit. Multiple error occurred: [Get file://../logo.png: unsupported protocol scheme "file"]
2019/03/20 15:02:45 [INFO] Updating catalog library
2019/03/20 15:02:45 [INFO] Catalog sync done. 0 templates created, 0 templates updated, 0 templates deleted
2019/03/20 15:02:47 [INFO] Updating ingress [cattle-system:cattle-ingress-http] with public endpoints [[{"addresses":["","",""],"port":443,"protocol":"HTTPS","serviceName":"cattle-system:cattle-service","ingressName":"cattle-system:cattle-ingress-http","hostname":"my-rancher-cluster.com","allNodes":false}]]
2019/03/20 15:02:47 [INFO] Updating workload [cattle-system/cattle] with public endpoints [[{"addresses":["","",""],"port":443,"protocol":"HTTPS","serviceName":"cattle-system:cattle-service","ingressName":"cattle-system:cattle-ingress-http","hostname":"my-rancher-cluster.com","allNodes":false}]]
2019/03/20 15:02:48 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:04:56 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:09:12 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:16:20 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:16:20 [ERROR] ClusterController local [cluster-deploy] failed with : waiting for server-url setting to be set
2019/03/20 15:17:27 [ERROR] error updating ns user-j8fq7 status: Operation cannot be fulfilled on namespaces "user-j8fq7": the object has been modified; please apply your changes to the latest version and try again
2019/03/20 15:17:28 [INFO] Creating token for user user-j8fq7
time="2019-03-20 15:17:28" level=info msg="Telemetry Client v0.5.1"
time="2019-03-20 15:17:28" level=info msg="Listening on <IP_ADDRESS>:8114"
2019/03/20 15:19:14 [INFO] Stopping cluster agent forlocal
2019/03/20 15:19:14 [INFO] Shutting down CronJobController controller
2019/03/20 15:19:14 [INFO] Shutting down JobController controller
2019/03/20 15:19:14 [INFO] Shutting down IngressController controller
2019/03/20 15:19:14 [INFO] Shutting down PodSecurityPolicyController controller
2019/03/20 15:19:14 [INFO] Shutting down RoleController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterRoleController controller
2019/03/20 15:19:14 [INFO] Shutting down RoleBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down ServiceController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterRoleBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down EndpointsController controller
2019/03/20 15:19:14 [INFO] Shutting down ServiceAccountController controller
2019/03/20 15:19:14 [INFO] Shutting down SecretController controller
2019/03/20 15:19:14 [INFO] Shutting down ResourceQuotaController controller
2019/03/20 15:19:14 [INFO] Shutting down StatefulSetController controller
2019/03/20 15:19:14 [INFO] Shutting down DaemonSetController controller
2019/03/20 15:19:14 [INFO] Shutting down ReplicaSetController controller
2019/03/20 15:19:14 [INFO] Shutting down EventController controller
2019/03/20 15:19:14 [INFO] Shutting down DeploymentController controller
2019/03/20 15:19:14 [INFO] Shutting down ReplicationControllerController controller
2019/03/20 15:19:14 [INFO] Shutting down NamespaceController controller
2019/03/20 15:19:14 [INFO] Shutting down NamespaceController controller
2019/03/20 15:19:14 [INFO] Shutting down SecretController controller
2019/03/20 15:19:14 [INFO] Shutting down NodeController controller
2019/03/20 15:19:14 [INFO] Shutting down AppController controller
2019/03/20 15:19:14 [INFO] Shutting down PodSecurityPolicyTemplateProjectBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down PodSecurityPolicyTemplateController controller
2019/03/20 15:19:14 [INFO] Shutting down PodController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterPipelineController controller
2019/03/20 15:19:14 [INFO] Shutting down SourceCodeRepositoryController controller
2019/03/20 15:19:14 [INFO] Handling backend connection request [local]
2019/03/20 15:19:14 [INFO] Shutting down SourceCodeCredentialController controller
2019/03/20 15:19:14 [INFO] Shutting down PipelineExecutionLogController controller
2019/03/20 15:19:14 [INFO] Shutting down PipelineExecutionController controller
2019/03/20 15:19:14 [INFO] Shutting down PipelineController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectNetworkPolicyController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterLoggingController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectLoggingController controller
2019/03/20 15:19:14 [INFO] Shutting down NodeController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterLoggingController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectController controller
2019/03/20 15:19:14 [INFO] Shutting down ResourceQuotaTemplateController controller
2019/03/20 15:19:14 [INFO] Shutting down GlobalRoleBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down RoleTemplateController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterRoleTemplateBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectRoleTemplateBindingController controller
2019/03/20 15:19:14 [INFO] Shutting down NodeController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterController controller
2019/03/20 15:19:14 [INFO] Shutting down NotifierController controller
2019/03/20 15:19:14 [INFO] Shutting down NotifierController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectAlertController controller
2019/03/20 15:19:14 [INFO] Shutting down ClusterAlertController controller
2019/03/20 15:19:14 [INFO] Shutting down ProjectController controller
2019/03/20 15:19:14 [INFO] Shutting down NetworkPolicyController controller
2019/03/20 15:19:14 [INFO] Starting cluster agent forlocal
2019/03/20 15:19:14 [INFO] Registering project network policy
2019/03/20 15:19:14 [INFO] Registering namespaceHandler for adding labels
2019/03/20 15:19:14 [INFO] registering podsecuritypolicy cluster handler for cluster local
2019/03/20 15:19:14 [INFO] registering podsecuritypolicy project handler for cluster local
2019/03/20 15:19:14 [INFO] registering podsecuritypolicy namespace handler for cluster local
2019/03/20 15:19:14 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2019/03/20 15:19:14 [INFO] registering podsecuritypolicy template handler for cluster local
2019/03/20 15:19:14 [INFO] Starting cluster controllers for local
E0320 15:19:14.725053       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field App.status of type v3.AppStatus
W0320 15:19:14.725080       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.App ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
W0320 15:19:14.726401       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.PodSecurityPolicyTemplateProjectBinding ended with: too old resource version: 1059 (1262)
E0320 15:19:14.761770       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field SourceCodeCredential.status of type v3.SourceCodeCredentialStatus
W0320 15:19:14.761793       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.SourceCodeCredential ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
E0320 15:19:14.763769       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field Notifier.status of type v3.NotifierStatus
W0320 15:19:14.763790       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.Notifier ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
E0320 15:19:14.776583       5 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: json: cannot unmarshal string into Go struct field ProjectLogging.status of type v3.ProjectLoggingStatus
W0320 15:19:14.776606       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.ProjectLogging ended with: very short watch: github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: Unexpected watch close - watch lasted less than a second and no items received
W0320 15:19:14.778642       5 reflector.go:341] github.com/rancher/rancher/vendor/github.com/rancher/norman/controller/generic_controller.go:139: watch of *v3.ResourceQuotaTemplate ended with: too old resource version: 1268 (6409)
2019/03/20 15:19:14 [INFO] Updating workload [ingress-nginx/nginx-ingress-controller] with public endpoints [[{"nodeName":"local:machine-2r7ll","addresses":["ip-XX-XX-1-XX.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ssfrt","allNodes":false},{"nodeName":"local:machine-2r7ll","addresses":["ip-XX-XX-1-XX.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ssfrt","allNodes":false},{"nodeName":"local:machine-kwjw9","addresses":["ip-XX-XX-1-ZZ.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-tfpbg","allNodes":false},{"nodeName":"local:machine-kwjw9","addresses":["ip-XX-XX-1-ZZ.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-tfpbg","allNodes":false},{"nodeName":"local:machine-k6dgz","addresses":["ip-XX-XX-2-YY.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ftv4x","allNodes":false},{"nodeName":"local:machine-k6dgz","addresses":["ip-XX-XX-2-YY.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ftv4x","allNodes":false}]]
2019/03/20 15:19:14 [INFO] Updating pod [ingress-nginx/nginx-ingress-controller-ssfrt] with public endpoints [[{"nodeName":"local:machine-2r7ll","addresses":["ip-XX-XX-1-XX.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ssfrt","allNodes":false},{"nodeName":"local:machine-2r7ll","addresses":["ip-XX-XX-1-XX.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ssfrt","allNodes":false}]]
2019/03/20 15:19:14 [INFO] Updating pod [ingress-nginx/nginx-ingress-controller-tfpbg] with public endpoints [[{"nodeName":"local:machine-kwjw9","addresses":["ip-XX-XX-1-ZZ.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-tfpbg","allNodes":false},{"nodeName":"local:machine-kwjw9","addresses":["ip-XX-XX-1-ZZ.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-tfpbg","allNodes":false}]]
2019/03/20 15:19:14 [INFO] Updating pod [ingress-nginx/nginx-ingress-controller-ftv4x] with public endpoints [[{"nodeName":"local:machine-k6dgz","addresses":["ip-XX-XX-2-YY.us-west-1.compute.internal"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ftv4x","allNodes":false},{"nodeName":"local:machine-k6dgz","addresses":["ip-XX-XX-2-YY.us-west-1.compute.internal"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-ftv4x","allNodes":false}]]
2019/03/20 15:20:23 [INFO] Handling backend connection request [machine-kwjw9]

Other details that may be helpful:
I noticed that my 3 rancher containers get deployed onto the same host when the cluster initializes. I am forced to manually delete 2 of them so that 2 more rancher servers are deployed on the other 2 hosts.
I am terminating TLS on the ingress-controllers using the "cattle-keys-ingress" created in the cattle-system namespace. This seems to be working fine as I can easily curl https://my-rancher-cluster.com from inside the cattle-agents and retrieve the decrypted payload.
However, when I curl https://my-rancher-cluster.com/v3/connect from inside the cattle-agents (or outside), I receive a 503 error that returns "tunnel server not active". I am pretty sure this is due to curl not having websocket capability but figure it worth mentioning.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):

v2.0.8

Installation option (single install/HA):

HA AirGap

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported):

Import (initializing local rancher cluster)

Machine type (cloud/VM/metal) and specifications (CPU/memory):

t2.large AWS ec2-instance

Kubernetes version (use kubectl version):

v1.11.2

Docker version (use docker version):

1.12.6

After some discussion with @superseb, i believe the root cause of my issue is that I am trying to deploy replicas of the rancher container to manage the same k8s clusters?
I think that this feature is only available with v2.1.
@superseb Can you verify that this is true?

Yes, you can't run more than 1 replica using a Rancher version before v2.1.0 and RKE add-on install is only supported up to v2.0.8. The RKE templates for v2.0.8 have replicas set to 1. The Helm chart install (which is the official and supported install method from v2.1.0 and up) will run 3 replicas by default.

@superseb Thank you so much for the help!

@superseb One thing I would add is that I don't see any documentation on that lacking capability for Rancher v2.0. I was only able to confirm what you said by going to the rancher v2.1 release page and seeing the improved features. I think it would be very helpful to include this information in the install docs as I would assume many individuals are going through the HA installation assuming the deployment of multiple rancher containers.

We have similar issue with the cluster-agent pod, Rancher version is 2.2.4 and we run 3 replicas of rancher with helm chart (RKE 0.2.4).
The logs of the nginx-ingress pods return 500 alternately (most of the time it's ok and return 200).
We have some issues with the Rancher monitoring which it doesn't create the new CRD, so we checked the cluster-agent pod and it seems pretty much the same.
Thanks!

We are facing the same issue with the cluster-agent pod, Rancher version is 2.6.2 and we ran 1 replicas of rancher.  Kindly help us to investigate further.
Details:

Created local RKE cluster with the version 1.3.2.
Installed cert-manager of the version 1.6.1
Installed rancher of the version 2.6.2.

Rancher Logs:
2021/12/15 17:53:37 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("rpc error: code = Unknown desc = container not running (0081239ae2f66fe7bde77f3716551108ffa071142c7b9cd351953a3bcff0bf64)") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:37 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:37 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:37 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:37 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:38 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:38 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:38 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2021/12/15 17:53:38 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2021/12/15 17:53:38 [ERROR] error syncing 'cattle-fleet-system/helm-operation-5lp2n': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-5lp2n), requeuing
2021/12/15 17:53:39 [INFO] kontainerdriver googlekubernetesengine listening on address <IP_ADDRESS>:44004
2021/12/15 17:53:39 [INFO] kontainerdriver amazonelasticcontainerservice listening on address <IP_ADDRESS>:42652
2021/12/15 17:53:39 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2021/12/15 17:53:39 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2021/12/15 17:53:39 [INFO] kontainerdriver googlekubernetesengine stopped
2021/12/15 17:53:39 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2021/12/15 17:53:39 [INFO] kontainerdriver azurekubernetesservice listening on address <IP_ADDRESS>:39393
2021/12/15 17:53:39 [INFO] kontainerdriver azurekubernetesservice stopped
2021/12/15 17:53:39 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2021/12/15 17:53:44 [WARNING] failed to refresh data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json: driverMetadata err Get "https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json": net/http: TLS handshake timeout, Fallback to refresh from local file path /var/lib/rancher-data/driver-metadata/data.json
2021/12/15 17:53:44 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:53:45 [ERROR] error syncing 'cattle-system/helm-operation-kkvhf': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-kkvhf), requeuing
2021/12/15 17:56:04 [INFO] Creating token for user user-slbv4
time="2021-12-15 17:56:04" level=info msg="Telemetry Client v0.5.16"
time="2021-12-15 17:56:04" level=info msg="Listening on <IP_ADDRESS>:8114"
time="2021-12-15 17:56:04" level=info msg="Collecting anonymous data from https://localhost:443/v3"
2021/12/15 17:56:04 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2021-12-15 17:52:33 +0000 UTC notAfter=2022-12-15 17:56:04 +0000 UTC
2021/12/15 17:56:04 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2021-12-15 17:52:33 +0000 UTC notAfter=2022-12-15 17:56:04 +0000 UTC
2021/12/15 17:56:04 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2021-12-15 17:52:33 +0000 UTC notAfter=2022-12-15 17:56:04 +0000 UTC
2021/12/15 17:56:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-rdpg5 listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-dcalca2118.amat.com:dcalca2118.amat.com listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=7920DC6A1D7D95C52875F491F22AB269F54182A4]
2021/12/15 17:56:04 [INFO] Active TLS secret serving-cert (ver=5465) (count 8): map[field.cattle.io/projectId:local:p-rdpg5 listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-dcalca2118.amat.com:dcalca2118.amat.com listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=7920DC6A1D7D95C52875F491F22AB269F54182A4]
time="2021-12-15 17:56:04" level=error msg="Error collecting data: Get "https://localhost:443/v3\": EOF"
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("rpc error: code = Unknown desc = container not running (f754f946ea725df05fbf84f850fb2c6b8364fb1194a73bfb3bc644769ed28c00)") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:06 [ERROR] error syncing 'cattle-fleet-system/helm-operation-thc87': handler helm-operation: an error on the server ("container not found ("proxy")") has prevented the request from succeeding (get pods helm-operation-thc87), requeuing
2021/12/15 17:56:34 [INFO] Calling Duplicate CRB and RB cleanup
2021/12/15 17:56:34 [INFO] Starting bindings cleanup
2021/12/15 17:56:34 [INFO] binding with deterministic name not found, will delete all except the oldest binding
2021/12/15 17:56:34 [INFO] binding with deterministic name not found, will delete all except the oldest binding
2021/12/15 17:56:34 [INFO] Total PRTB duplicate clusterRoleBindings 0, roleBindings 0
2021/12/15 17:56:34 [INFO] binding with deterministic name not found, will delete all except the oldest binding
2021/12/15 17:56:34 [INFO] binding with deterministic name not found, will delete all except the oldest binding
2021/12/15 17:56:34 [INFO] Total CRTB duplicate clusterRoleBindings 0, roleBindings 0
2021/12/15 17:56:34 [INFO] Done Duplicate CRB and RB cleanup
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Creating namespace c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Creating namespace p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-slbv4 for project p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Creating namespace p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-slbv4 for project p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-tks9p-projectowner
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Creating creator clusterRoleTemplateBinding for user user-slbv4 for cluster c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-tks9p for subject user-slbv4
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Updating project p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole c-9s84q-clustermember
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating role/clusterRole c-9s84q-clusterowner
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Updating project p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-9s84q for subject user-slbv4
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on cluster
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-9s84q for subject user-slbv4
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Setting InitialRolesPopulated condition on project p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-9s84q
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Updating project p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role project-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-gshbx-projectowner
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-slbv4 with role cluster-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-project-rbac-create] Updating project p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-gshbx for subject user-slbv4
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-tks9p
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-slbv4 with role cluster-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding crb-cmdrdlfhk3 for cluster membership in cluster c-9s84q for subject user-slbv4
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role project-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role project-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role admin in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-slbv4 with role cluster-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-gshbx
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role admin in namespace
2021/12/15 17:56:47 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-slbv4 with role project-owner in namespace
2021/12/15 17:56:47 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-9s84q
2021/12/15 17:57:25 [ERROR] Failed to handling tunnel request from :33306: response 400: Operation cannot be fulfilled on nodes.management.cattle.io "m-2c4d8fa7db48": the object has been modified; please apply your changes to the latest version and try again
2021/12/15 17:57:35 [INFO] Handling backend connection request [c-9s84q:m-2c4d8fa7db48]
W1215 18:27:01.775102      33 transport.go:260] Unable to cancel request for *client.addQuery
W1215 18:27:04.333464      33 transport.go:260] Unable to cancel request for *client.addQuery
W1215 18:27:04.333719      33 transport.go:260] Unable to cancel request for *client.addQuery
Agent Logs:
NFO: Arguments: --server https://proxyIP:port --token REDACTED --ca-checksum 01f5558ef8cb6e48604444ff3a960e4a7e7d86f808bc8036687224a756bc258f --etcd --controlplane
INFO: Environment: CATTLE_ADDRESS=<CATTLE_IP> CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME= CATTLE_ROLE=,etcd,controlplane CATTLE_SERVER=https://proxyIP:port CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: search mis.amat.com amat.com nameserver  nameserver  nameserver  options rotate timeout:5 attempts:3
INFO: https://proxyIP:port/ping is accessible
INFO: dcalca2118.amat.com resolves to <IP_ADDRESS>
INFO: Value from https://proxyIP:port/v3/settings/cacerts is an x509 certificate
time="2021-12-15T17:57:25Z" level=info msg="Listening on /tmp/log.sock"
time="2021-12-15T17:57:25Z" level=info msg="Rancher agent version v2.6.2 is starting"
time="2021-12-15T17:57:25Z" level=info msg="Option customConfig=map[address:<CATTLE_IP> internalAddress: label:map[] roles:[etc
d controlplane] taints:[]]"
time="2021-12-15T17:57:25Z" level=info msg="Option etcd=true"
time="2021-12-15T17:57:25Z" level=info msg="Option controlPlane=true"
time="2021-12-15T17:57:25Z" level=info msg="Option worker=false"
time="2021-12-15T17:57:25Z" level=info msg="Option requestedHostname="
time="2021-12-15T17:57:25Z" level=info msg="Option dockerInfo={XPFX:VGHG:LKMI:6JIT:BZB3:KTJW:OSRL:2B7R:AZ3K:SWZK:2SNM:6T3E 13 4
0 9 1 overlay2 [[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] [] {[local] [bridge host ipvlan macv
lan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} true true true true true
true true true true true true true false 43 true 68 2021-12-15T09:57:25.206696589-08:00 json-file cgroupfs 1 0 3.10.0-1127.19.1.
el7.x86_64 Red Hat Enterprise Linux 7.8 linux x86_64 https://index.docker.io/v1/ 0xc00038c3f0 8<PHONE_NUMBER>6 [] /Application/dock
er_root     [] false 20.10.5   map[io.containerd.runc.v2:{runc [] } io.containerd.runtime.v1.linux:{runc []
} runc:{runc [] }] runc {  inactive false  [] 0 0  []} false  docker-init {d71fcd7d8303cbf684402823e425e9dd2e99285d d7
1fcd7d8303cbf684402823e425e9dd2e99285d} {b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7} {de4
0ad0 de40ad0} [name=seccomp,profile=default]  [] []}"
time="2021-12-15T17:57:25Z" level=info msg="Connecting to wss://proxyIP:port/v3/connect/register with token starting
with dnbn8vfrtmvsvlpdvp6xwbctsdl"
time="2021-12-15T17:57:25Z" level=info msg="Connecting to proxy" url="wss://proxyIP:port/v3/connect/register"
time="2021-12-15T17:57:25Z" level=error msg="Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body:
Operation cannot be fulfilled on nodes.management.cattle.io "m-2c4d8fa7db48": the object has been modified; please apply your
changes to the latest version and try again" error="websocket: bad handshake"
time="2021-12-15T17:57:25Z" level=error msg="Remotedialer proxy error" error="websocket: bad handshake"
time="2021-12-15T17:57:35Z" level=info msg="Connecting to wss://proxyIP:port/v3/connect/register with token starting
with dnbn8vfrtmvsvlpdvp6xwbctsdl"
time="2021-12-15T17:57:35Z" level=info msg="Connecting to proxy" url="wss://proxyIP:port/v3/connect/register"
time="2021-12-15T17:57:35Z" level=info msg="Waiting for node to register. Either cluster is not ready for registering, cluster i
s currently provisioning, or etcd, controlplane and worker node have to be registered"
Ingress Controller logs:

[15/Dec/2021:18:47:40 +0000] "GET /ping HTTP/1.1" 308 164 "-" "curl/7.66.0" 92 0.000 [cattle-system-rancher-80] [] - - - - 8b8bc9c425af72debe1e11853f2294e4

[15/Dec/2021:18:47:41 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 888e746e4f3f54e3cd198f12ef53461a

[15/Dec/2021:18:47:43 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 cd843970b8c3ff4374767e988df1ce67

[15/Dec/2021:18:47:45 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 103694acba1e96647e4a24a636e03b9d

[15/Dec/2021:18:47:47 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3242 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 2ac9f9a61dbfd25375db2fab6373fc7d

[15/Dec/2021:18:47:49 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 76ee64a4dd6436edbf6bb9971ea37ec4

[15/Dec/2021:18:47:51 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 53e37d5c5baf6805ec29dcd8bdb1f548

[15/Dec/2021:18:47:53 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 ef466849ee40ff3f8606467521c87b40

[15/Dec/2021:18:47:55 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 59a8b529c0b8ddfcfe7bd806a320baf7

[15/Dec/2021:18:47:57 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 ce522aa4ab8388c560d480627484dc73

[15/Dec/2021:18:48:00 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 8464f3829d33c57a923267455063a8b6

[15/Dec/2021:18:48:02 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 57e6e083c7450079cd66ecf598fded3e

[15/Dec/2021:18:48:04 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 2e8e1ead87a4d99c4035364777374f89

[15/Dec/2021:18:48:06 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3242 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 65761ea62a849412b786120d8626bb67

[15/Dec/2021:18:48:08 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.000 503 16d1b319230aadd7c2a3b4d8dcf86daf

[15/Dec/2021:18:48:10 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.003 [cattle-system-rancher-80] [] :80 0 0.003 503 a9f6d3fbe5b82b0f5390b53caa98352a

[15/Dec/2021:18:48:12 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 fdfcbcbdaaf4ab9dc2614dc0e2932ca7

[15/Dec/2021:18:48:14 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 d4670efd8d540954ad8157ef09b6c442

[15/Dec/2021:18:48:16 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 40a79b534198abdc84b27dd7fe6096e5

[15/Dec/2021:18:48:18 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.000 503 92ddbad8bbb60cc4c45073444942ed65

[15/Dec/2021:18:48:20 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 2d07a236463586aed5f823c1b7e29775

[15/Dec/2021:18:48:22 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 cbf18e66dc06648342783c9c91b1854c

[15/Dec/2021:18:48:24 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 e74f4f414d5261274d4b531db3ca700b

[15/Dec/2021:18:48:26 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3242 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 44be3ba02f473aea62c778207b2ecd0c

[15/Dec/2021:18:48:28 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 cea40cd246e1ba623e1d7b0e4de6eef2

[15/Dec/2021:18:48:30 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001<PHONE_NUMBER>b3f4ed72439e186d95f6d3c9d

[15/Dec/2021:18:48:32 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 bc698e1bf76469535a73de28480856da

[15/Dec/2021:18:48:34 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 ee43948d87a2cec7dbdcf510b7c52c50

[15/Dec/2021:18:48:36 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.002 503 283e80fee26b3580907b2a53f6118931

[15/Dec/2021:18:48:38 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3242 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 87e7c47f6aa79efbd145d7532b107d4c

[15/Dec/2021:18:48:40 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.003 [cattle-system-rancher-80] [] :80 0 0.002 503 f97903823939446ecfdece9e7c564e0f

[15/Dec/2021:18:48:41 +0000] "GET /ping HTTP/1.1" 308 164 "-" "curl/7.66.0" 92 0.000 [cattle-system-rancher-80] [] - - - - 954a2a0211b2daae5c26e1bd5ec71e58

[15/Dec/2021:18:48:42 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 f01897fa36ef956084ae4c5160d337cc

[15/Dec/2021:18:48:44 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 1be48003ed26735fd925439e5f2a8af6

[15/Dec/2021:18:48:46 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 3c9ee184c7913a3b9f1f3a4d39401ee2

[15/Dec/2021:18:48:48 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 b70358e33f951b64be3402ee4997b842

[15/Dec/2021:18:48:50 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.002 [cattle-system-rancher-80] [] :80 0 0.001 503 6a4e2df9bbe0bb8c9893f95a6e6c1efd

[15/Dec/2021:18:48:52 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 f6333278124539b13189083bef9f7c4c

[15/Dec/2021:18:48:54 +0000] "GET /v3/connect/config HTTP/2.0" 503 0 "-" "Go-http-client/2.0" 3299 0.001 [cattle-system-rancher-80] [] :80 0 0.001 503 8afe420b87935a58096c8fd77d28e657


PEP8 Version Conflict with pep8 1.6.2 (current version)

I'm getting the following error when starting sublime text 3:
Traceback (most recent call last):
  File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 158, in on_api_ready
    m.plugin_loaded()
  File "/Users/username/Library/Application Support/Sublime Text 3/Packages/SublimeLinter/sublimelinter.py", line 34, in plugin_loaded
    linter.initialize()
  File "linter in /Users/username/Library/Application Support/Sublime Text 3/Installed Packages/SublimeLinter-flake8.sublime-package", line 92, in initialize
  File "/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2354, in load
    self.require(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2371, in require
    items = working_set.resolve(reqs, env, installer)
  File "/usr/local/lib/python3.4/site-packages/pkg_resources/__init__.py", line 844, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pep8 1.6.2 (/usr/local/lib/python3.4/site-packages), Requirement.parse('pep8!=1.6.0,!=1.6.1,!=1.6.2,>=1.5.7'))
SublimeLinter: pyflakes activated: <builtin> (disabled in settings)
SublimeLinter: pep257 activated: <builtin>
SublimeLinter: flake8 activated: <builtin>
SublimeLinter: ERROR: could not launch ['<builtin>', 'pep8', '--version']
SublimeLinter: reason: [Errno 2] No such file or directory: '<builtin>'
SublimeLinter: PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
SublimeLinter: WARNING: no pep8 version could be extracted from:

This is a requirement of flake8, and has nothing to do with us.
See this StackOverflow answer for a potential solution.