Rancher Cattle Cluster Agent Could not Resolve Host

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):
Provision new HA cluster using RKE. The cluster.yml file should only have the nodes stanza. 3 nodes with all roles assigned. Delete one of the 3 rancher pods.
root@massimo-server:~# kubectl delete pod/rancher-6dc68bb996-95rbw -n cattle-system

Result:
The rancher pod is recreated due to the ReplicaSet but the cattle cluster agent fails and goes into CrashLoopBackOff state.
root@massimo-server:~# kubectl logs --follow \
  pod/cattle-cluster-agent-7bcbf99f56-vdrbs -n cattle-system

INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-7bcbf99f56-vdrbs CATTLE_SERVER=https://massimo.rnchr.nl
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
ERROR: https://massimo.rnchr.nl/ping is not accessible (Could not resolve host: massimo.rnchr.nl)

Using curl from outside of the cattle agent pod
root@massimo-server:~# curl https://massimo.rnchr.nl/ping
pong

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
rancher/rancher v2.1.1
Installation option (single install/HA): HA

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): RKE provisioned
Machine type (cloud/VM/metal) and specifications (CPU/memory): cloud, 2 CPU/4 GB
Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Docker (use docker info):

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 5
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-138-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: massimo-server
ID: 747D:4N33:DFHF:IWOG:Z3RK:G22I:3663:ZSHQ:5HKF:SOGK:M3V4:EKQA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

WARNING: No swap limit support

gz#11051

Deleting the cattle cluster agent pod, fixes it. Why is the cattle cluster agent having issues resolving the host when a rancher pod is recreated?

Here is the yaml file used to provision the cluster. All of the hosts are running Ubuntu 16.04 and using Docker 17.03.2-ce.
root@massimo-server:~# cat cluster.yml
nodes:
  - address: <IP_ADDRESS>
    user: root
    role:
      - controlplane
      - worker
      - etcd

  - address: <IP_ADDRESS>
    user: root
    role:
      - controlplane
      - worker
      - etcd

  - address: <IP_ADDRESS>
    user: root
    role:
      - controlplane
      - worker
      - etcd

Helm was used with an already existing Let's Encrypt SSL cert.
helm install rancher-stable/rancher \
  --name rancher \
  --namespace cattle-system \
  --set hostname=massimo.rnchr.nl \
  --set ingress.tls.source=secret

root@massimo-server:~# kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=fullchain.pem  --key=privkey.pem

I had exactly the same issue with Ubuntu server 18.04 LTS and docker 18.06.1-ce. Cluster was created with rke. Deleting and recreating the cattle-cluster-agent pod did not help.
I set the CATTLE_SERVER env of cattle-cluster-agent to the configured rancher ingress service, so kube-dns can resolve it: rancher.cattle-system.svc.cluster.local or <service>.<namespace>.<cluster>
Afterwards the pod started, but I don't know if this is correct. Normally you would go over external LB? But in my case this did not work because my cluster dns domain overlaps with dns domain outside in real network...

I had the same issue when the br_netfilter was not loaded. This caused some nat rules not to work, so the DNS clients where asking to the kube-dns cluster IP but got a response from the service IP which was rejected.
"modprobe br_netfilter" solved it for me.

I have been having this issue as well. I think I have solved why this issue is happening and have found two temporary fixes for it. The real solution will be for rancher to change their deployment logic or to not specify a default cluster domain name.
This issue, at least for me has been being caused by specifying a domain in the rke config file like:
services:
kubelet:
cluster_domain: mycluster.com
The problem to the fact that the cattle-cluster-agent pod is querying kube-dns to resolve the name to your rancher cluster (let us say it is rancher.mycluster.com). Kube-dns does not have an entry for rancher.mycluster.com, and thanks to the cluster_domain specification, believes it is authoritative for mycluster.com and answers cattle-cluster-agent that there are no records for that name. This  causes the cattle-cluster-agent pod to bootloop and fail.
To fix this you can change the yml for cattle-cluster-agent and either edit the dnspolicy or the value for the  CATTLE_SERVER variable provides to the pod.
dnspolicy
If you change the dnspolicy to Default from ClusterFirst this will cause the pod to go to external (not kube-dns) for name resolution. This makes resolution of rancher.mycluster.com work, however this will break attempts by the cattle-cluster-agent pod to contact other pods (I do not know if it needs to do so)
CATTLE_SERVER
The other fix for this is to edit the yml entry for CATTLE_SERVER and change the value from https://rancher.mycluster.com to https://rancher. This will have the pod requesting kube-dns to resolve for rancher which it does have an entry for, resulting in a working pod.
Long term rancher should probably make the second fix presented here the default value for CATTLE_SERVER when it comes up by default. That should prevent this issue from occurring.

@ scotthesterberg  That's greet.Would you tell us nginx.conf and conf.d/rancher ?

@armanriazi  Here is nginx.conf, though it is simple what is autogenerated by the RKE/Rancher install so I am not sure what you are hoping to find.
# Configuration checksum:<PHONE_NUMBER>2266334817

# setup custom paths that do not require root access
pid /tmp/nginx.pid;

daemon off;

worker_processes 4;

worker_rlimit_nofile 15360;

worker_shutdown_timeout 10s ;

events {
        multi_accept        on;
        worker_connections  16384;
        use                 epoll;
}

http {

        lua_package_cpath "/usr/local/lib/lua/?.so;/usr/lib/lua-platform-path/lua/5.1/?.so;;";
        lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/lib/lua/?.lua;;";

        init_by_lua_block {
                require("resty.core")
                collectgarbage("collect")

                local lua_resty_waf = require("resty.waf")
                lua_resty_waf.init()

                ok, res = pcall(require, "monitor")
                if not ok then
                error("require failed: " .. tostring(res))
                else
                monitor = res
                end
        }

        real_ip_header      X-Forwarded-For;

        real_ip_recursive   on;

        set_real_ip_from    <IP_ADDRESS>/0;

        geoip_country       /etc/nginx/geoip/GeoIP.dat;
        geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;
        geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;
        geoip_proxy_recursive on;

        aio                 threads;
        aio_write           on;

        tcp_nopush          on;
        tcp_nodelay         on;

        log_subrequest      on;

        reset_timedout_connection on;

        keepalive_timeout  75s;
        keepalive_requests 100;

        client_body_temp_path           /tmp/client-body;
        fastcgi_temp_path               /tmp/fastcgi-temp;
        proxy_temp_path                 /tmp/proxy-temp;

        client_header_buffer_size       1k;
        client_header_timeout           60s;
        large_client_header_buffers     4 8k;
        client_body_buffer_size         8k;
        client_body_timeout             60s;

        http2_max_field_size            4k;
        http2_max_header_size           16k;

        types_hash_max_size             2048;
        server_names_hash_max_size      1024;
        server_names_hash_bucket_size   64;
        map_hash_bucket_size            64;

        proxy_headers_hash_max_size     512;
        proxy_headers_hash_bucket_size  64;

        variables_hash_bucket_size      128;
        variables_hash_max_size         2048;

        underscores_in_headers          off;
        ignore_invalid_headers          on;

        limit_req_status                503;

        include /etc/nginx/mime.types;
        default_type text/html;

        gzip on;
        gzip_comp_level 5;
        gzip_http_version 1.1;
        gzip_min_length 256;
        gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
        gzip_proxied any;
        gzip_vary on;

        # Custom headers for response

        server_tokens on;

        # disable warnings
        uninitialized_variable_warn off;

        # Additional available variables:
        # $namespace
        # $ingress_name
        # $service_name
        # $service_port
        log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';

        map $request_uri $loggable {

                default 1;
        }

        access_log /var/log/nginx/access.log upstreaminfo if=$loggable;

        error_log  /var/log/nginx/error.log notice;

        resolver <IP_ADDRESS> valid=30s;

        # Retain the default nginx handling of requests without a "Connection" header
        map $http_upgrade $connection_upgrade {
                default          upgrade;
                ''               close;
        }

        map $http_x_forwarded_for $the_real_ip {

                default          $remote_addr;

        }

        # trust http_x_forwarded_proto headers correctly indicate ssl offloading
        map $http_x_forwarded_proto $pass_access_scheme {
                default          $http_x_forwarded_proto;
                ''               $scheme;
        }

        # validate $pass_access_scheme and $scheme are http to force a redirect
        map "$scheme:$pass_access_scheme" $redirect_to_https {
                default          0;
                "http:http"      1;
                "https:http"     1;
        }

        map $http_x_forwarded_port $pass_server_port {
                default           $http_x_forwarded_port;
                ''                $server_port;
        }

        # map port 442 to 443 for header X-Forwarded-Port
        map $pass_server_port $pass_port {
                442              443;
                default          $pass_server_port;
        }

        # Obtain best http host
        map $http_host $this_host {
                default          $http_host;
                ''               $host;
        }

        map $http_x_forwarded_host $best_http_host {
                default          $http_x_forwarded_host;
                ''               $this_host;
        }

        # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
        # If no such header is provided, it can provide a random value.
        map $http_x_request_id $req_id {
                default   $http_x_request_id;

                ""        $request_id;

        }

        server_name_in_redirect off;
        port_in_redirect        off;

        ssl_protocols TLSv1.2;

        # turn on session caching to drastically improve performance

        ssl_session_cache builtin:1000 shared:SSL:10m;
        ssl_session_timeout 10m;

        # allow configuring ssl session tickets
        ssl_session_tickets on;

        # slightly reduce the time-to-first-byte
        ssl_buffer_size 4k;

        # allow configuring custom ssl ciphers
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;

        ssl_ecdh_curve auto;

        proxy_ssl_session_reuse on;

        upstream cattle-system-rancher-80 {
                least_conn;

                keepalive 32;

                server <IP_ADDRESS>:80 max_fails=0 fail_timeout=0;
                server <IP_ADDRESS>:80 max_fails=0 fail_timeout=0;
                server <IP_ADDRESS>:80 max_fails=0 fail_timeout=0;

        }

        upstream upstream-default-backend {
                least_conn;

                keepalive 32;

                server <IP_ADDRESS>:8080 max_fails=0 fail_timeout=0;

        }

        ## start server _
        server {
                server_name _ ;

                listen 80 default_server  backlog=511;

                listen [::]:80 default_server  backlog=511;

                set $proxy_upstream_name "-";

                listen 442 proxy_protocol   default_server  backlog=511 ssl http2;

                listen [::]:442 proxy_protocol  default_server  backlog=511 ssl http2;

                # PEM sha: c014591aaf641d7b7c85d9d6d193b327b88de3e4
                ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
                ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

                location / {

                        set $namespace      "";
                        set $ingress_name   "";
                        set $service_name   "";
                        set $service_port   "0";
                        set $location_path  "/";

                        rewrite_by_lua_block {

                        }

                        log_by_lua_block {

                                monitor.call()
                        }

                        if ($scheme = https) {
                                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
                        }

                        access_log off;

                        port_in_redirect off;

                        set $proxy_upstream_name "upstream-default-backend";

                        client_max_body_size                    "1m";

                        proxy_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        proxy_set_header                        Upgrade           $http_upgrade;

                        proxy_set_header                        Connection        $connection_upgrade;

                        proxy_set_header X-Request-ID           $req_id;
                        proxy_set_header X-Real-IP              $the_real_ip;

                        proxy_set_header X-Forwarded-For        $the_real_ip;

                        proxy_set_header X-Forwarded-Host       $best_http_host;
                        proxy_set_header X-Forwarded-Port       $pass_port;
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

                        proxy_set_header X-Original-URI         $request_uri;

                        proxy_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        proxy_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   5s;
                        proxy_send_timeout                      60s;
                        proxy_read_timeout                      60s;

                        proxy_buffering                         "off";
                        proxy_buffer_size                       "4k";
                        proxy_buffers                           4 "4k";
                        proxy_request_buffering                 "on";

                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

                        proxy_pass http://upstream-default-backend;

                        proxy_redirect                          off;

                }

                # health checks in cloud providers require the use of port 80
                location /healthz {

                        access_log off;
                        return 200;
                }

                # this is required to avoid error if nginx is being monitored
                # with an external software (like sysdig)
                location /nginx_status {

                        allow <IP_ADDRESS>;

                        allow ::1;

                        deny all;

                        access_log off;
                        stub_status on;
                }

        }
        ## end server _

        ## start server rancher.rancher09
        server {
                server_name rancher.rancher09 ;

                listen 80;

                listen [::]:80;

                set $proxy_upstream_name "-";

                listen 442 proxy_protocol   ssl http2;

                listen [::]:442 proxy_protocol  ssl http2;

                # PEM sha: 506a240a1d644efc7c4504160b671c06f295ffae
                ssl_certificate                         /etc/ingress-controller/ssl/cattle-system-tls-rancher-ingress.pem;
                ssl_certificate_key                     /etc/ingress-controller/ssl/cattle-system-tls-rancher-ingress.pem;

                ssl_trusted_certificate                 /etc/ingress-controller/ssl/cattle-system-tls-rancher-ingress-full-chain.pem;
                ssl_stapling                            on;
                ssl_stapling_verify                     on;

                location / {

                        set $namespace      "cattle-system";
                        set $ingress_name   "rancher";
                        set $service_name   "rancher";
                        set $service_port   "80";
                        set $location_path  "/";

                        rewrite_by_lua_block {

                        }

                        log_by_lua_block {

                                monitor.call()
                        }

                        if ($scheme = https) {
                                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
                        }

                        port_in_redirect off;

                        set $proxy_upstream_name "cattle-system-rancher-80";

                        # enforce ssl on server side
                        if ($redirect_to_https) {

                                return 308 https://$best_http_host$request_uri;

                        }

                        client_max_body_size                    "1m";

                        proxy_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        proxy_set_header                        Upgrade           $http_upgrade;

                        proxy_set_header                        Connection        $connection_upgrade;

                        proxy_set_header X-Request-ID           $req_id;
                        proxy_set_header X-Real-IP              $the_real_ip;

                        proxy_set_header X-Forwarded-For        $the_real_ip;

                        proxy_set_header X-Forwarded-Host       $best_http_host;
                        proxy_set_header X-Forwarded-Port       $pass_port;
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

                        proxy_set_header X-Original-URI         $request_uri;

                        proxy_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        proxy_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   30s;
                        proxy_send_timeout                      1800s;
                        proxy_read_timeout                      1800s;

                        proxy_buffering                         "off";
                        proxy_buffer_size                       "4k";
                        proxy_buffers                           4 "4k";
                        proxy_request_buffering                 "on";

                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_tries               3;

                        proxy_pass http://cattle-system-rancher-80;

                        proxy_redirect                          off;

                }

        }
        ## end server rancher.rancher09

        # default server, used for NGINX healthcheck and access to nginx stats
        server {
                # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
                # Changing this value requires a change in:
                # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
                listen 18080 default_server  backlog=511;
                listen [::]:18080 default_server  backlog=511;
                set $proxy_upstream_name "-";

                location /healthz {

                        access_log off;
                        return 200;
                }

                location /is-dynamic-lb-initialized {

                        access_log off;

                        content_by_lua_block {
                                local configuration = require("configuration")
                                local backend_data = configuration.get_backends_data()
                                if not backend_data then
                                ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
                                return
                                end

                                ngx.say("OK")
                                ngx.exit(ngx.HTTP_OK)
                        }
                }

                location /nginx_status {
                        set $proxy_upstream_name "internal";

                        access_log off;
                        stub_status on;
                }

                location / {

                        set $proxy_upstream_name "upstream-default-backend";

                        proxy_pass          http://upstream-default-backend;

                }

        }
}

stream {
        log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

        access_log /var/log/nginx/access.log log_stream;

        error_log  /var/log/nginx/error.log;

        # TCP services

        # UDP services

}

@armanriazi I am afraid I do not know where to locate conf.d/rancher.conf. Should is ve in a container (which)? Or on the host os? What are we trying to find?

@scotthesterberg Well,you can configure nginx two way Host and Container although there isn't any diff.
I have question: How you configure LB kind of autogenerate so I dont't know what is this link(manual config)?
https://rancher.com/docs/rancher/v2.x/en/installation/ha/rke-add-on/layer-4-lb/#2-configure-load-balancer

I solved value of server-url/v3/settings/cacerts and intermediate CA but I still getting this error (I saw issues similar but there was't full solver and I think it not granular nature system yet)
Thanks friends.

INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM=f4400caeebc0e481b1edfa15e5ba19b9756bc132d0a9448dc82479b718f8cdc9 CATTLE_CLUSTER=true CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-88dbc897d-x9hp7 CATTLE_SERVER=https://rancher.nik.com CATTLE_SERVICE_PORT=tcp://<IP_ADDRESS>:80 CATTLE_SERVICE_PORT_443_TCP=tcp://<IP_ADDRESS>:443 CATTLE_SERVICE_PORT_443_TCP_ADDR=<IP_ADDRESS> CATTLE_SERVICE_PORT_443_TCP_PORT=443 CATTLE_SERVICE_PORT_443_TCP_PROTO=tcp CATTLE_SERVICE_PORT_80_TCP=tcp://<IP_ADDRESS>:80 CATTLE_SERVICE_PORT_80_TCP_ADDR=<IP_ADDRESS> CATTLE_SERVICE_PORT_80_TCP_PORT=80 CATTLE_SERVICE_PORT_80_TCP_PROTO=tcp CATTLE_SERVICE_SERVICE_HOST=<IP_ADDRESS> CATTLE_SERVICE_SERVICE_PORT=80 CATTLE_SERVICE_SERVICE_PORT_HTTP=80 CATTLE_SERVICE_SERVICE_PORT_HTTPS=443

INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
error : ERROR: https://rancher.nik.com/ping is not accessible (Could not resolve host:)

I have tried to solve every solution but it dosn't work
solutions:
Change cluster_domain
Solved /etc/resolv.conf on Host and agent containers with internal org DNS and <IP_ADDRESS>
Update to v2.1.7
Flush docker containers
Change server_url to https://rancher and https://rancher.server.com
Configure LB 4 and LB7 with different mode/solutions
Solved CA private with intermedate CA

Can I  replace cattle pods ,deploy with adding manual Docker Swarm?

Dear @armanriazi, did you  find the solution for  it?
I have the same problem.

@gatici  I am going to wait for new releases then change OS to 1804 and using kube V13 finally I thought I might have to use Tk8

I'm having the same issue

I'm having the same issue

same issue
ubuntu: 18.04
docker: 18.09.6

same issue
ubuntu: 18.04
docker: 18.9.7

Same issue here:
ubuntu: 18.04
Docker version 18.09.7, build 2d0083d
This happens for me on 2 different machines when provisioning a new custom cluster through the Rancher UI with the long Docker install command.

same issue
redhat 8
docker-ce-3:18.09.1-3.el7.x86_64

I found a solution, reference: https://www.jianshu.com/p/5c13ebfd9947
Specifying the domain name and ip mapping in the k8s api will solve.
# Specify kubectl configuration file
$ export KUBECONFIG=xxx/xxx/xx.kubeconfig.yaml
$ kubectl -n cattle-system patch  deployments cattle-cluster-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "your domain name"
                        ],
                        "ip": "your real ip"
                    }
                ]
            }
        }
    }
}'
$ kubectl -n cattle-system patch  daemonsets cattle-node-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                        "hostnames":
                        [
                            "your domain name"
                        ],
                        "ip": "your real ip"
                    }
                ]
            }
        }
    }
}'

having the same issue on RancherOS.

Glad it at least partly worked for you @TheAifam5! Always good to hear a post helps someone.

same issue here:
I did manage to get it working by changing  CATTLE_SERVER  to another domain which I can resolve to the same rancher IP on my DNS server, don't know what the implications are though

I also have run into this issue after performing a restore from etcd snapshot. For me, cattle-cluster-agent is fine during cluster initialization, but after restoring from etcd snapshot, the cattle-cluster-agent will not be able to resolve the hostname.
It's a bit confusing because I am unsure how it was able to do so during cluster initialization. It is as if the agent uses Default dnsPolicy during cluster initialization but then afterward, if it is ever rebooted, it resorts to the default configuration of ClusterFirst dnsPolicy.

I also had to change dnsPolicy from clusterFirst to Default

It looks like when using Debian 10 it may be a problem with iptables. I followed this which solved the problem for me.

For me, changing the CATTLE_SERVER solves at first, but other places seem to point to the same address inside the pod.
I fixed it by adding a host alias (/etc/hosts entries) to point to the internal IP address for both cattle-cluster-agent and cattle-node-agent.
Looks like rancher should have an internal IP setting that it should use for this. In AWS, this causes additional cost unnecessarily.

@zbialik
It magically worked for me as well after changing dnsPolicy to Default, instead of ClusterFirst.
Thanks!

Is there any fix for this at a higher level? Now the DigitalOcean Kubernetes distribution has this issue which I ran into on my local cluster, and my cluster is unable to check in to Rancher.

This is still an issue: https://github.com/rancher/rancher/issues/24311

same problem

I also have run into this issue after performing a restore from etcd snapshot. For me, cattle-cluster-agent is fine during cluster initialization, but after restoring from etcd snapshot, the cattle-cluster-agent will not be able to resolve the hostname.
It's a bit confusing because I am unsure how it was able to do so during cluster initialization. It is as if the agent uses Default dnsPolicy during cluster initialization but then afterward, if it is ever rebooted, it resorts to the default configuration of ClusterFirst dnsPolicy.
As a workaround, I edit the cattle-cluster-agent workload after initializing such that my dns nameservers are configured (could also just set dnsPolicy to Default)
Would be nice to see this as a possible configuration when configuring cluster at initialization.
works for me, thanks much~

This ended up being DNS for me as well - the cattle-cluster-agent was constantly rebooting and not allowing for items like ingress to work properly. I had tried the dnsPolicy setting as suggested, tried putting in entries in the hosts file, tried variations on the host OS, Kubernetes version, and Rancher version, all to no avail.
What did work was setting up DNS "properly." Using the "roll your own / Custom" option in the Rancher Cluster Creation, not specifying any additional settings (such as private /public addresses), on a greenfield Ubuntu 18.04 setup (with both ipv4 and ipv6 addresses) / Rancher 2.3.3 / Kubernetes 1.16, I started by setting a DNS 'search' setting (in the netplan yaml, ethernets/eth0/nameservers/search) for all of the boxes in the mix. I then stood up an internal DNS server with a DNS zone to match, and then ensured that all of the boxes had host (A/AAAA) entries for both IPv4 and IPv6. With that in place, everything magically started working. Did NOT adjust the dnsPolicy setting in the end.

Still experiencing this.
Impossible to add new custom nodes to a newly created cluster when it's happening obviously. Occurred in two different attempts at provisioning nodes on Rancher clusters. Glad I found this thread!
Overriding host entries via hostAliases for cattle-node-agent and cattle-cluster-agent brought them to life.

Still same thing on the latest rancher, rke, rancheros....
cattle-node-agent works after setting the hostAliases, but cattle-cluster-agent still crashing even has the hostAliases set.

Hi
Having the same issue on a CentOS Linux release 7.7.1908 (Core). Nameserver in the cattle-cluster-agent is set to nameserver <IP_ADDRESS> when using the ClusterFirst.
Kernel version: 3.10.0-1062.9.1.el7.x86_64
RKE version: v1.0.4
Rancher version: v2.3.5
Kubernetes version:
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Docker version:
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Logs cattle-cluster-agent:
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
ERROR: https://rancher-tooling/ping is not accessible (Could not resolve host: rancher-tooling)

Workaround
Changed the dnsPolicy from the cattle-cluster-agent deployment:
From:
dnsPolicy: ClusterFirst

To:
dnsPolicy: Default

Regards
Matthias

On Debian 10, this issue is caused by the change from iptables to nftables.
This is resolved with:
update-alternatives --set iptables /usr/sbin/iptables-legacy
reboot

Regards
Jason

yep, see https://github.com/rancher/rancher/issues/16454#issuecomment-544621587 ;-)

This can happen also using the external and internal DNS.
extenal
ERROR: https://rancher.domain.com:443/ping is not accessible (Could not resolve host: rancher.domain.com)

internal
ERROR: https://cattle-system.svc.cluster.local:443/ping is not accessible (Could not resolve host: cattle-system.svc.cluster.local)

It also happen on my 2 different os Bottlerocket and AmazonLinux 2
NAME                                        STATUS   ROLES    AGE   VERSION               INTERNAL-IP      OS-IMAGE                         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-10-0-0-67.eu-west-1.compute.internal     Ready    <none>   39m   v1.15.10              <IP_ADDRESS>        Bottlerocket OS 0.3.1            5.4.16                          containerd://1.3.3+unknown
ip-10-0-10-161.eu-west-1.compute.internal   Ready    <none>   27h   v1.15.10-eks-bac369   <IP_ADDRESS>      Amazon Linux 2                   4.14.165-133.209.amzn2.x86_64   docker://18.9.9

docker & kubernetes version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.10-eks-bac369", GitCommit:"bac3690554985327ae4d13e42169e8b1c2f37226", GitTreeState:"clean", BuildDate:"2020-02-26T01:12:54Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"

cattle-cluster-agent deployment
Name:                   cattle-cluster-agent
Namespace:              cattle-system
CreationTimestamp:      Mon, 16 Mar 2020 17:00:38 +0000
Labels:                 app=cattle-cluster-agent
Annotations:            deployment.kubernetes.io/revision: 2
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"cattle-cluster-agent","namespace":"cattle-system"},"spec"...
Selector:               app=cattle-cluster-agent
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=cattle-cluster-agent
  Service Account:  cattle
  Containers:
   cluster-register:
    Image:      rancher/rancher-agent:v2.3.5
    Port:       <none>
    Host Port:  <none>
    Environment:
      CATTLE_SERVER:       https://cattle-system.svc.cluster.local:443
      CATTLE_CA_CHECKSUM:  b5b5eefc0bff6fc86164b9779b288f293e153ef77304531fc4728ecc823cc2a8
      CATTLE_CLUSTER:      true
      CATTLE_K8S_MANAGED:  true
    Mounts:
      /cattle-credentials from cattle-credentials (ro)
  Volumes:
   cattle-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cattle-credentials-5a098aa
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      False   MinimumReplicasUnavailable
OldReplicaSets:  <none>
NewReplicaSet:   cattle-cluster-agent-85f7d86f4f (1/1 replicas created)
Events:          <none>

I'm trying now the @matthiasdeblock solution to test it.

It's still there!

same here on a sles15 sp1 with rke install.

I can confirm it's on rancher 2.4.2 as well.

I had this issue because I wanted to create master node with controlplane + etcd only.
It seems that worker role is compulsory anyway.
So I've created master node with the 3 roles and no more worries !
After I could create additional nodes with worker role only.
Then I tainted my master node to prevent new pods to be deployed on it.

I think Rancher should be more explicit about roles requirements for RKE because it's an outstanding key feature but the UX is discutable for custom node creation.

After open all ports to nodes in the same cluster, I fixed this problem, hope it helpful to some guys.

This happen to me when i add manually the cloudProvider to the cluster:
cloud_provider:
      name: aws

But then i created a new cluster with the cloudProvider from rancher ui config, and add some labels to the cloud_provider:
cloud_provider:
    awsCloudProvider:
      global:
        disable-security-group-ingress: false
        disable-strict-zone-check: false
      service_override: null
    name: aws

And works without problems

I ran into this issue when I used a pfsense firewall as a load balancer in front of my cluster. I was able to resolve it by changing my outbound NAT as recommended here in the pfsense docs:
https://docs.netgate.com/pfsense/en/latest/book/loadbalancing/troubleshooting-server-load-balancing.html#unable-to-reach-a-virtual-server-from-a-client-in-the-same-subnet-as-the-pool-server
Essentially the fact that my nodes were on the same subnet as my load balancer caused traffic to not route properly because of incorrect source and destination IPs. This led to the failure in deployment described in this issue even though I was able to use my cluster for the most part normally otherwise. It may not necessarily represent what others have seen here, but could be a potential item to look at for others.
Note: This change does result in you losing some information regarding what host traffic originates from in traffic that hits your load balancer. This was not important to me so I did not worry about that loss of information, but others may.

Still there in v2.4.3 as well.

I had same issue and after changing from ClusterFirst to Default for cattle-cluster-agent, it was able to proceed further but it failed soon throwing -
level=fatal msg="steve exited: listen tcp <IP_ADDRESS>:6443: bind: address already in use"
I have added a loopback to self in my /etc/hosts file and I have single node cluster.
I had to add this file in hosts file at the time of installation itself since it wasn't able to create ssh tunnel (although why it tries to create tunnel to self?).
I am using rancher v2.4.3 with kubernetes (1.18 client & 1.17server) and docker  19.03.9

Same issue.
I am using rancher v2.4.3 with latest RKE v1.2.0-rc1 and docker 19.03.9

still here

@kubealex looks like a networking problem for me, you can try the work around via setting "HostNetwork"

still here

Workaround
Changed the dnsPolicy from the cattle-cluster-agent deployment:
From:
dnsPolicy: ClusterFirst
To:
dnsPolicy: Default

in my case, coredns wasn't working due to a missing capability enabled in cri-o (NET_RAW)

same issue
Rancher 2.4.4
Kubernetes Version: v1.18.3
minikube 1.11
Docker 19.03.8

Unbelievable that this is not resolved yet. I faced this issue more than 1 year back (I don't remember how I resolved back then). Now I am installing rancher 2.4 on a brand new Windwos 10 (docker-desktop) and Im facing the exact same issue.
Im installing using helm, and I cant find a way to use the workaround:
Workaround
Changed the dnsPolicy from the cattle-cluster-agent deployment:
From:
dnsPolicy: ClusterFirst
To:
dnsPolicy: Default

@brunobertechini , it's really weird, whatever the provider aws, minikube, docker-desktop, I have a host resolution error.
`INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM=04629d3294e6f84c25d055f13539346540170ad380a43375cbedf47f9aa5ca54 CATTLE_CLUSTER=true CATTLE_FEATURES=dashboard=true CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-6d85bc69d-fkx6r CATTLE_SERVER=https://k3s
INFO: Using resolv.conf: search home nameserver <IP_ADDRESS>
ERROR: https://k3s/ping is not accessible (Could not resolve host: k3s)`
After changed the dnsPolicy from the cattle-cluster-agent deployment:
From:
dnsPolicy: ClusterFirst
To:
dnsPolicy: Default
I still got the same error from dns resolution from cattle-cluster-agent
https://k3s/ping is reachable by my browser i got a pong

Update : I have no more errors as soon as I totally disable my firewall. In this case LittleSnitch on macOS with which I have a dnsPolicy : ClusterFirst
I will try again with providers like docker-desktop and AWS.

We have the same issue on Rancher 2.4.2. As soon as a node from the local cluster gets deleted, we are getting error messages from all custom clusters that try to connect to the CATTLE_SERVER that is set on each of the cattle_cluster_agent. Same issue when the retries pass onto the cattle_node_agent after a few minutes. We tried to set the dnspolicy to "clusterfirst with host" and also tried to resolve the DNS Address internally. It seems like the only solution is to pass on the /etc/hosts/ alias file to each of the cattle cluster and node agents with the relevant dns entries to resolve this issue and then resolve it manually.
Additional Note:
The first deployment was only solely the etcd Node on local that we are trying to delete. We need to get this local node off and add 3 further nodes to expand the local system to HA.
Our procedure was the following:

Add 3 nodes including all components to the local cluster with 1 Node. Everything works fine
Add ingresses and deploy rancher to all the nodes
Removed the one node from the local cluster that was provisioned before
All custom clusters are not reachable anymore, with the following error:

{"log":"time="2020-06-17T12:18:25Z" level=info msg="Connecting to proxy" url="wss://replacedurl.com"\n","stream":"stdout","time":"2020-06-17T12:18:25.738167056Z"}
{"log":"time="2020-06-17T12:18:25Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp replacedip:443: connect: connection refused"\n","stream":"stdout","time":"2020-06-17T12:18:25.739445736Z"}
{"log":"time="2020-06-17T12:18:25Z" level=error msg="Remotedialer proxy error" error="dial tcp replacedip:443: connect: connection refused"\n","stream":"stdout","time":"2020-06-17T12:18:25.739461749Z"}
{"log":"time="2020-06-17T12:18:35Z" level=info msg="Connecting to wss://replacedurl.com/v3/connect with token replacedtoken"\n","stream":"stdout","time":"2020-06-17T12:18:35.739883831Z"}

I ran into this issue when I used a pfsense firewall as a load balancer in front of my cluster. I was able to resolve it by changing my outbound NAT as recommended here in the pfsense docs:
https://docs.netgate.com/pfsense/en/latest/book/loadbalancing/troubleshooting-server-load-balancing.html#unable-to-reach-a-virtual-server-from-a-client-in-the-same-subnet-as-the-pool-server
Essentially the fact that my nodes were on the same subnet as my load balancer caused traffic to not route properly because of incorrect source and destination IPs. This led to the failure in deployment described in this issue even though I was able to use my cluster for the most part normally otherwise. It may not necessarily represent what others have seen here, but could be a potential item to look at for others.
Note: This change does result in you losing some information regarding what host traffic originates from in traffic that hits your load balancer. This was not important to me so I did not worry about that loss of information, but others may.

I am using the loadbalancer on a Fortigate FW, and was facing the same issues. Velocity's comment jogged my memory, I have seen this same thing in prod with other VIPs. Since my cluster shares IP space with my LB VIP, I set up a hairpin for that traffic. At least this way I don't have to NAT and loose that data. Hairpinning is different between vendors, but at least it's a place to start.

I'm using Rancher 2.4.4, K8s 1.17.6, Docker 19.03.8..same issue. I tried every solution here but no luck.

I upgraded two separate Rancher clusters as follows and ran into this issue on one Rancher cluster:
Rancher: From 2.2.8 to 2.3.6
Rancher K8S: From 1.13.10 to 1.15.11
Docker: From 17.3.2 to 19.3.11
From my 3 Rancher nodes, I can no longer curl or telnet to my Rancher cluster DNS (myranchercluster.domain.com). But, when I stop Docker on one of the Rancher nodes curl and telnet atempts to my Rancher DNS work. This issue is causing the cluster and node agent pods to constantly restart.

ERROR: https://myranchercluster.domain.com/ping is not accessible (Failed to connect to myranchercluster.domain.com port 443: Connection timed out)

I upgraded both my Rancher clusters the same exact way - but only one cluster has this issue. It doesn't make sense what changes when I stop Docker on one Rancher node and then I can successfully curl and telnet to myranchercluster.domain.com:443.

Same issue for me on a fresh install.
Resolved by changing the dnsPolicy from the cattle-cluster-agent deployment to Default and setting the appropriated records on the dns servers that are declared on the nodes, using hosts files did not work.

Seeing the same issues on fresh installs from time to time, rancher itself is returning 404's on ping

any solution?

Unfortunately this issue is still there in 2.4.5
I am managing 3 clusters (1 is local) using rancher and 2 out of 3 had this problem. I didn't see this issue on my development cluster but both local and production had this issue (both of them are on AWS).
The workaround that we have applied is to edit the "cattle-cluster-agent" workload to give local IP for your rancher URL -

This worked like a charm for us and no more failure/restart of the cluster-agent were observed after that.
Hope this helps for you too !!!

I'm able to fix issue on rhel8 after running below commands,
`
sudo iptables -P FORWARD ACCEPT
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/50-docker-forward.conf
for mod in ip_tables ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr; do sudo modprobe $mod; echo $mod | sudo tee -a /etc/modules-load.d/iptables.conf; done
sudo dnf -y install network-scripts
sudo systemctl enable network
sudo systemctl disable NetworkManager`
Got this from here https://github.com/superseb/rancher/commit/cacc5837ab50db42b078f32ac236dc3367130db1

Im having the same problem currently for adding a local cluster, it happened both with kind and minikube so i dont think thats part of the problem.
it is a fresh install in centos 7
The thing is that in the yaml from rancher there's no dnsPolicy : ClusterFirst line inside the cattle-cluster-agent configuration, and even if i add it (either with ClusterFirst or Default), the problem persists.

I have spent days troubleshooting this. I have ubuntu bionic VMs with dual NICs.
My cluster would only work if I deployed it by specifying cluster host addresses on the primary network interface of each host.
Changing default route from primary to secondary NICs does not resolve issue.
CNI is working, just cant resolve dns from within the pods. DNS resolution host side is fine on either NIC.
My only solution has been to remove the secondary interface and deploy on the primary.

Oh my God，I'm tired, I re-run it now and then with a virtualbox VM and it still doesn't work the same way!
unbelievable

I'm able to fix issue on rhel8 after running below commands,
`
sudo iptables -P FORWARD ACCEPT
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/50-docker-forward.conf
for mod in ip_tables ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr; do sudo modprobe $mod; echo $mod | sudo tee -a /etc/modules-load.d/iptables.conf; done
sudo dnf -y install network-scripts
sudo systemctl enable network
sudo systemctl disable NetworkManager`
Got this from here superseb@cacc583

useful

I'm able to fix issue on rhel8 after running below commands,
`
sudo iptables -P FORWARD ACCEPT
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/50-docker-forward.conf
for mod in ip_tables ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr; do sudo modprobe $mod; echo $mod | sudo tee -a /etc/modules-load.d/iptables.conf; done
sudo dnf -y install network-scripts
sudo systemctl enable network
sudo systemctl disable NetworkManager`
Got this from here https://github.com/superseb/rancher/commit/cacc5837ab50db42b078f32ac236dc3367130db1

Didnt work for me

Same issue in v2.4.5

can confirm with v.2.4.5

以下转自这个 issues
https://github.com/rancher/rancher/issues/18832#issuecomment-547728856
Adding host alias to the cattle agent can be done in this way:

kubectl -n cattle-system patch  deployments cattle-cluster-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                      "hostnames":
                      [
                        "{{ rancher_server_hostname }}"
                      ],
                      "ip": "{{ rancher_server_ip }}"
                    }
                ]
            }
        }
    }
}'

kubectl -n cattle-system patch  daemonsets cattle-node-agent --patch '{
 "spec": {
     "template": {
         "spec": {
             "hostAliases": [
                 {
                    "hostnames":
                      [
                        "{{ rancher_server_hostname }}"
                      ],
                    "ip": "{{ rancher_server_ip }}"
                 }
             ]
         }
     }
 }
}'

Hi,
same for me on "Custom Cluster" (on AWS but without Cloud Provider) => dnsPolicy to default seems to resolve the problem but is the cause for other problems (prometheus monitoring for example).
In fact what i have noticed is that there is no problem (with ClusterFirst dnspolicy and Prometheus...) if i'm modifying node scheduling policy to deploy on "etcd" node only => when cattle-cluster-agent is on controlplane node there is the connectivity problem.
any idea what could be the reason for that ?
Julien

Hello,
This issue still persists without any concrete solution. I imported an EKS cluster to a Rancher server v 2.4.3 but the status of eks cluster on Rancher server was stuck on pending. I even patched the cluster agent deployment and node agent daemonset to replace the DNS name with the IP address but it didn't work. The only error message the logs show is:
ERROR: https://IP-address/ping is not accessible (Failed to connect to IP-address port 443: Connection timed out)

I am on Rancher server v:2.4.8 and still have the same problem with running the quick-start install, the cattle cluster agent could not resolve the host name as found in my /etc/hosts file .  I re-imaged my Linux and re-started the server container and I did 2 things that worked as a workaround
a. I added  CATTLE_AGENT_IP=<IP_ADDRESS> to the docker run command on the server.
b. I made sure when I did the initial login in the rancher gui I used the IP address (again) as the URL to connect to the server . Not the name, and the web gui said .. are you are you want to do this, it looks like a private ip. Yes. this perhaps is also a key point.
ps. before I re-installed I tried some of the suggestions to change the .yaml as mentioned above. that was only a partial fix.
pps. I would recommend installing kubectl on your master node.

@onkar-dhuri please help out on this. And if anyone from Rancher is reading this.... This is a problem so many people have had for a really long time. Can you dedicate a page in your docs to addresses thia 1 particular problem. It's a humble request. Thanks.

@onkar-dhuri please help out on this. And if anyone from Rancher is reading this.... This is a problem so many people have had for a really long time. Can you dedicate a page in your docs to addresses thia 1 particular problem. It's a humble request. Thanks.

Nslookup IP of your rancher URL won't work because it's a public IP. Try giving local IP which you can get by executing the command ifconfig on the machine which has rancher installed.

For me to get away from this issue was to add a local dns server in my lab environment. I know this is not something that will work for everyone, but thought of adding a comment here. I am also surprised to see this issue is still open since Nov 2018.
Here are the steps if anyone wants to follow:

I created dns server and dns records in ubuntu 20.04 by following this youtube guide.
Added dns server IP address to all kubernetes nodes.

Hope this helps someone who is still experiencing the issue. 👍
My suggestion to rancher dev team is to maybe add a global custom dns entry input/value in settings that can allow everyone to customize host entries for cluster agent without going through kubectl patch workaround? In my case kubectl was not useful because my kubernetes nodes didnt have kubectl and cluster configurations available.

I also ended up deploying a dns server in my local network.

I already have a DNS set up, so I enabled automatic DHCP hostname registration in DNS. Issue is not resolved even if I can nslookup from the host.
Docker container logs:
INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM=5bd5632b37de5f40b5ef0233aa01fe86ae081c22e2f6633dcb8fbde586a6e80a CATTLE_CLUSTER=true CATTLE_FEATURES= CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=true CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-7c56d97f45-wfx95 CATTLE_SERVER=https://rancher.<MY DOMAIN>
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local <MY DOMAIN> options ndots:5
ERROR: https://rancher.<MY DOMAIN>/ping is not accessible (The requested URL returned error: 404)

Ip address
$ ip a
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:88:a5:11 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global dynamic ens192
       valid_lft 5601sec preferred_lft 5601sec
    inet6 fe80::250:56ff:fe88:a511/64 scope link
       valid_lft forever preferred_lft forever

nslookup
$ nslookup srv01-node-1.<MY DOMAIN>
Server:		<IP_ADDRESS>
Address:	<IP_ADDRESS>#53

Non-authoritative answer:
Name:	srv01-node-1.<MY DOMAIN>
Address: <IP_ADDRESS>

@TomasTokaMrazek - Your issue seems to be different then what most folks on this tread are experiencing. 404 error code means that dns is resolving and its having issue accessing the application. I would suggest using different tread or start a conversation in stackoverflow or slack channel with rancher.

@tekspacedemo I am creating new issue, but I think that the root cause is the same. I have 404, becuase I actually use the same domain in homelab as in production. And since I have a public wildcard A DNS record for the domain, it gets resolved, but sends /ping to wrong address, see https://adamo.wordpress.com/2020/01/14/ranchers-cattle-cluster-agent-and-error-404/
This is in my opinion the root cause is the same, i.e. sending /ping to wrong address. I am verifying it by removing pulic DNS records.

Same issue for me on a fresh install.
Resolved by changing the dnsPolicy from the cattle-cluster-agent deployment to Default and setting the appropriated records on the dns servers that are declared on the nodes, using hosts files did not work.

How to change dmsPolicy. ? any Command to get into the POD?

I upgraded two separate Rancher clusters as follows and ran into this issue on one Rancher cluster:
Rancher: From 2.2.8 to 2.3.6
Rancher K8S: From 1.13.10 to 1.15.11
Docker: From 17.3.2 to 19.3.11
From my 3 Rancher nodes, I can no longer curl or telnet to my Rancher cluster DNS (myranchercluster.domain.com). But, when I stop Docker on one of the Rancher nodes curl and telnet atempts to my Rancher DNS work. This issue is causing the cluster and node agent pods to constantly restart.

ERROR: https://myranchercluster.domain.com/ping is not accessible (Failed to connect to myranchercluster.domain.com port 443: Connection timed out)

I upgraded both my Rancher clusters the same exact way - but only one cluster has this issue. It doesn't make sense what changes when I stop Docker on one Rancher node and then I can successfully curl and telnet to myranchercluster.domain.com:443.

We resolve this issue by disabling session persistence/sticky sessions in the HA Rancher cluster's Azure load balancer.

this is mine (fix)
rancher  server :     <IP_ADDRESS>   rancher.t.org

Edit ( button)

click save  button
then    return the       Workload: cattle-cluster-agent

Success

In my case i was able to solve it by removing any loopback IP (<IP_ADDRESS>/8) in the resolv.conf file (which is stated in rancher dns doc https://rancher.com/docs/rancher/v2.x/en/troubleshooting/dns/). on my ubuntu 18 machine i did it by disabling and stopping the systemd-resolved and rebooting the machine.

I am facing the same problem and stuck for 2 days. I have tried everything suggested in this thread so far but nothing seems to work.
I have installed Rancher server (v2.5.2) on a 2 node K3S cluster. I am trying to create a custom cluster with all 3 components (etcd, control plane and worker) on a single VM.
The k8s_cluster-register_cattle-cluster-agent container is exiting in a loop as seen below:
~]$  docker container ls -a | grep register
e72a790daccc        444dfdf0ae3c                           "run.sh"                 About a minute ago   Exited (1) About a minute ago                                    k8s_cluster-register_cattle-cluster-agent-6c798f5596-x6b7w_cattle-system_8a2cceb2-70cd-4b39-9e5f-3d7446a5d54b_50

$ docker logs e72a790daccc
INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM=dddba4a71ddb84ea9ad5c88e2fd38a7df0e547676bc286a5303911040470f342 CATTLE_CLUSTER=true CATTLE_FEATURES= CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=true CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-6c798f5596-x6b7w CATTLE_SERVER=https://rancher.mydomain.com
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
ERROR: https://rancher.mydomain.com/ping is not accessible (Could not resolve host: rancher.mydomain.com)

DNS resolution is working fine from the host machine. I even tried the troubleshooting steps mentioned here: https://rancher.com/docs/rancher/v2.x/en/troubleshooting/dns/
But I can only try this from the K3S cluster. Since the custom cluster is not up, I cannot execute kubectl command on it.
~]$ kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup rancher.mydomain.com
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS> kube-dns.kube-system.svc.cluster.local

Name:      rancher.mydomain.com
Address 1: <IP_ADDRESS>
Address 2: <IP_ADDRESS>
pod "busybox" deleted

I have added DNS server name in /etc/resolv.conf on both, the Rancher server and downstream host.
From the Rancher UI, for Workload: rancher, Namespace: Cattle-system, under Networking, I have tried configuring Host alias, DNS server, dnsPolicy: Default, but nothing works.
Please let me know how I can investigate this further? Is there a way that I can run the k8s_cluster-register_cattle-cluster-agent container manually, so that I can troubleshoot the DNS failure reason.
Thanks.

I observed that 5 containers inside /var/lib/docker/containers/ folder had their DNS server name as <IP_ADDRESS>.
I manually changed that to the correct one. Now the k8s_cluster-register_cattle-cluster-agent is going a few steps further and existing because of this error:
INFO: rancher.mydomain.com resolves to <IP_ADDRESS> <IP_ADDRESS>
INFO: Value from https://rancher.mydomain.com/v3/settings/cacerts is an x509 certificate
time="2020-12-02T09:46:30Z" level=info msg="Rancher agent version v2.5.2 is starting"
time="2020-12-02T09:46:30Z" level=info msg="Listening on /tmp/log.sock"
time="2020-12-02T09:46:30Z" level=info msg="Connecting to wss://rancher.mydomain.com/v3/connect/register with token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
time="2020-12-02T09:46:30Z" level=info msg="Connecting to proxy" url="wss://rancher.mydomain.com/v3/connect/register"
time="2020-12-02T09:46:30Z" level=info msg="Starting user controllers"
time="2020-12-02T09:47:00Z" level=fatal msg="Get \"https://<IP_ADDRESS>:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions\": dial tcp <IP_ADDRESS>:443: i/o timeout"

I managed to solve the problem by doing the following procedure.
systemctl disable firewalld
systemctl stop firewalld
iptables -P FORWARD ACCEPT
modprobe br_netfilter
sysctl net.ipv4.ip_forward
sysctl net.bridge.bridge-nf-call-iptables=1
sysctl --system
sudo iptables -P FORWARD ACCEPT
set paramater : vi /etc/sysctl.d/50-docker-forward.conf
**
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
**
for mod in ip_tables ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr; do sudo modprobe $mod; echo $mod | sudo tee -a /etc/modules-load.d/iptables.conf; done
sudo systemctl enable network
sudo systemctl disable NetworkManager
reboot;

Same problem here  Imported EKS cluster:
Rancher 2.4.12
Ubuntu 20
Docker 19.03.8
Log for cattle-cluster-agent
INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_FEATURES= CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-754b8c6c8d-gq8wb CATTLE_SERVER=https://rch2.vetpro.us
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local vetpro.us options ndots:5
ERROR: https://rch2.vetpro.us/ping is not accessible (Could not resolve host: rch2.vetpro.us)

Same problem here  Imported EKS cluster:
Rancher 2.4.12
Ubuntu 20
Docker 19.03.8
Log for cattle-cluster-agent
INFO: Environment: CATTLE_ADDRESS=<IP_ADDRESS> CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_FEATURES= CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-754b8c6c8d-gq8wb CATTLE_SERVER=https://rch2.vetpro.us
INFO: Using resolv.conf: nameserver <IP_ADDRESS> search cattle-system.svc.cluster.local svc.cluster.local cluster.local vetpro.us options ndots:5
ERROR: https://rch2.vetpro.us/ping is not accessible (Could not resolve host: rch2.vetpro.us)

I got the same problem.
Rancher 2.5.5 on K3S
Docker 20.10
Kubernetes 1.19.7
Ubuntu 20 on each node
User cluster provided by RKE via RancherUI. Everyhing is fine except of the mentioned error message.

以下转自这个 issues
#18832 (comment)
Adding host alias to the cattle agent can be done in this way:

kubectl -n cattle-system patch  deployments cattle-cluster-agent --patch '{
    "spec": {
        "template": {
            "spec": {
                "hostAliases": [
                    {
                      "hostnames":
                      [
                        "{{ rancher_server_hostname }}"
                      ],
                      "ip": "{{ rancher_server_ip }}"
                    }
                ]
            }
        }
    }
}'

kubectl -n cattle-system patch  daemonsets cattle-node-agent --patch '{
 "spec": {
     "template": {
         "spec": {
             "hostAliases": [
                 {
                    "hostnames":
                      [
                        "{{ rancher_server_hostname }}"
                      ],
                    "ip": "{{ rancher_server_ip }}"
                 }
             ]
         }
     }
 }
}'

I almost gave up, this saved me as well, thanks!

I run into the same issues: on one machine it always worked, on another, similar machine, it never worked.
After "diffing" the two machines, I found out that on the not working IP forwarding was disabled for the eth0 and lo network device, while it was enabled for all interfaces on the working machine.
Enabling IP forwarding solved the problem for me.
After updating from docker 20.10.2 to docker 20.10.3, IP forwarding got enabled on all interfaces on both machines when starting docker. Not sure where the main problem was, if this was a docker bug or anything else, and why this only happened on the one machine and never on the other.
With this setup it works for me:

sysctl -a |grep \.forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.lo.forwarding = 1

#18832 (comment)
I just want to confirm that adding host aliases worked for me as well.

Same problem, 7 clusters all set up in the same way, but 1 cluster get lookup errors. Solved this by editing the deployment and changing the DNS policy from ClusterFirst to Default:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
dnsPolicy: Default

I got the same error where it could not resolve the host. However in my situation it happens during cluster creation, so I cannot use kubectl since the cluster is not ready yet. What can I do when kubectl isnt available yet to fix this issue?

I have found a way to apply the 'hack' without the use of kubectl.

ssh to your masters, and find the one that is attempting to run k8s_cluster-register_cattle-cluster-agent-******
use docker ps -a
docker container inspect  and look for this segment

      "Mounts": [
           .....
           {
               "Type": "bind",
               **"Source": "/var/lib/kubelet/pods/f179aff5-b149-4cbc-8d0f-09f1ca0d54e7/etc-hosts",**
               "Destination": "/etc/hosts",
               "Mode": "Z",
               "RW": true,
               "Propagation": "rprivate"
           },
    .....

open that source file with a text editor, and add the host entry for your rancher.example.com

restart the container
At this point, your cluster should continue putting itself together.

But this does not address the underlying issue regarding coredns. What I am currently testing out is adding a config to coredns which declares our dns server in it. And it seems to pick up the slack for that.
But some internal communications remain broken. For instance, for a redis deployment redis replica can not connect to redis master over ClusterIP. The cause for my woes is attempting to get iptables+firewalld to behave with k8s 1.19 and centos 8.3. Your issue may be different.
TLDR; the 'hack' only addresses a side effect of the root cause, not the actual issue itself, unfortunately :'(

I am having this same issue when registering a new custom cluster on aws ec2. Is there any way to solve this?

I resolved my issue. It was a problem with my firewall configuration.

I hit this issue sometimes with Rancher HA setups on AWS. Usually downstream clusters or even the local ha agent fails to resolve the name. Security policy is open-all for both incoming and outgoing network, No firewall on ubuntu image used for the RKE cluster setup.
In my case I don't think is a firewall issue.
If I change the DNS server in the cattle-cluster-agent to <IP_ADDRESS> i usually get the thing fixed.

This issue https://github.com/k3s-io/k3s/issues/535 seems related and the fix in this comment worked for me.

same problem exactly , using rancher 2.5.7 and 2.5.8 !

I hit this issue sometimes with Rancher HA setups on AWS. Usually downstream clusters or even the local ha agent fails to resolve the name. Security policy is open for both incoming and outgoing network, No firewall on ubuntu image used for

exactly this setup on AWS , same issue and same errors

Hi all,
I had faced the same issue and I have resolved it by, going to the namespace "cattle-system" in the Kubernetes cluster you provisioned, edit the deployment "cattle-cluster-agent" and click on show advanced options, and expand networking, and under "Host Aliases ( /etc/hosts entries ) maintain the IP address ( load balancer IP or any node IP of your rke cluster on which rancher server is deployed ) and DNS name of your rancher server endpoint.
The "cattle-cluster-agent" will change status from "crashloopbackoff" to running status.

When I deployed a port with nodeport , nodeport wasn't generated as the "cattle-cluster-agent" wasn't in running status. As soon as I made above change, the nodeport was immediately visible under "netstat -tnlup"
Attached couple of screenshots that may help you.

I'm also having this issue.
Rancher Server Setup

Rancher version: Rancher version 2.6.0
Installation option (Docker install/Helm Chart):

Rancher Server 2.6.0, rke1.

Proxy/Cert Details: Self signed

Information about the Cluster

Kubernetes version: kubernetes version v1.20.11-rancher1-2
Cluster Type (Local/Downstream):

Downstream, Custom via docker on new Ubuntu 20.04 QEMU KVM vm nodes.

Describe the bug
Cattle cluster agent fails to resolve rancher server dns hostname.
To Reproduce

Install rancher server using a dns hostname that is resolvable on both public facing and internal facing dns servers in a split horizon configuration.
Setup nodes into a cluster.
Attempt to access node explorer.

Result
This causes the explorer to not be available for the cluster on Rancher server.
Expected Result
I should be able to access the explorer.
Additional context
All 3 of my masters and all 3 of my workers can resolve the domain just fine. CoreDNS pods just show some io timeout errors:
linux/amd64, go1.15.3, 054c9ae
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0924 16:47:56.859902       1 trace.go:205] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (24-Sep-2021 16:47:26.859) (total time: 30000ms):
Trace[1427131847]: [30.000324176s] [30.000324176s] END
I0924 16:47:56.859923       1 trace.go:205] Trace[911902081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (24-Sep-2021 16:47:26.859) (total time: 30000ms):
Trace[911902081]: [30.000158123s] [30.000158123s] END
E0924 16:47:56.859925       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://<IP_ADDRESS>:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp <IP_ADDRESS>:443: i/o timeout
E0924 16:47:56.859933       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://<IP_ADDRESS>:443/api/v1/services?limit=500&resourceVersion=0": dial tcp <IP_ADDRESS>:443: i/o timeout
I0924 16:47:56.859937       1 trace.go:205] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (24-Sep-2021 16:47:26.859) (total time: 30000ms):
Trace[939984059]: [30.000156648s] [30.000156648s] END
E0924 16:47:56.859941       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://<IP_ADDRESS>:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp <IP_ADDRESS>:443: i/o timeout

Does Coredns forward it's dns requests to the DNS servers that the host has configured? It would appear so judging by the logs output, but it's not behaving that way.
I don't have any wildcards involved in my DNS setup yet, however I am doing split horizon DNS, where rancher.mydomain.com resolves to the public or private facing IP depending on what side of the network you're on. It resolve correctly when tested with nslookup and curl from the host:
ubuntu@k8s-master01:~$ curl -k https://rancher.mydomain.com/ping
pong

I worked around this issue by editing the cattle-cluster-agent deployment and adding a hostalias for rancher.mydomain.com to the pods spec.  EDIT:  I just worked around this issue by editing the cattle-cluster-agent deployment and adding a hostalias for rancher.mydomain.com to the pods spec. Adding entries to Pod /etc/hosts with HostAliases | Kubernetes, however I'm not convinced that this should be an accepted fix for an issue like this.

been 3 years and issue still surface once in a while, depending on DNS setup (CNAME or A records, *. prefix etc) - fact is something in cattle-agent code does not resolve DNS properly, while at the same time other containers on the same node resolves it fine

more details:
inside cattle agent i have changed 'k8s internal coredns ip to the ip of the working nameserver and it resolves the issue
nameserver <IP_ADDRESS>
#nameserver <IP_ADDRESS>
search cattle-system.svc.cluster.local svc.cluster.local cluster.local hoy5k2ydjtsuzalmvghk3dbabe.bx.internal.cloudapp.net
so cattle-agent needs some way to resolve rancher mgmt url and with the default assigned internal coreDNS it can not !

My master node is also running on Ubuntu 18.04. By running systemctl disable systemd-resolved and rebooting the node, the registration works.

Running on k8s (AKS).  This worked for me.   Changed CATTLE_SERVER  Key Value Pair under General Tab.
From https://localhost to https://rancher.cattle-system

Same problem, 7 clusters all set up in the same way, but 1 cluster get lookup errors. Solved this by editing the deployment and changing the DNS policy from ClusterFirst to Default:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
dnsPolicy: Default

Thank you Bro
That's really worked

I'm still seeing this problem randomly on Ubuntu 20.04. I know how to fix it using hostAliases or dnsPolicy but I rather stay away, we're 100% IaC. Are we any closer to a permanent fix?

Hi Folks,
We just encountered this issue and I believe I've identified the source of the issue. We started by creating a Custom RKE cluster on bare-metal, architecture for this cluster includes 3 masters and 3 workers.
We started by deploying a single master and worker to get the ball rolling and form the initial cluster. That is when we encountered the issue with the daemonset failing to come up.
2 pods were being created, 1 on the master, and 1 on the worker, and both were failing to resolve our rancher hosted DNS name.
As we've seen in other answers, deleting the pod on the worker node solved that particular issue, however we couldn't get the other pod to come up.
That's when it hit me - why is the daemonset trying to schedule the cattle-cluster-agent on a control plane node?! That shouldn't even be happening, as the control plane node should be unschedulable. It would seem there is an issue where a minimum of 2 cluster agents are started, some race condition, which tells it to start on the control plane, even though it shouldn't.
I hypothesised that if I deployed a second worker node, that failing pod would move over to the worker, and name resolution would resume working like normal (even though name resolution on the control plane node had no problem hitting the ping endpoint of our hosted rancher).
So, I deployed a second worker node, deleted the failing cattle-cluster-agent, and it came up without any new issues on the new worker, just like a daemonset should, and that allowed everything to come up as normal.
This was a weird one! Hope this helps.

Also happens for me, all the time. Until now, every single attempt to import a Kubernete cluster into rancher or creating a new cluster from rancher itself through docker, fails with this issue. I'm using Rocky Linux 8.5

Solved this by editing the deployment,setting hostNetwork to true and changing the DNS policy from ClusterFirst to ClusterFirstWithHostNet:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
...

Solved this by editing the deployment,setting hostNetwork to true and changing the DNS policy from ClusterFirst to ClusterFirstWithHostNet:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
hostNetwork: true dnsPolicy: ClusterFirstWithHostNet
...

Thanks! That solved my issue as well 👍

Having the same problem, and every time I create a new cluster from Rancher. I install rancher in another cluster with the Helm chart.
Is there any way to override the dns settings used in the cluster agents that are deployed, BEFORE they get deployed?  The newly created cluster never gets to a state where I can dig out a kubeconfig and modify the cluster-agent deployment.  I tried with everything I could find in /etc/kubernetes/ssl, but none of those had the rigts to edit the deployments.  I guess the next sted would be to find and modify the deployment in etcd on the newly created cluster, right?

Okay .... before the pod restarted, I found the underlying docker container for the cluster-agent, and it had mapped the hosts file ... yeay ...
"Source": "/var/lib/kubelet/pods/4894360c-ffcb-4b18-a2c0-c16b1b83b96a/etc-hosts",
"Destination": "/etc/hosts",
In with the ip address of my rancher server and the cluster got registered in Rancher. Then I could fetch the kubeconfig and go and fix the deployment for the cluster-agent.
So ... ops guys just get to sit around drink coffee and watch kubernetes do all the hard work?  Riiiiight.... 😂

Same problem, 7 clusters all set up in the same way, but 1 cluster get lookup errors. Solved this by editing the deployment and changing the DNS policy from ClusterFirst to Default:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
dnsPolicy: Default

Thank you, this was the only way I could get my cluster back online after upgrading to 22.04

I'm also affected by this issue in multi-node clusters, but can't edit anything with kubectl as Rancher (2.6.5) doesn't let me download the kubeconfig.  Im stuck in a "Waiting for API to be available" cluster state.
It's as if the cluster-agent container can't reach the service CIDR of the Cluster (<IP_ADDRESS>/16), in which CoreDNS also has it's endpoint.
Could this be a node-to-node communication issue? And the CoreDNS problem just a symptom?

Could this be related to the CNI used in the downstream clusters ?
I'm running Rancher 2.7.0 with Docker Compose :
version: '3.8'

services:
  rancher:
    privileged: true
    container_name: rancher
    image: rancher/rancher
    ports:
      - "80:80"
      - "443:443"
    environment:
      - TZ=Europe/Paris
    volumes:
      - rancher_cni:/var/lib/cni
      - rancher_kubelet:/var/lib/kubelet
      - rancher_data:/var/lib/rancher
      - rancher_logs:/var/log
    restart: unless-stopped

volumes:
  rancher_cni:
  rancher_kubelet:
  rancher_data:
  rancher_logs:

My test environment is based on K3s clusters :

Flannel (K3s'default) CNI

# curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable INSTALL_K3S_VERSION=v1.24.7+k3s1  K3S_TOKEN="*************" sh -s - server --cluster-init

No issue when importing the cluster into Rancher.

Antrea (1.9.0) CNI

# curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable INSTALL_K3S_VERSION=v1.24.7+k3s1  K3S_TOKEN="*************" sh -s - server --cluster-init --flannel-backend=none --disable-network-policy
# repo add antrea https://charts.antrea.io
# helm repo update antrea
# helm install antrea antrea/antrea --namespace kube-system --version 1.9.0

No issue when importing the cluster into Rancher.

Cilum (1.12.4) CNI

# curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable INSTALL_K3S_VERSION=v1.24.7+k3s1  K3S_TOKEN="*************" sh -s - server --cluster-init --flannel-backend=none --disable-network-policy
# helm repo add cilium https://helm.cilium.io
# helm repo update cilium
# helm install cilium cilium/cilium --set operator.replicas=1

No issue when importing the cluster into Rancher.

Calico (3.24.5) CNI

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable INSTALL_K3S_VERSION=v1.24.7+k3s1  K3S_TOKEN="*************" sh -s - server --cluster-init --flannel-backend=none --disable-network-policy
helm repo add projectcalico https://projectcalico.docs.tigera.io/charts
helm repo update projectcalico
kubectl create namespace tigera-operator
helm install calico projectcalico/tigera-operator --version v3.24.5 --namespace tigera-operator

When importing the cluster into Rancher :

```bash
INFO: Using resolv.conf: search home.lab nameserver <IP_ADDRESS>
ERROR: https://rancher.foo.com/ping is not accessible (Could not resolve host: rancher.foo.com)

Resolved by modifying the cattle-cluster-agent deployment :
dnsPolicy: Default
hostNetwork: true
``

Solved this by editing the deployment,setting hostNetwork to true and changing the DNS policy from ClusterFirst to ClusterFirstWithHostNet:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
hostNetwork: true dnsPolicy: ClusterFirstWithHostNet
...

works for me:
dnsPolicy: Default
hostNetwork: true
Thanks !!

Hi folks,
I have faced same issue and I just solved after patched as follows:
kubectl -n cattle-system patch deployments cattle-cluster-agent --patch '{"spec": {"template": {"spec": {"hostAliases": [{"hostnames":[""],"ip": ""}]}}}}'
Hope that it works for you as well.
Cheers.
Bruno Almeida

Faced the same issue when installing RKE2 on vSphere
ESXi and vSphere version: 6.7u2
Template OS: openSUSE Leap 15.4
Preparation steps: [https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template](Listed in the documentation)
Can this be solved without manually logging into the nodes and changing deployment?

Restart all worker and master nodes one by one then finally restart Rancher installed VM helped to resolve issue for us. It's due to network calico update caused for us.

This worker for me as well. Thanks. If possible could you clarify the mechanism that this changes made it work

Not to dig up an old thread... but in our case recently our DNS changed at some point after the cluster was deployed, but it didn't cause a problem until we updated one of the RKE nodes.
All we had to do was bounce the coredns pods on the affected cluster and everything came up, as they were holding on to the old nameservers

Solved this by editing the deployment,setting hostNetwork to true and changing the DNS policy from ClusterFirst to ClusterFirstWithHostNet:
kubectl edit deployment cattle-cluster-agent -n cattle-system
...
hostNetwork: true dnsPolicy: ClusterFirstWithHostNet
...

it resolve issue
dnsPolicy: Default
hostNetwork: true

This works for me:
$ cat cattle-cluster-agent.yaml
spec:
template:
spec:
dnsPolicy: Default
$ kubectl patch deployment -n cattle-system cattle-cluster-agent --patch-file cattle-cluster-agent.yaml
Still I think this problem should be fixed soon at the cattle-cluster-agent level, we already had it on many clusters and obviously many others do have it, too.

Based on trying out several of the methods provided above to solve the problem, the following method worked for me.
Its an issue with the cattle-cluster-agent dns resolution by coredns at <IP_ADDRESS> and <IP_ADDRESS>, this core dns does not have a resolution mechanism for the external ip and hostnames like rancher.cluster.org
to resolve the problem:
1>add the hosts to the \etc\hosts
<IP_ADDRESS> rancher.cluster.org
<IP_ADDRESS> rancher.cluster.org
<IP_ADDRESS> rancher.cluster.org
2> Update the Cluster Agent Deployment, screenshot based on rancher ui
Set:
Network Mode - Host Network
DNS Policy - Cluster First with Host Network

By having the DNS policy as ClusterFirst, we can ensure the internal pod communication happens by the clusterFirst policy, and if an ip does not resolve it would check the host network
after doing the above the cattle-cluster-agent is able to resolve the address for the rancher.cluster.org

Hopefully, this helps someone, but I found that the "Could not Resolve Host" issue could be potentially related to https://github.com/k3s-io/k3s/issues/5013#issuecomment-1026061183.
In my case, I have a cluster where I need to apply the fix above.  After setting up the cluster,  I was doing a series of etcd snapshot rollbacks because I was troubleshooting a separate issue that eventually lead me to this current thread with the "Could not Resolve Host" issue in the cattle-cluster-agent.
It had turned out that using an etcd snapshot, reverts the tx-checksum-ip-generic off change applied (even if the snapshot was created with tx-checksum-ip-generic off ).
I suggest double-checking the machine to ensure tx-checksum-ip-generic is off, if you have applied the fix before and you are using a RHEL 8.X machine.


Possible regression in version 9.9.2: [ 'Invalid purl pkg:generic/:opencv' ]

Hi there, thank you for this nice tool! We've been happily using it for a while, but now with version 9.9.2, it suddenly started erroring out our builds.
How to reproduce:
sudo npm install -g<EMAIL_ADDRESS>git clone https://github.com/opencv/opencv.git
cd opencv
cdxgen -o sbom.json

At least on our machines, the above prints out an error [ 'Invalid purl pkg:generic/:opencv' ] and returns an exit code 1.
This started with version 9.9.2, but seems to be occurring with the latest 9.9.3 as well.
Workaround: use version 9.9.1 instead.
Alternative workaround: just ignore the exit code – the SBOM itself seems to be ok (at least superficially). But surely it's not nice to ignore error codes, right?

@reunanen, Thank you for the kind words. Could you attach the sbom? I can guide you to sending a PR.

Opened a PR (see #708), although not really sure why exactly you wanted me to do it. Didn't update version numbers or anything, at least just yet.

Thank you! Line 126 in this project is referring to a project with a colon prefix. colon is widely used with gradle, so we take care of this in java. Now with your PR, we don't crash with these for c.
cmake/android/android_gradle_projects.cmake
116:include ':opencv'
125:include ':opencv'
126:project(':opencv').projectDir = new File(opencvsdk + '/sdk')


 Use http://autoclaim.win instead. (If the claim URL changes and you visit this page directly, you might be mistaken for a bot and banned.)

Pada tanggal 14 Feb 2018 8.19 PM, "newplayer191"<EMAIL_ADDRESS>menulis:
Closed #5 https://github.com/dwisiswant0/Faucet-DOGE-Bot/issues/5.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dwisiswant0/Faucet-DOGE-Bot/issues/5#event-1473762953,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYo_5HC7-mu8Mq8QY5pufpmY-QOTNk0Nks5tUt1dgaJpZM4SFTF9
.
Wtf


[macOS/Safari] mat-input value gets highlighted when you double click floating label

What is the expected behavior?
Double clicking a label doesn't do anything. Just like in other browsers.
What is the current behavior?
Double click highlights value inside mat-input.
What are the steps to reproduce?
Open on Safari https://material.angular.io/components/input/overview double click label.
https://streamable.com/rtwvw
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Browsers: Safari( i was able to reproduce on newest version)
Is there anything else we should know?

clicking the label focuses the input field, e.g. in Chrome, which is what I would expect.

@macjohnny this is inconsistent behavior as in every browser except safari clicking label doesn't select text inside input.

if you previously selected the text, clicking the label will select the text again in Chrome.
However, I believe this is a general issue concerning the focus behavior of safari, although I cant find the issue that discussed a similar case.

This is how native input labels work as well (see https://stackblitz.com/edit/angular-2jgbpy). Closing since this isn't something caused by Material.


refactor(sls): convert config to TS

Closes #799

Yeah this doesn't seem to work - I only noticed now that we still use webpack to transpile serverless handlers (and serverless.js file), albeit only the js files, and create a bundle.
In that case we still need to update the tsconfig file, maybe just for the serverless deployment run (as in - attach a tsconfig file that overrides the default one), which is what i am going to do.
Since we need to transpile the link-plugins file i see little point of doing it with babel-node only then to run serverless with webpack config that still uses the tsconfig file. In fact i am now leaning towards pointing the serverless handlers to their ts versions, using ts-node to run the link-plugins transpilation and then ditching babel-node for asap-server entirely.
Opinions?

Piotr feel free to close one of the PRs, not sure which one you were gonna use


Will this work with client credentials out of the box?

Client creds mean you get a token with no subject.
Is that supported? Why can't we look at the client_id claim and treat that as a subject?
Would a PR be accepted?

You can pass in whatever claims you like to the Evaluate method. But client id and sub have different semantics.
We are not planning to make any changes to this repo - this is sample. If you wish to change it, feel free to do that in your own fork.

Before I go and make my own changes, what semantics between sub and client_id do you mean?

They mean different things. That's all.


campaign version stage with not existed provider can still be created

Related issue: #192
create a campaign with following specs and it can still get successful return.
apiVersion: workflow.symphony/v1
kind: CampaignContainer
metadata:
  name: campaign06
spec:
---
apiVersion: workflow.symphony/v1
kind: Campaign
metadata:
  name: campaign06-v-v1
spec:
  rootResource: campaign06
  firstStage: first
  stages:
    first:
      name: first
      provider: providers.stage.mock
      stageSelector: second
    second:
      name: second
      provider: providers.stage.mock
      stageSelector: third
    third:
      name: third
      provider: providers.stage.notExist
  selfDriving: true

If we use an activation to trigger it,
apiVersion: v1
items:
- apiVersion: workflow.symphony/v1
  kind: Activation
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"workflow.symphony/v1","kind":"Activation","metadata":{"annotations":{},"name":"workflow","namespace":"default"},"spec":{"campaign":"campaign06:v1","stage":"first"}}
    creationTimestamp: "2024-08-14T08:27:44Z"
    generation: 1
    name: workflow
    namespace: default
    resourceVersion: "1181"
    uid: d88d623d-e0ba-4d66-a483-7cbc7c65f033
  spec:
    campaign: campaign06:v1
    stage: first
  status:
    stageHistory:
    - nextStage: second
      stage: first
      status: 9996
      statusMessage: Done
    - nextStage: third
      stage: second
      status: 9996
      statusMessage: Done
    - errorMessage: provider providers.stage.notExist is not found
      stage: third
      status: 400
      statusMessage: Bad Request
    status: 9994
    statusMessage: Running
    updateTime: "2024-08-14T08:27:54Z"
kind: List
metadata:
  resourceVersion: ""

We will get an error message but the activation will be stuck in "Running"

Fix in webhook changes which is working in @RemindD .


fix(swagger): use getAbsolutePath from swagger-ui-dist/get-absolute-path.js

Closes: #2695
Information

Type
Breaking change

Fix
No

Todos

[x] Tests
[ ] Coverage
[ ] Example
[ ] Documentation

:tada: This PR is included in version 7.68.6 :tada:
The release is available on:

v7.68.6
GitHub release

Your semantic-release bot :package::rocket:


Wikivoyage by Kiwix 2.5.2 UWP

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/110980&drop=dogfoodAlpha

[Policy] feedback reset

I get confused when github.com hide its setup installer.

@BrandonWanHuanSheng Thank you. Is there anything I need to change? Yes, some assets are hidden until you press "Show all 15 assets". It seems the PR is just waiting for mod approval now?

@BrandonWanHuanSheng Thank you. Is there anything I need to change? Yes, some assets are hidden until you press "Show all 15 assets". It seems the PR is just waiting for mod approval now?

No, need.


device called "UE - LiveLink" not shown motionbuilder2022 + UE4.27

try prebuild version, device not shown,
I guess someting missing, just like runtime or some win sdk. so I build mobulink by hand. （ motionbuilder2022 + UE4.27 ）and get MobuLiveLinkPlugin2022.dll + .exp + .lib + .pdb + .target
then I put path E:\UnrealEngine-4.27\Engine\Binaries\Win64\MotionBuilder\2022 into MB200 sdk path,
device called "UE - LiveLink" still not shown.
I have no idea now, really need help

I aslo check folder C:\Users\Hadis\Documents\MB,  there is 2022 folder, although I create 2022-64, device still not come to me

please close this pr， fixed by using unity3D. unity3D is much more stable than unreal.

I got the same problem, mine was solved by copying these files to
C:\Program Files\Autodesk\MotionBuilder 2022\bin\x64\plugins
- MobuLiveLinkPlugin2022.dll
- UE_128.png
- UE_128-l.png


add ec2-metadata tags support.

Description of changes:
implement for this release : Instance Tags now available on the Amazon EC2 Instance Metadata Service

Thank you for this contribution! It's a been a while since we added any new features to the ec2-metadata tool.
Please confirm that you are willing to license your changes under the same license (The MIT License) as the existing code.


Installer.bat and README Upgrade: Uninstaller Guide, and Apple Music Upgraded

Upgraded the Installer.bat to work on any directory and eliminated error messages if programs like "gameservices.exe" were not present.
Upgraded the README, fixed typos. added new screenshots, uninstaller guide, and added instructions on how to upgrade Apple Music.
Upgraded the included Apple Music.

Thank you! Merged <3


env_file parsing fails with spaces or quotes in value

given my env_file:
GIT_COMMIT_MESSAGE="move .version files into app dir"

results in:
/dev/fd/63: line 1: export: `.version': not a valid identifier

seems to fail due to spaces in the value or quotes
if I remove the quotes
GIT_COMMIT_MESSAGE=move .version files into app dir

then it errors as
parse error: Expected separator between values at line 16, column 15

nevermind -- apparently the quotes WERE breaking it, and the error when removing the quotes was due to the payload


add bugsnag logrus hook

Along with configuring a bugsnag http Handler, configure a logrus -> bugsnag hook. This will ensure that any Error log messages are also sent to bugsnag.
Signed-off-by: Matt Tescher<EMAIL_ADDRESS>

@dmcgowan ^

LGTM