Error: failed to start containers: kubelet

Rancher versions:
rancher/server or rancher/rancher: rancher/rancher:latest@sha256:38839bb19bdcac084a413a4edce7efb97ab99b6d896bda2f433dfacfd27f8770
rancher/agent or rancher/rancher-agent:
rancher/rancher-agent:v2.0.0
Infrastructure Stack versions:
whatever the defaults are
Docker version: (docker version,docker info preferred)
Client:
Version:      17.03.2-ce
API version:  1.27
Go version:   go1.8.3
Git commit:   f5ec1e2-snap-345b814
Built:        Thu Jun 29 23:40:29 2017
OS/Arch:      linux/amd64
Server:
Version:      17.03.2-ce
API version:  1.27 (minimum version 1.12)
Go version:   go1.8.3
Git commit:   f5ec1e2-snap-345b814
Built:        Thu Jun 29 23:40:29 2017
OS/Arch:      linux/amd64
Experimental: false
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
4.15.0-20-generic
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
ssdnodes/custom cluster
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
can't set up node
Steps to Reproduce:
snap install docker --channel=17.03/stable
mkdir /etc/kubernetes
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.0.0 --server https://rancher.sixcorners.info --token abc --ca-checksum xyz --worker
docker logs -f share-mnt
Results:
Found state.json: 931882e24ff0ef67b0e8744dbf1f7e04fd68afe714a29a2522293312824f3c51
time="2018-05-06T06:09:15Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/21787/ns/mnt -F -- /var/snap/docker/common/var-lib-docker/aufs/mnt/5d00bd40adec6662aaec8ea2a5f5ce6a332e9dbfad087a008c5c89b7cac4c22f/usr/bin/share-mnt --stage2 /var/lib/kubelet /var/lib/rancher -- norun]"
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

I got this when adding a new node. I started the agent container as per the command in the UI and it seems to have started a  second agent container named share-mnt. At the time there were only two containers running, the original agent container and this one. After a while I did docker rm -f on the share-mnt container and that seems to have cleared it up.

What is the output of docker ps -a when this happens? And docker logs --tail=all kubelet?

Here are some logs from my terminal:
jonkel@rancher2:~$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
7378e65f3edc        rancher/rancher-agent:v2.0.0   "run.sh -- share-r..."   5 minutes ago       Up 2 minutes                            share-mnt
117a4debc918        rancher/rancher-agent:v2.0.0   "run.sh --server h..."   5 minutes ago       Up 5 minutes                            tender_fermi
jonkel@rancher2:~$ docker logs 73
Found container ID: 1365e16b71514e04bcc4d4553f757f5684b55deeaa38523d71fda23d41ee678e
Checking root: /host/run/runc
Checking file: 117a4debc9180552ac578ab8e7145c4ceb9b384981c70e23ebb03a8251ff2909
Checking file: 1365e16b71514e04bcc4d4553f757f5684b55deeaa38523d71fda23d41ee678e
Found state.json: 1365e16b71514e04bcc4d4553f757f5684b55deeaa38523d71fda23d41ee678e
time="2018-05-06T20:48:25Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/10275/ns/mnt -F -- /var/lib/docker/aufs/mnt/25409162e1b2f1c3c0cbe62719f52bf6710bd4e97fe49ca612db6711d11a4a31/usr/bin/share-mnt --stage2 /var/lib/kubelet /var/lib/rancher -- norun]"
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}

There was no container named kubelet started at this time.

Ok and docker logs 117a4debc918 ?

I'm afraid it is too late to do that since that container is gone. Here is some more logs from what I did next when I removed the stuck container:
jonkel@rancher2:~$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
7378e65f3edc        rancher/rancher-agent:v2.0.0   "run.sh -- share-r..."   8 minutes ago       Up 5 minutes                            share-mnt
117a4debc918        rancher/rancher-agent:v2.0.0   "run.sh --server h..."   8 minutes ago       Up 8 minutes                            tender_fermi
jonkel@rancher2:~$ docker rm -f 7378e65f3edc
7378e65f3edc
jonkel@rancher2:~$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
50db7bace022        rancher/hyperkube:v1.10.1-rancher2   "/opt/rke/entrypoi..."   10 seconds ago      Up 1 second                             kubelet
c6e4a2980730        rancher/hyperkube:v1.10.1-rancher2   "/opt/rke/entrypoi..."   21 seconds ago      Up 11 seconds                           kube-proxy
117a4debc918        rancher/rancher-agent:v2.0.0         "run.sh --server h..."   11 minutes ago      Up 11 minutes                           tender_fermi
jonkel@rancher2:~$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED              STATUS              PORTS               NAMES
ebec6195fec5        d1a7302844b3                         "sh -c 'sysctl -w ..."   12 seconds ago       Up 1 second                             k8s_sysctl_nginx-ingress-controller-q5cgn_ingress-nginx_173d5bb8-5170-11e8-836e-000c2923739a_0
5e79d4008621        rancher/pause-amd64:3.1              "/pause"                 34 seconds ago       Up 2 seconds                            k8s_POD_canal-hdllj_kube-system_173d714f-5170-11e8-836e-000c2923739a_0
06d5b729ecf8        rancher/pause-amd64:3.1              "/pause"                 34 seconds ago       Up 4 seconds                            k8s_POD_cattle-node-agent-ttjfl_cattle-system_18853f76-5170-11e8-836e-000c2923739a_0
e78308215400        rancher/pause-amd64:3.1              "/pause"                 34 seconds ago       Up 13 seconds                           k8s_POD_nginx-ingress-controller-q5cgn_ingress-nginx_173d5bb8-5170-11e8-836e-000c2923739a_0
85e376a3857d        rancher/rke-tools:v0.1.4             "nginx-proxy CP_HO..."   58 seconds ago       Up 53 seconds                           nginx-proxy
50db7bace022        rancher/hyperkube:v1.10.1-rancher2   "/opt/rke/entrypoi..."   About a minute ago   Up 59 seconds                           kubelet
c6e4a2980730        rancher/hyperkube:v1.10.1-rancher2   "/opt/rke/entrypoi..."   About a minute ago   Up About a minute                       kube-proxy
117a4debc918        rancher/rancher-agent:v2.0.0         "run.sh --server h..."   12 minutes ago       Up 12 minutes                           tender_fermi
jonkel@rancher2:~$ docker ps
CONTAINER ID        IMAGE                                                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
748f86982bd9        rancher/nginx-ingress-controller@sha256:58944b175505087dfa7afc444577d1933fd3bf1f1f668ef57b9eaaa8b36f59ce   "/usr/bin/dumb-ini..."   21 minutes ago      Up 21 minutes                           k8s_nginx-ingress-controller_nginx-ingress-controller-q5cgn_ingress-nginx_173d5bb8-5170-11e8-836e-000c2923739a_0
c63b7163f96b        rancher/coreos-flannel@sha256:93952a105b4576e8f09ab8c4e00483131b862c24180b0b7d342fb360bbe44f3d             "/opt/bin/flanneld..."      29 minutes ago      Up 29 minutes                           nginx-proxy
50db7bace022        rancher/hyperkube:v1.10.1-rancher2                                                                         "/opt/rke/entrypoi..."   29 minutes ago      Up 29 minutes                           kubelet
c6e4a2980730        rancher/hyperkube:v1.10.1-rancher2                                                                         "/opt/rke/entrypoi..."   29 minutes ago      Up 29 minutes                           kube-proxy

I got same error when adding second node to k8s cluster.
here's some logs.
xxx@localhost ~ $ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
7e3995f9612e        rancher/rancher-agent:v2.0.0   "run.sh -- share-roo…"   7 minutes ago       Up 7 minutes                            share-mnt
3db29cd16328        rancher/rancher-agent:v2.0.0   "run.sh --server htt…"   7 minutes ago       Up 7 minutes                            laughing_goldwasser

xxx@localhost ~ $ docker logs 7e
Found container ID: 86e121fbbbbcff73e36fa17ca51b69a823bc3a0fc6a1312eab0bf7f694eeebe8
Checking root: /host/run/runc
Checking root: /host/var/run/runc
Checking root: /host/run/docker/execdriver/native
Checking root: /host/var/run/docker/execdriver/native
Checking root: /host/run/docker/runtime-runc/moby
Checking file: 3db29cd163283ea4f5d145d71c741e91b039648a45ce2480f95b25338e505009
Checking file: 7e3995f9612e592c21922d91ec72c29ef7a883fcff73ce029cfd3453277b32a2
Checking file: 86e121fbbbbcff73e36fa17ca51b69a823bc3a0fc6a1312eab0bf7f694eeebe8
Found state.json: 86e121fbbbbcff73e36fa17ca51b69a823bc3a0fc6a1312eab0bf7f694eeebe8
time="2018-05-07T05:33:56Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/12825/ns/mnt -F -- /var/lib/docker/overlay2/2193cefa0869f186d06ef729c7765e059d01ec8a32a848f3ab34e15435fe5312/merged/usr/bin/share-mnt --stage2 /var/lib/kubelet /var/lib/rancher -- norun]"
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

okano@localhost ~ $ docker logs 3d
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
time="2018-05-07T05:33:55Z" level=info msg="Option customConfig=map[address:<node IP Address> internalAddress:<node IP Address> roles:[worker]]"
time="2018-05-07T05:33:55Z" level=info msg="Option etcd=false"
time="2018-05-07T05:33:55Z" level=info msg="Option controlPlane=false"
time="2018-05-07T05:33:55Z" level=info msg="Option worker=true"
time="2018-05-07T05:33:55Z" level=info msg="Option requestedHostname=localhost"
time="2018-05-07T05:33:55Z" level=info msg="Connecting to wss://<server IP Address>/v3/connect/register with token <access token>"
time="2018-05-07T05:33:55Z" level=info msg="Connecting to proxy" url="wss://<server IP Address>/v3/connect/register"
time="2018-05-07T05:33:56Z" level=info msg="Starting plan monitor"

I've never seen anything interesting in the initial rancher agent's logs.
time="2018-05-07T05:59:14Z" level=info msg="Option requestedHostname=sn"
time="2018-05-07T05:59:14Z" level=info msg="Option customConfig=map[address:abc internalAddress: roles:[etcd worker controlplane]]"
time="2018-05-07T05:59:14Z" level=info msg="Option etcd=true"
time="2018-05-07T05:59:14Z" level=info msg="Option controlPlane=true"
time="2018-05-07T05:59:14Z" level=info msg="Option worker=true"
time="2018-05-07T05:59:14Z" level=info msg="Connecting to wss://rancher.sixcorners.info/v3/connect/register with token xyz"
time="2018-05-07T05:59:14Z" level=info msg="Connecting to proxy" url="wss://rancher.sixcorners.info/v3/connect/register"
time="2018-05-07T05:59:15Z" level=info msg="Starting plan monitor"

Interestingly I also got this when adding my second node, but the third was added fine. Basically what I did was follow the quickstart to set everything up on a single VM, then add a second VM as worker node and got the error. Then added a third node as worker and it went well.

Hmm... I will try setup machine again(now using baremetal).
Or should I use VM to setup k8s cluster?

I'm not sure if bare metal or VM matters. I think I got similar errors when trying to deploy it on bare metal (my laptop)

Hi, I also had this error because my VMs had the same name. After changing the name and cleaning entire VM (volume prune, system prune etc.) everything went back to normal.

I also had the same issue. But it turned out my localhost was setup right in /etc/hosts. once I fixed that, it came up!

in my case, problem solved by changing name. thanks!

I get this error on any node in a cluster but the first one.

So it is currently not possible to run the rancher/server and the rancher/agent on the same host without additional VM overhead? Is there a fix planned or is this an uncommon / unsupported use case, because I also ran into this problem when I wanted to deploy the agent to the same baremetal host as the server.

I am getting the same error. I just tried to deploy a cluster. All the masters that I started have the 'share-mnt' containers saying:
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

All I ran was the docker run script generated by rancher to start the nodes.

I have gotten this error many times in many different places. The only way I've found to fix it is by destroying the cluster and recreating it.

This cluster I just created in OpenStack right before running the docker command on each host. If the error is for something else like network or host am like above it would be nice if it said so. Right now with the cannot find kublet error isn't very helpful since it was able to pull hyperkube and start other containers.

I get the same error, then I destroy the cluster and recreating it from the latest version. Then I add 3 nodes, and they are ok.
the system and docker version:
~ # cat /etc/redhat-release                                                                                root@k8s
CentOS Linux release 7.4.1708 (Core)
~ # docker info                                                                                root@k8s
Containers: 30
Running: 24
Paused: 0
Stopped: 6
Images: 25
Server Version: 17.03.2-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
the rancher images;
REPOSITORY                                        TAG                 IMAGE ID            CREATED             SIZE
rancher/rancher-agent                        v2.0.2              df087865517d        15 hours ago        228 MB
rancher/rancher                                   latest              88526c7bea4e        15 hours ago        521 MB
rancher/rke-tools                                 v0.1.8              5df9ccc4e588        7 days ago          136 MB

I'm getting same error, I tried destroying and recreating the cluster but no luck. I'm using 2.0.2 version of rancher.
Error response from daemon: {"message":"No such container: kubelet"} Error: failed to start containers: kubelet Error response from daemon: {"message":"No such container: kubelet"} Error: failed to start containers: kubelet Error response from daemon: {"message":"No such container: kubelet"} Error: failed to start containers: kubelet Error response from daemon: {"message":"No such container: kubelet"}

I used registry which had authentication enabled, after making the rancher repo public, my issue is resolved.

Same here on Ubuntu 18.04 with docker 17.03.2-ce (tried with 18.X but no luck).
Can I get out more log from the startup?

Hi, same problem here DinD (Alpine Linux) Docker 18.03.1-ce
What could we do?

I have run into the same issue as well.
<EMAIL_ADDRESS>kubernetes]# docker logs -f 3a43a93af75b
Found container ID: c066d5a54f9a3e2ff78c093be68ff9442cc065656f3f654f1487487f36acd337
Checking root: /host/run/runc
Checking file: 3a43a93af75be42ceb79aa1f0d1e9a6bb19c51983cd72537e6ac154c4074362c
Checking file: c066d5a54f9a3e2ff78c093be68ff9442cc065656f3f654f1487487f36acd337
Found state.json: c066d5a54f9a3e2ff78c093be68ff9442cc065656f3f654f1487487f36acd337
time="2018-06-12T23:18:36Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/10039/ns/mnt -F -- /var/lib/docker/overlay/a57306998356a5882524dddddc3268f986d2e18c394bee6e8db5d4d72d7cbbe3/merged/usr/bin/share-mnt --stage2 /var/lib/kubelet /var/lib/rancher -- norun]"
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
Error response from daemon: {"message":"No such container: kubelet"

I dont understand where it is finding the container id from, i have checked the local mounts on the node.
I was testing some scenarios so tried to re-register a node, after cleaning the local mounts.
I have also tried:

docker system prune -a -f
docker rm -f share-mnt

the actual error comes from /usr/bin/share-root.sh
#!/bin/bash

ID=$(grep :devices: /proc/self/cgroup | head -n1 | awk -F/ '{print $NF}' | sed -e 's/docker-\(.*\)\.scope
/\1/')
IMAGE=$(docker inspect -f '{{.Config.Image}}' $ID)

docker run --privileged --net host --pid host -v /:/host --rm --entrypoint /usr/bin/share-mnt $IMAGE "$@"
 -- norun
while ! docker start kubelet; do
    sleep 2
done

I am not really sure if this is an issue with the logic, but no container named kubelet was created by agent however the agent is stuck trying to start it up.

I am seeing the same issue when attempting to re-register a node.  The steps I use are:

Create a custom cluster
Register a node
Delete the node from the rancher console
Attempt to clean up the node with the following: https://pastebin.com/AuQUJiM4
Reregister the node

In this case the rancher server reports:
2018/06/27 14:45:35 [ERROR] ClusterController c-v2998 [cluster-agent-controller] failed with : could not contact server: Get https://x.x.x.x:6443/version: dial tcp <IP_ADDRESS>:6443: getsockopt: connection refused
The node has only 2 containers:
share-mnt: Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
quirky_hopper: ... time="2018-06-27T14:57:46Z" level=info msg="waiting for node to register"
time="2018-06-27T14:57:48Z" level=info msg="Starting plan monitor"
The only way to proceed at this point is stop all activity on the node, delete the node from the rancher ui, and recreate the cluster and re register.
Running server 2.0.4 with only the one node, Docker version 17.03.2-ce on Ubuntu 16.04.

Same for me.
Ubuntu 16.04, Docker 17.03.2-ce
I have 2 host - rancher + creating cluster with 2 machines and get same error.
Logs from docker shows
dockerd[2710]: time="2018-07-05T13:59:37.185566233Z" level=error msg="Handler for POST /v1.24/containers/kubelet/start returned error: No such container: kubelet"

The share-mnt folder is waiting for the actual agent to get the kubelet, this is not an error. The actual error can be found either in the rancher/rancher-agent container (and not called share-mnt) and/or in the rancher/rancher container (the container running Rancher UI/API). Please post full logging of both if you experience issues so I can investigate.

Docker 1.12.6
Rancher 2.0.5
I'm experiencing this issue too. Agent and controller running on the same host.
Steps:

Cluster running for quite a while (had no problems with rancher up until this point)
Attempt to upgrade container
Container starts running extremely slow
Check kubelet logs and it's complaining about an orphaned pod
Stop kubelet to attempt to fix issue
Restart kubelet and notice that the kubelet container and rancher-server container just deleted themselves

Logs for agent: https://pastebin.com/UCDhXvVg
Rancher server: https://pastebin.com/W0p7tbnm
Nothing interesting seems to be in the agent logs or the server. I'm going to try a couple of other things to see if I can resurrect this node.

Docker 17.03.2-ce
Rancher 2.0.5 (And also the 2.0.4)
Running on virtual machines with ubuntu 16.04
I'm experiencing this issue too.  Etcd and controller running on the same host.
Steps:

Cluster recently created with running etcd+control nodes and quite a few workers (working just fine)
Attempt to upgrade container and add a new etcd+control node
Clusters gets stuck at "[certificates] Deploying kubernetes certificates to Cluster nodes" and the node never registered (waited more than 2 days for it)
The failing node has only two containers running one being share-mnt
The failing node keeps complaining about "Error response from daemon: {"message":"No such container: kubelet"}"

Logs from one node that is working (With the etcd+control setup): https://pastebin.com/ZxZ87uUD
Logs from the node that I was trying to add (With the same etcd+control setup): https://pastebin.com/SGS1uY1t
Differences I see in the logs, the node working logs etcd and control as false (even though they were true on creation and the node is working ok).
I updated from rancher 2.0.4 to 2.0.5 to see if the problem goes away but, it didn't.
The name of the agent container in the working node is rancher/rancher-agent@sha256:f539238b956f701d2a5675d3efdf469ff0772e56a9b8712b1b18a5f802fc554d and in the failing node is just rancher/rancher-agent:v2.0.5 (This might be because of the update).
That's all the differences I see.

Deleting all Rancher server state and recreating the cluster and reprovisioning the agent fixed the issue and it was able to create kubelet agent.
It's not able to fully provision the Kubernetes cluster and is erroring with factory_object_mapping.go:423] Failed to download OpenAPI (the server has asked for the client to provide credentials), falling back to swagger error: You must be logged in to the server (Unauthorized).
This seems to suggest it's a rancher server problem not an agent side problem.

Server logs with error:
2018/07/17 14:50:57 [INFO] Handling backend connection request [m-02f52479421f]
2018/07/17 14:50:57 [INFO] Handling backend connection request [m-02f52479421f]
# Nothing more

Server logs after wiping out and resetting the relevant cluster:
2018/07/17 15:00:01 [INFO] Handling backend connection request [m-02f52479421f]
2018/07/17 15:00:01 [INFO] Provisioning cluster [c-48qms]
2018/07/17 15:00:01 [INFO] Creating cluster [c-48qms]
2018/07/17 15:00:01 [INFO] cluster [c-48qms] provisioning: Building Kubernetes cluster
2018/07/17 15:00:01 [INFO] cluster [c-48qms] provisioning: [dialer] Setup tunnel for host [x.x.x.x]
2018/07/17 15:00:03 [INFO] cluster [c-48qms] provisioning: [network] Deploying port listener containers
2018/07/17 15:00:03 [INFO] cluster [c-48qms] provisioning: [network] Pulling image [rancher/rke-tools:v0.1.10] on host [x.x.x.x]
2018/07/17 15:00:04 [INFO] Handling backend connection request [m-02f52479421f]
2018/07/17 15:00:07 [INFO] cluster [c-48qms] provisioning: [network] Successfully pulled image [rancher/rke-tools:v0.1.10] on host [x.x.x.x]
... # Continues to deploy (successfully)

So I believe I've run into this issue and might be able to provide some more info.
Our setup:
Custom cluster with 4 nodes:
01 (IP: x.x.x.140) - controlplane,etcd,worker
02 (IP: x.x.x.141) - controlplane,etcd,worker
03 (IP: x.x.x.142) - etc,worker
We removed 01 from the cluster through the rancher UI - this removal might not have been done cleanly, but it'd be impossible to tell at this point.
We've since cleanly wiped 01 following the removal docs, and then attempted to re-add the node with the same roles. The node addition seems to get stuck on "waiting to register with Kubernetes".
We removed 01 again since it seemingly wasn't registering, so the only nodes in our cluster were 02 and 03.
Interestingly, when I hit the Rancher API (/v3/clusters/c-gbq42) for this cluster, I see a couple strange things (trimmed for brevity):
{
    "apiEndpoint": "https://x.x.x.140:6443",
    "appliedSpec": {
        "rancherKubernetesEngineConfig": {
            "nodes": [
                {
                    "address": "x.x.x.140",
                    "internalAddress": "x.x.x.140",
                    "role": [ "etcd", "controlplane", "worker" ],
                },
                {
                    "address": "x.x.x.142",
                    "internalAddress": "x.x.x.142",
                    "role": [ "etcd", "worker" ],
                },
                {
                    "address": "x.x.x.141",
                    "internalAddress": "x.x.x.141",
                    "role": [ "etcd", "controlplane", "worker" ],
                },
            ]
        },
    },
    "id": "c-gbq42",
}

It looks like rancher still believes that the 01 node is in the cluster, and the "apiEndpoint" is pointing to that node, even though it's no longer available.
Is there some way to modify this cluster metadata? I have a feeling that forcibly removing the "x.x.x.140" node and pointing "apiEndpoint" to "x.x.x.141" (our other controlplane node) might resolve things. Failing that - we'll probably rebuild the entire cluster, but it would be nice if there's some surgery we could try performing beforehand.

We also ran into this same error - it may have been prompted by stopping / starting docker processes on the etcd / controlplane nodes during provisioning testing.  While acknowledging that we contributed to getting the environment wedged, it would be really great if we had a process to troubleshoot and resolve this error message, as well as a clean shutdown / restart procedure per node.

Same here, there is no fix? Issue opened in May...

There is not an actual root cause determined yet. Reading through the replies, I see a lot of responses where nodes or responses start to slow down and I'm curious what specifications the machines are that are being used to setup the cluster.

Hy thanks @superseb , seems like its more an generic error. After using more powerfull vms there is no issue anymore..

@Berndinox Thanks, can you share what specifications the machines had when it wasn't working, thanks.

Digital Ocean 1vCPU 3 GB RAM was not enough, an Upgrade to 2vCPU 4GB did it. BR ---- On Di, 11 Sep 2018 17:08:58 +0200 Sebastiaan van Steenis<EMAIL_ADDRESS>wrote ---- @Berndinox Thanks, can you share what specifications the machines had when it wasn't working, thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

i solved the problem by restarting the rancher ui/backend container seems something stuck there, after the restart of the rancher backend container the "broken" node returned in a healthy state and spawned the missing containers

I solved this by restarting the racher ui/backend container.

this time it does not solve the problem using rancheros :/

I also got the same error，I have three machine，one for rancher server，others for node agent。
now I've fixed the bug，the point is the same node name.
rancher label the node from the hostname，so I login the two node agent machine, using 'hostnamectl --static set-hostname <your_defined_hostname> to different the hostname between the two machine
now it runs ok

FYI for anyone that has the issue, the solution above worked for me. I simply re-registered the same node again but with a different hostname. As soon as the new one came up, the deletion of the old node instance completed and it dropped out. Everything came back up at that point.

I got same error while installing Rancher 2.0
docker logs 5393ba3fd645
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2

If you are running your nodes in CentOS, to avoid this error then you need to open TCP ports and make sure firewalld is enabled, according to these requirements.
Delete all nodes, then run the following on each machine or VM, then create the node again.  This solved the problem for me.
docker rm -f $(docker ps -qa)
docker volume rm $(docker volume ls -q)
cleanupdirs="/var/lib/etcd /etc/kubernetes /etc/cni /opt/cni /var/lib/cni /var/run/calico /opt/rke"
for dir in $cleanupdirs; do
echo "Removing $dir"
rm -rf $dir
done
systemctl restart firewalld
systemctl enable firewalld
iptables -A INPUT -p tcp --dport 6443 -j ACCEPT
iptables -A INPUT -p tcp --dport 2379 -j ACCEPT
iptables -A INPUT -p tcp --dport 2380 -j ACCEPT
iptables -A INPUT -p tcp --dport 10250 -j ACCEPT
firewall-cmd --zone=public --add-port=6443/tcp --permanent
firewall-cmd --zone=public --add-port=2379/tcp --permanent
firewall-cmd --zone=public --add-port=2380/tcp --permanent
firewall-cmd --zone=public --add-port=10250/tcp --permanent
firewall-cmd --reload

with cleaning the rancher install info, the probelm is sovled.
#!/bin/env bash
docker rm -f $(docker ps -qa)
docker volume rm $(docker volume ls -q)
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
       /run/calico \
       /run/flannel \
       /var/lib/calico \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
       /var/lib/rancher/rke/log \
       /var/log/containers \
       /var/log/pods \
       /var/run/calico

ip address show
ifconfig -a
iptables -L -t nat
iptables -L -t mangle

I encountered the same error creating my first production rancher cluster on ubuntu 16.04. The solution was to stop and restart the rancher server. (Restarting the rancher agents had no effect).

Ubuntu 18.10
docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --name=rancher rancher/rancher:latest
$ docker logs -f share-mnt

Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

delete microk8s
neo@m-1d41c853af58:~$ snap list
Name      Version         Rev    Tracking   Publisher   Notes
core      16-2.37.4       6531   stable     canonical✓  core
go        1.12            3318   stable     mwhudson    classic
kubectl   1.13.4          780    stable     canonical✓  classic
lxd       3.11            10343  stable/…   canonical✓  -
microk8s  v1.14.0-beta.1  442    1.14/beta  canonical✓  classic

neo@m-1d41c853af58:~$ snap remove microk8s kubectl lxd
error: access denied (try with sudo)

neo@m-1d41c853af58:~$ sudo snap remove microk8s kubectl lxd
sudo: unable to resolve host m-1d41c853af58: Invalid argument
microk8s removed
kubectl removed
lxd removed

Now
+ docker kill --signal=SIGTERM 95b76b26d6cababc716edc517776dcd34e155bd47b914f0510a9465da7d49475
95b76b26d6cababc716edc517776dcd34e155bd47b914f0510a9465da7d49475
++ exit 0

I keep running into this issue continuously. I have tried to delete rancher, delete docker, delete prune the nodes, even reinstalled the operating system but sometimes it works and sometimes it just keeps getting stuck at

2

+ docker start kubelet

Error response from daemon: {"message":"No such container: kubelet"}

Error: failed to start containers: kubelet

Are there any permanent solutions known as this is really frustrating...

Whenever you are trying to reuse nodes for Rancher server install or to add back into Rancher as a cluster, make sure you clean up per our recommendations:
https://rancher.com/docs/rancher/v2.x/en/removing-rancher/

I also got the same error，I have three machine，one for rancher server，others for node agent。
now I've fixed the bug，the point is the same node name.
rancher label the node from the hostname，so I login the two node agent machine, using 'hostnamectl --static set-hostname <your_defined_hostname> to different the hostname between the two machine
now it runs ok

Stumbled on to this issue too. My problem was the also the hostname, just as @phutchins describes. I had set up two different nodes using the same cloud-config.yaml with RancherOS which caused both nodes to have the exact same hostname. Changed one and re-registered and problem was solved.

After update 2.2.2 and rotate certificate and rollback snapshot all nodes with control plane and etcd is damage.
2019/04/17 15:53:59 [INFO] Handling backend connection request [c-25d48:m-37b0224f2574]
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: [reconcile] Rebuilding and updating local kube config
I0417 15:54:02.471711       6 http.go:110] HTTP2 has been explicitly disabled
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-163976307/kube_config_cluster.yml]
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [INFO] kontainerdriver rancherkubernetesengine stopped
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-163976307/kube_config_cluster.yml]
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: Successfully Deployed local admin kubeconfig at [management-state/rke/rke-163976307/kube_config_cluster.yml]
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: [certificates] Successfully deployed kubernetes certificates to Cluster nodes
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: Successfully Deployed state file at [management-state/rke/rke-163976307/cluster.rkestate]
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: [etcd] Restarting up etcd plane..
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: [etcd] Successfully restarted etcd plane..
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [INFO] cluster [c-25d48] provisioning: [controlplane] Check if rotating a legacy cluster
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [cluster-provisioner-controller] failed with : Failed to inspect [kube-apiserver] container on host [<IP_ADDRESS>]: Error: No such container: kube-apiserver
2019/04/17 15:54:02 [ERROR] ClusterController c-25d48 [user-controllers-controller] failed with : failed to start user controllers for cluster c-25d48: failed to contact server: Get https://<IP_ADDRESS>:6443/version?timeout=30s: dial tcp <IP_ADDRESS>:6443: connect: connection refused

Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2

This is quite blocking issue also happening to me.
Unfortunately with this is not realistic to maintain a production cluster in operation since at some point nodes can't join anymore. Really sad about that though since I reckon Rancher is a good product.

Happened to me few moments ago, cleaning helped. See my steps in https://github.com/rancher/rancher/issues/19882#issuecomment-501056386

I have a similar issue: I had a cluster that was composed of VM nodes - I needed to rebuild the VMs, so I removed the node from the UI, waited until it was fully removed, then destroyed the VM and recreated it with vagrant.  Re-added it by using the rancher command (checking all three boxes for worker/etcd/control plane).  The node cycles with "No such container: kubelet".
Going to try restarting the rancher nodes to see if that fixes it; although they're HA so I guess I need to restart all three at once to do so?

Was having a similar issue, but for me, I was running a worker node on the same host (as VM) that rancher was on. Since the server's hostname was "rancher", the worker node would come up with the name "rancher." Seeing some of the traffic here about hostname similarities, I tore down everything rancher related and ensured that when I was loading the rancher container, I gave it the name rancher-server to avoid any confusion. This seems to have gotten me past the Error response from daemon: {"message":"No such container: kubelet"} message.

I'm having the same issue in Linux RH Atomic installation.
INFO: Arguments: -- share-root.sh docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.4 --server https://rancher-dev.home.net --token REDACTED --ca-checksum 67d64376c10cfba4e3f613b6aacb9beef09b8fd2094efcf31750ec89f6c40e39 --no-register --only-write-certs /var/lib/kubelet /var/lib/rancher

trap 'exit 0' SIGTERM
++ grep :devices: /proc/self/cgroup
++ head -n1
++ awk -F/ '{print $NF}'
++ sed -e 's/docker-(.*).scope/\1/'
ID=6b0210acd179c71d2eec8782aa57256b8ef75cff756aa0363247bf780f763e4b
++ docker inspect -f '{{.Config.Image}}' 6b0210acd179c71d2eec8782aa57256b8ef75cff756aa0363247bf780f763e4b
IMAGE=rancher/rancher-agent:v2.2.4
bash -c 'docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.4 --server https://rancher-dev.home.net --token dch54f9xl8j29sjtvhxm7tjqm9j7wmsw9d6brr4k4mdrn4r48xjglf --ca-checksum 67d64376c10cfba4e3f613b6aacb9beef09b8fd2094efcf31750ec89f6c40e39 --no-register --only-write-certs'
8b238bc28c42777ecd8830ba7504d17b947d68723a35d2413b466f182cc6c84a
docker run --privileged --net host --pid host -v /:/host --rm --entrypoint /usr/bin/share-mnt rancher/rancher-agent:v2.2.4 /var/lib/kubelet /var/lib/rancher -- norun
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2
[endless loop]

I tried to clean up the machine and install in a new server and nothing seams to work. I review my network,hostnames and proxy and all configured.
Please let me know if I can help someway. (logs,configs)

For my case,  I can create first master node, but cannot add second/third master node and other worker node , after some time's diagnostics, I found that the issue is configuration of net.ipv4.ip_forward=0 on our CentOS,  after changed net.ipv4.ip_forward=1 and systemctl restart network, I can add other nodes to cluster successfully.

For my case, I can create first master node, but cannot add second/third master node and other worker node , after some time's diagnostics, I found that the issue is configuration of net.ipv4.ip_forward=0 on our CentOS, after changed net.ipv4.ip_forward=1 and systemctl restart network, I can add other nodes to cluster successfully.

My  /etc/sysctl.conf
Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1
Uncomment the next line to enable packet forwarding for IPv6
Enabling this option disables Stateless Address Autoconfiguration
based on Router Advertisements for this host
net.ipv6.conf.all.forwarding=1
Disable IPV6
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.docker0.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
No matter the order in my case, the issue are happening with controlplane,etcd and worker.

I got this error on my first HA RKE install when adding my first cluster.  Wiping nodes and re-creating the cluster had no effect.  Removing the cluster and restarting rancher server nodes fixed it for me

I solved removing the hostname from <IP_ADDRESS> on /etc/hosts and including the FQDN also. So the machine now is responding the hostname and FQDN to the eth0 not loopback.
Resuming:
192.168.1.X  hostname FQDN

I hit this issue as well. In my case it started when the last node of my
experimental cluster stopped working after I restarted into a new version of
Docker and it couldn't get up and running again. I managed to "fix" it without
deleting and creating a new cluster (as I had to several times before when
I encountered the same issue).
Here's what I did:

Cleaned the node by running

docker system prune
docker volume prune

Note that this will delete all the Docker volumes, take care if you have
important data in your volumes.

Cleaned Rancher/Kubernetes runtime data on the node.

rm -rf /etc/cni/ /etc/kubernetes/ /opt/cni/ /var/lib/calico/ /var/lib/cni/ /var/lib/rancher/ /var/run/calico/

Note that official docs on node cleanup recommend also removal of /opt/rke and
/var/lib/etcd. I did NOT remove them because they contain cluster etcd
snapshots and data. This is especially important in case there's only one node
in the cluster.

I exec-ed into the rancher container and hacked the cluster status (thx
@ibrokethecloud for the hint):

docker exec -it rancher bash

Inside the container:
apt-get update && apt-get -y install vim
kubectl edit cluster c-XXXX  # replace the cluster-id with an actual cluster ID

Now in the editor I found the key apiEndpoint (it should be directly under
the status key) and removed it. Exit the editor and container. Make sure
kubectl says that it updated the cluster.

From the Rancher UI I got the command for registering new node.
I set a different name for the node than it was before by adding a
--node-name to the docker run command (actually there's an edit box for this
under advanced settings). It looked like this:

docker run -d --privileged --restart=unless-stopped --net=host \
  -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.2.6 \
  --server https://rancher.example.com --token XXXXXXXXXXXXXXX --node-name mynode2 \
  --etcd --controlplane --worker

I run the above command on the cleaned node and finally it registered
successfully and RKE started up all the kube-* and kubelet containers.

I also tried to just clean-up the node and register it again, but it always
ended up with the "Error: failed to start containers: kubelet".
Given the above, I think Rancher doesn't handle well the case when all the
cluster nodes are become unresponsive. In this case nodes can't even be removed
from the cluster. When I tried to remove the faulty node it got stuck in the
"Removing" state indefinitely, probably because cluster's etcd couldn't be reached.

I just created a new local VM with Ubuntu 18.04 to test if rancher 2.2.6 is 'stable' now. But I can't add the first worker with all 3 roles on my single machine setup because of this issue.

faild to start Kubelet means something has stopped it from starting up.
In my case, the problem is that my etcd cluster lost quorum, removing dead node from etcd state manually using etcdctl fixed this issue for me and the kubelet issue is gone.

Man.. there sure are a lot of posts here.
One of the first few responses was asking for more information. I also want more information because it seems like a lot of different kinds of problems can cause this. Could this error message be enhanced?

Rancher stable 2.2.8 also found this issue()

sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

docker ps -a

CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                   PORTS                                      NAMES
f428817311ac        rancher/hyperkube:v1.14.6-rancher1     "/opt/rke-tools/entrâ€¦"   2 hours ago         Up 2 hours                                                          kube-proxy
08f59cee9f37        rancher/hyperkube:v1.14.6-rancher1     "/opt/rke-tools/entrâ€¦"   2 hours ago         Up 2 hours                                                          kubelet
e1ad2c121a66        rancher/hyperkube:v1.14.6-rancher1     "/opt/rke-tools/entrâ€¦"   2 hours ago         Up 2 hours                                                          kube-scheduler
1088e8e2ef66        rancher/hyperkube:v1.14.6-rancher1     "/opt/rke-tools/entrâ€¦"   2 hours ago         Up 2 hours                                                          kube-controller-manager
5c30c26352c1        rancher/hyperkube:v1.14.6-rancher1     "/opt/rke-tools/entrâ€¦"   2 hours ago         Up 2 hours                                                          kube-apiserver
0465bfcce0d2        rancher/rke-tools:v0.1.42              "/bin/bash"              2 hours ago         Created                                                             service-sidekick
809c08e9f9bf        rancher/coreos-etcd:v3.3.10-rancher1   "/usr/local/bin/etcdâ€¦"   2 hours ago         Up 2 hours                                                          etcd
12a2ff2ae478        rancher/rancher-agent:v2.2.8           "run.sh --server httâ€¦"   2 hours ago         Up 2 hours                                                          flamboyant_engelbart
ab18ff7b118c        rancher/rancher-agent:v2.2.8           "run.sh -- share-rooâ€¦"   2 hours ago         Exited (0) 2 hours ago                                              share-mnt
b998e63a1a7a        rancher/rancher-agent:v2.2.8           "run.sh --server httâ€¦"   2 hours ago         Up 2 hours                                                          affectionate_edison
6a8d364109ab        rancher/rancher:stable                 "entrypoint.sh"          2 hours ago         Up 2 hours               <IP_ADDRESS>:80->80/tcp, <IP_ADDRESS>:443->443/tcp   silly_mclean

I have run into the same issue as well , but solved.
try to see the rancher server log ,not the log of agent.
agent log:

sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
sleep 2
docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet

rancher server log:
2019/11/04 09:25:07 [INFO] kontainerdriver rancherkubernetesengine stopped
2019/11/04 09:25:07 [INFO] cluster [c-w6rtq] provisioning: [network] Deploying port listener containers
2019/11/04 09:25:07 [INFO] cluster [c-w6rtq] provisioning: [network] Pulling image [http://<IP_ADDRESS>/rancher/rke-tools:v0.1.34] on host [<IP_ADDRESS>]
2019/11/04 09:25:07 [ERROR] cluster [c-w6rtq] provisioning: [invalid reference format]
2019/11/04 09:25:07 [ERROR] ClusterController c-w6rtq [cluster-provisioner-controller] failed with : [invalid reference format]
2019-11-04 09:28:59.185541 I | mvcc: store.index: compact 5146
Found that the images address is more http.

Spent days tracking this backwards, looking for answers.  Anyone still getting this error, you should first note that the share-mnt container does this in a tight loop while your node is pulling the kubernetes images and setting up other things by the remote Rancher server, so this isn't abnormal to see on first boot.  In the case of a ram-only machine, you'll always see this when you boot until the provisioning progresses to the kubernetes being available stage.  It's not necessarily stuck, it might just be waiting.
I have noticed that if you fail to provision a node, it will always fail well before it pulls the kubernetes container images, due to the cluster recognizing the node but assigning it a different certificate (or something), leaving it hung.  I solved it by wiping my Rancher server and starting over.  This has absolutely got to be fixed properly, somehow.
HOWEVER...
If you actually do get a container named kubelet created, and the error switches over to this:
Error response from daemon: {"message":"linux mounts: open /proc/self/mountinfo: no such file or directory"}
Error: failed to start containers: kubelet

from what I can tell, it means you're running a system where the OS is on a tmpfs volume (either diskless, or not installed to disk yet).  There are other errors similar to this in the Docker and Kubernetes forums.  If I get to the bottom of where it happens, I'll open a ticket.

Got to the bottom of it.  I was running diskless iPXE booted RancherOS.  Although rancher-agent started up, by the time it started installing Kubernetes onto the machine, it exhausted something in the tmpfs file system, causing a failure so bad that nothing shows up when you "ls -al /proc", and "df" or "du" would just fail.  Three different perfectly good servers with almost exactly the same results.  There may be a kernel setting that would expand the OS file system somehow, but by installing to hard drive and booting there instead, all the issues went away.
Still worth noting that if you are watching the logs of share-mnt while spinning up a node, quite a number of errors will pop up while k8s is installing, even on a completely healthy node.

I bumped into the very same issue today.
I have basically been trying to add a worker node to rancher all morning.
In the end it seemed that using a certain hostname for the node didn't work, but using a different one registered the node straight away.
It seems in my tests I have been impatiently adding/removing nodes from the cluster by just deleting them from the UI. My suspicion is that etcd cluster got dirty as a result.
A more graceful procedure should be followed, such as
Cordon the node from rancher
Drain the node from rancher
Delete node from rancher (wait for cluster to update)
SSH to the node (this will stop all containers): docker stop $(docker ps -q)
SSH to the node (this will remove all containers): docker rm $(docker ps -q)
Reboot node

I did not rebuild the cluster from scratch, as in my case I knew exactly when the tests started. I merely restored an earlier etcd snapshot, tried again and it all worked as expected.

Got to the bottom of it. I was running diskless iPXE booted RancherOS. Although rancher-agent started up, by the time it started installing Kubernetes onto the machine, it exhausted something in the tmpfs file system, causing a failure so bad that nothing shows up when you "ls -al /proc", and "df" or "du" would just fail. Three different perfectly good servers with almost exactly the same results. There may be a kernel setting that would expand the OS file system somehow, but by installing to hard drive and booting there instead, all the issues went away.

Were you able to get worker nodes stood up on an iPXE booted RancherOS? I am currently trying to boot 10 nodes this way (without local storage), and it does not seem to be working (seeing the same errors you were).

Never got it working.  From what I could tell, the tmpfs is exhausted of inodes or something, to the extent the OS stopped working.  Someone with more time on their hands can maybe sort it out for the rest of us, but I just threw a small HDD in each box and installed ROS there instead.  Worked fine.  Ultimately gave up on iPXE because of it, unfortunately, and ROS as well, because 99% of the value of it, to me at least, was to treat nodes as extremely dumb cattle.

Ah, that's unfortunate.
My only thought is maybe switching to K3sOS, but that is rather poorly documented, especially for iPXE booting.
Yea, the only reason I'm using iPXE right now is because of these node's lack of local storage, otherwise, I'd just install an OS and it'd be fine.
Further to your note on dumb cattle, I am not sure rancher would handle non-persistent nodes trying to join with existing node names anyways, so that might not even work.

I had set up my DHCP to assign the same IP address to the machine's MAC addresses, then ran an internal web server that trivially returned a hostname according to the requestor's IP address.  The nodes would boot, curl the server for its name, then set it right up front in the cloud init.  It's an afternoon to set up, but not hard.  Overall didn't seem worth pursuing it, given the failure to run entirely in RAM.
I am running a k3os cluster in AWS, and probably would not recommend it for running iPXE, since it really wants an external high availability MySQL setup that it can keep its brains in.  RDS is good for that, but running locally.... ehh.

Maybe I am misunderstanding, but k3os requires an external HA MySQL setup? I thought it was just a lighter weight version of RancherOS.
Hopefully someone on the OS team is able to take a look at that issue and provide a workaround or resolution.
What did you end up doing OS wise with your hosts you were trying to get working diskless?

I'm not sure if k3os and k3s is the same thing (I suspect not, on reflection), but to configure an HA Rancher management cluster, they recommend using k3s rather than ROS, which does need something to store shared data into.  The easy way is MySQL, the hard way is etcd.  https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/kubernetes-rke/
I ended up dropping an older server SAS drives about 80gb, which is plenty large for an OS.  I dropped Ubuntu 18.04 on it and did a simple install.  Life became much easier after that.

Oh, my Rancher master cluster is on separate nodes, and my master nodes for this cluster have local storage, so I am fine on that front.
Ah, I might have to end up adding local storage I guess. Not looking forward to that cost lol.

I'm having the same issue: I added a third node to a cluster, worked great.

NAME:
/usr/bin/share-mnt - A new cli application

USAGE:
/usr/bin/share-mnt [global options] command [command options] [arguments...]

VERSION:
1d97ce9

COMMANDS:
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--stage2
--help, -h show help
--version, -v print the version

time="2020-05-28T12:42:35Z" level=fatal msg="flag provided but not defined: -node-name"
+ docker start kubelet
Error response from daemon:

{"message":"No such container: kubelet"}

Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon:

{"message":"No such container: kubelet"}

Error: failed to start containers: kubelet
+ sleep 2
+ docker start kubelet
Error response from daemon:

{"message":"No such container: kubelet"}

Error: failed to start containers: kubelet
+ sleep 2

I've checked the diskspace, all ok. Also the hostname is correct and set up correctly in the hosts file.
Docker ps output when this happens:
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES

9e7d5c8d8b2d        rancher/rancher-agent:v2.3.5   "run.sh --server h..."   4 seconds ago       Up 3 seconds                            clever_davinci

18ec181d0456        rancher/rancher-agent:v2.3.5   "run.sh -- share-r..."   4 seconds ago       Up 3 seconds                            share-mnt

19f4d9d89a00        rancher/rancher-agent:v2.3.5   "run.sh --server h..."   15 seconds ago      Up 14 seconds                           zealous_cray

I also tried changing the hostname but no joy there. Any pointers would be greatly appreciated. And I join the chorus of those who ask for a better error message that lets us actually determine what's wrong.

I had the exact same issue and I found the cause:
I was trying to add a node, which had the exact same hostname as the first node. Rancher seems to use the hosts hostname as a name for the node in rancher itself. Try using --node-name NEW_NAME to override. At least for me, this worked.

I got the same error while creating a new cluster and add a node. And then resolve it!
Here are some info:
CentOS: 7.3.1611
kernel: 3.10.0-862
docker: 18.09.9
rancher: v2.2.2 (set up by rke)
rke: v0.2.4
Check steps:

make sure the /etc/hosts on all nodes (local-cluster's nodes and the new node) have all record about each node.
vi /etc/hosts
make sure each node (local-cluster's nodes and the new node) can ssh to another without password.
ssh-keygen -t rsa & ssh-copy-id<EMAIL_ADDRESS>make sure the system-default-registry (below Global -> Settings) is not end with '/'.
make sure the private registry (under advanced options) is not end with '/'.

Maybe you need to clean up the new cluster and node after failed options.
Clean steps:

delete node on rancher UI and wait it disappear.
delete cluster on rancher UI and wait it disappear.
run some scripts shown as below and here to clean the node and reboot.

docker ps -aq | xargs docker stop
docker ps -aq | xargs docker rm -v
docker volume rm $(sudo docker volume ls -q)
mount | grep '/var/lib/kubelet'| awk '{print $3}'|xargs umount
rm -rf /var/lib/etcd \
    /var/lib/cni \
    /var/run/calico \
    /etc/kubernetes/ssl \
    /etc/kubernetes/.tmp/ \
    /opt/cni \
    /var/lib/kubelet \
    /var/lib/rancher \
    /var/lib/calico

Finally, up the new cluster and GLHF! Please let me know if anything can be optimized, thanks!


Q2: Brand level up task list

Summary
For this next quarter I will be focusing on leveling up our branding. This will consist of a bunch of small changes, updates and edits across the website and general user interaction. I am compiling a list to help me achieve this below.
Goal
Get users to (continue to) talk about PostHog's branding online.
Plan
Homepage

[ ] Beam me up hedgehog V 2.0, UFO, star trek beam, animated in anyway
[ ] Animate Billboard truck section to look like its driving with moving ground and cloud/wind rushing past
[ ] Future of PostHog section on homepage - make hedgehog more futuristic/explorer
[ ] 'Please help our conversion rate hog' - praying on his knees
[ ] Enclosed HogQL section on homepage (reworked layout)
[ ] EU vs US updated to be not curved and more stereotypical depictions of each location (humorous)

Product Pages

[ ] add logo images to customer stories sidebar, will make them more easily seen and more distinguishable between eachother
[ ] Animated beta flag on homepage carousel of products
[ ] Update hedgehogs on carousel to be size and design consistent
[ ] Make hedgehogs on product carousel interactive? aka follow cursor or move when clicked
[ ] graphic dashboard examples on product pages instead of screenshots
[ ] animated graph graphics on carousels
[ ] 'Answer all questions' sections are not made obvious that the questions are clickable- hover colour state
[ ] Pricing hog update on product page
[ ] Sticky FAQ's on scroll (product page)
[ ] Highlighted rows when hovering on comparison chart
[ ] 'Whats best for you' reimagined as Pokemon style cards (powers, strengths etc)
[ ] Web analytics features section - screenshots replaced with better graphics
[ ] Product OS hedgehog needs an update
[ ] 'Boring but important' section could be made better - think silly CTA section
[ ] templates layout - drop down of features/quick scan of what they look like/carousel?

Docs

[ ] Newsletter update how infographics are designed
[ ] Framework guides hedgehog replacement
[ ] Partners of PostHog -hero image or just hedgehog related to this
[ ] Resources images on docs are usually all the same- change this to something else
[ ] Data warehouse illustration on docs needs updating

Company

[ ] Company page hog animation - hogs are older style branding - needs updating
[ ] redraw james and tim - don't match newer team images
[ ] redo transparency hog - could be funnier
[ ] update map on company page - I am not in congo - misleading

Other

[ ] Hedgehog avatars for profiles without images (questions section)
[ ] Avatar library/make your own hedgehog avatar minigame
[ ] anonymous profile image (aka no profile image image) changed to hedgehog version?
[ ] Team profiles- some say full country names and some are just letters
[ ] hedgehog icon for how tall company is in hedgehogs
[ ] update older team images to look more in keeping with newer versions
[ ] HogCoin visuals/merch collab

Worth adding some of the more 'boring' transactional pages, like this or this?
(But yes I think this looks like a great list!)

This is a fantastic list!!

Animation
Let's make sure we're checking with Stefan on availability/interest before we prioritize any of the animation work.
Homepage

Enclosed HogQL section on homepage (reworked layout)

What do you mean by this?
Product pages

add logo images to customer stories sidebar, will make them more easily seen and more distinguishable between each other

This view is shared between multiple types of content so we don't want to make any changes to this. But we're probably going to separate out the customer stories from this template anyway, so we'll probably do something different here in the not-too-distant future.

graphic dashboard examples on product pages instead of screenshots

We definitely want to retain product screenshots here. Okay adding graphics, but don't want to lose the ability to directly see what the product looks like. (Super important for our demographic)

Sticky FAQ's on scroll (product page)

May end up being technically challenging with how the site is built
Docs

Resources images on docs are usually all the same- change this to something else

This whole page template needs to be re-worked. We used to have the different articles in the main column (vs just the sidebar) but they've been removed and now the main column of the page is missing meat.
Other

Team profiles- some say full country names and some are just letters

These are editable in team member profiles - just need to be updated there to be uniform


0746- Min Cost Climbing Stairs

Description of the Problem
On a staircase, the i-th step has some non-negative cost cost[i] assigned (0 indexed).
Once you pay the cost, you can either climb one or two steps. You need to find minimum cost to reach the top of the floor, and you can either start from the step with index 0, or the step with index 1.
Code
    def minCostClimbingStairs(self, cost: List[int]) -> int:
        l=len(cost)

        if l<=2:
            return 0
        a=[0]*l
        a[0]=cost[0]
        a[1]=cost[1]
        for i in range(2,l):
            a[i]=min(a[i-1],a[i-2])+cost[i]
        return min(a[l-1],a[l-2])

Link To The LeetCode Problem
https://leetcode.com/problems/min-cost-climbing-stairs/

Please assign this to me


Images are not properly generated on v1.4.0

Please make sure you are testing with the latest release of html2canvas.
Old versions are not supported and issues reported for them will be closed.
Please follow the general troubleshooting steps first:

[x] You are using the latest version
[x] You are testing using the non-minified version of html2canvas and checked any potential issues reported in the console

Bug reports:
I am facing an with 1.4.0 version. Canvas is not properly generated on on this. when 1 am using vesion 1.3.3 my feature works fine but after we switch to v1.4.0 its is not generating complete canvas with proper height canvas. Canvas height always remain 1px
Images on v1.4.0

Image on v1.3.3

Specifications:

html2canvas version tested with:
Browser & version:
Operating system:

uninstall grammarly chrome extension, then all will be normal again

卸载语法Chrome扩展，然后一切都会再次正常

How to uninstall grammarly chrome extension？

uninstall grammarly chrome extension, then all will be normal again

How to uninstall grammarly chrome extension？ @K0wd

uninstall grammarly chrome extension, then all will be normal again

How to uninstall grammarly chrome extension？ @K0wd

https://www.youtube.com/watch?v=j1yfnO9Eomc @jacky-yanglei

I don't install grammarly chrome extension, I still have the same problem.

I don't install grammarly chrome extension, I still have the same problem. @K0wd

remark

uninstall grammarly chrome extension, then all will be normal again

Just spent several hours trying to figure out why it would render properly on load but not on function call. Figured I'd try this on a whim, worked like a charm. Seems really bizarre, but I guess grammarly was somehow conflicting with this?

I don't install grammarly chrome extension, I still have the same problem. @K0wd

Try uninstalling other plugins
or try in incognito

uninstall grammarly chrome extension, then all will be normal again

same here! thanks 👍🏽

For those who are looking for a real workaround:
function fnIgnoreElements(el: any) {
  return typeof el.shadowRoot === "object" && el.shadowRoot !== null
}

const canvas = await html2canvas(input, {
        ignoreElements: fnIgnoreElements,
})

I have also encountered this issue with grammarly and html2canvas 1.4.1

I confirm that @ospfranco solution fixed the problem with the Grammarly Chrome Extension. Thanks!
Grammarly Chrome Extension - 14.1050.0
html2canvas - 1.4.1

@antoniotorres I encountered it only recently (yesterday) on 1.4.1, but I don’t know which version of the Grammarly was causing issue.
Issue:
Chrome profile with Grammarly installed was not saving and generating base64 encoding for images using html2canvas correctly. Worked ok after Grammarly was uninstalled.
For now I have reverted back to html2canvas 1.3.4 which works ok.

There seems to be an issue with Grammarly and the Shadow DOM, so in order to workaround it, you need to skip the elements which are not properly created:
function fnIgnoreElements(el: any) {
  return typeof el.shadowRoot === "object" && el.shadowRoot !== null
}

const canvas = await html2canvas(input, {
	ignoreElements: fnIgnoreElements,
})

I confirm that too
I will add if you are using jspdf and .html() method, make sure to pass the ignoreElements() as follow
doc.html(pdf_el, {
	callback: function (doc) {
		// Save the PDF
		doc.save('search_result.pdf');
	},
	// ...
	html2canvas: { // <<<<<<------- html2canvas options
		ignoreElements(el) {
			return typeof el.shadowRoot === 'object' && el.shadowRoot !== null;
		},
	},
});


Add TLS renegotiation support

Add flag to support TLS renegotiation for chart repos and refactor chartrepo find code.
Fixes https://github.com/helm/helm/issues/7344
add a chart repository
Usage:
helm repo add [NAME] [URL] [flags]
Flags:
--ca-file string       verify certificates of HTTPS-enabled servers using this CA bundle
--cert-file string     identify HTTPS client using this SSL certificate file
-h, --help                 help for add
--key-file string      identify HTTPS client using this SSL key file
--no-update            raise error if repo is already registered
--password string      chart repository password
--renegotiate string   TLS renegotiation strategy, valid options include Never, OnceAsClient and FreelyAsClient (default "Never")
--username string      chart repository username

Hey @bacongobbler, sorry to trouble you but I was wondering if there was any chance you have the bandwidth to comment on the changes I made and if they address your concerns.

Thanks for the ping. Given the size of this PR it may be a while before I can find the time to review and test this PR. Just letting you know ahead of time. Thanks!

A couple of quick notes here as well to expedite the review:

looks like a few commits are missing the DCO sign-off. Would you mind amending your commits and sign off on them? More info here: https://github.com/helm/helm/pull/7472/checks?check_run_id=485556727
The tests are failing because the code has not been formatted. You should be able to run go fmt on the code and it should reformat itself.

Let us know if you run into any issues.

Is this abandoned?

Is this abandoned?

Yes, I don't think there was enough interest to ever merge this.


Chrome Crashes

Chrome: 46.0.2490.80 (64-bit)
OS: Ubuntu 14.04
CPU: i7-4790
RAM: 16GB

The tab process crash after switching between some components.

Hi @oswaldoacauan. Tks for reporting, we are working to improve the performance...


[Object Map Iterator] in Tabs

Hi @PedroBern @andreialecu
When upgrading to 4.0.0-next.0 and running the basic version in a react-native application (not expo) as per:
import { Tabs } from 'react-native-collapsible-tab-view';
.
.
.

return (
  <Tabs.Container HeaderComponent={_renderHeader}>
      <Tabs.Tab name="A">
        <Tabs.FlatList
          data={[0, 1, 2, 3, 4]}
          keyExtractor={(v) => v + ''}
          renderItem={null}
        />
      </Tabs.Tab>
      <Tabs.Tab name="B">
        <Tabs.ScrollView>{null}</Tabs.ScrollView>
      </Tabs.Tab>
 </Tabs.Container>
)

I see [Object Map Iterator], where the tabs are meant to be:

I also see:

I'm also having a problem scrolling the header. I will add additional notes here as I investigate.

I created a bare rn app here and cannot reproduce this:
https://github.com/andreialecu/rnctv-react-native-bare-test

What might be different between the bare repro above and yours?

I also do this in PR #101 and it also seemed to be ok for me, I'm trying to get to the bottom of the problem. It's strange because it happens on a screen in that just implements the QuickStart example. I will fork your repo and perhaps add the packages that I am using and update this issue if I find anything.

Might be related to babel. Do you use any other babel (or metro) plugins?

I'm also having a problem scrolling the header. I will add additional notes here as I investigate.

@alexpchin I just realised on the bare rn app that using 2.0.0-rc.3 of reanimated breaks pretty much everything. Please try going back to rc.0 and testing things with that version.
Expo only supports rc.0 and I think @PedroBern wants to mainly follow Expo support.

Hi @andreialecu I've found out what the problem is, I think last night @PedroBern released the latest version as "4.0.0-next.0", in my package JSON. (Please see the original issue post?) I copied from the Release Tags very soon after he had pushed, so I think 4.0.0-next.0 changed to 4.0.0-rc.0?
"react-native-collapsible-tab-view": "4.0.0-next.0",

I have now changed to:
"react-native-collapsible-tab-view": "^4.0.0-rc.0"

Sorry! Really strange how that change I made to hooks seemed to fix things... Really threw me. I will close this now and then move onto the next issue!


Doesn't preload in invisible iframe

I have a leaflet map inside an iframe. I start the iframe with css display:none and after ten seconds I show it with css display:block.
The map tiles only start to load when display:block is set. What browser event causes them to load? How can I set the iframe up so that they preload before it's visible?

Can you publish a fiddle/codepen/playground that displays this behaviour?

I'm not saying this is a bug. But I can't find what causes tiles to load in the source. I'm looking to make my iframe work with all sorts of sites and clearly I'm not simulating the environment correctly

@BryanCrotaz I would check to see that the map container has the correct size; often, an invisible container will have a zero size, which causes Leaflet to not load any tiles.
But, as @IvanSanchez says, it's hard to tell the cause without seeing an example of what you're doing.

So are the tiles loaded on body.onload, or on a resize event?

They are loaded once you set the view on the map (setView, fitBounds, etc.) as long as the tile layer is also added. As mentioned, the containers size is then used to figure out which tiles actually fit into the view.
Leaflet normally tracks window resize (http://leafletjs.com/reference-1.2.0.html#map-trackresize), but not if the map container changes size. If the container changes size, you have to call  invalidateSize to inform Leaflet of the size change, and it will load tile accordingly.
I'm going to close this issue now, since it's turning into more of a support question than an issue report/feature request.


Readme: use relative paths to CONTRIBUTING.md and contributors

It's easier to maintain.

Love it. Thanks @TomasVotruba!

You're fast @reinink, thanks!


Jdbc internal error

when using with dbeaver 4.2.4 with jdbc driver of 0.6.4, got Query execution failed
Internal jdbc driver error
read timeout must be >= 0
read timeout must be >= 0
what am i missing?
Thanks

Do you have stack trace? Is the same query working in cqlsh?


official Container at Docker Hub

Currently there are many inofficial containers at Docker Hub.
None of them seem to be official, some of them are abandoned.
So maybe it would be good to have an official container that will be built automatically for every release (i.e. with GitHub actions or so).
PS: Thank you so much for timeoff-management :raised_hands:

Did you at least succeed trying to run it @alexanderadam? I'm trying to run current version code with dockerfile and it doesn't seem to work at all...

I got it up and running with some warnings and probably security risks. It could be done better than I did for sure but I can share my at least working config later today here! :)

cat docker-compose.yml
version: "3.8"

services:
  app:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: timeoff
#    ports:.     #opening ports is not needed with traefik. If you use nginx/apache uncomment this
#      - 3000:3000
    networks:
      - proxy
    volumes:
      - ./Container-Daten/timeoff/config/app.json:/app/timeoff-management/config/app.json
      - ./Container-Daten/timeoff/db/db.development.sqlite:/app/timeoff-management/db.development.sqlite  #launch First time without this!
    labels:
      - com.centurylinklabs.watchtower.enable=false
      - "traefik.enable=true"
      - "traefik.http.routers.timeoff.rule=Host(`timeoff.example.com`)"  #set url (timeoff.example.com)
      - "traefik.http.routers.timeoff-secure.rule=Host(`timeoff.example.com`)"  #set url (timeoff.example.com)
      - "traefik.http.routers.timeoff.entrypoints=http"
      - "traefik.http.middlewares.timeoff-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.timeoff.middlewares=timeoff-https-redirect"
      - "traefik.http.routers.timeoff-secure.entrypoints=https"
      - "traefik.http.routers.timeoff-secure.tls=true"
      - "traefik.http.routers.timeoff-secure.tls.certresolver=http"
      - "traefik.http.routers.timeoff-secure.service=timeoff"
      - "traefik.http.services.timeoff.loadbalancer.server.port=3000"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.timeoff-secure.middlewares=secHeaders@file"

networks:
  proxy:  ###
    external: true

cat app/Dockerfile
FROM node:13-alpine

EXPOSE 3000

RUN apk update
RUN apk upgrade
#Install dependencies
RUN apk add \
    nodejs npm \
    git \
    make \
    python3 \
    g++ \
    gcc \
    libc-dev \
    clang

#Update npm
RUN npm install -g npm

#Add user so it doesn't run as root
RUN adduser --system app --home /app
USER app
WORKDIR /app

#clone app
RUN git clone https://github.com/timeoff-management/application.git timeoff-management

WORKDIR /app/timeoff-management

#bump formidable up a version to fix user import error.
RUN sed -i 's/formidable"\: "~1.0.17/formidable"\: "1.1.1/' package.json

#install app
RUN npm install -y

CMD npm start

cat Container-Daten/timeoff/config/app.json (https://raw.githubusercontent.com/timeoff-management/timeoff-management-application/master/config/app.json)
{
  "allow_create_new_accounts" : true,
  "send_emails"              : true,
  "application_sender_email"<EMAIL_ADDRESS>  "email_transporter" : {
    "host" : "mail.example-server.de",
    "port" : 587,
     "secureConnection" : false,
     "tls" : {
       "ciphers" : "SSLv3"
     },
    "auth" : {
      "user"<EMAIL_ADDRESS>      "pass" : "******YOUR_PASSWORD******"
    }
  },
  "sessionStore": {
    "useRedis": false,
    "redisConnectionConfiguration": {
      "host": "mail.example-server.de",
      "port": 6379
    }
  },
  "ga_analytics_on" : false,
  "crypto_secret" : "!2~`HswpPPLa22+=±§sdq qwe,appp qwwokDF_",
  "application_domain" : "https://timeoff.example.com",
  "promotion_website_domain" : "https://timeoff.example.com",
  "locale_code_for_sorting": "de"
}

The db file will be created after first launch. That means remove the db volume in the docker-compose.yml for the first launch. Then you need to copy the file from the container:
docker cp timeoff:/app/timeoff-management/db.development.sqlite Container-Daten/timeoff/db/
Now you can add the db volume to the docker-compose.yml again and your changes will be stored permanently.
If you have any tips or a better configuration, please share it. Also feel free to ask any questions about my setup which is again NOT ideal 😜

Thanks a lot @Sim0nW0lf, it really helped me to run it too :)
The only problem I'm facing now is smtp config... did try with a gmail account? I've tried many config examples and none of them seem to work...

SMTP is working for me, so it should be possible!
I am not using gmail here but I might be able to help you:
To use gmail as smtp server you have to generate an app password and use that for loging in!
https://support.google.com/accounts/answer/185833?hl=en
This is what you config should probably look like:
cat Container-Daten/timeoff/config/app.json
{
  "allow_create_new_accounts" : true,
  "send_emails"              : true,
  "application_sender_email"<EMAIL_ADDRESS>  "email_transporter" : {
    "host" : "smtp.gmail.com",
    "port" : 465,
     "secureConnection" : false,
     "tls" : {
       "ciphers" : "SSLv3"
     },
    "auth" : {
      "user"<EMAIL_ADDRESS>      "pass" : "******YOUR_PASSWORD******"
    }
  },
  "sessionStore": {
    "useRedis": false,
    "redisConnectionConfiguration": {
      "host": "smtp.gmail.com",
      "port": 6379
    }
  },
  "ga_analytics_on" : false,
  "crypto_secret" : "!2~`HswpPPLa22+=±§sdq qwe,appp qwwokDF_",
  "application_domain" : "https://timeoff.example.com",
  "promotion_website_domain" : "https://timeoff.example.com",
  "locale_code_for_sorting": "de"
}

Please let me know it that worked out for you :)

Funciono Works! Thank you


Do not panic with null volume configurations

Fix for #430.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "volume_panic" git@github.com:yudai/libcompose.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.


Homework 4 is done without book's examples

@VazgenHarutyunyan4, I don't see your changes.
There is no new commit done by you.

done

I change the Stack data type from int to Integer. As pop() should return null, it should be an array of Integers.
In the pop() method it should be check on top, not on capacity, as capacity is never decremented.


Fixed exception while processing missing file definitions.

While building ZACC documentation, exhale failed during parsing doxygen's XML files. Fixed by converting object (ExhaleNode) to string. Works on my machine after modifying package files (needed a fix ASAP).
Error message:
(!) Exception caught while parsing:Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/exhale/deploy.py", line 378, in explode
textRoot.parse()
File "/usr/local/lib/python3.6/site-packages/exhale/graph.py", line 979, in parse
self.discoverAllNodes()
File "/usr/local/lib/python3.6/site-packages/exhale/graph.py", line 1226, in discoverAllNodes
"{0}{1}".format(bullet, "\n{0}".format(bullet).join(c for c in candidates))
TypeError: sequence item 0: expected str instance, ExhaleNode found

Whoops -- that's not very helpful!  Your PR looked good, but I went with something a little different.  Can you try the master branch as of 5db06fa5938b18ffa068b10803f45640097955b7?
If it's not clear, in testing I never actually ran into this scenario.  I wanted to include the location in the print out, which required a different change.  I was also being lazy about building the error message, so I fixed that too.  It should work as expected now.
Thanks for bringing it to my attention!  Please feel free to submit an issue with a link to the docs / the actual error message being printed here.  The "automatic" file definition inference code could probably be better, having a specific use-case would be helpful in determining how it could change.

needed a fix ASAP

I'm going to update the licenses and fix a couple of links and then release a new version on PyPi if the above commit works for you.  Something on the master that's not released officially is namespace documentation...

Now, it does not fail anymore during parse XML sequence, but failed with "Path too long" errors during the rst generation. (A filename containing the entire SFINAE sequence in a specialization has been generated).
In my project, I use a lot of template-metaprogramming and had to disable some faulty parts with doxygen commands for now. After those changes, the documentation build process succeeded.

Cool, thanks for the feedback.

I use a lot of template-metaprogramming and had to disable some faulty parts with doxygen commands for now

Yeah...this has been a particularly uphill battle for me as well.  Though the generated URLs would be less clear for a human, I think I should just change everything to be something like api/class_{doxygen refid}.rst.  Template classes are pretty atrocious right now with Exhale :/  Especially the internal links generated...

Though the generated URLs would be less clear for a human
I guess, SFINAE is not human-readable :)

Released in v0.1.6 on PyPi.  Thanks again for bringing up the issue!
I'm getting close to being able to release my own C++ project, which has a lot of SFINAE / meta stuff going on.  I'll be revisiting the naming schemes before that goes public.  Did you want me to ping you when that gets released (v0.1.7)?

The new exhale release works as expected :)
Of course, you can ping me so that I can give the new version a try


Inheriting Element ?

class MyElement : Element
{
}

public class App
{
        [Ready]
        public static void Main()
        {
            Document.Body.AppendChild(new MyElement());
        }
}

giving me:
Uncaught TypeError: Illegal constructor
Uncaught TypeError: Failed to execute 'appendChild' on 'Node': parameter 1 is not of type 'Node'.

Yes, you would rather do this like below:
    [Ignore]
    [Name("HTMLMyElement")]
    class MyElement : Element
    {
        [Template("document.createElement('myelement')")]
        public MyElement()
        {
        }
    }

Reference: BR element definition on Bridge
You can see that your element tag was created by your browser's debug tools' DOM explorer after running the script.

does that also register the element with document.registerElement('HTMLMyElement') ?

No, it doesn't. It just allows you to use an element (like BR, LI, UL, BUTTON and so on) assuming it is already registered. You can try the example I provided above and check what it does output in the DOM and how the page is affected with the addition. The hidden class for the element must contain stubs for any specific attribute/method the element might support in javascript.
If you have the opportunity, please browse the source code for other defined elements to see how they've been defined in Bridge.
In the code we would not have any user-registered elements as far as I know -- as the pointed directory is for the HTML5 module, all elements there are expected to be supported by an html5-aware browser.

It looks like we missed document.registerElement() method to be added to the Document API. Though, it might be not added for reason - the method is not supported in IE and Safari. I will try to clarify.

Created a dedicated issue for document.registerElement() support - #389.
@matantsu, I am closing this issue if you don't mind.


Load Environment files according to XDG_CURRENT_DESKTOP, not WM ID

cycle through $XDG_CURRENT_DESKTOP and load matching wayland-session-env-*

45eb30f8c21f626c6b8fe48667f4653e46db81b0


Update how-to-set-up-link-branding.md

Typo
Description of the change:
Reason for the change:
Link to original source:

Closes #

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closing: more than 90 days without a signed CLA


Extra Dependency

In the documentation are reported the following dependencies:

numpy
matplotlib
B-splines

It seems that there is one extra dependency on the package tqdm

Fixed it and updated the equation also now it is a 3TM equation


SpongeForge Fix Please

I dont really know if this is a problem with spongeforge itself or anything, its just that it doesnt wanna fully boot up my server once i try, i tried before with different mods and everything worked without issues. idk if its a plugin/mod thats broken, or it just spongeforge itself, but if its due to the mod not being compatible with spongeforge (which i think so because it never mentioned anything about spongeforge but spongeforge can run spigot plugins and forge mods.) please fix/add this asap!
thank you for reading

Sponge and Spigot have completely different API. Spigot plugins can't run od Sponge. Only some engines like Magma or Mohist can do that.

As @P3ridot said, there's no way that you can run a spigot-only plugin on the Sponge engine. Some plugins just have a support for the Sponge and the Spigot API at the same time. Currently we don't support Sponge API, but we may add a support for that in the future - we'll think of that, so this issue won't be closed for now.


Log level at loggable instances is ignored

dropping level of the instance to DEBUG still does not emit entries into log flow.

can't reproduce in 1.1.0+


Jump list?

Would it make sense for Cursor:setPos() to push a position to the jump list as a rule? Or is that the wrong layer to do that? Would separate cursors need to maintain their own jump lists, or would the global jump list just pertain to the main cursor?
Context: in my mappings for prev/next/first/last cursor with setCursorsEnabled(false) (cf. #24), I manually added a vim.cmd.normal({ "m'", bang = true }) before moving the main cursor. I haven't thought very far through the implications for the whole plugin, but it got me wondering about how the jump list gets along with multiple cursors in general and whether there was anything "smart" to do automatically. Might be a rabbit hole, I dunno. 😅

It's completely by accident that the jumplist works at all for multiple cursors. I never put any effort into it and it just works (although I think there are oddities if you try to jump back multiple times). I should probably keep a separate jumplist per cursor to make sure it is consistent. Then yeah, setpos should push to the jumplist with an option not to.

undolist is pretty shit to work with but we have mc.jumpForward and mc.jumpBackward which you can map to <c-i> and <c-o>. not perfect and doubt it can be.


💄 convert editor acceptance tests to async/await

no issue

async/await cuts down on some boilerplate andThen(() => { ... }) code and makes it more explicit which methods are async and will wait for run-loop/promise behaviour to finish before continuing

Starting with the single acceptance test file to demonstrate the concept and get some buy-in for a decision to make this the default going forwards.

Coverage remained the same at 73.187% when pulling 61c93f2aebfdc4af8219f0d32fc3ae0ba478a912 on kevinansfield:async-await-acceptance-editor into c1a9726f1bc80ce08e175a0e43e3c1ee51b8344f on TryGhost:master.


files copied to where the daemon is started/restarted

When a gmx_### command is run, the files returned from these commands should return to the cwd from where the command is run. This isn't the case when the daemon is restarted from another directory. This is an issue when aiida-gromacs commands are run from different directories.

Seems that when the cwd is set in the parser, aiida defaults to using the path where the daemon was started from, this doesn't seem to be the case when a cwd is set from a calculation class.


[DRAFT] Refactoring Table problem handling

Pull Request Description
Closes #7514
Important Notes

Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[ ] The documentation has been updated, if necessary.
[ ] Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
[ ] All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:

[ ] Unit tests have been written where possible.
[ ] If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Note that during the migration we may need some workarounds to tie the old and new problem aggregation methods. These workarounds may not look perfect and they do not show a good representation of how the final approach will look like - it will start to look good once the ProblemAggregator is used all the way down the call chain. Whereas in the first steps, I migrate only some parts and do some conversions to translate between the approaches - these translations are not most pretty, but they allow us to perform the change in an incremental manner.

First small piece done: I managed to remove getProblems from Table (which was badly tying data to problems about operations) - now some of these operations already rely on the aggregator, altough most of the inner workings still do the handling the old way.
All tests are still passing so no regressions.
We can continue this way, moving stuff piece by piece to the new approach.

Another big piece done:

removed Problems attached to Builder and the whole sealWithProblems method that was a bit of a workaround. Now the problems are properly handled through ProblemAggregator passed as input.
Also adapted CastProblemBuilder and MapOperationProblemBuilder to fit into the new architecture.
Removed many instances of WithProblems in favour of ProblemAggregator - e.g. in Excel methods.

All tests are passing.
Still to do:

MultiValueIndex, internals of joins etc.
Aggregator
a tiny bit in DelimitedWriter


Colab Notebook: "Contine Training" does not use the most recent checkpoint version and always continues from Version 0

I have a finetuned model that I trained to produce a Version_0 checkpoint.
I then "Continue Training" it to produce a Version_1 checkpoint.  (To step 424902, as reported by TensorBoard)
I then "Continue Training" it again and it started training from the Verson_0 and went back to step 421483.
In both cases I left the colab unchanged, and my GDrive was also unchanged, and I can see the Version_1 data there.
The only difference is that with my Version_0 I got cold feet when Colab gave me an inactivity warning and stopped the training manually.  But with Version_1 I went past that point and shortly after (at about 2.5 hours run time) Colab kicked me off. So is it possible that if the script gets stopped by colab then the next "continue training" is not aware of that version?
The log gives this warning which seems to be related:-
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:731: LightningDeprecationWarning: `trainer.resume_from_checkpoint` is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with `trainer.fit(ckpt_path=)` instead.
  ckpt_path = ckpt_path or self.resume_from_checkpoint
Restoring states from the checkpoint path at /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt

This is the start of my log:
DEBUG:piper_train:Namespace(dataset_dir='/content/drive/MyDrive/colab/piper/Test', checkpoint_epochs=6, quality='high', resume_from_single_speaker_checkpoint=None, logger=True, enable_checkpointing=True, default_root_dir=None, gradient_clip_val=None, gradient_clip_algorithm=None, num_nodes=1, num_processes=None, devices='1', gpus=None, auto_select_gpus=False, tpu_cores=None, ipus=None, enable_progress_bar=True, overfit_batches=0.0, track_grad_norm=-1, check_val_every_n_epoch=1, fast_dev_run=False, accumulate_grad_batches=None, max_epochs=10000, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, val_check_interval=None, log_every_n_steps=1000, accelerator='gpu', strategy=None, sync_batchnorm=False, precision=32, enable_model_summary=True, weights_save_path=None, num_sanity_val_steps=2, resume_from_checkpoint='/content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt', profiler=None, benchmark=None, deterministic=None, reload_dataloaders_every_n_epochs=0, auto_lr_find=False, replace_sampler_ddp=True, detect_anomaly=False, auto_scale_batch_size=False, plugins=None, amp_backend='native', amp_level=None, move_metrics_to_cpu=False, multiple_trainloader_mode='max_size_cycle', batch_size=12, validation_split=0.01, num_test_examples=1, max_phoneme_ids=None, hidden_channels=192, inter_channels=192, filter_channels=768, n_layers=6, n_heads=2, seed=1234, num_ckpt=0, save_last=True)
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:52: LightningDeprecationWarning: Setting `Trainer(resume_from_checkpoint=)` is deprecated in v1.5 and will be removed in v1.7. Please pass `Trainer.fit(ckpt_path=)` directly instead.
  rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
DEBUG:piper_train:Checkpoints will be saved every 6 epoch(s)
DEBUG:piper_train:0 Checkpoints will be saved
DEBUG:vits.dataset:Loading dataset: /content/drive/MyDrive/colab/piper/Test/dataset.jsonl
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:731: LightningDeprecationWarning: `trainer.resume_from_checkpoint` is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with `trainer.fit(ckpt_path=)` instead.
  ckpt_path = ckpt_path or self.resume_from_checkpoint
Restoring states from the checkpoint path at /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt
DEBUG:fsspec.local:open file: /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/callbacks/model_checkpoint.py:345: UserWarning: The dirpath has changed from '/content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints' to '/content/drive/MyDrive/colab/piper/Test/lightning_logs/version_2/checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.
  warnings.warn(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
2023-12-21 11:23:21.786165: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-21 11:23:21.786214: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-21 11:23:21.787624: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2023-12-21 11:23:22.971882: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
DEBUG:h5py._conv:Creating converter from 7 to 5
DEBUG:h5py._conv:Creating converter from 5 to 7
DEBUG:h5py._conv:Creating converter from 7 to 5
DEBUG:h5py._conv:Creating converter from 5 to 7
INFO:numexpr.utils:NumExpr defaulting to 2 threads.
DEBUG:fsspec.local:open file: /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_2/hparams.yaml
Restored all states from the checkpoint file at /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/data.py:98: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 1. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
  warning_cache.warn(
Epoch: 2378. Steps: 842942

I later tried this again, refreshing the colab notebook page and setting it up the same as before.
Again I got this in the log:
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py:731: LightningDeprecationWarning: `trainer.resume_from_checkpoint` is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with `trainer.fit(ckpt_path=)` instead.
  ckpt_path = ckpt_path or self.resume_from_checkpoint
Restoring states from the checkpoint path at /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt
DEBUG:fsspec.local:open file: /content/drive/MyDrive/colab/piper/Test/lightning_logs/version_0/checkpoints/last.ckpt

Hi,
I've had it in mind to add a filter to check for the latest CKPT and a latest version. I'll probably do it in the future or end up removing the continue training function due to some optimization problems with the model. At the moment, you can move checkpoints that are at a version greater than 0 to version 0 so that they are recognized.

Thanks.  That sorted it.  I renamed my version_1 folder as version_0 and it continued from what was my version_1 to produce effectively a version_2, but named version_1.
It wasn't clear from the notebook that it would only ever continue from version_0, so perhaps that would be a good note to put in there?