[BUG] DNS not resolving

What did you do?

How was the cluster created?

 k3d create -n mycluster

What did you do afterwards?

Start a pod and try a DNS query:
$ export KUBECONFIG="$(k3d get-kubeconfig --name='mycluster')"
$ kubectl run --restart=Never --rm -i --tty tmp --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup www.gmail.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>:53

;; connection timed out; no servers could be reached

/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver <IP_ADDRESS>
options ndots:5
/ # exit

Exec into the k3d container and do the same DNS query:
docker exec -it k3d-endpoint-server sh
/ # nslookup www.gmail.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>:53

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: <IP_ADDRESS>

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: 2607:f8b0:4005:80b::2005

/ # cat /etc/resolv.conf
nameserver <IP_ADDRESS>
options ndots:0
/ # exit

What did you expect to happen?
I would expect the pods in the k3d cluster to be able to resolve DNS names
Which OS & Architecture?
MacOS 10.15.3
Which version of k3d?

output of k3d --version

k3d version v1.7.0

Which version of docker?

output of docker version

docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Hi there, thanks for opening this issue.
As mentioned in the related issue (https://github.com/rancher/k3d/issues/101#issuecomment-603668428), CoreDNS is doing the name resolution inside the cluster.
I couldn't reproduce this on Linux (with the same docker and k3d versions), so I guess that the difference in DNS settings is caused by Docker for Desktop.

To workaround this issue I'm patching the coredns configmap.

@luisdavim what's the patch that you apply?

@luisdavim I'm also interested in what patch you applied

I'm pretty sure that removing the forward . /etc/resolv.conf part is enough.

After more investigation I found that this could be related to the way k3d creates the docker network.
Indeed, k3d creates a custom docker network for each cluster and when this happens resolving is done through the docker daemon. The requests are actually forwarded to the DNS servers configured in your host's resolv.conf. But through a single DNS server (the embedded one of docker).
This means that if your daemon.json is, like mine, not configured to provide extra DNS servers it defaults to <IP_ADDRESS> which does not resolve any company address for example.
It would be useful to have a custom options to provide to k3d when it starts the cluster and specify the DNS servers there, as proposed here https://github.com/rancher/k3d/issues/165

Thanks for your additional input @irizzant , how would you add additional DNS servers here on k3d's side?
The network opts of docker don't seem to have such an option (after having a quick glance over what's available there) :thinking:

Personally I fixed this by injecting a custom ConfigMap for CoreDns, by changing

forward . /etc/resolv.conf

to:

forward . /etc/resolv.conf xxx.xxx.xxx.xxx

replacing the x with the IP of your DNS servers.
I had a quick look to the docker options and I confirm that I don't see an option to configure the custom DNS servers on the docker network.
Maybe a feasible option would be to add a custom flag to k3d command which adds the custom DNS servers to the CoreDns ConfigMap directly.

Currently, k3d doesn't interact with any Kubernetes resources inside the cluster (i.e. in k3s) and I tend to avoid this because of the huge dependencies on Kubernetes libraries it could draw in. Upon cluster creation this could work however by modifying the Chart that's being auto-deployed by k3s. Not sure if this could go into k3s itself instead :thinking:

Maybe interacting with k8s itself it's not needed.
k3s deploys whatever is under /var/lib/rancher/k3s/server/manifests, so you could add a valid CoreDns configuration just customizing the DNS part according to the command line flag.

I seem to be running into this issues about every couple of weeks.  This is the only workaround that seems to "just work" so I can get back to the job I'm paid to do:
sudo iptables -t nat -A PREROUTING -p udp -d <IP_ADDRESS>  --dport 53 -j DNAT --to <your DNS server IP>

This seems to be broken, because the coredns pod does not have an /etc/resolv.conf in it, while the ConfigMap is configured to forward to that.  All the docs have pointed told me that coredns will use the $HOST resolv.conf, but when I used k3d, which uses k3s, the coredns "pod" doesn't run as a container, or as a process on the $HOST.  It runs as a process of containerd, and therefore it doesn't get any of the correct settings.

 I've used it to take advantage of other things registries.yaml, etc) but haven't taken the time to dig into what all it gets mapped to

From what I understand there is nothing soecial. It just mount volume to the container running the k3s server.
Thus I guess mount anything and even maybe do some airgapped setup

Hey folks, sorry for the radio silence, just getting back to k3d now...
Let me reply to some of the messages here.

@irizzant

Maybe interacting with k8s itself it's not needed.
k3s deploys whatever is under /var/lib/rancher/k3s/server/manifests, so you could add a valid CoreDns configuration just customizing the DNS part according to the command line flag.

That's a good starting point. Unfortunately, this would require us to write the file to disk and bind mount it into the container, as exec'ing into it afterwards to update the ConfigMap manifest, wouldn't update the actual thing inside the cluster (IIRC, there is no loop to do so). It's definitely doable, but we'd need to keep state somewhere and react to changes k3s does in the auto-deploy manifests.

@dminca

This only happens to me if I deploy something in the default namespace, in other namespaces it worked just fine, idk why

This is for real the weirdest thing on this thread that you're experiencing :thinking: No clue, what's going on there..

@yamlCase

I seem to be running into this issues about every couple of weeks. This is the only workaround that seems to "just work" so I can get back to the job I'm paid to do:
sudo iptables -t nat -A PREROUTING -p udp -d <IP_ADDRESS>  --dport 53 -j DNAT --to <your DNS server IP>

Are you executing this on your local host (I assume so because of the sudo) to just route all the Google-DNS traffic (default Docker DNS) to your own DNS server?

@eigood & @Athosone

This seems to be broken, because the coredns pod does not have an /etc/resolv.conf in it, while the ConfigMap is configured to forward to that. All the docs have pointed told me that coredns will use the $HOST resolv.conf, but when I used k3d, which uses k3s, the coredns "pod" doesn't run as a container, or as a process on the $HOST. It runs as a process of containerd, and therefore it doesn't get any of the correct settings.

What do you mean by "it doesn't run as a container"? It surely is running in a container :thinking:

For those who have the problem a simple fix is to mount your /etc/resolve.conf onto the cluster:
k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf

Also, those two statements seem to conflict, right?

@yamlCase

What does that --volume command do exactly? I've used it to take advantage of other things registries.yaml, etc) but haven't taken the time to dig into what all it gets mapped to

It's doing basically the same as docker's --volume flag: bind-mounting a file or a directory into one or more containers, overlaying anything that might already be there (you can specify the access mode).

Is anyone experiencing this on a Linux machine or in WSL2, i.e. in Docker versions which do not run inside a VM?
I could imagine, that we can modify the CoreDNS ConfigMap in the running cluster like we might be doing it to inject the host.k3d.internal entry in #360 .
However, just mounting in your resolv.conf might be the easiest solution :thinking:

Well maybe it conflicts but for me it solves the problem

@Athosone , I meant, that CoreDNS does indeed pick up the resolv.conf, as your solution works :+1:
I guess the problem is more that the "default" resolv.conf in the k3s container (node) does not work for it (and that one you're effectively replacing with the volume mount) :+1:

Just as a clarification, if you modify the docker daemon configuration (daemon.json) to add the company DNS and then you launch the k3d cluster you can "docker exec" into the containers where k8s running and you'll see that nslookup finds DNS entries served by company DNS.
Consequently the docker container in which k8s is running picks up the right resolv.conf configuration.
The problem is that the default resolv.conf in k8s just reports:
forward . /etc/resolv.conf

which forces CoreDNS to use its own resolv.conf bypassing the docker one.

When I run k3d cluster create, I can certainly volume mount $HOST files into the docker containers(agent+server).  However, containerd is then used to start the CoreDNS pod, running inside k8s.  It is this internal container that needs to have an /etc/resolv.conf.  Nothing I do to the k3d command will allow me to adjust the internal containerd/pod that is created.
I did a bit of research, trying to figure out that this was the case, by figuring out where containerd stores it's filesystems.

I am running k3d in a Docker that runs in Virtual Box ( actually I'm using Docker Toolbox for Windows product).
And experiencing the same problem

inside a docker container  rancher/k3s:v1.18.8-k3s1 resolutions of external and custom domains is happening
inside pods - fails.

Mounting the resolv.conf works but this can't be the best solution. If you missed that oppertunity at the beginning you need to re-create a k3d cluster just to mount that volume.

forward . /etc/resolv.conf xxx.xxx.xxx.xxx

This one did not work for me.

For those who have the problem a simple fix is to mount your /etc/resolve.conf onto the cluster:
k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf

VERY HELPFULL HINT

Can someone give those binaries a try and use the experimental --enable-dns-magic flag to try to solve this problem similar to how kind does it?
https://drive.google.com/drive/folders/1Hu4Z0fAYr4ktw7r07naBUxOJY9EYr0Bq?usp=sharing
They are built from feature/dns

@iwilltry42

Windows 10 20H2
Docker Toolbox  ( ie VirtualBox VM with Linux + Docker)
Msys2 based bash as a shell

$ ./k3d-windows-amd64.exe --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port <IP_ADDRESS>:6550
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Network with name 'k3d-k3s' already exists with ID '856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a'
INFO[0000] Created volume 'k3d-k3s-images'
INFO[0001] Creating node 'k3d-k3s-server-0'
DEBU[0001] Created container k3d-k3s-server-0 (ID: 4711af3bcb9c3efb41d4e5bee47ce846e96afda76d9c69fc703d3ea72c656612)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0066] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0066] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
DEBU[0066] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
DEBU[0067] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
ERRO[0067] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0067] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0067] Enabling DNS Magic...
DEBU[0067] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
DEBU[0069] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
ERRO[0069] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0069] Failed to enable DNS Magic: %!w(*errors.errorString=&{Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response})
INFO[0069] Cluster 'k3s' created successfully!
DEBU[0069] Updating default kubeconfig with a new context for cluster k3s
DEBU[0069] Setting new current-context 'k3d-k3s'
DEBU[0070] Wrote kubeconfig to 'C:\Users\eshep/.kube/config'
INFO[0070] You can now use it like this:
kubectl cluster-info

Apparently DNS resolution not working since magic not applied.

@eshepelyuk that seems to be quite a special case. Can you please try again with --trace enabled (You can trim the log output after the DNS magic.
I have no experience with Docker Toolbox for Windows. Do you know, how it refers to the Host system?
We're trying docker-machine ip and the host.docker.internal on Windows.

Hello @iwilltry42

Adding requested logs.

Please note that I ran docker-machine ip before k3d and it returned proper address of VirtualBox VM with Docker inside,
the same address I've passed via --api-port.

Another question, is there a way to pass that host.docker.internal to k3d command ?

Here are the logs, click to expand
$ docker-machine.exe ip
<IP_ADDRESS>

$ ./k3d-windows-amd64.exe --trace --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port $(docker-machine ip):6550
DEBU[0000] Selected runtime is 'docker.Docker'
TRAC[0000] PortFilterMap: map[]
TRAC[0000] LabelFilterMap: map[]
TRAC[0000] EnvFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Network with name 'k3d-k3s' already exists with ID '856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a'
INFO[0000] Created volume 'k3d-k3s-images'
INFO[0001] Creating node 'k3d-k3s-server-0'
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-server-0 Role:server Image:docker.io/rancher/k3s:v1.19.3-k3s3 Volumes:[k3d-k3s-images:/k3d/images] Env:[K3S_TOKEN=jWiJFvqWengrHwISsBKh] Cmd:[] Args:[] Ports:[<IP_ADDRESS>:6550:6443/tcp] Restart:false Labels:map[k3d.cluster:k3s k3d.cluster.imageVolume:k3d-k3s-images k3d.cluster.network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a k3d.cluster.network.external:true k3d.cluster.token:jWiJFvqWengrHwISsBKh k3d.cluster.url:https://k3d-k3s-server-0:6443] Network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:<IP_ADDRESS> HostIP:<IP_ADDRESS> Port:6550}} AgentOpts:{} GPURequest: State:{Running:false Status:}}

TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=jWiJFvqWengrHwISsBKh K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san <IP_ADDRESS>] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.19.3-k3s3 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s k3d.cluster.imageVolume:k3d-k3s-images k3d.cluster.network:856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a k3d.cluster.network.external:true k3d.cluster.token:jWiJFvqWengrHwISsBKh k3d.cluster.url:https://k3d-k3s-server-0:6443 k3d.role:server k3d.server.api.host:<IP_ADDRESS> k3d.server.api.hostIP:<IP_ADDRESS> k3d.server.api.port:6550] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:<IP_ADDRESS> HostPort:6550}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc0002cc06a} NetworkingConfig:{EndpointsConfig:map[856209b2e35cf43c5d5b51d78757c4e57b828c2d8c24c7888a8c3ea2e420816a:0xc000194840]}}
DEBU[0001] Created container k3d-k3s-server-0 (ID: c8d4f2fd7253433d4fb94be5015743882750e31ebdf6a4f1bad599da67efe720)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0059] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0059] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0059] Runtime GOOS: windows
DEBU[0059] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0059] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0060] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0060] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0060] Scanning Log Line 'Server:           <IP_ADDRESS>'
TRAC[0060] Scanning Log Line 'Address:  <IP_ADDRESS>:53'
TRAC[0060] Scanning Log Line ''
TRAC[0060] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0060] Scanning Log Line ''
TRAC[0060] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0060] Scanning Log Line ''
ERRO[0060] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0060] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0060] Enabling DNS Magic...
TRAC[0060] Runtime GOOS: windows
DEBU[0060] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0060] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0061] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0061] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0061] Scanning Log Line 'Server:           <IP_ADDRESS>'
TRAC[0061] Scanning Log Line 'Address:  <IP_ADDRESS>:53'
TRAC[0061] Scanning Log Line ''
TRAC[0061] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0061] Scanning Log Line ''
TRAC[0061] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0061] Scanning Log Line ''
ERRO[0061] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0061] Failed to enable DNS Magic: %!w(*errors.errorString=&{Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response})
INFO[0061] Cluster 'k3s' created successfully!

..........

@eshepelyuk , thank you!
I just realized, that for the DNS magic (and all other related parts), we're not trying docker-machine ip. I added this now and uploaded a new build to the google drive folder (or you build it yourself).
Thanks for testing!

Hello @iwilltry42
Just downloaded a new build from the link provided above.
The error and behaviour are the same.
$ ./k3d-windows-amd64.exe --version
k3d version v3.3.0-2-g296bd5c
k3s version v1.19.3-k3s3 (default)

$ ./k3d-windows-amd64.exe --trace --verbose cluster create k3s --enable-dns-magic --wait --no-lb --api-port $(docker-machine ip):6550

.....

DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0062] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0062] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0062] Runtime GOOS: windows
DEBU[0062] Docker Machine found: default
DEBU[0062] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0063] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0064] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0064] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0064] Scanning Log Line 'Server:           <IP_ADDRESS>'
TRAC[0064] Scanning Log Line 'Address:  <IP_ADDRESS>:53'
TRAC[0064] Scanning Log Line ''
TRAC[0064] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0064] Scanning Log Line ''
TRAC[0064] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0064] Scanning Log Line ''
ERRO[0064] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0064] Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0064] Enabling DNS Magic...
TRAC[0064] Runtime GOOS: windows
DEBU[0064] Docker Machine found: default
DEBU[0064] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-server-0'
TRAC[0064] Exec process '[sh -c nslookup host.docker.internal]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0065] Exec Process Failed, but we still got logs, so we're at least trying to get the IP from there...
TRAC[0065] -> Exec Process Error was: Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
TRAC[0065] Scanning Log Line 'Server:           <IP_ADDRESS>'
TRAC[0065] Scanning Log Line 'Address:  <IP_ADDRESS>:53'
TRAC[0065] Scanning Log Line ''
TRAC[0065] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0065] Scanning Log Line ''
TRAC[0065] Scanning Log Line '** server can't find host.docker.internal: NXDOMAIN'
TRAC[0065] Scanning Log Line ''
ERRO[0065] Exec process in node 'k3d-k3s-server-0' failed with exit code '1'
WARN[0065] Failed to enable DNS Magic: Failed to get HostIP: Failed to read address for 'host.docker.internal' from nslookup response
INFO[0065] Cluster 'k3s' created successfully!

....

@eshepelyuk thanks again for testing!  No errors in logs, though.
DEBU[0001] Created container k3d-k3s-server-0 (ID: f34492b4b1bf149d86f0957a5b20aa82d56ef3213cfb4f65fc08bb32bf1104d8)
DEBU[0002] Created node 'k3d-k3s-server-0'
DEBU[0002] Waiting for server node 'k3d-k3s-server-0' to get ready
DEBU[0080] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-server-0'
INFO[0080] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0080] Runtime GOOS: windows
DEBU[0080] Docker Machine found: default
DEBU[0081] Adding extra host entry '<IP_ADDRESS> host.k3d.internal'...
DEBU[0081] Executing command '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-server-0'
TRAC[0081] Exec process '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0082] Exec process in node 'k3d-k3s-server-0' exited with '0'
DEBU[0082] Executing command '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' in node 'k3d-k3s-server-0'
TRAC[0082] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0083] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0084] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0085] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0086] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0087] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0088] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
TRAC[0089] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0090] Exec process in node 'k3d-k3s-server-0' exited with '0'
INFO[0090] Successfully added host record to /etc/hosts in 1/1 nodes and to the CoreDNS ConfigMap
INFO[0090] Enabling DNS Magic...
TRAC[0090] Runtime GOOS: windows
DEBU[0090] Docker Machine found: default
DEBU[0091] Executing command '[sh -c iptables-save | sed -e 's/--to-source :53/--to-source <IP_ADDRESS>:53/g' -e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/g' -e 's/-d <IP_ADDRESS>/-d <IP_ADDRESS>/g' | iptables-restore && cp /etc/resolv.conf /etc/resolv.conf.original && sed -e 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/resolv.conf.original >/etc/resolv.conf]' in node 'k3d-k3s-server-0'
TRAC[0091] Exec process '[sh -c iptables-save | sed -e 's/--to-source :53/--to-source <IP_ADDRESS>:53/g' -e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/g' -e 's/-d <IP_ADDRESS>/-d <IP_ADDRESS>/g' | iptables-restore && cp /etc/resolv.conf /etc/resolv.conf.original && sed -e 's/<IP_ADDRESS>/<IP_ADDRESS>/g' /etc/resolv.conf.original >/etc/resolv.conf]' still running in node 'k3d-k3s-server-0'.. sleeping for 1 second...
DEBU[0092] Exec process in node 'k3d-k3s-server-0' exited with '0'
INFO[0092] Successfully applied DNS Magic to 1/1 nodes
INFO[0092] Cluster 'k3s' created successfully!

Yeah, so it seems like it at least achieved what should've been achieved.
The change is similar to what kind does to "fix" DNS resolution on some systems: Set the host IP as a nameserver in /etc/resolv.conf and configure the iptables rules accordingly.
Seems like this doesn't give us anything though :thinking:
I'll research a bit more on this. Any input is highly appreciated :)

I am having the exact same issue @eshepelyuk has - inside k3s container all custom domains are seen without any issue but inside any pod they're not resolved. Linux here, newest k3d (installed today). Forward trick didn't work either.

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

@avodaqstephan

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

That's k3s itself, which deploys some manifests automatically upon startup: https://rancher.com/docs/k3s/latest/en/advanced/#auto-deploying-manifests

@avodaqstephan

Workaround: Edit the coredns configmap

forward . xxx.xxx.xxx.xxx /etc/resolv.conf

Problem: This is not persistent if you restart k3d. That is actually the real issue! Which component is responsible for re-writing / resetting changes in the configmap of coredns?

That's k3s itself, which deploys some manifests automatically upon startup: https://rancher.com/docs/k3s/latest/en/advanced/#auto-deploying-manifests

Just tried v4.0.0 and the error still exists

Just tried v4.0.0 and the error still exists

But I have to say that adding volume /etc/resolv.conf solved the issue completely and doesn't reqire patching CoreDNS.
My config file
apiVersion: k3d.io/v1alpha1
kind: Simple
name: k3s
volumes:
  - volume: /c/Users/eshep/.k3s/manifests/cm.yaml:/var/lib/rancher/k3s/server/manifests/cm.yaml
    nodeFilters:
      - all
  - volume: /etc/resolv.conf:/etc/resolv.conf
    nodeFilters:
      - all
options:
  k3d:
    wait: true
    disableLoadbalancer: false

But I have to say that adding volume /etc/resolv.conf solved the issue completely and doesn't reqire patching CoreDNS.
My config file
apiVersion: k3d.io/v1alpha1
kind: Simple
name: k3s
volumes:
  - volume: /c/Users/eshep/.k3s/manifests/cm.yaml:/var/lib/rancher/k3s/server/manifests/cm.yaml
    nodeFilters:
      - all
  - volume: /etc/resolv.conf:/etc/resolv.conf
    nodeFilters:
      - all
options:
  k3d:
    wait: true
    disableLoadbalancer: false

I seem to be having a similar issue with Docker Desktop for macOS and the Palo Alto Networks Global Protect VPN.
When the VPN is up, regular Docker containers don't have any problems resolving DNS via the resolver that is pushed to the client at VPN connect.  However, under k3d, no containers in any pods can resolve those same addresses.
I haven't tried to map the resolv.conf into the coredns container yet.  Why doesn't k3d/k3s just use the host resolver?

I've attempted quite a few of the above solution recommendations with no success.
I've documented the setup and logs here:
https://gist.github.com/ericis/c3c0b9c2fbb78fdf56b2193a88c35f6f
INFO[0005] Starting Node 'k3d-dev-serverlb'
INFO[0005] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
WARN[0006] Failed to patch CoreDNS ConfigMap to include entry '<IP_ADDRESS> host.k3d.internal': Exec process in node 'k3d-dev-server-0' failed with exit code '1'

Windows 10 Enterprise v1903  + WSL2 + Ubuntu Focal + k3d v4.2.0 (k3s v1.20.2-k3s1)

@cro

Why doesn't k3d/k3s just use the host resolver?

How would you suggest that k3d achieves this? We could search for e.g. the /etc/resolv.conf, but that's not platform-agnostic. Then people may as well use different nameresolvers, which k3d would have to account for, so we probably could only have a workaround for some people and not for others.
Additionally, there's a challenge to react to changes during the cluster runtime, e.g. when someone (de-)activates a VPN connection that pushes DNS settings.
I'd be really happy in general to finally find a generic fix for this :thinking:
@ericis That warning in your logs is not critical for the cluster and does not affect the general DNS issues (it's just an optional tweak that allows you to address your host system from inside the cluster by name). However, the fact that the exec process fails may be a hint, that something else is ggoing wrong there. Can you share the logs of the server container (docker logs k3d-dev-server-0) in the Gist?

Hi, all
One interesting observation. Am on Win10, using Docker Toolbox (i.e. VirtualBox VM running Docker).
After removing all the VPN - I've discovered that mounting /etc/resolv.conf is braeking DNS resolution from within pods :)
As soon as I removed mounting /etc/resolv.conf - resolution worked fine.

some quick-and-dirty bash for patching the coredns configmap with the nameservers in /etc/resolv.conf:
NAMESERVERS=`grep nameserver /etc/resolv.conf | awk '{print $2}' | xargs`
cmpatch=$(kubectl get cm coredns -n kube-system --template='{{.data.Corefile}}' | sed "s/forward.*/forward . $NAMESERVERS/g" | tr '\n' '^' | xargs -0 printf '{"data": {"Corefile":"%s"}}' | sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$cmpatch"

(code adapted from here)

Another request for this

add a custom flag to k3d command which adds the custom DNS servers to the CoreDns ConfigMap directly.

and another solution for simple editing the coredns configmap (this one adding a known set of needed DNS servers <IP_ADDRESS> and <IP_ADDRESS>)
KUBE_EDITOR="sed -i 's|forward.*|forward . <IP_ADDRESS> <IP_ADDRESS> /etc/resolv.conf|'" \
  kubectl edit -n kube-system cm coredns

Ideally this could be some using something like
k3d cluster create --servers 3 --dns <IP_ADDRESS> --dns <IP_ADDRESS>

Hey there :wave:
Anyone fancy testing https://github.com/rancher/k3d/releases/tag/v5.0.0-alpha.0 and see if that makes the DNS issue a little better?
Inspired by kind's dns magic, we're re-writing the entrypoint to modify iptables and resolv.conf to route DNS requests to the Docker Host instead of the embedded docker dns (that one is <IP_ADDRESS>, which will be filtered out by k3s, which in turn sets <IP_ADDRESS> in the cluster).

@iwilltry42 I just now got to try this and I see the same behavior I commented on earlier (https://github.com/rancher/k3d/issues/209#issuecomment-780173218)  though I did try the current beta.
FWIW I also saw this message bringing up the cluster:
WARN[0041] Failed to get HostIP to inject as host.k3d.internal: failed to resolve 'host.docker.internal' from inside the k3d-tools node: Failed to read address for 'host.docker.internal' from nslookup response

Hi @cro , can you please provide the full --trace logs for this? You're on a Mac, right? Unfortunately I can only test on Linux right now, where it worked with my company VPN as well as with my local non-default PiHole DNS.

@iwilltry42 This doesn't work for me on Linux (company DNS still doesn't resolve). I pasted the trace log below. Anything else I can provide for debugging?
$ ./k3d-linux-amd64 --version
k3d version v5.0.0-alpha.0
k3s version v1.21.4-k3s1 (default)
$ ./k3d-linux-amd64 cluster create --trace dnstest
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:19.03.4 OSType:linux OS:Ubuntu 18.04.1 LTS Arch:x86_64 CgroupVersion: CgroupDriver:cgroupfs Filesystem:extfs}
DEBU[0000] Configuration:
agents: 0
cli:
  k3sargs: []
image: docker.io/rancher/k3s:v1.21.4-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    serversmemory: ""
registries:
  config: ""
  create: false
  use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  ports: []
  runtime-labels: []
  volumes: []
TRAC[0000] Trying to read config apiVersion='k3d.io/v1alpha3', kind='simple'
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha3} Name: Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.21.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[]} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: Labels:[]}} Env:[] Registries:{Use:[] Create:false Config:}}
==========================
TRAC[0000] VolumeFilterMap: map[]
TRAC[0000] PortFilterMap: map[]
TRAC[0000] K3sNodeLabelFilterMap: map[]
TRAC[0000] RuntimeLabelFilterMap: map[]
TRAC[0000] EnvFilterMap: map[]
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha3} Name: Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:45791} Image:docker.io/rancher/k3s:v1.21.4-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[]} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: Labels:[]}} Env:[] Registries:{Use:[] Create:false Config:}}
==========================
DEBU[0000] generated loadbalancer config:
ports:
  6443.tcp:
  - k3d-dnstest-server-0
settings:
  workerProcesses: 1024
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:dnstest Network:{Name:k3d-dnstest ID: External:false IPAM:{IPPrefix:invalid IP prefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0004c4480 0xc0004c4600] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc0004a2c80 ServerLoadBalancer:0xc00039f9b0 ImageVolume:} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] ===== Processed Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:dnstest Network:{Name:k3d-dnstest ID: External:false IPAM:{IPPrefix:invalid IP prefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0004c4480 0xc0004c4600] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc0004a2c80 ServerLoadBalancer:0xc00039f9b0 ImageVolume:} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-dnstest' (7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf)
INFO[0000] Created volume 'k3d-dnstest-images'
TRAC[0000] Using Registries: []
TRAC[0000]
===== Creating Cluster =====

Runtime:
{}

Cluster:
&{Name:dnstest Network:{Name:k3d-dnstest ID:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf External:false IPAM:{IPPrefix:<IP_ADDRESS>/20 IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0004c4480 0xc0004c4600] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc0004a2c80 ServerLoadBalancer:0xc00039f9b0 ImageVolume:}

ClusterCreatOpts:
&{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d k3d.cluster.imageVolume:k3d-dnstest-images k3d.cluster.network:k3d-dnstest k3d.cluster.network.external:false k3d.cluster.network.id:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf k3d.cluster.network.iprange:<IP_ADDRESS>/20] GlobalEnv:[] Registries:{Create:<nil> Use:[] Config:<nil>}}

============================

INFO[0001] Creating node 'k3d-dnstest-server-0'
DEBU[0001] Failed to parse cgroupVersion: strconv.Atoi: parsing "": invalid syntax
TRAC[0001] Creating node from spec
&{Name:k3d-dnstest-server-0 Role:server Image:docker.io/rancher/k3s:v1.21.4-k3s1 Volumes:[k3d-dnstest-images:/k3d/images] Env:[K3S_TOKEN=OfgasfwbWnkBJKjGvAkh] Cmd:[] Args:[] Ports:map[] Restart:true Created: RuntimeLabels:map[app:k3d k3d.cluster:dnstest k3d.cluster.imageVolume:k3d-dnstest-images k3d.cluster.network:k3d-dnstest k3d.cluster.network.external:false k3d.cluster.network.id:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf k3d.cluster.network.iprange:<IP_ADDRESS>/20 k3d.cluster.token:OfgasfwbWnkBJKjGvAkh k3d.cluster.url:https://k3d-dnstest-server-0:6443] K3sNodeLabels:map[] Networks:[k3d-dnstest] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:0xc0004a2c80} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[]}
DEBU[0001] DockerHost:
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-dnstest-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=OfgasfwbWnkBJKjGvAkh K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san <IP_ADDRESS>] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.21.4-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:dnstest k3d.cluster.imageVolume:k3d-dnstest-images k3d.cluster.network:k3d-dnstest k3d.cluster.network.external:false k3d.cluster.network.id:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf k3d.cluster.network.iprange:<IP_ADDRESS>/20 k3d.cluster.token:OfgasfwbWnkBJKjGvAkh k3d.cluster.url:https://k3d-dnstest-server-0:6443 k3d.role:server k3d.server.api.host:<IP_ADDRESS> k3d.server.api.hostIP:<IP_ADDRESS> k3d.server.api.port:45791 k3d.version:v5.0.0-alpha.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-dnstest-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00050374f} NetworkingConfig:{EndpointsConfig:map[k3d-dnstest:0xc0003760c0]}}
DEBU[0001] Created container k3d-dnstest-server-0 (ID: 16f1bcd50b9b040b789fe70bd8f09f50fa3709f5ed61ffa0c344f13ca5b3adf3)
DEBU[0001] Created node 'k3d-dnstest-server-0'
INFO[0001] Creating LoadBalancer 'k3d-dnstest-serverlb'
DEBU[0001] Failed to parse cgroupVersion: strconv.Atoi: parsing "": invalid syntax
TRAC[0001] Creating node from spec
&{Name:k3d-dnstest-serverlb Role:loadbalancer Image:docker.io/rancher/k3d-proxy:5.0.0-alpha.0 Volumes:[k3d-dnstest-images:/k3d/images] Env:[] Cmd:[] Args:[] Ports:map[6443:[{HostIP:<IP_ADDRESS> HostPort:45791}]] Restart:true Created: RuntimeLabels:map[app:k3d k3d.cluster:dnstest k3d.cluster.imageVolume:k3d-dnstest-images k3d.cluster.network:k3d-dnstest k3d.cluster.network.external:false k3d.cluster.network.id:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf k3d.cluster.network.iprange:<IP_ADDRESS>/20 k3d.cluster.token:OfgasfwbWnkBJKjGvAkh k3d.cluster.url:https://k3d-dnstest-server-0:6443] K3sNodeLabels:map[] Networks:[k3d-dnstest] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:invalid IP Static:false} HookActions:[{Stage:preStart Action:{Runtime:{} Content:[112 111 114 116 115 58 10 32 32 54 52 52 51 46 116 99 112 58 10 32 32 45 32 107 51 100 45 100 110 115 116 101 115 116 45 115 101 114 118 101 114 45 48 10 115 101 116 116 105 110 103 115 58 10 32 32 119 111 114 107 101 114 80 114 111 99 101 115 115 101 115 58 32 49 48 50 52 10] Dest:/etc/confd/values.yaml Mode:-rwxr--r--}}]}
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-dnstest-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-proxy:5.0.0-alpha.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:dnstest k3d.cluster.imageVolume:k3d-dnstest-images k3d.cluster.network:k3d-dnstest k3d.cluster.network.external:false k3d.cluster.network.id:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf k3d.cluster.network.iprange:<IP_ADDRESS>/20 k3d.cluster.token:OfgasfwbWnkBJKjGvAkh k3d.cluster.url:https://k3d-dnstest-server-0:6443 k3d.role:loadbalancer k3d.version:v5.0.0-alpha.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-dnstest-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443:[{HostIP:<IP_ADDRESS> HostPort:45791}]] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc000503dca} NetworkingConfig:{EndpointsConfig:map[k3d-dnstest:0xc0003766c0]}}
DEBU[0001] Created container k3d-dnstest-serverlb (ID: b4aa22d80bdacac20ff8d6405613f87e484f4036b48797a5688e84d9ec9ab20c)
DEBU[0001] Created loadbalancer 'k3d-dnstest-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
TRAC[0001] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-dnstest-serverlb
TRAC[0001] failed to parse IP '' for container '/k3d-dnstest-serverlb', likely because it's not running (or restarting): ParseIP(""): unable to parse IP
TRAC[0001] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-dnstest-server-0
TRAC[0001] failed to parse IP '' for container '/k3d-dnstest-server-0', likely because it's not running (or restarting): ParseIP(""): unable to parse IP
DEBU[0001] Attaching to cluster's image volume 'k3d-dnstest-images'
INFO[0001] Starting new tools node...

INFO[0002] Deleted k3d-dnstest-tools
INFO[0002] Starting cluster 'dnstest'
TRAC[0002] Servers before sort:
TRAC[0002] Server 0 - k3d-dnstest-server-0
TRAC[0002] Servers after sort:
TRAC[0002] Server 0 - k3d-dnstest-server-0
INFO[0002] Starting servers...
DEBU[0002] Failed to parse cgroupVersion: strconv.Atoi: parsing "": invalid syntax
DEBU[0002] No fix enabled.
DEBU[0002] Node k3d-dnstest-server-0 Start Time: 2021-09-13 23:56:26.193549046 -0500 CDT m=+2.477672464
TRAC[0002] Starting node 'k3d-dnstest-server-0'
INFO[0002] Starting Node 'k3d-dnstest-server-0'
DEBU[0003] Truncated 2021-09-14 04:56:26.950881351 +0000 UTC to 2021-09-14 04:56:26 +0000 UTC
DEBU[0003] Waiting for node k3d-dnstest-server-0 to get ready (Log: 'k3s is up and running')
TRAC[0003] NodeWaitForLogMessage: Node 'k3d-dnstest-server-0' waiting for log message 'k3s is up and running' since '2021-09-14 04:56:26 +0000 UTC'
TRAC[0007] Found target log line: `Mtime="2021-09-14T04:56:31.174637417Z" level=info msg="k3s is up and running"`
DEBU[0007] Finished waiting for log message 'k3s is up and running' from node 'k3d-dnstest-server-0'
INFO[0007] Starting agents...
INFO[0007] Starting helpers...
DEBU[0007] Node k3d-dnstest-serverlb Start Time: 2021-09-13 23:56:31.508273926 -0500 CDT m=+7.792397354
TRAC[0007] Node k3d-dnstest-serverlb: Executing preStartAction 'types.NodeHook'
TRAC[0007] Starting node 'k3d-dnstest-serverlb'
INFO[0007] Starting Node 'k3d-dnstest-serverlb'
DEBU[0008] Truncated 2021-09-14 04:56:32.321394855 +0000 UTC to 2021-09-14 04:56:32 +0000 UTC
DEBU[0008] Waiting for node k3d-dnstest-serverlb to get ready (Log: 'start worker processes')
TRAC[0008] NodeWaitForLogMessage: Node 'k3d-dnstest-serverlb' waiting for log message 'start worker processes' since '2021-09-14 04:56:32 +0000 UTC'
TRAC[0014] Found target log line: `;2021/09/14 04:56:38 [notice] 32#32: start worker processes`
TRAC[0014] Found target log line: `;2021/09/14 04:56:38 [notice] 32#32: start worker processes`
DEBU[0014] Finished waiting for log message 'start worker processes' from node 'k3d-dnstest-serverlb'
INFO[0014] Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0014] Runtime GOOS: linux
DEBU[0014] Adding extra host entry '<IP_ADDRESS> host.k3d.internal'...
DEBU[0014] Executing command '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' in node 'k3d-dnstest-serverlb'
TRAC[0014] Exec process '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-dnstest-serverlb'.. sleeping for 1 second...
DEBU[0015] Exec process in node 'k3d-dnstest-serverlb' exited with '0'
DEBU[0015] Executing command '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' in node 'k3d-dnstest-server-0'
TRAC[0015] Exec process '[sh -c echo '<IP_ADDRESS> host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-dnstest-server-0'.. sleeping for 1 second...
DEBU[0016] Exec process in node 'k3d-dnstest-server-0' exited with '0'
DEBU[0016] Successfully added host record "<IP_ADDRESS> host.k3d.internal" to /etc/hosts in all nodes
DEBU[0016] Executing command '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\shost.k3d.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' in node 'k3d-dnstest-server-0'
TRAC[0016] Exec process '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\shost.k3d.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' still running in node 'k3d-dnstest-server-0'.. sleeping for 1 second...
TRAC[0017] Exec process '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\shost.k3d.internal$/!p' -e '$a<IP_ADDRESS> host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' still running in node 'k3d-dnstest-server-0'.. sleeping for 1 second...
DEBU[0018] Exec process in node 'k3d-dnstest-server-0' exited with '0'
DEBU[0018] Successfully added host record "<IP_ADDRESS> host.k3d.internal" to the CoreDNS ConfigMap
DEBU[0018] Found network {Name:k3d-dnstest ID:7aa7d87c40fc7b56c06d5d8061470b6a993253e9d03b38924e98b97bd01d32cf Created:2021-09-13 23:56:23.751862607 -0500 CDT Scope:local Driver:bridge EnableIPv6:false IPAM:{Driver:default Options:map[] Config:[{Subnet:<IP_ADDRESS>/20 IPRange: Gateway:<IP_ADDRESS> AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[16f1bcd50b9b040b789fe70bd8f09f50fa3709f5ed61ffa0c344f13ca5b3adf3:{Name:k3d-dnstest-server-0 EndpointID:6e3dd27c3085c467d18486a498c119969e8bb9ed5fba2bd36962723d7d189479 MacAddress:02:42:c0:a8:60:02 IPv4Address:<IP_ADDRESS>/20 IPv6Address:} b4aa22d80bdacac20ff8d6405613f87e484f4036b48797a5688e84d9ec9ab20c:{Name:k3d-dnstest-serverlb EndpointID:d5bb42f637d95244abe8fa205527ac6293c1c5b5aa7b6934f541d33b033e735f MacAddress:02:42:c0:a8:60:03 IPv4Address:<IP_ADDRESS>/20 IPv6Address:}] Options:map[com.docker.network.bridge.enable_ip_masquerade:true] Labels:map[app:k3d] Peers:[] Services:map[]}
DEBU[0018] Adding 2 network members to coredns
DEBU[0018] Executing command '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\sk3d-dnstest-serverlb$/!p' -e '$a<IP_ADDRESS> k3d-dnstest-serverlb' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' in node 'k3d-dnstest-server-0'
TRAC[0018] Exec process '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\sk3d-dnstest-serverlb$/!p' -e '$a<IP_ADDRESS> k3d-dnstest-serverlb' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' still running in node 'k3d-dnstest-server-0'.. sleeping for 1 second...
DEBU[0019] Exec process in node 'k3d-dnstest-server-0' exited with '0'
DEBU[0019] Executing command '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\sk3d-dnstest-server-0$/!p' -e '$a<IP_ADDRESS> k3d-dnstest-server-0' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' in node 'k3d-dnstest-server-0'
TRAC[0019] Exec process '[sh -c patch=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\sk3d-dnstest-server-0$/!p' -e '$a<IP_ADDRESS> k3d-dnstest-server-0' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$patch"]' still running in node 'k3d-dnstest-server-0'.. sleeping for 1 second...
DEBU[0020] Exec process in node 'k3d-dnstest-server-0' exited with '0'
INFO[0020] Cluster 'dnstest' created successfully!
 Although it is still not working on Mac: https://gist.github.com/sqd/1e17af4ab24ede63aedec73fd6396141

@sqd , thanks for testing! Can you please try again with the recent https://github.com/rancher/k3d/releases/tag/v5.0.0-rc.1 release and paste logs again? That'd make it easier to debug the issue (if it still persists) :)

@iwilltry42 during testing the same issue I found another one in v5.0.0-rc.1. CLI switch --k3s-arg "--disable=traefik@all" not applied during the creation of a cluster, but when I'm using the same 'options.k3s.extraArgs' in config file - it's working.

@gickis ,thanks for the hint!
But is the DNS issue solved for you?

@iwilltry42 Yes! DNS is working like a charm with K3D_FIX_DNS=1
I've tested it in WSL2 with Cisco AnyConnect VPN. Resources in private VPN network are accessible.

@gickis fix for your mentioned bug released in https://github.com/rancher/k3d/releases/tag/v5.0.0-rc.2

@sqd the logs show that you're connected to a Docker Host called cof-dev-l709:2375. Is this your local Mac or some other (possibly remote) machine?
I'd love to know if the issue persists with the latest rc.X release and if you could test if it works for you when using the Docker network gateway IP (docker network inspect) as the target for DNS requests, so I could add this variant for MacOS. Alternatively I could expose another feature flag or environment variable to override the chosen host :thinking:

@iwilltry42 I'm on big sur 11.6 with k3d version v5.0.0-rc.2. When I run with --trace and K3D_FIX_DNS=1 it seems to work, but K3D_FIX_DNS=1 alone doesn't work, i.e.,
$ K3D_FIX_DNS=1 k3d cluster create --config=configs/k3d.yaml
INFO[0000] Using config file configs/k3d.yaml (k3d.io/v1alpha3#simple)
INFO[0000] portmapping '8080:80' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] portmapping '8443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
INFO[0000] Prep: Network
INFO[0000] Re-using existing network 'k3d-local' (939a2b94377aa6937d2329e957369e7215353ed63c13dc2e4731f8a633d98115)
INFO[0000] Created volume 'k3d-local-images'
INFO[0000] Container 'k3d-registry.localhost' is already connected to 'k3d-local'
INFO[0001] Creating node 'k3d-local-server-0'
INFO[0001] Creating node 'k3d-local-agent-0'
INFO[0001] Creating node 'k3d-local-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-local-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting Node 'k3d-local-tools'
INFO[0007] Deleted k3d-local-tools
ERRO[0007] failed to gather environment information used for cluster creation: failed to get host IP: failed to resolve 'host.docker.internal' from inside the k3d-tools node: Failed to read address for 'host.docker.internal' from nslookup response
ERRO[0007] Failed to create cluster >>> Rolling Back
INFO[0007] Deleting cluster 'local'
INFO[0007] Deleted k3d-local-serverlb
INFO[0007] Deleted k3d-local-server-0
INFO[0007] Deleted k3d-local-agent-0
INFO[0007] Deleted k3d-local-agent-1
ERRO[0007] docker failed to remove the container 'k3d-local-tools': Error: No such container: k3d-local-tools
INFO[0007] Deleting image volume 'k3d-local-images'
FATA[0007] Cluster creation FAILED, all changes have been rolled back!

Is that intended?

@iJebus , this is absolutely not intended and super weird behavior actually :thinking:
So does it always work with --trace and never without? (meaning: is it certainly not just coincidence?)
I released https://github.com/rancher/k3d/releases/tag/v5.0.0-rc.4 today, which has quite some changes to this process in it again. May be worth testing.

I tried a handful of times and it seemed to be the case but I ended up going back to v4.4.8 and patching the CoreDNS config. If no-one else is reporting that behaviour with --trace then it's probably safe to ignore 😅.

@iJebus , I'd love to get some more feedback with the latest rc.4 release (https://github.com/rancher/k3d/releases/tag/v5.0.0-rc.4), as it works for me on MacOS Catalina 10.15.7 with Docker for Desktop 20.10.8 :thinking:
This is critical for releasing v5.0.0, so I'd be happy about any input from regular Mac users (I only have it running in a VM for testing).

I'll test this weekend and let you know. Are there any specific things
you'd like me to try?
On Sat, 2 Oct 2021, 00:13 Thorsten Klein, @.***> wrote:

@iJebus https://github.com/iJebus , I'd love to get some more feedback
with the latest rc.4 release (
https://github.com/rancher/k3d/releases/tag/v5.0.0-rc.4), as it works for
me on MacOS Catalina 10.15.7 with Docker for Desktop 20.10.8 🤔
This is critical for releasing v5.0.0, so I'd be happy about any input
from regular Mac users (I only have it running in a VM for testing).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/k3d/issues/209#issuecomment-932365377, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABIKCVJAUNJY7IU2YTQPWGTUEXM2RANCNFSM4LSSKICQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Actually @iwilltry42, can you provide the config and command you're using? That'll be a good way to make sure we're on the same page.

@iJebus , thank you!
I just use export K3D_FIX_DNS=1 and k3d cluster create testdns👍 (or with --trace when I need some more details)

@iwilltry42, awesome, running the above has both external DNS and host.k3d.internal working without any patching required. I'll keep using it and see how it goes.
k3d: v5.0.0-rc.5
os: MacOS Big Sur 11.6
docker: Docker for Desktop 20.10.8
On a side note, I'm happy to help out in general so feel free to tag me.

Fix working and tested on

Linux (Ubuntu): tested on my own machine and in a VM
MacOS 10 & 11: tested in a VM and on "real" machines by community
Windows w/ WSL2: tested on a "real" machine and in a VM

The K3D_FIX_DNS "feature" flag environment variable will stay at least until v5.1.0 to ensure it's stable. Once we see that it's working out for everyone, it will be promoted and made a fixed (and required) step of the cluster initialization.

@iwilltry42 seems that the fix is not working. I'm on 5.0.0-rc5

The fix with the environment variable doesn't seem to be working for me as well.
Apple laptop with M1 silicon CPU running Docker Desktop:
`❯ system_profiler SPSoftwareDataType
Software:
System Software Overview:

  System Version: macOS 11.6 (20G165)
  Kernel Version: Darwin 20.6.0`

❯ k3d --version k3d version v5.0.3 k3s version latest (default)
❯ k3d cluster create test INFO[0000] Prep: Network INFO[0000] Created network 'k3d-test' INFO[0000] Created volume 'k3d-test-images' INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-test-tools' INFO[0001] Creating node 'k3d-test-server-0' INFO[0001] Creating LoadBalancer 'k3d-test-serverlb' INFO[0001] Using the k3d-tools node to gather environment information **WARN[0001] failed to resolve 'host.docker.internal' from inside the k3d-tools node: Failed to read address for 'host.docker.internal' from command output** INFO[0001] HostIP: using network gateway... INFO[0001] Starting cluster 'test' INFO[0001] Starting servers... INFO[0001] Deleted k3d-test-tools INFO[0001] Starting Node 'k3d-test-server-0' INFO[0005] Starting agents... INFO[0005] Starting helpers... INFO[0005] Starting Node 'k3d-test-serverlb' INFO[0012] Injecting '<IP_ADDRESS> host.k3d.internal' into /etc/hosts of all nodes... INFO[0012] Injecting records for host.k3d.internal and for 2 network members into CoreDNS configmap... INFO[0012] Cluster 'test' created successfully!
Also, the fix with volume creation also doesn't seem to be working:
❯ k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf WARN[0000] No node filter specified INFO[0000] Prep: Network INFO[0000] Created network 'k3d-k3s-default' INFO[0000] Created volume 'k3d-k3s-default-images' INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-k3s-default-tools' INFO[0001] Creating node 'k3d-k3s-default-server-0' INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' INFO[0001] Using the k3d-tools node to gather environment information WARN[0001] failed to resolve 'host.docker.internal' from inside the k3d-tools node: Failed to read address for 'host.docker.internal' from command output INFO[0001] HostIP: using network gateway... INFO[0001] Starting cluster 'k3s-default' INFO[0001] Starting servers... INFO[0001] Deleted k3d-k3s-default-tools INFO[0001] Starting Node 'k3d-k3s-default-server-0' ERRO[0001] Failed Cluster Start: Failed to start server k3d-k3s-default-server-0: Node k3d-k3s-default-server-0 failed to get ready: error waiting for log line k3s is up and running from node 'k3d-k3s-default-server-0': stopped returning log lines ERRO[0001] Failed to create cluster >>> Rolling Back INFO[0001] Deleting cluster 'k3s-default' INFO[0001] Deleted k3d-k3s-default-serverlb INFO[0001] Deleted k3d-k3s-default-server-0 INFO[0001] Deleting cluster network 'k3d-k3s-default' INFO[0001] Deleting image volume 'k3d-k3s-default-images' FATA[0001] Cluster creation FAILED, all changes have been rolled back!

Hi @parg0MakSystem, can you provide some more information as per https://github.com/rancher/k3d/issues/209#issuecomment-947401078, please?
Additionally, you're hitting another issue here:

Also, the fix with volume creation also doesn't seem to be working:
❯ k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf
ERRO[0001] Failed Cluster Start: Failed to start server k3d-k3s-default-server-0: Node k3d-k3s-default-server-0 failed to get ready: error waiting for log line k3s is up and running from node 'k3d-k3s-default-server-0': stopped returning log lines

This is because with K3D_FIX_DNS, k3d writes an entrypoint script that modifies /etc/resolv.conf inside the container, which won't work if it's a mounted file (just implemented a quick check in 3cc4c5c5).
Even with K3D_FIX_DNS=0 this may fail, but this time at the loadbalancer, which you'd have to exclude from the volume mount (as otherwise it doesn't use the docker resolver and cannot resolve the names of the other containers anymore): k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf@server:0

Sorry for reopening this issue, but I'm really disappointed.
I'm running K3D behind a corporate DNS+HTTP Proxy.
Of course, without any configuration, the cluster fails at downloading the images of the K8S component (traefik, coredns...).
When setting proxy and K3D_FIX_DNS=1, cluster state is much much better as all components run.
But when loading workload requiring an Internet access (in intContainers), it appears it is unable to resolve the DNS name of the proxy server. While it is able to resolve the DNS name of Google's server (as example).
I suspect that CoreDNS is still unable to use my local-host DNS resolution (Ubuntu 20.04).
I'm still unable to explain how it is able to resolve public DNS names. My corporate network blocks any DNS requests. And the CoreDNS Config File does not mention any DoH provider.
Any help appreciated.
PS: I restarted the cluster between its creation and the add of workload: is there any impact related to K3D_FIX_DNS feature?

Sorry (again): it seems my previous issue was due to Docker daemon. I'm using a VPN. It seems that Docker daemon pick the DNS resolution config at start (outside VPN) and does not update when VPN was activated. It still continue to use the DNS of my local network, not the corporate one.
After a simple restart of Docker daemon while VPN is active solves the issue.

I shouldn't start a separate thread, so I'm adding my comment here.  New feature doesn't work for me.  My set-up is described in #1042.   After setting environment and create a test cluster, I saw this:
export K3D_FIX_DNS=1 && k3d cluster create test
kubectl run -it --image=rockylinux rockylinux -- bash
[rockylinux@rockylinux8 infra_k3d]$ kubectl exec -it rockylinux -- bash
[root@rockylinux /]#
[root@rockylinux /]# ping splunkapp-preprod02.med.umich.edu
PING splunkapp-preprod02.med.umich.edu (<IP_ADDRESS>) 56(84) bytes of data.
From k3d-test-server-0 (<IP_ADDRESS>) icmp_seq=1 Destination Host Unreachable

It did resolve <IP_ADDRESS> but I can't reach it.  Though I suspect there is an address conflict with docker network  172.20.XXX.XXX?

I shouldn't start a separate thread, so I'm adding my comment here. New feature doesn't work for me. My set-up is described in #1042. After setting environment and create a test cluster, I saw this:
export K3D_FIX_DNS=1 && k3d cluster create test
kubectl run -it --image=rockylinux rockylinux -- bash
[rockylinux@rockylinux8 infra_k3d]$ kubectl exec -it rockylinux -- bash
[root@rockylinux /]#
[root@rockylinux /]# ping splunkapp-preprod02.med.umich.edu
PING splunkapp-preprod02.med.umich.edu (<IP_ADDRESS>) 56(84) bytes of data.
From k3d-test-server-0 (<IP_ADDRESS>) icmp_seq=1 Destination Host Unreachable

It did resolve <IP_ADDRESS> but I can't reach it. Though I suspect there is an address conflict with docker network 172.20.XXX.XXX?

If so, why not configuring your docker (cf. parameter bip in daemon.json) in order to avoid conflict between docker and corporate address plan?

FYI experiencing a similar issue on an M1 Macbook as @parg0MakSystem reported, although I'm using colima (which could be related ie. https://github.com/abiosoft/colima/issues/341)
Tried to use K3D_FIX_DNS but still wasn't able to resolve DNS in pods
Used k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf@server:0 instead and was able to get DNS resolution working

Today I migrated one of my projects from KinD to K3D, and I immediately found this issue. Using K3D_FIX_DNS=1 fixes the issue.
In my case, I'm not running behind Docker Desktop, or even a corporate VPN. It failed to resolve updates.jenkins.io.
I just wonder if K3D_FIX_DNS=1 shouldn't be promoted to default (I wonder how KinD handles it, because I never ran into this when using KinD for two years).

K3D_FIX_DNS=1 works for us as well

It will definitely move to default :+1:
@felipecrs KinD does it exactly the same way.

Got it. Thank you!

We've run into an issue with K3D_FIX_DNS=1 , on the host running k3d I've noticed that after an upgrade received to containerd the docker daemon was restarted and the DNS resolution stopped working.
This is because of 2 extra lines added to iptables for DNS NAT, which caused DNS resolution to fail.
Could this be caused by this change ?
Maybe under certain conditions this script append wrong content to iptables?

Thanks a lot for the great work while searching a solution for the issue! Using K3D_FIX_DNS=1 solve it definitively, even after restarting the Docker daemon (Docker in WSL on my side).
However, it seems that it's not yet the default behavior with k3d v5.6.0. Any plan on this front @iwilltry42?

I am facing the issue that DNS does not work on mobile network...
Unfortunatly neither export K3D_FIX_DNS=1 nor --volume /etc/resolv.conf:/etc/resolv.conf@server:0 does work for me for k3d to work on mobile network...
any hints/help are highly appreciated.

Hey sorry to reopen this. At my company some scripts use export K3D_FIX_DNS=1 and that makes my cluster internal DNS stop working. Once I turn it off, it all start working perfectly. Real strange... Please don't make this the default !!!!

Both K3D_FIX_DNS=1 and k3d cluster create --volume /etc/resolv.conf:/etc/resolv.conf@server:* break the internal image registry for me, because k3s fails to resolve the internal registry hostname in both cases.


:sparkles: Add shared validateHex function

Useful in keeping code DRY when solving #68 and #29
Potentially breaks https://github.com/Earnifi/essential-eth/blob/master/src/shared/tiny-big/tiny-big.ts, marking as draft while I investigate

Issue with tinyBig function exists on master as well, will open an issue to solve that


DictSecurityKey has been deprecated

DictSecurityKey has been deprecated, and is invalid in the example for DictTable.securityKey

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 100622a3-c345-efbc-0eae-fb19eff4a0f9
Version Independent ID: 62e270b7-a183-bc07-195b-e5c78f985c9b
Content: D classes - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/fin-ops-core/dev-itpro/dev-ref/d-classes.md
Service: dynamics-ax-platform
Product: ****
Technology: ****
GitHub Login: @RobinARH
Microsoft Alias: rhaertle

@DAXRunBase, thank you for the feedback.
@RobinARH, please review and consider updating this topic.

Bug 397248 has been logged to track this issue.

I removed the code example, and the change should be published this evening. - Robin


HSEARCH-2084 Upgrade to WildFly 10.0.0.Final

https://hibernate.atlassian.net/browse/HSEARCH-2084

Merged https://github.com/hibernate/hibernate-search/pull/976


Regex search is dead...

When you click the "regex" checkbox for a field, that field becomes ineffective and just acts as if there's nothing.
I was going to search for username regex w.ngd.ng, but MS returned all posts (130k+) as result.

I've tried this search as you suggested, but I haven't been able ot reproduce your issue?

I fixed this a few days back, just didn't close the issue


ram_report fails due to pyelftools version

Describe the bug
Not able to get ram_report when using pyelftools=0.27.
To Reproduce
Steps to reproduce the behavior:

rm -rf build; west build -b esp32c3_devkitm samples/hello_world
west build -t ram_report
Error:

~/espressif/zephyr/zephyr (declare-off-t-and-ssize-t-in-fdtable-h) $ west build -t ram_report
-- west build: running target ram_report
Traceback (most recent call last):
  File "/home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report", line 876, in <module>
    main()
  File "/home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report", line 852, in main
    do_simple_name_matching(dwarfinfo, symbol_dict, processed)
  File "/home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report", line 361, in do_simple_name_matching
    symbol = match_symbol_address(symlist, die,
  File "/home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report", line 115, in match_symbol_address
    low, high = get_die_mapped_address(die, parser, dwarfinfo)
  File "/home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report", line 85, in get_die_mapped_address
    loc = parser.parse_from_attribute(loc_attr, die.cu['version'], die)
TypeError: LocationParser.parse_from_attribute() takes 3 positional arguments but 4 were given
FAILED: zephyr/cmake/reports/CMakeFiles/ram_report /home/sylvio/espressif/zephyr/zephyr/build/zephyr/cmake/reports/CMakeFiles/ram_report
cd /home/sylvio/espressif/zephyr/zephyr/build && /usr/bin/python3 /home/sylvio/espressif/zephyr/zephyr/scripts/footprint/size_report -k /home/sylvio/espressif/zephyr/zephyr/build/zephyr/zephyr.elf -z /home/sylvio/espressif/zephyr/zephyr -o /home/sylvio/espressif/zephyr/zephyr/build --workspace=/home/sylvio/espressif/zephyr -d 99 ram
ninja: build stopped: subcommand failed.

Expected behavior
ram_report output working
Environment (please complete the following information):

OS: Linux SDK 0.16.8
v3.7.0-rc2

many thanks for the PR...


${Callsite} fix class naming when includeNamespace=false and cleanNamesOfAnonymousDelegates=true

Fixed Callsite class name of compiler generated type when includeNamespace=false and cleanNamesOfAnonymousDelegates=true.
Fixes #2762

Thanks!!


Fine-tuning pre-trained models on AVA dataset?

Hi!
I would like to fine-tune pre-trained model using AVA dataset format. How to achieve this using pytorchvideo?
Current tutorial shows on how to run inference on already fine-tuned models.
Thank you!

Hi @TheKursta, I will add an example for the same soon.

Hi! Is there any estimate when it could be available in documentation?

Hi @kalyanvasudev  Is there an example for video detection training?

Any news on this?

I succeed make transfer learning works on slowfast model. Please find in the snippet below:
import torch
model = torch.hub.load('facebookresearch/pytorchvideo', 'slowfast_r50', pretrained=True)
## Freeze model
for param in model.parameters():
    param.requires_grad = False
model.blocks[6] = create_res_basic_head(in_features=2304, out_features=101, pool=None)

Normally, it just loads the pretrained model, modify the head with your number of output features (you can freeze the body weights if you don't want to loose the pretrained weights)

I succeed make transfer learning works on slowfast model. Please find in the snippet below:
import torch
model = torch.hub.load('facebookresearch/pytorchvideo', 'slowfast_r50', pretrained=True)
## Freeze model
for param in model.parameters():
    param.requires_grad = False
model.blocks[6] = create_res_basic_head(in_features=2304, out_features=101, pool=None)

Normally, it just loads the pretrained model, modify the head with your number of output features (you can freeze the body weights if you don't want to loose the pretrained weights)

Did you train on the exact AVA formatted dataset? I mean how did you set out_features=101. Is that your custom data has 101 classes?

@soumyadbanik  No I don't use the AVA dataset, I just want to share how I made transfer learning works. It's been a while but I used the Kinetics Dataset


feat(questDirectives): slider

This is an offshoot of #6.
We've decided to use http://danielcrisp.github.io/angular-rangeslider/demo/ as a base for our slider.

[ ] There should be an option to flip the values from left to right.
[ ] The user should control the width of the slider.
[ ] We should find a way to load this conditionally so that the external libs aren't required if they're not needed.
[ ] We need a labelling filter:

The filter should be a hash of number->label, where each key covers the values below it:
var map = {10:'a', 20:'b',30:'c'};

// 5 => a
// 10 => a
// 15 => b
// 40 => undefined

feat(slider): option to hide the slider handle until first click
feat(slider): option to submit upon click/drop


Snippets

https://github.com/angelozerr/CodeMirror-XQuery/wiki/Completion-with-Templates

Already brought in and working at a basic level 🌹


Fix hasfiller value for BFiniteType created for const literals

Purpose
BFiniteType were compared using == operator in BSameTypeVisitor. This fails to identify that two BFiniteTypes belong to the same type (eg: int). Due to this reason sample code fails to compile. Failing condition is that a union must have a single basic type for all of its members to have a filler value.
Fixes #21251
Approach
BFiniteTypes value space is flattened recursively and passed to BSameTypeVisitor.
Samples
const FOO = 0;
type Bar FOO|1;
public function main() {
    Bar[2] x = [];
}

List expression now supports this syntax..
Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

const FOO_ZERO = 0;
type FOO_FOUR_THREE  4 | 3;
type BarMultiple FOO_ZERO | 1 | FOO_FOUR_THREE;
function createMultipleConstLiteralAutoFilledSealedArray() {
    BarMultiple a = 1;
    BarMultiple[5] sealedArray = [a, a];
    sealedArray[3] = a;
    //assertEqualPanic(0, sealedArray[2]);
}

The compiler still fails to fill arrays with valid values for complex unions. In above code array index 2 is not filled with any value.
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[x] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Aren't we adding this to the 1.2.x branch btw? Couldn't find a PR to the ballerina-1.2.x branch.
This doesn't break existing code right?

Aren't we adding this to the 1.2.x branch btw? Couldn't find a PR to the ballerina-1.2.x branch.
This doesn't break existing code right?

I am waiting for the code review to finish.

Codecov Report

Merging #22289 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #22289   +/-   ##
=======================================
  Coverage   14.59%   14.59%
=======================================
  Files          51       51
  Lines        1411     1411
  Branches      219      219
=======================================
  Hits          206      206
  Misses       1189     1189
  Partials       16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40c02e1...7d8c6b9. Read the comment docs.


Token gets invalid

After a certain time my token turns invalid and doesn't refresh. Do I have to configure something in the Facebook Developer Console (something like the Valid OAuth Redirect URIs)?

I have experienced a similar issue this morning. I've been getting this console error: api request error: 4 Application request limit reached
Sorry to hihjack your question Preg, just thought it would be nice to kill two birds with one stone.

@benjamin-hull any idea when you will be able to take a look into this?

Hi @awoodford - I think your issue is a separate one (hitting the API request limit). There's a fairly simple workaround on the FB/IG side - I put some details in the wiki: https://github.com/companionstudio/instagram-token-agent/wiki/Instagram-API-Limits

@PregOfficial, that's an interesting one.
If you have help pages enabled on your app (they are enabled by default), you should be able to visit (your-app-name).herokuapp.com to check the status: The token agent will fetch your most recent post, and also show the time that Temporize is due to refresh your token.
Would you check that page and report back what it says?

Hi @benjamin-hull thanks for your reply.

Thanks for your reply @benjamin-hull, it was my fault. I assigned a short lived token instead of a long lived. This one expired after an hour and didn't refresh.

@awoodford I wrote a detailed instruction for me, how to set up the token and the token agent. If you want, I can share it with you.

@PregOfficial Thank you that would be great - it would be interesting to see. Although i've not had any issues setting it up it just hits the API call limit and adding new test users didn't fix that.
Either way that would be very much appreciated thank you :)

This is my instruction to set up the token and the agent

Facebook Console
Create new App
Add a Product -> Instagram
Roles

Roles -> add Instagram Testers
search for the Account and add it
go to https://www.instagram.com/accounts/manage_access/
Tester Invites
accept Invite

Go Tohttps://www.instagram.com/oauth/authorize?client_id={app-id}&redirect_uri={redirect-uri}&scope=user_profile,user_media&response_type=code

Click Authorize App

Redirect to given URL
URL Param ?code=..., this code will be used to generate a short-lived access token

Use code without #_ for cURLcurl -X POST \
  https://api.instagram.com/oauth/access_token \
  -F client_id={app-id} \
  -F client_secret={app-secret} \
  -F grant_type=authorization_code \
  -F redirect_uri={redirect-uri} \
  -F code={code without #_}

Result is a short-lived access token
Generate a long-lived access token curl -i -X GET "htts://graph.instagram.com/access_token
  ?grant_type=ig_exchange_token
  &client_secret={app-secret}
  &access_token={short-life-access-token}"

Use the long-lived access token to set up the agent.
Hope this helps

@PregOfficial Thats awesome thank you!

Hey guys - and thanks @PregOfficial!
There is a simpler method to replace steps 5-8, and you can get a long-lived token directly from the app dashboard, with instructions here: https://developers.facebook.com/docs/instagram-basic-display-api/overview#user-token-generator

@benjamin-hull that's how i've done it.
I've added 2 new testers and still getting the same issue. Removing the main tester account (where I'm getting the images from) and adding it again get's it working again.
Any ideas?

@benjamin-hull - with the facebook instructions my token turned invalid, but with the more complex method it worked.

Okay - so re-reading the documentation, I understand now that the 'adding extra users' workaround will not work, because the Basic Display API uses User Access Tokens, rather than Application Access Tokens which use a shared pool.
Unfortunately, according to the docs, user-level call limits are not a hard, published number:

Due to privacy concerns, we do not reveal actual call count values for users.

So - this leaves us with a problem. The Basic Display API always authorizes as a single user, and that user has a limited but unknown number of calls they can make per hour.
The 'real' answer to this issue (and I think the one Facebook/Instagram intends people to use), is to process these requests server-side, and cache the results before delivering them to user's browsers. That way the number of requests is 1 * number of feed refreshes per hour not ``1 * number of feed refreshes per hour * number of page views per hour`.
I will check in with a contact at Facebook to see if there's any other possible solution.


Still maintaining?

Hello! Thanks for your work on this package.
I'm curious if you're still maintaining this project and if you have any desire to update it to consumer the newest shopify-node-api package. There were several changes to the api, notably around how you can now update the call limits and bucket sizes for uploads.   With the advent of shopify plus, we now see those call limits being adjusted per merchant (and even per store) so it would be nice to have this package able to take those arguments as well.
If you're not maintaining or have no desire to perform these updates then please respond so I can find a work around.
Thanks!

Thanks for the heads up. I am still using the package (though not at the moment), so yes it makes sense to give it a little tlc.
Would you like to be added to the repo?

I’ve just done a comprehensive maintenance pass. Have a look, some new features added like support for custom timeouts and buckets, as well as done and error events


Update TLS version

Fixes #318

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?


Add "Open Data Commons Attribution License" from the Open Knowledge Foundation

https://opendatacommons.org/licenses/

already have https://spdx.org/licenses/PDDL-1.0.html and https://spdx.org/licenses/ODbL-1.0.html - but seem to have missed https://opendatacommons.org/licenses/by/ - it may not have been in existence when we added the others, as that was early on.

closing here, as logged this in new repo here: https://github.com/spdx/license-list-XML/issues/548


[WIP][SPARK-4351] RDD cached read logging + display RDD miss rates on web UI

This set of patches adds cached RDD block attempted read/write trace logging to TaskMetrics and therefore the event log, and uses this information to display/compute RDD miss rates on the Web UI.
This patch removes the elegance of the contents of the storage web UI being entirely from StorageStatuses (as RDD access records shouldn't go away when a block manager does for the purpose of tracking hit/miss rates); I'd appreciate comments on cleaner solutions.
This still needs some additions to the UI tests, and I'd like to make sure it doesn't do anything bad for perf.

@woggle thanks for working on this. There is some related discussion #2411, which was closed because the semantics of a miss rate here is not decisively defined. Would you mind closing this issue?


How to save the "Op-codes" after making changes ?

After making changes it doesn't seems that I am able to save the changes at all, any help on that (Export workspace and export program doesn't do it) ?
-Also as a small note the "Search" for a string inside a JAR file seems not working (or need enhancement).
Thanks for help.

Did you read the documentation on editing?
https://www.coley.software/Recaf-documentation/use-editing.html
Regarding string search, what did use as a search parameter and what were you trying to find?

No response, closing until requested information is provided


Scale Node Pool Up or Down for RKE2 provisioned clusters is not working

Setup

Rancher version: v2.6-head (052336574a21bf5675af9dc80632c69215a9715d)
Browser type & version: Chrome (94.0.4606.81)

Describe the bug
"+" and "-" buttons do nothing on node pools of RKE2-provisioned clusters.
To Reproduce

Navigate to RKE2-provisioned cluster in Cluster Manager.
For any node pool, click "+" or "-" buttons to attempt to scale up/down the number of nodes in the pool.

Result
Nothing happens, no new nodes are provisioned or existing ones removed. Looking at "Network" tab confirms there is no request going to backend.
Expected Result
Node pool is scaled up/down accordingly.
Screenshots

@snasovich I wasn't able to reproduce this, but I think I'm misunderstanding your repro steps based off your screenshot; your machine pool names don't look like ones produced by provisioning an RKE2 cluster through the dashboard. What exactly do you mean by 'an RKE2-provisioned cluster'?

@mantis-toboggan-md , yes, I've just provisioned RKE2 cluster using dashboard with 2 node pools. The node pool names ended up like this based on the cluster name and node pool names I've given.
I've just tried on another cluster and it worked. Maybe there was something special with these names. Let me try reproducing it a bit more to see if I can spot what resulted in +/- buttons not working.

@mantis-toboggan-md , it seems to be related to cluster/pool name. When cluster/pool names are really long, pool names end up being mostly the same, with only auto-generated 5-character suffix being different.
I've just tried another cluster created with these settings:

This created cluster with the following pools and "+" buttons are not working for them - no network requests are generated (no errors in console either):

Could you try reproducing it with this additional information?

@snasovich Thank you for taking the time to investigate further; I was able to repro the issue by using longer cluster and pool names and I can see where the shortcoming in the dashboard code is. I think I have what I need to fix this now, thanks again.

After further investigation this doesn't appear to be something we can solve on the UI side (we could detect that we can't scale these node pools and page the page error better but that sounds inadequate). For the scale up/down buttons to work, the UI has to be able to map between the resource in the table group headers,  cluster.x-k8s.io.machinedeployment and the machine pools defined in the cluster (provisioning.cattle.io.cluster) -- specifically in .spec.rkeConfig.machinePools .
You can see here the UI expects the machine deployment id to be in the form <provisioning cluster id>-<machine pool name as defined in spec.rkeConfig.machinePools> but when the cluster and machine pools have long names, this is no longer true: the machine deployment's id is a truncated copy of the name in the rkeConfig with some random letters appended. The provisioning cluster's rkeConfig also contains a reference to the machineConfig (in this case a rke-machine-config.cattle.io.digitaloceanconfig); I thought I might be able to map that to the right machine deployment but no dice.


force time column uint32

since we are doing this change.
I think it might be better to provide more supported underlying transformation for time column.
Like from string (ISO timestamp format) to uint32 seconds will be quite useful to users

Transformation needs define the mapping b/w from item in message and to column in table schema. This definition has not been supported in RTA yet.

merge with backfill support


add notes to readme comparing ScottPlot to similar libraries

Similar open-source libraries to be compared are:

LiveCharts
OxyPlot
ZedGraph

Any news about this?

Any news about this?

Great question! I started working on a little YouTube mini-series about this topic but I ran out of steam... although I hope to come back to it shortly.
Some good news is that I did pretty well on making markdown notes as sort of a script for the videos. Although the videos aren't ready, the code and comments are pretty useful in in their unfinished/unpolished state.
https://swharden.com/CsharpDataVis/#plotting-data-with-c
Hope it helps!
Scott

Great question! I started working on a little YouTube mini-series about this topic but I ran out of steam... although I hope to come back to it shortly.
Some good news is that I did pretty well on making markdown notes as sort of a script for the videos. Although the videos aren't ready, the code and comments are pretty useful in in their unfinished/unpolished state.
https://swharden.com/CsharpDataVis/#plotting-data-with-c
Hope it helps!
Scott

I briefly looked to this articles and found no comparison between libraries :(
I think this (performance / functionality / ease of use comparison) is important when ppl want to migrate to some thing new.
For example, we consider go away from winforms-datavisualization because it's discontinues and probably will have no new .NET designer support. We think about ScottPlot and OxyPlot for now.
May be some comparison with winforms-datavisualization?

I briefly looked to this articles and found no comparison between libraries :(
I think this (performance / functionality / ease of use comparison) is important when ppl want to migrate to some thing new.
For example, we consider go away from winforms-datavisualization because it's discontinues and probably will have no new .NET designer support. We think about ScottPlot and OxyPlot for now.
May be some comparison with winforms-datavisualization?


Adding cells doesn't mark a Notebook as dirty

Create New Notebook
Add a new cell to it
Close Notebook editor

Expected : Prompted to ask to save Notebook
Actual : No prompt - editor is closed immediately
This is a regression from 1.25.3
Note that typing in the cell does then correctly mark it as dirty - it's just the initial adding that doesn't.

No repro on Mac.


state_exists

Would be nice to have such a feature
Relying on dynamically created states- need to check if they exist before transitioning to it.

Implemented for v3.0.2.


secure=False required?

Hello,
I'm just getting started with elasticmq.  I'm using boto (the official aws client)  as the client, and I was running into a bunch of errors I didn't understand:
[WARN] [05/06/2015 20:36:37.435] [elasticmq-akka.actor.default-dispatcher-51] [akka://elasticmq/user/IO-HTTP/listener-0/23] Illegal request, responding with status '501 Not Implemented': Unsupported HTTP method

I was able to get it working by setting is_secure=False in boto (it defaults to is_secure=True).
I didn't see anything in the README about secure=False, or SSL.  Is this expected? Is there some way to get it to work with secure?  If not, could you add a note to the docs somewhere about that?
Thanks!

I use this with boto and it works fine:

region = boto.sqs.regioninfo.RegionInfo(name='elasticmq',
                                        endpoint=sqs_endpoint)
conn = boto.connect_sqs(aws_access_key_id='x',
                        aws_secret_access_key='x',
                        is_secure=False,
                        port=sqs_port,
                        region=region)

On Wed, May 6, 2015 at 4:56 PM Daniel Nephin<EMAIL_ADDRESS>wrote:

Hello,
I'm just getting started with elasticmq. I'm using boto
http://boto.readthedocs.org/en/latest/index.html# (the official aws
client) as the client, and I was running into a bunch of errors I didn't
understand:
[WARN] [05/06/2015 20:36:37.435] [elasticmq-akka.actor.default-dispatcher-51] [akka://elasticmq/user/IO-HTTP/listener-0/23] Illegal request, responding with status '501 Not Implemented': Unsupported HTTP method
I was able to get it working by setting is_secure=False in boto (it
defaults to is_secure=True).
I didn't see anything in the README about secure=False, or SSL. Is this
expected? Is there some way to get it to work with secure? If not, could
you add a note to the docs somewhere about that?
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/adamw/elasticmq/issues/48.

ElasticMQ exposes the API via HTTP, so it doesn't handle https in any way (would be quite complex given that you would have to provide certificates etc).
If you could make a PR with README changes describing how to use EMQ from Boto (I never used that :) ), it would be great!

Cool, #49 looks good. It doesn't explicitly call out is_secure=False in the text, but the example should do.
Thanks!