After a complete power off: the server is currently unable to handle the request

Environmental Info:
K3s Version: k3s version v1.19.1-rc1+k3s1 (041f18f6)
Node(s) CPU architecture, OS, and Version:
Linux cluster01 5.4.0-1018-raspi #20-Ubuntu SMP Sun Sep 6 05:11:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
Cluster Configuration:
3 masters
Describe the bug:
After a fresh install, all nodes were up correctly. After complete power off of all nodes, when devices come up, nodes are not able to reconnect.
Steps To Reproduce:
Installation of K3s on first master node:
curl -sfL https://get.k3s.io | K3S_CLUSTER_INIT=true INSTALL_K3S_CHANNEL=testing sh -

On 2 others masters, installation of K3s
curl -sfL https://get.k3s.io | K3S_URL=${k3s_url} K3S_TOKEN=${k3s_token} INSTALL_K3S_CHANNEL=testing bash -s server

with the correct first node URL and token

Installed K3s:
Before the shutdown, all nodes are Ready

$ sudo kubectl get nodes
NAME        STATUS   ROLES         AGE     VERSION
cluster01   Ready    etcd,master   4h47m   v1.19.1-rc1+k3s1
cluster02   Ready    etcd,master   4h46m   v1.19.1-rc1+k3s1
cluster03   Ready    etcd,master   4h46m   v1.19.1-rc1+k3s1

After the power off, here is the error on first created master node
Error from server (ServiceUnavailable): the server is currently unable to handle the request

And on other master nodes
The connection to the server <IP_ADDRESS>:6443 was refused - did you specify the right host or port?

Expected behavior:
After a shutdown, all clusters should reconnect. At least, one of then should start correctly
Actual behavior:
Connection error to nodes
Additional context / logs:
On first master, here are cyclic messages
Sep 15 00:03:16 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49416: remote error: tls: bad certificate
Sep 15 00:03:16 cluster01 k3s[1706]: raft2020/09/15 00:03:16 INFO: 6571fb7574e87dba is starting a new election at term 1287
Sep 15 00:03:16 cluster01 k3s[1706]: raft2020/09/15 00:03:16 INFO: 6571fb7574e87dba became candidate at term 1288
Sep 15 00:03:16 cluster01 k3s[1706]: raft2020/09/15 00:03:16 INFO: 6571fb7574e87dba received MsgVoteResp from 6571fb7574e87dba at term 1288
Sep 15 00:03:16 cluster01 k3s[1706]: raft2020/09/15 00:03:16 INFO: 6571fb7574e87dba [logterm: 3, index: 4142] sent MsgVote request to 5b9648b5cb0fc41a at term 1288
Sep 15 00:03:16 cluster01 k3s[1706]: raft2020/09/15 00:03:16 INFO: 6571fb7574e87dba [logterm: 3, index: 4142] sent MsgVote request to 7e3172f2dd9d22d5 at term 1288
Sep 15 00:03:18 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:39492: remote error: tls: bad certificate
Sep 15 00:03:18 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49424: remote error: tls: bad certificate
Sep 15 00:03:19 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:55948: remote error: tls: bad certificate
Sep 15 00:03:19 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:19 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:19 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:19 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:20 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49432: remote error: tls: bad certificate
Sep 15 00:03:22 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49440: remote error: tls: bad certificate
Sep 15 00:03:24 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:39496: remote error: tls: bad certificate
Sep 15 00:03:24 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49448: remote error: tls: bad certificate
Sep 15 00:03:24 cluster01 k3s[1706]: {"level":"warn","ts":"2020-09-15T00:03:24.661Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Sep 15 00:03:24 cluster01 k3s[1706]: time="2020-09-15T00:03:24.661907139Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 15 00:03:24 cluster01 k3s[1706]: WARNING: 2020/09/15 00:03:24 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
Sep 15 00:03:24 cluster01 k3s[1706]: WARNING: 2020/09/15 00:03:24 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context deadline exceeded". Reconnecting...
Sep 15 00:03:24 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:24 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:24 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:24 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:25 cluster01 k3s[1706]: raft2020/09/15 00:03:25 INFO: 6571fb7574e87dba is starting a new election at term 1288
Sep 15 00:03:25 cluster01 k3s[1706]: raft2020/09/15 00:03:25 INFO: 6571fb7574e87dba became candidate at term 1289
Sep 15 00:03:25 cluster01 k3s[1706]: raft2020/09/15 00:03:25 INFO: 6571fb7574e87dba received MsgVoteResp from 6571fb7574e87dba at term 1289
Sep 15 00:03:25 cluster01 k3s[1706]: raft2020/09/15 00:03:25 INFO: 6571fb7574e87dba [logterm: 3, index: 4142] sent MsgVote request to 5b9648b5cb0fc41a at term 1289
Sep 15 00:03:25 cluster01 k3s[1706]: raft2020/09/15 00:03:25 INFO: 6571fb7574e87dba [logterm: 3, index: 4142] sent MsgVote request to 7e3172f2dd9d22d5 at term 1289
Sep 15 00:03:25 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:55952: remote error: tls: bad certificate
Sep 15 00:03:26 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49456: remote error: tls: bad certificate
Sep 15 00:03:28 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49464: remote error: tls: bad certificate
Sep 15 00:03:29 cluster01 k3s[1706]: publish error: etcdserver: request timed out
Sep 15 00:03:29 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:29 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:29 cluster01 k3s[1706]: health check for peer 7e3172f2dd9d22d5 could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:29 cluster01 k3s[1706]: health check for peer 5b9648b5cb0fc41a could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 15 00:03:30 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:39500: remote error: tls: bad certificate
Sep 15 00:03:30 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49472: remote error: tls: bad certificate
Sep 15 00:03:31 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:55956: remote error: tls: bad certificate
Sep 15 00:03:32 cluster01 k3s[1706]: http: TLS handshake error from <IP_ADDRESS>:49480: remote error: tls: bad certificate

On other masters, cyclic messages
Sep 15 00:05:19 cluster02 k3s[41498]: time="2020-09-15T00:05:19.655182251Z" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs at https://cluster01:6443/cacerts: https://cluster01:6443/cacerts: 503 Service Unavailable"
Sep 15 00:05:19 cluster02 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 00:05:19 cluster02 systemd[1]: k3s.service: Failed with result 'exit-code'.
Sep 15 00:05:19 cluster02 systemd[1]: Failed to start Lightweight Kubernetes.
Sep 15 00:05:24 cluster02 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1579.
Sep 15 00:05:24 cluster02 systemd[1]: Stopped Lightweight Kubernetes.
Sep 15 00:05:24 cluster02 systemd[1]: Starting Lightweight Kubernetes...
Sep 15 00:05:25 cluster02 k3s[41527]: time="2020-09-15T00:05:25.507939367Z" level=info msg="Starting k3s v1.19.1-rc1+k3s1 (041f18f6)"
Sep 15 00:05:25 cluster02 k3s[41527]: time="2020-09-15T00:05:25.666961273Z" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs at https://cluster01:6443/cacerts: https://cluster01:6443/cacerts: 503 Service Unavailable"
Sep 15 00:05:25 cluster02 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 00:05:25 cluster02 systemd[1]: k3s.service: Failed with result 'exit-code'.
Sep 15 00:05:25 cluster02 systemd[1]: Failed to start Lightweight Kubernetes.
Sep 15 00:05:30 cluster02 systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1580.
Sep 15 00:05:30 cluster02 systemd[1]: Stopped Lightweight Kubernetes.
Sep 15 00:05:30 cluster02 systemd[1]: Starting Lightweight Kubernetes...
Sep 15 00:05:31 cluster02 k3s[41549]: time="2020-09-15T00:05:31.512641858Z" level=info msg="Starting k3s v1.19.1-rc1+k3s1 (041f18f6)"
Sep 15 00:05:31 cluster02 k3s[41549]: time="2020-09-15T00:05:31.671164367Z" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs at https://cluster01:6443/cacerts: https://cluster01:6443/cacerts: 503 Service Unavailable"
Sep 15 00:05:31 cluster02 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 00:05:31 cluster02 systemd[1]: k3s.service: Failed with result 'exit-code'.
Sep 15 00:05:31 cluster02 systemd[1]: Failed to start Lightweight Kubernetes.

Nobody has this issue?
Do I missed something ??

I have this same issue with 1.19.1-k3s1. I am using keepalived to make them HA.
master-a logs: k3s-a.log
master-b logs: k3s-b.log
master-c logs: k3s-c.log

Thanks for the feedback y'all! We'll investigate this as soon as possible. We expected there might be a few corner cases with the embedded etcd cluster which is why it's still experimental, but it should stabilize soon.

I redeployed the master nodes and this time with --debug turn on, logs are below.
k3s-a.log
k3s-b.log
k3s-c.log

Does this work without the keepalived VIP?
Also, Pi's have some odd issues with systemd starting things before the network is actually up. Do you see any change in behavior if you modify your k3s systemd unit as described here: https://github.com/rancher/k3s/pull/2210/files

I will report back on removing keepalived and seeing if I experience the same issue. No Pi here, I am using Intel NUCs, and my systemd file already had After=network.target in it.

@onedr0p if you're not on the same platform as this issue can you open a new issue and fill out the template? It's hard to track affected system configurations when people me-too on other issues without providing all the info.

Removing the keepalived VIP didn't work, meaning the master nodes still do not come back online after a reboot.
@brandond Apologies for not including my hardware initially, but these issues are too identical, no? I would expect a fix for one architecture would be a fix for the other architecture.

That depends on what the root cause is. Different OS distribution and different hardware merit a new issue.

I made a test with the new RC
cluster01   Ready    etcd,master   83m   v1.19.2-rc2+k3s1
cluster02   Ready    etcd,master   82m   v1.19.2-rc2+k3s1
cluster03   Ready    etcd,master   81m   v1.19.2-rc2+k3s1

The result is the same, the cluster is still down. But logs are different:
Sep 19 01:31:42 cluster01 k3s[2164]: time="2020-09-19T01:31:42.947291646+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:42 http: TLS handshake error from <IP_ADDRESS>:49048: remote error: tls: bad certificate"
Sep 19 01:31:43 cluster01 k3s[2164]: time="2020-09-19T01:31:43.000036202+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:42 http: TLS handshake error from <IP_ADDRESS>:54066: remote error: tls: bad certificate"
Sep 19 01:31:43 cluster01 k3s[2164]: time="2020-09-19T01:31:43.418649172+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:43 http: TLS handshake error from <IP_ADDRESS>:47548: remote error: tls: bad certificate"
Sep 19 01:31:45 cluster01 k3s[2164]: time="2020-09-19T01:31:45.438618592+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:45 http: TLS handshake error from <IP_ADDRESS>:47556: remote error: tls: bad certificate"
Sep 19 01:31:46 cluster01 k3s[2164]: {"level":"warn","ts":"2020-09-19T01:31:46.046+0200","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Sep 19 01:31:46 cluster01 k3s[2164]: time="2020-09-19T01:31:46.047078765+02:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 19 01:31:46 cluster01 k3s[2164]: WARNING: 2020/09/19 01:31:46 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
Sep 19 01:31:46 cluster01 k3s[2164]: health check for peer a1da96e589c651ed could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:46 cluster01 k3s[2164]: health check for peer 3928f76ae7bbea7b could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:46 cluster01 k3s[2164]: health check for peer a1da96e589c651ed could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:46 cluster01 k3s[2164]: health check for peer 3928f76ae7bbea7b could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:47 cluster01 k3s[2164]: raft2020/09/19 01:31:47 INFO: 6571fb7574e87dba is starting a new election at term 60
Sep 19 01:31:47 cluster01 k3s[2164]: raft2020/09/19 01:31:47 INFO: 6571fb7574e87dba became candidate at term 61
Sep 19 01:31:47 cluster01 k3s[2164]: raft2020/09/19 01:31:47 INFO: 6571fb7574e87dba received MsgVoteResp from 6571fb7574e87dba at term 61
Sep 19 01:31:47 cluster01 k3s[2164]: raft2020/09/19 01:31:47 INFO: 6571fb7574e87dba [logterm: 3, index: 26854] sent MsgVote request to 3928f76ae7bbea7b at term 61
Sep 19 01:31:47 cluster01 k3s[2164]: raft2020/09/19 01:31:47 INFO: 6571fb7574e87dba [logterm: 3, index: 26854] sent MsgVote request to a1da96e589c651ed at term 61
Sep 19 01:31:47 cluster01 k3s[2164]: time="2020-09-19T01:31:47.456692788+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:47 http: TLS handshake error from <IP_ADDRESS>:47564: remote error: tls: bad certificate"
Sep 19 01:31:48 cluster01 k3s[2164]: time="2020-09-19T01:31:48.954812267+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:48 http: TLS handshake error from <IP_ADDRESS>:49052: remote error: tls: bad certificate"
Sep 19 01:31:49 cluster01 k3s[2164]: time="2020-09-19T01:31:49.011361853+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:49 http: TLS handshake error from <IP_ADDRESS>:54070: remote error: tls: bad certificate"
Sep 19 01:31:49 cluster01 k3s[2164]: time="2020-09-19T01:31:49.476812173+02:00" level=info msg="Cluster-Http-Server 2020/09/19 01:31:49 http: TLS handshake error from <IP_ADDRESS>:47572: remote error: tls: bad certificate"
Sep 19 01:31:51 cluster01 k3s[2164]: publish error: etcdserver: request timed out
Sep 19 01:31:51 cluster01 k3s[2164]: health check for peer a1da96e589c651ed could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:51 cluster01 k3s[2164]: health check for peer a1da96e589c651ed could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:51 cluster01 k3s[2164]: health check for peer 3928f76ae7bbea7b could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused
Sep 19 01:31:51 cluster01 k3s[2164]: health check for peer 3928f76ae7bbea7b could not connect: dial tcp <IP_ADDRESS>:2380: connect: connection refused

Yes, it appears to be deadlocked. Node 1 is waiting for 2 and 3 to come up before etcd will start, as a single node does not have quorum. Nodes 2 and 3 are waiting for node 1 to come up so that they can bootstrap the certs, and can't proceed past that to start etc. I'm not sure why nodes 2 and 3 are trying to load certs again though, as this should have been completed when they came up the first time.

This is definitely a blocker for GAing etcd support. We’ll figure it out

Cc @davidnuzik

This happens because we try to validate the client token here:
https://github.com/rancher/k3s/blob/master/pkg/cluster/bootstrap.go#L43
before checking to see if we've already bootstrapped here:
https://github.com/rancher/k3s/blob/master/pkg/cluster/bootstrap.go#L56
The workaround is to remove K3S_TOKEN and K3S_URL from /etc/systemd/system/k3s.service.env after the first startup so it doesn't try to bootstrap again, but we need to fix the check order.

I try the woraround and it works fine. Thanks!
Do you think the real fix will be done in the next release?

We're all so eager to get this to work :D
@brandond I am doing a different method of install (https://github.com/PyratLabs/ansible-role-k3s), which includes the flags in the systemd files. I am still unable to make the cluster come back after reboot. Maybe you can see why looking at my systemd files?
ansible all -i inventory/custom/hosts.yml -b -m shell -a "cat /etc/systemd/system/k3s.service"

k8s-master-a | CHANGED | rc=0 >>

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --flannel-backend none --cluster-cidr <IP_ADDRESS>/16 --service-cidr <IP_ADDRESS>/16 --disable servicelb --disable traefik --disable local-storage --disable metrics-server --cluster-init --tls-san <IP_ADDRESS>/24 --node-ip <IP_ADDRESS> --kubelet-arg feature-gates=ExternalPolicyForExternalIP=true
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

k8s-master-c | CHANGED | rc=0 >>

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --flannel-backend none --cluster-cidr <IP_ADDRESS>/16 --service-cidr <IP_ADDRESS>/16 --disable servicelb --disable traefik --disable local-storage --disable metrics-server --server https://<IP_ADDRESS>:6443 --token K10a4e55644c307802de8b9b60d40a902e4e72dd3204994feeb481e63e5823ed4ea::server:12006d0e29e047265baf3c1fc06c6ec9 --tls-san <IP_ADDRESS>/24 --node-ip <IP_ADDRESS> --kubelet-arg feature-gates=ExternalPolicyForExternalIP=true
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

k8s-master-b | CHANGED | rc=0 >>

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --flannel-backend none --cluster-cidr <IP_ADDRESS>/16 --service-cidr <IP_ADDRESS>/16 --disable servicelb --disable traefik --disable local-storage --disable metrics-server --server https://<IP_ADDRESS>:6443 --token K10a4e55644c307802de8b9b60d40a902e4e72dd3204994feeb481e63e5823ed4ea::server:12006d0e29e047265baf3c1fc06c6ec9 --tls-san <IP_ADDRESS>/24 --node-ip <IP_ADDRESS> --kubelet-arg feature-gates=ExternalPolicyForExternalIP=true
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.targe

@onedr0p in that case you will need to remove the --server and --token flags from the server command line
Only servers run the apiserver and datastore (etcd). Labeling agents won't make them a server.

@brandond I removed --server and --token from one of the secondary masters. Then issued sudo systemctl daemon-reload and then sudo systemctl restart k3s. The service came back healthy and the node was still working. However on reboot of this node it did not want to join back.
Systemd service:
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --flannel-backend none --cluster-cidr <IP_ADDRESS>/16 --service-cidr <IP_ADDRESS>/16 --disable servicelb --disable traefik --disable local-storage --disable metrics-server --tls-san <IP_ADDRESS> --node-ip <IP_ADDRESS> --kubelet-arg feature-gates=ExternalPolicyForExternalIP=true
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Here is the logs for that node:
Sep 21 08:16:38 k8s-master-c systemd[1]: Starting Lightweight Kubernetes...
Sep 21 08:16:38 k8s-master-c k3s[521]: time="2020-09-21T08:16:38.429345552-04:00" level=info msg="Starting k3s v1.19.2-rc2+k3s1 (d38505b1)"
Sep 21 08:16:38 k8s-master-c k3s[521]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Sep 21 08:16:38 k8s-master-c k3s[521]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Sep 21 08:16:38 k8s-master-c k3s[521]: peerTLS: cert = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key, trusted-ca = /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file =
Sep 21 08:16:38 k8s-master-c k3s[521]: time="2020-09-21T08:16:38.447441699-04:00" level=info msg="Running kube-apiserver --advertise-address=<IP_ADDRESS> --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=<IP_ADDRESS> --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://<IP_ADDRESS>:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=<IP_ADDRESS>/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Sep 21 08:16:38 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:38 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:39 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:39 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:39 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:39 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:40 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:40 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:41 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:41 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:43 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:43 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:44 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:44 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:47 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:47 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:48 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:48 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:48 k8s-master-c k3s[521]: {"level":"warn","ts":"2020-09-21T08:16:48.447-0400","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused\""}
Sep 21 08:16:48 k8s-master-c k3s[521]: time="2020-09-21T08:16:48.447933340-04:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 21 08:16:53 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:53 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:53 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:53 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:54 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:54 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:55 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:55 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:16:58 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:16:58 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:17:02 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:17:02 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 08:17:03 k8s-master-c k3s[521]: WARNING: 2020/09/21 08:17:03 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...

@onedr0p can you change your command from k3s server to k3s --debug server? It looks like etcd is not finishing startup for some reason.

Sure here you go, the systemd file is the same but with --debug added, here is the log after reboot.
-- Logs begin at Mon 2020-09-21 10:32:02 EDT, end at Mon 2020-09-21 13:52:32 EDT. --
Sep 21 13:51:47 k8s-master-c systemd[1]: Starting Lightweight Kubernetes...
Sep 21 13:51:47 k8s-master-c k3s[526]: time="2020-09-21T13:51:47-04:00" level=debug msg="Asset dir /var/lib/rancher/k3s/data/07bf5246a6ab2127234428fbf3023ce85b0376a7b946bbdaee2726b7d9a6fad8"
Sep 21 13:51:47 k8s-master-c k3s[526]: time="2020-09-21T13:51:47-04:00" level=debug msg="Running /var/lib/rancher/k3s/data/07bf5246a6ab2127234428fbf3023ce85b0376a7b946bbdaee2726b7d9a6fad8/bin/k3s-server [/usr/local/bin/k3s --debug server --flannel-backend none --cluster-cidr <IP_ADDRESS>/16 --service-cidr <IP_ADDRESS>/16 --disable servicelb --disable traefik --disable local-storage --disable metrics-server --cluster-dns <IP_ADDRESS> --tls-san <IP_ADDRESS> --node-ip <IP_ADDRESS> --kubelet-arg feature-gates=ExternalPolicyForExternalIP=true]"
Sep 21 13:51:47 k8s-master-c k3s[526]: time="2020-09-21T13:51:47.400486793-04:00" level=info msg="Starting k3s v1.19.2+k3s1 (d38505b1)"
Sep 21 13:51:47 k8s-master-c k3s[526]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Sep 21 13:51:47 k8s-master-c k3s[526]: [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
Sep 21 13:51:47 k8s-master-c k3s[526]: peerTLS: cert = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key, trusted-ca = /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file =
Sep 21 13:51:47 k8s-master-c k3s[526]: time="2020-09-21T13:51:47.419057703-04:00" level=info msg="Running kube-apiserver --advertise-address=<IP_ADDRESS> --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=<IP_ADDRESS> --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://<IP_ADDRESS>:2379 --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=<IP_ADDRESS>/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Sep 21 13:51:47 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:47 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:47 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:47 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:48 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:48 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:48 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:48 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:49 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:49 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:50 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:50 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:51 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:51 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:52 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:52 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:56 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:56 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:51:57 k8s-master-c k3s[526]: {"level":"warn","ts":"2020-09-21T13:51:57.419-0400","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused\""}
Sep 21 13:51:57 k8s-master-c k3s[526]: time="2020-09-21T13:51:57.420272065-04:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 21 13:51:57 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:51:57 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:02 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:02 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:03 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:03 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:03 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:03 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:04 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:04 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:07 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:07 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:11 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:11 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:12 k8s-master-c k3s[526]: {"level":"warn","ts":"2020-09-21T13:52:12.420-0400","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused\""}
Sep 21 13:52:12 k8s-master-c k3s[526]: time="2020-09-21T13:52:12.420999478-04:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 21 13:52:12 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:12 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:17 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:17 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:18 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:18 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:20 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:20 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:22 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:22 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:26 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:26 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:27 k8s-master-c k3s[526]: {"level":"warn","ts":"2020-09-21T13:52:27.421-0400","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused\""}
Sep 21 13:52:27 k8s-master-c k3s[526]: time="2020-09-21T13:52:27.421641003-04:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Sep 21 13:52:29 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:29 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...
Sep 21 13:52:32 k8s-master-c k3s[526]: WARNING: 2020/09/21 13:52:32 grpc: addrConn.createTransport failed to connect to {https://<IP_ADDRESS>:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: connect: connection refused". Reconnecting...```

If you want to live dangerously, you can try out the fix via INSTALL_K3S_COMMIT=48ed47c4a3e420fa71c18b2ec97f13dc0659778b

@brandond the commit reboots successfully, everything looks good. The log is full of message I do not understand. If anyone like to check, I attached it.
node1.txt
I noticed one thing, but it's probably unrelated. The first node takes ~10 minutes to respond to any ssh requests after a reboot. :/

@brandond I have install that commit and the masters are coming online after reboot. I don't see the 10 minute delay @cubed-it mentioned.

@cubed-it that doesn't sound like a k3s issue. It sounds like something is timing out on boot and delaying network startup. Have you checked your system logs?

@brandond just in the last few minutes I have been analyzing the problem further and it seems to be the operating system.
When I issue a "reboot" command, everything runs fine until "Stopping Restore / save the current clock...". Whatever that means...
So everything is good with k3s ;) Even more than good I have to say! The 1.19 etcd performance compared to 1.18 sqlite is amazing. I observe only half of the base load on the cpu.

Reproduced the issue using v1.19.1-rc1+k3s1 and validated the fix using commit ID 6b11d860374f938623479b0d06aeb3c1ac87ce86.
Created cluster with 3 server nodes.  Rebooted all nodes. All nodes were able to reconnect.
Commands used:
Server 1:
curl -sfL https://get.k3s.io | K3S_CLUSTER_INIT=true INSTALL_K3S_COMMIT=6b11d860374f938623479b0d06aeb3c1ac87ce86 sh -

Server 2 and 3:
curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=6b11d860374f938623479b0d06aeb3c1ac87ce86 sh -s - server --token=TOKEN --server=https://FIRSTNODEIP:6443

After reboot:
kubectl get nodes
NAME               STATUS   ROLES         AGE   VERSION
ip-172-31-10-253   Ready    etcd,master   13m   v1.19.2+k3s-6b11d860
ip-172-31-11-247   Ready    etcd,master   13m   v1.19.2+k3s-6b11d860
ip-172-31-5-59     Ready    etcd,master   16m   v1.19.2+k3s-6b11d860

@ShylajaDevadiga @brandond
Is there a reason why this fix (the commit ID from @ShylajaDevadiga's comment) is not included in the latest v1.19.3+k3s1 release?
This issue has the milestone v1.19.3+k3s1 and is listed as closed, but it's not included in the latest release channel (which is currently v1.19.3+k3s1) and therefore an installation with INSTALL_K3S_CHANNEL=latest still has this issue.

The upstream release had several CVEs, so we wanted to turn it around quickly. We've got another release planned soon that will incl

Not everything makes the planned milestone. The upstream release had several CVEs, so we wanted to turn it around quickly. We've got another release planned soon (independent of upstream patches) that will include backported fixes from the master branch.

Hi @brandond ,  I am using this setup Exercise I Performed  , PS: I am using K3Os in Node2 and Node3, and I am facing this issue. After reboot the setup is not coming up. Can you suggest any particular for this case.

@dharmendrakariya this issue was fixed in the PR linked above. Please confirm that you are running a release that includes the fix.

@brandond okay great, but I think K3os doesn't have this patch. Because today only I faced this one.

Ah, in that case you should probably open an issue in the k3os repo so that it can be tracked there.

Oops my bad, will do that.


M3-5922: Add machine image update and delete E2E tests

Description 📝
What does this PR do?
Adds flows to the machine image E2E test to verify that updating and deleting machine images works as expected.
I opted to add flows to the existing image creation test rather than creating separate tests since the data setup (i.e. creating an image) can take a minute or two.
How to test 🧪
How do I run relevant unit or e2e tests?
Run yarn dev, and then run:
yarn cy:run -s "cypress/e2e/images/machine-image-upload.spec.ts"

And confirm that all of the tests pass.
To run the tests interactively, run yarn cy:debug and search for machine-image-upload.spec.ts.

All 5 tests passed ✅
Regarding Cloud's e2e testing, what do we generally try to do about picking regions? Should we use the same region every time, randomize, test all regions?

Thanks! Most of the time we randomly choose a region (this test does too), but we do have quite a few cases where the region used in the test is hardcoded.
There are arguments to be made for or against any approach, but I think using a random region for each test strikes a decent middle ground where we get reasonable coverage across our regions without having to run an excessive number of tests. I think there's an argument to be made that certain critical operations (like creating a Linode) could be tested against every region, though.

I ran interactively with yarn cy:debug and was seeing a Cypress error for the first test (uploads, updates, and deletes a machine image, end-to-end) returned at cypress/e2e/images/machine-image-upload.spec.ts:189:66where "cy.click() can only be called on a single element. Your subject contained 2 elements. Pass { multiple: true } if you want to serially click each element". All 3 attempted runs of that test failed.
Overall, 4/5 tests passed for me due to the failure above. Any idea what could be causing a failing test on my end when others have 5/5 successfully passing?

Thanks @mjac0bs! I'll take a closer look at this tomorrow, but I think I have an understanding of what's going on. By any chance, did you already have one or more machine images on your account when you ran the test?

did you already have one or more machine images on your account when you ran the test?

@jdamore-linode - It looks like I did have one additional machine image already! When I deleted that image and reran the tests, that first test did pass. Based on that original error, it seems like Cypress was trying to select both images or didn't know which to choose.

did you already have one or more machine images on your account when you ran the test?

@jdamore-linode - It looks like I did have one additional machine image already! When I deleted that image and reran the tests, that first test did pass. Based on that original error, it seems like Cypress was trying to select both images or didn't know which to choose.

That's pretty much exactly what's happening! The action menus (the "..." menus) are all present in the DOM, but only visible when they're open -- the test was tripping up because Cypress was selecting the wrong action menu.
Just pushed a fix for that, thanks again! I'm super glad you caught that because it could've led to some hard-to-troubleshoot failures later on.


Update index.html

I'm Jim, all has fixed

Just a black screen without javascript
Uses a subdomain
Incomplete PR text


Feature Request: Universal Action

Feature Requested
Hi Chris. I really like your workflow. Have you considered making this invokable as a universal action for any selected text? This may be easier for some users than remembering hotkeys. I manually added this and found it quite useful.
Danny
Relevant Screenshot
No response
Checklist

[X] The feature would be useful to more users than just me.

I manually added this and found it quite useful.

If you already did this, feel free to make a PR for that?


Please change the name of this GitHub Repository

fdisk is a well-known Linux application.
Please change the name to libfdisk or rust-fdisk or whatever the rust convention is for libraries.
Thank you.
/cc @stevemar @christo4ferris

Please change this repository name.

I will change this repositories name by the end of the week if you don't. 2020-05-08

+1 changing the name won't break any links or forks, but we really should use a different name here to avoid confusion.

renamed to fdisk-lib per the Rust conventions

Please rename this repository. ‘frisk’ is a well known Linux/Unix application

@christo4ferris @stevemar they haven't changed the name of this repo, can i go ahead and change (just the repo name) to something like fdisk-lib or fdisk-rs?

Yeah
Chris

On Aug 31, 2021, at 10:39 AM, JJ Asghar @.***> wrote:

@christo4ferris @stevemar they haven't changed the name of this repo, can i go ahead and change (just the repo name) to something like fdisk-lib or fdisk-rs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

Renamed.


Is this stable?

I am amazed to see that this actually successfully opens a new window of the same vault. That is great news. But I worry about corruption or other weird things given that two windows are now accessing the same files. Is this safe to use? How much has this been tested?
Really interested in this plugin. Props. 👊

It has not been extensively tested. Unfortunately the only way I can see obsidian code is minified, so I can’t even reassure you that I’ve read through the code and know where it might lead to bugs.
That said I’ve used it a bit here and there, mostly when I want two documents side by side.
But I wouldn’t use it too long without an application restart, as weird state might accumulate.

Thank you. I'll use it and try and provide useful feedback if I can.

Just tested on a new vault with no syncing setup. What I found:
What works:

Updating a file while open on new windows: there's some delay, but it seems to work
Deleting a file open on two windows: again, slight latency but seems to work fine
Creating new files works

Bugs:

Installing a new theme across two windows: doesn't propagate, but also for some reason the themes don't seem to apply instantly in the vault I set it in anyway (needs a refresh)
Installing a new plugin in one window doesn't show up in the other window (doing a force reload makes it work though)
Opening obsidian urls will only happen in a single window, even if they're clicked from the second window
If you close the original window and click an obsidian url, then a new window will be opened
For window state last write wins (whichever window was modified last, on restart that state will be restored)

I also noticed that right click only works in one of the windows. In the other window, nothing happens on right click.

I also noticed that right click only works in one of the windows. In the other window, nothing happens on right click.

ooo thanks. Looks like after initializing the browser window obsidian adds a custom context menu handler: https://github.com/ibash/obsidian-new-window/blob/f204bc07b49e5535fa660482eab797cf424cf9cb/obsidian-main.js#L1189
Just did a quick experiment of assigning the context-menu handler from the original browser window to the new window and it lets the menu open, but it doesn't have all the same options, so requires more investigation.
Original window:

Quick hack:

For posterity I did that by
x = window.require('electron')
win = x.remote.BrowserWindow.getAllWindows()[0]
z = win.webContents._events['context-menu']
// win is the original window and z is the context menu event handler
// open a new window
win2 = x.remote.BrowserWindow.getAllWindows()[0]
// win2 is the second window, electron adds it to the start of the list (compare ids to verify)
win2.webContents.on('context-menu', z)


Compound assignment operators don't obey StrictMode in dynamic scope

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
Continued discussion from https://github.com/PowerShell/PowerShell/issues/17911
Compound assignment operators (e.g. +=, -=, *=) behave weirdly when operating on non-existent variables in dynamic scope, e.g. functions. This is even when using Set-StrictMode -Version 3.0 which does protect against most other undefined variable uses:
> Set-StrictMode -Version 3.0
> function Test-Increment { $x++; $x }
> Test-Increment
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.

> function Test-Assign { $x = $x + 1; $x }
> Test-Assign
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.

> function Test-CompoundAssign { $x += 1; $x }
> Test-CompoundAssign
1

Do note that it only happens in dynamic scope, as in regular prompt all of those statements are caught:
> $x++; $x
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.
dkaszews@Cuddles-Berry:~/Downloads

> $x = $x + 1; $x
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.

> $x += 1; $x
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.

Expected behavior
> function Test-CompoundAssign { $x += 1; $x }
> Test-CompoundAssign
InvalidOperation: The variable '$x' cannot be retrieved because it has not been set.

Actual behavior
> function Test-CompoundAssign { $x += 1; $x }
> Test-CompoundAssign
1

Error details
No response
Environment data
Name                           Value
----                           -----
PSVersion                      7.3.0-preview.3
PSEdition                      Core
GitCommitId                    7.3.0-preview.3-304-gd02c59addc24e13da3b8ee5e1a8e7aa27e00c745
OS                             Linux 5.15.0-1013-raspi #15-Ubuntu SMP PREEMPT Mon Aug 8 06:33:06 UTC 2022
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Visuals
No response

Proposed solution is to create a new Set-StrictMode -Version 4.0 to avoid any breaking changes.

I have 6 test cases:
$notexist = $notexist + 1       # Global =+
$notexist++                     # Global ++
$notexist += 1                  # Global +=
&{ $notexist = $notexist + 1 }  # Dynamic =+
&{ $notexist++ }                # Dynamic ++
&{ $notexist += 1 }             # Dynamic +=

I found a location where the 5 of 6 cases check strict mode: sessionState.GetVariableItem(variablePath, out scope, origin); in VariableOps. GetVariableValue(VariablePath variablePath, ExecutionContext executionContext, VariableExpressionAst varAst) return null, which triggers ThrowStrictModeUndefinedVariable(executionContext, varAst). All global cases have identical callstack, the two working dynamic cases are also identical to each other and share top 9 functions of the global case.
But the failing dynamic += has completely different callstack, and does not even look for $notexist but for $lastError, probably completely unrelated but instead part of regular cleanup - confirmed by running $x = 0; $x += 1 and still seeing $lastError lookup. So dynamic += uses completely different codepath, that I am having trouble tracking down.

Can someone please point me where are dynamic scope variables set? The global ones use SessionStateInternal but I can't find where dynamic ones are - &{ $var = 1 } does not trigger any of my breakpoints.

@SeeminglyScience I don't get it, why is it generated code? Are PowerShell functions precompiled to .NET functions instead of interpreted on the fly? Does it mean that the fix needs to deal with the generator code?

@SeeminglyScience I don't get it, why is it generated code? Are PowerShell functions precompiled to .NET functions instead of interpreted on the fly? Does it mean that the fix needs to deal with the generator code?

Not precompiled no, but they are "just-in-time" compiled. It would be a change in the compiler that would need to happen, but I'd recommend letting the Engine WG discuss it before working on it as it has the same potential breaks as #17911

So interpretation vs jitted isn't actually the difference here. The difference is whether the compiler is marked as "optimized". In unoptimized mode (dot sourced, debugging, a few other things) the compiler generates different expressions for variable accesses. The easiest way to see the difference is by grabbing ScriptBlockDisassembler.

(Click to see disassembly)
{ $var = 10 } | Get-ScriptBlockDisassembly -Unoptimized -Minimal

// ScriptBlock.UnoptimizedEndBlock
try
{
    VariableOps.SetVariableValue(
        Fake.Const<VariablePath>("var"),
        10,
        context,
        (AttributeAst[])null);
}
catch (FlowControlException)
{
    throw;
}
catch (Exception exception)
{
    ExceptionHandlingOps.CheckActionPreference(funcContext, exception);
}

(Click to see disassembly)
# Optimized is the default
{ $var = 10 } | Get-ScriptBlockDisassembly -Minimal

// ScriptBlock.EndBlock
try
{
    locals.Item009 = 10;
}
catch (FlowControlException)
{
    throw;
}
catch (Exception exception)
{
    ExceptionHandlingOps.CheckActionPreference(funcContext, exception);
}

This happens both when interpreted and when jitted. afaik there are zero behavioral differences between the two other than speed.

Proposed solution is to create a new Set-StrictMode -Version 4.0 to avoid any breaking changes.

I do also want to note that this isn't exactly the easy solution it seems to be off the bat. Every time a new version is added, it splinters into a new dialect which needs to be tested separately. It's a high maintenance burden that needs significant value-add to be considered.

I run the test cases through disassembler, here's what I got:

$notexist = $notexist + 1
locals.Item009 = Fake.Dynamic<Func<CallSite, object, object>>(PSVariableAssignmentBinder.Get())(
    Fake.Dynamic<Func<CallSite, object, int, object>>(PSBinaryOperationBinder.Get(ExpressionType.Add, ignoreCase: true,scalarCompare: false))(
        VariableOps.GetVariableValue(
            Fake.Const<VariablePath>("notexists"),
            context,
            Fake.Const<VariableExpressionAst>("$notexists")),
        1));

$notexist++
object @object = VariableOps.GetVariableValue(
    Fake.Const<VariablePath>("notexists"),
    context,
    Fake.Const<VariableExpressionAst>("$notexists"));
VariableOps.SetVariableValue(
    Fake.Const<VariablePath>("notexists"),
    Fake.Dynamic<Func<CallSite, object, object>>(PSUnaryOperationBinder.Get(ExpressionType.Increment))(@object),
    context,
    (AttributeAst[])null);

return (@object == null) ? 0 : @object;

$notexist += 1
locals.Item009 = Fake.Dynamic<Func<CallSite, object, object>>(
    PSVariableAssignmentBinder.Get())(
    Fake.Dynamic<Func<CallSite, object, int, object>>(
        PSBinaryOperationBinder.Get(
            ExpressionType.Add,
            ignoreCase: true,
            scalarCompare: false))(
        locals.Item009,
        1));

For -Unoptimized, all three generate roughly the same disassembly (only difference that ++ has some extra preamble and return, probably because it's an expression with a return value):
csharp     VariableOps.SetVariableValue(         Fake.Const<VariablePath>("notexists"),         Fake.Dynamic<Func<CallSite, object, int, object>>(PSBinaryOperationBinder.Get(ExpressionType.Add, ignoreCase: true,scalarCompare: false))(             VariableOps.GetVariableValue(                 Fake.Const<VariablePath>("notexists"),                 context,                 Fake.Const<VariableExpressionAst>("$notexists")),             1),         context,         (AttributeAst[])null);
This clearly shows that optimized += is the only case which skips getting the variable altogether, instead using PSBinaryOperationBinder and passing uninitialized (default initialized?) `local.Item009`` to it.

This clearly shows that optimized += is the only case which skips getting the variable altogether, instead using PSBinaryOperationBinder and passing uninitialized (default initialized?) local.Item009 to it.

It does still use the binder, it doesn't use VariableOps.GetVariableValue. Basically notexist is incorrectly marked as a local too soon and therefore it does not look for it in other scopes. That's the difference that was previously deemed too likely to break to be fixed :/

@SeeminglyScience You're confusing this issue with https://github.com/PowerShell/PowerShell/issues/17911 . The problem this issue is trying to resolve is not the scope of the variable, but the fact that it is not checked for existence at all.

@SeeminglyScience You're confusing this issue with #17911 . The problem this issue is trying to resolve is not the scope of the variable, but the fact that it is not checked for existence at all.

The root of the issue is the same though. It's not checked if it exists because it's marked as a local variable that is assigned in the current code block prior to usage.

@daxian-dbw Strongly disagree. First, do you have any data to back up that claim? Second of all, documentation clearly states that Latest is not recommended for any production code, as you are explicitly opting into breaking changes.

@dkaszews Sorry that I missed your last comment.
Here is a quick search of Set-StrictMode -Version latest in GitHub: https://grep.app/search?q=Set-StrictMode -Version latest. You can take a look and see how much it's used.
I know the doc called out the caveat of using Latest, and yeah according to the behavior we defined in the doc, whoever uses latest opts into receiving stricter rules. But they will find stuff stop working when upgrading to a new PS version, so it's still practically a breaking change.
I'm not saying any breaking change is a hard big NO. It will need closer look from the team to weigh the value and cost.


Small refactoring of network::protocol module.

While browsing through network::protocol I refactored some small bits and pieces:

Embedded PeerInfo into Peer instead of reproducing the fields which makes it a tad more convenient to get a PeerInfo of a Peer.
Removed the inherent methods of ProtocolContext in preference of the corresponding methods of the Context trait impl.
Lowered the visibility of some structs from pub(crate) to private as they did not seem to be used outside of the module.
Addressed #1698 and removed temporary vectors.

[clabot:check]


Support select box for prompt.

Support dropdown in UI for the prompt.
Syntax:
prompt:
 code_branch: dev
 your_name: ''
 test1:
   type: select
   options:
    - 'Abba'
    - 'Abba2'
    - 'Abba3'

I think it's less about this being a select box with options, and more about the "prompt" (which can be done through CLI, a Web UI, a 3rd party like Slack bot...) to only accept a certain range/list of values.
I'd much prefer that we define patterns that are more general;
prompt:
  token:<PHONE_NUMBER>3jljkkjh234jkh
  branch:
    default: master
    optional: true
  phone:
    default: "(+1)<PHONE_NUMBER>"
    type: phone
  callback:
    pattern: "https?://.+"

@JuhaS - we're using it as a simple test, but it's not suitable to be merged. The default doesn't seem to work, and the call made to the backend makes no sense (submitting the whole object without the selected field).
For the moment, we use it as below to "force" the user to select a field:
prompt:
  branch:
    type: select
    options:
      - '--select the branch--'
      - master
      - dev

Other UI bugs - when selecting a field, it automatically convert it to a text field, and never comes back to the select field.
On the backend, when submitting the "default" (nothing selected manually) - we have the following errors:
ERROR:tornado.application:Uncaught exception POST /api/pipelines/example/run (<IP_ADDRESS>)
HTTPServerRequest(protocol='http', host='<IP_ADDRESS>:8000', method='POST', uri='/api/pipelines/example/run', version='HTTP/1.1', remote_ip='<IP_ADDRESS>', headers={'Origin': 'http://<IP_ADDRESS>:8000', 'Content-Length': '118', 'Accept-Language': 'en-US,en;q=0.8,fr;q=0.6', 'Accept-Encoding': 'gzip, deflate', 'Host': '<IP_ADDRESS>:8000', 'Accept': 'application/json', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36', 'Connection': 'keep-alive', 'Referer': 'http://<IP_ADDRESS>:8000/index.html', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Cookie': 'user="2|1:0|10:1479269766|4:user|12:ImFkbWluIg==|df5d8b4c95e5093c299a98056178f93140970680300259959ca52ce2e2e005cd"; ui-auth-<IP_ADDRESS>%3A8091=f924443d226c8f507b60fc97ca54d165', 'Content-Type': 'application/json'})
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/tornado/web.py", line 1445, in _execute
    result = yield result
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 282, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/site-packages/pipelines/api/server.py", line 126, in _run_pipeline
    runner.load(pipeline_filepath, folder_path, params)
  File "/usr/lib/python2.7/site-packages/pipelines/api/server.py", line 70, in load
    self.pipe = Pipeline.from_yaml(pipeline_filepath, base_params)
  File "/usr/lib/python2.7/site-packages/pipelines/pipeline/pipeline.py", line 124, in from_yaml
    return Pipeline.form_dict(pipeline_def)
  File "/usr/lib/python2.7/site-packages/pipelines/pipeline/pipeline.py", line 83, in form_dict
    PIPELINES_SCHEMA.validate(definition_dict)
  File "/usr/lib/python2.7/site-packages/schema.py", line 136, in validate
    nvalue = Schema(svalue, error=e).validate(value)
  File "/usr/lib/python2.7/site-packages/schema.py", line 136, in validate
    nvalue = Schema(svalue, error=e).validate(value)
  File "/usr/lib/python2.7/site-packages/schema.py", line 169, in validate
    raise SchemaError([None] + x.autos, [e] + x.errors)
SchemaError: {u'type': u'select', u'options': [u'--select your branch--', u'master', u'select-prompt']} should be instance of 'basestring'
ERROR:tornado.access:500 POST /api/pipelines/example/run (<IP_ADDRESS>) 4.39ms

Pushed a fix for the issues mentioned above:

Set the initial value properly according to "default" field. Also fixes the "send the whole object" problem.
Dropdown now stays dropdown even after selecting once.
Added "default" field to the sample-deploy.yml


Closing a scene tab makes another become "edited"

Godot version:
9eb4b6d
Issue description:
When closing a scene tab, the next tab that gain focus will be treated as "edited":

Might be due to 9e85d17.

@akien-mga Yep, just tested it, doesn't happen without @reduz's commit.

I suppose function in UndoRedo for clearing history should have a flag to increment version (true by default) but in the case of closing a tab should pass false.. anyone wants to give it a try?

Didn't find this at first, but made almost the same video of the problem.  X)


Passing a numpy array as the "scale" argument crashes napari

🐛 Bug
This one is probably my fault for promoting the x = y or default_value syntax for None arguments: it crashes when y is a NumPy array. :grimacing: Now, when you pass a NumPy array as the scale= parameter, it crashes with ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
To Reproduce
import numpy as np
from skimage import data, util
import napari

camera = util.img_as_float(data.camera())

with napari.gui_qt():
    napari.view_image(camera, scale=np.array([1, 1]))

Traceback (most recent call last):
  File "test_array_spacing.py", line 9, in <module>
    napari.view_image(camera, scale=np.array([1, 1]))
  File "/home/jni/projects/napari/napari/view_layers.py", line 130, in view_image
    path=path,
  File "/home/jni/projects/napari/napari/components/viewer_model.py", line 512, in add_image
    visible=visible,
  File "/home/jni/projects/napari/napari/layers/image/image.py", line 163, in __init__
    visible=visible,
  File "/home/jni/projects/napari/napari/layers/base/base.py", line 133, in __init__
    self._scale = scale or [1] * ndim
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Expected behavior
It should work the same as if the input was a list.

yup, should be fixed


Ocm 3963 client

Client and types changes for the completion of the aws sdk go v2 refactor
Related to: https://github.com/openshift/rosa/pull/1564

@den-rgb can you update the PR title to Include a title that provides a one sentence overview of the purpose of the PR?
Please see the guidelines here https://github.com/Wikia/guidelines/blob/master/Process/PullRequests.md


CSS

CSS
User Story
As a: Golfer
I want: a Scorecard App
So that: I can track my score on my phone instead of on paper &
So that: all the necessary signatures and calculations are taken care for me
##Acceptance Criteria
GIVEN a scorecard app
WHEN I view the Scorecard
THEN it resembles a real (i.e paper) scorecard

acceptance criteria met


[feature] - Support currency code, name generate

I think we should add support for currency code and name to make our faker more flexible.
About data i copy from google https://developers.google.com/public-data/docs/canonical/currencies_csv
What's your opinion about that?

@MQuy Probably need to update the documentation/Readme as well.

What's the status on this?


Focus mode on app startup/launch

The latest terminal preview:  1.2.2022.0 has "togglefocusmode" option which is great. Can we have a setting that starts the terminal in this mode by default? I want to have maximized and focused mode when app is launch
Or is this already available? I couldn't find any links/references to this

Oh, that's a really good idea


Also support loading sources with "wrong" content-type at start-up

When specifying a URI to ingest with the DATA_LOAD_PATHS config, it might be that the file is being returned with a content type like text/plain in stead of something that is a recognizable format by the built-in rdflib parser.
(for example when storing a .ttl file in a github repo)
Currently shamrql uses the built-in Graph.parse() from rdflib dirfectly, it is desirable to fix this behaviour and add some more guessing, possible using the extension of the supplied URI

This has been resolved in 87dee6657df14967696fec6ffc6c332f374ac329
/ping @metacontext


Deep neural network to training models

Hi,
What kind of deep learning do you use to neural networks? What kind of deep learning framework is used? How to use deep learning  neural network to training  speech data sets?
Deep neural network to training models code，can these be open source code?
Thank you!

this is all proprietary information owned by picovoice. we do not have any plans to open source the training code.


Use pre-push instead of pre-commit

I think it is ok, if not every commit is "valid". What do you think about it?

yeah, will save some time 👍


Windows Cross Compile doesn't work

When trying to compile windows with Ubuntu 20 i get this error while following the docs for windows compile
make[5]: Leaving directory '/home/travis/Neoxa/depends/work/build/x86_64-w64-mingw32/gmp/6.1.2-c999a7b8425'
make[4]: Leaving directory '/home/travis/Neoxa/depends/work/build/x86_64-w64-mingw32/gmp/6.1.2-c999a7b8425'
make[3]: Leaving directory '/home/travis/Neoxa/depends/work/build/x86_64-w64-mingw32/gmp/6.1.2-c999a7b8425'
make[2]: Leaving directory '/home/travis/Neoxa/depends/work/build/x86_64-w64-mingw32/gmp/6.1.2-c999a7b8425'
make[1]: Leaving directory '/home/travis/Neoxa/depends/work/build/x86_64-w64-mingw32/gmp/6.1.2-c999a7b8425'
Postprocessing gmp...
Caching gmp...
Fetching v20181101.tar.gz from https://github.com/dashpay/bls-signatures/archive
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  724k    0  724k    0     0   416k      0 --:--:--  0:00:01 --:--:--  841k
/home/travis/Neoxa/depends/work/download/chia_bls-v20181101/v20181101.tar.gz.temp: OK
Extracting chia_bls...
/home/travis/Neoxa/depends/sources/v20181101.tar.gz: OK
Preprocessing chia_bls...
Configuring chia_bls...
/bin/sh: 1: cmake: not found
funcs.mk:244: recipe for target '/home/Neoxa/depends/work/build/x86_64-w64-mingw32/chia_bls/v20181101-cd627115668/./.stamp_configured' failed
make: *** [/home/Neoxa/depends/work/build/x86_64-w64-mingw32/chia_bls/v20181101-cd627115668/./.stamp_configured] Error 127

Make sure you have cmake installed.


⚠ LPDM Handbook (hosted on Divio) has degraded performance

In 573d365, LPDM Handbook (hosted on Divio) (https://repohubdev.tk) experienced degraded performance:

HTTP code: 500
Response time: 6396 ms

Resolved: LPDM Handbook (hosted on Divio) performance has improved in 851a231.


Style not working on Multiple Select

my multiselect is not like the example

I'm using Laravel vuejs with webpack
npm install vue-multiselect --save
help me please

Seems you have the CSS added properly. It might be that for some reason you have some additional custom CSS rules that override the default styles. Also, please make sure you’re not using the slot="tag" slot, as it’s not needed if you want default tags.


rename mq_management to rabbitmq_api

The new title fits betetr whith what the goal of this project is

change composer.json
change namespaces in php code
change readme

tested by hand, all commands should work


Git-based files-to-artifacts database deployment prototype

The idea is to use git object database to manipulate the repo instead of doing it via the filesystem. The prototypes:

Tooling: https://github.com/zklaus/cfgraphman
https://github.com/zklaus/cfgraph (subdir at the end of the path)
https://github.com/zklaus/cfgraph-by-subdir (subdir at the beginning)

Processing blobs: 842724
Processing trees: 4674405
Processing commits: 1
Matching commits to trees: 1
Processing annotated tags: 0
Processing references: 2
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Commits                    |           |                                |
|   * Count                    |     1     |                                |
|   * Total size               |   216 B   |                                |
| * Trees                      |           |                                |
|   * Count                    |  4.67 M   | ***                            |
|   * Total size               |  1.48 GiB |                                |
|   * Total tree entries       |  37.2 M   |                                |
| * Blobs                      |           |                                |
|   * Count                    |   843 k   |                                |
|   * Total size               |   414 MiB |                                |
| * Annotated tags             |           |                                |
|   * Count                    |     0     |                                |
| * References                 |           |                                |
|   * Count                    |     2     |                                |
|     * Branches               |     1     |                                |
|     * Remote-tracking refs   |     1     |                                |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Commits                    |           |                                |
|   * Maximum size         [1] |   216 B   |                                |
|   * Maximum parents      [1] |     0     |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [2] |   120 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Blobs                      |           |                                |
|   * Maximum size         [3] |   204 KiB |                                |
|                              |           |                                |
| History structure            |           |                                |
| * Maximum history depth      |     1     |                                |
| * Maximum tag depth          |     0     |                                |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of directories  [4] |  42.4 M   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Maximum path depth     [4] |    39     | ***                            |
| * Maximum path length    [4] |   449 B   | ****                           |
| * Number of files        [4] |  37.0 M   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Total size of files    [4] |  17.8 GiB | *******************            |
| * Number of symlinks         |     0     |                                |
| * Number of submodules       |     0     |                                |

[1]  5ccd42e5c465418405284631d1ab3ee42062abdd (refs/heads/main)
[2]  6c3f9e68b3ed1b7e0a9dba13c10a606d3af3c8bb (refs/heads/main:noarch/site-packages)
[3]  06bbd9078a55c19b846d9131e01880909aa376c3 (refs/heads/main:linux-64/bin/ex/conda-forge-artifacts.txt)
[4]  88c00c398469e8f45d1d4785ad0098009bda721a (refs/heads/main^{tree})

As suspected, I was contacted these days by the github support team, asking me to remove the data repository. While it is quite feasible from a pure git perspective, the structure places too high demands on the github infrastructure in terms of indexing etc.


Add updestable example for PrivateDNSZoneVirtualNetworkLink

Description of your changes
Adds updestable example for the PrivateDNSZoneVirtualNetworkLink resource.
I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable to ensure this PR is ready for review.
[ ] ~Added backport release-x.y labels to auto-backport this PR if necessary.~

How has this code been tested
Uptest

/test-examples="examples/network/v1beta1/privatednszonevirtualnetworklink.yaml"

/test-examples="examples/network/v1beta1/privatednszonevirtualnetworklink.yaml"


Test coverage for setRegion method

Jest said 100%, check it again: https://app.codecov.io/gh/blizzapi/blizzapi/blob/master/src/classes/BlizzAPI.ts#L44

Nevermind, it was on master branch that is no longer used in the project. On main it's all good.


Configurable TabLine italics styling

Hello, thanks for the great scheme.
I've noticed you recently added Zenflesh variant and also the ability to enable/disable italics for comments.
Would you consider to do the same for TabLine? https://github.com/mcchrish/zenbones.nvim/blob/4cd223c9a27d8f73219c529eefc3c4d6f949edad/lua/zenbones/init.lua#L95

Make sense. For now, you can simply override the highlight with the help of lush.
colorscheme zenflesh

lua << EOF
local lush = require "lush"
local base = require "zenflesh"

local specs = lush.parse(function()
	return {
		TabLine { bg = base.TabLine.bg, fg = base.TabLine.fg, gui = nil }, -- setting gui to nil, not "italic"
	}
end)

lush.apply(lush.compile(specs))

EOF

The snippet just creates a small lush specs then applies it. This is more flexible and allows you to add more overrides if you want.
It's actually just simply:
TabLine { base.TabLine, gui = nil },

But not sure why it's not being accepted by lush.

Thank you for the snippet, I didn't realize it's that simple :+1:


Readme updates and BYOO orgInit.sh logic update

After several attempts to install the deployer, I made some small tweaks to the readme file to help the next person who forks the project.
I have additionally, updated the PrepLocalRepo.ts > prepOrgInit function to default to the orgInitDefault commands if the byooInit.sh and orgInit.sh files are not found in the source template.

I kinda did this all manually.  Your code changes I did differently, but it accomplishes the same thing (hopefully).
https://github.com/mshanemc/deploy-to-sfdx/blob/aeb58b69099c439e775148bc05df070d513b2317/src/server/lib/prepLocalRepo.ts#L59
I got most of your README stuff into the current version--there's more byoo setup help since you're not the only one who struggled a bit with trying to guess the required oauth scopes :)


Update undo redo feature

Updated the syntax of sell command

Pull Request Test Coverage Report for Build 654

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 79.46%

Totals

Change from base Build 650:
0.0%

Covered Lines:
1973

Relevant Lines:
2483

💛  - Coveralls


Question: On the reason for not encouraging container and reverse proxy usage

Could someone explain why the deployment of headscale with container and reverse proxy is not encouraged? Any difficulty or negative implication from that practice?
Sorry to use the bug report tag since I couldn't find appropriate tag for this issue. I searched through the issues but there is no answer to this. I believe this question is very common for  people read REDEME, by making it a public issue for people to search is going to save many explanation and confusions. Thanks for this great project BTW!

Because it adds an extra layer of complexity and more failure modes to the system.
For instance, many default configurations of reverse proxies do not handle WebSockets - which is an absolute requirement for Headscale. Or in the case of containers, you really have to have several mount points to get Headscale working.
Less tech saavvy users then have the impression that Headscale is broken, rather than their's setup. Which is why we ended having dedicated channels in Discord to support reverse proxies and docker.
For users familiar with containers and reverse proxies, there is nothing particularly difficult on getting Headscale working :)

I got to say, I find the wording a bit... blunt. I understand that you do not support it officially. Not encouraging it however may lead to some people maybe not even trying, which might keep them from figuring stuff out others haven't yet. Maybe even bugs. After all, except for production use, stuff like this is practically made to tinker with. I would recommend removing at least the "not encouraging" point, or at least emphasize not encouraging it particularly in production.
My two cents.


Unenforce the otlp_receiver feature flag

Description
Unenforce the otlp_receiver feature flag
Related issue
b/292091381
How has this been tested?
Unit tests pass, will let the presubmits run the integration tests.
Checklist:

Unit tests

[ ] Unit tests do not apply.
[x] Unit tests have been added/modified and passed for this PR.

Integration tests

[ ] Integration tests do not apply.
[x] Integration tests have been added/modified and passed for this PR.

Documentation

[ ] This PR introduces no user visible changes.
[ ] This PR introduces user visible changes and the corresponding documentation change has been made.
[x] This PR introduces user visible changes and the corresponding documentation changes are planned for a later date.

Minor version bump

[ ] This PR introduces no new features.
[x] This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
[ ] This PR bumps the version.

The ARM failure looks like a flake (was passing in the previous run); ignoring.


npm not working for v5

I'm not seeing any errors, but when I run nvm use 5.3.0, the node files are available in the symlinked directory, but the npm files are not. Npm files are included for other versions (currently using 4.2.3 and 0.10.41).

Do you see thi version in nvm list availalbe? Is v5 is supported at all?

I cannot recreate this error on Windows 10 (i.e. works as expected).
@whitecolor - see #111.

nvm-windows currently doesn't show that it supports the latest version of nodejs and maybe that's why it's necessary to update ~node_modules/npm ourselves.
In my case node v.5.5.0 goes with npm v.3.3.12.
And I have ENV PATH=C:\Users\Username\AppData\Roaming\npm;
So I downloaded source files from npm repository - npm 3.3.12
And copied them to C:\Users\Username\AppData\Roaming\npm\node_modules\npm.
Finally I restarted PC - and everything started working

closing due to age (cleanup)


Substitute taxonomy relations with tags

Syntax: #tag: <entity text>

 Substitute taxonomy relations with tags

How to guide the user?

We closed this for frontend, so I'm closing this one here as well


fix the date inputs icon overlap in safari

fixes the date input icon overlap on safari.
https://github.com/petalframework/petal_components/issues/372
the issue -

the fix -

Thanks for point this out! Merged.


Merge pull request #6463 from gunan/cp

Cherrypick doc updates to r0.12 branch.

Can one of the admins verify this patch?


When will the promise-based APIs be available for Outlook?

According to the docs, the promise-based API is currently only supported in Word and Excel. Are there any plans to implement this in Outlook? If so, is there a timeline for that?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7c223d75-70f3-ccdc-0137-4fc1a2aa885f
Version Independent ID: c55660a6-136b-ef66-8be4-f7bf3b2968c1
Content: Asynchronous programming in Office Add-ins - Office Add-ins
Content Source: docs/develop/asynchronous-programming-in-office-add-ins.md
Product: non-product-specific
Technology: add-ins
GitHub Login: @o365devx
Microsoft Alias: o365devx

Hi @vijaypemmaraju,
Thanks for your question. Outlook currently only supports the callback model, not the promise-based model Excel and Word use. We have no concrete plans to switch Outlook to this model, but we will if there is enough interest from the community. To help influence that discussion, please head over to UserVoice. There, you can ask for promise-based Outlook APIs (or upvote an existing request).


Mark scf.for as dispatchable ops.

This is a step toward https://github.com/google/iree/issues/6154

please don't do this - I want to allow scf.for to be preserved through flow/hal (vs today where we lower to cfg immediately) - if you want to dispatch scf.for you'll need to determine that with some heuristic or produce them as dispatches when creating them. In other words, it's not sufficient to see an scf.for and dispatch it, as that scf.for could be around entire invocations of the source model.

Offline discussions about this with Hanhan and @nicolasvasilache suggest we might be able to use linalg.tiled_loop for such cases. That decouples the use of scf.for coming from model level control flow, and those coming from op lowerings. SO maybe table this one for a bit?

I looked a bit into linalg.tiled_loop, we probably can have something like:
    %0 = linalg.tiled_loop (%i) = (%c0) to (%c1) step (%c1)
        ins(%operand : tensor<4xi32>)
        outs(%out : tensor<4xi32>)
        iterators("parallel")
        distribution("block_x") {
      %result = scf.for ... -> (tensor<4xi32>)
      linalg.yield %result : tensor<4xi32>
    }

will have more discussion in tomorrow's meeting.


C# UWP Release build fails due to: warning ILT0028

When creating a Release build of a UWP application that uses the C# bindings for grpc, the following warning is initially issued:
X:\Program Files (x86)\Microsoft SDKs\UWPNuGetPackages\microsoft.net.native.compiler\2.2.3\tools\Microsoft.NetNative.targets(801,5): warning : ILTransform : warning ILT0028: Found native library 'D:\xxx\obj\x86\Release\ilc\in\grpc_csharp_ext.x64.dll' with unexpected CPU architecture 'amd64', while the current build target architecture is set to 'x86'. Your application may fail to launch. Please make sure to build your application with the matching CPU architecture.

This warning eventually manifests in a failure to build.
Something about the way the nuget package is configured is causing the wrong architecture to be selected for both x86 and x64 architecture Release builds.  They appear to be inverted, so when building for x86 Release, grpc_csharp_ext.x64.dll is selected and when building for x64 Release, grpc_csharp_ext.x86.dll is selected.
Debug builds are correct.
This problem persists up to v1.20 of grpc.

I was able to work around the nuget package issue related to the wrong grpc_csharp_ext DLL being selected.  Unfortunately, that did not resolve the many "MCG0007: Unresolved P/Invoke method" warnings which cause the build to ultimately fail with a "ILT0014: Failed to compile interop source code." error.
It appears this form of DllImport is a problem for a UWP build (example):
        [DllImport(ImportName)]
        public static extern void grpcsharp_shutdown();

@Noemata  can you share what was the problem with the "nuget package issue related to the wrong grpc_csharp_ext DLL being selected" and how you resolved? Was that something that needs to be fixed on our side?

I can't recall the specific details of my fix, since I was flailing when trying to find a fix.  I believe I adjusted the way files were moved into specific folders within the nuget package.
As for the dynalink import problem, that too can be corrected by changing the import statement to be more explicit.

Bump, I'm seeing this issue too...
@Noemata can you try to remember what you did to fix this?

I figured out my error ILT005 - I was trying to pull in illegal unmanaged dlls into my UWP project.  So that's not actually related to this architecture issue.

I was suspecting a problem with the Nuget packing too, so I hunted down where it’s defined. I found it in the Grpc.Core.csproj file.
https://github.com/grpc/grpc/blob/d1bca6ff92a8acbda2ad87a9b23b17e5a68024f2/src/csharp/Grpc.Core/Grpc.Core.csproj#L54-L61
It looks okay as far as I can tell.
I’ve never seen the name.architecture.dll convention before though.
I’m not real sure where to look next, because the ../nativelibs directory doesn’t seem to exist in the tree. I’m assuming it gets created as part of the build?

This particular issue has been resolved.  The others that remain have workarounds that are described elsewhere.

can't run uwp app in release mode.
https://github.com/HppZ/grpc-bug


[#1530] Fix bug preventing remarkable athlete from being used when non-default ability is selected with skill checks

Resolves #1530

Closing because this is going to be resolved during the roll pipeline refactor #2138


Add set pose

About the PR
CM13 parity, add the set pose action
https://github.com/user-attachments/assets/fc8b7eae-3be0-48ff-8814-2df5c6bbb737
Requirements

[X] I have read and am following the Pull Request and Changelog Guidelines.
[X] I have added media to this PR or it does not require an ingame showcase.

:cl:

add: Added setting your character's pose. This can be updated in-round with a right-click menu verb, useful for giving your character time-specific flavor text.

 Biggest marine buff since CAS.

This is John Marine. He is a xenonid.

Man we really gotta port the little side panel with buttons from CM

Man we really gotta port the little side panel with buttons from CM

Good luck forcing everyone to play on separated :pain:

Separated is superior.

I'd rather have this as a right click verb instead of a action bar thing, there's going to be too much going on up there as we continue development.

made it a verb instead


Update toc.yml

added end of call survey

#sign-off

#sign-off

@mkhribech - You've opened this PR in the public repo. You need to work in the private repo instead, as you've done on previous PRs. I'll close this PR now.


problems in sharing lstms' weight

a = nn.LSTM(...)
b = a:clone('weight','bias'...)    --not working now, because the clone function can't touch recurrentModule
b = nn.LSTM(....)
b = share(a,'weight')  -- also not working, the same reason
I think LSTM(FastLSTM) should overwrite share() and clone() function

@liangjz92 have you tried sharedClone?

@nicholas-leonard
My mistake, I called share function before :cuda(). so it actually act in a unshare way, the rnn package is right. just wasted a lot of time ...


[loc][Query] Please clarify context for string "vs_platform"

For string below community member asked to provide more context, can you please help us with it?
Original question
"Can we get some more details about this, please? I cannot fully understand the original string. What does the command do exactly?"
Context
src/vs/platform/environment/node/argv:English (en):plaintext:goto
Resource
vs_platform
String Hash
c1394f990b45fab771734ea37a5bb3f9
String text
Open a file at the path on the specified line and character position.

@josefhosek the --goto command line option can be used to open a file in VS Code on a specific line and column in the text editor.


TF 2.3.0 build failure when its conda package is built using Anaconda's toolchain

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux RHEL 7 ppc64le
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): Source v2.3.0
TensorFlow version: 2.3.0
Python version: 3.6/3.7
Installed using virtualenv? pip? conda?: Building conda package of TF 2.3
Bazel version (if compiling from source): 3.1.0
GCC/Compiler version (if compiling from source): 7.3
CUDA/cuDNN version: 10.2/7

Describe the problem
I'm trying to build conda package of TF 2.3 using Anaconda's toolchain with gcc 7.3 on ppc64le.  Also, note that only GPU build is getting affected. CPU builds are working fine. And this error occurs only within conda environment. Standalone build of TF using gcc 7.4 works fine on ppc64le.
Any other info / logs
When I digged into the TF commits that went into 2.3.0, I found the change done in this commit https://github.com/tensorflow/tensorflow/commit/e0b19f6ef223af40e2e6d1d21b8464c1b2ebee8f#diff-ee64a9d8638aa0a655dfaac625f72bb2 has been causing the above problem. Since the exact problem is not mentioned in the bazel output, I am unable to find out what has gone wrong in the above commit. Reverting the change done in this cc_toolchain_config.bzl.tpl file fixes the problem for me.
If somebody could help me figure out what (specifically from above commit) might caused this error, it would be really appreciated.

@npanpaliya
Sorry, but we don't provide support for issues with the conda environment.
This issue is more suitable on Continuum Anaconda repo since its related to TF installation with Anaconda.
Please post it on Continuum Anaconda.
Thanks!

@ravikyram Thanks for your response. I'll raise the issue with Anaconda too.
I raised the issue here because conda builds used to work before that particular commit and then it has started giving error due to changes done in cc_toolchain_config.bzl.tpl file. So, just want to know if anyone has any pointers on it. The refactoring done in that file is for sure the cause of this error.

I've further figured out the exact linker related changes that are causing the error. Changes done in https://github.com/tensorflow/tensorflow/commit/bb4c751414c3562ab3ab4298f866f47438078c37 have introduced the build issues in conda environment which otherwise work fine.

@npanpaliya One thing I noted was that you are using CUDA10.2 where as tested build configuration of TF2.3 is as follows.

Version
Python version
Compiler
Build tools
cuDNN
CUDA

tensorflow-2.3.0
3.5-3.8
GCC 7.3.1
Bazel 3.1.0
7.6
10.1

May be there is a in compatibility issue. I am not sure how TF-Anaconda package was built but pip binaries of TF were built with the above config. Thanks

Thanks @jvishnuvardhan . But we have been building conda package for TF 2.2 too with cuda 10.2 and even TF 2.3 works with cuda 10.2 if I revert the commit I mentioned in my above comment. The build error I'm seeing is due to linker options' ordering changes done in that commit.

@npanpaliya Unfortunately we don't support or maintain Conda packages. I believe you need to post this issue is Anaconda repository. Thanks!

I've already posted this issue in Anaconda repository.
Also, tried building TF 2.3 conda package with cuda 10.1 but the error persists. So, cuda version doesn't seem to be problematic.

I have only reviewed the said change, but I have no knowledge of the conda toolchain.
@hlopko could you take a look if you have time?

@npanpaliya https://github.com/tensorflow/tensorflow/issues/41856#issuecomment-706422403 may be helpful for you.

Thanks @njzjz . I was able to get this working by reverting commit https://github.com/tensorflow/tensorflow/commit/bb4c751414c3562ab3ab4298f866f47438078c37 in my builds. Indeed what you've pointed is also on the same lines.

@npanpaliya You may also test my fix from #43951

@Flamefire - This seems to be correct. I'd noticed -B to be alone in bazel's params file which contains linker options for the target but I think I overlooked it. I'll try this fix. Thanks a lot!