How to use azurechat without Cosmos DB, Azure AI Document and Azure Cognitive

I am interested in using azurechat in a company scenario, but it has not been historically approved (Cosmo DB) and due to region reasons I cannot use the Document intelligence and Search service.
In this regard, I would like help from the community to create a simpler scenario containing only Azure OpenIA plus Web App with the azurechat interface.

@Gt180Full I would suggest using the default azurechat bot instead of this one as you would need those features for this app to work with the majority of functionality.

@Gt180Full
In order for chat history to work, you need to store the chatlogs somewhere, for this project it uses Cosmos DB. In addition the file upload functionality requires Document intelligence/search to work. The new extensions feature also utilizes this functionality to allow for search of internal documents with AI.
These are the primary benefits over the "default" bot you can create using the azure AI studio "one click deploy"
As all of these components are key for this particular bot to work. I don't think it makes a lot of sense to utilize this repo in your environment if you are not able to enable the deployment requirements . The default bot you get in the azure can be deployed without the search service, without the document intelligence, and without Cosmos DB. Just the functionality will be more limited due to this.


Document how collision layers work in VehicleBody3D

Godot version
3.3.3.stable, 4.0.dev (c1e6c2c49a7a2c05649d79733e7edda382aa92df)
System information
Windows 10
Issue description
Following up on https://github.com/godotengine/godot-proposals/issues/3341#issuecomment-926070052.
VehicleBody3D inherits a collision layer and mask from RigidDynamicBody3D for collision setup.
VehicleWheel3D uses the collision mask from VehicleBody3D to detect collision information via ray-casting.
But this is not properly explained in the documentation.
Both VehicleBody3D and VehicleWheel3D should be updated with a note to explain how collision layers work for vehicles.
Steps to reproduce
n/a
Minimal reproduction project
n/a

So, I have created a new project to try to replicate what I am experiencing. I believe I have it correctly configured. How may I upload the project for someone to check out? Thank you!

@mojoyup The best would be to open a new issue, since this one is about documentation.
You can put your project in a zip file and drag and drop it in the minimal reproduction project field of that issue.


Some adjustments for heroku

create sql tables by ourselves
remove tomcat connection pool config (we don't use tomcat as connection pool)
configure Hikari connection pool

 [Butler] Heroku Deployment - Preview Stage


[Feature][Task Plugin] Abstract python related tasks

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

Currently we cannot alter python virtual environment for python task plugin once ds starts. However, in practice, sometimes we need different python tasks to run in different virtual env.
We might add a field env or something like this so that users could choose the specific virtual environments for their python tasks. To implement this, we could execute something like conda activate xxx before executing the python script in python task plugin.

Use case

Already described above.

Related issues

related: #10302

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Currently, we have supported several python dependency management approaches in Jupyter Task, which is a specific type of python task. Some related discussions: https://github.com/apache/dolphinscheduler/pull/10658#issuecomment-1172014440

There are some points to be discussed here:

We could abstract python type task, other specific python related task could inherit it, together with its several  dependency management approaches.
Which one to use? Choose Anaconda  or venv to manage virtual python environment. Or maybe we could provide both of them for users to choose.

https://github.com/apache/dolphinscheduler/blob/63dfa45474752bfd514f1c8d4503e038b3b14e7c/dolphinscheduler-task-plugin/dolphinscheduler-task-pytorch/src/main/java/org/apache/dolphinscheduler/plugin/task/pytorch/PythonEnvManager.java#L1-L82
I write a PythonEnvManager in #11498, include the method below

getBuildEnvCommand: build a python environment
getSourceEnvCommand: after we build a environment, we can get source command to change python environment
getPythonCommand: after we build a environment, we can get python command directly

If on a WORKER node I want to run two python nodes at the same time, and the two nodes require different python versions. So the source is not enough, is it necessary to write the absolute path to meet this requirement?
Whether to alias the environment variable and configure it in the corresponding profile, because the same environment variable of each WORKER node may be in different paths. For example, the python2 of worker A is under a/xxx/python2, and the python2 of worker B is under b/xx/cc/python2。we can config it in every node's dolpscheduler.env file

If on a WORKER node I want to run two python nodes at the same time, and the two nodes require different python versions. So the source is not enough, is it necessary to write the absolute path to meet this requirement?
Whether to alias the environment variable and configure it in the corresponding profile, because the same environment variable of each WORKER node may be in different paths. For example, the python2 of worker A is under a/xxx/python2, and the python2 of worker B is under b/xx/cc/python2。we can config it in every node's dolpscheduler.env file

In Jupyter Task Plugin, we will source conda.sh, which is configurable, each time before running a task. Therefore, you could use different python virtual environments for different python task nodes even on the same worker, they will not affect one another.
After sourcing conda.sh, we use conda activate xxx to activate the specific environment and execute python script with the python interpreter in that environment, therefore, users do not need to worry about the python path.

https://github.com/apache/dolphinscheduler/blob/2a7e0bee917c0a75280e848ea40e5492a3302c76/dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/test/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterTaskTest.java#L112-L131


Error while converting ONNX to TRT

$ "/home/ubuntu/stable-diffusion-webui/extensions/stable-diffusion-webui-tensorrt/TensorRT-<IP_ADDRESS>/bin/trtexec" --onnx="/home/ubuntu/stable-diffusion-webui/models/Unet-onnx/woopwoop-general-2.0.onnx" --saveEngine="/home/ubuntu/stable-diffusion-webui/models/Unet-trt/woopwoop-general-2.0.trt" --minShapes=x:2x4x64x64,context:2x77x768,timesteps:2 --maxShapes=x:32x4x256x256,context:32x770x768,timesteps:32
...
[05/30/2023-17:02:44] [I] TensorRT version: 8.6.1
[05/30/2023-17:02:44] [I] Loading standard plugins
[05/30/2023-17:02:44] [I] [TRT] [MemUsageChange] Init CUDA: CPU +152, GPU +0, now: CPU 158, GPU 5415 (MiB)
[05/30/2023-17:02:52] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +2478, GPU +756, now: CPU 2712, GPU 6171 (MiB)
[05/30/2023-17:02:52] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading
[05/30/2023-17:02:52] [I] Start parsing network model.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than<PHONE_NUMBER> bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was<PHONE_NUMBER>
[05/30/2023-17:02:53] [I] [TRT] ----------------------------------------------------------------
[05/30/2023-17:02:53] [I] [TRT] Input filename:   /home/ubuntu/stable-diffusion-webui/models/Unet-onnx/woopwoop-general-2.0.onnx
[05/30/2023-17:02:53] [I] [TRT] ONNX IR version:  0.0.8
[05/30/2023-17:02:53] [I] [TRT] Opset version:    17
[05/30/2023-17:02:53] [I] [TRT] Producer name:    pytorch
[05/30/2023-17:02:53] [I] [TRT] Producer version: 2.0.1
[05/30/2023-17:02:53] [I] [TRT] Domain:
[05/30/2023-17:02:53] [I] [TRT] Model version:    0
[05/30/2023-17:02:53] [I] [TRT] Doc string:
[05/30/2023-17:02:53] [I] [TRT] ----------------------------------------------------------------
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than<PHONE_NUMBER> bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was<PHONE_NUMBER>
[05/30/2023-17:02:55] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[05/30/2023-17:02:57] [I] Finished parsing network model. Parse time: 4.66374
[05/30/2023-17:02:57] [I] [TRT] Graph optimization time: 0.317878 seconds.
[05/30/2023-17:02:57] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +256, now: CPU 6403, GPU 39595 (MiB)
[05/30/2023-17:02:57] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +264, now: CPU 6403, GPU 39859 (MiB)
[05/30/2023-17:02:57] [W] [TRT] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.7.0
[05/30/2023-17:02:57] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/30/2023-17:02:57] [E] Error[4]: kOPT values for profile 0 violate shape constraints: /input_blocks.1/input_blocks.1.1/transformer_blocks.0/attn1/Einsum_output_0: tensor volume exceeds (2^31)-1, dimensions are [(* 8 batch_size),E0,E0] where E0=(* height width) Volume exceeds 2^31-1.
[05/30/2023-17:02:58] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1276] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: /input_blocks.1/input_blocks.1.1/transformer_blocks.0/attn1/Einsum_output_0: tensor volume exceeds (2^31)-1, dimensions are [(* 8 batch_size),E0,E0] where E0=(* height width) Volume exceeds 2^31-1.)
[05/30/2023-17:02:58] [E] Engine could not be created from network
[05/30/2023-17:02:58] [E] Building engine failed
[05/30/2023-17:02:58] [E] Failed to create engine from model or file.
[05/30/2023-17:02:58] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # /home/ubuntu/stable-diffusion-webui/extensions/stable-diffusion-webui-tensorrt/TensorRT-<IP_ADDRESS>/bin/trtexec --onnx=/home/ubuntu/stable-diffusion-webui/models/Unet-onnx/woopwoop-general-2.0.onnx --saveEngine=/home/ubuntu/stable-diffusion-webui/models/Unet-trt/woopwoop-general-2.0.trt --minShapes=x:2x4x64x64,context:2x77x768,timesteps:2 --maxShapes=x:32x4x256x256,context:32x770x768,timesteps:32

You probably set the max width and max height too high.
for some reason, i am only able to set the max width and height to both 832, anything higher will fail.
i am also trying to find out what can i do to increase the max height and width.

I did a bunch of testing and found out how to figure out the max combinations of heights with widths. So like it says the volume is to high, I figured out that the max size of the --maxShapes can only total to at highest 92160... that is you take 24width*height has to be equal to or less than 92160, for instance you can have the following:
Maximum width: 1280
Maximum height: 576
which produces in the command: --maxShapes==x:2x4x72x160,...
2x4x72x160 = 92160
and say we do the following:
Maximum width: 1344 <--- this is the next step up that the width can be.
Maximum height: 576
we get --maxShapes=x:2x4x72x168
2x4x72x168 = 96768 which is over the limit of 92160
one more example for good measures, I'm just going to flip the width and height
Maximum width: 576
Maximum height: 1344
we get --maxShapes=2x4x168x72
2x4x168x72 = 96768 which is over the limit of 92160
so basically what we need for the UI is to have a limiter for people when they are sliding the max width and the max height bar. then this will prevent the error from every happening. outside of someone manually running a command line with custom modifications.
I'll see if I can add this into the UI.
wish me luck lol

[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than<PHONE_NUMBER> bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was<PHONE_NUMBER>

Thanks for the valuable observation. I am now able to get the dimension I want not exceeding the max shape size of 92160.
Thanks!!!

so, in other words, according to this formula:  x_max = f"{max_bs * 2}x4x{max_height // 8}x{max_width // 8}"
batch size * height * width / 8 <= 92160 (Conversion Will Succeed)
batch size * height * width / 8 > 92160 (Conversion Will Fail)
I am able to convert with batch size 1, max width 768, max height 960 since 1768960/8=92160

here is a list of useful dimensions that should work with batch size 1
512x1440
576x1280
640x1152
704x1024
768x960
832x832

Yea we actually made a UI update today, with the formula so you can check right on the page if you go over the allotted amount. Waiting for a PR to go through. We also added batch processesing which will create all the onnx models from your stable diffusion directory and also all the trt files with the set parameters. I also added an optional check box for adding the shape of the trt model to the end of the models file name, so you'll see model_name_batch_sizeXmax_widthXmax_height.trt

I also have some recommendations in the UI for people. I recommend only using a max batch size of 2 because 3 your max image size has one dimension less than 512.. which isn't very useful.

You can read my full explanation here
https://github.com/AUTOMATIC1111/stable-diffusion-webui-tensorrt/pull/32

You can read my full explanation here
#32

Thank you so much !!!!!


function id signature set

which functions do we want signatures for?
this enables:

avoid FP matches in these library functions
match against calls to these library functions

libcmt

ucrt

ideally use a matrix of visual studio releases, too

popular open source libraries. which ones? capa identifies the following, which seems to suggest some prevalence in malware:

need to find a good source .lib archives.

boost (https://www.boost.org/) and other crypto and compression libraries come to mind
also at least run-time support for gcc (mingw) and clang

vcpkg makes this so easy!
vcpkg.exe install install --triplet x32-windows-static cryptopp curl detours openssl mbedtls zlib
vcpkg.exe install install --triplet x64-windows-static cryptopp curl detours openssl mbedtls zlib

i put all this in a Windows docker container (which took quite a bit of experimenting/learning) only to find that GH Actions doesn't support Windows containers.
so can build this stuff reproducibly on my computer. if its meaningful, we can probably deploy CI to appveyor and build on windows there.

protobuf

this is closed in #446


Accept deferred function for password_hash

Pull Request (PR) description
Allow the specification of a password as a deferred function.
e.g.
class{'mongodb::server':
  create_admin  => true,
  admin_username => 'admin',
  admin_password_hash => Deferred('myfunc',['lookup_key']),
}

After #659

Not needed providing running puppet 7.23 I think it is.


Update server-rendering-tips.md

This change is

Coverage remained the same at 82.305% when pulling cd72861076c43cb3765a90ad69cda6a20b3bac95 on justin808-patch-1 into 0604006f5d14049dccb03e37f696502566a4416e on master.


Wait for jblas to hit maven central and upgrade to that

Upgrade jblas to 1.2.4.

Done in bugfixes.


x/tools/gopls: Hover bug reported by telemetry

This stack A-IqbA was reported by telemetry:
hover:133 is https://cs.opensource.google/go/x/tools/+/refs/tags/gopls/v0.14.1:gopls/internal/lsp/source/hover.go;l=225 is
return protocol.Range{}, nil, bug.Errorf("type name %q without type spec", obj.Name())

gopls/bug
golang.org/x/tools/gopls/internal/bug.report:35
golang.org/x/tools/gopls/internal/bug.Errorf:2
golang.org/x/tools/gopls/internal/lsp/source.hover:133
golang.org/x/tools/gopls/internal/lsp/source.Hover:4
golang.org/x/tools/gopls/internal/lsp.(*Server).hover:18
golang.org/x/tools/gopls/internal/lsp.(*Server).Hover:1
golang.org/x/tools/gopls/internal/lsp/protocol.serverDispatch:300
golang.org/x/tools/gopls/internal/lsp/lsprpc.(*StreamServer).ServeStream.ServerHandler.func3:5
golang.org/x/tools/gopls/internal/lsp/lsprpc.(*StreamServer).ServeStream.handshaker.func4:52
golang.org/x/tools/gopls/internal/lsp/protocol.Handlers.MustReplyHandler.func1:2
golang.org/x/tools/gopls/internal/lsp/protocol.Handlers.AsyncHandler.func2.2:3
runtime.goexit:0

golang.org/x/tools/gopls@devel devel linux/amd64 (1)
<EMAIL_ADDRESS>go1.21.3 darwin/arm64 (1)

Reported using golang.org/x/tools/gopls/internal/telemetry/cmd/stacks.

This stack indicates that obj is a TypeName (Name or alias, but not a TypeParam), but the spec returned by findDeclInfo was not a TypeSpec. I can't explain how this can happen.  (At first I assume it was due to obj.Pos() being NoPos, which can happen for "unsafe" package members; they are not caught by the obj.Pkg() == nil check because they do have a package. That is indeed a bug, but it's not this bug.)

Conceivably another case of AST fixing perturbing positions? (See related #64479.)

I wrote a program that ran findDeclInfo on every TypeName in kubernetes plus its dependencies, and not a single call returned spec=nil. This lends support to the idea that it's related to ill-formed files and AST repair.

Thanks for investigating. With the above CL, this is good to bump to v0.16.

Well, https://github.com/golang/go/issues/66314 shows that AST repair is not to blame.
One thought: is this an alias to a TypeParam, and therefore not a TypeParam under the GODEBUG=gotypesaliases=1 regime? But the version string says v0.15.1, and (IIUC) a user-built binary with a GODEBUG, even at that exact commit, would not have that version.

This stack G_PnXA was reported by telemetry:
gopls/bug
golang.org/x/tools/gopls/internal/util/bug.report:+35
golang.org/x/tools/gopls/internal/util/bug.Errorf:+2
golang.org/x/tools/gopls/internal/golang.hover:+224
golang.org/x/tools/gopls/internal/golang.Hover:+4
golang.org/x/tools/gopls/internal/server.(*server).Hover:+30
golang.org/x/tools/gopls/internal/protocol.serverDispatch:+335
golang.org/x/tools/gopls/internal/lsprpc.(*streamServer).ServeStream.ServerHandler.func3:+5
golang.org/x/tools/gopls/internal/lsprpc.(*streamServer).ServeStream.handshaker.func4:+52
golang.org/x/tools/gopls/internal/protocol.Handlers.MustReplyHandler.func1:+2
golang.org/x/tools/gopls/internal/protocol.Handlers.AsyncHandler.func2.2:+3
runtime.goexit:+0

<EMAIL_ADDRESS>go1.22.5 darwin/arm64 vscode (2)
<EMAIL_ADDRESS>go1.22.4 darwin/amd64 (1)

Issue created by golang.org/x/tools/gopls/internal/telemetry/cmd/stacks.

This stack gzmDpA was reported by telemetry:
gopls/bug
golang.org/x/tools/gopls/internal/util/bug.report:+35
golang.org/x/tools/gopls/internal/util/bug.Errorf:+2
golang.org/x/tools/gopls/internal/golang.hover:+224
golang.org/x/tools/gopls/internal/golang.Hover:+4
golang.org/x/tools/gopls/internal/server.(*server).Hover:+30
golang.org/x/tools/gopls/internal/protocol.serverDispatch:+335
golang.org/x/tools/gopls/internal/protocol.ServerHandler.func1:+5
golang.org/x/tools/gopls/internal/lsprpc.handshaker.func1:+52
golang.org/x/tools/internal/jsonrpc2.MustReplyHandler.func1:+2
golang.org/x/tools/internal/jsonrpc2.AsyncHandler.func1.2:+3
runtime.goexit:+0

<EMAIL_ADDRESS>go1.20.7 darwin/arm64 other,vscode (2)

Issue created by golang.org/x/tools/gopls/internal/telemetry/cmd/stacks.

Still no idea what could cause this, short of a bug in findDeclInfo.
Note that in<EMAIL_ADDRESS>we were already using Unalias, so something related to type parameters and Unalias seems unlikely.
This hasn't recurred recently, even with many gopls users, so let's wait and see.


Sparse merkle tree implementation - base

A golang based implementation optimized for the UTXO management with Zeto.

pluggable value objects, rather than simple hashes
append only
storage layer must support SQL databases
memory based storage implementation for use in unit tests

fixed by #17 #18 #19 #20


WIP: Activity context

This pull request is work in progress. Use it to discuss code changes.
Current status:

Unified what used to be "Compilation" and "Iface"
Introduced activity contexts that hold activity local data (including prev values of external variables)
Working on new data rendering logic

A short summary of what is left to do:

in CodeGenContext

fix adding pc's to a pc tree where the pc to be added is in a sub-sub-thread which is not yet known in the tree

in ActivityTranslator

refactor curComp to be a value instead of a ref type
refactor how pc's are read/written (there is still some string magic scattered around)

then integrate
master and other features

curComp will remain a ref type because the functions that manipulate it already return something else. The update of the current compilation is therefore a side-effect of code generation.

This PR has been continued in PR #27


fix: fix the problem in installing a script from a website

Description

Fix the problem in installing a script from a script distribution website.

This bug has been happening since I migrated to async functions.
Related Issue

fix #807

Motivation and Context

Refer to #807
How Has This Been Tested?

I could install the scripts from the script distribution website fine.
Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

solved problems:

Promise.all is required before call Array#some
extExists is async function


Escape any open wizards after each test

This prevents the python wizard from causing other non-python tests to fail.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 42.102%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
26470

Relevant Lines:
56980

💛  - Coveralls


Randomwalk diffuse tweaks

Remove the white edges and the bleaching when mfp is large and the geometry has sharp edges.

The only curious thing is the dark borders that appeared in 49 - mfp 002 - randomwalk - ibl - no nee.png and 050 - mfp 002 - randomwalk - ibl - nee.png.

The edges were white before because they were "zero scattering edges". Now we use the reflectance instead of white for them.

Ok. Let's merge this. @biart: would be nice if you could post-review this when you have 5 minutes, it's a one-line change.


[v1.27.6] Unable to join server to cluster (etcd cluster join failed)

Environmental Info:
K3s Version:
[root@k3s-1 ~]# /usr/local/bin/k3s -v
k3s version v1.27.6+k3s1 (bd04941a)
go version go1.20.8

Node(s) CPU architecture, OS, and Version:
[root@k3s-1 ~]# uname -a
Linux k3s-1.lab 5.14.0-284.11.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 9 05:49:00 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers
k3s-1: <IP_ADDRESS>
k3s-2: <IP_ADDRESS>
k3s-3: <IP_ADDRESS>
1 HAProxy (<IP_ADDRESS>) LB on 6443 to servers

/etc/hosts
[root@k3s-1 ~]# cat /etc/hosts
<IP_ADDRESS>   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# BEGIN ANSIBLE MANAGED BLOCK
<IP_ADDRESS> cluster cluster.lab
<IP_ADDRESS> lb-1 lb-1.lab
<IP_ADDRESS> lb-2 lb-2.lab
<IP_ADDRESS> k3s-1 k3s-1.lab
<IP_ADDRESS> k3s-2 k3s-2.lab
<IP_ADDRESS> k3s-3 k3s-3.lab
# END ANSIBLE MANAGED BLOCK

hostnamectl
[root@k3s-1 ~]# hostnamectl
 Static hostname: k3s-1.lab
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: 3485f84a7b5b45aaae9ab40d5dcb2d72
         Boot ID: 02347b7043da467f8f487af85e772e91
  Virtualization: kvm
Operating System: AlmaLinux 9.2 (Turquoise Kodkod)
     CPE OS Name: cpe:/o:almalinux:almalinux:9::baseos
          Kernel: Linux 5.14.0-284.11.1.el9_2.x86_64
    Architecture: x86-64
 Hardware Vendor: Red Hat
  Hardware Model: KVM
Firmware Version: 1.16.1-1.el9

cluster-info
[root@k3s-1 ~]# kubectl cluster-info
Kubernetes control plane is running at https://<IP_ADDRESS>:6443
CoreDNS is running at https://<IP_ADDRESS>:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://<IP_ADDRESS>:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

Describe the bug:
Unable to join second server to initialized cluster.
"msg="etcd cluster join failed: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs"

--bind-address <IP_ADDRESS> had no effect.
(Not verified to be a bug, likely to be PEBKAC...)
Steps To Reproduce:
Install a Libvirt VM based on Both nodes AlmaLinux 9 based on https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-9.2-20230513.x86_64.qcow2
k3s-1 (<IP_ADDRESS>):
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --selinux --tls-san <IP_ADDRESS> --cluster-init

k3s-2 (<IP_ADDRESS>)
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --selinux --tls-san <IP_ADDRESS> --server https://<IP_ADDRESS>:6443

Expected behavior:
Expect second server to join.
Actual behavior:
Join is not completed.
Logs
k3s-2:
Sep 26 16:30:16 k3s-2.lab systemd[1]: Starting Lightweight Kubernetes...
░░ Subject: A start job for unit k3s.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit k3s.service has begun execution.
░░
░░ The job identifier is 4065.
Sep 26 16:30:16 k3s-2.lab sh[3658]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Sep 26 16:30:16 k3s-2.lab systemctl[3659]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Starting k3s v1.27.6+k3s1 (bd04941a)"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Managed etcd cluster not yet initialized"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg=start
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="schedule, now=2023-09-26T16:30:16+02:00, entry=1, next=2023-09-27T00:00:00+02:00"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=<IP_ADDRESS> --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://<IP_ADDRESS>:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=<IP_ADDRESS>/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=<IP_ADDRESS> --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=<IP_ADDRESS> --cluster-cidr=<IP_ADDRESS>/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=<IP_ADDRESS>/16 --use-service-account-credentials=true"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=<IP_ADDRESS> --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=<IP_ADDRESS>/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="To join server node to cluster: k3s server -s https://<IP_ADDRESS>:6443 -t ${SERVER_NODE_TOKEN}"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="To join agent node to cluster: k3s agent -s https://<IP_ADDRESS>:6443 -t ${AGENT_NODE_TOKEN}"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Run: k3s kubectl"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Password verified locally for node k3s-2.lab"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="certificate CN=k3s-2.lab signed by CN=k3s-server-ca@1695737969: notBefore=2023-09-26 14:19:29 +0000 UTC notAfter=2024-09-25 14:30:16 +0000 UTC"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="certificate CN=system:node:k3s-2.lab,O=system:nodes signed by CN=k3s-client-ca@1695737969: notBefore=2023-09-26 14:19:29 +0000 UTC notAfter=2024-09-25 14:30:16 +0000 UTC"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Module overlay was already loaded"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Module nf_conntrack was already loaded"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Module br_netfilter was already loaded"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Module iptable_nat was already loaded"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Module iptable_filter was already loaded"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Sep 26 16:30:16 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:16+02:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=info msg="containerd is now running"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=info msg="Connecting to proxy" url="wss://<IP_ADDRESS>:6443/v1-k3s/connect"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=info msg="Running kubelet --address=<IP_ADDRESS> --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=<IP_ADDRESS> --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=<IP_ADDRESS> --hostname-override=k3s-2.lab --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=info msg="Starting etcd to join cluster with members [k3s-2.lab-45915cb7=https://<IP_ADDRESS>:2380 k3s-1.lab-8cdb347d=https://<IP_ADDRESS>:2380]"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"warn","ts":"2023-09-26T16:30:17.983275+0200","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"warn","ts":"2023-09-26T16:30:17.983309+0200","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.98332+0200","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://<IP_ADDRESS>:2380","https://<IP_ADDRESS>:2380"]}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.983366+0200","caller":"embed/etcd.go:495","msg":"starting with peer TLS","tls-info":"cert = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.983655+0200","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://<IP_ADDRESS>:2379","https://<IP_ADDRESS>:2379"]}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.983736+0200","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.20.8","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"k3s-2.lab-45915cb7","data-dir":"/var/lib/rancher/k3s/server/db/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["https://<IP_ADDRESS>:2380","https://<IP_ADDRESS>:2380"],"advertise-client-urls":["https://<IP_ADDRESS>:2379"],"listen-client-urls":["https://<IP_ADDRESS>:2379","https://<IP_ADDRESS>:2379"],"listen-metrics-urls":["http://<IP_ADDRESS>:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"k3s-1.lab-8cdb347d=https://<IP_ADDRESS>:2380,k3s-2.lab-45915cb7=https://<IP_ADDRESS>:2380","initial-cluster-state":"existing","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.984446+0200","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","took":"559.214µs"}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=info msg="Handling backend connection request [k3s-2.lab]"
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"warn","ts":"2023-09-26T16:30:17.987383+0200","caller":"etcdserver/cluster_util.go:79","msg":"failed to get cluster response","address":"https://<IP_ADDRESS>:2380/members","error":"Get \"https://<IP_ADDRESS>:2380/members\": EOF"}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.98983+0200","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"k3s-2.lab-45915cb7","data-dir":"/var/lib/rancher/k3s/server/db/etcd","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["https://<IP_ADDRESS>:2379"]}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: {"level":"info","ts":"2023-09-26T16:30:17.989991+0200","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"k3s-2.lab-45915cb7","data-dir":"/var/lib/rancher/k3s/server/db/etcd","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["https://<IP_ADDRESS>:2379"]}
Sep 26 16:30:17 k3s-2.lab k3s[3662]: time="2023-09-26T16:30:17+02:00" level=fatal msg="etcd cluster join failed: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs"
Sep 26 16:30:17 k3s-2.lab systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 26 16:30:17 k3s-2.lab systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Sep 26 16:30:17 k3s-2.lab systemd[1]: Failed to start Lightweight Kubernetes.
░░ Subject: A start job for unit k3s.service has failed
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ A start job for unit k3s.service has finished with a failure.
░░
░░ The job identifier is 4065 and the job result is failed.
Sep 26 16:30:17 k3s-2.lab systemd[1]: k3s.service: Consumed 2.686s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░
░░ The unit k3s.service completed and consumed the indicated resources.

k3s-1:
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.545858    2603 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.556571    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.556627    2603 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.556740    2603 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.556781    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.556918    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.557053    2603 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.557100    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.557150    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.557183    2603 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: I0926 16:29:31.569520    2603 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Sep 26 16:29:31 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:31.8305+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:29:31 k3s-1.lab k3s[2603]: time="2023-09-26T16:29:31+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 2m45.830530888s"
Sep 26 16:29:32 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:32.380676+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:32 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:32.380722+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:34.508316+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:34.50837+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:35 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:35.854632+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:43814","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:29:37 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:37.380832+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:37 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:37.380879+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:38 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:38.509748+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:38 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:38.509798+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:42.381625+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:42.381949+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:42.511236+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:42.511367+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:42.842266+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:55788","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:29:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:46.512916+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:46.512951+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:46.832187+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:29:46 k3s-1.lab k3s[2603]: time="2023-09-26T16:29:46+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 3m0.832331322s"
Sep 26 16:29:47 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:47.382835+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:47 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:47.382875+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:49 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:49.813933+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:55790","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:29:50 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:50.514566+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:50 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:50.514619+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:52 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:52.383625+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:52 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:52.383665+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:54 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:54.515977+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:54 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:54.516191+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:56 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:56.902279+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:55158","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:29:57 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:57.384274+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:57 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:57.384402+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:58 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:58.517122+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:58 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:58.517151+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:01 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:01.831871+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:30:01 k3s-1.lab k3s[2603]: time="2023-09-26T16:30:01+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 3m15.831895456s"
Sep 26 16:30:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:02.384416+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:02.384482+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:02.518597+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:02.518655+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:04 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:04.079114+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:54886","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:06 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:06.519896+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:06 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:06.519929+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:07 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:07.385008+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:07 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:07.385026+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:10 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:10.520719+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:10 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:10.520755+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:11 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:11.064854+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:54898","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:12 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:12.385338+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:12 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:12.385375+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:14 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:14.522573+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:14 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:14.522601+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:16 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:16.831917+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:30:16 k3s-1.lab k3s[2603]: time="2023-09-26T16:30:16+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 3m30.83195296s"
Sep 26 16:30:17 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:17.38647+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:17 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:17.386627+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:18 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:18.0715+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:46768","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:18 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:18.524377+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:18 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:18.524409+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:22 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:22.387275+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:22 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:22.387334+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:22 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:22.52651+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:22 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:22.526538+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:25 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:25.071404+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:53078","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:26 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:26.528004+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:26 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:26.52806+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:27 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:27.387914+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:27 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:27.387932+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:30 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:30.529772+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:30 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:30.52982+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:31 k3s-1.lab k3s[2603]: I0926 16:30:31.546911    2603 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Sep 26 16:30:31 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:31.830769+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:30:31 k3s-1.lab k3s[2603]: time="2023-09-26T16:30:31+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 3m45.830912155s"
Sep 26 16:30:32 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:32.089276+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:45552","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:32 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:32.388608+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:32 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:32.388652+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:34.530187+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:34.530215+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:37 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:37.38955+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:37 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:37.389576+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:38 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:38.531286+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:38 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:38.531324+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:39 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:39.070681+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:45560","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:42.389804+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:42.389799+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:42.532613+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:42 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:42.53268+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:46.111852+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:33622","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:46.534744+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:46.534794+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:46 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:46.831495+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:30:46 k3s-1.lab k3s[2603]: time="2023-09-26T16:30:46+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 4m0.831520429s"
Sep 26 16:30:47 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:47.390194+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:47 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:47.39024+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:50 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:50.536449+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:50 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:50.536509+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:52 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:52.391624+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:52 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:52.391669+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:53 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:53.138261+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:39236","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:30:54 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:54.537799+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:54 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:54.537845+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:57 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:57.392219+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:57 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:57.392271+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:58 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:58.53895+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:30:58 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:30:58.539039+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:00 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:00.421608+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:39252","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}
Sep 26 16:31:01 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:01.831869+0200","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005ea540/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}
Sep 26 16:31:01 k3s-1.lab k3s[2603]: time="2023-09-26T16:31:01+02:00" level=warning msg="Learner  stalled at RaftAppliedIndex=0 for 4m15.831901633s"
Sep 26 16:31:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:02.392762+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:02.392827+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:02.539902+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:02 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:02.539936+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:06 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:06.540781+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:06 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:06.540834+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:07 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:07.393223+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:07 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:07.393258+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"11df2e51eff149b7","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:31:07 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:31:07.58403+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:48306","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}

Sep 26 16:29:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:34.508316+0200","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"https://<IP_ADDRESS>:2380/version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}
Sep 26 16:29:34 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:34.50837+0200","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"11df2e51eff149b7","error":"Get \"https://<IP_ADDRESS>:2380/version\": dial tcp <IP_ADDRESS>:2380: connect: connection refused"}

It looks like you haven't opened all the correct ports between your nodes? Make sure you've disabled ufw or firewalld.

Sep 26 16:29:35 k3s-1.lab k3s[2603]: {"level":"warn","ts":"2023-09-26T16:29:35.854632+0200","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"<IP_ADDRESS>:43814","server-name":"","ip-addresses":["<IP_ADDRESS>","<IP_ADDRESS>","::1","<IP_ADDRESS>","<IP_ADDRESS>"],"dns-names":["localhost","k3s-2.lab"],"error":"tls: \"<IP_ADDRESS>\" does not match any of DNSNames [\"localhost\" \"k3s-2.lab\"] (lookup <IP_ADDRESS>: Name does not resolve)"}

Why is this node seeing connections from <IP_ADDRESS>? Are you sure that your nodes are using the correct interfaces and routes to talk to each other? This feels like the nodes are going through another interface and the traffic is being NATed by the router.


fix(charts): fix auth endpoints cli templating

This PR corrects the templates to ensure the following behavior.

If a user provides a jwks or userinfo endpoint (or both), Chronicle uses them
If a user provides no endpoints but enables the devIdProvider, then Chronicle uses the endpoints for the devIdProvider
If a user provides no endpoints and the devIdProvider is not enabled, we do not want any cli args for jwks and userinfo endpoints passed to Chronicle.

PR Checklist
Please complete this checklist after opening your PR

[x] I have updated documentation as necessary
[x] I have updated helm chart(s) if needed
[x] I have added tests if needed
[x] All new and existing tests are passing
[x] Any breaking changes are clearly flagged and documented
[x] I’ve included a link to any relevant ticket(s)

It looks as if there's another case, which I'm okay with but may need clearly documenting: If the user supplies one of the endpoints and enables the dev-id-provider then the provider's used for the other endpoint that the user didn't supply. Basically: if you enable the dev-id-provider then both endpoints will be set, where user-provided URIs take priority over dev-id-provider ones.

It looks as if there's another case, which I'm okay with but may need clearly documenting: If the user supplies one of the endpoints and enables the dev-id-provider then the provider's used for the other endpoint that the user didn't supply. Basically: if you enable the dev-id-provider then both endpoints will be set, where user-provided URIs take priority over dev-id-provider ones.

@mtbc That is not the expected behavior for this PR. I didn't intend to be. And following your comment, I went through the different options and templated them out to check that things are behaving as I describe them. To confirm, the expected behavior is: if either auth.jwks.url or auth.userinfo.url are provided, Chronicle will be passed the values supplied by the user and not those of the devIdProvider. Even if the devIdProvider is enabled. Chronicle uses the devIdProvider endpoints only when the user provides no values for auth.jwks.url and/or auth.userinfo.url and devIdProvider is enabled, in which case Chronicle uses both devIdProvider endpoints.

Ah yes, just finishing retesting and it's exactly matching, earlier I must have failed to save a values.yaml from the editor or not waited long enough for the helm uninstall or something and ended up deploying not what I thought. :+1:


I cannot reproduce the performance of the model in the paper

First I used environment.yaml to configure the environment and start.sh to test on the diginetica dataset, here are the results for the last four epochs:
20%|██        | 6/30 [14:52<59:37, 149.08s/it]  Epoch 5: MRR = 19.008%, Hit = 55.406%
Batch 8500: Loss = 4.0826, Time Elapsed = 21.61s
Batch 8600: Loss = 4.0721, Time Elapsed = 9.60s
Batch 8700: Loss = 4.0687, Time Elapsed = 9.53s
Batch 8800: Loss = 4.0753, Time Elapsed = 9.51s
Batch 8900: Loss = 4.0801, Time Elapsed = 10.03s
Batch 9000: Loss = 4.0732, Time Elapsed = 9.77s
Batch 9100: Loss = 4.0731, Time Elapsed = 9.63s
Batch 9200: Loss = 4.0772, Time Elapsed = 9.47s
Batch 9300: Loss = 4.0917, Time Elapsed = 10.03s
Batch 9400: Loss = 4.0949, Time Elapsed = 9.88s
Batch 9500: Loss = 4.0945, Time Elapsed = 9.58s
Batch 9600: Loss = 4.0916, Time Elapsed = 9.70s
Batch 9700: Loss = 4.0954, Time Elapsed = 9.76s
Batch 9800: Loss = 4.1006, Time Elapsed = 9.96s
23%|██▎       | 7/30 [17:21<57:01, 148.75s/it]Epoch 6: MRR = 19.013%, Hit = 55.383%
Batch 9900: Loss = 4.0622, Time Elapsed = 21.49s
Batch 10000: Loss = 4.0686, Time Elapsed = 9.53s
Batch 10100: Loss = 4.0630, Time Elapsed = 9.81s
Batch 10200: Loss = 4.0753, Time Elapsed = 9.31s
Batch 10300: Loss = 4.0691, Time Elapsed = 9.69s
Batch 10400: Loss = 4.0818, Time Elapsed = 9.66s
Batch 10500: Loss = 4.0660, Time Elapsed = 9.58s
Batch 10600: Loss = 4.0944, Time Elapsed = 9.87s
Batch 10700: Loss = 4.0769, Time Elapsed = 9.77s
Batch 10800: Loss = 4.0792, Time Elapsed = 9.62s
Batch 10900: Loss = 4.0737, Time Elapsed = 9.85s
Batch 11000: Loss = 4.0862, Time Elapsed = 9.36s
Batch 11100: Loss = 4.0784, Time Elapsed = 9.50s
Batch 11200: Loss = 4.0959, Time Elapsed = 10.21s
Epoch 7: MRR = 19.006%, Hit = 55.379%
27%|██▋       | 8/30 [19:48<54:23, 148.35s/it]Batch 11300: Loss = 4.0781, Time Elapsed = 21.73s
Batch 11400: Loss = 4.0772, Time Elapsed = 9.75s
Batch 11500: Loss = 4.0463, Time Elapsed = 9.50s
Batch 11600: Loss = 4.0733, Time Elapsed = 9.85s
Batch 11700: Loss = 4.0664, Time Elapsed = 9.95s
Batch 11800: Loss = 4.0548, Time Elapsed = 9.84s
Batch 11900: Loss = 4.0820, Time Elapsed = 9.80s
Batch 12000: Loss = 4.0775, Time Elapsed = 9.51s
Batch 12100: Loss = 4.0711, Time Elapsed = 9.55s
Batch 12200: Loss = 4.0749, Time Elapsed = 9.63s
Batch 12300: Loss = 4.0626, Time Elapsed = 9.86s
Batch 12400: Loss = 4.0845, Time Elapsed = 9.82s
Batch 12500: Loss = 4.0734, Time Elapsed = 9.45s
Batch 12600: Loss = 4.0856, Time Elapsed = 9.66s
27%|██▋       | 8/30 [22:16<1:01:14, 167.02s/it]
Epoch 8: MRR = 19.006%, Hit = 55.379%
MRR@20	HR@20
19.015%	55.406%
Process finished with exit code 0

You can try to add "--extra" and "--fusion" on the command line of start.sh and increase the order to get better performance.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v5.15.19",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d5755832a1e47f5d8773f0776e211ecd4e02da72"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "d5755832a1e47f5d8773f0776e211ecd4e02da72",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "usb: xhci-plat: fix crash when suspend if remote wake enable\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.15.19 by commit d5755832a1e47f5d8773f0776e211ecd4e02da72. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000249


Es/plug translation

v1.1.0 Plug translation to spanish related #1043
cc @erickgnavar

Howdy @edwinallenz!  Thank you again for your contributions.
It looks like you've got a lot of extra commits on your branch, could you rebase?

will do and let you know @doomspork

@erickgnavar Done.
cc @doomspork

Awesome!  Thank you @edwinallenz and @erickgnavar


Documentation for subnet

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description
Please add more documentations for the subnet submodule and variable.

Just in case it helps anyone else passing here, saving you the time it took me to work this out.
  subnets                  = { "red": { cidr_block = "<IP_ADDRESS>/24",
			                dns_label = "red"
                                      },
                              "blue": { cidr_block = "<IP_ADDRESS>/24",
			                dns_label = "blue"
                                      }
                             }


[lilv] update to 0.24.24

Fixes https://github.com/microsoft/vcpkg/issues/42530

Update the following ports
lilv 0.24.24
lv2 1.18.10
sord 0.16.16
serd 0.32.2
add new port zix.
Modify all port compilation methods to use upstream meson compilation.
Repair lv2 dynamic library installation path.
Fixed the issue of zix function adaptation under Windows.

[x] Changes comply with the maintainer guide.
[x] SHA512s are updated for each updated download.
[ ] ~The "supports" clause reflects platforms that may be fixed by this new version.~
[ ] ~Any fixed CI baseline entries are removed from that file.~
[ ] ~Any patches that are no longer applied are deleted from the port's directory.~
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

Usage test passed with x64-windows triplet.

Please note, that there is now also another PR for the serd update: https://github.com/microsoft/vcpkg/pull/42539, as I created two seperated issues for updating serd and lilv.

Thank you for the big effort of updating all the sub-ports as well!


UI. Update Input component

Input should not accept className & wrapperClass
Move styles from wrapper component to native <input />
Adjust prefix/suffix elements in order to input have proper padding
Fix SearchInput clear button (doesn't work currently)

related to #2490

Status verification: ✅


Fix readme links

The getting started link wasn't working and then I noticed the repo changed owner and links hadn't been updated, so I went ahead and updated them :+1:

Sweet! Thanks. Yeah, Clearsight and InfiniteRed have merged!

Thanks @acrogenesis !


GPU PBD Solver for Cloth

Tonemaps + Expose adjustment

Cloth Sim Sample completed. Requires some cleanup which will occur in another issue.


DropKick errors out when not mounted on mobile.

When using the new Dropkick(elem, {mobile: false}) configuration, DropKick will crash when calling methods that manipulate Dropkick like .refresh() and .dispose() on mobile. DropKick will attempt to remove itself even though it was never mounted causing an error.
The hack fix I found was to check that dropkick.data.elem.parentNode has a value before calling any of the DropKick functionality. This check should probably be added to DropKick before performing DOM manipulations or offer a function for users to easily check if DropKick was actually mounted.

Another option would be to just have DropKick sto

Hi @nrdobie! I can't replicate this issue. Here is a JSBin I used to try and replicate it: http://jsbin.com/zeruhum/2/edit?html,js,output

After doing more digging it seems this was fixed with #310