Autoscaler does not scale up. predicate checking error: Insufficient ephemeral-storage.

Description
I just realized that my kube-hetzner cluster does not scale anylonger. As far as I remember it was actually working for a long time (could be that I just thought it was working, but never really saw it).
Inspecting journalctl logs I found that the key error seems to be inside systemd k3s-agent.service. It logs:
k3s-agent.service: Failed to locate executable /usr/local/bin/k3s: Permission denied

I made a ls -l to inspect the directory, showing
# ls -l /usr/local/bin
total 58068
lrwxrwxrwx. 1 root root        3 Feb 24 09:32 crictl -> k3s
lrwxrwxrwx. 1 root root        3 Feb 24 09:32 ctr -> k3s
-rwxr-xr-x. 1 root root 59441152 Feb 24 09:32 k3s
-rwxr-xr-x. 1 root root     1565 Feb 24 09:32 k3s-agent-uninstall.sh
-rwxr-xr-x. 1 root root     2274 Feb 24 09:32 k3s-killall.sh
lrwxrwxrwx. 1 root root        3 Feb 24 09:32 kubectl -> k3s

Which looks fine, doesn't it?
Googling shows this issue https://github.com/k3s-io/k3s/issues/5903  which links to a workaround ln -s /data/k3s /var/lib/rancher/k3s. I have no clue how this relates.

I am really buffled to see such critical bugs of the cluster because I thought I fixed almost any version I could find.
Hence two questions:

how to fix this? ANSWER: upgrade to most recent kube-hetzner version
which version do I need to fix where in order to prevent such problems appearing spontanuously? ANSWER: set automatically_upgrade_k3s = false in your kube.tf

EDIT: New Problem, autoscaler still not working
After updating everything to the most recent kube-hetzner (including microos snapshots), the above error disappears, however a new one still prevents the autoscaling from working.
The autoscaler logs show something like:
fails because of `k3s-agent.service: Failed to locate executable /usr/local/bin/k3s: Permission denied

Which prevents the scale up.

Kube.tf file

locals {
  # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env
  # within your shell, such as such: export TF_VAR_hcloud_token=xxxxxxxxxxx
  # If you choose to define it in the shell, this can be left as is.

  # Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).
  hcloud_token = "xxxxxxxxxxx"

  # to get the corresponding etcd_version for a k3s version you need to
  # - start k3s
  # - run `curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://<IP_ADDRESS>:2379/version`
  # for details see https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408
  etcd_version = "v3.5.9"
  initial_k3s_channel = "v1.26"

}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  k3s_token = var.k3s_token

  # using restoration, the timing for the nodes to respond is a bit larger
  # load_balancer_health_check_interval = "45s"
  # load_balancer_health_check_timeout = "30s"
  # load_balancer_health_check_retries = "40"

  # Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you
  # could adapt them to your needs.

  # The cluster name, by default "k3s"
  cluster_name = var.cluster_name

  ###################
  # FIXING VERSIONS #
  ###################

  # * For local dev, path to the git repo
  # source = "../../kube-hetzner/"
  # If you want to use the latest master branch
  # source = "github.com/schlichtanders/terraform-hcloud-kube-hetzner?ref=load-balancer-create"
  # source = "github.com/schlichtanders/terraform-hcloud-kube-hetzner?ref=postinstall_exec2"

  # # For normal use, this is the path to the terraform registry
  # # You can optionally specify a version number - for the registry
  source = "kube-hetzner/kube-hetzner/hcloud"
  version = "2.9.2"

  # If you want to use a specific Hetzner CCM and CSI version, set them below; otherwise, leave them as-is for the latest versions.
  # https://github.com/hetznercloud/hcloud-cloud-controller-manager
  hetzner_ccm_version = "v1.18.0"  # buggy?
  # hetzner_ccm_version = "v1.17.2"

  # https://github.com/hetznercloud/csi-driver
  hetzner_csi_version = "v2.5.1"

  # If you want to specify the Kured version, set it below - otherwise it'll use the latest version available.
  # https://github.com/kubereboot/kured
  kured_version = "1.14.0"

  # Allows you to specify either stable, latest, testing or supported minor versions.
  # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels
  # ⚠ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to latest - 0.01,
  # ⚠ Rancher currently only supports v1.25 and earlier versions: https://github.com/rancher/rancher/issues/41113
  # The default is "v1.26".
  initial_k3s_channel = local.initial_k3s_channel

  # You can choose the version of Calico that you want. By default, the latest is used.
  # More info on available versions can be found at https://github.com/projectcalico/calico/releases
  # Please note that if you are getting 403s from Github, it's also useful to set the version manually. However there is rarely a need for that!
  calico_version = "v3.26.1"

  #######################
  # END FIXING VERSIONS #
  #######################

  # Note that some values, notably "location" and "public_key" have no effect after initializing the cluster.
  # This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update
  # those, you should instead change the value here and manually re-provision each node. Grep for "lifecycle".

  # Customize the SSH port (by default 22)
  # ssh_port = 2222

  # * Your ssh public key
  ssh_public_key = file(var.ssh_public_key_file)
  # * Your private key must be "ssh_private_key = null" when you want to use ssh-agent for a Yubikey-like device authentification or an SSH key-pair with a passphrase.
  # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md
  ssh_private_key = file(var.ssh_private_key_file)
  # You can add additional SSH public Keys to grant other team members root access to your cluster nodes.
  # ssh_additional_public_keys = []

  # You can also add additional SSH public Keys which are saved in the hetzner cloud by a label.
  # See https://docs.hetzner.cloud/#label-selector
  # ssh_hcloud_key_label = "role=admin"

  # If you want to use an ssh key that is already registered within hetzner cloud, you can pass its id.
  # If no id is passed, a new ssh key will be registered within hetzner cloud.
  # It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` vars.
  hcloud_ssh_key_id = var.hcloud_ssh_key_id

  # These can be customized, or left with the default values
  # * For Hetzner locations see https://docs.hetzner.com/general/others/data-centers-and-connection/
  network_region = "eu-central" # change to `us-east` if location is ash

  # If you must change the network CIDR you can do so below, but it is highly advised against.
  # network_ipv4_cidr = "<IP_ADDRESS>/8"

  # If you must change the cluster CIDR you can do so below, but it is highly advised against.
  # Cluster CIDR must be a part of the network CIDR!
  # cluster_ipv4_cidr = "<IP_ADDRESS>/16"

  # For the control planes, at least three nodes are the minimum for HA. Otherwise, you need to turn off the automatic upgrades (see README).
  # **It must always be an ODD number, never even!** Search the internet for "splitbrain problem with etcd" or see https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/
  # For instance, one is ok (non-HA), two is not ok, and three is ok (becomes HA). It does not matter if they are in the same nodepool or not! So they can be in different locations and of various types.

  # Of course, you can choose any number of nodepools you want, with the location you want. The only constraint on the location is that you need to stay in the same network region, Europe, or the US.
  # For the server type, the minimum instance supported is cpx11 (just a few cents more than cx11); see https://www.hetzner.com/cloud.

  # IMPORTANT: Before you create your cluster, you can do anything you want with the nodepools, but you need at least one of each, control plane and agent.
  # Once the cluster is up and running, you can change nodepool count and even set it to 0 (in the case of the first control-plane nodepool, the minimum is 1).
  # You can also rename it (if the count is 0), but do not remove a nodepool from the list.

  # The only nodepools that are safe to remove from the list are at the end. That is due to how subnets and IPs get allocated (FILO).
  # You can, however, freely add other nodepools at the end of each list if you want. The maximum number of nodepools you can create combined for both lists is 255.
  # Also, before decreasing the count of any nodepools to 0, it's essential to drain and cordon the nodes in question. Otherwise, it will leave your cluster in a bad state.

  # Before initializing the cluster, you can change all parameters and add or remove any nodepools. You need at least one nodepool of each kind, control plane, and agent.
  # The nodepool names are entirely arbitrary, you can choose whatever you want, but no special characters or underscore, and they must be unique; only alphanumeric characters and dashes are allowed.

  # If you want to have a single node cluster, have one control plane nodepools with a count of 1, and one agent nodepool with a count of 0.

  # Please note that changing labels and taints after the first run will have no effect. If needed, you can do that through Kubernetes directly.

  # ⚠ When choosing ARM cax* server types, for the moment they are only available in fsn1.
  # Muli-architecture clusters are OK for most use cases, as container underlying images tend to be multi-architecture too.

  # * Example below:

  control_plane_nodepools = [
    {
      # name        = "control-plane"
      name        = "control-${var.default_location}-cax41",
      server_type = "cax11",
      location    = var.default_location,
      labels      = [
        # no longer needed, but may be interesting when using klipper as the load balancer
        # as this seems to be the final option which counts
        # taken from https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/447#issuecomment-1350597300
        # "node.kubernetes.io/exclude-from-external-load-balancers=true",
      ],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
  ]

  agent_nodepools = [
    # Arm based nodes, currently available only in FSN location
    {
      # name        = "agent-arm-largest"
      name        = "agent-${var.default_location}-cax41",
      server_type = "cax41",
      location    = var.default_location,
      labels      = [],
      taints      = [],
      count       = 1,
    }
  ]
  # Add custom control plane configuration options here.
  # E.g to enable monitoring for etcd, proxy etc:
  # control_planes_custom_config = {
  #  etcd-expose-metrics = true,
  #  kube-controller-manager-arg = "bind-address=<IP_ADDRESS>",
  #  kube-proxy-arg ="metrics-bind-address=<IP_ADDRESS>",
  #  kube-scheduler-arg = "bind-address=<IP_ADDRESS>",
  # }

  # You can enable encrypted wireguard for the CNI by setting this to "true". Default is "false".
  # FYI, Hetzner says "Traffic between cloud servers inside a Network is private and isolated, but not automatically encrypted."
  # Source: https://docs.hetzner.com/cloud/networks/faq/#is-traffic-inside-hetzner-cloud-networks-encrypted
  # It works with all CNIs that we support.
  # Just note, that if Cilium with cilium_values, the responsability of enabling of disabling Wireguard falls on you.
  enable_wireguard = true

  # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer
  load_balancer_type     = "lb11"
  load_balancer_location = var.default_location

  ### The following values are entirely optional (and can be removed from this if unused)

  # Cluster Autoscaler
  # Providing at least one map for the array enables the cluster autoscaler feature, default is disabled
  # By default we set a compatible version with the default initial_k3s_channel, to set another one,
  # have a look at the tag value in https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/values.yaml
  # ⚠ Based on how the autoscaler works with this project, you can only choose either x86 instances or ARM server types for ALL autocaler nodepools.
  # Also, as mentioned above, for the time being ARM cax* instances are only available in fsn1.
  # If you are curious, it's ok to have a multi-architecture cluster, as most underlying container images are multi-architecture too.
  # * Example below:
  autoscaler_nodepools = [
    {
      # "ca", as short for "cluster-autoscaler" - this is the common abbreviation
      # this needs to be really short, because a string like "-432f51dcc918aeba" is appended,
      # and the total string must be 63 characters maximum!
      # SUPER IMPORTANT: The prefix is used internaly to distinguish autoscaler nodes from other node types
      # search for "${var.cluster_name}-ca-" and change it too if you change the name here
      name        = "ca-${var.default_location}-cax41"  # it seems the node
      # name        = "autoscaled-arm-largest"
      server_type = "cax41"
      location    = var.default_location
      min_nodes   = 0  # somehow a first autoscaler node is spawned even if `min_nodes = 0`, for updates see https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/756
      max_nodes   = 4
    }
  ]

  # Enable etcd snapshot backups to S3 storage.
  # Just provide a map with the needed settings (according to your S3 storage provider) and backups to S3 will
  # be enabled (with the default settings for etcd snapshots).
  # Cloudflare's R2 offers 10GB, 10 million reads and 1 million writes per month for free.
  # For proper context, have a look at https://docs.k3s.io/backup-restore.
  etcd_s3_backup = {
    etcd-s3-endpoint   = var.etcd_s3_endpoint
    etcd-s3-access-key = var.etcd_s3_access_key
    etcd-s3-secret-key = var.etcd_s3_secret_key
    etcd-s3-bucket     = var.etcd_s3_bucket
  }

  # To enable Hetzner Storage Box support, you can enable csi-driver-smb, default is "false".
  # enable_csi_driver_smb = true

  # To use local storage on the nodes, you can enable Longhorn, default is "false".
  # See a full recap on how to configure agent nodepools for longhorn here https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/373#discussioncomment-3983159
  # Also see Longhorn best practices here https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b
  enable_longhorn = true

  # By default, longhorn is pulled from https://charts.longhorn.io.
  # If you need a version of longhorn which assures compatibility with rancher you can set this variable to https://charts.rancher.io.
  # longhorn_repository = "https://charts.rancher.io"

  # The namespace for longhorn deployment, default is "longhorn-system".
  # longhorn_namespace = "longhorn-system"

  # The file system type for Longhorn, if enabled (ext4 is the default, otherwise you can choose xfs).
  # longhorn_fstype = "xfs"

  # how many replica volumes should longhorn create (default is 3).
  longhorn_replica_count = 2

  # When you enable Longhorn, you can go with the default settings and just modify the above two variables OR you can add a longhorn_values variable
  # with all needed helm values, see towards the end of the file in the advanced section.
  # If that file is present, the system will use it during the deploy, if not it will use the default values with the two variable above that can be customized.
  # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.

  # Also, you can choose to use a Hetzner volume with Longhorn. By default, it will use the nodes own storage space, but if you add an attribute of
  # longhorn_volume_size (⚠ not a variable, just a possible agent nodepool attribute) with a value between 10 and 10000 GB to your agent nodepool definition, it will create and use the volume in question.
  # See the agent nodepool section for an example of how to do that.

  # To disable Hetzner CSI storage, you can set the following to "true", default is "false".
  # disable_hetzner_csi = true

  # If you want to use a specific Hetzner CCM and CSI version, set them below; otherwise, leave them as-is for the latest versions.
  # https://github.com/hetznercloud/hcloud-cloud-controller-manager
  # hetzner_ccm_version = "v1.15.0"
  # https://github.com/hetznercloud/csi-driver
  # hetzner_csi_version = "v2.3.2"

  # If you want to specify the Kured version, set it below - otherwise it'll use the latest version available.
  # https://github.com/kubereboot/kured
  # kured_version = "1.13.1"

  # If you want to enable the Nginx ingress controller (https://kubernetes.github.io/ingress-nginx/) instead of Traefik, you can set this to "nginx". Default is "traefik".
  # By the default we load optimal Traefik and Nginx ingress controller config for Hetzner, however you may need to tweak it to your needs, so to do,
  # we allow you to add a traefik_values and nginx_values, see towards the end of this file in the advanced section.
  # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.
  # If you want to disable both controllers set this to "none"
  # ingress_controller = "nginx"

  # You can change the number of replicas for selected ingress controller here. The default 0 means autoselecting based on number of agent nodes (1 node = 1 replica, 2 nodes = 2 replicas, 3+ nodes = 3 replicas)
  # ingress_replica_count = 1

  # Use the klipperLB (similar to metalLB), instead of the default Hetzner one, that has an advantage of dropping the cost of the setup.
  # Automatically "true" in the case of single node cluster (as it does not make sense to use the Hetzner LB in that situation).
  # It can work with any ingress controller that you choose to deploy.
  # Please note that because the klipperLB points to all nodes, we automatically allow scheduling on the control plane when it is active.
  # enable_klipper_metal_lb = "true"

  # If you want to configure additional arguments for traefik, enter them here as a list and in the form of traefik CLI arguments; see https://doc.traefik.io/traefik/reference/static-configuration/cli/
  # They are the options that go into the additionalArguments section of the Traefik helm values file.
  # Example: traefik_additional_options = ["--log.level=DEBUG", "--tracing=true"]
  # traefik_additional_options = []

  # By default traefik is configured to redirect http traffic to https, you can set this to "false" to disable the redirection.
  # traefik_redirect_to_https = false

  # If you want to disable the metric server set this to "false". Default is "true".
  # enable_metrics_server = false

  # If you want to allow non-control-plane workloads to run on the control-plane nodes, set this to "true". The default is "false".
  # True by default for single node clusters, and when enable_klipper_metal_lb is true. In those cases, the value below will be ignored.
  # allow_scheduling_on_control_plane = true

  # If you want to disable the automatic upgrade of k3s, you can set below to "false".
  # Ideally, keep it on, to always have the latest Kubernetes version, but lock the initial_k3s_channel to a kube major version,
  # of your choice, like v1.25 or v1.26. That way you get the best of both worlds without the breaking changes risk.
  # For production use, always use an HA setup with at least 3 control-plane nodes and 2 agents, and keep this on for maximum security.

  # The default is "true" (in HA setup i.e. at least 3 control plane nodes & 2 agents, just keep it enabled since it works flawlessly).
  # automatically_upgrade_k3s = false

  # The default is "true" (in HA setup it works wonderfully well, with automatic roll-back to the previous snapshot in case of an issue).
  # IMPORTANT! For non-HA clusters i.e. when the number of control-plane nodes is < 3, you have to turn it off.
  automatically_upgrade_os = false

  # If you need more control over kured and the reboot behaviour, you can pass additional options to kured.
  # For example limiting reboots to certain timeframes. For all options see: https://kured.dev/docs/configuration/
  # The default options are: `--reboot-command=/usr/bin/systemctl reboot --pre-reboot-node-labels=kured=rebooting --post-reboot-node-labels=kured=done --period=5m`
  # Defaults can be overridden by using the same key.
  # kured_options = {
  #   "reboot-days": "su"
  #   "start-time": "3am"
  #   "end-time": "8am"
  #   "time-zone": "Local"
  # }

  # Allows you to specify either stable, latest, testing or supported minor versions.
  # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels
  # ⚠ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to latest - 0.01,
  # ⚠ Rancher currently only supports v1.25 and earlier versions: https://github.com/rancher/rancher/issues/41113
  # The default is "v1.26".
  # initial_k3s_channel = "stable"

  # Whether to use the cluster name in the node name, in the form of {cluster_name}-{nodepool_name}, the default is "true".
  # use_cluster_name_in_node_name = false

  # Extra k3s registries. This is useful if you have private registries and you want to pull images without additional secrets.
  # Or if you want to proxy registries for various reasons like rate-limiting.
  # It will create the registries.yaml file, more info here https://docs.k3s.io/installation/private-registry.
  # Note that you do not need to get this right from the first time, you can update it when you want during the life of your cluster.
  # The default is blank.
  /* k3s_registries = <<-EOT
    mirrors:
      hub.my_registry.com:
        endpoint:
          - "hub.my_registry.com"
    configs:
      hub.my_registry.com:
        auth:
          username: username
          password: password
  EOT */

  # Additional environment variables for the host OS on which k3s runs. See for example https://docs.k3s.io/advanced#configuring-an-http-proxy .
  # additional_k3s_environment = {
  #   "CONTAINERD_HTTP_PROXY" : "http://your.proxy:port",
  #   "CONTAINERD_HTTPS_PROXY" : "http://your.proxy:port",
  #   "NO_PROXY" : "<IP_ADDRESS>/8,<IP_ADDRESS>/8,",
  # }

  # Additional commands to execute on the host OS before the k3s install, for example fetching and installing certs.
  # preinstall_exec = [
  #   "curl https://somewhere.over.the.rainbow/ca.crt > /root/ca.crt",
  #   "trust anchor --store /root/ca.crt",
  # ]

  preinstall_exec = [

    # This is adding node-taints to the config
    # it is safest to do it here, because node-taints may have already been added, and setting --node-taint attribute to k3s agent args will clean out all the other node-taints
    # we check whether the csi-node-driver can be loaded
    # if we get 403 response, this means that we got a bad IP
    # adapted from https://stackoverflow.com/questions/53526188/can-i-have-curl-print-just-the-response-code

    # simply fail if a wrong IP could be identified
    # (this prevents request to hcloud api as long as this node still exists)
    # node that `%%{` is the escaped version of a literal `%{`
    <<-EOF
    RESPONSE_CODE=$(curl -IL --silent --write-out "%%{http_code}\n" -o /dev/null https://registry.k8s.io/v2/sig-storage/csi-node-driver-registrar/manifests/v2.7.0)
    echo "hostname = '$(hostname)'. prefix = '${var.cluster_name}-ca-'. RESPONSE_CODE = '$RESPONSE_CODE'" > /tmp/iamhere
    if [ $(hostname) != "${var.cluster_name}-ca-" ]; then
      # fail immediately if not on autoscaler, because the initial nodes really need to be clean
      [ "403" != "$RESPONSE_CODE" ]
    else
      # if on an autoscaler node, just taint the node, so that autoscaler does not fail and the node is cleanup later automatically
      # we can simply add a second config to be aggregated see https://docs.k3s.io/installation/configuration
      mkdir -p /etc/rancher/k3s/config.yaml.d
      [ "403" != "$RESPONSE_CODE" ] || cat > /etc/rancher/k3s/config.yaml.d/jolin.yaml <<END
    node-taint+:
    - jolin=badip:NoSchedule
    END
    fi
    EOF
    ,
    # we can simply add a second config to be aggregated
    # see https://docs.k3s.io/installation/configuration
    # node that `%%{` is the escaped version of a literal `%{`

    # <<-EOF
    # mkdir -p /etc/rancher/k3s/config.yaml.d && \
    # [ "403" == $(curl -IL --silent --write-out "%%{http_code}\n" -o /dev/null https://registry.k8s.io/v2/sig-storage/csi-node-driver-registrar/manifests/v2.7.0) ] && \
    # cat > /etc/rancher/k3s/config.yaml.d/jolin.yaml <<END
    # node-taint+:
    # - jolin=badip:NoSchedule
    # END
    # EOF
    # ,
  ]

  postinstall_exec = var.etcd_snapshot_name == "" ? [] : [
    <<-EOF
    export CLUSTERINIT=$(cat /etc/rancher/k3s/config.yaml | grep -i '"cluster-init": true')
    if [ -n "$CLUSTERINIT" ]; then
      echo indeed this is the first control plane node > /tmp/restorenotes
      k3s server \
        --cluster-reset \
        --etcd-s3 \
        --cluster-reset-restore-path=${var.etcd_snapshot_name} \
        --etcd-s3-endpoint=${var.etcd_s3_endpoint} \
        --etcd-s3-bucket=${var.etcd_s3_bucket} \
        --etcd-s3-access-key=${var.etcd_s3_access_key} \
        --etcd-s3-secret-key=${var.etcd_s3_secret_key}
      # renaming the k3s.yaml because it is used as a trigger for further downstream
      # changes. Better to let `k3s server` create it as expected.
      mv /etc/rancher/k3s/k3s.yaml /etc/rancher/k3s/k3s.backup.yaml

      # download etcd/etcdctl for adapting the kubernetes config before starting k3s
      ETCD_VER=${local.etcd_version}
      case "$(uname -m)" in
          aarch64) ETCD_ARCH="arm64" ;;
          x86_64) ETCD_ARCH="amd64" ;;
      esac;
      DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download
      rm -f /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz
      curl -L $DOWNLOAD_URL/$ETCD_VER/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -o /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz
      tar xzvf /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -C /usr/local/bin --strip-components=1
      rm -f /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz

      etcd --version
      etcdctl version

      # delete traefik service so that no load-balancer is accidently changed
      nohup etcd --data-dir /var/lib/rancher/k3s/server/db/etcd &
      echo $! > save_pid.txt
      etcdctl del /registry/services/specs/traefik/traefik
      etcdctl del /registry/services/endpoints/traefik/traefik
      # delete old nodes (they interfere with load balancer)
      # minions is the old name for "nodes"
      OLD_NODES=$(etcdctl get "" --prefix --keys-only | grep /registry/minions/ | cut -c 19-)
      for NODE in $OLD_NODES; do
        for KEY in $(etcdctl get "" --prefix --keys-only | grep $NODE); do
          etcdctl del $KEY
        done
      done

      kill -9 `cat save_pid.txt`
      rm save_pid.txt
    else
      echo this is not the first control plane node > /tmp/restorenotes
    fi
    EOF
  ]

  # firstcontrolplane_kubectlisready_exec = [
  #   "kubectl delete service/traefik -n traefik || true"
  # ]

  # Additional flags to pass to the k3s server command (the control plane).
  # k3s_exec_server_args = "--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector"
  # we need to repeat the default kubelet-arg as CLI arguments take precendence. See https://docs.k3s.io/installation/configuration#configuration-file
  # default kubelet_arg = ["cloud-provider=external", "volume-plugin-dir=/var/lib/kubelet/volumeplugins"]

  # If you want to allow all outbound traffic you can set this to "false". Default is "true".
  # restrict_outbound_traffic = false

  # Adding extra firewall rules, like opening a port
  # More info on the format here https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/firewall
  extra_firewall_rules = [
    # {
    #   description = "For Postgres"
    #   direction       = "in"
    #   protocol        = "tcp"
    #   port            = "5432"
    #   source_ips      = ["<IP_ADDRESS>/0", "::/0"]
    #   destination_ips = [] # Won't be used for this rule
    # },
    # {
    #   description = "To Allow ArgoCD (or ssh-keyscan) access to resources via SSH"
    #   direction       = "out"
    #   protocol        = "tcp"
    #   port            = "22"
    #   source_ips      = [] # Won't be used for this rule
    #   destination_ips = ["<IP_ADDRESS>/0", "::/0"]
    # }
    {
      description = "Allow any outward access. To Allow ArgoCD (or ssh-keyscan) access to resources via SSH, access to Databases via special ports, etc."
      direction       = "out"
      protocol        = "tcp"
      port            = "any"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["<IP_ADDRESS>/0", "::/0"]
    }
  ]

  # If you want to configure a different CNI for k3s, use this flag
  # possible values: flannel (Default), calico, and cilium
  # As for Cilium, we allow infinite configurations via helm values, please check the CNI section of the readme over at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/#cni.
  # Also, see the cilium_values at towards the end of this file, in the advanced section.
  # cni_plugin = "cilium"

  # You can choose the version of Calico that you want. By default, the latest is used.
  # More info on available versions can be found at https://github.com/projectcalico/calico/releases
  # Please note that if you are getting 403s from Github, it's also useful to set the version manually. However there is rarely a need for that!
  # calico_version = "v3.25.0"

  # If you want to disable the k3s default network policy controller, use this flag!
  # Both Calico and Ciliun cni_plugin values override this value to true automatically, the default is "false".
  # disable_network_policy = true

  # If you want to disable the automatic use of placement group "spread". See https://docs.hetzner.com/cloud/placement-groups/overview/
  # We advise to not touch that setting, unless you have a specific purpose.
  # The default is "false", meaning it's enabled by default.
  # placement_group_disable = true

  # By default, we allow ICMP ping in to the nodes, to check for liveness for instance. If you do not want to allow that, you can. Just set this flag to true (false by default).
  # block_icmp_ping_in = true

  # You can enable cert-manager (installed by Helm behind the scenes) with the following flag, the default is "true".
  # enable_cert_manager = false

  # IP Addresses to use for the DNS Servers, set to an empty list to use the ones provided by Hetzner, defaults to ["<IP_ADDRESS>", "<IP_ADDRESS>", "<IP_ADDRESS>"].
  # The number of different DNS servers is limited to 3 by Kubernetes itself.
  # dns_servers = []

  # When this is enabled, rather than the first node, all external traffic will be routed via a control-plane loadbalancer, allowing for high availability.
  # The default is false.
  # see https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/302
  use_control_plane_lb = false

  # Let's say you are not using the control plane LB solution above, and still want to have one hostname point to all your control-plane nodes.
  # You could create multiple A records of to let's say cp.cluster.my.org pointing to all of your control-plane nodes ips.
  # In which case, you need to define that hostname in the k3s TLS-SANs config to allow connection through it. It can be hostnames or IP addresses.
  # additional_tls_sans = ["cp.cluster.my.org"]

  # Oftentimes, you need to communicate to the cluster from inside the cluster itself, in which case it is important to set this value, as it will configure the hostname
  # at the load balancer level, and will save you from many slows downs when initiating communications from inside. Later on, you can point your DNS to the IP given
  # to the LB. And if you have other services pointing to it, you are also free to create CNAMES to point to it, or whatever you see fit.
  # If set, it will apply to either ingress controllers, Traefik or Ingress-Nginx.
  lb_hostname = var.lb_hostname
  # You can refine a base domain name to be use in this form of nodename.base_domain for setting the reserve dns inside Hetzner
  base_domain = var.lb_hostname

  # You can enable Rancher (installed by Helm behind the scenes) with the following flag, the default is "false".
  # ⚠ Rancher currently only supports Kubernetes v1.25 and earlier, you will need to set initial_k3s_channel to a supported version: https://github.com/rancher/rancher/issues/41113
  # When Rancher is enabled, it automatically installs cert-manager too, and it uses rancher's own self-signed certificates.
  # See for options https://rancher.com/docs/rancher/v2.0-v2.4/en/installation/resources/advanced/helm2/helm-rancher/#choose-your-ssl-configuration
  # The easiest thing is to leave everything as is (using the default rancher self-signed certificate) and put Cloudflare in front of it.
  # As for the number of replicas, by default it is set to the numbe of control plane nodes.
  # You can customized all of the above by adding a rancher_values variable see at the end of this file in the advanced section.
  # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.
  # IMPORTANT: Rancher's install is quite memory intensive, you will require at least 4GB if RAM, meaning cx21 server type (for your control plane).
  # ALSO, in order for Rancher to successfully deploy, you have to set the "rancher_hostname".
  # enable_rancher = true

  # If using Rancher you can set the Rancher hostname, it must be unique hostname even if you do not use it.
  # If not pointing the DNS, you can just port-forward locally via kubectl to get access to the dashboard.
  # If you already set the lb_hostname above and are using a Hetzner LB, you do not need to set this one, as it will be used by default.
  # But if you set this one explicitly, it will have preference over the lb_hostname in rancher settings.
  # rancher_hostname = "rancher.xyz.dev"

  # When Rancher is deployed, by default is uses the "latest" channel. But this can be customized.
  # The allowed values are "stable" or "latest".
  # rancher_install_channel = "stable"

  # Finally, you can specify a bootstrap-password for your rancher instance. Minimum 48 characters long!
  # If you leave empty, one will be generated for you.
  # (Can be used by another rancher2 provider to continue setup of rancher outside this module.)
  # rancher_bootstrap_password = ""

  # Separate from the above Rancher config (only use one or the other). You can import this cluster directly on an
  # an already active Rancher install. By clicking "import cluster" choosing "generic", giving it a name and pasting
  # the cluster registration url below. However, you can also ignore that and apply the url via kubectl as instructed
  # by Rancher in the wizard, and that would register your cluster too.
  # More information about the registration can be found here https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/
  # rancher_registration_manifest_url = "https://rancher.xyz.dev/v3/import/xxxxxxxxxxxxxxxxxxYYYYYYYYYYYYYYYYYYYzzzzzzzzzzzzzzzzzzzzz.yaml"

  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
  # extra_kustomize_parameters={}

  # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.
  # See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/349
  # When "false". The kubeconfig file can instead be created by executing: "terraform output --raw kubeconfig > cluster_kubeconfig.yaml"
  # Always be careful to not commit this file!
  # create_kubeconfig = false

  # Don't create the kustomize backup. This can be helpful for automation.
  # create_kustomization = false

  ### ADVANCED - Custom helm values for packages above (search _values if you want to located where those are mentioned upper in this file)
  # ⚠ Inside the _values variable below are examples, up to you to find out the best helm values possible, we do not provide support for customized helm values.
  # Please understand that the indentation is very important, inside the EOTs, as those are proper yaml helm values.
  # We advise you to use the default values, and only change them if you know what you are doing!

  # Cilium, all Cilium helm values can be found at https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   cilium_values = <<EOT
ipam:
  mode: kubernetes
devices: "eth1"
k8s:
  requireIPv4PodCIDR: true
kubeProxyReplacement: strict
l7Proxy: false
encryption:
  enabled: true
  type: wireguard
  EOT */

  # Cert manager, all cert-manager helm values can be found at https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   cert_manager_values = <<EOT
installCRDs: true
replicaCount: 3
webhook:
  replicaCount: 3
cainjector:
  replicaCount: 3
  EOT */

  # csi-driver-smb, all csi-driver-smb helm values can be found at https://github.com/kubernetes-csi/csi-driver-smb/blob/master/charts/latest/csi-driver-smb/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   csi_driver_smb_values = <<EOT
controller:
  name: csi-smb-controller
  replicas: 1
  runOnMaster: false
  runOnControlPlane: false
  resources:
    csiProvisioner:
      limits:
        memory: 300Mi
      requests:
        cpu: 10m
        memory: 20Mi
    livenessProbe:
      limits:
        memory: 100Mi
      requests:
        cpu: 10m
        memory: 20Mi
    smb:
      limits:
        memory: 200Mi
      requests:
        cpu: 10m
        memory: 20Mi
  EOT */

  # Longhorn, all Longhorn helm values can be found at https://github.com/longhorn/longhorn/blob/master/chart/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   longhorn_values = <<EOT
defaultSettings:
  defaultDataPath: /var/longhorn
persistence:
  defaultFsType: ext4
  defaultClassReplicaCount: 3
  defaultClass: true
  EOT */

  # Traefik, all Traefik helm values can be found at https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   traefik_values = <<EOT
deployment:
  replicas: 1
globalArguments: []
service:
  enabled: true
  type: LoadBalancer
  annotations:
    "load-balancer.hetzner.cloud/name": "k3s"
    "load-balancer.hetzner.cloud/use-private-ip": "true"
    "load-balancer.hetzner.cloud/disable-private-ingress": "true"
    "load-balancer.hetzner.cloud/location": "nbg1"
    "load-balancer.hetzner.cloud/type": "lb11"
    "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"

ports:
  web:
    redirectTo: websecure

    proxyProtocol:
      trustedIPs:
        - <IP_ADDRESS>/32
        - <IP_ADDRESS>/8
    forwardedHeaders:
      trustedIPs:
        - <IP_ADDRESS>/32
        - <IP_ADDRESS>/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - <IP_ADDRESS>/32
        - <IP_ADDRESS>/8
    forwardedHeaders:
      trustedIPs:
        - <IP_ADDRESS>/32
        - <IP_ADDRESS>/8
  EOT */

  # Nginx, all Nginx helm values can be found at https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml
  # You can also have a look at https://kubernetes.github.io/ingress-nginx/, to understand how it works, and all the options at your disposal.
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   nginx_values = <<EOT
controller:
  watchIngressWithoutClass: "true"
  kind: "DaemonSet"
  config:
    "use-forwarded-headers": "true"
    "compute-full-forwarded-for": "true"
    "use-proxy-protocol": "true"
  service:
    annotations:
      "load-balancer.hetzner.cloud/name": "k3s"
      "load-balancer.hetzner.cloud/use-private-ip": "true"
      "load-balancer.hetzner.cloud/disable-private-ingress": "true"
      "load-balancer.hetzner.cloud/location": "nbg1"
      "load-balancer.hetzner.cloud/type": "lb11"
      "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
  EOT */

  # Rancher, all Rancher helm values can be found at https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/
  # The following is an example, please note that the current indentation inside the EOT is important.
  /*   rancher_values = <<EOT
ingress:
  tls:
    source: "rancher"
hostname: "rancher.example.com"
replicas: 1
bootstrapPassword: "supermario"
  EOT */

}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.38.2"
    }
    github = {
      source  = "integrations/github"
      version = ">= 5.38.0, < 5.41.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

output "k3s_token" {
  value     = module.kube-hetzner.k3s_token
  sensitive = true
}

variable "cluster_name" {
  type   = string
}

variable "lb_hostname" {
  type   = string
}

variable "default_location" {
  type   = string
}

variable "ssh_private_key_file" {
  sensitive = true  # in case someone accidentally inputs the key itself
  type   = string
}

variable "ssh_public_key_file" {
  type   = string
}

variable "hcloud_ssh_key_id" {
  description = "create via `hcloud context use cloud.jolin.io && hcloud ssh-key describe cloud.jolin.io`"
  type   = string
  sensitive = true  # should not be needed, but better safe than sorry
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

variable "k3s_token" {
  sensitive = true
  type      = string
  default   = null
}

variable "etcd_snapshot_name" {
  sensitive = true
  default   = ""
}

variable "etcd_s3_endpoint" {
  sensitive = true
  default   = ""
}

variable "etcd_s3_bucket" {
  sensitive = true
  default   = ""
}

variable "etcd_s3_access_key" {
  sensitive = true
  default   = ""
}

variable "etcd_s3_secret_key" {
  sensitive = true
  default   = ""
}

Screenshots
No response
Platform
Linux

Just saw that this is kind of a duplicate. The error seems to happen on all nodes, not just autoscaler.
Can someone help me - is it safe to just upgrade to the latest kube-hetzner version and run terraform apply? Will it update the autoscaler? (and hopefully not destroy anything else?)

@schlichtanders If you are using v2.x it should be safe to update. Just run terraform plan to confirm. And if you see it replaces the kustomization, it's safe and a good sign. If it wants to replace a node, it's the only thing that's a big no-no, but that's unlikely. And yes, that should update the autoscaler.
And after you have done that, you can also replay with changing initial_k3s_channel to to v1.28 and that should also be safe, it will upgrade without downtime, super smooth (the system upgrade controller, just replaces the k3s binary and that's it, no need for a restart even). You can watch your nodes upgrade in realtime after applying by running watch kubectl get nodes.
That should hopefully fix it for you. Let us know!

Just to be super sure. Does the following terraform plan include some changes to nodes?
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.kube-hetzner.hcloud_load_balancer.cluster[0] will be updated in-place
  ~ resource "hcloud_load_balancer" "cluster" {
        id                 = "1471768"
      ~ name               = "20231001t152028z-cloud-jolin-io" -> "20231001t152028z-cloud-jolin-io-traefik"
        # (9 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # module.kube-hetzner.hcloud_network.k3s has moved to module.kube-hetzner.hcloud_network.k3s[0]
    resource "hcloud_network" "k3s" {
        id                       = "3403608"
        name                     = "20231001t152028z-cloud-jolin-io"
        # (4 unchanged attributes hidden)
    }

  # module.kube-hetzner.local_file.kustomization_backup[0] must be replaced
-/+ resource "local_file" "kustomization_backup" {
      ~ content              = <<-EOT # forces replacement
            "apiVersion": "kustomize.config.k8s.io/v1beta1"
            "kind": "Kustomization"
          - "patchesStrategicMerge":
          - - |
          -   apiVersion: apps/v1
          -   kind: Deployment
          -   metadata:
          -     name: system-upgrade-controller
          -     namespace: system-upgrade
          -   spec:
          -     template:
          -       spec:
          -         containers:
          -           - name: system-upgrade-controller
          -             volumeMounts:
          -               - name: ca-certificates
          -                 mountPath: /var/lib/ca-certificates
          -         volumes:
          -           - name: ca-certificates
          -             hostPath:
          -               path: /var/lib/ca-certificates
          -               type: Directory
          - - "kured.yaml"
          - - "ccm.yaml"
          + "patches":
          + - "patch": |
          +     apiVersion: apps/v1
          +     kind: Deployment
          +     metadata:
          +       name: system-upgrade-controller
          +       namespace: system-upgrade
          +     spec:
          +       template:
          +         spec:
          +           containers:
          +             - name: system-upgrade-controller
          +               volumeMounts:
          +                 - name: ca-certificates
          +                   mountPath: /var/lib/ca-certificates
          +           volumes:
          +             - name: ca-certificates
          +               hostPath:
          +                 path: /var/lib/ca-certificates
          +                 type: Directory
          +   "target":
          +     "group": "apps"
          +     "kind": "Deployment"
          +     "name": "system-upgrade-controller"
          +     "namespace": "system-upgrade"
          +     "version": "v1"
          + - "path": "kured.yaml"
          + - "path": "ccm.yaml"
            "resources":
            - "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.18.0/ccm-networks.yaml"
            - "https://github.com/kubereboot/kured/releases/download/1.14.0/kured-1.14.0-dockerhub.yaml"
            - "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
            - "hcloud-csi.yml"
            - "traefik_ingress.yaml"
            - "longhorn.yaml"
            - "cert_manager.yaml"
        EOT
      ~ content_base64sha256 = "lNpUo1iYAV0KdsMXjK89vXx721X7clMRjsYcoekMh9M=" -> (known after apply)
      ~ content_base64sha512 = "qpCSZ3ofys8sSJHjJuJgfNz51WWYFeyUjxP7NpKJcxrvDNeAxukSGO7NB/+VWpDhcsrX2fTUnawkI35KCcIviA==" -> (known after apply)
      ~ content_md5          = "dfb992de46ba593c58a3be8d34bdb078" -> (known after apply)
      ~ content_sha1         = "5d34df489f16e4b4927bac5bd6a6c5b0c5f84541" -> (known after apply)
      ~ content_sha256       = "94da54a35898015d0a76c3178caf3dbd7c7bdb55fb7253118ec61ca1e90c87d3" -> (known after apply)
      ~ content_sha512       = "aa9092677a1fcacf2c4891e326e2607cdcf9d5659815ec948f13fb369289731aef0cd780c6e91218eecd07ff955a90e172cad7d9f4d49dac24237e4a09c22f88" -> (known after apply)
      ~ id                   = "5d34df489f16e4b4927bac5bd6a6c5b0c5f84541" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.kube-hetzner.null_resource.agent_config["0-0-agent-nbg1-cax41"] will be created
  + resource "null_resource" "agent_config" {
      + id       = (known after apply)
      + triggers = {
          + "agent_id" = "37727681"
          + "config"   = (sensitive value)
        }
    }

  # module.kube-hetzner.null_resource.autoscaled_nodes_registries["20231001t152028z-cloud-jolin-io-ca-nbg1-cax41-2a8faee08034e953"] will be created
  + resource "null_resource" "autoscaled_nodes_registries" {
      + id       = (known after apply)
      + triggers = {
          + "registries" = " "
        }
    }

  # module.kube-hetzner.null_resource.configure_autoscaler[0] must be replaced
-/+ resource "null_resource" "configure_autoscaler" {
      ~ id       = "4444793812316690135" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "template" = (sensitive value)
        }
    }

  # module.kube-hetzner.null_resource.control_plane_config["0-0-control-nbg1-cax41"] will be created
  + resource "null_resource" "control_plane_config" {
      + id       = (known after apply)
      + triggers = {
          + "config"           = (sensitive value)
          + "control_plane_id" = "37727682"
        }
    }

  # module.kube-hetzner.null_resource.kustomization must be replaced
-/+ resource "null_resource" "kustomization" {
      ~ id       = "3574815439652385236" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "helm_values_yaml" = (sensitive value)
          ~ "options"          = <<-EOT
              -
              + period=5m
              + post-reboot-node-labels=kured=done
              + pre-reboot-node-labels=kured=rebooting
              + reboot-command=/usr/bin/systemctl reboot
              + reboot-sentinel=/sentinel/reboot-required
            EOT
          ~ "versions"         = <<-EOT
                v1.26
              - v1.27.3
              + 20231027
                v1.18.0
                v2.5.1
                1.14.0
                v3.26.1
              - v1.14.0
              + 1.15.1
              + N/A
              + N/A
            EOT
        }
    }

  # module.kube-hetzner.module.agents["0-0-agent-nbg1-cax41"].null_resource.zram will be created
  + resource "null_resource" "zram" {
      + id       = (known after apply)
      + triggers = {
          + "zram_size" = ""
        }
    }

  # module.kube-hetzner.module.control_planes["0-0-control-nbg1-cax41"].null_resource.zram will be created
  + resource "null_resource" "zram" {
      + id       = (known after apply)
      + triggers = {
          + "zram_size" = ""
        }
    }

Plan: 8 to add, 1 to change, 3 to destroy.

@schlichtanders Nope, all seems good! It will recreate the right stuff.

If the update does not solve your problem, it means that it's an selinux issue, and that cannot be solved with a simple update because the latter is applied only during cloudinit, so first boot. So as a remedy, you can use the following shell script on each of your nodes.
#!/bin/bash

# Create the SELinux policy module file
cat <<EOF >/root/kube_hetzner_selinux.te
module kube_hetzner_selinux 1.0;

require {
  type kernel_t, bin_t, kernel_generic_helper_t, iscsid_t, iscsid_exec_t, var_run_t,
  init_t, unlabeled_t, systemd_logind_t, systemd_hostnamed_t, container_t,
  cert_t, container_var_lib_t, etc_t, usr_t, container_file_t, container_log_t,
  container_share_t, container_runtime_exec_t, container_runtime_t, var_log_t, proc_t, io_uring_t;
  class key { read view };
  class file { open read execute execute_no_trans create link lock rename write append setattr unlink getattr watch };
  class sock_file { watch write create unlink };
  class unix_dgram_socket create;
  class unix_stream_socket { connectto read write };
  class dir { add_name create getattr link lock read rename remove_name reparent rmdir setattr unlink search write watch };
  class lnk_file { read create };
  class system module_request;
  class filesystem associate;
  class bpf map_create;
  class io_uring sqpoll;
  class anon_inode create;
}

#============= kernel_generic_helper_t ==============
allow kernel_generic_helper_t bin_t:file execute_no_trans;
allow kernel_generic_helper_t kernel_t:key { read view };
allow kernel_generic_helper_t self:unix_dgram_socket create;

#============= iscsid_t ==============
allow iscsid_t iscsid_exec_t:file execute;
allow iscsid_t var_run_t:sock_file write;
allow iscsid_t var_run_t:unix_stream_socket connectto;

#============= init_t ==============
allow init_t unlabeled_t:dir { add_name remove_name rmdir };
allow init_t unlabeled_t:lnk_file create;
allow init_t container_t:file { open read };

#============= systemd_logind_t ==============
allow systemd_logind_t unlabeled_t:dir search;

#============= systemd_hostnamed_t ==============
allow systemd_hostnamed_t unlabeled_t:dir search;

#============= container_t ==============
# Basic file and directory operations for specific types
allow container_t cert_t:dir read;
allow container_t cert_t:lnk_file read;
allow container_t cert_t:file { read open };
allow container_t container_var_lib_t:file { create open read write rename lock };
allow container_t etc_t:dir { add_name remove_name write create setattr watch };
allow container_t etc_t:file { create setattr unlink write };
allow container_t etc_t:sock_file { create unlink };
allow container_t usr_t:dir { add_name create getattr link lock read rename remove_name reparent rmdir setattr unlink search write };
allow container_t usr_t:file { append create execute getattr link lock read rename setattr unlink write };

# Additional rules for container_t
allow container_t container_file_t:file { open read write append getattr setattr };
allow container_t container_file_t:sock_file watch;
allow container_t container_log_t:file { open read write append getattr setattr };
allow container_t container_share_t:dir { read write add_name remove_name };
allow container_t container_share_t:file { read write create unlink };
allow container_t container_runtime_exec_t:file { read execute execute_no_trans open };
allow container_t container_runtime_t:unix_stream_socket { connectto read write };
allow container_t kernel_t:system module_request;
allow container_t container_log_t:dir { read watch };
allow container_t container_log_t:file { open read watch };
allow container_t container_log_t:lnk_file read;
allow container_t var_log_t:dir { add_name write };
allow container_t var_log_t:file { create lock open read setattr write };
allow container_t var_log_t:dir remove_name;
allow container_t var_log_t:file unlink;
allow container_t proc_t:filesystem associate;
allow container_t self:bpf map_create;
allow container_t io_uring_t:anon_inode create;
allow container_t self:io_uring sqpoll;
EOF

# Compile the SELinux policy module
checkmodule -M -m -o /root/kube_hetzner_selinux.mod /root/kube_hetzner_selinux.te

# Package the compiled policy module
semodule_package -o /root/kube_hetzner_selinux.pp -m /root/kube_hetzner_selinux.mod

# Load the policy package into the SELinux policy store
semodule -i /root/kube_hetzner_selinux.pp

echo "SELinux policy module 'kube_hetzner_selinux' has been installed."

Thank you for all the hints.
little update: I terraform applied to the new kube-hetzner version successfully, but the new nodes still do not connect. Could it be that the bug was actually not fixed for the autoscaler part? Just asking

@schlichtanders It could be. I will try to see if we need to update the autoscaler version. However, did you try the selinux bash script above?

Update: I realized that the terraform apply seems to have created a new cluster-autoscaler, which however has no resources because the original cluster autoscaler is still ill-functioning...
This is surprising - why hasn't it replaced the original cluster-autoscaler instead?

Thank you for your further support. I now changed the version to 1.28 and also fixed cluster_autoscaler_version.
Unfortunately the scaling still does not work.
The autoscaler is not even deployed. It shows the following error in the logs
Failed to create Hetzner manager: `HCLOUD_CLOUD_INIT` is not specified

Looking into the code, this HCLOUD_CLOUD_INIT variable is regarded as legacy... why? version 1.28.2 seems to expect it?
My production cluster is down since days because of this... It would be so great to get it working again :cry:

I think I am into a new bug, which might just have occured as part of the update process.
The autoscaler logs mention:
Couldn't get autoscaling options for ng

I think this means the new node could register, but the cluster still fails to autoscale...

I think I understood that problem... I deleted a autoscaling node on Hetzner manually, because it was not cleanedup (thought this happened because the autoscaler was restarted in the wrong moment).
I think by doing this I also deleted the node placement group which might be critical for the autoscaling to work

I know removed the autoscaling group from kube.tf, run terraform apply, and added it again, and run terraform apply, but no placement group is created on hetzner.
But the autoscaler still expects it to be there somehow... any help is highly appreciated.

It might be that this error log is actually not telling anything.
I found a recent blog post which also has the log, but as part of a verification which is evaluated successful, so this might not be the problem.
Maybe it is just because of the newer autoscaler version... that it does not scale up somehow

So back to the logs I think I now came down the following error as the most descriptive of why it does not scale:
I0226 11:12:14.198771       1 orchestrator.go:542] Pod mypod/mynamespace can't be scheduled on NAME_OF_AUTO_SCALING_GROUP, predicate checking error: Insufficient ephemeral-storage; predicateName=NodeResourcesFit; reasons: Insufficient ephemeral-storage; debugInfo=

This is only an info, but actually might be the key error. It seems like the autoscaler still thinks that there is a node with the same name as the auto scaling group itself. But there is nothing like this.

This is so complicated... I do not get it to work.
Please help - Autoscaling does not work with kube-hetzner, even worse, this bug appeared without me doing anything and destroyed my cluster.
Here the key logs put together from autoscaler
I0226 11:32:54.763746       1 klogx.go:87] Pod mypod is unschedulable
I0226 11:32:54.763783       1 orchestrator.go:108] Upcoming 0 nodes
I0226 11:32:54.763806       1 orchestrator.go:440] Skipping node group draining-node-pool - max size reached
E0226 11:32:54.763812       1 orchestrator.go:446] Couldn't get autoscaling options for ng: NAME_OF_AUTOSCALING_GROUP
I0226 11:32:54.763861       1 orchestrator.go:542] Pod mypod/mynamespace can't be scheduled on NAME_OF_AUTOSCALING_GROUP, predicate checking error: Insufficient ephemeral-storage; predicateName=NodeResourcesFit; reasons: Insufficient ephemeral-storage; debugInfo=
I0226 11:32:54.763891       1 orchestrator.go:542] Pod mypod/mynamespace can't be scheduled on NAME_OF_AUTOSCALING_GROUP, predicate checking error: Insufficient ephemeral-storage; predicateName=NodeResourcesFit; reasons: Insufficient ephemeral-storage; debugInfo=
I0226 11:32:54.763903       1 orchestrator.go:150] No pod can fit to NAME_OF_AUTOSCALING_GROUP
I0226 11:32:54.763913       1 orchestrator.go:164] No expansion options

@mysticaltech Which autoscaler - k3s version combination actually works? I couldn't get any to work... my cluster is still down

(I tried also by setting the minSize of the autoscaler group to 1 - it is ignored somehow, and no error is apparent in the logs actually, super weird.)

@schlichtanders
The ephemeral storage seems to be a bug which has been in autoscaler for ages. Can you post the pod spec it is trying to schedule? I think it seems to request for ephemeral storage

@schlichtanders  Please see @Silvest89 answer above. And seeing your other issues, as hotfix, set initial_k3s_channel to v1.28, with automatically_upgrade_k3s = true, and delete the lines with cluster_autoscaler_image and cluster_autoscaler_version to force the use of the default values. This will force your cluster to downgrade to v1.28 and it should work with the default autoscaler config we have.

	return apiv1.ResourceList{
		// TODO somehow determine the actual pods that will be running
		apiv1.ResourcePods:    *resource.NewQuantity(defaultPodAmountsLimit, resource.DecimalSI),
		apiv1.ResourceCPU:     *resource.NewQuantity(int64(typeInfo.Cores), resource.DecimalSI),
		apiv1.ResourceMemory:  *resource.NewQuantity(int64(typeInfo.Memory*1024*1024*1024), resource.DecimalSI),
		apiv1.ResourceStorage: *resource.NewQuantity(int64(typeInfo.Disk*1024*1024*1024), resource.DecimalSI),
	}, nil

I just looked through the autoscaler code
apiv1.ResourceEphemeralStorage is not implemented for hetzner

@mysticaltech
EphemeralStorage can be assigned to the same *resource.NewQuantity(int64(typeInfo.Disk*1024*1024*1024), resource.DecimalSI),
And that will be a one line fix :P

@Silvest89 You are the man, please do (I'm not good at golang)! So at least our own fork will, and then later if you could submit a PR to the mother repo it would be great, as you did last time 🙏

@mysticaltech But I like to test my shit as well :P, so it will probably take longer than this one line fix :P

@Silvest89  If you push the fix and do a successful release, I can do the testing lol

@mysticaltech
The workflow is running. When it is done, you will see a new tag (current date)

Thanks a lot @Silvest89! @schlichtanders So no need to downgrade, keep k3s at v1.29, fixing incoming, will tell you what changes need to be done, just two lines in your kube.tf. Keep you posted ASAP as soon as the build is successful.

@schlichtanders Here we go, in your kube.tf, remove the line with cluster_autoscaler_image as to force it to use the default value. And change cluster_autoscaler_version  to "20240226".
Also, make sure your kube.tf follows the same autoscaler nodepools definition as in the latest kube.tf.example. Change to min 0 and max 0, apply. Let it scale down if needed, if already scaled down, set your wanted values for min and max and apply again.
Let us know please 🙏

@schlichtanders @mysticaltech
Don't forget to set
cluster_autoscaler_image to ghcr.io/kube-hetzner/autoscaler/cluster-autoscaler

@schlichtanders  @Silvest89  Works like a charm on my end. Tried with both kube v1.28 and v1.29, it's autoscalling correctly!
@Silvest89 I can make this one the default, yes? On your side, if you could PR to the mother repo it would be dope!  🚀

Thank you for all your help. Still I am facing problems

the autoscaler fork uses 2 replicas. As my cluster does not scale up and has no resources I manually needed to delete the old autoscaler
this somehow was still not enough, hence I added further agent nodes so that the autoscalers have a place to be deployed
This also didn't work - I get Pending stage with node(s) didn't match Pod's node affinity/selector

Not sure what changed, but this is a new behaviour I haven't seen on the previous autoscalers...
debugging further

I unfortunately get the same affinity problem with 20231027 version

@schlichtanders It's one replica on my end. And what I do to test is I use a dummy pod definition, give it a lot of memory needs, enough for a full node, like 1500Mi for cpx11, and it works as in the nodes gets created and the pod is scheduled after the nodes joins.
You need to inspect the logs of the autoscaler pod. Also Insec my log level to 5, in the variable for this.

My autoscaler pod is not even starting - it is in Pending mode because of this not fulfilled Affinity restriction.
I am really surprised that the same autoscaler has different parameters on your side.
I am using
  source = "kube-hetzner/kube-hetzner/hcloud"
  version = "2.12.2"

which version are you using?

@schlichtanders It's one replica on my end. And what I do to test is I use a dummy pod definition, give it a lot of memory needs, enough for a full node, like 1500Mi for cpx11, and it works as in the nodes gets created and the pod is scheduled after the nodes joins.
You need to inspect the logs of the autoscaler pod. Also Insec my log level to 5, in the variable for this.
If you see two replicas it's probably two distinct deployments, you need to delete the old one.

I am using same latest version. And you need to debug by increasing your log level. And describing the autoscaler pod to make sure it's not an old version.

I inspected the yaml on kubernetes and it indeed says replicas: 1 under the definition, but under status it is still replicas: 2. Weird...
The log level is not useful yet, as the autoscaler pod is not even starting (sorry for repeating myself again).
I now delete the cluster-autoscaler manually in the cluster and reapplied terraform.
Result: Only one replica, but as expected, the affinity is still leaving it in Pending. There are no resources with this taint.
Have you tainted your nodes with node-role.kubernetes.io/master?

Researching the specific taint it looks like the cluster autoscaler should be scheduled on the control plane
https://kubernetes.io/docs/reference/labels-annotations-taints/#node-role-kubernetes-io-master-taint
It is a deprecated taint and should be replaced by node-role.kubernetes.io/control-plane
But I am confused about about the distinction between taints and labels here...

This is definitely the next bug. Also looking at the tolerations it should apparently be scheduled on the control plane.
Hence the affinity to node-role.kubernetes.io/master should be replaced with node-role.kubernetes.io/control-plane

@mysticaltech can you include that to your new pullrequest?

@schlichtanders Thanks for in-depth debugging. @Silvest89 what do you think?

@mysticaltech
The autoscaler yaml should be changed. All kubernetes versions now have control-plane since "master" is considered as one of harmful language today.
And there is no node with a master label anymore.
Change in autoscaler.yaml.tpl:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/master
                    operator: Exists

@mysticaltech please give me a ping if it is included in your pullrequest, then I can test whether it works for me

@Silvest89 Thanks for clarifying. On it. @schlichtanders Will ping you ASAP.

Thank you very much. It still does not work :grin:
But this one might be a special thing of my slightly older control-plane node. As already mentioned it has also the taint node.kubernetes.io/unschedulable:NoSchedule.
k8s docs suggest that this taint is rather temporary, "to avoid race conditions". However for me it is still there...
Any ideas, why it is there?

I just create a fresh cluster to see whether this taint appears and super surprising it appeared on one of the two agent nodes of my cluster. The control-plane does not have it this time...
I have no clue yet how to debug it why this taint node.kubernetes.io/unschedulable:NoSchedule is not cleaned up (or set initially)...

@schlichtanders Yes, try with kubectl taint nodes <node-name> node.kubernetes.io/unschedulable:NoSchedule-

Your command does not seem to have any effect. The node is also marked as Unscheduable: true maybe this is stronger.

I've found another command kubectl uncordon <node name> to make a node schedulable - is this safe to do on the control plane?

I just tried it :grin: and indeed this seems to have been the important part. Not sure why it was cordoned. If kube-hetzner was not doing this somewhen in the past, I might have, for some security guide I found somewhere maybe.
The autoscaler now works again :partying_face: :muscle: Thank you so much for your support. I feel released now
invalid capacity 0 on image filesystem
I just saw that the new nodes come up with an warning event InvalidDiskCapacity with description invalid capacity 0 on image filesystem.
There is one open question still
This should not have happened in the first place. I fixed all versions everywhere. But it still autoupdated itself somehow so that the whole cluster failed. This should never happen.
The only self-update I can see is the k3s channel. Was this creating the bug? Is there a way to prevent it from failing? How can I set a specific k3s version?
I think it is really better not to have the most recent security updates, but a functional cluster. I can then update manually after testing it beforehand.

Thank you. I actually already set automatic_os_upgrade = false, hence it was probably myself. Sorry for that.
Regarding the k3s_channel, if I understood correctly this "always works well" assessment is no longer true, is it?
Like that the autoscaler failed without any update to the cluster, just by it self-updating, that was happening exactly because such a minor patch release broke the register step for k3s.
@Silvest89 Please help me here understanding the issue. Do you also think that the k3s self-update was in the beginning introducing this bug?
I really would like to have a way of preventing this for the future.

You probably had a new pod/deployment which had a storage request. The hetzner provider never supported it. so that pod never made your cluster scale up.
requests:
  storage:

As for this one
Failed to create Hetzner manager: HCLOUD_CLOUD_INIT is not specified
This is because you used the latest kube-hetzner version, with an old autoscaler version which still expects the legacy implementation. The newer autoscaler version has backwards compatibility.
And yeah k3s_upgrade = true usually works fine. os_upgrade not. Since well we have had alot of SELINUX shit that we had to update :P But I am no SELINUX expert here.

Thank you for your assessment. I have request: ephemeral-storage:  kind of things in the deployments. But they had worked for a long time.
It feels a bit unsatisfying, as I really have now the same setup as before (with newer versions everywhere). So I must expect things to possibly fail somewhen because of these implicit updates...
I looked again into the other issues and it seems that people thought that it has something todo with os upgrades https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/1145#issuecomment-1878487927.
I have them disabled via automatically_upgrade_os = false, but if I understood it now correctly, this does not prevent the autoscaler from doing an OS upgrade on a new node. I guess this is the underlying instability.
Is there a way to fix the OS for the autoscaler?

@schlichtanders Maybe I did not explain myself clearly enough. automatically_upgrade_os = false works only for new nodes, not retroactively on old nodes, hence my recommendation to disable updates at the os level (see readme) for old nodes.

I feel misunderstood.
This is about the autoscaler. It failed to deploy new nodes because of Permission Denied on k3s executable.
It is not about already running nodes. These nodes got created and failed immediately after creation.
automatically_upgrade_os = false is set since ever on my cluster because I am using a single control node only.
So I think there must be another hidden way, how the autoscaler could spawn new nodes with an updated OS (i.e. I guess around January there was a time where on the day before it still worked, but the day after it failed).
I would like to prevent this kind of implicit update to happen again in the future

@schlichtanders Just set both upgrade variables to false that way no more implicit upgrades.
Normally now, the new autoscaler nodes will get a correct cloudinit with the right selinux rules, so no more k3s permission denied, I would imagine.
Also, make sure to update the underlying snapshot images, please see the 2.x upgrade guide pinned in the discussions to refresh on how to do that.
Closing this for now.

Thank you for confirming that the autoscaler should not be able to spawn nodes with an updated OS as long as automatically_upgrade_os = false. This was my best guess.

I updated snapshot images :+1:  thank you for the pointer back then.
unfortunately it is not possible to fix k3s for the autoscaler nodes with automatically_upgrade_k3s = false, as it only prevents the update on an existing node. New nodes will still fetch the most recent k3s version from the channel.

So I still see myself in the same situation as before. Hopefully it is just bad memory, i.e. that I made the lethal change myself somehow. Fingers crossed that this is not happening again :crossed_fingers:

@schlichtanders  Like two months ago, there was a k3s denied issue linked to selinux that showed up after an automatic node upgrade, hence my bash script above. So I think you are now safe since you have those turned off haha, don't worry! 🤞


Maintenance

Hi @mpociot
Are you still happy to maintain this package?
Thanks!

See https://github.com/laravel-notification-channels/channels/issues/67


Added in controller bookshelfChildren

This will allow the use of trees  bookshelfChildren. with entities.

Thanks for offering this @Abe-Telo but I'm not really willing to accept and maintain added data, that's used in customizations, into the core project repo. Doing so is not really feasible for long-term application performance or maintenance.
Let me know though if I've misunderstood the idea behind this pull request.

Will therefore close this off


:rocket: 2022.1.0-beta.0

Preliminary Changelog

Feb 8, 2022
A big thanks to the 15 contributors who made this release possible. Here are some highlights ✨:
Additions and Improvements

Improved onboarding flow is now a bit simpler. (#4307) @dimitropoulos

Notable Fixes

Fixed a bug in gRPC requests where the "Send Request" hotkey was not triggering (#4126) @vincendep
Fixed a bug related to circular reference in OpenAPI specs (#4040) @MrSnix
Fixed a regression where spaces (not tabs) was being used to write YAML OpenAPI specs (#4315) @develohpanda
Fixed situation where an extra v could appear in the dashboard for a Design Document's version (#4319) @wdawson

Other Changes

Allow users to configure what is shown on environment variable template tag (#4277) @develohpanda
Upgraded to Electron 12 (#4232) @jackkav
autofocuses the URL bar when interacting with new requests or switching between requests (#4338) @dimitropoulos
[inso] Allows overriding annotations on Kubernetes Ingress objects (#4349) @ttyS0e
adds support for Apple Silicon (Apple M1) (#4140) @johnwchadwick
Show gRPC requests in the request switcher (#4127) @vincendep
[inso] show request logs in verbose mode (#4368) @jackkav
The url bar in the debug tab will now receive focus automatically (#4400) @dimitropoulos
removes migration for insomnia designer (#4404) @dimitropoulos

All contributors of this release in alphabetical order: @boflynn, @curiositycasualty, @develohpanda, @dimitropoulos, @DMarby, @falondarville, @filfreire, @gatzjames, @jackkav, @JasonCubic, @johnwchadwick, @MrSnix, @ttyS0e, @vincendep, @wdawson


Refactor write pipeline to support multiple formats and preprocessors

Current implementation did not support adding new data formats and data
preprocessors needed to support upcoming new features planned. This change
makes the ingest pipeline easier to work with.

bors r+

bors r+


program_exit hook triggered 2 times in some cases

Since 6a53d6766c7e9dc048217cd18fa5ea7c79e48a24, program_exit hook is triggered inside jv_kill_jarvis(), but in some case we can use jarvis -q while not started with jarvis -b

Jarvis is running inside a terminal, we use jarvis -q in another terminal
Jarvis is started thanks to jarvis-systemd plugin (so with -q option) and we close it with jarvis -q, service stop or service restart

And, when Jarvis is not started with -b option, SIGTERM signal send here is caught by jarvis.sh and program_exit hook is called another time

Indeed, if you refer to my previous comment on your PR, this is because I use a dirty workaround in -q because when jarvis is started with -b the TERM is not properly caught.
I have to intevestigate further and make this work, removing this workaround (and so your PR) will avoid this situation...

Ok I think I found a solution... for some reason the killing simply the parent process instead of the whole group works, and doesn't leave orpheans as it used to in the past.
Will push that to beta for you to test

Implémenté sur la branch beta. Pour le tester dès à présent:
Menu Settings > General > Branch > beta
Sinon attendre la mise à jour de ce weekend.

Nice, I test it soon

Tested with success, thank you


Support multiple sensors for 1804 branch

Same as pull request #79 but for merging to branch 1804. We need to update the Dockerfile before merge.

Need to re-open this pull request. Abandoning and closing.


Implement Painter's Partition Problem (Binary Search) using Java

🚀 Feature
(A clear and concise description of what the feature is.)
Have you read the Contributing Guidelines on Pull Requests?
(Yes.)
Motivation
This problem consists to paint n boards of given length A[i...n] and there will be k no of painters available and each painter takes B unit of time to paint a particular board. Our task is to find the minimum time to get this job done under the constraints that any painter will only paint continuous sections of boards, say board {2, 3, 4} or only board {1} or nothing but not board {2, 4, 5}.
This problem will be solved using the concept of binary search algorithm with Time Complexity O(NLogN) and space complexity O(1).
Pitch
This problem will be give the real time use of binary search algorithm in order to find the minimum time required to paint the given boards.

Greetings sir/Mam,
I would like to contribute the code for Bubble Sort Algorithm #933 in C++/CPP.
The code is ready. I have added the code , along with proper explanation of the algorithm , test cases , time complexity and space complexity of the same, as per the guidelines.
Kindly assign it to me.
This is regarding GSSOC'21.
Regards,
Hitain Kakkar


Update CRD's API version to v1 and keep old one as v1beta1.yaml

@s1061123 CI failed:
hack/../pkg is out of date. Please run hack/update-codegen.sh

Thank you for your comment, @phoracek !
Diff is updated.


[WebSocketLink] inactivityTimeout causes a reconnect

When a WebSocketLink is closed due to the inactivityTimeout running out, it's automatically reconnected due to autoReconnect = true. This defeats the purpose of an inactivityTimeout.

Can you use the TransportWebsocketLink instead?
The architecture of TransportWebsocketLink is generally better, and the protocol that transport WebSocketLink implements is deprecated.
I imagine there will probably not a lot of maintenance work flow into WebSocketLink (unless someone steps up  ;) )

Then please add the Deprecated annotation. I wasn't aware of that, and the documentation also doesn't say it.

Well, there are two different protocols for graphql over Websockets.
graphql-transport-ws ( https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md )
and
graphql-ws
https://github.com/apollographql/subscriptions-transport-ws/blob/master/PROTOCOL.md.
To make matters worse, the javascript library implementing the graphql-transport-ws is named graphql-ws ( https://www.npmjs.com/package/graphql-ws ) so there is a lot of potential for confusion.
The library that implemented the graphql-ws protocol was called subscriptions-transport-ws https://github.com/apollographql/subscriptions-transport-ws/tree/master
The graphql-ws protocol is deprecated. This is stated in the README.
But I would not completely deprecate the WebsocketLink class, there are still a lot of servers that only support this protocol.
However, if your server supports both, you should definitely use the graphql-transport-ws and TransportWebsocketLink class. Aside from the improved protocol, the dart implementation of the ``TransportWebsocketLink` is better.
If you have suggestions on how the documentation around this can be improved, feel free to open issues or PRs.

If it's broken and not going to be fixed, isn't that the definition of deprecation? If you don't want people to encounter warnings when using that class, maybe just include what you told me here in the documentation.
TransportWebsocketLink works fine for me, btw. It also doesn't have the bug that the WebSocket protocol name isn't set automatically on connection (and so the server rejects the connection before it's even established).

Well, I'm not going to fix it in the near future. But I'm also not the original contributor and I'm not using the Websocketlinks myself. If someone from the community is willing to fix it or take over maintenance, they are welcome of course.


[Enhancement] Update log4j to 2.17.1 (or newer)

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request
https://github.com/apache/incubator-eventmesh/blob/master/build.gradle#L412 -- shows that log4j 2.17.0 is used but this too has a CVE. https://logging.apache.org/log4j/2.x/security.html
Describe the solution you'd like
Update to v 2.17.1 or higher
Are you willing to submit PR?

[x] Yes I am willing to submit a PR!

closed by #736


Config specifications

Overview
In order to be able to pass in specifications for the underlying processing after stage 0, there needs to be a mechanism to do so. https://github.com/OSOceanAcoustics/echopype/pull/817 has the proposed subpackages, so below are the proposed specifications that can happen:
Spec
name: (str) Name of the full pipeline must be unique
sonar_model: (str) The sonar model, can only be the available ones in echopype
raw_regex: (str) The regular expression of the pattern for raw files
args: # input arguments for raw files
  urlpath: (str) urlpath to the input raw file... can be a jinja template where the values retrieved form parameters below
  parameters: {} # (dict) Set default parameter values as found in urlpath
  storage_options: {} # (dict) fsspec filesystem storage options for the source
  transect: # **Optional** field, if exists it indicates that converted files should be organized by transect (yyyy/transect)
    file: (str) urlpath to transect files
    storage_options: {} # (dict) fsspec filesystem storage options for the transect files
output:
  urlpath: (str) The urlpath to the output converted raw file
  storage_options: {} # (dict) fsspec filesystem storage options for the output where files will be stored
  overwrite: (bool) Flag to allow for overwriting or not

# Below are echopype specifications, if not provided will be using defaults
echopype:
  consolidate:
    add_location: {}
    add_splitbeam_angle: {}
  calibrate:
    compute_Sv:
      env_params: {}
      cal_params: {}
      waveform_mode: {}
    compute_TS: {}
  filter:
    median: {}
    conv: {}
    remove_noise: {}
    noise:
      estimate_noise: {}
      mean_bkg: {}
      spike: {}
  unify:
    compute_MVBS: {}
    compute_MVBS_index_binning: {}
    regrid_Sv: {}
  mask:
    from_Sv:
      freq_diff: {}
    from_labels:
      boundary: {}
      region: {}
  metrics:
    summary statistics: {}
    compute_NASC: {}

Closing this, since this is linked with [Separating Dataset and Pipeline configuration yaml files] (https://github.com/OSOceanAcoustics/echoflow/issues/11)


Add GPU Mining maybe?

HI!.
For solomining, could you add in the config File to Set GPU Mining for Nvidia graphic cards to true or false without needing CUDA?
Thanks!

I think it is possible to mine with Nvidia GPU in python.However,I don't understand what you mean that mining without cuda

I think it is possible to mine with Nvidia GPU in python.However,I don't understand what you mean that mining without cuda.Can you mention more about that?

Cuds is a driver, which sometimes is needed by NVIDIA for certain operations.

operations

I see.

I think it is possible to mine with Nvidia GPU in python.However,I don't understand what you mean that mining without cuda.Can you mention more about that?

Cuda is a driver, which sometimes is needed by NVIDIA graphic cards, for certain operations.

I see.

I think it is possible to mine with Nvidia GPU in python.However,I don't understand what you mean that mining without cuda.Can you mention more about that?

Cuda is a driver, which sometimes is needed by NVIDIA graphic cards, for certain operations.

I see.

https://developer.nvidia.com/cuda-toolkit


how to start archive mode?

I want to understand if it is possible to enable archive mode? If so, how to run cardano node in archive mode?

What is "archive mode"? As far as I am aware, there is no "archive mode".

Archive nodes are full nodes running with a special option known as "archive mode". Archive nodes have all the historical data of the blockchain since the genesis block.

The standard cardano-node has downloads and stores (unfortunately not in an easily form) historical data on the blockchain including genesis.


Question about prin function

Description
Is there a way to use prin without it using trim to remove spaces ?
For example, prin "   -----   ----" will output ---- ---- and remove the first and middle spaces.
Or should I use echo or prinf if I want to do that ?
Neofetch version
Neofetch 7.1.0
Screenshot

Config file
config.txt

Well I used on empty char other than space and it worked, so my problem is solved.


fix: widget refactor actions fetch change

Description
Widget refactor was not reflected in API pane
Fixes #7627
Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:red_circle: Total coverage has decreased
// Code coverage diff between base branch:release and head branch: fetch-actions-after-update-canvas

Status
File
% Stmts
% Branch
% Funcs
% Lines

:red_circle:
total
54.8 (0)
36.55 (-0.01)
33.93 (0)
55.35 (0)

:green_circle:
app/client/src/sagas/ActionSagas.ts
10.88 (0.03)
0.78 (0)
3.7 (0)
13.14 (0.04)

:green_circle:
app/client/src/sagas/JSActionSagas.ts
16.17 (0.1)
2.04 (0)
6.67 (0)
19.12 (0.14)

:green_circle:
app/client/src/sagas/PageSagas.tsx
23.95 (0.08)
5.95 (0)
14.29 (0)
25.78 (0.08)

:red_circle:
app/client/src/utils/autocomplete/TernServer.ts
50.47 (-0.24)
38.26 (-0.87)
36.21 (0)
55.08 (-0.27)

Actions are retrieved twice after refactor @ApekshaBhosale

Actions are retrieved twice post widget name refactor @ApekshaBhosale

those 2 are actions and collections/actions i.e js collections


May I ask if there is a way to visualize SDF in the paper if I want to visualize my own model, like the figure 4 in your paper

Thanks for your question. Unfortunately I'm not sure if I still have access to the code for producing Figure 4. But I guess it's possible to find good open-source tools or github repos for SDF visualization.


Add ability to set ETA

Please,
add ability to set message ETA.
Alternately add public method accepting *TaskMessage.
Thanks.

@sickyoon you want help with this?

I thought I already added ETA. @phamdt you're always welcome to create a PR.

You have the ETA property set but I believe the worker logic needs to be updated to respect the ETA by not running the job until the specified time. So there'd be a fair amount of work (which I'm happy to take on) to adjust the worker loop execution logic, probably port over some timestamp validation, and then some doc additions as well.

@phamdt cool as long as you don't introduce breaking changes.


about bezier GT

How can i get the bezier curve GT in custom dataset? will it be published?

@ccblublu If your custom dataset is already labeled by sample points, you can refer to this documentation: https://github.com/voldemortX/pytorch-auto-drive/blob/master/docs/CURVE.md
But you will need to get the loader working for your dataset. That will depend on how your custom dataset is organized.
Essentially, you need to fit Bézier curves for your GT lines and save them in the json format.

This issue seems to be addressed. I'll close it for now. Feel free to continue commenting for reopen/open a new one.


Explicitly disable CGO in build

Fleet does not use CGO. In some build situations, not explicitly
disabling CGO results in binaries that have linking errors when running
on an Alpine Docker container. This change should fix the build for
those containers.

I agree, though looks like that was attempted previously (https://github.com/kolide/fleet/blob/master/Makefile#L99) and didn't work. Unfortunately I don't have the bandwidth right now to find the cleanest way to do it so the explicit approach will be used here.

I would not expect setting variables in a rule to work like that. But, I assume,  APP_NAME does. Skimming the docs, I think that APP_NAME is a make variable, and eval is can effect make level things.
But, CGO_ENABLED and GOGC are shell level variables, and setting them in a line like that has no effect. (by default, each line of a rule is a new shell.
If it's meant to be global, the correct thing is to export it toplevel, like https://github.com/kolide/fleet/blob/master/Makefile#L3


Spelling correction

Change "ot" to "of"

@MikeBarker-MSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@MikeBarker-MSFT thanks for the fix! #sign-off

@JasonWHowell: I'm sorry - only the author of this article, @joannapea, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


Few warnings

gathering elements...
(node:15370) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Evaluation failed: DOMException: Failed to execute 'querySelector' on 'Document': '#21993399543' is not a valid selector.
    at selector (<anonymous>:2:30)
    at <anonymous>:7:5
(node:15370) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

It does look promising!

@sibelius Ah, I think I know what's going on - would it be possible for me to get the sketch file you were trying to convert?  Either here or in an email?

@chrisachard what's your email?

@sibelius<EMAIL_ADDRESS>

I had the same problem 👍

@pablostanley This should be fixed in master (although fixing it revealed some other bugs...) - did you have the error recently?  If so, could you post or send your sketch file?  Thanks!

Fixed in a75b0f1e5be7d3cab1d20eec64bd1510613e0f56


customisation of a scanner view

I kinda need only a scanner view (no cancel button and no switch to front camera button) to be displayed on a screen. So I added this method:
public func returnCameraView () -> UIView {
return cameraView;
}
I believe that adding only a clean scanner view to my view controller would be more preferable for the component.

Hi,
the cameraView is a composition of the codeReader.previewLayer and the overlay. If you want only have the camera view you can add the codeReader.previewLayer directly as sub layer of your view.


add performance instrumentation and perfutil_readme.md

Signed-off-by: Dongming Hwang<EMAIL_ADDRESS>

@dongmingh Thanks for submitting the pull request.
I would like to improve a bit on the following:

perfutil.go should go into /core/util/ directory
perfutil_readme.md should go into /docs/tech and rename to perf_trace.md
fix misspellings in perfutil_readme.md, include capitalizing Hyperledger Fabric and remove IBM
for env var and REST endpoint, it would be better to call it "perftrace" rather than "perfutil"
why would REST endpoint /perfutil has the body of a "chaincodeSpec" ?
better names for the "actions" and should be defined as constants
call to perfutil.PerfTraceHandler() should be guarded as it is being called multiple times in the critical path
we can enhance later to provide summary average stats
devops is not going to be in the main path when we complete the client SDK, so I would suggest the look into instrumenting core/peer.ProcessTransaction() where all transactions go through

This PR is using the incorrect license header. Use this one, please. Substitute "IBM 2016" for the "[COPYRIGHT OWNER]".
I'm also a bit unclear why we are developing code to handle profiling when there are so many libraries available that we might use.

If this PR is to be merged, it needs tests for perfutil.go at the very least. @bmos299 @binhn we should not be merging code without tests.

@christo4ferris
We will create a perftrace_test.go for test and evaluate Go-Metrics.
@binhn concerning your comments:
1.perfutil.go should go into /core/util/ directory
2.perfutil_readme.md should go into /docs/tech and rename to perf_trace.md
For above two comments, can we create a subdirectory, core/util/perftrace, to contain all related go files and readme,md (rename to README.md)? Then the user can read the README.md without clicking the file when they switch to that directory
3.fix misspellings in perfutil_readme.md, include capitalizing Hyperledger Fabric and remove IBM
We will take care of this.
4.for env var and REST endpoint, it would be better to call it "perftrace" rather than "perfutil"
We will rename perfutil to perftrace.
5.why would REST endpoint /perfutil has the body of a "chaincodeSpec" ?
We will change this not to use chaincodeSpec.
6.better names for the "actions" and should be defined as constants
We will rename action to reqType
7.call to perfutil.PerfTraceHandler() should be guarded as it is being called multiple times in the critical path
We will guard this API with lock()/unlock() and logic to prevent invalid uuid (nil or a different uuid from PerfUuid)
8.we can enhance later to provide summary average stats
We will add more functions, such as average and concurrence, to the perftrace later.
9.devops is not going to be in the main path when we complete the client SDK, so I would suggest the look into instrumenting core/peer.ProcessTransaction() where all transactions go through
We will move starting/ending tracepoints from devops to core/peer.ProcessTransaction().  Moreover, when tracepoints (neither starting nor ending) are deleted or added, the perftrace will calculate the elapsed time based on the new segments, i.e., there will be no harm to the perftrace.

@christo4ferris A number of folks chased various tools but none gave us the info we need, especially related to gRPC and breaking the transaction path into segments.

@dongmingh let's talk -- i will google hangout with you

@dongmingh please transfer this to Gerrit or close. Thanks!

close.


community: set litellm max_tokens=None

Description: The default value of the max_tokens for litellm is 256. It should be increased to avoid truncation of replies.
resovle conflicts for #10812

Closing as this was implemented separately in https://github.com/langchain-ai/langchain/pull/21534. Apologies as this PR was first. Thank you for the fix!