After reducing the master node value to one, communication with the cluster was lost

Description
Hi, everything was working fine until I decided to reduce the number of master nodes from 3 to 1, after that I see
E0825 09:58:36.124964 8655 memcache.go:265] couldn't get current server API group list: the server is currently unable to handle the request
I cannot return value 3 because, I can't connect to the cluster and my helm which is installed by a separate provider is installed like this
provider "helm" {
  kubernetes {
    host     = var.host

    client_certificate     = var.client_certificate_data
    client_key             = var.client_key_data
    cluster_ca_certificate = var.certificate_authority_data
  }
}

gives an error such as
Error: Kubernetes cluster unreachable: The server is currently unable to handle the request
with helm_release.external_dns
on helm.tf line 1, in resource "helm_release" "external_dns":
resource "helm_release" "external_dns" {

When I try to comment the definition of provider and terraform the code is related to helm i see
Terraform:
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I have a question. Can I fix this and is this the expected behavior?
Kube.tf file
module "kube-hetzner" {

  version = "2.5.2"

  providers = {
    hcloud = hcloud
  }
  extra_firewall_rules = [
    {
      description     = "Access to MySQL cluster"
      direction       = "out"
      protocol        = "tcp"
      port            = "3306"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["<IP_ADDRESS>/0", "::/0"]
    },
    {
      description     = "Access to"
      direction       = "in"
      protocol        = "tcp"
      port            = "6001"
      source_ips      = ["<IP_ADDRESS>/0", "::/0"] # Won't be used for this rule
      destination_ips = []
    },
    {
      description     = "Access to"
      direction       = "in"
      protocol        = "tcp"
      port            = "30001"
      source_ips      = ["<IP_ADDRESS>/0", "::/0"] # Won't be used for this rule
      destination_ips = []
    },
    {
      description     = "Access to"
      direction       = "out"
      protocol        = "tcp"
      port            = "6001"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["<IP_ADDRESS>/0", "::/0"]
    }
  ]
  hcloud_token         = var.hcloud_token
  source               = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key       = var.pub
  ssh_private_key      = var.priv
  network_region       = "eu-central"
  create_kubeconfig    = false
  create_kustomization = false
#
  use_control_plane_lb  = true
  lb_hostname           = "hostname"
  control_plane_lb_type = "lb11"
  enable_rancher        = true
  initial_k3s_channel   = "v1.25"
  rancher_hostname      = "hostname"

  cluster_name = "inb-k3s"

  automatically_upgrade_k3s = false
  automatically_upgrade_os  = false

  ingress_controller = "nginx"

  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cx31",
      location    = "fsn1",
      labels      = ["role=master"],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },

  ]

  agent_nodepools = [
    {
      name        = "agent-development",
      server_type = "cpx41",
      location    = "fsn1",
      labels      = ["environment=development", "role=worker"],
      taints      = [],
      count       = 2

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "agent-production",
      server_type = "cpx51",
      location    = "fsn1",
      labels      = ["environment=production", "role=worker"],
      taints      = [],
      count       = 0

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
#
    {
      name        = "agent-jenkins-main",
      server_type = "cx41",
      location    = "fsn1",
      labels      = ["environment=jenkins"],
      taints      = [],
      count       = 0

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "agent-redis-dev",
      server_type = "cx21",
      location    = "fsn1",
      labels      = ["environment=redis"],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

}

Screenshots
No response
Platform
Linux

@grig0701 First of all, try to change the IP in your kubeconfig to the direct IP of the remaining control plane node, forget about the control plane lb for a sec, see if that works.
Second, did you drain the two control plane before taking them down? This is super important and explained in the readme and/or kube.tf.example.

@grig0701 First of all, try to change the IP in your kubeconfig to the direct IP of the remaining control plane node, forget about the control plane lb for a sec, see if that works.
Second, did you drain the two control plane before taking them down? This is super important and explained in the readme and/or kube.tf.example.
Finally, please see the debug section in the readme, to SSH into the main CP node and see what is happening. Maybe restarting the k3s process could do the trick. You can also inspect the /etc/rancher/k3s/config.yaml file.

Hi, thanks for your reply. Changing IP doesn't help. As far as I understand, the external IP should be specified in the config.yaml for further access via this IP, but it is not there.
Unfortunately, I missed the drain step before reducing the number of nodes to one.
systemctl restart k3s does not help
Logs:
Aug 28 08:00:00 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:00.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 is starting a new election at term 2"}
Aug 28 08:00:00 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:00.294Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 became pre-candidate at term 2"}
Aug 28 08:00:00 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:00.295Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 received MsgPreVoteResp from 6862d5ef6fbce322 at term 2"}
Aug 28 08:00:00 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:00.295Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 [logterm: 2, index: 23103299] sent MsgPreVote request to 706ca1e01eedd961 at term 2"}
Aug 28 08:00:00 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:00.295Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 [logterm: 2, index: 23103299] sent MsgPreVote request to cc3bf8125b7e74ec at term 2"}
Aug 28 08:00:03 inb-k3s-control-plane-arp k3s[1249]: time="2023-08-28T08:00:03Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://<IP_ADDRESS>:6443/v1-k3s/readyz: 500 Internal Server Error"
Aug 28 08:00:03 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:03.878Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"cc3bf8125b7e74ec","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:03 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:03.878Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"706ca1e01eedd961","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:03 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:03.879Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"706ca1e01eedd961","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:03 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:03.879Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"cc3bf8125b7e74ec","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:05.305Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 is starting a new election at term 2"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:05.306Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 became pre-candidate at term 2"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:05.306Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 received MsgPreVoteResp from 6862d5ef6fbce322 at term 2"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:05.306Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 [logterm: 2, index: 23103299] sent MsgPreVote request to 706ca1e01eedd961 at term 2"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:05.306Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6862d5ef6fbce322 [logterm: 2, index: 23103299] sent MsgPreVote request to cc3bf8125b7e74ec at term 2"}
Aug 28 08:00:05 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:05.843Z","caller":"etcdserver/server.go:2065","msg":"failed to publish local member to cluster through raft","local-member-id":"6862d5ef6fbce322","local-member-attributes":"{Name:inb-k3s-control-plane-arp-05efcb79 ClientURLs:[https://<IP_ADDRESS>:2379]}","request-path":"/0/members/6862d5ef6fbce322/attributes","publish-timeout":"15s","error":"etcdserver: request timed out"}
Aug 28 08:00:06 inb-k3s-control-plane-arp k3s[1249]: time="2023-08-28T08:00:06Z" level=info msg="Waiting for etcd server to become available"
Aug 28 08:00:07 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:07.776Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000872c40/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""}
Aug 28 08:00:07 inb-k3s-control-plane-arp k3s[1249]: {"level":"info","ts":"2023-08-28T08:00:07.776Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:08.548Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000872c40/<IP_ADDRESS>:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: time="2023-08-28T08:00:08Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: time="2023-08-28T08:00:08Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://<IP_ADDRESS>:6443/v1-k3s/readyz: 500 Internal Server Error"
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:08.879Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"706ca1e01eedd961","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:08.879Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"cc3bf8125b7e74ec","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:08.880Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"cc3bf8125b7e74ec","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}
Aug 28 08:00:08 inb-k3s-control-plane-arp k3s[1249]: {"level":"warn","ts":"2023-08-28T08:00:08.880Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"706ca1e01eedd961","rtt":"0s","error":"dial tcp <IP_ADDRESS>:2380: i/o timeout"}

I have a question. Is the lack of quorum the reason for this behavior?
Will Drain prevent this?

@grig0701 Indeed this was caused because of the non draining. If you had implemented etcd backup, you may be able to restore. Please find below the analysis of the issue by GPT-4.

@grig0701 Indeed this was caused because of the non draining. If you had implemented etcd backup, you may be able to restore. Please find below the analysis of the issue by GPT-4.
Dissecting the Components:
Raft Consensus Algorithm
The Raft consensus algorithm is central to how etcd manages distributed data. When scaling down without draining, one disrupts the quorum-based decision-making model, effectively paralyzing the data store. In your scenario of going from 3 to 1 master node, you've violated the mathematical basis of Raft.
Load Balancer
Load balancers add another layer of complexity. Even if one correctly drains nodes, the load balancer might still keep routing requests to non-existent or non-operational nodes unless it's reconfigured.
Helm Dependencies
Helm’s dependency on the Kubernetes API server is a well-known bottleneck. Helm stores release information in ConfigMaps or Secrets within Kubernetes itself. Hence, API server availability becomes crucial for any Helm-based operations.
Possible Mitigations:
1. Rolling Back State
Reverting to the original 3-node setup is an immediate mitigation strategy but does nothing to address the underlying ignorance (or mismanagement) of cluster-state expectations.
2. Draining and Graceful Exit
Future-proofing could include better use of kubectl drain before scaling down, which not only cordons the node but also ensures that pods are gracefully removed. This would prevent such a drastic violation of Raft consensus.
3. Kubeconfig and Direct IP Access
The idea to modify the kubeconfig file to connect directly to the remaining master node's IP is sound for immediate troubleshooting. However, it's a band-aid solution and doesn't replace a proper load balancer reconfiguration.
4. Debugging Tools
Low-level debugging using commands like etcdctl can be performed to check cluster health (etcdctl cluster-health) or even manually manipulate the etcd data if required.
5. Logs and Configuration Files
Logs located in /etc/rancher/k3s/config.yaml can offer clues on whether any configuration-level changes may have inadvertently triggered this cascade of failures.
6. Helm Reconfiguration
It's paramount to reconfigure the Helm provider once the Kubernetes cluster is operational. A helm repo update might be needed to ensure that Helm Charts are correctly synced with the restored API server.
Is this expected behavior?
In the framework of a system designed for fault tolerance and high availability, this behavior is not entirely unexpected, assuming that you've not adhered to standard procedures like draining nodes. Kubernetes expects you to play by certain operational rules, and failure to do so will cause the system to behave in ways that may seem chaotic or counterintuitive but are entirely logical within the bounds of its architecture.

Thanks for the answer. It was important for me to understand how I can prevent this in the future, the problem was solved by rebuilding the cluster, because it was a dev environment


Profile Updation

Hey @Saurav-Pant, I think you didn't commit the changes. Please let us know if you need help to do that. Find me on Twitter

#118  plz check this maybe I commit here.

Duplicate of #118


Graph RAG recipe

WIP Recipe for Graph RAG application

Please sign your commits.
git commit -a --amend -s
git push --force

@srampal is this still a WIP?

@srampal is this still a WIP?

Yes @rhatdan .. still a WIP


Set location and then get listings for that Latitude and Longitude

Hey!
Thanks for the really awesome package. I've a minor doubt regarding the usage, we want to set the location and then get listings for that particular location. As far as I understand, the package sends user_id to fetch the listings presently. Could you please suggest a way which I can get listings for any given set of Lat-Long, instead of sending user_id which will only get listings for the current location.
Thanks a ton!

Hey!
At the moment the package doesn't support searching for listings in a given location.
I've done some research and it seems that there at least two way to do that with a mobile app:

Search queries (e.g. "San Francisco, CA, United States"). This would require a call to their location name autocomplete endpoint. This is performed when you type in the search box on the top of the iOS app.
Search within a coordinate rectangle (defined by SW/NE lat/lng). This is performed when you switch to the map mode.

Which one is more useful?

Hi nderkach, +1 for coordinates 👍


Can you share  board of loss changing?

TKS

That will require me to rerun the experiment, which will take some time, but good suggestion anyway.

In my experiment，the  d_loss decreased very fast, and when d_loss less than 1(ofen in 10 epochs), the quality of images generated would not improve.
I reinitialize the d_vars at 15 epoch, now it seems better, however, the images generated is still much worse than your results.


py/tools: Namespace repo packages to prevent conflict

This allows repo tooling to work externally alongside our namepsaced pypi packages

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

this requires an upstream fix https://github.com/aspect-build/bazel-lib/pull/547
it also has a really dirty hack (repeated) - its v similar to the ones we used to have to patch sys.path - in fact that hack may work and be cleaner - this was was v much a distressed hack
pulling bits of here as they are resolved (eg #29786) - downstream target is https://github.com/envoyproxy/envoy-website/pull/336

@htuch this is what is left to make the external docs/protodoc build work
essentially the problem boils down to the fact that we have pip deps that are namespaced envoy.* and rules_python is creating non namespaced-packages for tool/* which externally it sees as envoy/tools/*
not sure how best to resolve - not as current in this Pr - at least without some cleanup - will try and follow up one way or anotherr - for now the website is carrying the patch from this pr

this should work now and is a i think a much more elegant/reliable/bazelish solution
essentially it injects namespaced init.py into all the places that were making it impossible to use tooling externally
i may have missed a couple but this will at least allow external building of docs, and use of most of the linting tools


ReadSubtree

I'm missing a method that exists in XmlReader called ReadSubtree: https://msdn.microsoft.com/en-us/library/system.xml.xmlreader.readsubtree(v=vs.110).aspx
This method is very handy when parsing large documents and handing over parts of the document to other methods. Here's an example:
        public static FeedFolderResponse DeserializeFromJson(string jsonContent)
        {
            FeedFolderResponse response = new FeedFolderResponse();

            using (StringReader textReader = new StringReader(jsonContent))
            using (JsonTextReader jsonReader = new JsonTextReader(textReader))
            {
                while (jsonReader.Read())
                {
                    if (jsonReader.TokenType == JsonToken.PropertyName && (string)jsonReader.Value == "folders")
                    {
                        // JsonTextReader has no ReadSubtree method, so ensure exiting DeserializeFoldersNode
                        // when reaching the end of the folders part.
                        DeserializeFoldersNode(response, jsonReader);
                    }
                    else if (jsonReader.TokenType == JsonToken.PropertyName && (string)jsonReader.Value == "feeds")
                    {
                        DeserializeFeedsNode(response, jsonReader);
                    }
                }  // while jsonReader.Read()
            }  // using JsonTextReader

            return response;
        }  // DeserializeFromJson

Instead of passing the jsonReader object over to DeserializeFoldersNode and DeserializeFeedsNode, it would be much safer to pass a new JsonReader containing the part of the document that belongs to the "folders" and "feeds" nodes and have the outer JsonReader skip those parts by calling its Skip() method. This way, you can be sure that the deserialization done in the inner methods doesn't affect the outer reader.

I make considerable use of ReadSubTree with XmlReader myself, and have similarly noticed the lack of a comparable ReadObject to give you a reader scoped on an object or array. I think I shall give this a look.


Incorrect autocorrect for Rails/RedundantActiveRecordAllMethod in 2.21.0

When using rubocop autocorrect, incorrect syntax is created.
Original code:
@labels = policy_scope(Label.all).merge(apply_scopes(Label).all)

Autocorrected code:
@labels = policy_scope(Label.all)merge(apply_scopes(Label).)

Expected result:
@labels = policy_scope(Label.all).merge(apply_scopes(Label))

Expected behavior
Rubocop removes redundant all method
Actual behavior
Rubocop removes incorrect . in line, resulting in incorrect code
Steps to reproduce the problem
Run rubocop -A on this commit https://github.com/dodona-edu/dodona/pull/4949/commits/c273a4892eef5e550328ec18b2669a76aa394789
RuboCop version
1.56.3 (using Parser <IP_ADDRESS>, rubocop-ast 1.29.0, running on ruby 3.1.2) [x86_64-linux]
  - rubocop-capybara 2.18.0
  - rubocop-factory_bot 2.23.1
  - rubocop-minitest 0.31.1
  - rubocop-rails 2.21.0

The check marking the all statement as redundant in this case is also incorrect
@labels = policy_scope(Label.all).merge(apply_scopes(Label))

results in the error ArgumentError: Label(id: integer, name: string, color: integer) is not an ActiveRecord::Relation
Thus the all was not even redundant here, as removing the all results in errors

I had the same problem with the extra period being removed.

Thus the all was not even redundant here, as removing the all results in errors

Note that is described in the lint rules and why it's an unsafe autocorrect.


Pword

Current coverage is 42.97%

Merging #214 into master will decrease coverage by -1.50% as of 2e5a0b4

@@            master    #214   diff @@
======================================
  Files           21      21
  Stmts          353     370    +17
  Branches         0       0
  Methods          0       0
======================================
+ Hit            157     159     +2
  Partial          0       0
- Missed         196     211    +15

Review entire Coverage Diff as of 2e5a0b4

Powered by Codecov. Updated on successful CI builds.


Don't auto-preload AGAIN during nested renders

Fixing an issue with auto and nested renders. When auto was on, nested associations were RE-preloading their subtrees. In many cases this was only a minor issue. But in certain situations (lots of manual preloads + field blocks that were really associations), it was a huge net loss of performance.
This is fixed by stricter matching in the extension. Preloading now only occurs if the rendered object's class is exactly ActiveRecord::Relation. Subclasses like ActiveRecord::Associations::CollectionProxy will no longer match, ensuring we don't preload on nested associations.
Since this is a significant issue, I've included a version bump so we can release asap. Also enabled appraisal in GH Actions.

At @njbbaer's suggestion I attempted to use instance_of? instead of comparing strings. Sadly, this doesn't work as ActiveRecord is doing some magic:
q = Project.all
q.class.name
=> "ActiveRecord::Relation"

q.class
=> "Project::ActiveRecord_Relation"


Change HCX version to 3.5.3

[Please change HCX version to 3.5.3]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ecae1fa9-e6f3-ecf3-fc19-afea5f16fe54
Version Independent ID: 14ffd8ac-9c84-7653-89cf-d01a915a4a2e
Content: Azure VMware Solution platform updates - Azure VMware Solution
Content Source: articles/azure-vmware/azure-vmware-solution-platform-updates.md
Service: azure-vmware
GitHub Login: @shortpatti
Microsoft Alias: v-patsho

Update: Change HCX version from 3.5.2 to R146

@AnejaAmit
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@shortpatti
Can you please check and add your comments on this doc update request as applicable.

Document updated
#please-close


Add MSBuild 15.5 branch.

@AndyGerlicher,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


Drop package Romanitho.WiGUI

Package WiGUI isnt supported anymore and not being worked on. The new package (which makes this one obsolete) has been submitted to winget: https://github.com/microsoft/winget-pkgs/pull/182735
Checklist for Pull Requests

[X] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[X] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Not sure what to do. This PR just removes the package (deleted the folder). There's nothing that could "fail"?

Automatic Validation ended with:

The pull request contains more than one manifest. Either singleton or multi file manifest under one version directory.

Unfortunately, the rule of 1 manifest change per PR applies to removals as well. Could either of these be moved to another PR? If so, then both should be able to move past this check.


[BUG] Volumes are not properly mounted/unmounted when kubelet restarts

Describe the bug
Pods within a statefulset get stuck in Terminating or Pending state after kubelet goes unready back to ready. Volume never becomes unattached for some reason.
To Reproduce
Steps to reproduce the behavior:
in order to reproduce we took a multi-node cluster with a stateful set (in this case prometheus from cattle-monitoring), stop the rke2-server process that is on the node where the pod is, wait for the node to go not ready, kubectl delete the pod, then start rke2-server and wait patiently
This by way of @oats87
Expected behavior
Pods should recover from a kubelet restart/crash
Log
I have logs, please ping me directly as they might contain sensitive information.
Environment:

Longhorn version: v1.1.1
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: rke2

Number of management node in the cluster: 1 node cluster with all roles

Node config

OS type and version: rhel 8
CPU per node: 16
Memory per node: 64
Disk type(e.g. SSD/NVMe): ssd
Network bandwidth between the nodes: NA

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Azure

The error message is no relationship found between node <xxx> and this object.
@joshimoo and @Oats87 are aware of the issue.
Related the k3s issue: https://github.com/rancher/rke2/issues/914

The error message is no relationship found between node <xxx> and this object.
@joshimoo and @Oats87 are aware of the issue.
Related the k3s issue: https://github.com/rancher/rke2/issues/914

@Oats87 is now debugging the issue. It might not be a Longhorn issue in the end. But we've seen that error message multiple times, including:
https://github.com/longhorn/longhorn/issues/2533
https://github.com/longhorn/longhorn/issues/2276
https://github.com/harvester/harvester/issues/543
None of the issues above have been resolved and all pointed to Longhorn. If in the end, it's not a Longhorn issue, we should produce a KB article to address it.

So more details on this. It appears that the attachdetach-controller and the longhorn-attacher are both doing their jobs, although I'm not sure why the force detach is happening.
Logs from the attachdetach-controller i.e. kube-controller-manager show that
2021-05-27T08:36:42.95441388-07:00 stderr F W0527 15:36:42.947984       1 reconciler.go:222] attacherDetacher.DetachVolume started for volume "pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5") on node "ck-c7-3" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching
2021-05-27T08:36:45.13540826-07:00 stderr F I0527 15:36:45.134288       1 operation_generator.go:470] DetachVolume.Detach succeeded for volume "pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5") on node "ck-c7-3"

Logs from the longhorn-attacher show the corresponding detach:
Warning  FailedMount             8s                   kubelet                  MountVolume.WaitForAttach failed for volume "pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5" : volume pvc-524a19af-0e75-4cff-b461-f9c7a4fedfd5 has GET error for volume attachment csi-204c3043c9b00f9435af375819f256bad1e8a0f7eb89ad448c42e5463d2fc00c: volumeattachments.storage.k8s.io "csi-204c3043c9b00f9435af375819f256bad1e8a0f7eb89ad448c42e5463d2fc00c" not found

So, summary of the above logs is that the volume is getting forced detached from the node due to the maxWaitForUnmountDuration, but I'm not sure why the termination doesn't just continue and detach gracefully when the kubelet starts back up -- instead the kubelet is having the carpet pulled out from under it and then goes into that error state reporting the no relationship found.
According to the design docs I've found online for the attachdetach-controller (https://github.com/kubernetes/kubernetes/issues/20262), the volumeattachment being deleted should indicate that the volume is not mounted, and the kubelet should be able to take that input and understand that the pod can be terminated.
More investigation is necessary here.

Just leaving some additional references here for a dangling attachment issue on the kubernetes side.
https://github.com/kubernetes/kubernetes/issues/80488
https://github.com/kubernetes/kubernetes/issues/80488#issuecomment-783621304
https://docs.google.com/document/d/1fEpM1s8qEU6pZvDh9NMsLug606NB5b2qP2GPdX3O_F8/edit#heading=h.b7dx73o7lamg

Solving this issue might help us to understand the issue https://github.com/longhorn/longhorn/issues/2178

i have the same issue once or twice a week, everytime when k3s panics because the etcd took to look to write to disk. (only 3 low budget vms).
The statefulset volume gets detached and the pod is stucked in:
Warning  FailedMount  2m41s (x531 over 18h)  kubelet  MountVolume.WaitForAttach failed for volume "pvc-fd5f5eeb-b6d1-499a-becb-134d760fb604" : volume pvc-fd5f5eeb-b6d1-499a-becb-134d760fb604 has GET error for volume attachment csi-eb34f320c64e8dd056dfd2dd1ab561d59f0cd8da616c61769686a031b531fe82: volumeattachments.storage.k8s.io "csi-eb34f320c64e8dd056dfd2dd1ab561d59f0cd8da616c61769686a031b531fe82" is forbidden: User "system:node:host1.host.xy" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node 'host1.host.xy' and this object

Is there a workaround to set the volume free/detached?

For those looking for a workaround, a reboot of the node should be able to help finish termination, as a reboot will "refresh" all of the mount points on the system and the kubelet should be able to send an Unmount request to the csi volume mounter plugin and finish unmount. I understand that a reboot is not a good workaround, but until we think of a better workaround for this, I believe this is the most effective way to help restore functionality.
Update on my investigation into this:
It appears that the restart/termination of the longhorn-manager engine pod is causing the kubelet to get stuck in a strange state, where it is continuously attempting to operationExecution.MountVolume the volume back into place for the pod. I believe this is essentially "expected" behavior as the desired state is attempting to be reconciled against the actual state. What is interesting, is that the termination of the consumer pod by longhorn seems to only exasperate the issue/call it to light, if this functionality is disabled as per the Longhorn settings, a manual termination of the pod will still cause the problem.
When I set my kubelet to debug mode (-v=6) and observed the logs of the kubelet, I can see that once I restart the kubelet, or even without restarting the kubelet perform a kubectl delete -n longhorn-system instance-manager-e-<hash> on the node that I am targeting, I can see the kubelet enter a frenzy attempting to MountVolume.Setup/Starting operationExecutor.MountVolume the volume. My current assumption is that the kubelet entering this eternal loop is what is causing it from never sending the Unmount call to the longhorn csi plugin.
The specific section of kubelet code that is reporting these logs is here: https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/kubelet/volumemanager/reconciler/reconciler.go#L254-L266
Tracing through this code, I would expect that an error be returned and a operationExecutor.MountVolume failed error be logged; however, I don't see these failed messages logged.
Note: the following logs are slightly condensed i.e. non-relevant log lines removed for readability.
In the kubelet logs we can see that when housekeeping comes around, it finds the pod and says that the mount point is still found on disk (which is more or less expected):
I0602 14:39:50.612680   31911 kubelet.go:1999] SyncLoop (housekeeping)
I0602 14:39:50.614211   31911 kubelet_volumes.go:70] pod "4e55325b-11cf-4e9b-988f-fc0fc93ace7a" found, but volumes are still mounted on disk [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/mount]
I0602 14:39:50.614263   31911 kubelet_pods.go:971] Pod "prometheus-rancher-monitoring-prometheus-0_cattle-monitoring-system(4e55325b-11cf-4e9b-988f-fc0fc93ace7a)" is terminated, but some volumes have not been cleaned up
I0602 14:39:50.639808   31911 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:39:50.650802   31911 desired_state_of_world_populator.go:174] Skipping findAndRemoveDeletedPods(). Not permitted until 2021-06-02 14:39:51.23982552 +0000 UTC m=+298.380649752 (getPodStatusRetryDuration 2s).
I0602 14:39:50.740484   31911 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:39:50.751448   31911 desired_state_of_world_populator.go:174] Skipping findAndRemoveDeletedPods(). Not permitted until 2021-06-02 14:39:51.23982552 +0000 UTC m=+298.380649752 (getPodStatusRetryDuration 2s).
I0602 14:39:50.780555   31911 kubelet_volumes.go:70] pod "4e55325b-11cf-4e9b-988f-fc0fc93ace7a" found, but volumes are still mounted on disk [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/mount]
I0602 14:39:50.780638   31911 kubelet_pods.go:1956] Orphaned pod "4e55325b-11cf-4e9b-988f-fc0fc93ace7a" found, but volumes not yet removed.  Reducing cpu to minimum

Inspecting the operation_executor we can see that there is really not much logging going on here: https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/volume/util/operationexecutor/operation_executor.go#L803-L841 but tracing through I can see that it the downstream functions are using a RWmutex to allow for parallelization.
If I inspect a fresh kubelet startup, I can see logs similar to:
I0602 14:33:06.220457   23712 reconciler.go:211] Starting operationExecutor.VerifyControllerAttachedVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.322014   23712 reconciler.go:211] Starting operationExecutor.VerifyControllerAttachedVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.414032   23712 operation_generator.go:1357] Controller attach succeeded for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a") device path: ""
I0602 14:33:06.423152   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.423298   23712 reconciler.go:269] operationExecutor.MountVolume started for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.423534   23712 csi_mounter.go:89] kubernetes.io/csi: mounter.GetPath generated [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/mount]
I0602 14:33:06.423586   23712 csi_plugin.go:444] kubernetes.io/csi: created path successfully [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362]
I0602 14:33:06.423629   23712 csi_util.go:61] kubernetes.io/csi: saving volume data file [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/vol_data.json]
I0602 14:33:06.423829   23712 csi_util.go:70] kubernetes.io/csi: volume data file saved successfully [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/vol_data.json]
I0602 14:33:06.424200   23712 csi_plugin.go:480] kubernetes.io/csi: mounter created successfully
I0602 14:33:06.424354   23712 operation_generator.go:556] MountVolume.WaitForAttach entering for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a") DevicePath ""
I0602 14:33:06.424387   23712 csi_attacher.go:146] kubernetes.io/csi: probing for updates from CSI driver for [attachment.ID=csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52]
I0602 14:33:06.424411   23712 csi_attacher.go:156] kubernetes.io/csi: probing VolumeAttachment [id=csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52]
I0602 14:33:06.512289   23712 plugins.go:433] Calling network plugin cni to tear down pod "instance-manager-r-470b430a_longhorn-system"
I0602 14:33:06.515534   23712 cni.go:382] Deleting longhorn-system_instance-manager-r-470b430a/61dcc564fc30f567ea2411933534dae0b8f18f6737a77efef60b71f2817aed3f from network loopback/cni-loopback netns "/proc/25394/ns/net"
I0602 14:33:06.524308   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.552704   23712 cni.go:390] Deleted longhorn-system_instance-manager-r-470b430a/61dcc564fc30f567ea2411933534dae0b8f18f6737a77efef60b71f2817aed3f from network loopback/cni-loopback
I0602 14:33:06.552899   23712 cni.go:382] Deleting longhorn-system_instance-manager-r-470b430a/61dcc564fc30f567ea2411933534dae0b8f18f6737a77efef60b71f2817aed3f from network calico/k8s-pod-network netns "/proc/25394/ns/net"
I0602 14:33:06.572488   23712 request.go:591] Throttling request took 352.497206ms, request: GET:https://<IP_ADDRESS>:6443/api/v1/namespaces/cattle-monitoring-system/pods/prometheus-rancher-monitoring-prometheus-0
I0602 14:33:06.579209   23712 round_trippers.go:445] GET https://<IP_ADDRESS>:6443/api/v1/namespaces/cattle-monitoring-system/pods/prometheus-rancher-monitoring-prometheus-0 200 OK in 6 milliseconds
I0602 14:33:06.585253   23712 status_manager.go:564] Patch status for pod "prometheus-rancher-monitoring-prometheus-0_cattle-monitoring-system(4e55325b-11cf-4e9b-988f-fc0fc93ace7a)" with "{\"metadata\":{\"uid\":\"4e55325b-11cf-4e9b-988f-fc0fc93ace7a\"}}"
I0602 14:33:06.585300   23712 status_manager.go:570] Status for pod "prometheus-rancher-monitoring-prometheus-0_cattle-monitoring-system(4e55325b-11cf-4e9b-988f-fc0fc93ace7a)" is up-to-date: (1)
I0602 14:33:06.585319   23712 status_manager.go:529] Status Manager: syncPod in syncbatch. pod UID: "9e66a98d-3299-4198-90db-8fa78c47f0b2"
I0602 14:33:06.598156   23712 kuberuntime_container.go:641] Container "docker://0a3e3dff5d3d80988a83aff1ba50f8cf1ee93aeac6f817fbab1ca85865b84870" exited normally
I0602 14:33:06.599886   23712 plugins.go:433] Calling network plugin cni to tear down pod "instance-manager-e-b79c1bac_longhorn-system"
I0602 14:33:06.602484   23712 cni.go:382] Deleting longhorn-system_instance-manager-e-b79c1bac/63bf44ea4f24d669ac4cf36dfa59d5a058c60d32f579a772465c5bd8cfdb57c3 from network loopback/cni-loopback netns "/proc/25309/ns/net"
I0602 14:33:06.611035   23712 cni.go:390] Deleted longhorn-system_instance-manager-e-b79c1bac/63bf44ea4f24d669ac4cf36dfa59d5a058c60d32f579a772465c5bd8cfdb57c3 from network loopback/cni-loopback
I0602 14:33:06.611352   23712 cni.go:382] Deleting longhorn-system_instance-manager-e-b79c1bac/63bf44ea4f24d669ac4cf36dfa59d5a058c60d32f579a772465c5bd8cfdb57c3 from network calico/k8s-pod-network netns "/proc/25309/ns/net"
I0602 14:33:06.624956   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.632418   23712 shared_informer.go:270] caches populated
I0602 14:33:06.725812   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:06.733055   23712 shared_informer.go:270] caches populated
I0602 14:33:06.773690   23712 request.go:591] Throttling request took 349.17894ms, request: GET:https://<IP_ADDRESS>:6443/apis/storage.k8s.io/v1/volumeattachments/csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52
I0602 14:33:06.780271   23712 round_trippers.go:445] GET https://<IP_ADDRESS>:6443/apis/storage.k8s.io/v1/volumeattachments/csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52 200 OK in 6 milliseconds
I0602 14:33:06.780588   23712 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a") DevicePath "csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52"
I0602 14:33:06.783218   23712 csi_attacher.go:226] kubernetes.io/csi: attacher.GetDeviceMountPath(&{nil &PersistentVolume{ObjectMeta:{pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362    b3ec7da2-9e76-4254-aea0-6efb0e5ba65c 20939 0 2021-06-01 19:24:32 +0000 UTC <nil> <nil> map[] map[pv.kubernetes.io/provisioned-by:driver.longhorn.io] [] [kubernetes.io/pv-protection external-attacher/driver-longhorn-io]  [{csi-provisioner Update v1 2021-06-01 19:24:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/provisioned-by":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:claimRef":{".":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:csi":{".":{},"f:driver":{},"f:fsType":{},"f:volumeAttributes":{".":{},"f:baseImage":{},"f:fromBackup":{},"f:numberOfReplicas":{},"f:staleReplicaTimeout":{},"f:storage.kubernetes.io/csiProvisionerIdentity":{}},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}}} {kube-controller-manager Update v1 2021-06-01 19:24:33 +0000 UTC FieldsV1 {"f:status":{"f:phase":{}}}} {csi-attacher Update v1 2021-06-01 19:24:38 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:\"external-attacher/driver-longhorn-io\"":{}}}}}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{10737418240 0} {<nil>} 10Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:driver.longhorn.io,VolumeHandle:pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{baseImage: ,fromBackup: ,numberOfReplicas: 3,staleReplicaTimeout: 30,storage.kubernetes.io/csiProvisionerIdentity:<PHONE_NUMBER>956-8081-driver.longhorn.io,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:cattle-monitoring-system,Name:prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0,UID:6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362,APIVersion:v1,ResourceVersion:20828,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:longhorn,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Bound,Message:,Reason:,},} false false})
I0602 14:33:06.783253   23712 csi_attacher.go:231] attacher.GetDeviceMountPath succeeded, deviceMountPath: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/globalmount
I0602 14:33:06.783282   23712 csi_attacher.go:236] kubernetes.io/csi: attacher.MountDevice(csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52, /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/globalmount)
I0602 14:33:06.783344   23712 csi_client.go:467] kubernetes.io/csi: calling NodeGetCapabilities rpc to determine if NodeSupportsStageUnstage
I0602 14:33:06.783361   23712 csi_client.go:513] kubernetes.io/csi: creating new gRPC connection for [unix:///var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock]
I0602 14:33:06.783402   23712 clientconn.go:106] parsed scheme: ""
I0602 14:33:06.783417   23712 clientconn.go:106] scheme "" not registered, fallback to default scheme
I0602 14:33:06.783453   23712 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
I0602 14:33:06.783483   23712 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0602 14:33:06.783650   23712 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0602 14:33:06.784524   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00004dbe0, {CONNECTING <nil>}
I0602 14:33:06.786722   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00004dbe0, {READY <nil>}
I0602 14:33:06.813803   23712 shared_informer.go:270] caches populated
I0602 14:33:06.813943   23712 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0602 14:33:06.829477   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")

What is interesting for me to note is that:
I0602 14:33:06.813943   23712 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

From that line I am inferring that the kubelet is unable to talk to the csi plugin due to "transport is closing"
Further on, I can see the mount attachment successful:
I0602 14:33:07.132350   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:07.172175   23712 request.go:591] Throttling request took 358.186659ms, request: GET:https://<IP_ADDRESS>:6443/apis/storage.k8s.io/v1/volumeattachments/csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52
W0602 14:33:07.203242   23712 docker_sandbox.go:402] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "instance-manager-e-b79c1bac_longhorn-system": unexpected command output Device "eth0" does not exist.
 with error: exit status 1
I0602 14:33:07.203339   23712 round_trippers.go:445] GET https://<IP_ADDRESS>:6443/apis/storage.k8s.io/v1/volumeattachments/csi-1322745dbf1eabd3bf8b2c0148909e9f0f47c46c6af8fbba90235eb843414c52 200 OK in 30 milliseconds
I0602 14:33:07.203574   23712 csi_attacher.go:293] kubernetes.io/csi: created target path successfully [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/globalmount]
I0602 14:33:07.203593   23712 csi_util.go:61] kubernetes.io/csi: saving volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/vol_data.json]
I0602 14:33:07.203711   23712 csi_util.go:70] kubernetes.io/csi: volume data file saved successfully [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/vol_data.json]
I0602 14:33:07.203857   23712 csi_attacher.go:320] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
I0602 14:33:07.203898   23712 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/globalmount"

yet, we can see yet another subsequent mount attempt fail:
I0602 14:33:07.534837   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")
I0602 14:33:07.572057   23712 request.go:591] Throttling request took 368.058563ms, request: GET:https://<IP_ADDRESS>:6443/api/v1/namespaces/cattle-monitoring-system/persistentvolumeclaims/prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0
I0602 14:33:07.576865   23712 round_trippers.go:445] GET https://<IP_ADDRESS>:6443/api/v1/namespaces/cattle-monitoring-system/persistentvolumeclaims/prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0 200 OK in 4 milliseconds
I0602 14:33:07.577133   23712 csi_mounter.go:89] kubernetes.io/csi: mounter.GetPath generated [/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/mount]
I0602 14:33:07.577183   23712 csi_mounter.go:110] kubernetes.io/csi: Mounter.SetUpAt(/var/lib/kubelet/pods/4e55325b-11cf-4e9b-988f-fc0fc93ace7a/volumes/kubernetes.io~csi/pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362/mount)
I0602 14:33:07.577221   23712 csi_client.go:467] kubernetes.io/csi: calling NodeGetCapabilities rpc to determine if NodeSupportsStageUnstage
I0602 14:33:07.577239   23712 csi_client.go:513] kubernetes.io/csi: creating new gRPC connection for [unix:///var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock]
I0602 14:33:07.577287   23712 clientconn.go:106] parsed scheme: ""
I0602 14:33:07.577305   23712 clientconn.go:106] scheme "" not registered, fallback to default scheme
I0602 14:33:07.577338   23712 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
I0602 14:33:07.577351   23712 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0602 14:33:07.577403   23712 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0602 14:33:07.577669   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00257c790, {CONNECTING <nil>}
I0602 14:33:07.579460   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00257c790, {READY <nil>}
I0602 14:33:07.580134   23712 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0602 14:33:07.635403   23712 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-6f5fbfc6-d0e9-4f3d-ab8b-8884ae5d3362") pod "prometheus-rancher-monitoring-prometheus-0" (UID: "4e55325b-11cf-4e9b-988f-fc0fc93ace7a")

Indeed, if we inspect the source code here we can find that loopyWriter.run in the grpc-go source code: https://github.com/grpc/grpc-go/blob/v1.27.1/internal/transport/controlbuf.go#L508
So I suppose a question is posed here: why is the transport closing when the kubelet should really be talking to the volume plugin?
If I look at the kubelet at startup when it registers csi plugins, it looks like:
I0602 14:32:57.582509   23712 plugin_watcher.go:52] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
I0602 14:32:57.582536   23712 plugin_watcher.go:101] Ensuring Plugin directory at /var/lib/kubelet/plugins_registry
I0602 14:32:57.582706   23712 plugin_watcher.go:160] Handling create event: "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock": CREATE
I0602 14:32:57.582794   23712 plugin_watcher.go:203] Adding socket path or updating timestamp /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock to desired state cache
I0602 14:32:57.582817   23712 plugin_manager.go:112] The desired_state_of_world populator (plugin watcher) starts
I0602 14:32:57.582826   23712 plugin_manager.go:114] Starting Kubelet Plugin Manager
I0602 14:32:57.584950   23712 reconciler.go:147] Starting operationExecutor.RegisterPlugin for plugin at "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock" (plugin details: &{/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock 2021-06-02 14:32:57.582809639 +0000 UTC m=+9.774620244 <nil> })
I0602 14:32:57.585061   23712 reconciler.go:156] operationExecutor.RegisterPlugin started for plugin at "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock" (plugin details: &{/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock 2021-06-02 14:32:57.582809639 +0000 UTC m=+9.774620244 <nil> })
I0602 14:32:57.585148   23712 operation_generator.go:181] parsed scheme: ""
I0602 14:32:57.585166   23712 operation_generator.go:181] scheme "" not registered, fallback to default scheme
I0602 14:32:57.585197   23712 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock  <nil> 0 <nil>}] <nil> <nil>}
I0602 14:32:57.585213   23712 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0602 14:32:57.585812   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000c29b50, {CONNECTING <nil>}
I0602 14:32:57.604755   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000c29b50, {READY <nil>}
I0602 14:32:57.615415   23712 round_trippers.go:445] POST https://<IP_ADDRESS>:6443/api/v1/namespaces/default/events 201 Created in 34 milliseconds
I0602 14:32:57.617244   23712 csi_plugin.go:101] kubernetes.io/csi: Trying to validate a new CSI Driver with name: driver.longhorn.io endpoint: /var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock versions: 1.0.0
I0602 14:32:57.617355   23712 csi_plugin.go:114] kubernetes.io/csi: Register new plugin with name: driver.longhorn.io at endpoint: /var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock
I0602 14:32:57.617420   23712 csi_client.go:159] kubernetes.io/csi: calling NodeGetInfo rpc
I0602 14:32:57.617459   23712 csi_client.go:513] kubernetes.io/csi: creating new gRPC connection for [unix:///var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock]
I0602 14:32:57.617495   23712 clientconn.go:106] parsed scheme: ""
I0602 14:32:57.617510   23712 clientconn.go:106] scheme "" not registered, fallback to default scheme
I0602 14:32:57.617612   23712 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/obsoleted-longhorn-plugins/driver.longhorn.io/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
I0602 14:32:57.617643   23712 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0602 14:32:57.617779   23712 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
I0602 14:32:57.617818   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc002c30810, {CONNECTING <nil>}
I0602 14:32:57.626036   23712 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc002c30810, {READY <nil>

A question here is why is the name "obsoleted-longhorn-plugins" used? obsolete implies no longer necessary/used/out of date, perhaps @shuo-wu you may know why that term is used here and if it is still necessary? I see the issue that created it here: https://github.com/longhorn/longhorn/issues/899#issuecomment-555200366

Investigated this issue with @joshimoo in a live debug session.
We're finding that the kubelet is in fact getting stuck in a MountVolume.SetUp loop, and is never able to finish this because it is unable to set ownership on the directory. This can be seen here: https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/volume/csi/csi_mounter.go#L271
Interestingly, the Longhorn csi plugin is receiving the NodePublish calls, but is returning essentially a "volume already mounted" as it is not checking the health of the mount, but rather just the fact that the mount point exists. This can be seen here: https://github.com/longhorn/longhorn-manager/blob/0f63e8fae7dc979b72d1529033d5669d22264d56/csi/node_server.go#L205
The csi_client.go attempts to call NodePublish and based on the error result of that, returns different types of errors. https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/volume/csi/csi_client.go#L255-L258 If the error is a final error, it will remove the mount path in question, which will free up the node and have the node attempt to either call NodePublish again (thus remounting the volume) or NodeUnpublish which would be ideal in our case.
Current thought on our end is potentially to be returning an error from the node_server.go if the mount point exists but is unhealthy, which will trigger the removal of the mount point, although I'm not actually sure the kubelet will be able to remove the folder considering it is a mount point, but at least the client will return an final error rather than a transient error.

Just leaving a quick update here:
@Oats87 and I continued looking into the Issue and we think that a change on the CSI driver would help us exit the error loop mentioned above. By unmouting the corrupt mount point during the NodePublishCall, then the following NodePublishCall will mount it again, we can combine both of these actions into 1 NodePublishCall for existing corrupt mounts.
Example commit: https://github.com/longhorn/longhorn-manager/commit/62eef5b7d7f931ef218e8183cb1056d72cee87cc
REF: https://github.com/kubernetes/kubernetes/issues/70013
With this PR kubernetes introduced the responsibility/ability for the csi driver to deal with corrupt mounts.
https://github.com/kubernetes/kubernetes/pull/88569
Here is an example PR for the azure driver that implements that cleanup routine mentioned above.
https://github.com/kubernetes-sigs/azuredisk-csi-driver/pull/308
Other drivers (intree) behave similarly.
This might leave some open file handles stale, since the block device major:minor can change between attachments.
But together with the https://github.com/longhorn/longhorn/issues/2650 lifecycle managment changes for the instance-managers this should be even less of a problem, since the engine (volume) should continue being available to the workload pods, therefore the mount wouldn't even become corrupt (i.e. since the block device never gets attached)

cc @khushboo-rancher

I believe I've determined why the volume is never Unmounted from the node.
When the kubelet starts up (and once it's running it continuously does this) and is building its desired state of the world/actual state of the world, it performs a syncState where it checks the pod directory for volumes. It will come across the mount for the Longhorn volume on disk, and proceed to add it to the desired state of the world marked as "InUse": https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/kubelet/volumemanager/reconciler/reconciler.go#L422 and https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/kubelet/volumemanager/reconciler/reconciler.go#L441
Now, because the MountVolume.Setup never actually succeeds for this volume (because we return a false positive "success" here: https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/volume/csi/csi_mounter.go#L271), the volume is never added to the actual state of the world, and thus, when the subsequent evaluation for Unmount the volume comes along, it doesn't get processed as the volume isn't actually in the actual state of the world: https://github.com/kubernetes/kubernetes/blob/v1.20.7/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go#L286-L292

IIRC, the socket in dir obsoleted-longhorn-plugins is for the old version CSI plugin compatibility. Since the old CSI plugin is already removed, and based on the latest investigation, this is not related to this bug.

One way I was reproducing this was:
Using Rancher v2.5.8, register my cluster and install cattle-monitoring v2 into it, with prometheus using persistent storage (longhorn class) of any size.
Allow things to come up/stabilize. On the node where the prometheus pod is running, restart the kubelet, and observe the pod goes into terminating state forever.
/cc @khushboo-rancher

@Oats87 Thanks for the workaround. I can confirm that rebooting the node does fix the issue

Verified with Longhorn master-head and v1.1.2-head 06/10/2021
Validation - Fail
I followed the reproduction steps and was able to repro the problem, however I don't the see it got fixed with Longhorn master.
Steps:

Deployed Longhorn v1.1.1 on a v1.20.7 rke1 cluster.
Deployed Monitoring.
Restarted the kubelet on a worker node where the pod prometheus-cluster-monitoring was attached. At this point,  I was able to reproduce the problem.
Rebooted the node to fix the problem.
Upgraded the Longhorn to v1.1.2-head.
Restarted the kubelet, the pod stuck in terminating state. (Rebooted the node to fix it)
Upgraded the Longhorn to master-head.
Restarted the kubelet, the pod stuck in terminating state.

longhorn-support-bundle_080f08d0-55da-4758-b013-e46a4b0cb14d_2021-06-10T22-15-43Z.zip
Volume name - pvc-2cf096a6-a0e4-4e65-99c3-097da9a024d4
Workload name - prometheus-cluster-monitoring
Time stamp - 2021-06-10 22:03:00 UTC

cc @joshimoo

@khushboo-rancher can you test this image: joshimoo/longhorn-manager:ui-path-v1
I broke the broke the corrupt mount point detection. I will have another look tomorrow.

Working with joshimoo/longhorn-manager:ui-path-v1 image on a RKE1 multi node cluster. The pods are getting restarted after termination in case of kubelet restart.

@khushboo-rancher Please help close if this is verified by the latest build. Thanks.

@khushboo-rancher can be tested on master once https://github.com/longhorn/longhorn-manager/pull/945 is merged
The fix is already in v1.1.2

@khushboo-rancher https://github.com/longhorn/longhorn-manager/pull/945 has been merged, so the fixes are now also available to test on master.

Validated with v1.1.2-rc1 and Longhorn-master - 07/28/2021
Validation - Pass
The tests from https://longhorn.github.io/longhorn-tests/manual/pre-release/node/kubelet-restart-on-a-node passed.
Validated the scenarios related to kubelet restart, node down and instance manager crash for RWO and RWX volumes on RKE1 and RKE2 cluster.

ref: https://github.com/kubernetes/kubernetes/pull/110670


[Enhancement] Voice Blender Merge Ratio

The description of the merge ratio is very unclear atm. and it sounds more like "pull it to the right to get more of the right model and to the left to get more of the left model".
I tested a bit and found that the merge ratio itself just sets the weight of the first model, so a ratio of 0.8 gives you 80% of the first / left model and 20% of the second / right model.
Should be changed to clear up misunderstandings.

feel free to open a pr


How to draw the triangles in the visualization?

Hi,
I am confusing this operation because it is not the transformation from  image coordination to camera coordination in the line

Hi @rockywind, I draw the triangle directly in the camera coordinates space using the focal length f extracted from the projection matrix. There is no need of any transformation from 2D to 3D.


Don't mask errors when downstream parsing fails

If the client of this library mistakenly gets a NPE when parsing a status
update the exception is hidden in the logging. That makes debugging near
impossible.
So this is a 2-part fix.

Log the actual exception instead the localizedErrorMessage.
Split apart the message parsing from the message handling. If a parse
error happens we'll close the connection. If the handler fails, just ignore
it and report the error.

This should have targeted the real master


Native Widgets

📜 Instructions Compliance

[ ] If a native widget has a visible label or instructions, they must be programmatically determinable - WCAG 1.3.1, WCAG 2.5.3

Path
.navbar-item:nth-child(2) > .theme-switch-button[aria-label="light/dark"][data-bs-original-title="light/dark"]

Snippet
<button class="btn btn-sm navbar-btn theme-switch-button" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip" data-bs-original-title="light/dark">

Element
button

Accessible Name
light/dark

Accessible description
(no value)

This is the theme (light/dark) switcher on the navbar. This was the test:
Note: Both WCAG 2.0 and 2.1 require a widget's visible label and instructions (if present) to be programmatically determinable. WCAG 2.1 also requires a widget's visible label and instructions (if present) to be included in its accessible name and description.

In the target page, examine each highlighted element to determine whether it has a visible label or instructions.
If a widget does have a visible label or instructions, verify that they are also displayed in the Instances list:
- The accessible name must be (or include) an exact match of any visible text label.
- The accessible description must include any additional visible instructions. If any non-text instructions are provided (for example, icons or color changes), the accessible description must include a text equivalent.

I think the switcher failed this test because there aren't any instructions in the Accessible Description.

@gabalafou to look into this and the research the WCAG requirement

Looking into this, I think the only thing we really need to do with the light/dark button is to convey the state of the button in a more accessible way. The button has three states: light, dark, and auto. Right now we only convey the state of the button with one of three SVG icons. I think the first step would be to give those icons some alt text.
And then later when we've fixed higher priority issues, we could give the whole light/dark widget a rethink. What do we think of this plan?


Charts

In the Stats window, it might be useful to have charts of various data points.

Stats Dock Mockup

Needs mockup/redesign to more appropriately handle data
General or Overview tab with summary info, similar to current stats dock view
Tabs or other system for viewing more detailed information for individual outputs (Ex. graphs)
Future-proofing for multi-output

I like cg2121 idea of a system monitor.
Due to the number of graphs it could be either a separate window or a full height dock.
I agree with Rodney's comment that tabs within tabs feel awkward. Unless the stats dock is docked on its own as a full size one ?

Warchamp stressed that one should future proof what's done so that other outputs can be added to graphs.
Also he suggested tabs were one option among others.
So it's in flux.
I hope i have not strayed too much from what @Warchamp7 said. Feel free to add your thoughts in here.


Improving gradle-home-cache-cleanup detection of unused files

Hi! 👋
Our Gradle home cache is getting so big in our project that is taking long times loading and storing the cache, so we enabled the gradle-home-cache-cleanup flag to remove unused files from Gradle User Home before saving to the cache.
After enabling this flag, the home cache entry dropped from 3.6 GB to 2.3 GB. While this is an improvement, we expected more. If we remove all the caches in Github actions and we start a fresh new runner without any cache, it only generates 457 MB. Therefore it is still keeping 80% of data that was actually not used in the last build. Please note that I only refer to the Gradle User Home entry.
After doing some debugging in a sample project (by enabling logging and exploring the sectionGradle User Home (directories >5M): after extracting common artifacts) I observed that it keeps multiple transforms-x folders, which represent a big part of the contents stored in the home cache. In theory a certain version of Gradle should use a consistent X value for that folder, so most likely the cleanup is not properly removing these unused folders left behind after a Gradle bump.
This is an Android project, mostly Kotlin code. The cache was never cleaned up in the last 4 months. It started with Gradle 8.4, later we migrated to 8.5 and last week to 8.6.
I also observed the home increasing significantly when bumping some dependencies (please note that I do not refer to the dependencies entry, which is stored separately, but still talking about the home), but I didn't research this thoroughly. Most likely this could be the next step to explore after figuring out why the transforms folder is still there.
You can check the thread in the slack community for a few other details.
Thanks!

Thanks for the report @rolgalan . I've done some investigation, and it seems that Gradle does not cleanup the unused transforms-3 directory when running builds with Gradle 8.6.
This will need to be fixed in Gradle before we can fix the behaviour in this action.
I've raised https://github.com/gradle/gradle/issues/28178 for this issue.

FYI this has been fixed in Gradle and once v8.8 is released we can incorporate the fix into this action.


Unmark flaky on streaming_pump

#22619 makes this benchmark non-flaky as far as we know, so let's bring it back into the test cycle once that merges.

nit: you also need to regenerate the projects

@jtattermusch I also rebased it after merging #22619. Sure, I won't merge it until we're sure it's not flaky.

Confirmed no longer flaky.


Wtf bots

Is it possible to replace bots by original Barbosik's bots ?
They were way smarter in split

@duwayy Just replace the BotPlayer files.

I already tried it but when adding bots in console it give a fatal error and server then crash

Can you provide said error?

Sure:
>addbot 25
Added 25 player bots
>[FATAL] TypeError: Object #<Cell> has no method 'getSize'
    at PlayerTracker.BotPlayer.decide (/var/www/html/agario/MultiOgar-Edited-master/src/ai/BotPlayer.js:84:27)
    at PlayerTracker.BotPlayer.sendUpdate (/var/www/html/agario/MultiOgar-Edited-master/src/ai/BotPlayer.js:58:10)
    at GameServer.updateClients (/var/www/html/agario/MultiOgar-Edited-master/src/GameServer.js:436:39)
    at GameServer.mainLoop (/var/www/html/agario/MultiOgar-Edited-master/src/GameServer.js:616:10)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

@duwayy use this one instead: https://github.com/Megabyte918/MultiOgar-Edited/blob/6bb6eef1d5c32c48e3c99b361e4ef361737d4eb5/src/ai/BotPlayer.js

Thank you !
Why don't you put back the old one ? In the newest I think bots split too much :)

@duwayy Lol I want to popsplit a bot. Then at the perfect moment. I double split. The bot splits all into me.

@Tyler3D If it's a joke, I'm afraid I forgot to laugh.
Not relating to popsplits, bots when they are <300 mass full splitting for savaging tiny cells does not look very "natural", you know.

@duwayy the original bots are dumb, they rarely split

@AlexandriteErikParacelsus they split.. new ones split too much for anything (in my opinion)

@duwayy "they rarely split"


Merging in Dev to Master for 6.3.3 release

@peterbae,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Codecov Report

Merging #507 into master will increase coverage by 0.14%.
The diff coverage is 74.74%.

@@             Coverage Diff              @@
##             master     #507      +/-   ##
============================================
+ Coverage      46.3%   46.44%   +0.14%
- Complexity     2209     2212       +3
============================================
  Files           108      108
  Lines         25231    25312      +81
  Branches       4173     4181       +8
============================================
+ Hits          11682    11755      +73
+ Misses        11537    11530       -7
- Partials       2012     2027      +15

Flag
Coverage Δ
Complexity Δ

#JDBC41
46.21% <74.74%> (+0.13%)
2203 <38> (+6)
:arrow_up:

#JDBC42
46.34% <74.74%> (+0.2%)
2211 <38> (+10)
:arrow_up:

Impacted Files
Coverage Δ
Complexity Δ

...om/microsoft/sqlserver/jdbc/SQLServerResource.java
100% <ø> (ø)
4 <0> (ø)
:arrow_down:

...osoft/sqlserver/jdbc/SQLServerSecurityUtility.java
50.74% <ø> (ø)
10 <0> (ø)
:arrow_down:

...a/com/microsoft/sqlserver/jdbc/SQLJdbcVersion.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

src/main/java/microsoft/sql/Types.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...va/com/microsoft/sqlserver/jdbc/StreamTabName.java
85% <ø> (ø)
7 <0> (ø)
:arrow_down:

...rc/main/java/com/microsoft/sqlserver/jdbc/DDC.java
44.14% <ø> (-0.58%)
103 <0> (-2)

.../microsoft/sqlserver/jdbc/SQLServerDataSource.java
45.98% <0%> (-1.01%)
66 <0> (ø)

...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
52.32% <0%> (-0.2%)
238 <0> (-1)

src/main/java/microsoft/sql/DateTimeOffset.java
41.42% <0%> (ø)
11 <1> (ø)
:arrow_down:

...n/java/com/microsoft/sqlserver/jdbc/Parameter.java
61.58% <100%> (-0.86%)
63 <2> (ø)

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9cdef4...33ff0e2. Read the comment docs.


DynamicBatteryStorage to Spacedock kref

Ref #5659

It looks like it's not getting the version from AVC since it already has one from spacedock.  Is there any way to force it to use the AVC version?


Compliance with SYCL 2020

Hi Ladies and Gents!
I am currently evaluating getting a VE Type 10C as an experimental card for testing part of my SYCL codebase. To do so I would be using neoSYCL.
A significant part of my codebase is SYCL2020 based and I'm wanting to verify how much, if not all of neoSYCL is compliant with 2020 vs 1.2.1.
From the paper it seems as if neoSYCL is built on SYCL1.2.1, is that correct? Specifically since the paper sites the 2019, OpenCL 1.2.1 revision of the specification, this is my assumption.
Thanks,
Felix

Hi @FCLC, your assumption is correct. neoSYCL implements the SYCL 1.2.1 specs. We currently do not have plans to support SYCL 2020.

Hi @FCLC, your assumption is correct. neoSYCL implements the SYCL 1.2.1 specs. We currently do not have plans to support SYCL 2020.

Hi @keichi,
Thank you for the prompt reply.
It may be worth adding this to the Readme.
Being as neoSYCL is downstream/forked from DPC++ (or seems to be so per the Kronos specification webpage) is their no possibility of changing this longer term?
(I'm not trying to add changes to the roadmap, more so trying to understand)
Thanks,
Felix

Thanks for your advice, we will clarify the supported SYCL version in our README.
In the long-term we might support SYCL 2020, but currently do not have resources.


Quantization, Pruning and distillation

@liuch37 Hi there,
Currently, i am interested in further improving the efficiency of text detection inferencing for cpu-only,
Methods such as: changing the backbone, quantization, pruning and distilling, are important for deployment
inorder to increase the fps, and lower the memory usage, while remaining accurate.
Therefore, are you interested in freelancing,
my email is<EMAIL_ADDRESS>Waiting for your reply.

May I know what is your goal for your test detection project? For example, commercialize a product, own interest, academia research...etc?

to include within the product perhaps, though i am currently only a single person.

That sounds interesting. Since I am quite busy personally, I could do some consultant/prototype for you - with some random delay expected.

i am interested in hiring you to deliver a complete code that is accurate, efficient and ready to be deployed.
select backbone > Quantization > train using gpu > pruning > finetune > export to onnx or pytorch lightning etc..> deploy on final devices Raspberry Pi to inference using cpu-only

So what do you think?


 This fixes that.

Did't see that. Thanks.


Syncing when not on site

Hi regarding Dexie-CLOUD
I have a question about how/when the syncing happens...
My use case.. .I want to have a public table... when a user first comes to the site of course he has to download it completely
When i change the table while he is somewhere else, when he comes back to the site does the sync still have to happen or have service workers already synced everything?
Cheers

Hi! Dexie Cloud client will have a service worker that subscribes to the periodicsync event. Service worker will be a separate delivareble in the dist folder of dexie-cloud-addon and can be included alone or combined with your other service worker code.
So in practice, your app will normally be up-to-date already from opening it.

Great thanks a lot for the info!
Happy with the answer...
LOOKING FORWARD to this..
Also holding off syncing in a different project that I have to deliver soon
hoping to use dexie as opposed to custom work :)
[image: photo] Bastiaan Oelrich
Human Content
Phone: 06-55 145 854 or +31 20 76 000 20  <+31+20+76+000+20>
Email: @.***  @.>
Website: http://www.humancontent.nl http://www.humancontent.nl/
Skype: @. https://mail.google.com/mail/u/0/#
Address: Lindengracht 9, Amsterdam
On Thu, Apr 15, 2021 at 2:31 PM David Fahlander @.***>
wrote:

Hi! Dexie Cloud client will have a service worker that subscribes to the
periodicsync event. Service worker will be a separate delivareble in the
dist folder of dexie-cloud-addon and can be included alone or combined with
your other service worker code.
So in practice, your app will normally be up-to-date already from opening
it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dfahlander/Dexie.js/issues/1274#issuecomment-820385634,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAG2Q3KX65BJQD2TFE7MGOTTI3MDTANCNFSM427HGPZA
.

I'm glad to hear! We're doing our best to get the beta released as soon as possible but it seems more likely to get it out in May than in April. I will give an update of the work going on in a blog post soon.


Uncaught TypeError: cyclic object value

I get the following error:
Uncaught TypeError: cyclic object value
    sendMessage http://<my-id>.disqus.com/embed.js:44
    getInstanceOrLoad http://<my-id>.disqus.com/embed.js:45
    e http://<my-id>.disqus.com/embed.js:44
    g http://<my-id>.disqus.com/embed.js:44
    m http://<my-id>.disqus.com/embed.js:44
    trigger http://<my-id>.disqus.com/embed.js:44
    init http://<my-id>.disqus.com/embed.js:44
    m http://<my-id>.disqus.com/embed.js:44
    trigger http://<my-id>.disqus.com/embed.js:44
    ia http://<my-id>.disqus.com/embed.js:44

when clicking on this button:

Afterward, the cursor can't click or mark anything.
I am directly importing vue-disqus into the component, like so:
import { Disqus } from "vue-disqus";
export default {
  name: "Part",
  components: {
    Disqus,
  },
...

Edit: Also happened when importing this component, globally.

@michidk
Did you manage to solve it?

@ktquez No, I ended up not using Disqus

I think it is a problem in the disqus script.


Allow Extending CircularBar to override the onDraw method.

Awesome library, would just make a few minor adjustments :) Apologize for the formatter, as well as the *.iml files getting modified, I don't usually track those in VC anyway.
Overview: Version code to 3, Gradle to 2.14.1, updating support lib/compiles to 25.0.0, allowing onDraw to be overridden to customize the way the circular bar is being drawn by setting the needed variables and methods to protected.

Cool thanks @CCorrado


Revamp salesforce integration scenarios in ballerina-guids/integration samples

The salesforce integration samples in ballerina-guides/integration-samples has to be revamped to shorten and simplify the code.
https://github.com/ballerina-guides/integration-samples

This is done. hence closing


chore: use github action

Coverage remained the same at 81.529% when pulling 45d88837a6af7dda3678ee1c47e46f0c3ed198d0 on ci into facf4fc4c34f4ae5ab3fdf1b96b659bd0ccbb1e1 on master.


CI: Building against JDK6 - 8

Extends the CI builds to following JDK's:

OpenJDK 6
OpenJDK 7
Oracle JDK 7
Oracle JDK 8

Cool beans, thanks!


added note about multiple waf policies with different CRS Engines

Added a note about the effect on the WAF engine when multiple policies are added with different Engine versions

@bram-boer Thanks for raising this; however, I confirmed with the WAF team that the information is not correct. If you configure different CRS versions at different scopes, the WAF uses the version you specify at each scope.
#please-close


[feature]: add ngpu train scripts

add n gpu train scripts.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Update master to main

Folks, just a heads up that we'll be swapping the primary branch from master to main. GitHub's tooling will remap any open PRs. If you're working locally and all your code has been merged, you can either reset your origin (GitHub will provide an instruction) or just do a fresh clone.
cc/ @leebyron

@brianwarner Done ✅


add support for api gateway

Add support for yaml based provider schemas
Adding the Java SDK for AWS Api Gateway

Description

Fixes # (issue)
Checklist

[ ] I have added tests that prove my fix is effective or that my feature works

[ ] I have updated the CHANGELOG-PENDING file with my change

[ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

Looks good! There's a bit more boilerplaty locations to mention the new provider in:
anton@Antons-MacBook-Pro> git grep kubernetes | grep -v '^providers' | grep -v '^pkg/codegen' | grep -v '^tests' | grep -v '^maven' | grep -v '^sdk'                                                           ~/pulumi-java
.github/workflows/cleanup-prerelease-builds.yml:        - kubernetes
.github/workflows/pr.yml:      - kubernetes/v*.*.*
.github/workflows/pr.yml:        - kubernetes
.github/workflows/release-providers-to-maven-central.yml:      - kubernetes/v*.*.*
.github/workflows/release-providers-to-maven-central.yml:        - kubernetes

I've added it already:
https://github.com/pulumi/pulumi-java/pull/579

Mine has slightly better error handling and fixes missed spots after bumping provider versions.
It also builds the provider in the second stage, since it depends on aws.

In the future, please look at the issue if it's assigned to anyone before duplicating work: https://github.com/pulumi/pulumi-java/issues/564


Variant Annotation - GET  VariantAnnotationSet

GET /variantannotationsets/{id} endpoint needed

I've added the label "Variant Annotation" for issues referring to the variation_annotation branch.

Closed with #846


Issue running the script. Session is set.

PS H:\GitHub\ChatGPT-Content-Generator> python content_gen.py
Would you like to provide a topic for the video? (y/n) y
What topic would you like to write about? Motivation
Would you like to provide a tone for the video? (y/n) y
What tone would you like to write about? empowering, not giving up, pushing hard
Would you like to specify the length of the video? (y/n) y
How long should the video be? (in seconds) 15
Topic: Motivation
Tone: empowering, not giving up, pushing hard
Length: 15 seconds
Traceback (most recent call last):
File "H:\GitHub\ChatGPT-Content-Generator\content_gen.py", line 41, in
video_script = get_video_script(session_token, video_topic, video_tone, video_length) # Get the video script from the AI.
File "H:\GitHub\ChatGPT-Content-Generator\chat_gpt_interface.py", line 34, in get_video_script
chatbot = Chatbot(config, conversation_id=None)
File "C:\Users\patri\AppData\Local\Programs\Python\Python310\lib\site-packages\revChatGPT\revChatGPT.py", line 34, in init
self.refresh_session()
File "C:\Users\patri\AppData\Local\Programs\Python\Python310\lib\site-packages\revChatGPT\revChatGPT.py", line 214, in refresh_session
raise Exception("Wrong response code")
Exception: Wrong response code

I'm looking into this. Will get back to you shortly


bottle: fix old_value/value ordering for --merge.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

Previously this was the wrong way around causing it to not be accepting enough on e.g. a bump from :any to :any_no_relocation.

@zmwangx This should fix the automake failure.

Lovely!


가스료를 줄이기 위한 함수 제거

ERC721Enumerable에서 total supply 빼고 다 지웠습니다. 가스비 차이 많이 납니다. ;)

Doesn't make a lot of sense, this removes the whole enumerability functionality...

Total supply is used in opensea, etherscan, etc., but it is better to erase others if it is not clear where else is used. Leave it if you need it. In my project, I've deleted every function in enumerable contract except total supply.

If you have to use this function at the expense of gas, you can include whole IERC721Enumerable/ERC721Enumerable files. But I don't recommend it due to high cost gas fee.

ERC721A is a good ref. for it.
https://www.erc721a.org/

ERC721Enumerable definitely adds a lot of gas. But it gives the ability for other smart contracts to query for a certain address's tokens. And also the ability to do so easily from the frontend.
Since they added it specifically I assume they added it on purpose cause they want that functionality, but yeah, maybe they don't need it.
There are anyway problems in this implementation though. For one, it says:
    function supportsInterface(bytes4 interfaceId) public view virtual override(IERC165, ERC721Initializable) returns (bool) {
        return interfaceId == type(IERC721Enumerable).interfaceId || super.supportsInterface(interfaceId);
    }

Which means it reports that it supports IERC721Enumerable - although it doesn't.
Additionally, it says:
abstract contract ERC721Enumerable is ERC721Initializable, IERC721Enumerable

But since it does not implement IERC721Enumerable functions, it will not compile.

Didn't you erase a meaningful function to save on gas costs?

@syl-doo2 Oh I didn't know Alchemy / Infura provide enum functions with their NFT API. I guess this could be really useful :)
The problem is we support multi-chains, including BSC and Klaytn - and they're not supported by Alchemy / Infura.. orz
I guess I'll have to check how much the additional gas cost is and if it's worth implementing an external indices by ourselves.
I will check for ERC721A as well. Thanks for your feedbacks!

I will close this issue in favor of #21
Thanks again for your feedback @syl-doo2 👍🏻


Add hex to client API

Signed-off-by: kamilsa<EMAIL_ADDRESS>

Description of the Change
All messages in client API containing bytes were changed to the hex.
Benefits
More consistent API for clients
Possible Drawbacks
None

May you please confirm that no fixes required for  .jenkinsci, deploy and docs folders? Ping @bakhtin

There are actually several locations that require fixes.

./deploy/ansible/roles/iroha-k8s/files/genesis.block should be replaced with example/genesis.block
./deploy/ansible/playbooks/iroha-k8s/scripts/genesis-add-peers.py should be refactored to support hex keys format. You may want to coordinate with @bagf as they created a PR with some improvements for deploying Iroha on k8s cluster (https://github.com/hyperledger/iroha/pull/1875). This PR also introduces changes in ./deploy/ansible/playbooks/iroha-k8s/scripts/genesis-add-peers.py.


Add constructor with a culture to RouteValueProvider

Addresses #5336

Hi @jburgard, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla2.dotnetfoundation.org.

TTYL, DNFBOT;

Added a constructor with culture info.  I wasn't sure if you wanted to add CultureInfo to the existing constructor.  With the new constructor, I didn't need to modify several test classes such as DefaultModelBindingContextTest, RouteValueProviderTest and TestValueProvider.
Do you prefer a single constructor with the addition of CultureInfo, which includes updating the associated tests and factory?

@jburgard, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.
Thanks, DNFBOT;

Apologies for this sitting so long, this was already addressed as part of 366dbde378259444485178a23e59708f62479fd5


OpenSearch Dashboards - pdf not downloading behind a gateway

I have a gateway, where I am using an internal and external URL (differs from each other). I am assuming that OpenSearch Dashboards are checking if the URL's exist internally as well as externally. In my case the internal URL will exist internally but will fail if accessed externally, the same with the external URL. This is causing that the pdf's cannot be downloaded behind the gateway. Any advice?

duplicate of https://github.com/opensearch-project/dashboards-reports/issues/374