Submariner Gateways Not Establishing Connections Between Clusters

I'm experiencing an issue where the Submariner gateways between my clusters are not establishing connections as expected and gets stuck in the Connecting status (the ipsec tunnel is not established) .

When running subctl show gateways, I observed that the gateways in member1 and member2 clusters are active, but 0 connections out of 1 are established for each.

Below is a summary of the scenario and the troubleshooting steps I have taken so far.

i'm using 3 clusters with minikube profile (each cluster have isolated network) for karmada and 2 members, setting also different service and pod CIDRs to avoid overlapping .

this is the subctl gather output
hale@hale-VirtualBox:~$ subctl gather

Cluster "karmada-cluster"

Gathering information from cluster "karmada-cluster"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ⚠ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ⚠ Found logs for previous instances of pod submariner-operator-587b7f4fc6-frhq2

 ✓ Gathering operator resources

 ✓ Found 0 submariners in namespace "submariner-operator"

 ✓ Found 0 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "[metadata.name](http://metadata.name/)=submariner-operator" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 0 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 0 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ⚠ Gathering connectivity logs

 ⚠ The Submariner connectivity components are not installed

 ⚠ Gathering connectivity resources

 ⚠ The Submariner connectivity components are not installed

 ⚠ Gathering service-discovery logs

 ⚠ The Submariner service discovery components are not installed

 ⚠ Gathering service-discovery resources

 ⚠ The Submariner service discovery components are not installed

Files are stored under directory "submariner-20240907184724/karmada-cluster"

Cluster "member1"

Gathering information from cluster "member1"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 2 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "[endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io](http://endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io)" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "[metadata.name](http://metadata.name/)=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 2 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 1 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ⚠ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ⚠ Found logs for previous instances of pod coredns-7db6d8ff4d-9cl6p

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "[endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io](http://endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io)" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "[metadata.name](http://metadata.name/)=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "[submariner.io/exportedServiceRef](http://submariner.io/exportedServiceRef)" in namespace ""

Files are stored under directory "submariner-20240907184724/member1"

Cluster "member2"

Gathering information from cluster "member2"

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 2 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "[endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io](http://endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io)" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "[metadata.name](http://metadata.name/)=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 1 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 2 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 1 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ⚠ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ⚠ Found logs for previous instances of pod coredns-7db6d8ff4d-p262h

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "[endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io](http://endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io)" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "[metadata.name](http://metadata.name/)=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "[submariner.io/exportedServiceRef](http://submariner.io/exportedServiceRef)" in namespace ""
Files are stored under directory "submariner-20240907184724/member2"

this is the subctl diagnose all output
hale@hale-VirtualBox:~$ subctl diagnose all

Cluster "karmada-cluster"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported

 ⚠ Submariner connectivity feature is not installed

 ⚠ Submariner service discovery feature is not installed

Cluster "member1"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported

 ✓ Globalnet deployment detected - checking that globalnet CIDRs do not overlap

 ✓ Checking DaemonSet "submariner-gateway"

 ✓ Checking DaemonSet "submariner-routeagent"

 ✓ Checking DaemonSet "submariner-globalnet"

 ✓ Checking DaemonSet "submariner-metrics-proxy"

 ✓ Checking Deployment "submariner-lighthouse-agent"

 ✓ Checking Deployment "submariner-lighthouse-coredns"

 ✓ Checking the status of all Submariner pods

 ✓ Checking that gateway metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that globalnet metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ⚠ Checking Submariner support for the CNI network plugin

 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.

 ✗ Checking gateway connections

 ✗ Connection to cluster "member2" is in progress

 ✓ Checking Submariner support for the kube-proxy mode

 ✓ The kube-proxy mode is supported

 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that Globalnet is correctly configured and functioning

 ✓ Checking that services have been exported properly

Cluster "member2"

 ✓ Checking Submariner support for the Kubernetes version

 ✓ Kubernetes version "v1.30.0" is supported

 ✓ Globalnet deployment detected - checking that globalnet CIDRs do not overlap

 ✓ Checking DaemonSet "submariner-gateway"

 ✓ Checking DaemonSet "submariner-routeagent"

 ✓ Checking DaemonSet "submariner-globalnet"

 ✓ Checking DaemonSet "submariner-metrics-proxy"

 ✓ Checking Deployment "submariner-lighthouse-agent"

 ✓ Checking Deployment "submariner-lighthouse-coredns"

 ✓ Checking the status of all Submariner pods

 ✓ Checking that gateway metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that globalnet metrics are accessible from non-gateway nodes

 ✓ Skipping this check as it's a single node cluster

 ⚠ Checking Submariner support for the CNI network plugin

 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.

 ✗ Checking gateway connections

 ✗ Connection to cluster "member1" is in progress

 ✓ Checking Submariner support for the kube-proxy mode

 ✓ The kube-proxy mode is supported

 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Skipping this check as it's a single node cluster

 ✓ Checking that Globalnet is correctly configured and functioning

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.18.0

and this is how i deploy the broker then how i join members
# deploy the broker in the karmada cluster

sudo /home/hale/.local/bin/subctl deploy-broker --globalnet --kubeconfig .kube/config --context karmada-cluster

# join member1 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member1 broker-info.subm --natt=false --health-check=false

# join member2 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member2 broker-info.subm --natt=false --health-check=false

I’m looking for recommendations or assistance on how to set up locally these clusters differently to avoid this gateway creation issue.
Thank you for your support!

Thanks for contacting @halekammoun ,

Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.

Submariner could not detect your CNI, what CNI are you using in your clusters?
Submariner uses the detected CNI information to automatically retrieve service and pod CIDRs of the cluster.
You can workaround service and pod CIDRs auto discovery by providing it in subctl join command, by adding the following params:
 --clustercidr <pod CIDR> --servicecidr <service CIDR>
Could you reinstall using the updated join command and see if that helps ?

Also please provide subctl gather output, specifically the pod logs.

This seems to be duplicated by https://github.com/submariner-io/submariner-operator/issues/3213. Closing that one has the discussion has started here.

@yboaron @tpantelis
i modified it but still stuck in the connecting status:
kubectl config use-context karmada-cluster

#deploy the broker in the karmada cluster

sudo /home/hale/.local/bin/subctl deploy-broker --kubeconfig .kube/config --context karmada-cluster

# join member1 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member1 broker-info.subm --natt=false \

 --clustercidr <IP_ADDRESS>/16 --servicecidr <IP_ADDRESS>/16 --health-check=false

# join member2 to the broker

sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member2 broker-info.subm --natt=false \

 --clustercidr <IP_ADDRESS>/16 --servicecidr <IP_ADDRESS>/16 --health-check=false

this is the output pf subctl show all
hale@hale-VirtualBox:~$ subctl show all

Cluster "member2"

 ✓ Detecting broker(s)

 ✓ No brokers found

 ✓ Showing Connections

GATEWAY   CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS       RTT avg.

member1   member1   <IP_ADDRESS>   no    libreswan      <IP_ADDRESS>/16, <IP_ADDRESS>/16   connecting

 ✓ Showing Endpoints

CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE

member2   <IP_ADDRESS>   <IP_ADDRESS>   libreswan      local

member1   <IP_ADDRESS>   <IP_ADDRESS>   libreswan      remote

 ✓ Showing Gateways

NODE      HA STATUS   SUMMARY

member2   active      0 connections out of 1 are established

 ✓ Showing Network details

    Discovered network details via Submariner:

        Network plugin:  generic

        Service CIDRs:   [<IP_ADDRESS>/16]

        Cluster CIDRs:   [<IP_ADDRESS>/16]

 ✓ Showing versions

COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH

submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64

submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64

submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

Cluster "karmada-cluster"

 ✓ Detecting broker(s)

NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS

submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          <IP_ADDRESS>/8      65536

 ✗ Showing Connections

 ✗ No connections found

 ✓ Showing Endpoints

CLUSTER           ENDPOINT IP   PUBLIC IP     CABLE DRIVER   TYPE

karmada-cluster   <IP_ADDRESS>     <IP_ADDRESS>   libreswan      local

 ✓ Showing Gateways

NODE              HA STATUS   SUMMARY

karmada-cluster   active      There are no connections

 ✓ Showing Network details

    Discovered network details via Submariner:

        Network plugin:  generic

        Service CIDRs:   [<IP_ADDRESS>/12]

        Cluster CIDRs:   [<IP_ADDRESS>/16]

 ✓ Showing versions

COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH

submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64

submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64

submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

Cluster "member1"

 ✓ Detecting broker(s)

 ✓ No brokers found

 ✓ Showing Connections

GATEWAY   CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS       RTT avg.

member2   member2   <IP_ADDRESS>   no    libreswan      <IP_ADDRESS>/16, <IP_ADDRESS>/16   connecting

 ✓ Showing Endpoints

CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE

member1   <IP_ADDRESS>   <IP_ADDRESS>   libreswan      local

member2   <IP_ADDRESS>   <IP_ADDRESS>   libreswan      remote

 ✓ Showing Gateways

NODE      HA STATUS   SUMMARY

member1   active      0 connections out of 1 are established

 ✓ Showing Network details

    Discovered network details via Submariner:

        Network plugin:  generic

        Service CIDRs:   [<IP_ADDRESS>/16]

        Cluster CIDRs:   [<IP_ADDRESS>/16]

 ✓ Showing versions

COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH

submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64

submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64

submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64

submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

subctl version: v0.18.0

and this is the output of subctl gather
hale@hale-VirtualBox:~$ subctl gather

Cluster "member1"

Gathering information from cluster "member1"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 0 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 3 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 0 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ✓ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 3 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 3 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

Files are stored under directory "submariner-20240910171903/member1"

Cluster "member2"

Gathering information from cluster "member2"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 0 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 2 endpoints in namespace "submariner-operator"

 ✓ Found 3 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 0 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ✓ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 3 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 3 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

Files are stored under directory "submariner-20240910171903/member2"

Cluster "karmada-cluster"

Gathering information from cluster "karmada-cluster"

 ✓ Gathering connectivity logs

 ✓ Found 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 pods matching label selector "app=submariner-routeagent"

 ✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"

 ✓ Found 0 pods matching label selector "app=submariner-globalnet"

 ✓ Found 0 pods matching label selector "app=submariner-addon"

 ✓ Gathering connectivity resources

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"

 ✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"

 ✓ Found 1 endpoints in namespace "submariner-operator"

 ✓ Found 3 clusters in namespace "submariner-operator"

 ✓ Found 1 gateways in namespace "submariner-operator"

 ✓ Found 0 clusterglobalegressips in namespace ""

 ✓ Found 0 globalegressips in namespace ""

 ✓ Found 0 globalingressips in namespace ""

 ✓ Gathering service-discovery logs

 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"

 ✓ Found 1 pods matching label selector "k8s-app=kube-dns"

 ✓ Gathering service-discovery resources

 ✓ Found 0 serviceexports in namespace ""

 ✓ Found 0 serviceimports in namespace ""

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""

 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"

 ✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"

 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""

 ✓ Gathering broker logs

 ✓ Gathering broker resources

 ✓ Found 3 endpoints in namespace "submariner-k8s-broker"

 ✓ Found 3 clusters in namespace "submariner-k8s-broker"

 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"

 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"

 ✓ Gathering operator logs

 ✓ Found 1 pods matching label selector "name=submariner-operator"

 ✓ Gathering operator resources

 ✓ Found 1 submariners in namespace "submariner-operator"

 ✓ Found 1 servicediscoveries in namespace "submariner-operator"

 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"

 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"

 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"

 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"

Files are stored under directory "submariner-20240910171903/karmada-cluster"

@tpantelis @yboaron
this is the logs of the gateway member1 & member2
hale@hale-VirtualBox:~$ kubectl logs submariner-gateway-gcjqn -n submariner-operator --context member2 | tail

2024-09-10T18:47:45.506Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-0-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:45.506Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-0-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:45.506Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-1-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:45.506Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-1-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:45.506Z DBG ..reswan/libreswan.go:282 libreswan            Connection "submariner-cable-member1-192-168-49-2" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:50.590Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-0-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:50.592Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-0-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:50.592Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-1-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:50.592Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member1-192-168-49-2-1-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:47:50.592Z DBG ..reswan/libreswan.go:282 libreswan            Connection "submariner-cable-member1-192-168-49-2" not found in active connections obtained from whack: map[], map[]

hale@hale-VirtualBox:~$ kubectl logs submariner-gateway-rnjwf -n submariner-operator --context member1 | tail

2024-09-10T18:48:39.284Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-0-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:39.284Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-0-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:39.284Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-1-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:39.284Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-1-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:39.284Z DBG ..reswan/libreswan.go:282 libreswan            Connection "submariner-cable-member2-192-168-58-2" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:44.387Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-0-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:44.387Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-0-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:44.387Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-1-0" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:44.387Z DBG ..reswan/libreswan.go:268 libreswan            Connection "submariner-cable-member2-192-168-58-2-1-1" not found in active connections obtained from whack: map[], map[]

2024-09-10T18:48:44.387Z DBG ..reswan/libreswan.go:282 libreswan            Connection "submariner-cable-member2-192-168-58-2" not found in active connections obtained from whack: map[], map[]

Sorry - I didn’t mean the actual console output from subctl gather but the files that it outputs which includes the pod logs.

@tpantelis i'm sorry! here is the folder
submariner-20240910185521.zip

I can see from the logs that the IPsec tunnels are set up in the OS but that never go active:
Connection "submariner-cable-member2-192-168-58-2-0-1" not found in active connections obtained from whack
Perhaps there's a firewall preventing traffic between them? I know certain ports have to open but I'll let the dataplane experts chime in. @yboaron @aswinsuryan

From GW pod logs it seems that NatDiscovery failed [1]  on both clusters ,
Can you verify that submariner firewall prerequisites are supported ?
You can use [2] subctl command to verify inter-cluster firewall prerequisites
[1]
USING_SRC: <IP_ADDRESS>:4490, USING_DST: <IP_ADDRESS>:4490 2024-09-10T17:02:18.124Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x25310406a19d650f, SENDER: "submariner-cable-member2-192-168-58-2", RECEIVER: "submariner-cable-member1-192-168-49-2", USING_SRC: <IP_ADDRESS>:4490, USING_DST: <IP_ADDRESS>:4490 2024-09-10T17:02:18.124Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x25310406a19d6510, SENDER: "submariner-cable-member2-192-168-58-2", RECEIVER: "submariner-cable-member1-192-168-49-2", USING_SRC: <IP_ADDRESS>:4490, USING_DST: <IP_ADDRESS>:4490 2024-09-10T17:02:20.125Z WRN ..ery/natdiscovery.go:166 NAT                  NAT discovery for endpoint "submariner-cable-member1-192-168-49-2" has timed out
[2]
subctl diagnose firewall inter-cluster --context <localcontext> --remotecontext <remotecontext>

yes i followed firewall prerequisites
i'm using just one vm should i set natt=true?
hale@hale-VirtualBox:~$ subctl diagnose firewall inter-cluster --context member1 --remotecontext member2

 ✓ Checking if tunnels can be setup on the gateway node of cluster "member1"

 ✓ Skipping this check as it's a single node cluster

 ✓ Tunnels can be established on the gateway node of cluster "member1"

hale@hale-VirtualBox:~$ subctl diagnose firewall inter-cluster --context member2 --remotecontext member1

 ✓ Checking if tunnels can be setup on the gateway node of cluster "member2"

 ✓ Skipping this check as it's a single node cluster

 ✓ Tunnels can be established on the gateway node of cluster "member2"

A.
I can see that Submariner RouteAgent pod failed to discover the CNI interface (interface with IP address from pod CIDR range) , Submariner uses CNI if IP address to set SNAT rule to allow hostnetworling to remote cluster connectivity, so without discovering CNI interface hostnetworking to remote cluster support will be broken.
which CNI do you use ?
B.
Also noticed that Submariner GW uses private IPs (<IP_ADDRESS>, <IP_ADDRESS>) for inter-cluster connectivity, to force IPSec traffic to be UDP-encapsulated (port  4500)  add --force-udp-encaps to subctl join command.
C.
subctl diagnose firewall inter-cluster isn't supported for single node clusters , so we need to find another way to verify that Submariner inter-cluster traffic is allowed.
So, you can change to multi-nodes clusters and run  subctl diagnose firewall inter-cluster or just sniff traffic on GW nodes and verify submariner inter-cluster traffic (udp port 4500,4490) sent/received.

Hi @halekammoun , any update?

Looks like this discussion has slowed down. Closing the issue for now, feel free to re-open or open a new one if required.


Adds Input Field Types mergeable validation rule

We need to explain how the nullability merges ...

Closed by #93


I don't see the possibility to read a remote csv file

I would like to have the possibility to read a remote csv file. it would be very useful. can you implement it please?

fopen should support external URLs.


try using a generic issue template to track data updates

background
we don't currently have a standardized way to track data updates from end-to-end. by track I primarily mean the ability for anyone in GDE to wonder "what's the status of that update?" and find an up-to-date answer
we currently have issue templates for each product, but the bulk of what makes them product-specific are lists and details on source data. those details may be more valuable as documentation stored in other forms (readmes, wiki)
approach
beyond using a single issue template to track updates, an issue form seems like an interesting way to create a template and get structured user input when creating the new issue here (how much use input is really needed/useful though if it's just a checklist?)

gonna merge this since it seems it won't have an effect until the changes are in main and this shouldn't disrupt anything


IMiscellaneousOperatons should be renamed to ISystemOperations

To better match the Docker API documents we should rename IMiscellaneousOperations to ISystemOperations including:
CheckAuthConfigurationAsync();
GetSystemInformationAsync();
GetVersionAsync();
PingAsync();
MonitorEventsAsync();
GetDataUsageAsync();

I can do this one :)

Sounds great! Thanks

This has been fixed.


fix non-slug app names sent in emails

if fixes this issue
https://github.com/fossasia/open-event-webapp/issues/604
Signed-off-by: Arnav Gupta<EMAIL_ADDRESS>

@aayusharora please check and merge :)

Current coverage is 47.26% (diff: 0.00%)

Merging #610 into development will not change coverage

@@           development       #610   diff @@
=============================================
  Files                5          5
  Lines              548        548
  Methods             52         52
  Messages             0          0
  Branches            88         88
=============================================
  Hits               259        259
  Misses             289        289
  Partials             0          0

Powered by Codecov. Last update 06edafb...9236f76


fix: API multipart spec flakiness

Description

This PR fixes the flakiness of the app/client/cypress/e2e/Regression/ServerSide/ApiTests/API_MultiPart_Spec.ts spec file.
It removes the dependency on external library cloudinary for multipart upload test
It removes the test step where the cloudinary api was expecting error if the array type was selected for multipart form data key instead of file type.
Moved the cloudinary api to ted api
Fixes the issue of the None type selection which goes out of the view port after the multipart form data is selected in the dropdown.

Fixes #35137
or
Fixes Issue URL

[!WARNING]
If no issue exists, please create an issue first, and check with the maintainers if the issue is valid.

Automation
/ok-to-test<EMAIL_ADDRESS>@tag.Binding, @tag.Visual"
:mag: Cypress test results

[!WARNING]
Tests have not run on the HEAD 99e0865b268deaafec33f24bd44e3707a6d44acb yet
Wed, 28 Aug 2024 08:09:53 UTC

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[ ] No

Summary by CodeRabbit

New Features

Introduced a new multipart API for image uploads, replacing the previous Cloudinary upload functionality.
Enhanced the DataManager with a new property for the multipart API endpoint, expanding API interaction capabilities.

Bug Fixes

Updated success and error handling messages in the API test cases to reflect the new multipart API.

Documentation

Adjusted the testing focus from client-side templates to server-side API multipart tests in the configuration.

Refactor

Simplified API interaction methods by removing unnecessary visibility checks in the UI interactions.

/ci-test-limit

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486

/ci-test-limit runId=10592758486


reorder force flush condition + add 502 as retriable error

fixes #31
add error code 502 as retriable error as per https://cloud.google.com/storage/docs/json_api/v1/status-codes#502_Bad_Gateway

I think the test must be adapted

Yes, the reordering did this, but I think that the logic now is essentially right, but that write should check for forced=True and raise the error there. Getting the tests to work is, in general, a bit of a pain, because they rely on vcrpy, which is rather flaky.
Sorry I haven't been able to dedicate any effort to this project. I hope to come back to it and clean up the several small-but-annoying issues.

Thanks @martindurant and team for Pythonic file-object support of GCS.
Why Google has not this covered yet in their client library is beyond me.
As a fyi, the arguments for it, lack of traction and perceived resistance: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3903
In the meantime I am researching how to use service accounts for GCSFS credentials as this fits a use case of mine.

Superseded by #37


fix bug where error at import dashboard fails to show toast in "welcome" app

CATEGORY
Choose one

[ x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY

The toast messages weren't working  when there was an error redirect  from the import template to the react dashboard component. To fix this common data had to be passed into the components the app level to show toast or not.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

To test out go to import and try to upload a non dashboard. You will be redirected to page to dashboardlist with the correct toast.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

Codecov Report

Merging #9714 into master will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9714      +/-   ##
==========================================
- Coverage   70.72%   70.66%   -0.06%
==========================================
  Files         584      182     -402
  Lines       30382    17818   -12564
  Branches     3112        0    -3112
==========================================
- Hits        21487    12591    -8896
+ Misses       8781     5227    -3554
+ Partials      114        0     -114

Flag
Coverage Δ

#cypress
?

#javascript
?

#python
70.66% <ø> (-0.22%)
:arrow_down:

Impacted Files
Coverage Δ

superset/views/database/mixins.py
58.92% <0.00%> (-21.43%)
:arrow_down:

superset/utils/cache.py
45.83% <0.00%> (-20.84%)
:arrow_down:

superset/db_engine_specs/mysql.py
78.26% <0.00%> (-13.05%)
:arrow_down:

superset/views/database/api.py
83.90% <0.00%> (-3.45%)
:arrow_down:

superset/db_engine_specs/postgres.py
97.29% <0.00%> (-2.71%)
:arrow_down:

superset/models/core.py
84.47% <0.00%> (-2.39%)
:arrow_down:

superset/views/core.py
74.90% <0.00%> (-0.23%)
:arrow_down:

superset/connectors/sqla/models.py
88.41% <0.00%> (-0.16%)
:arrow_down:

...ntend/src/dashboard/components/PropertiesModal.jsx

superset-frontend/src/components/AnchorLink.jsx

... and 398 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d167af...02cc383. Read the comment docs.


Providing micro-sam as napari plugin widgets

I think we need a thread to discuss implementing napari plugin widgets for micro-sam.
This is follow up work to https://github.com/computational-cell-analytics/micro-sam/pull/165

Also including an error Constantin ran into (I have not seen the same error, but will keep an eye out for it now)
https://github.com/computational-cell-analytics/micro-sam/pull/165#issuecomment-1697518398

When trying to start the 2d annotator plugin I get an error:
RuntimeError: Failed to import command at 'micro_sam.sam_annotator.annotator_2d:annotator_2d_widget': /lib/x86_64-linux-gnu/libtinfo.so.6: version `NCURSES6_TINFO_6.2.20211010' not found (required by /home/pape/software/conda/miniconda3/envs/sam-plugin/lib/python3.10/site-packages/vigra/../../../././libncurses.so.6)
(Just fyi, just importing vigra works).

Hi @GenevieveBuckley,
thanks for opening the issue and working on this! Here are my initial thoughts on how to change the architecture to enable the plugins.
1. Computing Image Embeddings and "raw" image layer
We add a new widget for the embedding computation. The user can just select an image layer that is used to precompute the embeddings (and optionally to specify a path where the embeddings are cached). This will also solve #161 (cause we get a progress bar "for free") and #17 as the name "raw" is not hard-coded anymore.
As far as I understand we can't rely on global variables any more when we switch to the plugin, so we need to store the image embeddings somewhere  else than in IMAGE_EMBEDDINGS, e.g. as a property of the viewer or the corresponding image layer. (Same is true for the other global state we currently have)
2. Other hard-coded layers
For the other hard-coded layers (prompts, box_prompts, current_object, commited_objects, etc.) it would be best to keep them hard-coded, and if possible to just have the plugin create them when called. From my understanding this should be possible by having type annotated return values for these layers.
3. Type annotations and magicfactory
My current opinion is: we solve this via wrapper functions. We have the "actual" function that corresponds to the plugin and is decorated with @magicfactory and then write wrapper functions around it to keep support for the current functionality like starting from python and CLI. I think this will be necessary in any case, even if you would solve the Union problem, because some of these wrappers will need specially functionality, e.g. taking the existing viewer and returning the viewer again.
4. Development
I would suggest to start with a PR that creates the plugin for the 2d annotation tool. Let's create the PR onto https://github.com/computational-cell-analytics/micro-sam/tree/dev, so that we can work on all the updates there and only merge into master once the plugins are all there, and the current functionality is adapted to the changes.
Let me know what you think about these points and plan @GenevieveBuckley! (And please let me know if you disagree with any of the technical points or have a better plan!!)

Response:

Computing Image Embeddings and "raw" image layer

"We add a new widget for the embedding computation" - ok, sure
"the name "raw" is not hard-coded anymore" - won't it still be hard coded in all of the sam_annotator python files? It's all the downstream stuff that makes this a little complicated.
That's a good point about the global variables. I didn't realise there were so many, that's good to know.

Other hard-coded layers

I don't quite understand what you mean by "From my understanding this should be possible by having type annotated return values for these layers."

Type annotations and magicfactory

Agree, wrapper functions for widgets seem like a good approach.

Development

I don't have access to push to your dev branch. You can either grant me write access, or I can open a PR (we both have permissions to push to that).

"the name "raw" is not hard-coded anymore" - won't it still be hard coded in all of the sam_annotator python files? It's all the downstream stuff that makes this a little complicated.

It's not hard-coded in many cases:
$ grep -Inr "\"raw\"" sam_annotator
sam_annotator/annotator_3d.py:160:    shape = v.layers["raw"].data.shape
sam_annotator/annotator_2d.py:64:        AMG.initialize(v.layers["raw"].data, image_embeddings=IMAGE_EMBEDDINGS, verbose=True)
sam_annotator/annotator_2d.py:68:    shape = v.layers["raw"].data.shape[:2]
sam_annotator/annotator_2d.py:170:    v.layers["raw"].data = raw
sam_annotator/annotator_tracking.py:226:    shape = v.layers["raw"].data.shape
sam_annotator/annotator_tracking.py:354:    shape = v.layers["raw"].data.shape

I think we can get rid of this without too much effort.

That's a good point about the global variables. I didn't realise there were so many, that's good to know.

The global state for each annotator is defined in the first few lines of the annotator_... function, e.g. https://github.com/computational-cell-analytics/micro-sam/blob/master/micro_sam/sam_annotator/annotator_2d.py#L220-L221.
It's not that much for annotator_2d and 3d, tracking has a bit more (but that's the most complex one anyways and we should probably tackle it last).
I am not really sure what the best strategy is to represent this state in a plugin.

I don't quite understand what you mean by "From my understanding this should be possible by having type annotated return values for these layers."

From what I understand from https://napari.org/stable/plugins/guides.html something along these is possible in the function that initializes the plugin:
@magicfactory
def my_plugin() ->  List[LayerTypes]:
   ...
   return [prompt_layeer, box_prompt_layer, ...]

and the viewer will create the corresponding layers, so that we can add these specific layers during plugin creation.
(If this is not clear: I can check this more myself and try to write a minimal example).

Agree, wrapper functions for widgets seem like a good approach.

:+1:

I don't have access to push to your dev branch. You can either grant me write access, or I can open a PR (we both have permissions to push to that).

Sorry, I wasn't quite clear. What I meant: the PRs should be for merging into dev rather than merging into master.
(The tool is used quite a bit now, and some people are using it by installing from current master to have newer features. By merging the PRs for the migration to the plugin in dev rather than master we avoid confusion due to incomplete plugin support etc.)

One other thing I notice is that we want to make sure the path through the code doesn't include napari.run() (or too many v = napari.Viewer()) statements. I think we can manage to balance making things work from scripts as well as interactively.

One other thing I notice is that we want to make sure the path through the code doesn't include napari.run()

Good point. I think we can avoid this with the wrapper design (exemplified for the 2d annotation tool).
@magicfactory
def annotator_2d_plugin(...) -> ,,,:
    """This function implements the core logic and is used to provide the plugin"""

def annotator_2d(image_data, ...):
    """This function can be called in python (or from CLI) to start the annotation tool"""
    v = napari.Viewer()
    v.add_image(image_data)
    layers = annotator_2d_plugin(...)
    for layer_data in layers:
        v.add_layer(layer_data)
    napari.run()

A quick summary on what @GenevieveBuckley , @paulhfu and me discussed in a meeting this morning.

We will break up the work towards the plugin into smaller packages, the next goals are:

Refactoring the global state #201
Exploring widget containers #200
Using thread workers #202
Implementing an embedding widget #203
and enabling GUI tests in the CI #205

Based on this we can then

Implement the plugin for the 2d annotator, continuing (or replacing) #177
And see how to share functionality for the different annotation plugins #206 before porting all tools to a plugin

This has been addressed by #304. I have opened several follow-up issues to improve the plugin.


Add Travis CI configuration

Add a Travis CI config to:

Run unit tests
Use test matrix for last few 3.x versions
Use test matrix for v 2.7

Problem: liboqs-python repo doesn't have the SO file. Options include:

Slower: build liboqs (master & nist-branch) in travis, link to SO & run (~5 mins per container?)
Middle: Provide pre-built binaries to download (~2.7mb on linux). Store where? Who updates them?
Fast: redo tests with unittest.mock to replace ctypes interface. This is fast for unittests at the expense of connecting to the SO. Use example py script for that?

Optional/nice to haves

Run test code coverage checker
Run any style check / static analysis tools?

Pre-built binaries are temporarily (until the next liboqs release) available at

https://openquantumsafe.org/binaries/liboqs-nist/c13eada65cd46931829c4a2fcae5b9d3516ff3a8/ubuntu_gcc/liboqs.so
https://openquantumsafe.org/binaries/liboqs-nist/c13eada65cd46931829c4a2fcae5b9d3516ff3a8/macos10.14.2_clang/liboqs.so
https://openquantumsafe.org/binaries/liboqs-nist/c13eada65cd46931829c4a2fcae5b9d3516ff3a8/macos10.14.2_clang/liboqs.a
https://openquantumsafe.org/binaries/liboqs-master/a7da6ec8ea8d4c143261c2041191e3a1c84e907e/ubuntu_gcc/liboqs.so
https://openquantumsafe.org/binaries/liboqs-master/a7da6ec8ea8d4c143261c2041191e3a1c84e907e/macos10.14.2_clang/liboqs.0.dylib
https://openquantumsafe.org/binaries/liboqs-master/a7da6ec8ea8d4c143261c2041191e3a1c84e907e/macos10.14.2_clang/liboqs.a

Fixed by ba4a8b8f85e3859ac946cacf9abd37201a92caec.  That commit adds continuous integration tests using CircleCI which is container-based and runs really quickly.  It builds liboqs master, then runs the Python test suite using python3.


Add missing checks to the Condensed BP Check script

We have a "condensed check" script of best practices that we're usually using for the SQL Server Review:
https://github.com/MadeiraData/MadeiraToolbox/blob/8f478c3166d8e3f0284bbea12a69f33afda7716d/Best Practices Checklists/Best Practices Condensed Check.sql
However, it's incomplete, and missing several checks, most of which exist in other scripts:

[ ] High VLF Count
[ ] Foreign Keys without corresponding indexes
[ ] Heap Tables
[ ] Invalid owner for system schema role or database
[ ] Outdated Statistics
[ ] Redundant Indexes
[ ] Orphaned jobs
[ ] Additional checks from Azure-SQL-Tips
[ ] Table and Index High Unused Space

Feel free to also add more checks and list them here.

I've also opened an issue to make it suitable for AzureSql #5

This issue was duplicated into our new repository: https://github.com/MadeiraData/microsoft-dbas-club/issues/1


Center point selected during an interactive session is not honored

Widget
Distance and Direction  - Circle Tab
Version of widget
NA
Bug or Enhancement
When a user selects a location on the map for the center point of a circle what is created after the second click is using a different center point.

Repo Steps or Enhancement details
Add the widget to an app
Launch the widget
Click a location using the interactive method of a circle

@topowright Lyle, I noticed this issue this morning and thought I had fixed it. Can you confirm you are using the latest version of the widget that includes PR #908

@adgiles This issue has been fixed with  06-22-2017_1603.  Thanks for the quick turn around.


[WIP] Geometry Engine tests and cleanup

This PR will

Complete the GeometryEngine tests
Fix any test failures
Return to sampling torsions from a pdf

Right now, the openeye problem seems to have reared its head again.

Ok, going to go through and see if I can fix the openeye thing:

Confirmed I am running Python 3.4

Confirmed the OEMol has no bonds when it is given to OEAssignPartialCharges

generateTopologyFromOEMol seems to produce a topology with all bonds from one atom to itself. Will move discussion to openmoltools.

Actually, I think I see the problem.

I neglected to
    oechem.OETriposAtomNames(mol)
    oechem.OETriposBondTypeNames(mol)

(debugging unit madness at the moment)

Found an error... will fix now.

definitely some bond length weirdness in the reverse logp..

Ok, now the logp_bond is only ~2, but logp_phi is ~-75 for the reverse when it's more like -4 for the forward proposal. Investigating...

I think it might be another unit error. One sec.

I'm going to merge this and then make sure the GeometryEngine API is consistent with the rest of the package (fixing the build failures) then add the last tests.


[Feature] Better handling of connector init failure / disconnect

Feature description
Currently we assume the connector init() function always works.
If the init() function doesn't work, the connector is left in a broken state, and multiple error toasts are displayed each time a call is requested (at least two - getinfo and invoices)
Describe the solution
I think we either should have a state where shows the connection failed, and the user can either retry or switch account. If the init fails or the connector disconnects at some point, we shouldn't still try to access the connector to get new data (which causes a cascading list of error messages)
Describe alternatives
No response
Additional context
No response
Are you working on this?
None

Notes from @reneaaron
I always wondered why after a failed request with Alby (offline, 503 error, etc) I have to reload the extension before it starts to work again. Seems the the failed promise is stored in state.connector which was introduced with this commit. (this breaks every subsequent request done via the connector and prevents you e.g. from unloading, because the failed promise is cached and immediately returns)
Now I just stumbled upon Roland's PR to fix that for LNC and think this needs to be solved at the root of the problem. (and not in the connector individually)

Another issue: It's impossible to switch accounts / cancel loading of an account. You have to wait for the account to fail loading before you can switch.

Notes from @reneaaron
I always wondered why after a failed request with Alby (offline, 503 error, etc) I have to reload the extension before it starts to work again. Seems the the failed promise is stored in state.connector which was introduced with this commit. (this breaks every subsequent request done via the connector and prevents you e.g. from unloading, because the failed promise is cached and immediately returns)
Now I just stumbled upon Roland's PR to fix that for LNC and think this needs to be solved at the root of the problem. (and not in the connector individually)

so, the same try /catch of lnc here? https://github.com/getAlby/lightning-browser-extension/blob/master/src/extension/background-script/state.ts#L112

@fczuardi that would be a good start (I think there is one other usage of connector.init() which would also need to have the same error handling)


Move hard-coded indexshow styles to Sass partial

The inline style attributes prevent users from customizing the presentation of index terms when these are emitted to HTML5 output with args.indexshow=yes.
These changes move the default presentation rules to CSS to allow users to override presentation in custom stylesheets.
The output is visually equivalent to the results generated by previous toolkit versions.
Signed-off-by: Roger Sheen<EMAIL_ADDRESS>

Note: The same inline styles are duplicated in org.dita.xhtml/xsl/xslhtml/dita2htmlImpl.xsl and org.lwdita/xsl/dita2markdownImpl.xsl, but I assume we'd want to leave XHTML alone, and I'm not sure whether the LwDITA variant would pick up the styling from the HTML5 CSS files.


Use object name in discriminator mapping

Problem description
In Area, the mapping uses CIRCLE to reference the Circle object... but CIRCLE should be Circle
Expected behavior
The left side of the mapping should be the object name, as explained in the design guidelines, to simplify implementation and consumption.
Alternative solution
        mapping:
          Circle: "#/components/schemas/Circle"

    AreaType:
      type: string
      description: |
        Type of this area.
        Circle - The area is defined as a circle.
      enum:
        - Circle

Hi @patrice-conil,
We recently had the same discussion around mapping for eventType discriminator in CloudEvents. mapping in OAS 3 was introduced precisely to allow having independence between enum values and schema names. If both are the same, the mapping is redundant, OAS3 does not mandate to use it.
The key for mapping is the enum value, and we historically use UPPER_SNAKE_CASE for them, or kebab-case in the case of CloudEvents, while for schema names we use UpperCamelCase. So there is an inconsistency in the guidelines if we ask on one hand to be the same value, and on the other, to have same different formats.
Here we could adopt UpperCamelCase for "Circle" and it would work, But in geofencing we hace CloudEvents and we have the same situation.

Hi @patrice-conil,
We recently had the same discussion around mapping for eventType discriminator in CloudEvents. mapping in OAS 3 was introduced precisely to allow having independence between enum values and schema names. If both are the same, the mapping is redundant, OAS3 does not mandate to use it.
The key for mapping is the enum value, and we historically use UPPER_SNAKE_CASE for them, or kebab-case in the case of CloudEvents, while for schema names we use UpperCamelCase. So there is an inconsistency in the guidelines if we ask on one hand to be the same value, and on the other, to have same different formats.
Here we could adopt UpperCamelCase for "Circle" and it would work, But in geofencing we hace CloudEvents and we have the same situation.

@jlurien, you're right, I close this one.


[Reworked] GCP metadata server credentials support

This is a picked version of #16 of @bryancymo to support google metadata token api integrated into one gcs-rsync example bin.
This version uses a dyn trait over TokenGenerator to avoid static dispatch issue when it comes to use the api.
In the end, only one gcs-rsync bin will be available and the feature can be activated by adding a flag : -u, --use-metadata-token-api
⚠ This version breaks the compatibility over the TokenGenerator dyn trait

[x] Use TokenGenerator dyn trait to switch between AuthorizedUser/Metadata in gcs-rsync example
[x] Fix new clippy errors
[x] Upgrade to Rust 2021
[x] Provide a new conf to make integration tests working

Hello @bryancymo, could you give it a try by cherry-picking this PR on our hand?
I did not tested it yet in a real gcp context.

Just checked the codecov report. Seems to be a change in codecov since only the metadata implementation has not been tested with automated tests

Hello @bryancymo, I merged the PR and published a new docker image v0.2.0. Could you give it a try to see if everything is good ?


Static cache & torch.compile

Does vLLM make use of the new speed up in transformers around using a static cache and torch compile?
https://x.com/art_zucker/status/1758510984631845278?s=46

@p-christ from my understanding StaticCache is an attempt to get something like the pre-allocated KVCache vLLM already has with PagedAttention. I wouldn't expect it to offer improvements to this project then.
There is an item on the roadmap for exploring torch.compile support though, which could be useful to fuse the small operations in models https://github.com/vllm-project/vllm/issues/2681


Latest build (7.2.0) breaks server-side rendering

Hi, it looks as though the latest build introduced debug code that relies on the DOM in src/animated/FrameLoop.js: https://github.com/react-spring/react-spring/commit/5af3f6bb1623cc3b36733518cdf71b8bf3e669db#diff-ca0fe0482065f1abd8b878839eb281ad
This is breaks server-side rendering/Node compatibility, as the package attempts to write to window at import time, resulting in this error:
ReferenceError: window is not defined
    at Object.<anonymous> (/Users/cecchi/Code/deal/consumer-app/node_modules/react-spring/web.cjs.js:1222:1)
    at Module._compile (internal/modules/cjs/loader.js:689:30)

whoops, that's left over debug code, released a small patch version to take this out

🎉  Thank you!


[Enhancement] use resolveExpr to resolve the materialized view's partition expr

Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 67 / 76 (88.16%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
58
66
87.88%
[1304, 1335, 1358, 1376, 1387, 1392, 1401, 1402]

:large_blue_circle:
com/starrocks/sql/analyzer/MaterializedViewAnalyzer.java
9
10
90.00%
[600]

@choury Please comment what issue do you want to fix, thanks

[FE Incremental Coverage Report]
:heart_eyes: pass : 69 / 78 (88.46%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
60
68
88.24%
[1304, 1338, 1361, 1379, 1390, 1395, 1404, 1405]

:large_blue_circle:
com/starrocks/sql/analyzer/MaterializedViewAnalyzer.java
9
10
90.00%
[600]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 65 / 74 (87.84%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
56
64
87.50%
[1303, 1337, 1360, 1378, 1389, 1394, 1403, 1404]

:large_blue_circle:
com/starrocks/sql/analyzer/MaterializedViewAnalyzer.java
9
10
90.00%
[607]

@Mergifyio backport branch-3.2

@Mergifyio backport branch-3.1


ls

Before opening an issue:

Read the common workarounds in the TROUBLESHOOTING.md
Search for duplicate or closed issues
Validate that it runs with both Ruby Sass and LibSass
Prepare a reduced test case for any bugs
Read the contributing guidelines

When reporting an bug, you must provide this information:

NPM version (npm -v):
Node version (node -v):
Node Process (node -p process.versions):
Node Platform (node -p process.platform):
Node architecture (node -p process.arch):
node-sass version (node -p "require('node-sass').info"):
npm node-sass versions (npm ls node-sass):

When encountering a syntax, or compilation issue:

Open an issue on LibSass. You
may link it back here, but any change will be required there, not here

If you delete this text without following it, your issue will be closed.

No issue reported


[ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './optimizer' is not defined by "exports"

Qwik Version
0.11.0
Operating System (or Browser)
Linux
Node Version (if applicable)
v16.13.1
Which component is affected?
Qwik Rollup / Vite plugin
Expected Behaviour
npm run dev and the other build commands should work.
Actual Behaviour
npm run dev returns this error:
❯ npm run dev

> dev
> vite --mode ssr

failed to load config from /home/user/qwik-app/vite.config.ts
error when starting dev server:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './optimizer' is not defined by "exports" in<EMAIL_ADDRESS>    at new NodeError (node:internal/errors:371:5)
    at throwExportsNotFound (node:internal/modules/esm/resolve:440:9)
    at packageExportsResolve (node:internal/modules/esm/resolve:645:7)
    at resolveExports (node:internal/modules/cjs/loader:482:36)
    at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/user/qwik-app/vite.config.ts:33:24)

Additional Information
Steps to reproduce:

Create a new project with npm create qwik@latest
run npm run dev

now you should see the error.
Checking the qwik package.json this is what I have in the exports field for the optimizer entry:
    "./optimizer.cjs": "./optimizer.cjs",
    "./optimizer.mjs": "./optimizer.mjs",
    "./optimizer": {
      "import": "./optimizer.mjs"
    },

the require entry is missing.
Changing the optimizer import in vite.config.ts into @builder.io/qwik/optimizer.cjs fixes the error.

@gdorsi I'm also running into the same error. Can you please share the code that fixed the problem for you?

Fixed in 0.11.1
thanks for the issue report


Fix/msvc compilation

This PR fixes two issues with compilation on Windows using MSVC.

forward declaration inconsistency with the actual definition of the cuda function MeshIntersectionKernelLauncher
fix of preprocessor condition syntax that was not working properly on MSVC

With these two fixes, I was able to compile on Windows just fine, the setup was the following:
Python version: 3.6
CUDA version: 10.1
Pytorch version: 1.4.0

Hi @coloss ,
Thank you for your interest in Kaolin!
Can you signoff your first commit ? (you can use git rebase -i -S).

Hi @Caenorst . Did that do it? I've actually never had to do that before so I'm not sure if I did it right.

The two commits doesn't seems to be signed properly.
your second commit is actually signed but with a GPG-key which apparently removed the signature ?
you can look at https://github.com/NVIDIAGameWorks/kaolin/pull/200/checks?check_run_id=588333880 for status
Basically at the end of all your commits messages there should be:
Signed-off-by: your_github_username <your_email_address>
I believe the email address should one of those linked to you github account.

Hi @coloss your PR looks good to me, but we can't merge it until the commits are passing your DCO bot. Signoff your commits as recommended above ?

@Caenorst , sorry for the delay. I forgot about this. I did add the changes. Is it fine like this?

@coloss The checks seem to have failed.
The first of them is easy to fix (signoff on all your commits; for convenience, see instructions here https://github.com/NVIDIAGameWorks/kaolin/pull/200/checks?check_run_id=632291095)
The second issue seems to be with compiling the library; do you already know that error?

Note the subtle issus in the signoff. It seems you signed off, but using different creds (likely cause you were on a different machine?)
Expected "Radek Danecek<EMAIL_ADDRESS>but got "coloss<EMAIL_ADDRESS>


Add new options to exec command

Add two new options to exec command:

-s, like other commands, for so called "silent mode"
-i  to have environment inherits variables before being populated/modified

@stoned cloud you mind posting a new PR for helmfile for updating vals?

@stoned I've already published vals v0.25.0 for this so it would be great if you could use v0.25.0 as the version number when you're submitting the PR!


[Enhancement] Config the run command

Is that possible to add a function to configure the run command ?
e.g. Currently the run code command execute clang++ current-file && wasm a.out, I would like to add some parameters to clang++ when building the out file

I will work on this.

I will work on this.

YES, ADD THIS
maybe a .run file?
Or a config menu?


validate plan concreteness before execution

Related to #613
This attempts to validate plan concreteness (e.g. missing fields) before execution.
The objective is to provide nicer errors when fields are missing: instead of complaining about low-level task fields (e.g. foo._bar._exec.input), this will provide the "source" error (e.g. foo.source).
There are some heuristics in place:

Recursively walk the entire tree (avoiding definitions and optional fields)
Scalars (string, bool, etc): Make sure they're concrete, or have default values, or is a reference
Core types (#FS, #Secret, etc): Make sure they're references
Core actions (e.g. core.#ReadFile): validate, but skip generated fields

Example In Action
Given this config:
#Test: {
	required: string
	_op: core.#Pull & {
		source: required
	}
}

dagger.#Plan & {
	actions: test: #Test & {}
}

Before:
$ dagger do -p ./definition.cue test
[✗] actions.test
6:08PM FTL failed to execute plan: task failed: actions.test._op: actions.test._op.source: non-concrete value string

After:
$ dagger do -p ./definition.cue test
failed to load plan: "actions.test.required" is not set:
    ./definition.cue:12:2

Validation happens before execution (e.g. Parse error rather than Runtime error)
Error is caught at the source (e.g. complains about test.required field rather than test._op.source
Error includes stack trace (where it happened). NOTE This will always be the file where the field is DEFINED, not where it's missing

Notes

Every field now must either be concrete, a reference, optional or come with a default value. If that's not the case, the validation will fail. We had a couple of instances of that in universe which I had to tweak
Core Actions are a special case: some fields will be made concrete by the runtime at execution. Those fields are now flagged with @dagger(generated). This instructs the validator that it's okay if they're not concrete.

/cc @helderco @jlongtine @shykes

Another example using a higher level (docker.#Run package):
dagger.#Plan & {
	actions: test: docker.#Run
}

Before: confusing error about actions.test._exec.input
$ dagger do -p ./docker_image.cue test
[✗] actions.test
6:30PM FTL failed to execute plan: task failed: actions.test._exec: invalid FS at path "actions.test._exec.input": FS is not set

After: missing actions.test.input
$ dagger do -p ./docker_image.cue test
failed to load plan: "actions.test.input" is not set:
    ../../cue.mod/pkg/universe.dagger.io/docker/run.cue:13:2

NOTE: The runtime now has knowledge of docker.#Image, otherwise the error would have been actions.test.input.rootfs since that's the core type.

This is pretty great! 🙂
I see you're adding attributes to outputs generated from core actions, as you can't tell between an input and an output on those (e.g., source and digest from core.#Pull). I was about to suggest this for:

https://github.com/dagger/dagger/issues/1351#issuecomment-1104578526

For now I'm just comparing against the computed values.

NOTE: The runtime now has knowledge of docker.#Image, otherwise the error would have been actions.test.input.rootfs since that's the core type.

Actually, I had to revert this bit. Can't handle #Image like a core type because it isn't (and it's being modified by universe packages so the heuristics don't work). Which means the runtime complains about .<image>.rootfs missing rather than just .<image>

I see you're adding attributes to outputs generated from core actions, as you can't tell between an input and an output on those (e.g., source and digest from core.#Pull). I was about to suggest this for:

Yes! Couldn't find an alternative, but the runtime really needs this for the special case of "consider this field concrete" even though it's not

Love this for debugging plans that don't run.

Did another batch of improvements.

Wrap the error with the position of the parent value. For instance, given actions: foo: yarn.#Run & {}, if actions.foo.source is missing, grab the position of actions.foo for the stack trace (e.g. so it complains about your_file.cue in addition to cue.mod/pkg/universe.dagger.io/yarn/yarn.cue)

Special casing for docker.#Image: Rather than complaining about <image>.rootfs, complain about <image>

Example of the two combined:
dagger.#Plan & {
	actions: test: docker.#Run
}

Will result in:
failed to load plan: "actions.test.input" is not set:
    ./test:13:11
    ./cue.mod/pkg/universe.dagger.io/docker/run.cue:13:2

Maybe this fixes some of the issues in https://github.com/dagger/dagger/issues/493#issuecomment-1086120601
We need to re-test them.


Rewrite the mechanism for rendering descriptions and match explanations

See https://github.com/google/googletest-rust/issues/325#issuecomment-1826423227.
This is a rather large PR, so I've split the steps into individual commits which are hopefully individually understandable. I wanted to present it as a whole so that it's clear what the whole plan is.
This does not replace all existing code for constructing descriptions and match explanations. It focuses just on the parts which I could see were broken. We can clean the rest up as we go along.
Fixes #325

I have seen a few times implementation still building the description like so: format!("which has inner state {actual.inner}, {}", self.inner.explain_matches(actual.inner)).into(). Should this use the Description builder instead? This can also be fixed in a follow up

Yeah, I didn't update all uses yet but rather just focused on fixing the acute problem. These should be updated too but we can clean this up as we go along.


chore(*): Revert for http2 CVE patch

This reverts commit ad916d29571ef66868d15755e91e471a73d0048a.

Summary
Revert commits to prepare for the 3.4.2 http2 CVE patch.

Checklist

[ ] The Pull Request has tests
[ ] A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
[ ] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

[Implement ...]

Issue reference

Fix #[issue number]

Wait for CI green


⚠ Bofrost.es has degraded performance

In a268eb3, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1522 ms

Resolved: Bofrost.es performance has improved in 2cf3e05 after 8 minutes.


Several Improvements

Proposed Changes:

Added: ComfyUI Device Manager to handle device management for Comfy.
Moved: Model location to /models/LLM.
Introduced: max_new_tokens, num_beams, do_sample, and fill_mask parameters to Florence2 node.
Updated: Logic to accurately draw masks from multiple polygons within the same image in the Florence2PostprocessAll Node.
Modified: apply method to handle polygons in the Florence2PostprocessAll Node when using REFERRING_EXPRESSION_SEGMENTATION.

Example:
These changes were made to improve performance and functionality based on recent discoveries:

Performance Optimization: Experimentation revealed that adjusting max_tokens not only improves processing speed at lower values but also allows for better mask quality when appropriately adjusted.
Enhanced Masking: Previously, only one eye was masked when searching for "both eyes".
Selective Masking: In Florence2Postprocess, users can now select either eye (left/right) using the index parameter.

Please review the changes and feel free to test them to ensure they meet your expectations. I believe these modifications align well with the intended improvements, and I welcome any feedback or further adjustments if needed.
A screenshot of my changes:

Thank you!

I like your version, it works very well, and I hope it can be merged. The only regret is that it made me download the model again. lol


Add some export fields

1. Why is this change necessary?
https://issues.shopware.com/issues/SW-24885
Now there are tests for the export! 💃
2. What does this change do, exactly?
3. Describe each step to reproduce the issue or behaviour.
4. Please link to the relevant issues (if any).
5. Which documentation changes (if any) need to be made because of this PR?
6. Checklist

[x] I have written tests and verified that they fail without my change
[x] I have squashed any insignificant commits
[x] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/SW-24937
Please use this issue to track the state of your pull request.

Thanks for your Contribution! :blue_heart:


Documentation: Investigate hierarchy issues in sidebar

Some interesting things going on in the sidebar navigation:
https://opencolorio.readthedocs.io/en/latest/concepts/_index.html#concepts
So here, on the side, only Overview and Publications show up, though seemingly Introduction, Internal Architecture Overview, and Glossary would be on the same level and therefore should show up.

Meeting of 2020/09/11, after further investigation, we have double-checked and it is normal that overview and publications are listed in the side bar as they are listed in the _index.rst and have a folder for each one. Therefore we could probably have a look at why "Introduction" and "Internal Architecture Overview" are listed on the same level.


the app's args string parser has a very serious bug

Checklist

[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log
PS C:\Users\Administrator> youtube-dl.exe https://youtu.be/VyWAvY2CF9c -v
[debug] System config: []
[debug] User config: ['--ignore-errors', '--continue', '--no-overwrites', '--download-archive', '-f', 'bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best', '--add-metadata', '--embed-thumbnail', '--external-downloader', 'aria2c', '--external-downloader-args', '-c -j 8 -x 8 -s 8 -k 1M', '-o', 'F:\youtube_videos\%(playlist_index)s-%(title)s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['https://youtu.be/VyWAvY2CF9c', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.22000
[debug] exe versions: ffmpeg 2022-03-31-git-e301a24fa1-full_build-www.gyan.dev, ffprobe 2022-03-31-git-e301a24fa1-full_build-www.gyan.dev
[debug] Proxy map: {}
ERROR: 'bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best" ) to search YouTube
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\generic.py", line 2364, in _real_extract
youtube_dl.utils.ExtractorError: 'bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:bestvideo[ext=mp4][width<2000][height<=1200]+bestaudio[ext=m4a]/bestvideo[ext=webm][width<2000][height<=1200]+bestaudio[ext=webm]/bestvideo[width<2000][height<=1200]+bestaudio/best[width<2000][height<=1200]/best" ) to search YouTube
[youtube] VyWAvY2CF9c: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[youtube] VyWAvY2CF9c: Thumbnail is already present
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://rr4---sn-hpa7kn76.googlevideo.com/videoplayback?expire=1649102219&ei=K_lKYsTxHMKm8gPenoTgAw&ip=<IP_ADDRESS>&id=o-AEr7efj_yVkvgdfm4bpREoJLllydFbdyNM83niPDTv2d&itag=248&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=fM&mm=31%2C26&mn=sn-hpa7kn76%2Csn-4g5ednds&ms=au%2Conr&mv=m&mvi=4&pl=24&initcwndbps=530000&spc=4ocVC-eoSq_BD6EZrNkOh5_uhPVP&vprv=1&mime=video%2Fwebm&ns=TojVDQad0x2ncsQyOWHrXtUG&gir=yes&clen=113996202&dur=5138.333&lmt=1596263790399043&mt=1649080155&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5432432&n=Bg4y8jbevD2H8p6gc&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cspc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgWbMbtNlw55-vt90uYUow26g4S6S_-PAsAKk_4Cy2jAoCIQDTMsqP-jYZ6cKr-TkjH2GA-N4koI8kczPg-WFgNzJ2FA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhALVxUhUhjygIRx4R8Fbf-XrTJ1aZm7QtPT4QgDu2pf5XAiB2QHIrjd38uoBqaMbyRhig4cCZOar1fYCr4uRn9gJQGg%3D%3D'
[download] Destination: F:\youtube_videos\NA-Deep Learning Crash Course for Beginners.f248.webm
[debug] aria2c command line: aria2c -c -c -j 8 -x 8 -s 8 -k 1M --dir "F:\youtube_videos" --out "NA-Deep Learning Crash Course for Beginners.f248.webm.part" --header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3665.3 Safari/537.36" --header "Accept-Encoding: gzip, deflate" --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7" --header "Accept-Language: en-us,en;q=0.5" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8" "--check-certificate=true" "--remote-time=true" -- "https://rr4---sn-hpa7kn76.googlevideo.com/videoplayback?expire=1649102219&ei=K_lKYsTxHMKm8gPenoTgAw&ip=<IP_ADDRESS>&id=o-AEr7efj_yVkvgdfm4bpREoJLllydFbdyNM83niPDTv2d&itag=248&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=fM&mm=31%2C26&mn=sn-hpa7kn76%2Csn-4g5ednds&ms=au%2Conr&mv=m&mvi=4&pl=24&initcwndbps=530000&spc=4ocVC-eoSq_BD6EZrNkOh5_uhPVP&vprv=1&mime=video%2Fwebm&ns=TojVDQad0x2ncsQyOWHrXtUG&gir=yes&clen=113996202&dur=5138.333&lmt=1596263790399043&mt=1649080155&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5432432&n=Bg4y8jbevD2H8p6gc&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cspc%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgWbMbtNlw55-vt90uYUow26g4S6S_-PAsAKk_4Cy2jAoCIQDTMsqP-jYZ6cKr-TkjH2GA-N4koI8kczPg-WFgNzJ2FA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhALVxUhUhjygIRx4R8Fbf-XrTJ1aZm7QtPT4QgDu2pf5XAiB2QHIrjd38uoBqaMbyRhig4cCZOar1fYCr4uRn9gJQGg%3D%3D"

## Description

this bug has been around since I've been using youtube-dl. the problem is this: when I write the list of arguments, for some reasons if they are randomly (not logically significant) at a place in line that the app can't parse them I get error. for example this config at %APPDATA%\youtube-dl\config.txt has no error: https://pastebin.com/DBxFUjsH -- but if I so much as change a line disaster happens take a look and test for yourselves: https://pastebin.com/4FNHrTyt.
keep in mind this is not an issue with the os but the youtube-dl's buggy args parser. I've also had this problem on variety of linux-distors such as Ubuntu, OpenSuSe, Fedora and so on...
again it's obvious that the args parser is buggy it can't parse the passed strings correctly. in this example (among many) the -f option is processed independent of its following options. but if the order of the arguments are changed through the process of trial and error the bug disappears

--download-archive needs a filename.

I get what you're getting at, That was obviously a mistake on my part. But experiment with it (besides --download-archive). I've had this problem for a very long time. I'm sure there's something to it.
Besides, technically the app itself should point these things out, not people's comments.
Think about it. It's simply not robust enough. I'm saying this because I too am a programmer and this behavior is very new and odd to me

I get what you're getting at as well. But remember that youtube-dl started as a hobby project and is in hard-maintenance mode for quite a while now. "PRs to improve the config parser are welcome" yada yada I guess.
p.s.: don't forget to delete all the leftover -f files the program have been generating with the wrong config args.

May I make commits to the project. I'm pretty good

You may submit Pull Requests, yes. If they get merged eventually depends on many things.

Unfortunately -f is a valid, if unusual, filename. If you don't want that to be the name of your download archive file, better put the one you want.
Arguably a GNU-style CLI with --arg_name=arg_value could be less prone to such confusion, but then Windows users would have to write "--arg_name=arg_value". Plainly the CLI syntax is not going to change now.

What is true is that a config file is not separately parsed but just replaced by the arguments in the file.
You might hope, and the examples in the manual support the idea, that each line should contain a full set of options and values, where applicable.
In case people rely on that not being so, a keen PR author could propose a --strict-config option that, when used in a config file, would cause each further line in the file to be validated as full set of options and values, where applicable (ie, if --arg_name needs a value, then it should be followed by its value on the same line); it might also restrict each line to at most one option. This would, as far as I can see, mean improving the parser in _readOptions(): perhaps read each line, apply compat_shlex_split() (allow a \ at the end of the line to continue the line?), validate the line; return the accumulated results.

You don't like to read the part of  "working with strings" in programming books, do you? plz tell me you didn't learn programming from youtube too.
FINE I APOLOGIZE AND I'LL FIX IT FOR YOU BUT NOT TODAY. I'm not giving up 10k fixing a qemu bug request from IBM on the account of your ******* fill the blank using strings.


Correctly handle new multiline secrets

This commit fixes as small issue in how multi-line secrets are handled. Before they were always written in to the secret body completly ignoring the first line that contains the password. Now we do respect that correctly. To implement that properly we need to have some additional code to satisfy the io.Writer assumptions around the AKV secret type.
Also this fixes some non-hermetic tests that showed up during testing of this change.
Fixes #2614

I did try the simpler approach first, indeed. It had side-effects for some of our tests. Unfortunately. I'll address the comments later.


[Question] Using gotty as a library/using gotty server bits only

I was wondering if it's okay to use gotty as a library to write my own terminal-over-websocket server application.
Essentially I'd like to extend the functionality (client specifies what commands to run, cross-origin connections etc.) and was wondering if gotty is intended to be used as a library or might have frequent breaking changes in the contact between the client and the server (gotty subprotocol, auth methods etc)
Currently the code seems to be tailored to simply the gotty client, is this intention or would be open to extending the gotty project to support it's use a library for things other than the gotty client?

I think this duplicates #128.
Some code refactoring is in (slowly) progress to make gotty as  a library at the wip branch.

@yudai Hi, I've left a comment on that PR. Would you like to close this one and continue discussion on that one? I've proposed a scope increase, but I'd be happy to contribute to the effort.

@ffledgling Let's close this issue and keep discussion at #128


Multiple processors issue

Hello everyone,
I'm new to using PeleLMeX and I'm trying to simulate the flamesheet model using multiple processors. However, I've noticed that some of the files generated during the simulation have names like 'plt00000.old.0782200/', which is causing issues when I try to visualize the results using visIt.
Does anyone have any advice or best practices for avoiding these kinds of issues when working with multiple processors? Any guidance you can offer would be greatly appreciated.
Thank you!

Those are old plot files that are renamed rather than removed.  If you re-run a time step and don't want to see those, you'll need to manually remove them yourself prior to the run.  This is done to avoid the code potentially overwriting valuable data, but can seem annoying.  The same thing is done with checkpoint files.

Thank you for responding!
In my situation, the plt files are not old as I have already removed them. However, when I run the command line with mpirun or mpiexec and specify the number of processors, I notice that for each step, I have n-1 old files saved. I suspect that there is an output generated for each processor, which is causing the issue I am experiencing.

Ahh, did you build with USE_MPI=TRUE?  If not, then you are running a bunch of serial codes at once and overwriting files on the fly.  Your executable should have .MPI in its name.

Yes, I built the executable with USE_MPI=TRUE

It is also possible that the mpi wrapper used to compile the code is not from the same library as the mpirun used for the simulation. Which OS are you running on ?
The standard PeleLMeX log should tell you how many MPI ranks are employed during the AMReX initialization:
MPI initialized with 4 MPI processes

I am running ubuntu22.04.2.LTS OS
However, I can't seem to determine whether AMREX is initialized with 4 processors.
How can I verify this step?
Thank you in advance.

It should be the first line of PeleLMeX stdout. The first few lines look like:
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing SUNDIALS with 1 threads...
SUNDIALS initialized.
AMReX (23.04-8-gbacaa10a7636) initialized
Successfully read inputs file ...

 ================= Build infos =================
 PeleLMeX    git hash: v23.03-16-g9fbe4ef-dirty
 AMReX       git hash: 23.04-8-gbacaa10a7
 PelePhysics git hash: v23.03-9-gb04c8ee8
 AMReX-Hydro git hash: 38ad308
 ===============================================

Since you're on linux, you can check which MPI library the PeleLMeX exec is linked to using ldd PeleLMeX*.exe, and which one your mpirun command point to using which and ls -l.

So you are definitely running multiple serial PeleLMeX simulations instead of a single parallel one. Did you try using mpiexec instead of mpirun ?

I tried both mpirun and mpiexec and i had the same result!

Did you install both open-mpi and mpich at any point on your machine ?

They were already installed in the usr directory!

Are you able to explicitly use one or the other by overwriting/reinstalling the library ? My best guess is still that the default mpirun is not from the same library as the one you used by the compiler MPI wrapper.

@bazharz did you manage to get an MPI run going ?

Yes it works now!
Thank you

I'll close the issue then.


Container With Most Water

Given n non-negative integers a_1, a_2, ..., a_n where each represents a point at coordinate (i, a_i) . ‘ n ‘ vertical lines are drawn such that the two endpoints of line i is at (i, a_i)and (i, 0).
Find two lines, which together with x-axis form a container, such that the container contains the most water.

nice