Rak/infer fixes

Several upgrades to infer:

#30 vm cmd line argument --output-type to toggle between labels (black background, labels are GV-->3X smaller size; saves with '[image_id]_labels.tif') and overlay (raw image background, labels are RGB). To get both labels and overlay, need to run infer twice--overlays will not overwrite labels, and metadata is timestamped. Default is overlay.
#50 vm cmd line argument --image-ids to specify comma-separated subset of images to infer. Default is all images.
vm cmd line argument --background-class to skip the labeling of background if the given model was trained with background as a class. Default is None, aka no background class assumed.

Upgrade to models.py:

#54 default loss is always binary ce now (cat ce was essentially breaking model training and thus infer)

Infer validation below.

Verification of image subset inference:

Verification of vm cmd line arguments:
Case 1: overlay with background class (note overlay is default, so no need to enter it actually)
python3 infer_segmentation.py --gcp-bucket gs://necstlab-sandbox --model-id segmentation-model-esrf16_segV1_study1_CropCase2_TrnValCaseB_BackgClass_BinCE_20200215T193828Z --stack-id 8bit_AS4_S2_P1_L6_2560_1750_2160 --image-ids 8bit_AS4_S2_P1_L6_2560_1750_2160-0123.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-2089.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-1060.tif --output-type overlay

Case 2: overlay without background class
python3 infer_segmentation.py --gcp-bucket gs://necstlab-sandbox --model-id segmentation-model-esrf16_segV1_study1_CropCase2_TrnValCaseB_BackgClass_BinCE_20200215T193828Z --stack-id 8bit_AS4_S2_P1_L6_2560_1750_2160 --image-ids 8bit_AS4_S2_P1_L6_2560_1750_2160-0123.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-2089.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-1060.tif --output-type overlay

Case 3: labels with background class. GVs scale over full 8 bit GV range.
python3 infer_segmentation.py --gcp-bucket gs://necstlab-sandbox --model-id segmentation-model-esrf16_segV1_study1_CropCase2_TrnValCaseB_BackgClass_BinCE_20200215T193828Z --stack-id 8bit_AS4_S2_P1_L6_2560_1750_2160 --image-ids 8bit_AS4_S2_P1_L6_2560_1750_2160-0123.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-2089.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-1060.tif --output-type labels

Case 4: labels without background class. GVs scale over full 8 bit GV range.
python3 infer_segmentation.py --gcp-bucket gs://necstlab-sandbox --model-id segmentation-model-esrf16_segV1_study1_CropCase2_TrnValCaseB_BackgClass_BinCE_20200215T193828Z --stack-id 8bit_AS4_S2_P1_L6_2560_1750_2160 --image-ids 8bit_AS4_S2_P1_L6_2560_1750_2160-0123.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-2089.tif,8bit_AS4_S2_P1_L6_2560_1750_2160-1060.tif --output-type labels   --background-class 0

Upon review, we may close issues #30, #50, and #54

UPDATE: MAKE NON-BACKG COLORS INDEP OF WHETHER BACKG IS CLASS DURING OVERLAY, SINCE WE WOULD NEVER WANT TO OVERLAY BACKGROUND PRACTICALLY SPEAKING.
EXAMPLE: 3 dmg classes + background class but now with dmg having same colors as models where background is not a class. this requires setting the --background-class 0 parameter.

this looks great -- minor suggestion is to change the naming of --background-class 0 to --background-class-index 0 or change the expected values to strings (e.g., `--background-class "class_0") so it's clear to a user if the value should be an int (representing the index) or a string (representing the class label). I'd vote for the latter but am flexible


feat: report duration for multi month sub gift

Similar to #2026 but for subgift (and these irc tags are actually documented)
Before

After

can you put one of these in src/util/SampleData.cpp so it can be tested with the respective fake-sub-messages button (or the command)

actually the system message is good when the gift is not anonymous, will adjust my patch
@badge-info=subscriber/32;badges=subscriber/3030,sub-gift-leader/2;color=#FF8EA3;display-name=iNatsuFN;emotes=;flags=;id=0d0decbd-b8f4-4e83-9e18-eca9cab69153;login=inatsufn;mod=0;msg-id=subgift;msg-param-gift-months=6;msg-param-goal-contribution-type=SUBS;msg-param-goal-current-contributions=881;msg-param-goal-target-contributions=900;msg-param-goal-user-contributions=1;msg-param-months=16;msg-param-origin-id=2524053421157386961;msg-param-recipient-display-name=kimmi_tm;msg-param-recipient-id=225806893;msg-param-recipient-user-name=kimmi_tm;msg-param-sender-count=334;msg-param-sub-plan-name=Channel\sSubscription\s(mxddy);msg-param-sub-plan=1000;room-id=210915729;subscriber=1;system-msg=iNatsuFN\sgifted\s6\smonths\sof\sTier\s1\sto\skimmi_tm.\sThey've\sgifted\s334\smonths\sin\sthe\schannel!;tmi-sent-ts=1712034497332;user-id=218205938;user-type=;vip=0 :tmi.twitch.tv USERNOTICE #mxddy

actually the system message is good when the gift is not anonymous, will adjust my patch

hm, not quite; observed a broadcaster gift a 12 month sub and the system-msg didn't include the number of months. may be related to msg-param-sender-count=0 (twitch doesn't appear to count the total number of subs gifted by the broadcaster) so They've gifted X months in the channel can't be populated (see quoted example above) => could adjust patch again to workaround this (or we can ignore this edge case)

@user-id=broadcasterId;msg-param-origin-id=2862055070165643340;display-name=broadcasterLogin;id=eeb3cdb8-337c-413a-9521-3a884ff78754;msg-param-gift-months=12;msg-param-sub-plan=1000;vip=0;emotes=;badges=broadcaster/1,subscriber/3042,partner/1;msg-param-recipient-user-name=ogprodigy;msg-param-recipient-id=53888434;badge-info=subscriber/71;room-id=broadcasterId;msg-param-recipient-display-name=OGprodigy;msg-param-sub-plan-name=Silver\sPackage;subscriber=1;system-msg=broadcasterLogin\sgifted\sa\sTier\s1\ssub\sto\sOGprodigy!;login=broadcasterLogin;msg-param-sender-count=0;user-type=;mod=0;flags=;rm-received-ts=1712803947891;color=#EB078D;msg-param-months=15;tmi-sent-ts=1712803947773;msg-id=subgift :tmi.twitch.tv USERNOTICE #broadcasterLogin

(removed broadcaster info)


Add file extension to import

This fix correctly resolves the import for ESM projects.
See https://nodejs.org/docs/latest-v14.x/api/esm.html#esm_mandatory_file_extensions

@dburles could you add a changeset?

Done


initial RPC implementation

RPC implemented over zmq sockets.

remote procedure calls can be synchronous, asynchronous, or no-return
exceptions propagate nicely back to caller
no specific event loop requirements
client socket uses zmq.ROUTER, which allows us to manage multiple rpc servers from a single client socket.

OK.
Need to implement the multiple client way later on.


Fixed deprecated selectors

I just pushed your merged as a tag 0.2.0.
Is this enough for atom package manager to detect the new version ?
Also do you want to be added as a collaborator (I am not using this theme for now) ?

I think you have to publish the package via apm (btw I'm not sure if it matters, but the version in package.json is still 0.1.0).
And yes, you can add me as a collaborator. I may not be really reactive but I can help ;)

0.2.0 has been published.
You have been added as a collab.


showInNetron function dosn't use custom netron port set as enviroment variable.

This requires a few changes to the function itself here: https://github.com/Xilinx/finn/blob/bdfbd4b79088accf92e60fc1fe790e697500dfe7/src/finn/util/visualization.py#L39
And a way to get the port number itself, likely similar to this: https://github.com/Xilinx/finn-base/blob/c928353369f9f4c76e6bbd5c2d0f68c20763c44b/src/finn/util/basic.py#L106

Resolved in #366


Fix debug and run locally after regenerating with controller-gen

What does this PR do?
This PR fixes debug and run locally after regenerating with controller-gen.
I've moved these changes out from https://github.com/devfile/devworkspace-operator/pull/193 since here is something to discuss:

without .ONESHELL env vars are not properly propagated to the dvl and eventually to controller. So, we should enable .ONSHELL or alernative - create script that will set env vars and run dvl or controller directly.
I've figured out that we modify default service account in namespace instead of created devworkspace specific one. I think that it's better to have a dedicated SA (if multiple operators are deployed in the same namespace, they may conflict due default roles modifications) but I did not manage to archive it properly with all deployment,roles,rolesbinding patching via kustomize. It can be solved independently from this PR.

What issues does this PR fix or reference?
N/A
Is it tested? How?
make deploy
oc patch deployment/devworkspace-controller-manager --patch "{\"spec\":{\"replicas\":0}}" -n devworkspace-controller
make debug
#or make run
#make sure everything works

:warning: that's not well tested after the latest changes, but it'll be done before merge

/test v5-devworkspaces-operator-e2e

I continued to pump up this PR, so @amisevsk please review again. The most important new change https://github.com/devfile/devworkspace-operator/pull/194/commits/7f19b2531b55e3e29a364d587d7e07aebc3a1427

/test v5-devworkspaces-operator-e2e

e2e is failing with error: kubectl is not found.

From https://docs.openshift.com/container-platform/3.10/cli_reference/differences_oc_kubectl.html#differences-oc-kubectl-using-kubectl it's safe to use oc instead of kubectl. I've checked and the difference we used to have in Makefile for oc/kubectl is not needed anymore(like for make uninstall and make restart). So, to easily fix e2e tests and avoid some users issues - now Makefile detects which k8s cli is available and use it or report an issue.

/test v5-devworkspaces-operator-e2e

Tests are failing with
+ make test_e2e
which: no kubectl in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin:/tmp/bin:/go/src/github.com/devfile/devworkspace-operator:/go/bin)
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin:/tmp/bin:/go/src/github.com/devfile/devworkspace-operator:/go/bin)
which: no kustomize in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin:/tmp/bin:/go/src/github.com/devfile/devworkspace-operator:/go/bin)
make: *** [controller-gen] Error 1

Prow instance faces: go get: disabled by -mod=vendor due:
sh-4.2$ go env
GO111MODULE="on"
GOFLAGS="-mod=vendor"

If https://github.com/devfile/devworkspace-operator/pull/194/commits/8094ec939b875ca914058cd56070fa11a6a66873 will fix the issue - we need to find a proper solution for that.

/test v5-devworkspaces-operator-e2e

Next e2e test barier: Error patching crds: yq is required

/test v5-devworkspaces-operator-e2e

/test v5-devworkspaces-operator-e2e

/test v5-devworkspaces-operator-e2e

Merging since this PR is already pretty huge and no blockers found according to PR checks and reviewers approvals.
Minor stuff could be fixed separately.


Paste error with CKEDITOR : this.getChildrenContainer(...) is null

Hi,
a have a weird JS error when pasting anything (even a single character) at the EDGEs of the text.
No problem pasting INSIDE existing text.
I tried with differents versions of JQUERY and CKEDITOR. Always the same bug. I think having all required plugins for CK. An idea ?

Hey @LoFromCaumont, I got a similiar issue when I paste a text using "Ctrl + V". This happens when you combine these the plugins: plugins: ["grapesjs-mjml", "gjs-plugin-ckeditor"]. The text got blank then when you click outside of the component it triggers the error:

@artf, do you have any idea about it?

Hey, happy to not be alone !
But no,
i didn't worked on this project since 6 months.
I suppose you tried with the lastest versions of Jquery, CK, Grapes... ?

I was running ckeditor 4.7, I switched to 4.6.2 (Standard) and this fixed the issue.

Yeah !
I'll try this change ...one day ;-(
Results in next post...
Thanks you

can anyone confirm this is still an issue?


Rancher keeps redirecting to port 443 despite "tls=external"

Rancher Server Setup

Rancher version: 2.6.0
Installation option (Docker install/Helm Chart):

Helm Chart on RKE2 cluster. RKE2 v1.21.5+rke2r1

Proxy/Cert Details:

Information about the Cluster

Kubernetes version: 1.21.5
Cluster Type (Local/Downstream):

Local

Describe the bug
Rancher keeps doing HTTPS redirect despite tls=external flag set in Helm chart and use-forwarded-headers=True in Nginx ingress controller configuration
To Reproduce

Deploy rancher on Kubernetes per instructions here.
Follow "tls=external" instructions here  to terminate SSL on an upstream proxy (HAProxy, in this case) whose backends are cluster nodes on 80/tcp.
Enable "use-forwarded-headers=true" on the Nginx ingress as explained here
Test that you can reach rancher directly on the cluster on 80/tcp using the "required" headers mentioned here.

 curl -H "Host:rancher.hostname.com" \
        -H "X-Forwarded-Proto:https" \
        -H "X-Forwarded-Port:443" \
        -H "X-Forwarded-For:<ip of client>" \
        -i <ip of worker node>

Result
The ingress redirects (302) to port 443. This causes loops when coming through the upstream proxy.
Expected Result
The ingress should not redirect to port 443.
Screenshots

Additional context
Helm command used to deploy rancher:
helm install rancher rancher-latest/rancher \
        --namespace cattle-system \
        --set hostname=rancher.hostname.com \
        --set tls=external`

Ingress describe command:
Name:             rancher
Namespace:        cattle-system
Address:          10.X.Y.15,10.X.Y.25,10.X.Y.27
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  infra-rke2-brno.kmglobalrd.com
                                     rancher:80 (<IP_ADDRESS>:80,<IP_ADDRESS>:80,<IP_ADDRESS>:80)
Annotations:                      field.cattle.io/publicEndpoints:
                                    [{"addresses":["10.X.Y.15","10.X.Y.25","10.X.Y.27"],"port":80,"protocol":"HTTP","serviceName":"cattle-system:rancher","ingress...
                                  meta.helm.sh/release-name: rancher
                                  meta.helm.sh/release-namespace: cattle-system
                                  nginx.ingress.kubernetes.io/proxy-connect-timeout: 30
                                  nginx.ingress.kubernetes.io/proxy-read-timeout: 1800
                                  nginx.ingress.kubernetes.io/proxy-send-timeout: 1800
                                  nginx.ingress.kubernetes.io/ssl-redirect: false
                                  nginx.ingress.kubernetes.io/use-forwarded-headers: true

I am running into this same issue with a fresh install of Rancher 2.6.2 with k3s v1.21.5+k3s2. I am new to k3s, but it appears to be using traefik instead of nginx, and it is having the same issue. I see the same behavior above testing with curl as well.

Hi @davemuench , I installed same yours(k3s + traefik ingress controller + rancher helm stable version) and success to access 80 port with no redirect to 443 port.
plz check below my steps

configure your LB

you should config the X-Forwarded-Proto header for LB.

install k3s

$ curl -sfL https://get.k3s.io | sh -s - server --docker \
 --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name"

install helm

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

install rancher-stable using helm(with TLS Termination options)

$ helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
$ kubectl create namespace cattle-system
$ helm install rancher rancher-stable/rancher \
  --namespace cattle-system \
  --set hostname=your-lb-domain \
  --set replicas=3 \
  --set tls=external

edit k3s traefik manifest file
add  valuesContent.additionalArguments into /var/lib/rancher/k3s/server/manifests/traefik.yaml file.

  valuesContent: |-
    additionalArguments:
      - "--entryPoints.web.proxyProtocol.insecure"
      - "--entryPoints.web.forwardedHeaders.insecure"

when  you save the file, the traefik pod will redeploy.

test with curl

$ curl https://your-lb-domain -H "X-Forwarded-Proto: https" -v

SKIP...

> Host: your-lb-domain
> User-Agent: curl/7.58.0
> Accept: */*
> X-Forwarded-Proto: https
>
< HTTP/1.1 200 OK
< Content-Type: application/json

... SKIP

edit k3s traefik manifest file
add  valuesContent.additionalArguments into /var/lib/rancher/k3s/server/manifests/traefik.yaml file.

  valuesContent: |-
    additionalArguments:
      - "--entryPoints.web.proxyProtocol.insecure"
      - "--entryPoints.web.forwardedHeaders.insecure"

This fixes the problem for me, but occasionally the change gets erased. Is Rancher auto-updating Traefik or its chart values?

ingress-nginx
helm upgrade --install
ingress-nginx ingress-nginx/ingress-nginx
--namespace ingress-nginx
--set controller.service.type=LoadBalancer
--set controller.service.annotations."service.beta.kubernetes.io/aws-load-balancer-ssl-cert"=arn:aws:acm:xxx
--set controller.service.annotations."service.beta.kubernetes.io/aws-load-balancer-backend-protocol"=http
--set controller.service.annotations."service.beta.kubernetes.io/aws-load-balancer-ssl-ports"=https
--set controller.service.targetPorts.https=http
--set-string controller.config.use-forwarded-headers="true"
--version 3.12.0
--create-namespace

rancher
helm install rancher rancher-stable/rancher   --namespace cattle-system   --set hostname=XXX.com   --set replicas=3 --set tls=external

Here is my solution.
As most people already found, missing use-forwarded-headers="true"  for the nginx-ingress is the culprit. My main problem was how to set it. It probably depends  on which kube you run - k3s, rke or rke2. Documentation tells about adding this setting to cluster.yml.  What, how, huh?  I couldn't find anything about cluster.yaml for the rke2 so finally I realized that it must be for k3s or something else. Anyway in RKE2 documentation (https://docs.rke2.io/networking/) I found that for RKE2 all we need to do is:
create /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml :
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-ingress-nginx
namespace: kube-system
spec:
valuesContent: |-
controller:
config:
use-forwarded-headers: "true"
If you do this before installing rke2 then of course you need to create that directory first, dump in the file and proceed with installation of rke2 and rancher
But you can do this at any moment, even after installing rke2 and rancher.   (don't forget about --set tls=external when you do helm install rancher)
I was able to confirm that the “use-forwarded-headers” has been added (or not) by running:
kubectl get configmap rke2-ingress-nginx-controller -n kube-system -o yaml
…
data:
allow-snippet-annotations: "false"
use-forwarded-headers: "true"
The second line will literally pop up in there few seconds after dropping that /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml  in. There is a process which monitors that directory and when there is a new manifest it automatically does apply on it.
This made my proxy setup for rancher (with ssl termination) work right away.
It also works if you just edit the config map instead of adding the file:
kubectl edit configmap rke2-ingress-nginx-controller -n kube-system   and adding that line manually but I am not sure if it would stick during cluster upgrades.
The nginx.conf for the proxy which terminates SSL was fine the way it was suggested in the documentation.
One more problem I had was “An error occurred logging in. Please try again.”   right after providing initial password for the rancher.
This again has something to do with proxy and ports. I was trying to set my proxy on port 4444 (outside port) but apparently it didn’t like it. Changing my outside port to 443 solved the issue.
I didn’t have any enough will power left in my to find out why I can’t use other port :(
Hope this helps

This fixes the problem for me, but occasionally the change gets erased. Is Rancher auto-updating Traefik or its chart values?

I added a file
/var/lib/rancher/k3s/server/manifests/traefik-config.yaml
with this inside problem solved and the pods were redeployed  automatically
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
additionalArguments:
- "--entryPoints.web.proxyProtocol.insecure"
- "--entryPoints.web.forwardedHeaders.insecure"

Still Facing the same issue
http call is redirecting to https
even after adding the annotations in ingress
nginx.ingress.kubernetes.io/ssl-redirect: false
nginx.ingress.kubernetes.io/use-forwarded-headers: true


linters: enable errcheck

/assign @geeknoid
Signed-off-by: Kuat Yessenov<EMAIL_ADDRESS>

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: kyessenov
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: geeknoid
If they are not already assigned, you can assign the PR to them by writing /assign @geeknoid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kyessenov: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3976d17e0c88d600c9ad5643090bae6c4ff1c728
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-simpleTests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-pilot-multicluster-e2e

prow/e2e_pilotv2_auth_sds.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio_auth_sds_e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-unit-tests

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-dashboard.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-pilot-multicluster-e2e

prow/e2e_pilotv2_auth_sds.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio_auth_sds_e2e

prow/e2e-mixer-no_auth.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-mixer-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/approve
/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: geeknoid, kyessenov
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [geeknoid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/retest

/retest

/retest

Try try again...
/retest

/test istio-integ-k8s-tests
/test e2e-dashboard
/test e2e-bookInfoTests-envoyv2-v1alpha3
/test e2e-simpleTests
/test istio-pilot-multicluster-e2e
/test istio_auth_sds_e2e
/test e2e-mixer-no_auth

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-integ-k8s-tests.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-integ-k8s-tests

prow/e2e-bookInfoTests-v1alpha3.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/e2e_pilotv2_auth_sds.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio_auth_sds_e2e

prow/e2e-mixer-no_auth.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-mixer-no_auth

prow/istio-pilot-multicluster-e2e.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyessenov: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-trustdomain.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test e2e-bookInfoTests-trustdomain

prow/istio-pilot-multicluster-e2e.sh
04342793642c73181db81d88e95d92673cce4c5d
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


DEPRECATION WARNING: alias_method_chain is deprecated

DEPRECATION WARNING: alias_method_chain is deprecated. Please, use Module#prepend instead. From module, you can access the original method using super. (called from require at ~/.rvm/gems/ruby-2.3.0/bundler/gems/devise_token_auth-129025b080aa/lib/devise_token_auth.rb:1)

Did you ever figure this out?

Same here, getting this message as soon as I try to run the migrations:

DEPRECATION
WARNING: alias_method_chain is deprecated. Please, use Module#prepend instead. From module, you can access the original method using super. (called from <top (required)> at /my-app/config/application.rb:17)
[DEPRECATION] last_comment is deprecated.  Please use last_description instead.
rails aborted!

another closed ticket indicates that we should add manually the dependency to devise, but even this doesn't change the situation...
Assistance would be much appreciated :)

OK, my bad, I've had it working eventually :)
doing this:

gem "omniauth"
gem 'devise_token_auth', :github => 'lynndylanhurley/devise_token_auth', :branch => 'master'

I still have a deprecation warning but otherwise it works 👍

Hi.
I have the same problem that Fudoshiki but not corrected with the change in the gem version.
To compile on test mode works but not for production.
Can someone help me?


Create nightly builds

Description
This creates nightly builds of Flow Go and creates an issue if the version breaks the emulator.

For contributor use:

[ ] Targeted PR against master branch
[ ] Linked to GitHub issue with discussion and accepted design OR link to spec that describes this work
[ ] Code follows the standards mentioned here
[ ] Updated relevant documentation
[ ] Re-reviewed Files changed in the GitHub PR explorer
[ ] Added appropriate labels

Codecov Report

Merging #380 (0c09e01) into master (3c8ca15) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #380   +/-   ##
=======================================
  Coverage   46.98%   46.98%
=======================================
  Files          26       26
  Lines        3418     3418
=======================================
  Hits         1606     1606
  Misses       1688     1688
  Partials      124      124

Flag
Coverage Δ

unittests
46.98% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

oh this is very nice, but can flow-go break emulator? I think emulator can break flow-go but flow-go tests are dependent on emulator,

oh this is very nice, but can flow-go break emulator? I think emulator can break flow-go but flow-go tests are dependent on emulator,

Flow-go can break the emulator by introducing braking changes to the APIs, and the idea is for the flow-go team to be aware of that. I'm not sure in what sense the emulator is used in flow-go tbh.
@bluesign if you can review it would help 🙏 also this was requested by flow-go team anyway.


DO NOT MERGE! Fast Forward stable to 6f8124be3ef18e7ab687b19bb8daeabd7890bad7.

This PR will be merged automatically once checks are successful.

This change is

Jenkins job istio/mixer-pr-stable passed

Jenkins job mixer/pr-stable passed


epic2 with bedpe input issue

Hello!
I am trying to call peaks from H3K27me3 ChIPseq data, I converted the bams to bedpe as it is suggested for paired-end datasets. I get this error message:
"terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
Aborted"
command used: epic2 --treatment t.bedpe --control c.bedpe --chromsizes hg38_sizes.txt -fdr 0.01 --output mydir/file
Unfortunately, I cannot share the data, could you please help me with this?
Thanks in advance.
Yussuf

Without the data it'd be too hard. Could you try to anonymize the data in some way or create a minimal reproducible example?
Does it happen on some of your non-confidential data?

Hello,
Thanks for the reply, I created data for chr1 only, both conditions control and treatment, please find them through this link : https://drive.google.com/drive/u/0/folders/1Mm0q1xKIQfKttNpEG5k5Z79JILqtghOA

Hi Endre,
Hope you good.
The error message only appears with the bedpe input, I used epic2 on my bam files and it works, but again not the suggested file format for Paired end data.
The same error message appear when I use your provided examples, and I really cannot understand the problem, and I hope you could help. Thanks in advance.
Yussuf

Oh sorry, are you facing a problem accessing the google drive link I provided above?

Oh, sorry I did not see that link. In the first message you said you could
not share the data.
I'll look at it next week :D
On Fri, Jul 1, 2022 at 3:31 PM yussufhajjaj @.***>
wrote:

Oh sorry, are you facing a problem accessing the google drive link I
provided above?
—
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/epic2/issues/61#issuecomment-1172353393,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHURUWLF62LWOGDIT756A3VR3XL5ANCNFSM5YGCB3YQ
.
You are receiving this because you commented.Message ID:
@.***>

I'll need a linux machine to debug it. I'll have to put this issue on hold
until I get access to a server with linux.
However, you could turn the bedpe into a bed where the interval is the
midpoint between the two mate pairs and it would work with the regular epic2
On Fri, Jul 1, 2022 at 5:14 PM Endre Bakken Stovner @.***>
wrote:

Oh, sorry I did not see that link. In the first message you said you could
not share the data.
I'll look at it next week :D
On Fri, Jul 1, 2022 at 3:31 PM yussufhajjaj @.***>
wrote:

Oh sorry, are you facing a problem accessing the google drive link I
provided above?
—
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/epic2/issues/61#issuecomment-1172353393,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHURUWLF62LWOGDIT756A3VR3XL5ANCNFSM5YGCB3YQ
.
You are receiving this because you commented.Message ID:
@.***>

Thanks for checking. Okay, I will be waiting for that.
But is that the format you mean or what? Or I miss understood sth,
from this:
chr9    123172513       123172563       chr9    123172652       123172698       39V34V1:121:C3V68ACXX:4:1101:1247:9184  42      +  -
To this:
chr9 123172563 123172652 39V34V1:121:C3V68ACXX:4:1101:1247:9184  42  +

You can find the midpoint between the start and the end.
start + (end - start)/2
123172513 + (123172698 - 123172513)/2 = 123 172 605
So your interval is 123 172 605, 123 172 606
On Tue, Jul 5, 2022 at 5:29 PM yussufhajjaj @.***>
wrote:

Thanks for checking. Okay, I will be waiting for that.
But is that the format you mean or what? Or I miss understood sth,
from this:
chr9 123172513 123172563 chr9 123172652 123172698
39V34V1:121:C3V68ACXX:4:1101:1247:9184 42 + -
To this:
chr9 123172563 123172652 39V34V1:121:C3V68ACXX:4:1101:1247:9184 42 +
—
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/epic2/issues/61#issuecomment-1175194564,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHURUSSZKUZHZO742TQE7LVSRIEDANCNFSM5YGCB3YQ
.
You are receiving this because you commented.Message ID:
@.***>

I do not know, unfortunately. I will look into it when I get a linux
machine or access to a linux server :D
On Tue, Jul 12, 2022 at 4:22 PM yussufhajjaj @.***>
wrote:

Even with a .bed file, using the midpoint, the same error message appear
"terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted"
if I use the paired end bam, and put the --guess-bampe it says that it is
an unused argument.
Mind that I installed epic2 from bioconda, maybe there is a version issue?
—
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/epic2/issues/61#issuecomment-1181823465,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHURUTQ4YLZRNAHCF3BSC3VTV5SBANCNFSM5YGCB3YQ
.
You are receiving this because you commented.Message ID:
@.***>

Hi there!
Has this issue being solved? It's happening to me.
Thanks for your help!
Roger

i need data that reproduces the bug ☺ Hard to debug otherwise
On Sun, 10 Sep 2023 at 03:23, rogeroria @.***> wrote:

Hi there!
Has this issue being solved? It's happening to me.
Thanks for your help!
Roger
—
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/epic2/issues/61#issuecomment-1712678557,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEHURURE3IVYJTQRRXVXMUDXZUJBFANCNFSM5YGCB3YQ
.
You are receiving this because you commented.Message ID:
@.***>

Hi!
Thank you for your quick response. I just sent you through WeTransfer two files (one of them used as a treatment and another as a control). Let me know if you can download them. (you should get an email in your gmail address).
Thank you again!
Roger

Hi Endrebak,
Where you able to download the data?
Roger


Fix bad link to pilot infrastructure page

ah thanks for spotting this! we just updated the pilot docs today :-)

ah thanks for spotting this! we just updated the pilot docs today :-)


Dummy payment button is not working

Summary
As an unlogged user I want to use dummy payment ex. on
https://react-storefront-e2e.vercel.app/checkout/?checkout=Q2hlY2tvdXQ6NmNiNTk2YzctMGFiMi00MzkxLTg2YjktMTg2YTUzMGEyODJl&locale=en-US&channel=default-channel&saleorApiUrl=https%3A%2F%2Fstorefront-automation.staging.saleor.cloud%2Fgraphql%2F&domain=storefront-automation.staging.saleor.cloud
Environment , when I click on dummy button nothing happened. Because of this the test should buy a product as anonymous SRS_1001 is failing
Steps to reproduce

As a unlogged user go to checkout
Fill up all required fields
Try to click on dummy button

Current result: Nothing happened after clicking
A link to a reproduction repository
No response
Output from envinfo
No response

I've investigated, and I confirm there's a bug. Hopefully, it'll be fixed via https://github.com/saleor/react-storefront/pull/679

hi @mmiszy , I have met a bug related to above button. When I click to choose Dummy provider (or maybe Dummy method), I saw that setPaymentData function was trigger (It set payment method and payment provider to "Dummy" value).  But when I click pay button, the payment method and payment provider still be null.


Publish Docs to The NPM Registry

This is ready for review.

This seems to be broken.
Even though publishDocs phase succeeded https://github.com/zio/zio-sql/actions/runs/3652024488/jobs/6169929786 , the state in docs does not represent the content from .md files.
https://zio.github.io/zio-sql/docs/overview/overview_index
vs
https://github.com/zio/zio-sql/blob/master/docs/index.md
I am unsure how this works, @khajavi would you please take a look when you have a chance?


feat: When uploading, include record information

When uploading, include record information
Type of Change

[x] New feature
[ ] Bug fix
[ ] Documentation update
[ ] Example update
[ ] Refactoring
[ ] Hotfix
[ ] Security patch
[ ] UI/UX improvement

Issue
[Linked issues]
Description
When uploading, include record information, such as the resource ID.
This is because it can be utilized to generate a URL when uploading a file.

Hello
Thank you for the PR. While the feature will be working when editing a record, this will not work on a record creation as there is no id. I guess we can still have this merged with a precision in the doc about this.
Could you please run yarn changeset to generate a changeset ? Thank you

Thanks for the feedback!
I hope to put this project to good use as an admin in the future. Thanks for the great project!


Fixed the resulting queryset in the documentation.

Under the section https://docs.djangoproject.com/en/4.1/topics/db/models/#field-options, there a example of a snippet where the value of a field is changed and saved. But when querying it afterwards it shows the old field value together with the newer one.

This PR will resolve the said error.

Thanks for this patch, however the current version is correct. name is a primary key, that's why the new objects is created. Please check docs above the example.


Feature/documentation upload meteor autoform file

Closes #938
Proposed changes

Add 'manage' button to API Documentation tab
add 'Manage API Documentation modal
add documentationFileId to apiBackend schema
configure autoform for API Backend documentation modal
use yogiben:autoform-file for file upload field

This is a work in progress, as I am tracking an upstream bug with yogiben:autoform-file.

Closing, since changes have been merged into @elnzv's branch.


ClimSA 5.2. The ability to ingest historical data from other sources

ClimSA 5.2. The ability to ingest historical data from other sources, including global datasets (GHCN, International Surface Database).

Duplicate


Document macro options in a consistent way across all components

The following components do not include a _macro-options.md file:

~call-to-action~
error
field
fieldset
icons
~modal~
~promotional-banner~
~quote~
~text-indent~
~timeline~
~video~

I have struck through the components that I added separate _macro-options.md files for in #2126.
The remaining components don't have examples or documentation, which is why their macro options are not in separate files and some are documented in tables in their index.njk file instead (otherwise they wouldn't visible on the website).
But we might want to consider a way to pull through markdown files into the website?

Closing this ticket to investigate #3158.


Dockerized the development environment

i think we could add a Dockerized option for developement where everyone would have the same environment regardless of the OS

@BlinfoldKing I'm not sure if it's a good idea, especially in the cross-platform development. The more platform the app is tested the better it is so it shouldn't be dockerized. We can't provide a single docker image for all the platforms since I don't think it provides any kind of UI. IMO, Docker is better for server-side things.
That being said, if you want to give it a try feel free to do it. Maybe try the vagrant image or something. In any case, we need to provide different images for all the platforms (because of the UI restrictions).
cc @timabbott, @gnprice.

i see, thanks for your opinion

@BlinfoldKing let me know if you're willing to work on #145 which will certainly make the development process better.


Add more OData expression support to $filter

Right now, the $filter REST option supports the native OData operations:
http://docs.oasis-open.org/odata/odata/v4.01/cs01/part1-protocol/odata-v4.01-cs01-part1-protocol.html#sec_BuiltinFilterOperations
feedback from some early adopters is emerging in the sense that we should support also some of the native OData query functions:
http://docs.oasis-open.org/odata/odata/v4.01/cs01/part1-protocol/odata-v4.01-cs01-part1-protocol.html#sec_BuiltinQueryFunctions
we should see how many of those can be pushed to the database engine to make sure we can push as much as possible the computation effort to the database

Data API builder should output standard OData.
It would conform also to Microsoft Graph (supporting expands, navigation properties, grouping, etc...) and there are already standard libraries to interact with it: https://github.com/OData/odata.net

At the very least it should support: contains and ideally should also support startsWith and endsWith


Datamanagement

This is something that popped into my mind recently, especially now that our field is delving into world of WGS.
There is a field standard for this (both in English and Dutch but god knows why these two are somewhat different: https://www.vkgl.nl/nl/kwaliteit/formulieren-documenten-kwaliteit/7-veldnormen). However, these standards are quite vague because they speak of "intermediate" results and "end" results.
What counts as what? they make some mentions of what counts as "end results", but there is still some debate for what counts as "intermediate" and if it should be kept or not. The Dutch version also mentions raw data, while the English version does not.
Some other questions I still have:
how well protected should the data be kept?
how and which validation data should be kept?

Yes, I wouldn't mind doing this. Our lab has been "struggling" with data management, so I wouldn't mind delving further into the matter. We also discussed this last Friday and came to the conclusion that we as bioinformaticians should give a well informed advice regarding the retention of different types of patient data.


Broken link

I'm getting a 404 error when trying to access this page:
http://go.microsoft.com/fwlink/?LinkId=267637
Redirection page is the following:
https://docs.microsoft.com/en-us/previous-versions/orphan-topics/azure.100/dn338078(v=azure.100)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 48d01116-e0d9-f62d-1f10-3f2e13a2d2f1
Version Independent ID: e4c42205-428c-8487-265a-173f324e8d66
Content: SQL error codes - database connection error
Content Source: articles/sql-database/sql-database-develop-error-messages.md
Service: sql-database
GitHub Login: @stevestein
Microsoft Alias: sstein

@ruijarimba This will be fixed on the referenced PR. We will now close this issue. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


PlayerList always hidden

Hi,
Thanks for this usefull module. A must have.
After install minimal ui and try differents options for the playerlist, My playerlist become always hidden, I can't see it. I have to use a script like that each time to activate it :
if (ui.players.rendered)
{
ui.players.close();
}
else
{
ui.players.render(true);
}

I dont have any error (F12). I use the last stable version of your module and foundry . I have try to disable or unintall your module, no result. Thanks for your help.

Hi @CaosFR - on which vesion of Foundry are you on?

I guess you have this, no?

Hi,
This problem was solve when update to v10.


fix(nrdb): update to nrdb query to include more options

TBA

Codecov Report
Attention: Patch coverage is 0% with 26 lines in your changes are missing coverage. Please review.

Project coverage is 38.84%.  Consider uploading reports for the commit e04bbed to get more accurate results

Files
Patch %
Lines

pkg/nrdb/nrdb_query.go
0.00%
26 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1110      +/-   ##
==========================================
- Coverage   38.90%   38.84%   -0.07%
==========================================
  Files          86       86
  Lines        5590     5612      +22
==========================================
+ Hits         2175     2180       +5
- Misses       3249     3266      +17
  Partials      166      166

Flag
Coverage Δ

unit
38.84% <0.00%> (-0.07%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Truncate changed behavior: warn users

as per issue https://github.com/ansible/ansible/issues/36599 , truncate changed behavior, that by itself is bad, but the documentation is both outdated and warns users in a way that can be easily ignored
outdated because of this:
 truncate(s, length=255, killwords=False, end='...', leeway=None)

Please update the leeway=None to leeway=5, as it is now the new default
the warning, it is in the end, after the example, and that can be easily ignored. Please move it to the top or before the example and probably add in bold change in behavior since version X: , so users may see more clearly what changed

i forgot where i'm checking the documentation: http://jinja.pocoo.org/docs/2.10/templates/#list-of-builtin-filters

I don't agree that the documentation needs updating.  The truncate function actually does default leeway=None, but then overrides it using env.policies to be 5.  However, this is clearly documented here http://jinja.pocoo.org/docs/2.10/templates/#list-of-builtin-filters with the following statement:
"The default leeway on newer Jinja2 versions is 5 and was 0 before but can be reconfigured globally."
I'm not a maintainer, but I wouldn't support falsely stating that leeway defaults to 5 when it actually defaults to None, and then is overridden to be 5; especially since this is already explicitly stated in the documentation that default leeway is 5.
From the code (version 2.10):
def do_truncate(env, s, length=255, killwords=False, end='...', leeway=None):
    if leeway is None:
        leeway = env.policies['truncate.leeway']


Update README.md

several

Coverage remained the same at 64.106% when pulling 2f8d727c17283a8d587ccebb5440a020c69dae03 on jeldred:patch-1 into 7acbca86b043af58d9b7ff3b0263b041a3c72800 on ernfrid:update_readme.


make use of map instead of slice to improve pod handling speed

Please provide a description of this PR:
Review as the commit depicted

PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hzxuzhonghu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
29e816c8f683570e30a99060d16470c6dbe731c3
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
29e816c8f683570e30a99060d16470c6dbe731c3
link
true
/test lint

gencheck_istio
29e816c8f683570e30a99060d16470c6dbe731c3
link
true
/test gencheck

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-multicluster_istio
e0316bee14b0bcc8ba0d7a722d6e8c9eac8cfee4
link
unknown
/test integ-security-multicluster

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-multicluster


NODE_PATH not being considered

I have NODE_PATH defined as
export NODE_PATH="/Users/usename/Dropbox/Mackup/lib/node_modules"

where
node
> require('request')

works fine on Terminal,
but when it comes to SublimeText-Nodejs it won't load any module I have installed, looking only for modules installed where the document is saved.
How can I use global NPM modules?
Thanks in advance

Yep. Thanks @zanona


False positive on using classes extending same trait

sealed trait Person

final case class Student(name: String) extends Person
final case class Teacher(name: String) extends Person

val list : List[Person] = List(Student("john"),Teacher("doe"))

The code no longer issues any warnings using latest master. Not even sure what was the inspection which triggered that, so I am closing without adding a unit test.
Thanks for reporting though!


fix(notificationdrawer): updated dropdowns so they dont overflow pare…

…nt (#2767)

Hmm, something seems to be the matter with version numbers. This PR shouldn't trigger a release since it's a PR, but it's trying to since it sees that the branch it's running on is master. I'm unsure of what versions CI is actually seeing (they're all git tags) and trying to release.
Let me take a look and think about how to best do this.

Ah, part of the reason is that master is currently broken: https://circleci.com/gh/patternfly/patternfly-next/4836
I'll get a PR up to fix that first, then we can try this again.


Custom runIde task requires idea and pluginDirectory

Plugin version: 0.4.9
Building plugin for PyCharm
I've created custom runIde task as described in custom-run-ide-task example:
task runIdeNonInternal(type: RunIdeTask) {
    jvmArgs "-Didea.is.internal=false"
}

Nevertheless, I receive the following errors when trying to run this task:
17:56:44: Executing task 'runIdeNonInternal'...

> Task :wrapper

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :patchPluginXml UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :inspectClassesForKotlinIC UP-TO-DATE
> Task :instrumentCode UP-TO-DATE
> Task :postInstrumentCode
> Task :jar UP-TO-DATE
> Task :prepareSandbox UP-TO-DATE
> Task :copyStubs NO-SOURCE
> Task :runIdeNonInternal FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Some problems were found with the configuration of task ':runIdeNonInternal'.
> No value has been specified for property 'ideaDirectory'.
> No value has been specified for property 'pluginsDirectory'.

runIde task works fine for me with this argument.

No value has been specified for property 'ideaDirectory'.
No value has been specified for property 'pluginsDirectory'.

You have to set them

runIde task works fine for me with this argument.

Those parameters are filled by gradle-intellij-plugin

And is there any way to run exactly runIde with these jvmArgs? Maybe the way to pass it as an argument to run configuration?

runIde.ideaDirectory or something like that

task runIdeNonInternal(type: RunIdeTask) {
    jvmArgs "-Didea.is.internal=false"
    ideaDirectory runIde.ideaDirectory
    pluginsDirectory runIde.pluginsDirectory
}

> Cannot get property 'classes' on null object
So it seems these parameters are mapped only when the task is called?

 ideaDirectory { runIde.ideaDirectory } to make it lazy?

> Cannot get property 'absolutePath' on null object

the same for pluginsDirectory of course

Yeah, but that still occurs :)

please attach the full stacktrace using (use --stacktrace argument to get it)

> Task :runIdeNonInternal FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runIdeNonInternal'.
> Cannot get property 'absolutePath' on null object

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':runIdeNonInternal'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
        at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: java.lang.NullPointerException: Cannot get property 'absolutePath' on null object
        at org.jetbrains.intellij.Utils.getIdeaSystemProperties(Utils.groovy:78)
        at org.jetbrains.intellij.Utils$getIdeaSystemProperties$9.call(Unknown Source)
        at org.jetbrains.intellij.tasks.RunIdeBase.configureSystemProperties(RunIdeBase.groovy:200)
        at org.jetbrains.intellij.tasks.RunIdeBase.exec(RunIdeBase.groovy:173)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:801)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:768)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:131)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:120)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99)
        ... 31 more

thanks, I'll take a look

also, you need
        configDirectory = {runIde.configDirectory}
        systemDirectory = {runIde.systemDirectory}

Got the same issue moving from 0.3.12 to 0.4.9.
IMHO would be better to maintain consistency and set pluginsDirectory, configDirectory and systemDirectory to defaults if not specified, as it was before.

To not create new similar issue will ask here:
If I need to create a few RunIde tasks how to avoid setting pluginsDirectory, configDirectory, and systemDirectory for each task repeatedly?
Example:
task runCLion_191(type: RunIdeTask){
    ideaDirectory '.../CLion/ch-0/191.7141.37/'
}
task runCLion_183(type: RunIdeTask){
    ideaDirectory '.../CLion/ch-1/183.5429.37/'
}
etc...

Should I made some kind of "proto-task/abstract-task" with
    pluginsDirectory = {runIde.pluginsDirectory}
    configDirectory = {runIde.configDirectory}
    systemDirectory = {runIde.systemDirectory}

and then declare all runClion_xxx tasks as type: from it? If so how to do it in groovy?
PS that's probably my lack of experience with groovy rather than issue with this plugin...

@ArtsiomCh to avoid it you can just use SNAPSHOT version of the plugin. Another way is you can just interate all RunIde tasks and set their properties: project.tasks.withType(RunIde.class).forEach{} or something like this


Add support for batteries?

See https://github.com/tijsverkoyen/HomeAssistant-FusionSolar/issues/6

I'm look for people that want to test this new feature. I personally don't have a battery. I used the API reference the add the functionality, but I would like to test with real data.
If you want to help, send me your OpenAPI credentials at fusionsolar [at] verkoyen.eu.


chore: rm modelina files and folders

Description
This PR removes modelina playground specific code.
Related issue(s)

fixes #1732

hey @anshgoyalevil
Nice work, but @derberg has gone for a conference so we need to wait for his approval

@sambhavgupta0705 Sure. 🛩

@derberg Yes it looks good to me 👍🏻
great work @anshgoyalevil

/rtm

@allcontributors please add @anshgoyalevil for code


Breakpoints and executable lines

I need to mark executable lines with blue dots and breakpoints with red.
What are the options?

Check these events:

OnCustomLineColors
OnAfterBookmarkPanelPaint
OnBeforeBookmarkPanelPaint
OnBookmarkPanelLinePaint
etc.

I think there should be everything you need for it. Check also PaintLeftMargin procedure and you get the idea in which order it is painted. You can also use your own bookmarks.


Cannot Login with two factor authentication

I'm trying to connect with Xrdp on ubuntu 16.04 , in this ubuntu running two factor authentication
using  Google Authenticator.   I`m logining successfully with different RDP services like NX, bat xrdp cannot understand verification code .
if somebody has experience please tell me how to configure xrdp.ini end sesman.ini
or /etc/pam.d/xrdp-sesman

Dup of #736.


Problems with robust-laplacian 0.1.0

I get compilation errors when installing the dependencies, because robust-laplacian 0.1.0 seems to have problems with the namespace of Vector3. I installed the latest version (0.2.4) and it seems to work.
So maybe the dependency should just be changed to >=0.2.4 if there is no reason against using a more recent version.

Ah, thanks for the quick heads up. Not sure what's going on there but the newer version should certainly be safe. I just updated the environment.yml in aba98e99c0427b192ef6b3d6e8e56d3d30812e64. Haven't re-tested yet but hopefully that fixes.

I didn't use conda but pip and I probably have another compiler than you used, so a lot of things could have been the issue.
As your fix only pins the version I've installed manually, I think we can close the bug. The other packages were installed without any error.


Added FB data extraction

I don't have an iPhone, but wanted to use this analysis (really cool presentation btw). So I applied it to my FB data. It's not quite as smooth as with the iPhone, since it requires to download your archive and then convert it to json (https://github.com/ownaginatious/fbchat-archive-parser). In addition I'm not very familiar with Python or Pandas. Any interest in integrating this? Then I can put some more effort into getting everything right.

Hey Neitsch,
Thanks for the contribution, and I'm glad to hear you enjoyed the talk! I think integrating facebook messenger is a fantastic idea.  I'd be happy to integrate this, but do have a few comments:

Instead of using table_connector, how about renaming that to iphone_connector and then creating a new file named fb_dump_connector, documenting it and adding your logic there?  You can then include clear documentation and error checking for opening the json bundle
Let's add the new pip dependency to an optional-requirements.txt file and update the README to mention it
I'll likely have to update some cases in the IPython notebook but would be happy to do so

Are you interested in following through on this?  If so, I'd recommend I create a branch for you and we can work in that.  The benefit is as you complete chunks you can send them my way for review and submission.  Also as a heads up, I tend to give very thorough reviews :), interested in line by line reviews of your Python?

Also thinking fb_archive_connector would be better than fb_dump_connector

Hey mdezube,

Separating those two into different classes is a great idea! Will do so
Will add that
I can take care of it

Yes, I absolutely want to move forward with this! Reviews are the best way to improve, thorough are the best 👍
I'll start by separating the two connectors and then we'll go from there :)
Thank you!

Hey, working on putting the parsed data in a DF. What exactly is required for the address_book and the message_df? I think you only need full_name for address_book and for the message_df you need (message_id,is_from_me,phone_or_email,text,date). I recall from the talk that address_book also requires birthday, is that right? Am I missing anything else?

Fantastic, looking forward to working with you on this.  And I agree regarding code reviews, glad to hear we're in alignment :)
There's a difference between what the dataframes contain and what they actually require.  E.g. although they include birthday, chat_id, message_id none of these are actually required for the rest of the analysis, they're included solely because they're available in the sqlite databases and could be used for additional analysis.
In your case what I'd do is:

Generate an address_book with full_name and birthday (if you have the birthday available, if not just include full_name).  Return this to the notebook and name it fb_address_book.  Note the other address_book is not used anywhere later in the notebook so keeping it at None is acceptable.
In the fb_archive_connector generate a fully_merged_messages_df which contains text, date, is_from_me and full_name (and birthday if you have it), none of the other fields are required.
Document the cell to clarify that there's now an option for which data source you want to use.

Offhand I don't think it's worth combining Facebook messages with texts since they're different mediums, have different vernaculars and reconciling names may not be worth it.
I'm also very welcome to other thoughts!

Closing in favor of #14


Duplicate error messages on embedded form

Describe the bug
I have an action that given invalid input will generate 2 errors. But when given the same input in an AshPhoenix.Form, the same invalid input will generate each error twice.
I wrote the following test to duplicate the issue in my app.
    test "should not duplicate error messages", %{
      environment: environment,
      feature: feature,
      owner: owner
    } do
      # This assertation passes. There are 2 errors: key is required, value is required
      assert {:error,
              %Ash.Error.Invalid{
                errors: [
                  %Ash.Error.Changes.Required{
                    field: :key,
                    type: :attribute,
                    resource: MyApp.Environments.FeatureVariant.String,
                    changeset: nil,
                    query: nil,
                    error_context: [],
                    vars: [],
                    path: [:variant],
                    class: :invalid
                  },
                  %Ash.Error.Changes.Required{
                    field: :value,
                    type: :attribute,
                    resource: MyApp.Environments.FeatureVariant.String,
                    changeset: nil,
                    query: nil,
                    error_context: [],
                    vars: [],
                    path: [:variant],
                    class: :invalid
                  }
                ]
              }} =
               MyApp.Environments.Feature.add_variant(
                 feature,
                 %{
                   variant: %{
                     description: "",
                     enabled: "true",
                     key: "",
                     metadata: "{}",
                     type: "string",
                     value: ""
                   }
                 },
                 actor: owner,
                 tenant: environment.organization_id
               )

      # This should be the same as above, except going AshPhoenix.Form, but there are now 4 errors.
      assert {:error,
              %{
                source: %{
                  forms: %{
                    variant: %{
                      # submit_errors does not match.
                      # I expect 2 errors, but there are 4. Each of the 2 errors is duplicated.
                      submit_errors: [value: {"is required", []}, key: {"is required", []}]
                    }
                  }
                }
              }} =
               AshPhoenix.Form.for_update(feature, :add_variant,
                 api: MyApp.Environments,
                 as: "feature",
                 forms: [
                   auto?: true
                 ],
                 actor: owner,
                 tenant: environment.organization_id
               )
               |> Phoenix.Component.to_form()
               |> AshPhoenix.Form.add_form(:variant, params: %{"type" => "string"})
               |> AshPhoenix.Form.submit(
                 params: %{
                   "variant" => %{
                     "_form_type" => "create",
                     "_persistent_id" => "0",
                     "_touched" => "_form_type,_touched,type",
                     "description" => "",
                     "enabled" => "true",
                     "key" => "",
                     "metadata" => "{}",
                     "type" => "string",
                     "value" => ""
                   }
                 }
               )
    end

To Reproduce
I can reliably reproduce this in my app. I don't have an easy way to replicate it outside of my app though.
Expected behavior
The errors should not duplicate
** Runtime
I'm on the latest Ash Versions.
Additional context
This form has an embedded form for an argument that is an embedded resource (but not a relationship). That resource is also a union of models.

Interestinggggggg....It's hard to say what the cause is without a reproduction, but I think it's safe for us to use an Enum.uniq when generating these errors anyway. The main reason being that two error messages could theoretically be added with the same error message for the same field and we'd want to hide those anyway.

Should be "fixed" in main. The production of duplicate errors somewhere in the process is not ideal, but this should at least fix the UX issue that you're seeing.


update tags

nit: i would delete 'printing' since that's usually an O(1) action and is less specific than your new tags


[Feature] Introduce New NSG Graph into KNN for faiss Engine

Description
[Feature] Introduce New NSG Graph into KNN for faiss Engine
As the paper shows:  Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graph
, NSG Graph has great Improvement in Query performance AND Memory Usage and with same recall precision with HNSW

And i see faiss has introduced NSG Graph, so i prefer to introduce this algorithm into KNN plugin.
As the perf-top benchmark with SIFT1M dataset shows:

Metrics
NSG
HNSW(NMSLIB)

query_took_total
66344
96996

query_took_p99
10
15

query_client_time_total
113976
147168

query_client_time_p99
16
21

query_memory_kb_total
589330
652872

And OpenSearch Benchmark with SIFT1M dataset  shows:

Metric
Task
Baseline
Contender
Diff
Unit

Mean Throughput
knn-query-from-data-set
81.4421
217.044
135.602
ops/s

Median Throughput
knn-query-from-data-set
83.8532
244.642
160.789
ops/s

Max Throughput
knn-query-from-data-set
85.6192
308.791
223.171
ops/s

99th percentile latency
knn-query-from-data-set
218.196
70.1691
-148.026
ms

Issues Resolved
More details: see PR: #945

@luyuncheng from issue description can you remove the things which are related to the PR you raised.

This is awesome @luyuncheng! Will need some time to look into this, but thanks for working on it!

@luyuncheng thanks for the contribution. I want to start discussing next steps.
First, I want to get opinions from @luyuncheng and the community on philosophy behind adding support for new algorithms in the plugin.
Previously, we have considered allowing users to pass in any valid faiss configuration string to create an index. The problem with this is that it greatly increases the amount of coverage that needs to be maintained and also may make overall user experience more complex. Instead, we decided to explicitly support algorithms. That being said, in my opinion, there is a balance to consider when supporting a new algorithm: improvements for potential users vs. increase in user complexity and increase in maintenance. So, I think before accepting a new algorithm, we need to come up with a process that we should go through before accepting a new algorithm into the plugin. For this process, I think we need to:

Provide a summary of improvements the algorithm can offer compared to the existing algorithms with an explanation of why this algorithm provides the improvements. For example, for NSG, we might say that it offers comparable query performance to HNSW, but requires less memory because it does not require a hierarchical structure.
Create proof of concept integration in OpenSearch
Execute a comprehensive performance profile of the new algorithm that can be used to compare against existing supported algorithms. For this, we can define a set of data sets that should be tested against.
Observe statistically significant improvements for some metrics/use-cases supported by experimental results - if we cannot find any, we shouldnt add it.
Document what cases this new algorithm should be used instead of others - with the addition of the algorithm, what changes about our recommendations to customers?

Obviously, we need to add a lot of details, but what are your thoughts?

@luyuncheng I think there might be some confusion. I wanted to discuss the general process for onboarding algorithms for the future - for instance, someday, we may want to support a new algorithm. Once we settle on the process, we can then use it to get NSG added. That being said, from the process I proposed in https://github.com/opensearch-project/k-NN/issues/946#issuecomment-1679422975, do you have any feedback? Do you think there are any steps missing?

i think that paper describe the proof of NSG algorithm,

By proof of concept, I meant a minimal implementation. You had actually already done this in #945.

In the NSG case, we only change the method parameter, usage is easier. but we need a description about the difference between HNSW and NSG. like trade balance with index / search memory?

yes exactly - we need to be able to describe when a user should use NSG vs. HNSW.

do you have any feedback? Do you think there are any steps missing?

@jmazanec15  I like the process steps. Nothing to add. Thanks for it !

@luyuncheng thanks, I am working on adding some more details. Will update this thread once I have more information.
I had a couple more questions on the implementation of the algorithm/algorithm:

Is the file size a good estimate of the memory that the graph will take up? The plugin uses the file size as an estimator of memory
Does the algorithm generalize to the inner product space? I see in the paper they reference euclidean, but do not see cosine/inner product. Have you tried NSG with either of these spaces?

Is the file size a good estimate of the memory that the graph will take up? The plugin uses the file size as an estimator of memory

i need some time to verify this. i plan to use some test, check the RES memory and segment file size.

Does the algorithm generalize to the inner product space? I see in the paper they reference euclidean, but do not see cosine/inner product. Have you tried NSG with either of these spaces?

@jmazanec15
i have tried inner product, and i think there is a unit test in following tests:
https://github.com/opensearch-project/k-NN/blob/3269d729187d3d43e922e7b6fa46787b09fb031e/src/test/java/org/opensearch/knn/jni/JNIServiceTests.java#L671-L679

@jmazanec15
i have tried inner product, and i think there is a unit test in following tests:

Thanks @luyuncheng. Yes I think we might need to test on some inner product data sets in order to ensure recall behaves correctly.

@luyuncheng I think we should benchmark a couple more data sets. Could you run benchmarks for 3 different data sets like you ran in #945:

glove-200 (angular - dataset will need to be normalized before running with IP space) and
gist-960 (euclidean)
sift-128 (euclidean)

With set to r=16,32,64? And forcemerge to 1 segment?

@jmazanec15 i use perf-top benchmark test NSG(R32), NSG(R64),HNSW(NMSLIB) with 8Core 16GB machine

Additionally, in #945 , could you provide more details around your cluster setup:

what machines did you use
how many data nodes
how many master nodes
what cloud did it run in

Also, what client did you use? What was network configuration between client and cluster?

Could you run benchmarks for 3 different data sets like you ran in #945

@jmazanec15  OK, give me some time, i'll benchmark it.

could you provide more details around your cluster setup:

machine some like AWS  c7g.2xlarge
Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz * 8
Memory 16GB

Single node.

so i am wondering should i use single-node benchmarking or clusters benchmark. pls let me know which is better for you to compare the performance in different machine.

machine some like AWS c7g.2xlarge
Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz * 8
Memory 16GB

Single node.

Thanks, Is the machine a c7g.2xlarge instance? Also, are you using docker to create opensearch node? What was the JVM heap percentage?  Is perf-tool running from a different machine? If so, what are the details about that machine?

so i am wondering should i use single-node benchmarking or clusters benchmark.
i also can run benchmark on physical machine with 92core * 3node. with less cpu i can only run on virtual machine or k8s with cpu limit.
pls let me know which is better for you to compare the performance in different machine.

Related to #1053. I think single-node makes sense. IMO, Multi-node will not add any additional insight into the algorithm - instead, it may just add noise. So I think the setup is good.
However, for comparison purposes, it would be good to run HNSW experiments as baselines to compare to NSG for expeeriments detailed in https://github.com/opensearch-project/k-NN/issues/946#issuecomment-1690375951. For HNSW, the following parameters are good:

m=16, ef_construction=512, ef_search=512
m=8, ef_construction=128, ef_search=128

@jmazanec15 i did 1st benchmark with dataset gist-960 , and next i will do other dataset: glove-200 , sift-128
Dataset: gist-960
Machine: 8Core16G, client + node in one single machine.
Parameter:  perf-tool, os jvm 6g
endpoint: localhost
test_name: "Faiss Test"
test_id: "Faiss NSG/HNSW Test"
num_runs: 2
show_runs: true
setup:
  - name: delete_index
    index_name: target_hnswindex
steps:
  - name: create_index
    index_name: target_index
    index_spec: sample-configs/faiss-{hnsw/nsg}/index-spec-glove.json
  - name: ingest
    index_name: target_index
    field_name: target_field
    bulk_size: 500
    dataset_format: hdf5
    dataset_path: /dataset/gist-960-euclidean.hdf5
  - name: refresh_index
    index_name: target_field
  - name: force_merge
    index_name: target_field
    max_num_segments: 5
  - name: query
    k: 100
    r: 1
    calculate_recall: true
    index_name: target_index
    field_name: target_field
    dataset_format: hdf5
    dataset_path: /dataset/gist-960-euclidean.hdf5
    neighbors_format: hdf5
    neighbors_path: /dataset/gist-960-euclidean.hdf5

Metrics
NSG(R16)
NSG(R32)
NSG(R64)
HNSW(Faiss)
HNSW(Faiss)

parameter
R:16
R:32
R:64
m:8,ef_c:128,ef_s:128
m:16,ef_c:512,ef_s:512

index_shard
3
3
3
3
3

processor
8
8
8
8
8

test_took
2959906.95
3039879.44
3200681.76
2360436.12
4496545.97

refresh_index_took_total
1285479.95
1553477.82
1754427.23
1192.52
1529585.77

query_took_total
88529.5
93536.5
92634.5
99443.0
131032.5

query_took_p99
101.0
103.0
105.0
105.0
152.0

query_memory_kb_total
3791290.0
3801255.5
3804834.0
3836531.0
3898744.0

query_recall@K_total
0.845
0.94322
0.959
0.899085
0.99738

query_recall@1_total
1
1
1
1
1

Thanks @luyuncheng! Could you also add p50 and p90 query metrics as well?
Additionally, as an update, following https://github.com/opensearch-project/.github/blob/main/RELEASING.md#security-reviews, I started the internal security consultation. I will provide updates as I receive them.

Could you also add p50 and p90 query metrics as well?

@jmazanec15  Updated,
AND also i added R:96 in NSG graph, it seems when R increase linearly, and query took time also increase linearly, also increased recall.

i did 2nd benchmark with dataset sift-128 1M , next i want to do some specific memory usage tests
Dataset: sift-128 1M
Machine: 8Core16G, client + node in one single machine.
Parameter:  perf-tool, os jvm 6g, 3Shard, 0 Replication
endpoint: localhost
test_name: "Faiss Test"
test_id: "Faiss NSG/HNSW Test"
num_runs: 2
show_runs: true
setup:
  - name: delete_index
    index_name: target_index
steps:
  - name: create_index
    index_name: target_index
    index_spec: sample-configs/faiss-{hnsw/nsg}/index-spec-sift.json
  - name: ingest
    index_name: target_index
    field_name: target_field
    bulk_size: 500
    dataset_format: hdf5
    dataset_path: /dataset/sift-128-euclidean.hdf5
  - name: refresh_index
    index_name: target_field
  - name: force_merge
    index_name: target_field
    max_num_segments: 5
  - name: query
    k: 100
    r: 1
    calculate_recall: true
    index_name: target_index
    field_name: target_field
    dataset_format: hdf5
    dataset_path: /dataset/sift-128-euclidean.hdf5
    neighbors_format: hdf5
    neighbors_path: /dataset/sift-128-euclidean.hdf5

Metrics
NSG(R16)
NSG(R32)
NSG(R64)
NSG(96)
HNSW(Faiss)
HNSW(Faiss)

parameter
R:16
R:32
R:64
R:96
m:8,ef_c:128,ef_s:128
m:16,ef_c:512,ef_s:512

index_shard
3
3
3
3
3
3

processor
8
8
8
8
8
8

test_took
483342
483342
519851.58
541131.67
597683.22
231579.61

refresh_index_took_total
336279.89
366993.81
387473.35
445455.44
65002.56
347285

query_took_total
50863
55634
57060
58365
63067
106466

query_took_p50
5
5
6
6
6
11

query_took_p90
6
6
7
7
7
12

query_took_p99
7
8
8
8
11
16

query_memory_kb_total
567662
588119
596139
601199
586523
648732

query_recall@K_total
0.9599
0.9871
0.9911
0.9925
0.982
0.999

query_recall@1_total
1
1
1
1
1
1

Thanks @luyuncheng. Overall, indexing seems to be more expensive for NSG, correct? Query times do look better though. Memory seems somewhat comparable.
Also, with all of the benchmarks, I think it would be helpful to collect the conclusions that we can draw from them.

indexing seems to be more expensive for NSG, correct? Query times do look better though.

Yes, correctly. it tooks more time to create NSG grpah at index time. and better query time without build multi layer graph.

Memory seems somewhat comparable.

i am wondering why paper can reduce many memory then hnsw. so maybe i need to check the rss memory.

@luyuncheng What is the formula for memory estimate for nsg? From the implementation, it looks like there is an adjacency matrix that is used to store the neighbors. So formula should roughly be:
memory_footprint = num_vectors * dimension * 4 + (R * num_vectors * 8)

4 bytes/float
8 bytes/int
plugging that in, memory estimate for sift with R=32 should be = 0.71 GB (actual is 0.58 GB).
So it looks like they do some compression on disk, meaning that the disk-based estimate for memory is inaccurate. See:

https://github.com/facebookresearch/faiss/blob/main/faiss/impl/NSG.h#L69
https://github.com/facebookresearch/faiss/blob/main/faiss/impl/index_write.cpp#L349

That being said, the HNSW memory estimate is:
memory_footprint = 1.1 * (4*dimension + 8 * M) * num_vectors

similarly for M=16, for sift, this value would be 0.65.
That being said, Im not sure that with this implementation with the adjacency matrix, there would be a memory saving during search.

similarly for M=16, for sift, this value would be 0.65.
For R=64, it would be 0.71 GB.

@jmazanec15  Thanks for the formula for memory estimate. in #1139 i added a memory tests to check the real memory usage with different algorithms. i use SIFT1M datasets and get following result.

Algotightm
IndexTime RES
FileSize
Query RES

HNSW32
1.8GB
634MB
769MB

NSG64
3.1GB
586MB
752MB

@jmazanec15 when i testing the memory and performance,  i think there is some trick parameter with paper,
so i read the faiss code, when we using IndexFactory to build nsg graph, it would using IndexNSGFlat to build with build_type=0 https://github.com/facebookresearch/faiss/blob/v1.7.4/faiss/IndexNSG.cpp#L53, and this make graph not use NNDescent.

@luyuncheng I see thanks for providing that Ill take a look at #1139 as well. Overall, memory limit seems pretty good. Im wondering if for querying they use 32-bit ints when loading the graph. For indexing, the RES seems very high for all algorithms. Do you know what might be causing this?

@luyuncheng I see thanks for providing that Ill take a look at #1139 as well. Overall, memory limit seems pretty good. Im wondering if for querying they use 32-bit ints when loading the graph. For indexing, the RES seems very high for all algorithms. Do you know what might be causing this?

@jmazanec15  i use gperftools with LD_PRELOAD=/usr/local/lib/libprofiler.so shows as following:
i am wondering whether the temporary variable and jni wrap occupy many memory
in HNSW

in NSG

Thanks @luyuncheng. For nsg and hnsw for sift, I got the following results:
$ ./bin/jni_memory_test --gtest_filter=FaissNSGIndexMemoryTest.*
Running main() from /home/ec2-user/mem-test/k-NN-1/jni/build/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FaissNSGIndexMemoryTest.*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FaissNSGIndexMemoryTest
[ RUN      ] FaissNSGIndexMemoryTest.BasicAssertions
[          ] [ INFO ]points_num：1000000 data dimension：128
[          ] [ INFO ]======Memory Usage:[37mb]======
[       OK ] FaissNSGIndexMemoryTest.BasicAssertions (227085 ms)
[----------] 1 test from FaissNSGIndexMemoryTest (227085 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (227085 ms total)
[  PASSED  ] 1 test.

$ ./bin/jni_memory_test --gtest_filter=FaissHNSWIndexMemoryTest.*
Running main() from /home/ec2-user/mem-test/k-NN-1/jni/build/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FaissHNSWIndexMemoryTest.*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FaissHNSWIndexMemoryTest
[ RUN      ] FaissHNSWIndexMemoryTest.BasicAssertions
[          ] [ INFO ]points_num：1000000 data dimension：128
[          ] [ INFO ]======Memory Usage:[559mb]======
[       OK ] FaissHNSWIndexMemoryTest.BasicAssertions (41356 ms)
[----------] 1 test from FaissHNSWIndexMemoryTest (41356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (41356 ms total)
[  PASSED  ] 1 test.

Am I doing anything wrong with this?

Also, for HNSW test, did you use nmslib or faiss?

Also, for HNSW test, did you use nmslib or faiss?

@jmazanec15  i think this caused by malloc_trim(0); at latest commits, i removed all malloc_trim(0); calling.
i test following case:
added malloc_trim(0); in all tests, the memory usage only 16MB .
deleted malloc_trim(0); in all tests, the memory usage upto 2143mb

@jmazanec15  i think there is some memory usage optimize as https://github.com/opensearch-project/k-NN/issues/946#issuecomment-1725658197 shows, when i write #1139 and i found some memory usage waste. so i will create a new issue for this.

so when i evaluation Index RES, i use the running time RES rather than the memory usage at the end of time.

What is the running time RES? Is this from the docker stats? I have ran a couple unrelated benchmarks with the docker-compose code you have worked on, and docker stats seems to work pretty well.

so i used tcmalloc to check the total memory allocated shows as following:
FaissHNSWIndexMemoryTest: 2143mb
FaissNSGIndexMemoryTest: 2925mb
NmslibHNSWIndexMemoryTest: 2156mb

the allocator will hold on to some memory (not sure what tcmalloc's behavior is, but we've seen glibc malloc do this quite a bit - jemalloc has much better properties). But in https://github.com/opensearch-project/k-NN/issues/946#issuecomment-1721015640 how did you get 1.8 to 3 GB with malloc_trim() enabled?

and i found some memory usage waste. so i will create a new issue for the memory usage optimize.

Oh nice. Did you find in the JNI code?


docs: fix hugo links

The relative links at the docs are wrong here

https://gohugo.io/content-management/cross-references/

Does not work yet. Do i need to rebuild locally and send pr with updated files?

@viceice Yes, you need to rebuild after every change

ok, so  I'll send a new pr with rebuild


Add nap6 menu contributions

This PR puts the zarpaint widgets and functions in defined menus in napari>=0.5.0.
See NAP-6 for details:
https://napari.org/dev/naps/6-contributable-menus.html
Result (needs napari/napari#7011):

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.43%. Comparing base (74a6b0a) to head (bd47b72).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #51   +/-   ##
=======================================
  Coverage   91.43%   91.43%
=======================================
  Files          14       14
  Lines         537      537
=======================================
  Hits          491      491
  Misses         46       46

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Build with correct CsvLint binary on linux-x64

Closes #121

Ah, lovely stuff. Nice elegant solution too. I'll merge this and kick off a build :+1:


Print provider config in provider get, if any

Summary
We used to only print provider config if there was any on provider get. That's not very user-friendly in case the user enables
auto-registration and then runs provider get to see what had changed.
In the usual case where there's no config, we don't print anything. If
there is configuration, we print key-value pairs for each config option.
The key-value pairs work fine for now since we have a low number of
config options. In the future, once our config gets more complicated, we
should just switch to printing the config as YAML, but for now this
looks awkward and the table gets too vertically long for just one
option.
Fixes: #3816
Change Type

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
minder provider update --set-attribute=auto_registration.entities.repository.enabled=false --name=github-app-jakubtestorg
minder provider get -n github-app-jakubtestorg

Review Checklist:

[x] Reviewed my own code for quality and clarity.
[ ] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 53.053% (+0.005%) from 53.048%
when pulling 9840bdcf21fee20da16bf56b34e8bbb15c0e37b3 on jhrozek:provider_get_show_config
into 41e48333e3c329cc02a6eb688bf0192dcbd77493 on stacklok:main.


No Hover hints , diagnostics and others

I have set daScript path int setting,but it seems not work for me....
I think the “server.das” take so much time to compile for run。。。

I set the connectTimeout to 50s,and it working...

It hard to believe, but this is easiest fix. I didn’t managed to fix it more correctly. Try to find lowest timeout then all works and use it. For debug build it usually 20-30sec, for “release with debug info” - 4-8

ok, that works ok~