[BUG] share-manager-pvc appears to be leaking memory

Describe the bug
Over time share-manager-pvc appears to leak memory, starting at ~100Mb and growing as far as we've seen to ~15Gb
To Reproduce
Not sure, we just wait over time. Seems like a leak as the graph as jumps in it, perhaps related to pod restarts / mounts?
Expected behavior
Stable memory usage
Support bundle for troubleshooting
Will do
Environment

Longhorn version: 1.6.1
Impacted volume (PV): share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: microk8s v1.28.8

Number of control plane nodes in the cluster: 3
Number of worker nodes in the cluster: 0

Node config

OS type and version: ubuntu 22.04
Kernel version: 5.15.0-102-generic
CPU per node: 12
Memory per node: 128Gb
Disk type (e.g. SSD/NVMe/HDD): NVMe
Network bandwidth between the nodes (Gbps): 1Gbps

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 5

Additional context

examples of the leak since last share-manager pvc restart:

not sure but I think the big spikes may correspond to attaching the volume to a different pod given the timings

supportbundle_5c8ad69a-4ef9-47a4-8c8c-6aaa06a38849_2024-04-19T06-15-54Z.zip
I've snipped some logs out of this that contain too much information (for example the node syslogs), I can supply relevant sections if needed or possibly share them privately (I'd need to get permission to do that)

Can you login into the share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f pod and provide us the results

ps aux
cat /proc/<PID of ganesha.nfsd>/status
cat /proc/1/status

@PhanLe1010 Have you ever observed the memory leak in the RWX performance investigation?

share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f:/ # ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 1237096 13200 ?       Ssl  Apr14   2:23 /longhorn-share-manager --debug daemon --volume pvc-0cb784d8-46f9-42e1-9476-591
root          38  1.6  1.6 5015480 2145440 ?     Sl   Apr14 113:57 ganesha.nfsd -F -p /var/run/ganesha.pid -f /tmp/vfs.conf
root      610865  0.8  0.0   6956  4268 pts/0    Ss   07:30   0:00 bash
root      610893  133  0.0  13748  4048 pts/0    R+   07:30   0:00 ps aux
share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f:/ # cat /proc/38/status
Name:   ganesha.nfsd
Umask:  0000
State:  S (sleeping)
Tgid:   38
Ngid:   0
Pid:    38
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 64
Groups: 0
NStgid: 38
NSpid:  38
NSpgid: 1
NSsid:  1
VmPeak:  5083072 kB
VmSize:  5015480 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   2145444 kB
VmRSS:   2145444 kB
RssAnon:         2134608 kB
RssFile:           10836 kB
RssShmem:              0 kB
VmData:  2303600 kB
VmStk:       136 kB
VmExe:        16 kB
VmLib:     18704 kB
VmPTE:      4784 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        20
SigQ:   1/514185
SigPnd:<PHONE_NUMBER>000000
ShdPnd:<PHONE_NUMBER>000000
SigBlk:<PHONE_NUMBER>005001
SigIgn:<PHONE_NUMBER>000000
SigCgt:<PHONE_NUMBER>000000
CapInh:<PHONE_NUMBER>000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb:<PHONE_NUMBER>000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   fff
Cpus_allowed_list:      0-11
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        56
nonvoluntary_ctxt_switches:     109
share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f:/ # cat /proc/1/status
Name:   longhorn-share-
Umask:  0022
State:  S (sleeping)
Tgid:   1
Ngid:   0
Pid:    1
PPid:   0
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 64
Groups: 0
NStgid: 1
NSpid:  1
NSpgid: 1
NSsid:  1
VmPeak:  1237096 kB
VmSize:  1237096 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     15228 kB
VmRSS:     13200 kB
RssAnon:            4784 kB
RssFile:            8416 kB
RssShmem:              0 kB
VmData:    46880 kB
VmStk:       136 kB
VmExe:      4380 kB
VmLib:         8 kB
VmPTE:       120 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        18
SigQ:   1/514185
SigPnd:<PHONE_NUMBER>000000
ShdPnd:<PHONE_NUMBER>000000
SigBlk: fffffffc3bba3a00
SigIgn:<PHONE_NUMBER>000000
SigCgt: fffffffd7fc1feff
CapInh:<PHONE_NUMBER>000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb:<PHONE_NUMBER>000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   fff
Cpus_allowed_list:      0-11
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        150
nonvoluntary_ctxt_switches:     59
share-manager-pvc-0cb784d8-46f9-42e1-9476-5918989ec94f:/ #

The VmRSS of the nfs-ganesha process is too high (2145444 kB). The high VmRSS value was not observed in my cluster, and the value went back to a lower value after an IO intensive operation.
To verify if it is caused by the upstream regression you mentioned, could you provide us with the steps to reproduce as well as the information

What's your workload (manifest is appreciated)
Is it IO intensive? Write or read?

I'm not sure if it's caused by the regression, I was just looking for leaks in upstream given it's the nfs server using all the ram.
The workload is that maybe 10-15 times a day rsync is run on ~4000 files, and half a dozen files will be copied to the nfs share (which in our case corresponds to releasing some code), then a few hundred times a day a few dozen of those files will be read at a time. It's quite light use and read mostly.
By manifest are you after a test workload?

I'm not sure if it's caused by the regression, I was just looking for leaks in upstream given it's the nfs server using all the ram.
The workload is that maybe 10-15 times a day rsync is run on ~4000 files, and half a dozen files will be copied to the nfs share (which in our case corresponds to releasing some code), then a few hundred times a day a few dozen of those files will be read at a time. It's quite light use and read mostly.
By manifest are you after a test workload?

OK. I will try the steps in our lab.
If you are available, you can also try v1.5.5 and see if it doesn't have the issue.

cc @james-munson

@derekbit I can't be sure but v1.5.5 looks better, below is the memory usage with v1.6.1 jumping when the files are read by some nightly jobs. One thing I hadn't thought of is that those pods are all transient, and there are a few thousand of them, so this also corresponds to a lot of mounting and unmounting:

and here is the comparison from yesterday's nightly jobs:

you can see there is a small increase in memory usage but it's nothing like before

@blushingpenguin
Thanks for the update.
I can build a customized share-manager image with the fix of the memory leak. Can you help test if it is the culprit in v1.6.1? What do you think?

@derekbit I'm happy to test a fixed build, however I think it's probably worth giving v1.5.5 another 24h to confirm that the leak doesn't occur with our weekday usage pattern. I'll check it tomorrow and report back.
Thanks,
Mark

@blushingpenguin
You can test the share-manager image derekbit/longhorn-share-manager:v1.6.1-fix-leak on Longhorn v1.6.1.
To replace the share-manager image, edit the longhorn-manager daemonset by kubectl -n longhorn-system edit daemonset longhorn-manager, then replace the share-manager image
....
    spec:
      containers:
      - command:
        - longhorn-manager
        - -d
        - daemon
        - --engine-image
        - longhornio/longhorn-engine:v1.6.1
        - --instance-manager-image
        - longhornio/longhorn-instance-manager:v1.6.1
        - --share-manager-image
        - longhornio/longhorn-share-manager:v1.6.1 <---- Replace this image with derekbit/longhorn-share-manager:v1.6.1-fix-leak
        - --backing-image-manager-image
        - longhornio/backing-image-manager:v1.6.1
        - --support-bundle-manager-image
        - longhornio/support-bundle-kit:v0.0.36
        - --manager-image
        - longhornio/longhorn-manager:v1.6.1
        - --service-account
        - longhorn-service-account
        - --upgrade-version-check
...

Many thanks.

v1.5.5 still looks a bit leaky, but nothing on the scale of before:

I will switch to your patched version now and report back tomorrow. Thanks very much for following up on this!

@derekbit  yes, it looks like that was the major cause.
v1.6.1-fix-leak:

(you can see the restart yesterday at around 05:30)
Thanks very much for digging into it so quickly.

Thanks @blushingpenguin for the quick update.
The fix will be included in v1.6.2. Temporarily, you can continue using derekbit/longhorn-share-manager:v1.6.1-fix-leak.
I also found nfs-ganesha still seems a bit leaky. It's minor. I will check this part later.

This is a very, very, very bad situation! Will we have and release like... today?!
In my cluster with just 5 nodes 12 disk the Longhorn is using more than 100GB of ram over all nodes!!!
This response time will show us if this is a reliable storage system, how can a critical bug like this happen in a release marked as stable?!
Please, just fix it!

This is a very, very, very bad situation! Will we have and release like... today?!
In my cluster with just 5 nodes 12 disk the Longhorn is using more than 100GB of ram over all nodes!!!
This response time will show us if this is a reliable storage system, how can a critical bug like this happen in a release marked as stable?!
Please, just fix it!

Can you elaborate more on your use case?
The issue is triggered only when there are tons of unmount and mount operation.
I've provided a share-manager image for mitigating the issue before the v1.6.2 release. Please see https://github.com/longhorn/longhorn/issues/8394#issuecomment-2074021679.

@brunosmartin It's upstream that's at fault here, and is possibly related to usage patterns (I was seeing much higher ram usage with volumes that were being mounted/unmounted a lot). This a problem of modern software development I suppose -- pretty much all software is built of other components these days, and validating they all work together is a pretty difficult task. I actually think the response has been excellent here, @derekbit was on the case immediately.
You can just patch your deployment as outlined in https://github.com/longhorn/longhorn/issues/8394#issuecomment-2069398514 until the next release -- this has mostly fixed the problem for me (it still looks a bit leaky, but nothing like before).

This is a very, very, very bad situation! Will we have and release like... today?!
In my cluster with just 5 nodes 12 disk the Longhorn is using more than 100GB of ram over all nodes!!!
This response time will show us if this is a reliable storage system, how can a critical bug like this happen in a release marked as stable?!
Please, just fix it!

Can you elaborate more on your use case? The issue is triggered only when there are tons of unmount and mount operation.
I've provided a share-manager image for mitigating the issue before the v1.6.2 release. Please see #8394 (comment).

Maybe this is underestimated, my cluster doesn't make "tons of unmount and mount operation", but see the images above how it acted in face of this bug:

we upgrade 04/23 from 1.5.4 to 1.6.1, and the memory leak clearly consumes lots of ram. Then we applied this fix 04/26, and its stable now. Every peak in this image was a very painful huge outage!
Besides we don't make "tons of unmount and mount operation", we have like five workloads where we have 3 pods using the same volume, maybe this is related somehow.
I repeat this critical bug is underestimated, I don't have any special use, just using normal (small) kubernetes cluster with rancher and made my company's services burn for a week, it's very likely there are tons off users affected and after one week we don't have even an RC build.
@blushingpenguin for modern software development problems we have modern management methods to deal with. My point was that this critical bug shows was a mistake to call the version 1.6.1 stable.
I also work on some open source projects and my intention here is to help this project to be more stable and reliable, as an storage system must be.
Please tell me if I can provide any additional information on this issue.

@brunosmartin
Thank you for the update. We will review the strategy of marking stable release. cc @innobead
The high memory usage in your cluster sounds triggered by the same nfs-ganesha bug.
Can you elaborate more on the workload such as the applications, io patterns, read intensive or write intensive? These information will help us define the test cases and avoid falling into the known issues again.

@brunosmartin
May I have some question about your setup

Is it correct that you have 5 deployment?
Each deployment uses 1 RWX PVC?
Each deployment have 3 pods?
What are the size of the RWX PVCs?

Also, do you know what happen at the timestamp in the below picture? Did your workload deployments restart?

@brunosmartin
May I have some question about your setup
* Is it correct that you have 5 deployment?

* Each deployment uses 1 RWX PVC?

* Each deployment have 3 pods?

* What are the size of the RWX PVCs?

Also, do you know what happen at the timestamp in the below picture? Did your workload deployments restart?

@brunosmartin
May I have some question about your setup

Sure

* Is it correct that you have 5 deployment?

Yep, 3 production

* Each deployment uses 1 RWX PVC?

Yes

* Each deployment have 3 pods?

exactly, scale = 3

* What are the size of the RWX PVCs?

the smaller is 200GB, the biggest is 500GB. They are wordpress sites, with Cloudflare as CDN (with page cache) so not expected to be IO intensive, only the apache has RWX PVCs, the database does the intensive IOPs operations as a Single Node Read Write PVC.

Also, do you know what happen at the timestamp in the below picture? Did your workload deployments restart?

Yes, they did restart at that point, but I'm not sure this was related to longhorn, at that point the patch the patch was applied already (since the huge memory drop at 04/26 @ 12pm).
Hope it helps, good lucky on this.

@PhanLe1010 just to clarify, I have 5 deployments with scale > 1, but I have much more RXW PVCs, I think about 50 RWX PVCs, most of then are deployed with scale = 1.

Verified on master-head 20240507

longhorn master-head https://github.com/longhorn/longhorn/commit/8026d1a700e66f9254fad75ee1e39390a0172208
nfs-ganesha longhorn-ganesha-v5 https://github.com/longhorn/nfs-ganesha/commit/996a59c04d4f3ccbaf23a6565f84fec00b2e02e4
longhorn-share-manager master-head https://github.com/longhorn/longhorn-share-manager/commit/88d4db12400c8241652eedb7cff7ab2b3a06644c

The test steps
https://github.com/longhorn/longhorn/issues/8394#issuecomment-2074065022

Create first workload with a RWX volume by https://github.com/longhorn/longhorn/blob/master/examples/rwx/rwx-nginx-deployment.yaml
Scale up the replicas to 3.
Check if 3 workloads are in the "Running" state.
Scale down the replicas to 1.
Check if one workload are in the "Running" state.
We can test steps 2-5 using the following shell script.

deployment_rwx_test.sh
#!/bin/bash

# Define the deployment name
DEPLOYMENT_NAME="rwx-test"
KUBECONFIF="/home/ryao/Desktop/note/longhorn-tool/ryao-161.yaml"

for ((i=1; i<=100; i++)); do
    # Scale deployment to 10 replicas
    kubectl --kubeconfig=$KUBECONFIF scale deployment $DEPLOYMENT_NAME --replicas=3

    # Wait for the deployment to have 3 ready replicas
    until [[ "$(kubectl --kubeconfig=$KUBECONFIF get deployment $DEPLOYMENT_NAME -o=jsonpath='{.status.readyReplicas}')" == "3" ]]; do
        ready_replicas=$(kubectl --kubeconfig=$KUBECONFIF get deployment $DEPLOYMENT_NAME -o=jsonpath='{.status.readyReplicas}')
        echo "Iteration #$i: $DEPLOYMENT_NAME has $ready_replicas ready replicas"
        sleep 1
    done

    # Check if all pods are in the "Running" state
    while [[ $(kubectl --kubeconfig=$KUBECONFIF get pods -l=app=$DEPLOYMENT_NAME -o=jsonpath='{.items[*].status.phase}') != "Running Running Running" ]]; do
        echo "Not all pods are in the 'Running' state. Waiting..."
        sleep 5
    done

    # Scale deployment down to 1 replicas
    kubectl --kubeconfig=$KUBECONFIF scale deployment $DEPLOYMENT_NAME --replicas=1

    # Wait for the deployment to have 1 ready replicas
    until [[ "$(kubectl --kubeconfig=$KUBECONFIF get deployment $DEPLOYMENT_NAME -o=jsonpath='{.status.readyReplicas}')" == "1" ]]; do
        ready_replicas=$(kubectl --kubeconfig=$KUBECONFIF get deployment $DEPLOYMENT_NAME -o=jsonpath='{.status.readyReplicas}')
        echo "Iteration #$i: $DEPLOYMENT_NAME has $ready_replicas ready replicas"
        sleep 1
    done

    # Check if all pods are in the "Running" state
    while [[ $(kubectl --kubeconfig=$KUBECONFIF get pods -l=app=$DEPLOYMENT_NAME -o=jsonpath='{.items[*].status.phase}') != "Running" ]]; do
        echo "Not all pods are in the 'Running' state. Waiting..."
        sleep 5
    done
done

Find the PID of the nfs-ganesha in the share-manager pod by ps aux
Observe the VmRSS of nfs-ganesha in the share-manager pod by cat /proc/<nfs-ganesha PID>/status | grep VmRSS

Result Passed

We were also able to reproduce this issue on v1.6.1.
After executing the script, the output for v1.6.1 is as follows:

Every 2.0s: cat /proc/29/status | grep VmRSS                     share-manager-pvc-119d403e-ae17-4f4f-aa7f-06e7bf40fca2: Tue May  7 09:54:38 2024

VmRSS:     47192 kB

For the master-head:
Every 2.0s: cat /proc/43/status | grep VmRSS                    share-manager-pvc-d0a41b4a-5bb8-4cd5-b60a-d260c6ce8a34: Tue May  7 10:08:18 2024

VmRSS:     42588 kB

When validating this issue, there is one thing that needs to be noted.
I originally followed the test plan steps below, but I couldn't reproduce the issue using that method.
The memory leak issue seems to occur only when scaling up or down the deployment replicas.
The test steps

Created an RWX volume named vol-0 with a PV/PVC in the default namespace.
Created the first pod using the vol-0 RWX volume.

pod_mount_vol-0.yaml
kind: Pod
apiVersion: v1
metadata:
  name: ubuntu-mountvol0
  namespace: default
spec:
  containers:
    - name: ubuntu
      image: ubuntu
      command: ["/bin/sleep", "3650d"]
      volumeMounts:
      - mountPath: "/data/"
        name: vol-0
  volumes:
    - name: vol-0
      persistentVolumeClaim:
        claimName: vol-0

Created the second pod using the vol-0 RWX volume.

2nd_pod_mount_rwx_vol.yaml
kind: Pod
apiVersion: v1
metadata:
  name: ubuntu-mount-rwx-vol
  namespace: default
spec:
  containers:
    - name: ubuntu
      image: ubuntu
      command: ["/bin/sleep", "3650d"]
      volumeMounts:
      - mountPath: "/data/"
        name: vol-0
  volumes:
    - name: vol-0
      persistentVolumeClaim:
        claimName: vol-0

Deleted and created the second pod repeatedly 200 times using the following shell script.

attach_detach_pod_rwx.sh
#!/bin/bash

# This script from Derek in the Github issue #6776.
# It assumes that pod_mount_rwx_vol.yaml has already been applied, so that the PVC in pod_mount_rwx_vol.yaml will exist.
# Set the path to the pod_mount_rwx_vol.yaml file
POD_YAML_FILE="pod_mount_rwx_vol.yaml"

# Set the path to the kubeconfig file
KUBECONFIG_PATH="/home/ryao/Desktop/note/longhorn-tool/ryao-master.yaml"

# Set the number of iterations to perform
NUM_ITERATIONS=200

# Set timeout values for attachment and detachment
ATTACH_WAIT_SECONDS=120
DETACH_WAIT_SECONDS=300

for ((i=1; i<=$NUM_ITERATIONS; i++))
do
  echo "Iteration $i/$NUM_ITERATIONS"

  echo "Attaching"
  kubectl apply -f "$POD_YAML_FILE" --kubeconfig="$KUBECONFIG_PATH"

  c=0
  while [ $c -lt $ATTACH_WAIT_SECONDS ]
  do
    phase=$(kubectl --kubeconfig="$KUBECONFIG_PATH" get pod ubuntu-mount-rwx-vol -o=jsonpath="{['status.phase']}"  2>/dev/null)
    if [ x"$phase" == x"Running" ]; then
      break
    fi

    sleep 1

    c=$((c+1))
    if [ x"$c" = x"$ATTACH_WAIT_SECONDS" ]; then
      echo "Found error"
      echo "Error: Pod failed to reach Running state within $ATTACH_WAIT_SECONDS seconds"
      exit 1
    fi
  done

  echo "Detaching"
  kubectl delete -f "$POD_YAML_FILE" --kubeconfig="$KUBECONFIG_PATH"

  # Wait for a few seconds before the next iteration
  sleep 5

done

cc @derekbit @longhorn/qa


idea: nuget package?

Hey Gilles
Thanks for publishing your library.
Would it be a ridiculous idea to publish it as a Nuget Package?
(btw always appreciated your knowledge etc on the forums.)
Ben

I hadn't considered that before, but I'll think about it when I have some free time.

Pls LMK (let me know) what you think. I can provide PRs (pull requests) for this.

Closed with https://github.com/gileCAD/GeometryExtensions/issues/6


[EVAL]: Add more African Benchmarks

Evaluation short description

Why is this evaluation interesting?
This focuses on 16 African languages, evaluated on three knowledge QA and reasoning tasks such as AfriMMLU, AfriMGSM and AfriXNLI, human translated from MMLU, MGSM and XNLI respectively.

How used is it in the community?

Evaluation metadata: IrokoBench
Provide all available

Paper url: https://arxiv.org/abs/2406.03368
Github url:
Dataset url: https://huggingface.co/collections/masakhane/irokobench-665a21b6d4714ed3f81af3b1

Evaluation metadata: Uhura
Provide all available

Paper url:
Github url:
Dataset url: https://huggingface.co/datasets/ebayes/uhura-arc-easy-clean ,  https://huggingface.co/datasets/ebayes/uhura-truthfulqa-clean

Evaluation metadata: SIB-200
Provide all available

Paper url: https://aclanthology.org/2024.eacl-long.14/
Github url: https://github.com/dadelani/sib-200
Dataset url: https://huggingface.co/datasets/Davlan/sib200

@NathanHB

Hi @dadelani , thanks for your issue!
This sounds like a great idea, do you want to add them yourself to the library and open a PR? We've got a guide here for adding tasks and one here specific to multilingual evaluations :)

Hey Iroko bench is already implemented
See: https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/multilingual/tasks.py#L239
Currently only the swahili subset is available because we don't have correct translations for anchor words that are required during the evaluation (e.g. answer/question).
If you are a native speaker in any of the languages we would love your help in improving that!
See the guide how to add a translations: https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations#contributing-a-small-translation

Great, thank you @clefourrier and @hynky1999 .
For the anchor words, maybe you can use google translate of the keywords. I can provide for Yoruba below:
Question: Ìbéèrè
Answer: Ìdáhùn

Great ! do you think you can open a PR for the evals and the keywords ? We prefere not to use google translate as it can be unreliable.
As clementine said, the documentation to add a task is here:
https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations


Feature flags: create a process for ensuring flags are enabled or disabled in production

Current solution allows users to toggle feature flags. This works fine for development and testing environments, but may not be desired for production environment.
@maneesha @froggleston Please consider if you want to allow users in production (admins or instructors) to change their feature flags as desired.
If we shouldn't allow users to change their feature flags in production, but instead control them ourselves, then I propose to create a process of managing feature flags after every production release. It's possible to globally enable or globally disable a feature flag for everyone without changing code through django-admin panel. It was discussed by me and @elichad here: https://github.com/carpentries/amy/pull/2516#discussion_r1301552547
Original thread: https://github.com/carpentries/amy/pull/2516#discussion_r1301488470
Current UI used for switching feature flags on and off: https://test-amy2.carpentries.org/dashboard/admin/feature_flags/

I can't think of a reason we'd want to use feature flags in production.  Our current need for them is so that we can test features that are being developed in parallel.


set params when training the flow model

I'm trying to reproduce the two-stream results of TSM on Something v1, but the performance of my flow model is far below. (segment based sampling method)
I understand the 10 channels stacked optical flow (TV-L1) / learning rate 5 times in the first conv layer.
Is there any difference between RGB and Flow model in setting params?
(e.x, epochs, learning rate...)

Hi, we basically used the same training hyper-parameters for flow models. How much accuracy can you get using optical flow? Note that optical flow model should give a lower performance compared to RGB input.

I just stopped the training process before 50 epochs cuz my performance is too low.
It was lower than 30% at 40epochs. (ResNet-18)
Could you tell me the performance of TSM Flow model on Something v1? (8 segments, single inference, ResNet-50)

The accuracy of TSM Flow model on Something V1 is about 42%.

Thanks for the answer.
I just modified the dense sampling to the uniform sampling, and achieves the similar performance as you said.

Thanks for the answer.
I just modified the dense sampling to the uniform sampling, and achieves the similar performance as you said.

Hi, I just trained the TSM-ResNet50 model on SomethingV1 with 8-segments optical flow input only, but the performance is around 37%, far below the 42%, and the two-stream is around 48%. Can you provide the link to download the optical flow for SomethingV1 dataset that you use?

@arunos728 How did you prepare the optical-flow data? Was it as suggested in the Temporal Segment Network (TSN) repository, which requires  the denseflow package?

@boheumd How did you extract flow data for sthv1?


Shader issue in non-browser mode

Thanks for fixing the demo. So I'm rendering from node, using three in a headless configuration. When I run from the browser, fxaa works great, but when run from node on the command line, I get
TypeError: string.replace is not a function
at parseIncludes (/Users/.../node_modules/three/three.js:29083:17)
at new WebGLProgram (/Users/.../node_modules/three/three.js:29386:18)
at acquireProgram (/Users/.../node_modules/three/three.js:29819:14)
at initMaterial (/Users/.../node_modules/three/three.js:25888:27)
at setProgram (/Users/.../node_modules/three/three.js:26041:4)
at renderBufferDirect (/Users/.../node_modules/three/three.js:25138:17)
at renderObjects (/Users/.../node_modules/three/three.js:25822:11)
at render (/Users/.../node_modules/three/three.js:25606:4)
at Object.ShaderPass.render (/Users/.../node_modules/three-effectcomposer/lib/shaderpass.js:43:18)
at Object.EffectComposer.render (/Users/.../node_modules/three-effectcomposer/index.js:70:14)
It looks like in three.js, in the WebGLProgram function:
return function WebGLProgram( renderer, code, material, parameters ) {

	var gl = renderer.context;

	var extensions = material.extensions;
	var defines = material.defines;

	var vertexShader = material.__webglShader.vertexShader;
	var fragmentShader = material.__webglShader.fragmentShader;

...
vertexShader is expected to be a string, and for my other shaders, it is, but for fxaa I get an object called DuplexWrapper:
DuplexWrapper {
_readableState:
ReadableState {
highWaterMark: 16,
buffer: [],
length: 0,
pipes: null,
pipesCount: 0,
flowing: null,
ended: false,
...
and so it seems to barf because it can't find a replace method on DuplexWrapper. Any idea what the issue might be? One difference is that, in headless mode, there is no global window object, so things have to be set up a little differently. Here is how i instantiate three:
window = { innerWidth: 800, innerHeight: 600 }

var canvas = { style : new Object() };
var gl = require('gl')(window.innerWidth, window.innerHeight);
var threegl = new THREE.WebGLRenderer(
{
    antialias : false,
    width: 0,
    height: 0,
    canvas: canvas,
    context: gl
});

This works well for everything so far, I can render in three, read the framebuffer into a buffer I can then write out to an image file, etc. But of course it's aliased, so this shader is exactly what I'm looking for. Thanks,

It's probably because this module uses glslify (in its deps) and is designed to be bundled with browserify (for the browser). When run through Node, the shaders are not inlined as they should be. I haven't tested or tried to support non-browserify/webpack usage.
To support, I would need to transpile the source on pre-publish – perhaps using something like transpilify.

I just pushed to master some changes that should work for your use case now. It relies on build/index.js which has the source inlined as strings. Let me know if that works! :smile:

I just copied index.js and package.json manually into node_modules/three-shader-fxaa, and it seems to be working :) I'm hoping you're good with making a release from this? Thanks!

Ok, pushed as 5.1.0. :smile:


Error executing script. Error for call_service. Failed to upload image: Image upload failed with status code: 409

Describe the bug
I have updated  from <IP_ADDRESS> to <IP_ADDRESS>_dev.
I have changed automations to use the device_id. In general it works but now I can see quite a lot of this kind of errors in logs:
Epaper Aurora: Error executing script. Error for call_service at pos 3: Error processing device open_epaper_link.00000282db983b16: Failed to upload image: Image upload failed with status code: 409
Logger: homeassistant.components.automation.epaper_aurora
Source: components/automation/__init__.py:763
integration: Automation ([documentation](https://www.home-assistant.io/integrations/automation), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+automation%22))
First occurred: 17:21:02 (1 occurrences)
Last logged: 17:21:02
 Error while executing automation automation.epaper_aurora: Error processing device open_epaper_link.00000282db983b16: Failed to upload image: Image upload failed with status code: 409

When it happens, tag is not updated. I have 11 tags and it happens randomly, not to particular tag.
My AP's data:

What run status was your AP in at the time the upload error occurred?

If you look at https://github.com/OpenEPaperLink/OpenEPaperLink/blob/6ec580d267698e972e4e25697e6c71051a74e033/ESP32_AP-Flasher/src/web.cpp#L837
this is most likely a problem with your AP. Can you check the web page of your AP?

What run status was your AP in at the time the upload error occurred?

If you are talking about "AP Run State" sensor, then it was Running.

If you look at https://github.com/OpenEPaperLink/OpenEPaperLink/blob/6ec580d267698e972e4e25697e6c71051a74e033/ESP32_AP-Flasher/src/web.cpp#L837 this is most likely a problem with your AP. Can you check the web page of your AP?

I don't understand. What exactly you want me to check?

Additionally, around the time my automations trigger, there are sometimes those kind of messages:
This error originated from a custom integration.
Logger: custom_components.open_epaper_link.hub
Source: custom_components/open_epaper_link/hub.py:194
integration: OpenEPaperLink (documentation, issues)
First occurred: November 6, 2024 at 20:20:46 (2 occurrences)
Last logged: November 6, 2024 at 21:41:53

WebSocket error: <aiohttp.client_ws.ClientWebSocketResponse object at 0x7f0987f5c0>
WebSocket error: <aiohttp.client_ws.ClientWebSocketResponse object at 0x7f2ac10110>

This might have something to do with the missing upload queue in this version, but will be fixed in the next release


test nodelist

Add test coverage for LfuNode and LfuNodeList functionality that is not used by ConcurrentLfu. These methods exist for completeness, they were implemented to support now deleted code. In the future, we will likely need these methods so keeping them.

coverage: 97.635%. first build when pulling c971c06edcc037e478036e1ea003bb1b0412a663 on users/alexpeck/lfunode into 602ec5b45dc8eada78f112221b8a63969316d6f7 on main.


[WIP] Claim/Execute transactions with a higher bounty first

Pull Request Test Coverage Report for Build 2275

44 of 48 (91.67%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.4%) to 76.639%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Scanner/CacheScanner.ts
28
32
87.5%

Files with Coverage Reduction
New Missed Lines
%

src/Util.ts
1
75.57%

Totals

Change from base Build 2266:
0.4%

Covered Lines:
986

Relevant Lines:
1220

💛  - Coveralls


Update IACA and DS certificates for ISO 18013-7 test event

Fixes #340
created new IACA and DS certificates. US is being used as country code and US-MA for stateOrProvinceName. the mDL dataset was updated to be aligned with the new country code.

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Hi,
Looks good but couple of things.

You need to go through the CLA flow
Please squash all commits into just one
Please test this on a real device, verify it works as intended, and include that you did this in the commit message
Please run all unit tests, verify they all pass, and include that you did this in the commit message


[SPARK-9910][ML]User guide for train validation split

Merged build triggered.

Merged build started.

Test build #41417 has started for   PR 8377 at commit fb3b1d6.

Test build #41417 has finished for   PR 8377 at commit fb3b1d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaTrainValidationSplitExample

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41417/
Test PASSed.

I'll take a look.  FYI this may conflict with [https://github.com/apache/spark/pull/8304]

Just a few comments.  Some Scala comments carry over to Java too.  Thanks!  (I like the examples.)

Merged build triggered.

Merged build started.

Thanks for the comments! I have updated the review, ready for second pass.

Test build #41652 has started for   PR 8377 at commit 6443d55.

Test build #41652 has finished for   PR 8377 at commit 6443d55.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaTrainValidationSplitExample

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41652/
Test PASSed.

Merged build finished. Test PASSed.

By "remove the imports," I like @feynmanliang meant to remove the imports which are no longer necessary.  It'd be nice to keep the ones explicitly used in the example code.

Merged build triggered.

Merged build started.

Test build #41665 has started for   PR 8377 at commit 1dd1cd1.

Test build #41665 has finished for   PR 8377 at commit 1dd1cd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaTrainValidationSplitExample

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41665/
Test PASSed.

@zapletal-martin Do you have time to update this time? It would be nice to merge this first before I make a pass to the whole user guide to avoid merge conflicts. Or I can prepare a PR based on yours.

@mengxr, I am planning to work on this in around 5 hours. Unfortunately I won't be able to progress it before that.

No problems. That should work. I will wait for your update before I make a global check:)

Merged build triggered.

Merged build started.

Test build #41776 has started for   PR 8377 at commit dd3a0e5.

Test build #41776 has finished for   PR 8377 at commit dd3a0e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaTrainValidationSplitExample

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41776/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Merged into master and branch-1.5. Thanks!


[Tracker]: Near Edge MVP

Description
This MVP delivery is focused on delivering a feature set into ODH that supports deploying models to near edge locations. This is a top level tracker to provide an single issue entry point for external visibility
This feature set will include but is not limited to:

Define the standard terms that will be used to through this AI Edge initiative to ensure that

#156

Fetch a trained model from a repository: Git, S3, Model Registry
Build a model into packaged into an immutable container with all of it's required dependencies
Test the model in a staging environment to verify that it is working as intended
Deploy the packaged model into a remote repository that will be made available at the Near Edge

https://github.com/opendatahub-io/odh-dashboard/issues/1812 (UX/UI) - Provide a unified user experience for the user to minimize the requirement to understand all of the backend services required to build, deploy and monitor models running in near edge environments

#157

Integration with ACM and ArgoCD to support deploying immutable model containers into near edge environments
Utilize ArgoCD to support the ACM Pull model for disconnected near edge environments
Utilize a GitOps to provide a source of truth for model deployment configuration

#158

Monitor the operational metrics of my containerized model at the near edge.

#159

Support for test a immutable monitor container
Updates to Pipelines required for the MLOps workflow
Integration is other dependency that are required for delivering immutable model containers to near edge locations:

Closing this as the work will be tracked in RHOAIENG-218


newton (N) cannot be used as a unit.

Hi @breathiness,
Thanks for reporting.  Looks like the N newton unit is being overridden by the N excel function.  I will release a fix as soon as possible.

Hi @breathiness,
I resolved the naming conflict  by adding a prefix to all Formulajs functions.  N unit and any other conflicting units should now work as expected in just released v5.2.0

Thank you! The new version is now working properly.

The screenshot made me realize the N syntax color was now incorrect.  So thanks for posting that.  I have fixed it in v5.2.1.
Let me know if you find anything else.  I will close this issue as resolved.


really serious bug in definite initialization checking

This passes the typechecker:
class Foo() {
    if (1==1) {
        return;
    }
    value name = "Gavin";
    shared void bar() => print(name);
}

Ouch!!

Ugh. It gets worse. The typechecker doesn't accept this code:
class Foo() {
    String? name;
    if (1==1) {
        name = null;
        return;
    }
    name = "Gavin";
    shared void bar() => print(name);
}

I have fixed the second problem reported here.

Fixed, finally.


blockOutsidePointerEvents

Is it planned to add the blockOutsidePointerEvents parameter, which contains  in the branch small_improvements ?

if you think it's a good addition, sure

Yes, sometimes a very necessary property.
Thanks a lot!


Reminder: Every second day

I would need the reminder to be set only for every second day.
Could this be included in a future release?:)
The best, Michael

Yes, I'll work on that in the coming weeks.

Works great with 1.3.0. thanks a lot.
Still I think the naming could be improved. It still says 1 reminder/day instead of reminder every 2 days.


MatchPhrasePrefix is deprecated

NEST/Elasticsearch.Net version: 5.0
Elasticsearch version: 5.1.1
Description of the problem including expected versus actual behavior:
Using Elasticsearch 5.1.1 with the MatchPhrasePrefix operator, I get deprecation warnings
Deprecated field [type] used, replaced by [match_phrase and match_phrase_prefix query]
Steps to reproduce:

Create a query using the MatchPhrasePrefix operator

Provide DebugInformation (if relevant):
Examining the generated query, it looks like this
     "match": {                             "navn": {                               "type": "phrase_prefix",                               "query": "Rasta"                        }                          }
The new and improved syntax should be
  "match_phrase_prefix": {    "navn": {                                                        "query": "Rasta"                             }                           }

Nice catch @babadofar looks like we missed this :\
I've opened #2515 which will fix this in master, but debating whether we should back port it to 5.x since it will have to be done in backwards compatible manner leaving irrelevant and obsoleted properties on MatchQuery.


Focus an element by name

Hi,
In some situations I need to focus an input, have you something like?:
<input ng-model="username" focus-name="userInput"/>
<button ng-click="fm.setFocus('userInput');

Thanks in advance.

You can achieve this by doing the following with the existing API:
var el = document.querySelector('[focus-name="userInput"]');
fm.manager.focus(el);

True, true. I will use it.
Many times in SPA's I have to reset the focus to the first element, or focus a specific element. I think to have the posibility of to set focus to specific element would be good.
At the moment I think to extend the manager with focusByName function, something like:
fm.manager.focusByName = function (name) {
    var el = document.querySelector('[focus-name="userInput"]');
    if (el)
        fm.manager.focus(el);
}

Perhaps in the future the manager support it natively ;)


CLI-Command

Suggeston
It would be great when you would integrate an commad for generating an extension skeleton.
In Symfony exists such an command ( generate:bundle).

Seems to me like a request for #468 or drunken-kungfu/pagekit-template-generator which requires #703. So either the author did not respond (and the pagekit team did also not step in as announced) or the pagekit team did not respond and so the author has no chance to solve the issues ...


Add code block long lines wrapping

This change is for the code block style - allowing to wrap long lines.

Looks good to me, thanks a lot! :+1:


SLES import: Support conversion to on-demand

This PR adds:

Support for importing on-demand SLES and SLES for SAP guests. The imported guest is not required to have an active SLES subscription.

OS targets
Eight targets are available for the --os flag, covering all combinations of:

Major version: 12, 15
Flavor: SLES, SLES for SAP
Subscription: BYOL, convert to on-demand

See cli_tools/common/utils/daisy/daisy_utils.go for the new targets.
Noteworthy changes

The on-demand conversion runs in a chroot. This is required since SLES's registration code
fingerprints the instance. Running in a guestfs appliance fails registration.
Similarly, to support this fingerprinting, the Debian 9 import worker includes
the license that will be applied to the final image.

Testing:

Added e2e tests:

Each minor version has an unregistered image that is imported and verified (8 total)
Each major version has one registered image that is imported as BYOL (4 total)

Ran SLES and OpenSUSE tests from ci-daisy-e2e

/test all

There are many "no changes" file included, is it by accident?

There are many "no changes" file included, is it by accident?

Good question -- the message means that it's an empty file. These occur for two reasons:

init.py: These are for Python's package search: https://docs.python.org/3/tutorial/modules.html
Some files in the test-data directory are empty. They're used when testing check_root, which in some cases is only looking for a file to be present.

https://github.com/GoogleCloudPlatform/compute-image-tools/blob/2390aef03cf228dc0dcf477bdfd9f1b552c07a07/daisy_workflows/image_import/suse/suse_import/on_demand/validate_chroot.py#L67

@zoran15  -- Added a debugging tip when the on-demand import tests fail (talked about this offline)


Add rpm for presto-cli - fixes #1882

Adds the rpm-maven-plugin to the package phase to create an RPM that
will install the presto-cli executable jar as presto-cli to
/usr/local/bin. Fixes #1882.

Do we have test for Presto PRM?

Yes, ServerIT.

I wonder if we could have some test for RPM
I think that would be an overkill. We could maybe could have a test that would install this RPM on some docker and the run simple presto-cli --help to see that jar is where it should be and it is executable. However, I am not sure of the value of this test.

I'll see if I can add a test similar to ServerIT in presto-server-rpm.

Tests are added. Left some comments inline seeking other people's inputs on some things.

Locally
./mvnw -pl presto-cli clean install -TC2 -DskipTests -Dair.check.skip-all=true -Dmaven.javadoc.skip=true
...
[INFO] --- rpm-maven-plugin:2.2.0:rpm (generate-rpm) @ presto-cli ---
[WARNING] rpm version string truncated to 335
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.501 s (Wall Clock)
[INFO] Finished at: 2020-06-10T08:43:31+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.2.0:rpm (generate-rpm) on project presto-cli: Unable to query for default vendor from RPM: Error while executing process. Cannot run program "rpm": error=2, No such file or directory -> [Help 1]

am i doing something wrong?

@findepi Oh yes, forgot to mention that the plugin depends on the command rpm being available in your path. I tried the pure Java redlinerpm plugin but it doesn't allow specifying a file path outside the target/classes directory.
See https://github.com/prestosql/presto/pull/3958#issuecomment-640295897 for why I chose to do it this way. If someone can help out with those I can change this to use the redlinerpm plugin which doesn't depend on rpm command.

If someone can help out with those I can change this to use the redlinerpm plugin which doesn't depend on rpm command.

@kokosing can you advise?
This is a must-have for this to be merged.

I then tried to do it similar to how presto-server, presto-server-main and presto-server-rpm are structured but ended up getting some errors with provisio. Apparently adding two modules with provisio makes one of those fail.

What was the problem?

forgot to mention that the plugin depends on the command rpm being available in your path

Can we use docker image with rpm command (aka custom rpm command)?

@kokosing The maven-rpm-plugin doesn't allow defining a custom rpmbuild command.
I could see two other options:

Modify the forked redlinerpm-maven-plugin at https://github.com/airlift/redlinerpm-maven-plugin/ to allow passing a base directory anywhere on the filesystem rather than limiting to target/classes.

Implement similar to how presto-server-rpm is implemented by adding two modules - one for creating a tar.gz from presto-cli artifacts and another for actually creating the rpm from the artifacts.

I've implemented the 2nd approach in a different branch and have opened another PR for the same so that it can be contrasted against the current one. You can find it at https://github.com/prestosql/presto/pull/4025
If the only reason to not use the existing approach is that it will force users to have the rpmbuild command on their $PATH then it can be solved by moving the rpm execution to a separate maven profile and have it run only in CI and releases.

If the only reason to not use the existing approach is that it will force users to have the rpmbuild command on their $PATH then it can be solved by moving the rpm execution to a separate maven profile and have it run only in CI and releases.

We want to have minimal set of prerequisites for running the release process.
IF avoiding having rpm on the PATH requires an additional maven module, THEN we should have an additional maven module.
Yet Another Module approach has lower maintenance price tag attached for (for maintainers and for everyone).

Modify the forked redlinerpm-maven-plugin at https://github.com/airlift/redlinerpm-maven-plugin/ to allow passing a base directory anywhere on the filesystem rather than limiting to target/classes.

Ideally we would want not to maintain fork, but sadly, uptsream had some things broken.
My personal preference is to not add changes in that project and have an additional module here,
with the hope this avoids some maintenance problem in the future, if we need to upgrade/replace
the rpm maven plugin.
I don't feel strongly about this though.
Also, two separate modules may be easier to exclude during local build (see "quick build" in
Presto dev cheat sheet https://gist.github.com/findepi/04c96f0f60dcc95329f569bb0c44a0cd).

I'm closing this in favor of #4025 which is the multi-module pure Java approach.


Translate to Chinese version

Type

Addition

Changes

I like your project very much and am trying to translate it into Chinese version. If you agree, I will continue to contribute the rest.
Supporting Material

Affiliation

No Affiliation.
Checklist

[x] I have read the Contributing guide, and confirmed my PR aligns with the requirements
[x] I have performed a self-review (valid markdown formatting, spelling and grammar)
[x] I have indicated whether I have any affiliation with any software/ services added
[x] I agree to follow the repositories Contributor Covenant Code of Conduct

I am aware that you are diligently endeavoring to sustain this project, hence I deem it necessary to initiate some translation efforts for the purpose of the forthcoming website construction, thus garnering increased attention towards privacy preservation.


Argon2id.ComputeHash requires the hash be exactly 128 bytes long

First of all, I'm not sure why MinHashSize is 93, or why there's a MaxHashSize of 128, seems a bit random to me.
But anyway, ComputeHash only allows a hash of size 128.
Here's the source code:
public static void ComputeHash(Span<byte> hash, ReadOnlySpan<byte> password, int iterations, int memorySize)
{
    Validation.EqualToSize(nameof(hash), hash.Length, MaxHashSize); // Issue?
    Validation.NotLessThanMin(nameof(iterations), iterations, MinIterations);
    Validation.NotLessThanMin(nameof(memorySize), memorySize, MinMemorySize);
    Sodium.Initialize();
    int ret = crypto_pwhash_str_alg(hash, password, (ulong)password.Length, (ulong)iterations, (nuint)memorySize, crypto_pwhash_argon2id_ALG_ARGON2ID13);
    if (ret != 0) { throw new InsufficientMemoryException("Insufficient memory to perform password hashing."); }
}

I think it should instead be:
Validation.SizeBetween(nameof(hash), hash.Length, MinHashSize, MaxHashSize);

Just like in VerifyHash.

Hi, thanks for using Geralt.
This is because of crypto_pwhash_STRBYTES in libsodium, which is 128 bytes. From the libsodium documentation:

out must be a dedicated storage area that’s large enough to hold crypto_pwhash_STRBYTES bytes, but the actual output string may be shorter.

So this is correct. I agree it seems weird, but this is what has to be done when using a span buffer rather than return. The output is padded and can be truncated to remove the padding.
VerifyHash() is a bit different because the output from ComputeHash() isn't a fixed length (if you ignore the padding) and the libsodium function accepts a shorter string. MinHashSize was calculated based on things like the string prefix and encoded parameters for Argon2id but could be smaller to allow Argon2i and non-libsodium strings to be verified if I'm remembering correctly.

@samuel-lucas6 Thanks for responding and creating this repository.
I've set the hash length to 128 and it seems to work correctly. Please can you tell me if I'm using this in the intended way?
Thank you.
public const int HashLength = 128;
public const int SaltLength = 32;
public const int Iterations = 3;
public const int MemorySize = 32_000;
public static readonly Encoding Encoding = Encoding.UTF8;

public static async Task<string> HashAsync(string Input, string? Salt = null) {
    return await Task.Run(() => {
        Span<byte> Output = stackalloc byte[HashLength];
        Argon2id.ComputeHash(Output, Encoding.GetBytes(Input + Salt), iterations: Iterations, memorySize: MemorySize);
        return Encoding.GetString(Output);
    });
}
public static async Task<bool> VerifyAsync(string Input, string Hash, string? Salt = null) {
    return await Task.Run(() => {
        return Argon2id.VerifyHash(Encoding.GetBytes(Hash), Encoding.GetBytes(Input + Salt));
    });
}

Please can you tell me if I'm using this in the intended way?

You don't need to provide a salt to these functions. ComputeHash() randomly generates a salt and VerifyHash() automatically retrieves the salt encoded in the string output. That's why there's no salt parameter, but I'll update the docs to make this clear.
In terms of other comments, you should never have an optional salt when password hashing, there's no need to define your own HashLength and SaltLength constants (in fact, libsodium only supports 16 byte salts unfortunately), and I'd recommend against using a string for storing the password if possible (see if you can use a char array instead so it can be wiped). I'm adding some functions that will assist with secure memory.

Please can you tell me if I'm using this in the intended way?

You don't need to provide a salt to these functions. ComputeHash() randomly generates a salt and VerifyHash() automatically retrieves the salt encoded in the string output. That's why there's no salt parameter, but I'll update the docs to make this clear.
In terms of other comments, you should never have an optional salt when password hashing, there's no need to define your own HashLength and SaltLength constants (in fact, libsodium only supports 16 byte salts unfortunately), and I'd recommend against using a string for storing the password if possible (see if you can use a char array instead so it can be wiped). I'm adding some functions that will assist with secure memory.

Thanks for your response.
If Geralt or libsodium is automatically generating and extracting a salt, how can I make sure a salt is being used? I hashed the UTF8 bytes of the string "abcdef" twice and got:
$argon2id$v=19$m=31,t=3,p=1$zpZmd/2ovOevxTkzNYG85A$EKjR6NL4T63VuV7vhoSinDbGh96scyUVmN1sMn6tWu8
$argon2id$v=19$m=31,t=3,p=1$/E27Byacr5pm/hBYJ2uxAg$+ka+awDRp6Hkyb9VXbESZDKiymRCtMptFnkgZcvH290

Which part of these strings is the salt?
As for the HashLength and SaltLength constants, which constant should I use for the hash size?
MinHashSize = 93 // Why 93?
MaxHashSize = 128 // Will this stay the same?

how can I make sure a salt is being used?

Notice how $argon2id$v=19$m=31,t=3,p=1$ remains the same, whereas the second half changes. That's because of the salt. The salt is the first part of the second half between the two $ symbols, and the actual hash is the last part of the string after the final $. The salt is 16 bytes, and the hash output is 32 bytes.
So the salts are:
zpZmd/2ovOevxTkzNYG85A
/E27Byacr5pm/hBYJ2uxAg

As for the HashLength and SaltLength constants, which constant should I use for the hash size?

You should use MaxHashSize for the reasons I've already explained. As mentioned in the docs, you can either keep the null padding characters or truncate the null characters after converting the bytes to a string.
MinHashSize is purely for validation when verifying a hash; you can ignore it completely if you're only verifying strings produced by ComputeHash(). It's 93 because that's what I calculated the minimum hash size to be for Argon2id hash strings produced by this library. If you're using another library, you could produce shorter strings (e.g., by using a smaller salt).
In this case, you don't need a salt length constant at all. For DeriveKey(), you can use SaltSize.

The salt is 16 bytes, and the hash output is 32 bytes.
So the salts are:
zpZmd/2ovOevxTkzNYG85A
/E27Byacr5pm/hBYJ2uxAg

Thanks for your response again. I'm just a bit confused about this. I thought that was the case, but those are not 16 characters. They are 22 characters.
Since they're UTF8 ASCII characters, one character should equal one byte. So what are the extra 6 characters?

Since they're UTF8 ASCII characters, one character should equal one byte. So what are the extra 6 characters?

These are Base64 characters. Everything is done with bytes and then converted to a string.


Cannot read property 'nativeTrackerId of undefined'

I'm probably missing something really obvious but having performed an rnpm installed and linked the libraries as described, I am getting this message.  Looking at the code it looks like it can't find the native module Google Analytics Bridge.  If I knew more about xcode I could probs work this out myself. Unfortunately.... Any help would be appreciated!

@Bobby-hewitt,
I think its a problem with rnpm & the readme instructions.  I noticed that the MainActivity.java was updated, instead of MainApplication.java as the manual instructions say to do.
In my case here is what I did to resolve the same error for android:
Remove the line import com.idehub.GoogleAnalyticsBridge.GoogleAnalyticsBridgePackage; from MainActivity.java
Add the line import com.idehub.GoogleAnalyticsBridge.GoogleAnalyticsBridgePackage; to MainApplication.java
Add the line new GoogleAnalyticsBridgePackage() under return Arrays.<ReactPackage>asList(
Step 4 of the manual instructions for Android says to update MainApplication.java, but the example shows the class MainActivity:

I'm on rn 0.30

Which version of RN are you using @Bobby-hewitt?
I've also updated the README to reflect more recent changes in RN (with rnpm being migrated to react-native link).

@cbrevik, thanks for the update!...I noticed it Step 4 of Android manual steps still shows public class MainActivity extends ReactActivity {, is that inaccurate?

@marcpechaitis No, that's incorrect. I'll have it fixed!

If you end up here looking for a solution to nativeTrackerId of undefined for your tests, you can use https://github.com/CureApp/react-native-google-analytics-bridge-mock

FTR, faced the same issue, I just did the 5th of Manual Installation iOs of the readme (https://github.com/idehub/react-native-google-analytics-bridge#manual-installation-ios).

Next you will have to link a few more SDK framework/libraries which are required by GA (if you do not already have them linked.) Under the same "Link Binary With Libraries", click the + and add the following:
CoreData.framework
SystemConfiguration.framework
libz.tbd
libsqlite3.0.tbd

In v4.0.0 released yesterday, the tracker should be constructed instead of used statically:
let tracker = new GoogleAnalyticsTracker('UA-12345-1');. So use of nativeTrackerId and setTrackerId has been deprecated.
I think if you changed your code to instead to use the new GoogleAnalyticsTracker class, it will stop this from occurring.
See v4.0.0 release notes.

+1

+1

+1 is really unhelpful. If you are having this problem I suggest looking at my previous comment, and tell me if that works.

Im using the GoogleAnalyticsTracker class but isnt working on iOS yet, Android was always fine.
RN 0.36.0
RN-google-analytics-bridge 4.0.0

I did the manual installation for both (android/ios)

Can you show a small code example to reproduce the error @marlonbrgomes?

@cbrevik thank you for your earlier answer, but I am using iOS and making the change you suggested did not change any behavior. I can comment out GA for now to keep making progress.

Aha, I see where the problem might be @ayercm. I'll take a look at this today

I've published a new version 4.0.2 @ayercm, can you test that version?

Hi @cbrevik , I just had a chance to run a first test where I watched it fail, installed 4.0.2, and watched it pass, so yes this seems to have fixed it. I'll test 4.0.2 elsewhere but at the moment it looks great. Thanks for fixing:-)


admin/enable-tests-for-windows-and-mac

Discovered during #273, MacOS and Windows builds weren't running the full test suite.
Couple of comments. Massive thanks to @hoefling for letting us know how to solve the problem. I have included a "Help from..." tag in the release. I hope that is okay, just want to credit you with the fix.
While these builds now run all defined tests (windows test log and mac test log) they do not run doctests. There were three errors that arose on the windows side of things when enabled -- different outputs from what was written as expected (all of which were dtype differences). Other than that, all good.

[x] Unit tests and/or doctests in docstrings

See comments above

[x] tox -e py39 passes locally
[x] Docstrings and API docs for any new/modified user-facing classes and functions
[x] Changes documented in docs/release.rst
[x] tox -e docs passes locally
[x] GitHub Actions CI passes

collecting ... collected 522 items in https://github.com/zarr-developers/numcodecs/pull/276/checks?check_run_id=2126976275

Hurray!


Add count option to subscriptions list command

This PR adds count option to subscriptions list command.

Coverage remained the same at 100.0% when pulling 89947fba66d85ccd2670fc14e2c902fa7e337871 on fisuda:feature/subscription into c12395c17c596a77adcdeba8d3652e5f629c3760 on lets-fiware:main.


Notebook crashes on large data

Issue Type: Bug
SQL Server
I have large database with millions of record in many tables.

click +Code
select * from table1
it will show top 5000 rows
Click +text
type something (Azure Data Studio will start pausing and late response)
Click +Code
select * from table2
it will try loading data... keeps on trying and crashes

Azure Data Studio version: azuredatastudio 1.9.0 (78a42e1d112ae3231777722b51eaf44f83ddbe55, 2019-07-10T04:31:36.998Z)
OS version: Windows_NT x64 6.1.7601

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz (4 x 3192)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: enabledsurface_synchronization: enabled_onvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.91GB (5.05GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none

Thanks, @rohailiqbal. It may be related to #6392 but let's keep this open for now in case the culprit is different.

We've spent some significant effort improving memory usage/performance for notebooks in the September release (current efforts can be seen in insiders). I tested a large (~60MB) notebook on both with the August (1.10) release and our current insiders release. Here's the difference:
August:

Insiders [i.e. September release soon]:

If you end up seeing any issues in the future with this, I would love to hear about it, as the team put significant effort into minimizing any crashes as much as posible.


How do I trigger an event handler in this.props

I'd like to propagate the location fix to parent elements, but I can't seem to get it right.  Would it be possible to adapt the demo to make that function more clear?

I'm not sure I understand what event you'd like to handle. Can you please elaborate?

The react-geolocation library has an onSuccess event, which allows the parent element to assign a handler to the Geolocation class and use the location data.  Is there an equivalent function in this library?

Oh, I see what you mean. No, this is not currently supported, but would be relatively simple to add. PR would be welcome, if you want to contribute, otherwise I'll try to look into it tomorrow.

:tada: This issue has been resolved in version 2.4.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Adds unit test for project create

What is the purpose of this change? What does it change?

UTs coverage for project create
Was the change discussed in an issue?
No

How to test changes?

make test-coverage

/test v4.2-integration

/approve

/hold

/hold cancel

/retest

/retest

/retest

/test v4.1-benchmark

/test v4.1-e2e-scenarios


Meta Edge inconsistancy

Load Graph Ml file    BUG.graphml.txt
Collapse all nodes
Expand c3
Meta Edge is inconsistent (should be b/w c1 & c4).

Resolved

I am getting an error when applying the exact scenario above. I don't see a linked commit to this issues either.

Similar Bug Found
Add 2 edges between Node d and g
check recursive flag
Collapse c3
collapse all edges
uncheck recursive flag
expand c3
MetaEdges are inconsistent.


False colors after image is filtered with filterWithPipeline

I'm having an issue with image filtering. I want to create a pipeline with 3 filter but after the image is processed, the colors of the image is more pale than it should be. My function applies 3 adjustment to the image but if I set their values to the default ones (as the user did not set any changes for the image), the image is not look like the raw image, it looks pale. This makes the whole picture look wrong.
I want to apply brightness, contrast and saturation adjustment to the picture with filterWithPipeline, as the following code snippet shows:
    class func createImageForPostcard(image: UIImage, intensities: Intensities) -> UIImage {
        let brightnessEffect = BrightnessAdjustment()
        let contrastEffect = ContrastAdjustment()
        let saturationEffect = SaturationAdjustment()

        brightnessEffect.brightness = intensities.brightness
        contrastEffect.contrast = intensities.contrast
        saturationEffect.saturation = intensities.saturation

        let filteredImage = image.filterWithPipeline { input, output in
            input --> brightnessEffect --> contrastEffect --> saturationEffect --> output
        }

        return filteredImage
    }

This is the raw image:

After applying the function with the default intensity values, it looks like this:

I do not understand why the color is modified because I set every intensity to the default value.
I would be really grateful if someone could tell me what should I do differently because I can't find any solution to get the well-filtered image.

I have created a project where the issue can be tested:
https://github.com/domivarga/test-filter


Pathway query

@jordan-strasser has created a sweet SQL query to generate new metabolic pathways. This PR is a work in progress and @Koeng101 is going to refactor the SQL into distinct queries after Jordan writes and confirms that his test case in pathways_test.go works as intended.

Didn’t intend to include that stuff, just didn’t know if it was safe to delete.

From: Tim @.***>
Sent: Tuesday, September 14, 2021 12:20 PM
To: allyourbasepair/allbase
Cc: Jordan Wayne Strasser; Mention
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
@TimothyStiles commented on this pull request.
Did you mean to include the swap and db-shm and db-wal files? This seems like accidental cruft to me.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#pullrequestreview-754336252, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOWZUMKC6CZSBLNKDWU3UB6N5XANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@jordan-strasser tests aren't passing and it looks like the one you wrote fails by default. Could you get the test to work before @Koeng101 takes a look at it?

Also it's safe to delete the cruft. Just delete and push your changes 🚀

This is strange, because all the tests I wrote passed on my computer. Not sure why it’s failing here

From: Tim @.>
Sent: Tuesday, September 14, 2021 12:26:48 PM
To: allyourbasepair/allbase @.>
Cc: Jordan Wayne Strasser @.>; Mention @.>
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
@jordan-strasserhttps://github.com/jordan-strasser tests aren't passing and it looks like the one you wrote fails by default. Could you get the test to work before @Koeng101https://github.com/Koeng101 takes a look at it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#issuecomment-919450089, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOW4CY75QQRMNUAZV4BLUB6OXRANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@jordan-strasser you can check why with "details"
For example:
pathways_test.go:11: Expected: len(result) = 9, Got:  0
--- FAIL: TestGetTotalPathways (0.00s)

Wonder why that is happening. Are the test files all there?

Oh the database isn’t there. Makes sense

From: Koeng101 @.***>
Sent: Tuesday, September 14, 2021 12:39 PM
To: allyourbasepair/allbase
Cc: Jordan Wayne Strasser; Mention
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
@jordan-strasserhttps://github.com/jordan-strasser you can check why with "details"
For example:
pathways_test.go:11: Expected: len(result) = 9, Got:  0--- FAIL: TestGetTotalPathways (0.00s)
Wonder why that is happening. Are the test files all there?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#issuecomment-919457784, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOW5ZCHKKSSCENHE3LPDUB6QFPANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Uh oh. We didn't strip it down yet did we?

(make sure to make backups before this) you can probably just run a query to delete the uniprot rows not associated with the coli genome.

Is that necessary? I think I just didn’t properly commit the database

From: Koeng101 @.>
Sent: Tuesday, September 14, 2021 12:43:06 PM
To: allyourbasepair/allbase @.>
Cc: Jordan Wayne Strasser @.>; Mention @.>
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
(make sure to make backups before this) you can probably just run a query to delete the uniprot rows not associated with the coli genome.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#issuecomment-919460180, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOWY2P2HD3G7DSIZYDXTUB6QUVANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Not necessary strictly speaking, but I'd be willing to bet Tim will require for the merge. If you get it working, I can do the row dropping.

Does this test db need to be smaller to merge? I wanted to use it to test hybrid organisms too once we add more genbank files

From: Koeng101 @.***>
Sent: Tuesday, September 14, 2021 12:45 PM
To: allyourbasepair/allbase
Cc: Jordan Wayne Strasser; Mention
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
Not necessary strictly speaking, but I'd be willing to bet Tim will require for the merge. If you get it working, I can do the row dropping.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#issuecomment-919461733, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOW3LUZ5PKKPG5TQB54LUB6Q6VANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

There's more to be done review wise but for now @jordan-strasser could you fix error handling in pathways.go?

@jordan-strasser I just ran my code coverage tool and it looks like you also needs tests to check for error handling in 'pathways.go'.

Does it give a line number? I just checked the code and it looks like all errors are checked for.

From: Tim @.>
Sent: Thursday, September 16, 2021 11:33 AM
To: allyourbasepair/allbase @.>
Cc: Jordan Wayne Strasser @.>; Mention @.>
Subject: Re: [allyourbasepair/allbase] Pathway query (#23)
@jordan-strasserhttps://github.com/jordan-strasser I just ran my code coverage tool and it looks like you also needs tests to check for error handling in 'pathways.go'.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/allyourbasepair/allbase/pull/23#issuecomment-921146284, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUQJOW4VRLFIKU3E634BUITUCIZ73ANCNFSM5D3HJ6WQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@jordan-strasser I did another round of review. Please make edits and ask as many questions as you need!

@jordan-strasser I did another round of review. Please make edits and ask as many questions as you need!

@jordan-strasser the test database is above ~10mb and you forgot to stop tracking your .swp file. Could you make the test database smaller and delete the .swp file?

I don't see the .swp file anywhere, and it's not possible for me to get the db down to 10mb without destroying the GetTotalPathways function. Just dropped 2 extraneous tables and got it from 44 mb to 40 mb

@jordan-strasser all tests and checks are passing but you still need to write tests to prove that error handling works correctly.

Do I need to specifically handle errors within the test functions? At each error in pathways.go I do log.Fatalf("...%d", err) to handle them internally so I can just have one thing returned


support absolute tag in img src

Hello again! This PR adds support for image tags that start with / (absolute url that resolves to the base path), e.g:

Before, these tags were not rewritten at all in html files. After this PR, they should be resolved correctly to:

Although it may not be the cleanest of fixes (maybe you can find a better way!), all tests are still passing 😄

Hey,
I ran the tests and ran it as well on my projects and it didn't break anything. So I'd say it's good for now :)
Thanks!

@n3d1117 Looked at it with fresh eyes today...the containing path indeed was still buggy.
Should be fixed now though, maybe you check with your project using the main branch in SPM

Looking good now thanks! Tested and working for me too.


Reposition toggle-button for Dark Mode in Docs

Description
Most users nowadays prefer surfing docs with Dark-Mode enabled. But, in the current design, the user has to first scroll down to the bottom & then can access the toggle button to switch between themes, which I believe is a bad UX.

Solution
I propose a solution where we can shift the button to the sticky navbar which will remain fixed even if we scroll down, so that the user can switch between themes anytime they want. Plus, we can also shift the "Built with Nuxt" promo to the sidebar to make to reduce unnecessary space wastage on the viewport.

Final look 👇🏼

P.S. : If this issue is validated & approved, I'd love to contribute. Additionally, I do have a similar design in mind, following the same approach for responsive mode (i.e. mobile view).

@ltagliaferri thoughts on this?

Hi @Neilblaze, the docs site is automatically set up to run dark or light mode based on system preferences. FWIW, we are thinking about replatforming the site in the near to mid-term future. I don't think this is an urgent need as docs are the more important concern, but if you feel strongly about it you can certainly open a PR.

Closed as of now, refer to this!


Type error while training

@ruotianluo Sorry again, after fixing the file problem, I got an error:
Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)
run tools/train.py --cfg configs/fc_rl.yml --id fc_rl
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocotalk_fc data/cocotalk_att data/cocotalk_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Read data: 0.003994464874267578
Save ckpt on exception ...
model saved to ./log_fc_rl\model.pth
Save ckpt done.
Traceback (most recent call last):
File "D:\Stephan\Final project\self-critical.pytorch-master\tools\train.py", line 183, in train
model_out = dp_lw_model(fc_feats, att_feats, labels, masks, att_masks, data['gts'], torch.arange(0, len(data['gts'])), sc_flag, struc_flag)
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "D:\Stephan\Final project\self-critical.pytorch-master\captioning\modules\loss_wrapper.py", line 45, in forward
loss = self.crit(self.model(fc_feats, att_feats, labels[..., :-1], att_masks), labels[..., 1:], masks[..., 1:])
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "D:\Stephan\Final project\self-critical.pytorch-master\captioning\models\CaptionModel.py", line 33, in forward
return getattr(self, '_'+mode)(*args, **kwargs)
File "D:\Stephan\Final project\self-critical.pytorch-master\captioning\models\AttModel.py", line 160, in _forward
output, state = self.get_logprobs_state(it, p_fc_feats, p_att_feats, pp_att_feats, p_att_masks, state)
File "D:\Stephan\Final project\self-critical.pytorch-master\captioning\models\AttModel.py", line 167, in get_logprobs_state
xt = self.embed(it)
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py", line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "C:\Users\ncku_ailab\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1484, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)

May I know the exact versions that you used?

I'm now using 1.8

I think it is the problem of window, failed running in pytorch 1.8 with Window but success with pytorch 1.9 in a linux machine. Thanks anyway!


Redirect opt-out per website

When using things such as riot (matrix client) that rely on the domain to separate logins it gets very annoying to have to turn off ipfs-companion to avoid redirection to the local gateway (which would cause riot to use a completely different account than when it's served from the intitial domain).
It would be nice to be able to blacklist certain domains from being redirected to the local gateway.

This sounds useful and easy to add.
Would appreciate some feedback:

What would be the intuitive UI for adding a website to redirect opt-out list?

A list of hostnames on Preferences screen?
A button in browser action?
How to name this feature? ("Redirect opt-out"?)

I think the preferences screen is suitable, and maybe "Gateway redirect blacklist"?
Maybe while you're at it add an option for whitelist as well? The user would then have radio buttons for choosing between "Filter: None, Blacklist, Whitelist".

When IPFS times out, the user should be prompted to temporarily disable it for the current domain (24 hours?) and be redirected back to the web version.

Some websites should possibly be on the blacklist by default …

*.neocities.org (public; but their nodes have been broken since 2017-11; hosts large number of websites)
*.shiftproject.com (private network; source)
*.mrh.io (private network; source)

WIP in https://github.com/ipfs-shipyard/ipfs-companion/pull/687 – just pushed the first stab at UX, would appreciate feedback, thanks!

Ready for testing: v<IP_ADDRESS>8 (Beta)


[5.1]Remove redundant leading/trailing trivia from StringLiteralExpr

Overview
This PR introduced a bug to set leading/trailing trivia to StringSegment as well. This causes unexpected behavior like below
SyntaxFactory.makeStringLiteralExpr(
    "foo",
    leadingTrivia: .newlines(1),
    trailingTrivia: .newlines(1)
)

Expected

"foo"

Actual

"
foo
"

master: https://github.com/apple/swift-syntax/pull/135

@swift-ci test and merge

@swift-ci Please test


Add support for the Dock Builder.

Support for the dock builder appears to be missing.
A couple of enums seems to have made it in but apart from that support seems to be entirely missing.
Any chance we could see support for the dock builder at some point soon?

The DockBuilder API is still in the "internal" part of the native library (imgui_internal.h). I'm going to close this isuse and keep tracking it over in #81 since this isn't specific to the DockBuilder.

@ohtrobinson  you can use this fork to generate DockerBuilder or use the packaged NuGet directly. I don't guarantee 100% support or uptime of it.
<PackageReference Include="ImGui.SDL.Dockspace.NET " Version="1.89.1" />
                    ImGui.DockBuilderRemoveNode(dockspaceId);
                    ImGui.DockBuilderAddNode(dockspaceId, (ImGuiDockNodeFlags)((int)dockspaceFlags | 1 << 10));
                    ImGui.DockBuilderSetNodeSize(dockspaceId, viewport.Size);

                    uint dock_id_left =
                        ImGui.DockBuilderSplitNode(dockspaceId, ImGuiDir.Left, 0.2f, out var _, out dockspaceId);
                    uint dock_id_down =
                        ImGui.DockBuilderSplitNode(dockspaceId, ImGuiDir.Down, 0.25f, out var _, out dockspaceId);
                    uint dock_id_up =
                        ImGui.DockBuilderSplitNode(dockspaceId, ImGuiDir.Up, 0.55f, out var _, out dockspaceId);

                    uint right;
                    uint left;

                    ImGui.DockBuilderSplitNode(dock_id_up, ImGuiDir.Right, 0.55f, out right, out left);
                    ImGui.DockBuilderFinish(dockspaceId);

This code splits the dockspace in 4 parts as an example:


switch out restore flag

You iiidiot shtimpy! Left in the flag I was testing with so that i didn't have to do full restores :(.

You plum 😀


Support for ARM isn't "limited", it's absent.

The text "The Linux emulator currently has limited support for developer machines running on M1 and M2 chip" suggests to me that the emulator will work - but perhaps with some quirks or missing features - on an M1 or M2 machine. However, there is zero functionality when running this container on an Apple Silicon-based Mac, and apparently no urgency to fix this, going by this comment.
Maybe change "has limited support for" to "does not currently support".

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c3cd845a-d0e8-923d-d950-adc28b7b1a05
Version Independent ID: dfc7ce77-8d3e-f81e-b586-a69254deeef2
Content: Emulator (Docker/local) - Azure Cosmos DB
Content Source: articles/cosmos-db/emulator.md
Service: cosmos-db
GitHub Login: @sajeetharan
Microsoft Alias: sasinnat

@john-manktelow
Thanks for your feedback! We will investigate and update as appropriate.

Hi @sajeetharan,
Could you please look into this, thanks.

@john-manktelow, the page will be updated soon to clarify that the emulator is currently not supported on ARM machines (Apple silicon series or Microsoft's latest ARM devices).


Subscribe to a regex topic pattern in pykafka?

Hello, folks,
please, suggest, if there is a possibility in pykafka to get a consumer, subscribed to the topic pattern (not a single topic) as kafka-python has ?
If not - could you please suggest a workaround for this feature?
Thanks,
Viktor

Consumers in pykafka can only subscribe to one topic at a time, so regex subscription from a single consumer is not supported. It's blocked by https://github.com/Parsely/pykafka/issues/354.
The closest you could get to this in pykafka's current form is something like this:
import re
from pykafka import KafkaClient
expr = re.compile(MY_TOPIC_REGEX)
client = KafkaClient(hosts=MY_HOSTS)
consumers = {}
for topic_name, topic in client.topics.items():
    if expr.match(topic_name):
        consumers[topic_name] = topic.get_simple_consumer()

@emmett9001 thanks a lot!


fix: 出现 tooltip 后点击画布外面的空白区域可能抛出错误

当文档的高度未被  body 元素完全占满的时候，tooltip 显示以后点击“外部空白区域”触发 click 事件的 event.target 可能是 html 元素，执行到该函数会抛出错误  Cannot read properties of null (reading 'parentElement')
👀 PR includes

✨ Feature

[ ] New feature

🎨 Enhance

[ ] Code style optimization
[ ] Refactoring
[ ] Change the UI
[ ] Improve the performance
[ ] Type optimization

🐛 Bugfix

[ ] Solve the issue and close #0

🔧 Chore

[ ] Test case
[ ] Docs / demos update
[ ] CI / workflow
[ ] Release version
[ ] Other ()

📝 Description
🖼 Screenshot

Before
After

❌
✅

🔗 Related issue link

🔍 Self Check before Merge

[ ] Add or update relevant Docs.
[ ] Add or update relevant Demos.
[ ] Add or update relevant TypeScript definitions.

:tada: This PR is included in version @antv/s2-v1.20.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


More words added to the blacklist

An older fork already has these, but did not make a pull request:
https://github.com/ealdent/profanity_filter/commit/33d956b20d36541f8c0aea4838dcff2b67160707
I created my own fork as I may add a few other simple features like exposing custom blacklists and whitelists.

Sorry for all the spam, but apparently the dictionary from the older fork wasn't valid and didn't pass tests.  Everything passes now.

No worries! Thanks for checking into it. I'll take a look!


HS stuck in the migration process

[x] I understand that this is only to report software bugs and not for customer support. Please use nebra.io/helium-support.

Describe the bug
My HS was onboarded with blue app unfortunately in the period of Solana migration, I think that blockchain snapshot didn't included HS as it should. Now, because of this in new helium app I can't see anything, including earnings and other stats. Can someone fix this issue related to my HS.

What miner do you have and what's the animal name?

Also, it sounds like you may not have migrated your wallet properly.
In the black Helium app you need to go to settings, and then click migrate wallet
Also, are you sure you restored the wallet from the old blue helium app too the new black Helium app correctly?

What miner do you have and what's the animal name?

acidic-strawberry-mule

Attempted both things, including resurrecting from the original seed phrase. According to the app, migration is fine.

Also, it sounds like you may not have migrated your wallet properly.
In the black Helium app you need to go to settings, and then click migrate wallet
Also, are you sure you restored the wallet from the old blue helium app too the new black Helium app correctly?

Migration shows as fine, however the wallets I have and the seed phrase resurrects do not match what is shown for the miner in Hotspotty, which is also incorrect as the miner is inactive.

The address for the owner of that hotspot is...
14FkiprdidAQsjpbGyJLBrC6vNAgYRgxKD313sYaqU834pFdz3G
Solana address...
CciPmboL2D9c6T1eQAVdntvAtRZCnsdQfhb2xb4bsSWu
Which says it does not exist.
This means that your wallet hasn't migrated https://explorer.solana.com/address/CciPmboL2D9c6T1eQAVdntvAtRZCnsdQfhb2xb4bsSWu

@jbb-kryo

In the black Helium app you need to go to settings, and then click migrate wallet

Last but not least... On hotspotty it shows you are earning rewards so the hotspot is definitely working fine. You just need to properly migrate the wallet

I've also just seeded/inflated your wallet using the widget here...
https://docs.helium.com/solana/migration/wallet-user/
So it now will show as existing.
But you still need to do the migration thing in the black Helium wallet app as mentioned.
Closing as this will fix the issue

Last but not least, with the new IoT token you have to go to the collectibles area (diamond logo at bottom of the app) and claim the tokens for them to go into your wallet :-)

That wallet showing is really old and not the current wallet. It was switched out way before migration, hence me being even more confused now. That is the problem. I'm now going through my archives for that old wallet. Is my hardware bricked if I can't find that?

I've also just seeded/inflated your wallet using the widget here... https://docs.helium.com/solana/migration/wallet-user/
So it now will show as existing.
But you still need to do the migration thing in the black Helium wallet app as mentioned.
Closing as this will fix the issue

Closing before the person who's raised it has confirmed a fix is really not great. And it leads to this - having to reopen it as it blatantly isn't resolved yet.

If you lost access to the wallet that owns the device there's nothing that can be done other than re-keying the miner (which costs $25 for labour and $50 for reonboarding it)
Bear in mind that previously (before solana migration) you could only transfer a hotspot if it had recent activity. If it didn't the transfer would have failed. In any case it definitely hadn't been transferred before the solana transition... So the only ways are as mentioned above.
In any case, this isn't Nebra support. You even ticked a box saying that this was a bug report and not a support request.
Since this isn't a bug report and isn't related to software development side of things I'll leave this issue closed. If you can't get access to your wallet and you need the miner re-keying you will need to create a support ticket in the link in your first post here.

If you lost access to the wallet that owns the device there's nothing that can be done other than re-keying the miner (which costs $25 for labour and $50 for reonboarding it)
Bear in mind that previously (before solana migration) you could only transfer a hotspot if it had recent activity. If it didn't the transfer would have failed. In any case it definitely hadn't been transferred before the solana transition... So the only ways are as mentioned above.
In any case, this isn't Nebra support. You even ticked a box saying that this was a bug report and not a support request.
Since this isn't a bug report and isn't related to software development side of things I'll leave this issue closed. If you can't get access to your wallet and you need the miner re-keying you will need to create a support ticket in the link in your first post here.

----> Ticket now answered. Thanks.


MasterConnectionPool exhausted

redis use  Sentinel when switch master slave  where cause it
2016-10-11 19:10:26.625 WARN  o.r.c.SentinelConnectionManager.onNodeDown:272 master: <IP_ADDRESS>:7091 has down
2016-10-11 19:10:27.666 WARN  o.r.c.SentinelConnectionManager.slaveDown:286 slave: <IP_ADDRESS>:7091 has down
2016-10-11 19:10:42.937 WARN  o.r.c.SentinelConnectionManager.slaveDown:286 slave: <IP_ADDRESS>:7091 has down
2016-10-11 19:12:18.385 WARN  i.n.c.DefaultChannelPipeline.warn:151 An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
org.redisson.client.RedisConnectionException: MasterConnectionPool exhausted!  Disconnected hosts: [<IP_ADDRESS>/<IP_ADDRESS>:7091]

this is my redisConfig .
Config redisConfig = new Config();
redisConfig.useSentinelServers().addSentinelAddress(StringUtils.split(StringUtils.trim(getServer()), ","))
.setMasterName(StringUtils.trim(getMasterName()))
.setDatabase(getDatabase()).setRetryAttempts(100).setFailedAttempts(100)
.setMasterConnectionMinimumIdleSize(10).setSlaveConnectionMinimumIdleSize(10)
.setMasterConnectionPoolSize(250).setSlaveConnectionPoolSize(250)
.setTimeout(TIME_OUT).setReadMode(ReadMode.MASTER);
redisConfig.setThreads(threadNum);
RedissonClient redissonClient = Redisson.create(redisConfig);

only when I restart my webServer can solve it

How do I reproduce it?

redis use sentinel mode when switch master/slave where cause it.

Can you reproduce it with test like below in comment?
https://github.com/redisson/redisson/issues/550#issuecomment-232905142

yes!  my config is  redisConfig.useSentinelServers()
.addSentinelAddress(<IP_ADDRESS>:26379).addSentinelAddress(<IP_ADDRESS>:26379)
.addSentinelAddress(<IP_ADDRESS>:26379).
master server is <IP_ADDRESS>:7091  slave server is <IP_ADDRESS>:7091
when I switch master/slave , stop <IP_ADDRESS>:7091 and wait one minute start  <IP_ADDRESS>:7091
can cause it

I use redisson with  netty-socketio ，when operations person switch  master/slave can cause it .
this will cause me must to restart netty-socketio server can solve it .

I have reproduced it. Sometimes failover doesn't work after master has switched. I'll investigate it

ok,thanks!  I have reproduced it each time by production environment

@pengfei-xu Which Redis version do you use?

2.4.0

I mean Redis server version

redis-2.8.19-1.el6.remi.x86_64

Please check

thanks!

I have reproduced it again by production environment! @mrniko

First screenshot looks strange. Because it seems like Redisson instance goes shutdown. I don't know any other reason why HashedWheelTimer stopped except this.
As for second screenshot. Is this related to PUBLISH commands only ?

yes,redisson is used for pub/sub .second screenshot is caused by  I switch master/slave redis server。the first screenshot is follow second screenshot。it also caused by switch master/slave.

I used it with netty-socketio.

@mrniko

@pengfei-xu I haven't resolved it yet

thanks! you can reproduce it ?  I use redisson with netty-socketio ，when operations person switch master/slave can cause it .this will cause me must to restart netty-socketio server can solve it .

Could you provide logs from Sentinel nodes too?


Feat/arbitrary message signing structured data/i2387

Try out this version of the Hiro Wallet - download extension builds.

Add support for signing structured data (ClarityValue)

The web wallet receives the ClarityValue serialized in hex format which is then deserialized to display in the UI in a human readable format. The serialized format is finallly hashed and signed when the user clicks 'Sign'
cc/ @kyranjamie @fbwoolf @beguene @He1DAr

@beguene Is there a way to see these screens with the test app?

@landitus Were you able to test with https://stacks-wallet-jznmyg9k2-blockstack.vercel.app/ ?

@landitus Were you able to test with https://stacks-wallet-jznmyg9k2-blockstack.vercel.app/ ?

Yes, thank you. Be sure to check the Figma for the text formatting. It doesn't need to be a  tag as the font should be our normal one. HTML wise it could be better as a list () with definition lists () for title and description). In the design, the definition is lighter (gray) while the description is in the regular text color (black).
Also, beware there's an extra space at the top of the text content which I'm not sure where it comes from.

@beguene I fixed this test on dev if you rebase it you should be ok

Thanks a lot @timstackblock I will check it out.

@landitus I updated the UI with dl dd markup. Let me know if it's fine.

Thanks a lot @landitus, I will try your solution.


Network requests cannot be sent on ios 16.0

What did you do?
Chinese version of iPhone from iOS10 onwards needs to agree to allow networking request pop-ups to send network requests properly.
After I upgraded Alamofire from version 4.7.2 to 5.6.2, iOS 16.0 users are still unable to send network requests after the first install of agreeing to allow network requests. Waiting for about 40 seconds for the second Allow Network Request pop-up to appear and agreeing to it restores the network to normal.
Users with iOS 16.1 and iOS 16.0 and below had no problems, Alamofire 4.9.1 and below had no problems, and Alamofire 5.0 and above only started having problems.
中国版的iPhone从iOS10以后需要同意允许联网请求的弹窗才能正常发送网络请求。
我把Alamofire从4.7.2版本升级到 5.6.2后，iOS16.0的用户在首次安装同意允许联网请求后，仍然无法发送网络请求。等过了大约40秒后弹出第二次允许网络请求弹窗，同意之后网络才恢复正常。
iOS16.1及iOS16.0以下的用户没有问题，Alamofire 4.9.1及以下的版本没有问题，Alamofire 5.0及以上的版本才开始有问题。
What did you expect to happen?
After the Chinese version of iOS 16.0 iPhone agrees to allow networking, the send request can be sent immediately.
中国版iOS16.0的iPhone同意允许联网后，发送请求能立刻发出。
Alamofire Environment
Alamofire Version: 5.6.2
Dependency Manager: Cocoapods
Xcode Version:  13.2.1 (13C100)
Swift Version: Swift4
Platform(s) Running Alamofire: iOS16

出现递归不断地发送网络请求导致的现象。