Required disk size not visible in vm

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind enhancement

What happened:
I’ve created a vm with a datavolume which does cloning from a pvc with resources/requests/storage of 3gb.
but the target pvc in my datavolume has resources/requests/storage set to 10gb which is the size of disk i was expecting the vm to see (from “inside”).
vms shows a 3gb disk instead of 10gb disk
What you expected to happen:
vm to show specified disk size
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

KubeVirt version (use virtctl version): latest stable (v0.39.0)
Kubernetes version (use kubectl version): latest stable (v1.20.4)
VM or VMI specifications:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

the vm spec
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kcli/image: focal-server-cloudimg-amd64.img
    kcli/plan: kvirt
    kcli/profile: ubuntu2004
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
  creationTimestamp: "2021-03-16T09:29:19Z"
  generation: 1
  labels:
    kubevirt.io/os: linux
    special: vmi-migratable
  managedFields:
  - apiVersion: kubevirt.io/v1alpha3
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kcli/image: {}
          f:kcli/plan: {}
          f:kcli/profile: {}
        f:labels:
          .: {}
          f:kubevirt.io/os: {}
          f:special: {}
      f:spec:
        .: {}
        f:dataVolumeTemplates: {}
        f:running: {}
        f:template:
          .: {}
          f:metadata:
            .: {}
            f:labels:
              .: {}
              f:kubevirt.io/domain: {}
              f:kubevirt.io/provider: {}
          f:spec:
            .: {}
            f:domain:
              .: {}
              f:cpu:
                .: {}
                f:cores: {}
              f:devices:
                .: {}
                f:disks: {}
                f:interfaces: {}
              f:machine:
                .: {}
                f:type: {}
              f:resources:
                .: {}
                f:requests:
                  .: {}
                  f:memory: {}
            f:networks: {}
            f:volumes: {}
    manager: OpenAPI-Generator
    operation: Update
    time: "2021-03-16T09:29:19Z"
  - apiVersion: kubevirt.io/v1alpha3
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubevirt.io/latest-observed-api-version: {}
          f:kubevirt.io/storage-observed-api-version: {}
      f:status:
        .: {}
        f:volumeSnapshotStatuses: {}
    manager: virt-controller
    operation: Update
    time: "2021-03-16T09:29:19Z"
  name: focused-ritchie
  namespace: default
  resourceVersion: "111431"
  uid: c43eb934-03b0-41ac-9586-829a3df11e39
spec:
  dataVolumeTemplates:
  - metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      name: focused-ritchie-disk0
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
      source:
        pvc:
          name: ubuntu2004
          namespace: default
    status: {}
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: focused-ritchie
        kubevirt.io/provider: kcli
    spec:
      domain:
        cpu:
          cores: 2
        devices:
          disks:
          - disk:
              bus: virtio
            name: focused-ritchie-disk0
          - cdrom:
              bus: sata
            name: cloudinitdisk
          interfaces:
          - bridge: {}
            name: default
        machine:
          type: q35
        resources:
          requests:
            memory: 512M
      networks:
      - name: default
        pod: {}
      volumes:
      - dataVolume:
          name: focused-ritchie-disk0
        name: focused-ritchie-disk0
      - cloudInitNoCloud:
          networkDataSecretRef:
            name: focused-ritchie-netdata-secret
          secretRef:
            name: focused-ritchie-userdata-secret
        name: cloudinitdisk

there are two pvcs, ubuntu2004 which is the origin with a 3gb request size and focused-ritchie-disk0, which is a clone with a 10gn request size
kubectl get pvc
NAME                    STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
focused-ritchie-disk0   Bound    pv011    30Gi       RWO                           76s
ubuntu2004              Bound    pv007    30Gi       RWO                           14h

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Import Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.import.endpoint: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
    cdi.kubevirt.io/storage.import.importPodName: importer-ubuntu2004
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.restarts: "0"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2021-03-15T18:42:28Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cdi.kubevirt.io/storage.import.endpoint: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:volumeMode: {}
    manager: OpenAPI-Generator
    operation: Update
    time: "2021-03-15T18:42:28Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:pv.kubernetes.io/bind-completed: {}
          f:pv.kubernetes.io/bound-by-controller: {}
      f:spec:
        f:volumeName: {}
      f:status:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-03-15T18:42:28Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cdi.kubevirt.io/storage.condition.running: {}
          f:cdi.kubevirt.io/storage.condition.running.message: {}
          f:cdi.kubevirt.io/storage.condition.running.reason: {}
          f:cdi.kubevirt.io/storage.import.importPodName: {}
          f:cdi.kubevirt.io/storage.pod.phase: {}
          f:cdi.kubevirt.io/storage.pod.restarts: {}
        f:labels:
          .: {}
          f:app: {}
    manager: cdi-controller
    operation: Update
    time: "2021-03-15T18:42:30Z"
  name: ubuntu2004
  namespace: default
  resourceVersion: "40370"
  uid: 5132417f-3a60-4d79-8e81-d72885fc450e
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  volumeMode: Filesystem
  volumeName: pv007
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  phase: Bound

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.clone.token: eyJhbGciOiJQUzI1NiIsImtpZCI6IiJ9.eyJleHAiOjE2MTU4ODcyNTksImlhdCI6MTYxNTg4Njk1OSwiaXNzIjoiY2RpLWFwaXNlcnZlciIsIm5hbWUiOiJ1YnVudHUyMDA0IiwibmFtZXNwYWNlIjoiZGVmYXVsdCIsIm5iZiI6MTYxNTg4Njk1OSwib3BlcnRhdGlvbiI6IkNsb25lIiwicGFyYW1zIjp7InRhcmdldE5hbWUiOiJmb2N1c2VkLXJpdGNoaWUtZGlzazAiLCJ0YXJnZXROYW1lc3BhY2UiOiJkZWZhdWx0In0sInJlc291cmNlIjp7Imdyb3VwIjoiIiwicmVzb3VyY2UiOiJwZXJzaXN0ZW50dm9sdW1lY2xhaW1zIiwidmVyc2lvbiI6InYxIn19.AGCf6cvIhx8l4oaD3Uk2xEOijnhfJumLCMjUpXEs_1zQMl2J8Xm7Yjvs2RZMGtFOe5BxM2FmqcugU2gbjhPM6JC3jmNl27fFukyB2jJAZqCStoKIhO7YlbZREJ8JfwoYJzOMixKgWXmuFLpgtym9aUJJOLCEv5b4C3ENTzimp3kR07TUmIIo6xH8JnTZuRsRZHJ08fR9ZoHfR4LwSZADaiGHMioJ80fEms-Ev5i5M5U-pnveka7YareaqozuEL-oweXfgpPmYKtyKe8Vrb1gXmH8io85zeeode9z32IinSEk5GO9Idxbc8ph-wPrWLRBL5dDX0O8VTKOzZ-lfwKMdw
    cdi.kubevirt.io/storage.condition.running: "true"
    cdi.kubevirt.io/storage.condition.running.message: ""
    cdi.kubevirt.io/storage.condition.running.reason: Pod is running
    cdi.kubevirt.io/storage.condition.source.running: "true"
    cdi.kubevirt.io/storage.condition.source.running.message: ""
    cdi.kubevirt.io/storage.condition.source.running.reason: Pod is running
    cdi.kubevirt.io/storage.pod.phase: Running
    cdi.kubevirt.io/storage.pod.ready: "true"
    cdi.kubevirt.io/storage.pod.restarts: "3"
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.sourceClonePodName: 663a0058-3cbe-429e-b0d5-e38b55f2beb1-source-pod
    cdi.kubevirt.io/storage.uploadPodName: cdi-upload-focused-ritchie-disk0
    cdi.kubevirt.io/uploadClientName: default/ubuntu2004-default/focused-ritchie-disk0
    k8s.io/CloneRequest: default/ubuntu2004
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2021-03-16T09:29:19Z"
  finalizers:
  - kubernetes.io/pvc-protection
  - cdi.kubevirt.io/cloneSource
  labels:
    app: containerized-data-importer
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:pv.kubernetes.io/bind-completed: {}
          f:pv.kubernetes.io/bound-by-controller: {}
      f:spec:
        f:volumeName: {}
      f:status:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-03-16T09:29:19Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cdi.kubevirt.io/storage.clone.token: {}
          f:cdi.kubevirt.io/storage.condition.running: {}
          f:cdi.kubevirt.io/storage.condition.running.message: {}
          f:cdi.kubevirt.io/storage.condition.running.reason: {}
          f:cdi.kubevirt.io/storage.condition.source.running: {}
          f:cdi.kubevirt.io/storage.condition.source.running.message: {}
          f:cdi.kubevirt.io/storage.condition.source.running.reason: {}
          f:cdi.kubevirt.io/storage.pod.phase: {}
          f:cdi.kubevirt.io/storage.pod.ready: {}
          f:cdi.kubevirt.io/storage.pod.restarts: {}
          f:cdi.kubevirt.io/storage.preallocation.requested: {}
          f:cdi.kubevirt.io/storage.sourceClonePodName: {}
          f:cdi.kubevirt.io/storage.uploadPodName: {}
          f:cdi.kubevirt.io/uploadClientName: {}
          f:k8s.io/CloneRequest: {}
          f:sidecar.istio.io/inject: {}
        f:finalizers:
          v:"cdi.kubevirt.io/cloneSource": {}
        f:labels:
          .: {}
          f:app: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"594aa394-70a7-4421-89e4-347c3aed43a8"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:volumeMode: {}
    manager: cdi-controller
    operation: Update
    time: "2021-03-16T09:29:29Z"
  name: focused-ritchie-disk0
  namespace: default
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: focused-ritchie-disk0
    uid: 594aa394-70a7-4421-89e4-347c3aed43a8
  resourceVersion: "113461"
  uid: 663a0058-3cbe-429e-b0d5-e38b55f2beb1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeMode: Filesystem
  volumeName: pv011
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  phase: Bound

In the resulting vm, the disk shows with a 3gb size

I'm being hit by this. And the workaround is to set the PVC size of e.g. the ubuntu2004 OS image to a bigger size (hardcoded). This is not too great for the future - as a code-change is needed for the KCLI project whenever I would like the disk, of whatever KCLI deployed VM/VM's, to be bigger.
Looking forward to a solution to this. From the discussion on Slack I understand that people closely involved in the KubeVirt project have been thinking about an approach that could help alleviate this challenge.
See:
karmab  1 day ago
hello buddies, I m seeing a weird behaviour:
I’ve created a vm with a datavolume which does cloning from a pvc with resources/requests/storage of 3gb.
but the target pvc in my datavolume has resources/requests/storage set to 10gb which is the size of disk i was expecting the vm to see (from “inside”).
vms shows a 3gb disk. Am i mistaken here?

fabiand  1 day ago
Salut. @Maya Rashish might be able to help

karmab  1 day ago
opened issue
karmabkarmab
#5233 Required disk size not visible in vm

Maya Rashish  1 day ago
This is intended -- containerized-data-importer will resize based on the requested size. I guess given the two choices (resize to full PVC size, resize to full requested PVC size), people thought this one would be less surprising

karmab  24 hours ago
i dont understand

karmab  24 hours ago
do i need to do something special to get the behaviour i m looking for?

Maya Rashish  24 hours ago
you could get it by specifying you want 10gb

karmab  24 hours ago
that s what i did

karmab  24 hours ago
in the datavolume/pvc for my vm

karmab  24 hours ago
but the source has its own size

Maya Rashish  24 hours ago
ah I see

Alexander Wels  23 hours ago
So cloning is a blind clone (it could clone any PVC) so it doesn't do a resize afterwards like the other operations like import and upload

Alexander Wels  23 hours ago
We are looking into a mechanism that will allow you to specify operations after a transfer (like resize, partition resize, other guestfish operations, etc) But nothing official on that yet.

karmab  22 hours ago
then it means you are limited by the size of the origin pvc?

karmab  21 hours ago
@Lars Bingchong ^

Alexander Wels  21 hours ago
Right now unfortunately yes.

Alexander Wels  21 hours ago
We could fudge it a little when doing a host assisted clone, however when doing a csi clone or snapshot clone, no pods are involved at all so we need an alternate mechanism to run operations after a clone.

Lars Bingchong  20 hours ago
Fair enough @Alexander Wels - makes sense.
I’m the one experiencing the issue when utilising KCLI to deploy a K3S cluster.
I guess I’m lucky. As I’m using Longhorn behind the scenes and Longhorn apparently is a thin provisioned storage system.
See > https://longhorn.io/docs/1.0.2/terminology/#thin-provisioning

Lars Bingchong  20 hours ago
But the approach you are talking about @Alexander Wels - to take some actions after the disk clone. Sounds great!

Looking forward to it and thank you very much

@awels is this something we are looking into fixing? If so, should it be tracked on kubevirt/kubevirt or CDI?

Thanks. Closing as a duplicate.


masking cardReference with transactionReference until creditcardtoken…

…s become available

This introduces a BC breaking change, and the travis tests don't pass. Furthermore the parameters passed to recurringtransactions are incorrect. Closing for now and if you want to do this correctly at some point then please resubmit.


[Hub Generated] Review request for Microsoft.DataFactory to add version stable/2018-06-01

This is a PR generated at OpenAPI Hub. You can view your work branch via this link.
Contribution checklist:

[ ] I commit to follow the Breaking Change Policy of “no breaking changes
[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

[ ] Ensure to check this box if one of the following scenarios meet updates in the PR, so that label “WaitForARMFeedback” will be added automatically to involve ARM API Review. Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs, all “removals” and “adding a new property” no more require ARM API review.

Adding new API(s)
Adding a new API version
Adding a new service

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If there are following updates in the PR, ensure to request an approval from API Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in stable version
[ ] Removing properties in stable version
[ ] Removing API version(s) in stable version
[ ] Updating API in stable version with Breaking Change Validation errors
[ ] Updating API(s) in preview over 1 year

Please follow the link to find more details on PR review process.

/azp run automation - sdk

We would like to support both object and string type for computeType, coreCount properties, this will enable us to support expressions for these attributes. This follows the same pattern as other properties that support expressions today.

@qianwens could you please review this change?


perf(interval): simplify isPerfect logic

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

3 of 3 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
346

Relevant Lines:
346

💛  - Coveralls


Changes for Release version 1.0.0

hi allen,
thanks for merging my pull requests. but we are sitll missing release version to our library. somehow its still denied for me to push change codes in "your repo". perhaps you should look at https://help.github.com/articles/what-are-the-different-access-permissions/ page. once i have write access to your repo i will take care of release version and we can resolve https://github.com/allenRoyston/ngCroppie/issues/15 issue ;)

ngCroppie v1.0.0 has been released, which fixes issues #4, #12 and #15


nimble/host: Add support for SM debug mode

New syscfg option allows to enable SM debug mode which uses predefined
DH key pair, as per core spec.

@sjanc yeah, we can handle this in separate PR :)


TLS ciphers broken with ioBroker.email

Not sure this might be related to the Docker environment, just wanted to double check here as I am not a Node.js expert:
https://github.com/iobroker-community-adapters/ioBroker.email/issues/17

It seems that supported TLS ciphers are incomplete/outdated. After STARTTLS, the connector is unable to find a common cipher suite:
Error<PHONE_NUMBER>18400:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

I am using https://github.com/buanet/docker-iobroker Docker image, not sure if this is related to outdated Node environment (as I am not a Node expert).
This is what the mailserver actually offers:
# for v in ssl2 ssl3 tls1 tls1_1 tls1_2; do
#  for c in $(openssl ciphers 'ALL:eNULL' | tr ':' ' '); do
#  openssl s_client -connect <IP_ADDRESS>:25 -starttls smtp \
#  -cipher $c -$v <<< "Q" > /dev/null 2>&1 && echo -e "$v:\t$c"
#  done
# done

tls1:	ECDHE-ECDSA-AES256-SHA
tls1:	ECDHE-ECDSA-AES128-SHA
tls1:	ECDHE-ECDSA-DES-CBC3-SHA
tls1_1:	ECDHE-ECDSA-AES256-SHA
tls1_1:	ECDHE-ECDSA-AES128-SHA
tls1_1:	ECDHE-ECDSA-DES-CBC3-SHA
tls1_2:	ECDHE-ECDSA-AES256-GCM-SHA384
tls1_2:	ECDHE-ECDSA-AES256-SHA384
tls1_2:	ECDHE-ECDSA-AES256-SHA
tls1_2:	ECDHE-ECDSA-AES128-GCM-SHA256
tls1_2:	ECDHE-ECDSA-AES128-SHA256
tls1_2:	ECDHE-ECDSA-AES128-SHA
tls1_2:	ECDHE-ECDSA-DES-CBC3-SHA

Any other cipher suites are disabled for security reasons (which is quite common I believe).

To be continued at iobroker-community-adapters/ioBroker.email#17.
Regards,
André.


AppleHealthKit.getRestingHeartRate returning array, not HealthValue

Describe the bug
The interface of
 getRestingHeartRate(
      options: HealthInputOptions,
      callback: (err: string, results: HealthValue) => void,
    ): void

is incorrect, results is Array<HealthValue>
To Reproduce
access it and the result is like:
[{"endDate": "2024-06-18T16:19:00.000+0200", "id": "5CA4B701-A576-4021-BEDE-F84DA2DE0693", "metadata": {"HKWasUserEntered": 1}, "sourceId": "com.apple.Health", "sourceName": "Health", "startDate": "2024-06-18T16:19:00.000+0200", "value": 66}, {"endDate": "2024-06-18T16:07:00.000+0200", "id": "8E2F92B6-8ECC-4597-956B-99F7687A3F01", "metadata": {"HKWasUserEntered": 1}, "sourceId": "com.apple.Health", "sourceName": "Health", "startDate": "2024-06-18T16:07:00.000+0200", "value": 60}]

Expected behavior
the results should be array
Screenshots
If applicable, add screenshots to help explain your problem.
Smartphone (please complete the following information):

Device: Simulator
OS: iOS 17.5
Version [e.g. 22]

Additional context
Add any other context about the problem here.

I've fixed it here https://github.com/agencyenterprise/react-native-health/pull/363


Dataset error reports not being sent?

Michelle says:

I had signed up for alerts related to IOM and our data set broke a couple of weeks ago but I never got any sort of alert. Is there a log of some sort that can check if Canary registered the break and if I’m properly registered to receive notices? I did check my spam and nothing came in there. And our data set it fixed.

IATI Canary logs show that the IoM organisation file had validation errors starting on 23 August 2019. These issues were resolved on 28 August 2019. IATI Canary contacted an @iom.int email address on 23 August 2019 at 7.00am UTC.
In the case of data being inaccessible, IATI Canary sends daily emails until the problem is resolved. Because this was a validation error (rather than data being inaccessible) IATI Canary only sends one email per week until the problem is resolved. So in this case, just one email was sent (since the issue was resolved within a week.)


Shaders

Please can you make a lite version of this shader oh and please make vine shader work for 1.19.60 and above

he has already made a lite version https://github.com/bambosan/BSBE-Lite


[Ray] refine ray dataset integration

What do these changes do?
Remove custom ray dataset subclasses for integration
Check code requirements

[ ] tests added / passed (if needed)
[ ] Ensure all linting tests pass, see here for how to run them

There are two issues about Ray version, could you check them in this PR?

The limitation here seems outdated:

https://github.com/mars-project/mars/blob/df1492c4974bb027e7916ae116b12a340e3abdea/setup.cfg#L75

The minimal version in doc(https://docs.pymars.org/en/latest/installation/ray.html#basic-steps) will raise AttributeError when I run test_convert_to_ray_dataset.

Ray Datasets is in beta in ray 1.8.0, I think we should use ray>=1.8.0 @hekaisheng @qinxuye @fyrestone

Ray Datasets is in beta in ray 1.8.0, I think we should use ray>=1.8.0 @hekaisheng @qinxuye @fyrestone

There are two issues about Ray version, could you check them in this PR?

The limitation here seems outdated:

https://github.com/mars-project/mars/blob/df1492c4974bb027e7916ae116b12a340e3abdea/setup.cfg#L75

The minimal version in doc(https://docs.pymars.org/en/latest/installation/ray.html#basic-steps) will raise AttributeError when I run test_convert_to_ray_dataset.

Could you paste the complete error stack here?

There are two issues about Ray version, could you check them in this PR?

The limitation here seems outdated:

https://github.com/mars-project/mars/blob/df1492c4974bb027e7916ae116b12a340e3abdea/setup.cfg#L75

The minimal version in doc(https://docs.pymars.org/en/latest/installation/ray.html#basic-steps) will raise AttributeError when I run test_convert_to_ray_dataset.

Ci cannot pass, could you please see what has happened there? @chaokunyang

Ci cannot pass, could you please see what has happened there? @chaokunyang

The ci failure seems not related to this pr:
FAILED mars/deploy/oscar/tests/test_ray_fault_injection.py::test_rerun_subtask_describe[fault_config0-fault_cluster0]

Could you take a loot at it? @fyrestone


WorkerThreadPool leaks memory and occasionally crashes

Godot version
v4.2.beta6.official [64150060f]
System information
Godot v4.2.beta6 - Windows 10.0.19045 - Vulkan (Mobile) - dedicated NVIDIA RTX A2000 Laptop GPU (NVIDIA; <IP_ADDRESS>37) - 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz (16 Threads)
Issue description
Adding tasks to the WorkerThreadPool leaks memory and occasionally crashes.
While the memory leak is very reproducible, the crash only occurs very sporadically.
Running the same tasks using dedicated threads or a custom thread pool does not leak memory.
This is not a regression, the same behavior occurs all the way back to v4.0.2.stable.official [7a0977ce2] and presumably earlier.
Steps to reproduce
extends Node2D

func work(data: Array):
	pass

func _process(delta: float):
	if Input.is_mouse_button_pressed(MOUSE_BUTTON_LEFT):
		var data = []
		data.resize(1000)
		data.fill(0)
		WorkerThreadPool.add_task(work.bind(data))

Run a scene containing a node with this script, view the static memory monitor in the Debugger pane, then click the mouse in the game window and watch the memory usage ramp up.
Minimal reproduction project
Trivial to reproduce without a dedicated project.

Confirmed in 4.2-beta6 on Linux.

You have to wait for WorkerThreadPool tasks at some point so that WorkerThreadPool has a chance to clean up any used resources. In this case, use WorkerThreadPool.wait_for_task_completion(). This seems to work without leaks:
extends Node2D

func work(data: Array):
    pass

func _process(delta: float):
    if Input.is_mouse_button_pressed(MOUSE_BUTTON_LEFT):
        var data = []
        data.resize(1000)
        data.fill(0)
        var task = WorkerThreadPool.add_task(work.bind(data))
        WorkerThreadPool.wait_for_task_completion(task)

However, the documentation doesn't directly mention this, so I'll add a documentation tag. Both WorkerThreadPool.add_group_task() and WorkerThreadPool.add_task() documentation should mention that every task should be waited at some point so that any allocated resources can be cleaned up.
I can't repro the crashing, which might be worth making a separate issue if you can repro it. Maybe you have to wait a lot longer for operating system thread, mutex etc. resources to reach their process limit or something like that? The limits tend to be quite high, so in normal use WorkerThreadPool should not crash very easily.

Tested using Godot v4.2.beta (64150060f) - Windows 10.0.19045 - GLES3 (Compatibility) - NVIDIA GeForce RTX 3060 (NVIDIA; <IP_ADDRESS>19) - 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz (16 Threads), it didn't crash too but the memory leak is obvious. I guess the crash may related to OS or something else.

Asking for clarification here, by "every task should be waited at some point" does the task of waiting (WorkerThreadPool.wait_for_task_completion()) itself clear up the resources, or would waiting enough time (i.e not a wait function) make the threads clearup resources ? I'm wondering because having a function which pauses the thread that calls this method seems to negate the benefit of using that thread if the following code does not depend on the task being completed (i.e saving a file)


[Wasm][Runtime] Interpret absolute function pointer in runtime structures

When SWIFT_COMPACT_ABSOLUTE_FUNCTION_POINTER is enabled, relative direct
pointers whose pointees are functions will be turned into absolute
pointer at compile-time.
Based on https://github.com/apple/swift/pull/42094

@swift-ci please test

@swift-ci Please test

@swift-ci Please test Windows platform

@swift-ci Please test Windows platform

@swift-ci Please test

The failing Windows CI seems unrelated.
Failed Tests (1):
  Swift(windows-x86_64) :: Driver/enable_regex_literals_flag.swift


Warning: NaN is an invalid value for the marginLeft css style property.

I think this had been posted in the mbrn project somewhere but maybe we can find the solution for it.
I'm not entirely sure yet when it happens. Just saw this in the console. Will try to isolate the issue and post more information
Stacktrace:
Warning: NaN is an invalid value for the marginLeft css style property.
in span (created by ForwardRef(ButtonBase))
in ForwardRef(ButtonBase) (created by WithStyles(ForwardRef(ButtonBase)))
in WithStyles(ForwardRef(ButtonBase)) (created by ForwardRef(IconButton))
in ForwardRef(IconButton) (created by WithStyles(ForwardRef(IconButton)))
in WithStyles(ForwardRef(IconButton)) (created by ForwardRef(SwitchBase))
in ForwardRef(SwitchBase) (created by WithStyles(ForwardRef(SwitchBase)))
in WithStyles(ForwardRef(SwitchBase)) (created by ForwardRef(Checkbox))
in ForwardRef(Checkbox) (created by WithStyles(ForwardRef(Checkbox)))
in WithStyles(ForwardRef(Checkbox)) (created by MTableBodyRow)
in td (created by ForwardRef(TableCell))
in ForwardRef(TableCell) (created by WithStyles(ForwardRef(TableCell)))
in WithStyles(ForwardRef(TableCell)) (created by MTableBodyRow)
in tr (created by ForwardRef(TableRow))
in ForwardRef(TableRow) (created by WithStyles(ForwardRef(TableRow)))
in WithStyles(ForwardRef(TableRow)) (created by MTableBodyRow)
in MTableBodyRow (created by MTableGroupRow)
in MTableGroupRow (created by MTableBody)

Hey @masbaehr would you mind using the starter template to create a reproducible example?

Hey @masbaehr would you mind using the starter template to create a reproducible example?

Here is an example. Sorry for the delay, had to remove a lot of unneeded code ;)
Also i think i found out why it happens. It happens when defaultGroupOrder is used, and one of the data-sets does not have a value for the field in question.
import React, { useState, useEffect } from "react";
import MaterialTable from "@material-table/core";

function Debug(props) {

  const [data, setData] = useState([{
      id: "1", alpha: "alpha", beta: "beta", gamma: "gamma",
      id: "2", alpha: "alpha", beta: "beta"
  }]);
  const [columns, setColumns] = useState( [
    { title: 'ID', field: 'id', hidden: true },
    { title: 'Alpha', field: 'alpha' },
    { title: 'Beta', field: 'beta' },
    { title: 'Gamma', field: 'gamma' , defaultGroupOrder: 0 }
  ]);

    const [isloading, setisloading] = useState(false);

  return (

            <MaterialTable

            title={"TestTable"}
            columns={columns}
            data={data}
            isLoading={isloading}
            options={{
              selection: true,
              grouping: true,
              defaultExpanded: true,
            }}
            onSelectionChange={(rows) => {

            }}

          />

  );
}

export default Debug;

Here is an example. Sorry for the delay, had to remove a lot of unneeded code ;)
Also i think i found out why it happens. It happens when defaultGroupOrder is used, and one of the data-sets does not have a value for the field in question.
import React, { useState, useEffect } from "react";
import MaterialTable from "@material-table/core";

function Debug(props) {

  const [data, setData] = useState([{
      id: "1", alpha: "alpha", beta: "beta", gamma: "gamma",
      id: "2", alpha: "alpha", beta: "beta"
  }]);
  const [columns, setColumns] = useState( [
    { title: 'ID', field: 'id', hidden: true },
    { title: 'Alpha', field: 'alpha' },
    { title: 'Beta', field: 'beta' },
    { title: 'Gamma', field: 'gamma' , defaultGroupOrder: 0 }
  ]);

    const [isloading, setisloading] = useState(false);

  return (

            <MaterialTable

            title={"TestTable"}
            columns={columns}
            data={data}
            isLoading={isloading}
            options={{
              selection: true,
              grouping: true,
              defaultExpanded: true,
            }}
            onSelectionChange={(rows) => {

            }}

          />

  );
}

export default Debug;

No worries. Would you mind putting this in a Sandbox, per our guidelines?

Hello! If you are still experiencing this issue, please reopen. Otherwise, please provide the required Sandbox as it is just easier on everyone.
I have not noticed this error, so it may be resolved.

Hi, I don't use material-table anymore , but here is the test copied & pasted into the starter template if someone wants to resolve the issue
https://stackblitz.com/edit/material-table-starter-template-zqbtun?file=index.js
Changed version to *.32 , still happens (See Console). I already explained above what triggers it.

Thank you. @Domino987 is this something you're aware of or can I merge that PR from mbrn/material-table?
For some reason this issue sounds familiar (outside of this convo).

@oze4 I bet it's familiar to you haha. If you look at the original issue in material-table #1707, you were the first one to submit a PR (#1759), then it did not get merged because another PR override yours.
Anyway, I can confirm that v3.0.5 fixed the bug.

Perfect, i will close this since it works. Thanks everybody for getting this fixed.


String Literal Type can't be used as index

TypeScript Version:  2.1.3
Code
type X = "a" | "b";

const x: { [key: X]: string } = {
    "a":"value"
}

Expected behavior:
Should compile as X is actually a string.
Actual behavior:
Compile-time error:
An index signature parameter type must be 'string' or 'number'.

Use the in operator. E.g.
type X = "a" | "b";

const zzz: { [key in X]: string } = {
    "a": "value",
    "b": "value 2"     // b is required
}

// to make `b` optional make the keys optional
const zzz2: { [key in X]?: string } = {
    "a": "value",
}

@gcnew Neither of those is quite the same as requiring a single key from the union.

@masaeedu Yes, that's true. Unfortunately picking a single key is not currently possible, AFAIK :(

@gcnew I am looking to construct the type type A = { foo: 10 } | { bar: 10 }. Using the literal types type B = "foo" | "bar" and type C = 10. I feebly remember recognizing some general distributive property of unions over other types in a discussion about invoking unions of functions, but I can't quite figure it out for key-value map types.
At the moment there is no value-level computation involved, this is purely type-level juggling of expressions, so I don't understand what you're saying in the edit. Could you provide an example of what you mean?

@olostan It turns out this is possible, you just need to introduce a type variable:
type X = "a" | "b";
function acceptX<T extends X>(x: {[K in T]: string }) {
    return x
}
const x = acceptX({ "b": "value" })
// x is inferred as { "b": string }

Passing { "c": "value" } or { "a": 10 } is a type error, as expected.


Failed to execute: 'control start' for ...

When I try to deploy or start my app I get this error:
DL is deprecated, please use Fiddle
Starting Node.js application...
Failed to execute: 'control start' for /var/lib/openshift/57e71d342d5271b2ee0000e3/nodejs

nodejs.log:
tarting 4 workers...
Application worker 55025 started...
Application worker 55036 started...
Application worker 55024 started...
Application worker 55037 started...
Got SIGTERM, stopping workers...
All workers stopped, exiting.
nohup: failed to run command `undefined': No such file or directory
nohup: failed to run command `undefined': No such file or directory
nohup: failed to run command `undefined': No such file or directory
nohup: failed to run command `undefined': No such file or directory
nohup: failed to run command `undefined': No such file or directory

Anyway, found it.
I forgot to set start script in package.json

Nop, no I'm getting
Error: listen EADDRINUSE :::8080
    at Object.exports._errnoException (util.js:1036:11)
    at exports._exceptionWithHostPort (util.js:1059:20)
    at Server._listen2 (net.js:1252:14)
    at listen (net.js:1288:10)
    at Server.listen (net.js:1384:5)
    at Object.<anonymous> (/var/lib/openshift/57e71d342d5271b2ee0000e3/app-root/runtime/repo/index.js:12:4)
    at Module._compile (module.js:556:32)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)
    at Module.runMain (module.js:590:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

"EADDRINUSE" means that the port you're trying to bind to is already in use

Please refer to Before raising issues note in the README.


NOMINMAX missing to successfully rebuild for Electron v33.2.0

SerialPort Version
v12.0.0
Node Version
v20.18.1
Electron Version
v33.2.0
Platform
Microsoft Windows NT 10.0.19045.0 x64
Architecture
x64
Hardware or chipset of serialport
No response
What steps will reproduce the bug?
Setup an Vite Typescript Electron project, install serialport and Firmata using npm, run npm run start which runs
electron-forge start
What happens?
While preparing the application, the native dependencies are being rebuilt for the electron-internal Node.js version. This fails because there apparently are syntax errors in the "@serialport" dependency in Firmata.
The output is similar to this issue in the nan project.
I tried Node 18 - 22 and the problem occurred every single time.
The only way I was able to fix it was by manually adding the following to<EMAIL_ADDRESS>      'defines': [
          'NOMINMAX'
      ],

so that it looks like this:
      ['OS=="win"',
        {
            'defines': [
                'NOMINMAX'
            ],
          'sources': [
            'src/serialport_win.cpp'
          ],
          'msvs_settings': {
            'VCCLCompilerTool': {
              'ExceptionHandling': '2',
              'DisableSpecificWarnings': [ '4530', '4506' ],
            }
          }
        }
      ]

What should have happened?
The Firmata / SerialPort dependency should just rebuild for the Electron Node.js version like every other native dependency.
Additional information
Another project fixed this in their bindings.gyp like this (see here), that's why I thought it might fix it for serialport as well.

Same issue!


[pgadmin4] README.md: fix small formatting and typo

What this PR does / why we need it:

fix small typo

Which issue this PR fixes

fix small typo

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Title of the PR starts with chart name (e.g. [pgadmin4])

Thank you for the fixes, but is it possible you bump the chart version a minor bit? Each adjustment will need a new chart version.
Thanks.

Thanks for bringing this to my attention, I made a PR with a Chart version adjustment: https://github.com/rowanruseler/helm-charts/pull/133


WIP: Native yml parsing

Adds the ability to parse yml files with ease 😎
Still todo:

[ ] Import the awk program

Option A: curl it in
Option B: hardcoded string it
Option C: separate file that needs to be installed beside strapped.sh

[ ] Handle " properly in ymls
[ ] Convert remaining straps over
[ ] Update strap generator

Right now, it is working and updated for the unix_utils strap.

before (from straps/unix_utils):
echo_count=$(jq -r '.echo | length' <<< "$user_json")
# ...
phrase=$(jq -r ".echo[${i}].phrase" <<< "$user_json")

after:
echo_count=$(q_count "$user_json" "echo")
phrase=$(q "$user_json" "echo.\\[$i\\].phrase")

Merging now so I can get a jump start on rewriting the straps. We need support for quotes and multi line stuff asap. 👍


Missing Creation of NSG

Need a line before the creation of the NSG rules that creates the NSG.
az network nsg create --resource-group rgname --name vmoracle19cNSG

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 279359a4-139a-ee4b-027b-4f43c538b90f
Version Independent ID: 950014e2-2457-6045-eefd-865bd1c1fb76
Content: Create an Oracle database in an Azure VM - Azure Virtual Machines
Content Source: articles/virtual-machines/workloads/oracle/oracle-database-quick-create.md
Service: virtual-machines
Sub-service: oracle
GitHub Login: @Dbakevlar
Microsoft Alias: kegorman

@chapmantim , thank you for your feedback. We have shared the same with the content author.


Release Google.Cloud.CertificateManager.V1 version 2.3.0

Changes in this release:
Bug fixes

Workaround crashes in gRPC for C++ (commit edf4bda)

Documentation improvements

Corrected information about the limit of certificates that can be attached to a Certificate Map Entry (commit 7219d64)

Created release for Google.Cloud.CertificateManager.V1-2.3.0


Railroads under construction don't render at any zoom level

Style change requested
Railroads under construction don't render at any zoom level
Affected Tag
railway=construction
https://openhistoricalmap.org/way/198190749#map=15/39.8409/-77.2546&layers=O
Where this is visible
You can see why we need this in this location on the map:
https://openhistoricalmap.org/#map=14/39.8253/-77.2236&layers=O
Contrast with this raster tile rendering (starts at zoom level 14 and higher):
http://<IP_ADDRESS>/#14/39.8272/-437.2235

Light gray railroads for "under construction"

lighter gray for railroads under construction on live site:

https://openhistoricalmap.org/#map=15/39.8389/-77.2596&layers=O, 1863

lighter gray for railroads under construction on live site:

https://openhistoricalmap.org/#map=15/39.8389/-77.2596&layers=O, 1863


Incompatible changes in Memrise API

Reported by jonahlash
Im getting the same problem with the 1.2.7 release (and any release I can find)
'Error
An error occurred. Please start Anki while holding down the shift key, which will temporarily disable the add-ons you have installed.
If the issue only occurs when add-ons are enabled, please use the Tools>Add-ons menu item to disable some add-ons and restart Anki, repeat until you discover the add-on that is causing the problem.
When you've discovered the add-on that is causing the problem, please report the issue on the add-ons section of our support site.
Debug info:
Anki 2.1.43 (0fbae6bc) Python 3.8.6 Qt 5.14.2 PyQt 5.14.2
Platform: Mac 10.16
Flags: frz=True ao=True sv=2
Add-ons, last update check: 2022-03-31 17:36:23
Add-ons possibly involved: memrise2anki
Caught exception:
Traceback (most recent call last):
File "/Users/jonahlash/Library/Application Support/Anki2/addons21/memrise2anki/importer.py", line 856, in importCourse
raise exc_info0.with_traceback(exc_info[2])
File "/Users/jonahlash/Library/Application Support/Anki2/addons21/memrise2anki/importer.py", line 151, in run
course = self.memriseService.loadCourse(self.url, MemriseCourseLoader.Observer(self))
File "/Users/jonahlash/Library/Application Support/Anki2/addons21/memrise2anki/memrise.py", line 870, in loadCourse
return courseLoader.loadCourse(self.getCourseIdFromUrl(url))
File "/Users/jonahlash/Library/Application Support/Anki2/addons21/memrise2anki/memrise.py", line 623, in loadCourse
course.title = sanitizeName(courseData["session"]["course"]["name"], "Course")
TypeError: 'NoneType' object is not subscriptable'
Could I change something in the code to fix it?

@wilddom Thank you so much for working on this! You're amazing :)

Hello, i would like to help but i know nothing about classes and have a question.

At line 621 the courseData variable is it the soup variable from the loadCourseData function at 872 ?

Well no, actually it should be the json data of the first level, but the corresponding end point is gone (see getJsonLevelUrl). There is more or less a complete rewrite needed to get the addon working again.

It appears that my "workaround" isn't ideal as there's no way of downloading the media.

The only interesting thing i found is :
https://app.memrise.com/v1.17/learning_sessions/speed_review/?session_source_id=2140476&session_source_type=course
The session_source_id is the course id.
The response is pretty big 691.89 KB and contains 47 learnables which are the course material.
From my testing it always gives the same learnables but the order is different.
Not sure if this contains all the course or only the ones that are learned.
Tested on a different course on which i have no progress gives this response : {     "code": "SPEED_REVIEW_DOESNT_LOAD_MEMRISE" }
The speed_review is the only one that gives a consistent response.
At this point i don't think that i am being useful so i'll stop.

As you figured yourself, the review and learning sessions are not very helpful, because they depend on the user's individual progress. But what could be used instead is the lesson preview:
https://app.memrise.com/v1.17/learning_sessions/preview/?session_source_id=[course_id]&session_source_sub_index=[level_index]&session_source_type=course_id_and_level_index

Oh i totally missed the preview button.
The keys for a valid response are : ['learnables', 'progress', 'session_source_info', 'settings']
If the response is invalid there is only one key : ['detail']
An example for one learnable : ghost bin
An example of the contents of 'session_source_info' : {'source_id': 2140476, 'source_type': 'course_id_and_level_index', 'name': 'Japanese 0', 'learnable_ids_to_course_ids': {}, 'num_due_for_review': 24, 'level_id': 7907421, 'level_name': 'The Basics 1', 'source_sub_index': 1, 'template_id': None, 'parent_source_id': None, 'parent_template_id': None}
PoolId, thing count, column_a, column_b and thingusers are no longer available.

Will there be an updated version soon? Im really desperate to find a way to import memrise courses to my anki.. :/

If you're really desperate, you can do it semi automatically by putting it into a UTF-8 CSV file and importing into anki. I personally do it by copying from memrise into an excel, export to CSV UTF-8, then import. It's a bit of manual work, but it's still magnitudes easier than creating cards manually. Google for youtube guides on how to do it. It's not hard.

I looked through the new API and got something that works for me (i only wanted front/back + audio):
https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).
I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

I looked through the new API and got something that works for me (i only wanted front/back + audio):
https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).
I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

I think everyone would appreciate something like that if you could provide a hack that recovers some functionality until it's fixed one day. Thanks!!

I looked through the new API and got something that works for me (i only wanted front/back + audio):
https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).
I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

I would appreciate a lot for such a version. Thanks!

See  #113

I looked through the new API and got something that works for me (i only wanted front/back + audio):
https://app.memrise.com/v1.18/learning_sessions/learn/?session_source_id=<course_id>&session_source_sub_index=<lesson_index>&session_source_type=course_id_and_level_index

I tried going through the plugin source to update it, but there were several parts that i wasn't sure what they were used for and the plugin does much more than i need (i don't need mems or scheduling information or to update the course) so ended up just writing a separate script to pull the data from the learning_sessions endpoint, which has everything i need (front/back and audio).
I could probably put together a stripped down version that at least pulls audio and the target language + translation data.

Would you mind sharing it please? I only need front/back + audio as well. Thanks!

I only touched memrise.py and importer.py:
memrise2anki.zip
You'll need to set your username and password in importer.py (so assuming your username was "YourUsername" and password was "YourPassword"):
USER_INFO = {
	'username': 'YourUsername',
	'password': 'YourPassword',
}

You'll also want to provide a list of course IDs at the top. I've got a couple of them in there already, so you'll probably want to replace those. The course ID is in the URL for the course:

https://app.memrise.com/course/157425/tagalogwith-audiob-phrases/

The course ID here is "157425":
COURSES = [157425]

You can add multiple courses, just separate them with a comma. You may need to install some of the Python libraries like beautifulsoup4 if they're not installed already. Finally, run importer.py with Python (Python 3). It'll download the course into a .csv file named output.csv and put the mp3s into the out folder.

You can add multiple courses, just separate them with a comma. You may need to install some of the Python libraries like beautifulsoup4 if they're not installed already. Finally, run importer.py with Python (Python 3). It'll download the course into a .csv file named output.csv and put the mp3s into the out folder.

Confirm it works as stated. Thanks!
I'll have to learn a bit of python so it can separate the cards into separate csv files. I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders:
memrise2anki-separate-files.zip

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders: memrise2anki-separate-files.zip

If anyone is on Windows, python 3 and get the below error, modify memrise.py, line 756:
with open(f"{course['name']}.csv", "a", encoding="utf-8") as f:
You need the encoding="utf-8" or you can't write it out.
E:\Memrise patched separate>importer.py
https://app.memrise.com/course/5732759/random/
Traceback (most recent call last):
File "E:\Memrise patched separate\importer.py", line 329, in
startCourseImporter()
File "E:\Memrise patched separate\importer.py", line 307, in startCourseImporter
memriseService.loadCourse(url)
File "E:\Memrise patched separate\memrise.py", line 874, in loadCourse
return courseLoader.loadCourse(self.getCourseIdFromUrl(url))
File "E:\Memrise patched separate\memrise.py", line 679, in loadCourse
level = self.loadLevel(level["learnables"], levelIndex, course_data)
File "E:\Memrise patched separate\memrise.py", line 757, in loadLevel
f.write(str(level))
File "C:\python3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to

I used your 2 courses to test and it only creates 1 output.csv, so I'm assuming both courses were put into the same file.

Yep, they're put into the same file as that was the use case i had. Here's a separate version which should store them into separate files and put the mp3s into separate folders: memrise2anki-separate-files.zip

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/
The csv only has 330 lines instead of 984 words as stated in the course. And it's missing a lot of audio links. Is this an issue with the scripts or memrise is not exposing everything through the API?

Here is an alternative solution in a form of a Chrome extension:
https://github.com/Eltaurus-Lt/CourseDump2022
It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/
It downloads 984 words and 960 audio files (some words in the course don't have audio at all).

Yep, I got the same results (984/960). And I'll read the thread to see how to import into Anki. Thanks!!

Yep, I got the same results (984/960). And I'll read the thread to see how to import into Anki. Thanks!!

Glad to be of help! Please feel free to tell us, if you encounter any problems with that.

Here is an alternative solution in a form of a Chrome extension: https://github.com/Eltaurus-Lt/CourseDump2022
It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

Thanks for the link!
I tried it for a japanese memrise course and it isn't recognizing the japanese words, however, It does download the audio files and reference which audio file goes with which english translated word, which is a big deal when working with a big list of words,
So to extract the words I'm using this other chrome extension (github link)
Let's hope memrise2anki gets fixed soon so I don't need to use 2 extensions, but for now,  I found a way to do this.

Here is an alternative solution in a form of a Chrome extension: https://github.com/Eltaurus-Lt/CourseDump2022
It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

Thanks for the link!
I tried it for a japanese memrise course and it isn't recognizing the japanese words, however, It does download the audio files and reference which audio file goes with which english translated word, which is a big deal when working with a big list of words,
So to extract the words I'm using this other chrome extension (github link)
Let's hope memrise2anki gets fixed soon so I don't need to use 2 extensions, but for now, I found a way to do this.

I tried it for a japanese memrise course and it isn't recognizing the japanese words,

Which course have you been trying to download?
I've tested the extention on all of the official ones and didn't have any problems with them.

Looks like It's some kind of microsoft excel problem. I tried opening the same csv with openoffice and it does correctly display the japanese text, so no issues at all

Yes, microsoft excel does not recognize the utf-8 encoding by default for some reason, although it can be made to work with these files through importing settings. I wanted to recommend to try opening the csv with notepad (that's what I use), but you seem to have found a better solution already.
Anyway, the important thing is that Anki doesn't have such encoding problems, so everything should be working properly.

Here is an alternative solution in a form of a Chrome extension: https://github.com/Eltaurus-Lt/CourseDump2022
It does seem to handle this case:

Could you take a quick look at:
https://app.memrise.com/course/1614/1000-most-common-korean-words/

It downloads 984 words and 960 audio files (some words in the course just don't have audio).

Thanks for the link!
I tried it for a japanese memrise course and it isn't recognizing the japanese words, however, It does download the audio files and reference which audio file goes with which english translated word, which is a big deal when working with a big list of words,
So to extract the words I'm using this other chrome extension (github link)
Let's hope memrise2anki gets fixed soon so I don't need to use 2 extensions, but for now, I found a way to do this.
EDIT: Looks like It's some kind of microsoft excel problem. I tried opening the same csv with openoffice and it does correctly display the japanese text, so no issues at all

Another option is to open a blank workbook with excel then use Data (tab) From Text/CSV and it'll pop up the wizard and it should default to 65001 (UTF-8) then import it properly. Then do Save As -> CSV (Utf-8) and it'll make a proper utf-8 csv that Excel can read.


[Bug] The WITH subquery is abnormal. Procedure. WITH 子查询异常

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
Doris-2.1.3
What's Wrong?
WITH 语法异常
有以下SQL语句
ALTER TABLE dwd.dwd_inv_proj_corp_d DROP PARTITION IF EXISTS p20240928;
ALTER TABLE dwd.dwd_inv_proj_corp_d ADD PARTITION IF NOT EXISTS p20240928 VALUES IN ('2024-09-28');
INSERT INTO dwd.dwd_inv_proj_corp_d PARTITION (p20240928)
WITH t1 AS (
SELECT *
FROM ods.ods_cfhec_pro_inv_com_inf_f_d PARTITION (p20240928)
WHERE del_flag = '0'
)
,t2 AS (
SELECT pro_com_id
,sum(case when gd_category = '01' then (cgbl / 100) else 0 end) AS sum_cg
FROM ods.ods_cfhec_pro_inv_com_inf_gd_f_d PARTITION (p20240928)
GROUP BY pro_com_id
)
SELECT NULL AS proj_md_code
,t1.inv_pro_id                                        AS proj_code          --项目编码
,t1.id                                                AS proj_corp_code            --项目公司编码
,t1.pro_com_name                                      AS proj_corp_name            --项目公司名称(唯一性)
,round(t1.zczj, 2)                                    AS reg_cap                   --注册资金
,t1.register_area                                     AS reg_location              --注册地
,t1.fddbr                                             AS lpr                       --法定代表人
,t1.register_date                                     AS reg_date                  --注册时间
,t1.com_zjl                                           AS corp_gm                   --公司总经理
,t1.zjl_tel                                           AS gm_tel                    --总经理联系方式
,t1.com_cwfzr                                         AS corp_fin_prin             --公司财务负责人
,t1.cw_tel                                            AS fin_tel                   --财务联系方式
,round(t2.sum_cg / 100, 4)                            AS cfhec_group_shr_right     --一公局集团所占股权(%)
,round(t1.zgjjszgq / 100, 4)                          AS cccc_consortium_shr_right --中交内联合体所占股权(%)
,t1.yyzz                                              AS busi_license              --营业执照
,t1.yyzzh                                             AS busi_license_num          --营业执照号
,t1.gszc                                              AS articles_of_corp          --公司章程
,t1.wfgddbmc                                          AS our_side_shrhd_rept_name  --我方股东代表名称
,t1.create_date                                       AS create_date
,t1.update_date                                       AS update_date
,t1.del_flag                                          AS del_flag                  --删除标记
,current_timestamp()                                  AS etl_time                  --etl加载时间
,''                                                   AS source_system             --来源系统
,'ods_cfhec_pro_inv_com_inf_f_d,ods_cfhec_pro_inv_com_inf_gd_f_d' AS source_table              --来源表名
,'2024-09-28'                                        AS etl_part                  --分区字段
FROM t1
LEFT JOIN t2
ON t1.id = t2.pro_com_id
;

当执行 INSERT INTO WITH SELECT 的时候, 居然报错
SQL 错误 [1105] [HY000]: errCode = 2, detailMessage = Syntax error in line 5:
WITH t1 AS (
^
Encountered: IDENTIFIER
Expected

What You Expected?
无
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

设置变量 enable_unique_key_partial_update = true之后，
INSERT 不支持CTE，不确定是否是同一个问题

好像不是，已反馈给社区这个问题了

使用With 语句insert into 的时候， 必须使用 label


Unable to get Plex nor IINA to function w/ Anime4k (MPV is configured and works fine)

I am unable to get Plex nor IINA to function w/ Anime4k. MPV is configured and works fine. I have rebooted my machine after following the guides to setup MPV, IINA, and Plex per the guides for MacOs here on this project's github pages. MPV functions as expected, unfortunately nothing visually happens when I press Ctrl 1,2,3,0 in Plex or IINA.
Desktop:
MacBook Pro
14-inch, 2021
Chip Apple M1 Pro
Memory 32 GB
Startup disk Macintosh HD
Serial number LPW5Y7FFPR
macOS Sonoma 14.5 (23F79)
Media Players and Versions

IINA 1.3.5 Build 141 mpv 0.35.0-419-gf79 FFmpeg 6.0
Plex (MacOS Client) Version <IP_ADDRESS>-ca5000d0
Plex Server Version <IP_ADDRESS>79 (on Synology Diskstation DS918+ w/ DSM 7.1.1-42962 Update 6

Could you confirm the path of your config file for Plex and IINA? IINA should have the option to use mpv config as well.


Create eafc evere

Adding University
EAFC Evere in Brussels using @eafcevere.eu as domain name

@IbraYou lib/domains/be/eafc: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.


support session token for object storage

close #2236

Codecov Report

Merging #2261 (9604eb4) into main (32fbe0d) will increase coverage by 0.10%.
The diff coverage is 44.97%.

@@            Coverage Diff             @@
##             main    #2261      +/-   ##
==========================================
+ Coverage   61.74%   61.84%   +0.10%
==========================================
  Files         138      137       -1
  Lines       25052    25241     +189
==========================================
+ Hits        15468    15610     +142
- Misses       7784     7814      +30
- Partials     1800     1817      +17

Impacted Files
Coverage Δ

cmd/mount.go
61.73% <0.00%> (-0.23%)
:arrow_down:

pkg/object/azure.go
1.69% <0.00%> (ø)

pkg/object/b2.go
2.04% <0.00%> (ø)

pkg/object/bos.go
1.50% <0.00%> (ø)

pkg/object/cos.go
1.36% <0.00%> (-0.02%)
:arrow_down:

pkg/object/eos.go
6.06% <0.00%> (-0.40%)
:arrow_down:

pkg/object/etcd.go
1.90% <0.00%> (ø)

pkg/object/gs.go
2.15% <0.00%> (ø)

pkg/object/ibmcos.go
1.28% <0.00%> (ø)

pkg/object/jss.go
6.06% <0.00%> (ø)

... and 54 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09dabc4...9604eb4. Read the comment docs.


Connect to Warframe account and download inventory

DE doesn't disclose user profike APIs. Theatre obtainable if u decompile game files, but can lead to banana and essentially breaking ToS.
However, the existing tesseract parsing mechanism could be theoretically used for parsing inventory windows screen and autofill information for #14

that'd be pretty cool

This has been discuses before, the reality is with the labels in warframe the ability to do so is unreliable at best. I had done some testing back in the day and got low accuracy as the text is smaller and overlaps with the icons more, thus giving more noise.

Removing due to being unrealistic and cluttering up the issues section


Fine-Grained Personal Access Tokens don't need repo:admin

README.md assumes "classic" PATs. During set up of buildbot-nix, @Erethon and me noticed that it would probably be enough to create a Fine-Grained Personal Access Token with the following permissions for the buildbot GitHub Account:

Repository Permissions for "Webhooks" (read and write).
Repository Permissions for "Commit Statuses" (read and write).
Repository Permissions for "Contents" (read). We are configuring a public repo, so did not have to set this, but I guess that this is what would be needed for a private repo. StackOverflow seems to agree.

The account also needs "Write" Permissions on the repository. This seems to work fine. What's preventing their use is an explicit check in buildbot-nix that guards repository discovery:
https://github.com/Mic92/buildbot-nix/blob/077a60a5d040f7161c50a83d827db77a2a68ed9f/buildbot_nix/github_projects.py#L270-L274
This checking method is incompatible with Fine-Grained PATs. The response looks like this:
{ "permissions": { "admin": false, "maintain": false, "push": false, "triage": false, "pull": true } }

One downside of Fine-Grained PATs is that such a check apparently cannot be implemented.
However, in case buildbot runs into a HTTP 401, it cloud also expose the contents of the response header X-Accepted-GitHub-Permissions in the logs:

To help you choose the correct permissions, you will receive the X-Accepted-GitHub-Permissions header in the REST API response. The header will tell you what permissions are required in order to access the endpoint.

A benefit is that, well, it's true that permissions can be set in a much more fine grained way. It avoids having to give the bot user account admin permissions on the repo(s).
Our workaround was to switch buildbot from a "classic" PAT to a Fine-Grained PAT after it had installed the Webhook. Another option would of course be to patch away the above check.
Here's how that looks in Org settings at https://github.com/organizations/${NAME}/settings/personal-access-tokens/:

@MagicRB is it still on your agenda to convert buildbot-nix into only using the GitHub app for everything?

Yep, I should get to it this weekend

@MagicRB is it still on your agenda to convert buildbot-nix into only using the GitHub app for everything?

@MagicRB you can steal the code for re-newing GITHUB_TOKEN's from an github app from here: https://github.com/NixOS/nixpkgs-merge-bot/blob/1aad9bf9b570889da6555650f82d670fc2f73658/nixpkgs_merge_bot/github/GitHubClient.py#L218

Sounds like #127?

The PR exists, the code is in place, a mess has been created: #156
Next step: refactoring

@lorenzleutgeb #156  is in place, is that enough for your use case? Or do you still desire file-grained PATs?

I believe so. Otherwise feel free to re-open this.


chore: improve error message in modify-windows-iso and README.md

What this PR does / why we need it:
chore: improve error message in modify-windows-iso and README.md
the task expect ISO in some format. Improve the error message to indicate user, that task will not work without correct ISO file. Add command to generate sha256 checksum of the file.
Release note:
NONE

/retest

/retest


[Blog] Optimize OpenSearch Refresh Interval

Describe the blog post your would like to write
Adjusting the refresh interval of an OpenSearch index can improve indexing performance. This blog would be a clear and concise post explaining and demonstrating

What is an refresh and refresh interval
How does refresh frequency affect indexing performance
How to adjust the refresh interval for an index

I already have some of the material created for this blog post at OpenSearch_Refresh_Interval. Including a YouTube video - https://www.youtube.com/watch?v=8uyemEfgcY8&t=3s
I want to turn this into a OpenSearch blog post so it is easier for people to find this information.
What is the title of the blog post?
Optimize OpenSearch index refresh interval
Who are the authors?
Christopher Sharkey, Samit Kumbhani
What is the proposed posting date?
September 13 but I am flexible on date. It shouldn't take me long write the post

@ev2900 - Just checking in on the draft to see how it's coming. If you could provide an update here, I'd appreciate it.

Hi @pajuric I wasn't sure if this idea was approved or now. It sounds like it is!
I'll start the draft today. I should have it done in a few days.

Samit and I drafted the blog post in this Google document
The share settings are configured so any one can comment. Please comment with any changes we should make. If anyone needs edit access you can request access.

@pajuric just wanted to check in. Samit and I finished the draft of the blog post in this Google document
Please let me know if you want to do a technical review of the draft or if I should proceed directly to a pull request

Pull request with the blog post content - Content for Blog Post - Optimize OpenSearch Refresh Interval

The blog post is live at - https://opensearch.org/blog/optimize-refresh-interval/
I am closing out this issue


Refreshing page makes App loose it's state.

When refreshing page with HABITUS2.0-dev on UCloud the app restarts en looses its state.
Investigate whether this can be resolved from within HabitusGUI.
Update: Is being addressed with nohup argument, see #60.
Follow-up step:

[ ] Also modify Shiny code to remember its own state (tick boxes ticked, paths specified, etc...) after been rephreshed.

Hi @vincentvanhees, I have written some code with a proposal for this here.
The dummyApp2_save load buttons.R script includes a shiny app in which you can save/load the options selected (even the file uploaded) to the app. Let me know what you think about this option for HABITUS.

Looks good @jhmigueles
To minimize the number of buttons in the HabitusGUI app. How about implementing it in HabitusGUI as follows?

Auto-save when user goes to next page. For this see observeEvent(input$page_... in myApp where we already do various checks before goin to the next page. Saving the state could be one of them. As a result, we do not need a "Save" button.
The "load" button could then be on each page, but we can name it "Undo" because clicking it will reset the state of the page to the last saved state, which was when we arrived at the page.

Thanks @vincentvanhees
I've made some progress here. The app is able to save the state of most of the input and this is done when going to the next page as you suggested. The app is also able to save the directories selected in previous runs (for now I have only implemented this for the rawaccdir. But I'm finding some challenges:

The conditional panel about the researchgoals get updated when it is generated, so I cannot save the state of it, because everything gets unselected every time the app is run. This is not a major problem for the functioning of the app but the user would miss the proposed pipeline suggestion.
The selected directories are saved and used, but the shinyDirButton would not show the selected directory from a previous run. For this, I have included some text to show up that a directory is selected even if not shown in the button:

After selecting everything, I cannot make GGIR work, I see some warning messages and the output is not generated, but I'm not sure if I'm doing something wrong or this is to be expected. Here what I see in the console:

The conditional panel about the researchgoals get updated when it is generated, so I cannot save the state of it, because everything gets unselected every time the app is run. This is not a major problem for the functioning of the app but the user would miss the proposed pipeline suggestion.

Can you make it in such a way that when previously stored state is loaded the app also triggers the re-generation of the proposed pipeline. Maybe via an observeEvent?

The selected directories are saved and used, but the shinyDirButton would not show the selected directory from a previous run. For this, I have included some text to show up that a directory is selected even if not shown in the button:

Did this work in the dummy app? If not, maybe first try to figure out how we can get it to work there?

After selecting everything, I cannot make GGIR work, I see some warning messages and the output is not generated, but I'm not sure if I'm doing something wrong or this is to be expected. Here what I see in the console:

This reminds me now that the app only works in Linux following the changes I made in December, because I had to include those steps for it to work in UCloud. Maybe it is easier if I try to work on a solution for this as I agree that it would be nice if we could get it to also work in Windows even if it is just for testing purposes.

Can you make it in such a way that when previously stored state is loaded the app also triggers the re-generation of the proposed pipeline. Maybe via an observeEvent?

This is fixed now, the app remembers and uses the researchgoals selected in previous runs.

Did this work in the dummy app? If not, maybe first try to figure out how we can get it to work there?

I have been trying this, but there is no way to modify the shinyDirButton to show the path from previous run, so that's why I included the message right below as you can see in the image above. In the dummy app I used fileInput but did not include any directory selection button. Note this is not a problem for the app, the app uses the directory from previous runs as it is supposed to (and the user can always see the directory selected).

I'm wondering what to do with the config file, should the app load the config file from a previous run or always require the user to upload a config file? What do you think?

Just recap from our meeting:

Saving/loading of state sounds good, please do for all directories and file paths.
We keep forcing user to upload and specify config file, we do not want software to start proposing defaults because the config file is the only step in the App where the user is asked to specify what needs to be done. Skipping this step would almost certainly result in bad output.


Error type detection only worked without type path

In most cases when you return Result<..., ServerFnError<E>> this worked but when you tried Result<..., leptos::ServerFnError<E>> then it didn't.

Thanks! As the change suggests, clearly I intended this and then missed that reference :smile: Good catch.

Happened to me various times. Macro code is very finicky to write.
I think the 2 failing checks are not because of my code.

Thanks!


add Resession.nvim support

Awesome plugin!
resession.nvim is a neovim plugin meant to take away the burden of persisting data without the hassle of dealing with :mksession or writing/parsing json files manually.
it's highly extensible, designed with plugin-data in mind and is already being used for big plugins such as aerial.nvim and overseer.nvim.
One of the reasons why I stopped using harpoon is because of harpoon.json (I've talked more about this in its own issue)
lmk what you think!

I like the idea of adding optional support for resession.nvim. it appears to be a good backend for saving state, so I wouldn't have any problems giving the choice.
I'll take a look at its implementation when I get the chance here soon.
Thanks for the suggestion!

That sounds great!, one more thing regarding resession, it already saves and restores the cursor position It would be pointless if grapple also did that.
I suggest a config option to disable this feature.

one more thing regarding resession, it already https://github.com/stevearc/resession.nvim/issues/2 It would be pointless if grapple also did that.

While that may be true, marks may exist for files that don't have a valid buffer. Grapple could opt to defer cursor tracking to resession, but I think this may be unnecessary coupling. It is probably better to keep cursor state separate for now.
I've added preliminary support for resession. Feel free to try it out with the branch feat_integrations_resession 🙂
To enable grapple as an extension:
require("grapple").setup({
    integrations = {
        resession = true
    }
})

require("resession").setup({
    extensions = {
        grapple = {}
    }
})

You may need to pass in enable_in_tab = true into grapple = { ... }, but I feel like that shouldn't be necessary.


Update hands-free.md

Docs Build status updates of commit 6710b67:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

mixed-reality-docs/hands-free.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


When running discovery scan i get the following error massage.

I run the following : python3 main.py --mode all --tactic discovery
And get the following output:

me too

fixed now


Info about error responses from Viagogo API

Hello,
I've successfully integrated Viagogo API into application, but I have question regarding error codes.
Here is the example of validation_failed response:
{
    "code": "validation_failed",
    "message": null,
    "errors": {
        "seller_listing.event.name": [
            "'name' must not be empty."
        ],
        "seller_listing.event.start_date": [
            "'start_date' must not be empty."
        ]
    }
}

So my question is, will there ever be more than one item in this array of error keys? Eg. Can seller_listing.event.name or any other contain more than one error inside of that array? I really can't reproduce scenario where key will have more than one item in array.
Also @akilburge sorry if this repo isn't for this type of question.
Thanks!

@emsoaldin no problem.

Can seller_listing.event.name or any other contain more than one error inside of that array?

It's definitely possible to have more than one error in these arrays but the majority of scenarios will only have one.


using Appveyor for continuous build

details about the service are here http://www.appveyor.com/
@lbrader you might be interested in this.
@mj1856 can you give this a quick look over?

Hi @bennage, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (christb). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, or work for Microsoft Open Technologies, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

Notice that the build status of this pull request is included. :smiley:

Hmmm..  I see in the build output none of the tests are running.
Skipping assembly (no tests match the specified filter).

I also see /trait "category=Unit", while in our code we have [Trait("Category", "Unit")].   Is it case-sensitive perhaps?

Blerg. The integration tests were failing before. When this passed, I assumed the tests were running.

It's not the case that causing the problem. I'm upgrading to xunit 2 to see if that changes anything.

Okay, I'm seeing passed tests now. I think that xunit 1.9.x was part of the problem.
The traits are not consistently applied across the tests though (and there are a few odd tests).
I'd like to work that out in a future pr. Do you see any other problems?

LGTM


load functions-only prezto modules

Some prezto modules (like archive) don't have a .zsh at all

Thanks for contribution. Nice work!


Could not find module RNQRGenerato

Hello,
I got an issue that I dont find a way to resolve.
Juste tried this module on my expo app and can't make it work.
I've done :
$ npm install rn-qr-generator --save
and
cd ios && pod install && cd ../
But when my app is launch on the iOS simulator, I got this error :
Could not find module "RNQrGenerator".
Did you forget to run "pod install" ?
at node_modules/rn-qr-generator/index.js:6:8 in
at http://<IP_ADDRESS>:19000/index.bundle?platform=ios&dev=true&hot=false&minify=false:478471:57 in
at App.js:28:0 in
at index.js:3:0 in
at http://<IP_ADDRESS>:19000/index.bundle?platform=ios&dev=true&hot=false&minify=false:479139:3 in global code
Any help ?
Thanks

Hello @marchetb .
Currently there is no support for the expo projects.
Thanks

@marchetb can you try
expo install react-native-view-shot
instead of
npm install rn-qr-generator --save ?

Hello @marchetb .
Currently there is no support for the expo projects.
Thanks

Hi @gevgasparyan,
Is there support for expo yet?
Thanks


Feature Request: ui

Is it possible to display author names in a consistent spelling?
In the metadata, the names of the authors are usually written in one of the two ways:
Forename Surname
Surname, Forename
It would be nice if the second variant is rewritten to "Forename Surname" when indexing.

How would this differentiate between lists of authors (possibly in the reversed format) separated by commas? Also, the reversed format isn't standard. EPUBs should have the proper format in the dc:author element, with the reversed format in the opf:file-as property (either as a property in EPUB2 or as a refines in EPUB 3).

Sorry for the late reply...
Yes, you are right. Multiple authors would be a problem, haven't thought about this case. Obviously none of my books has multiple authors (I don't have reference or technical books as ebook...)
The reverse order may be non-standard, but about half of my books are in this form. More than 90% of the books bought in the last 2 years are in this form. Something is going wrong... ;)


Starboard Edit Event reserved Word

manager.on('starboardEdited', (old, new) => {
message.channel.send(Starboard (channel ${new.channelID}) edited !)
});
"new" has an error "Identifier expected. 'new' is a reserved word that cannot be used here."

Hey, that's because the keyword "new" is a word reserved by Javacript to create a new instance of a class (see here). Rename your variable and it should work, I will update the documentation.
Example :
manager.on('starboardEdited', (oldStarboard, newStarboard) => {
     message.channel.send(`Starboard (channel ${newStarboard.channelID}) edited !`)
});


Swift needs a namespacing solution

Swift replicated the worst parts of Objective C, including the lack of namespacing for files within the same project.
We can either support user-provided prefixes for all classes and protocols, generate a separate framework so it can be imported and therefore have a namespace, or both.

Done!


Support relative import paths.

Resolve relative import paths to full import paths as part of pre-processing. That way, the rest of the code only has to deal with one type of import path.
Fixes #2.
There are many, many variations of style for how to implement this. I don't think this implementation is final or ideal, but it resolves the issue. I'd rather submit this for review now, rather than make the diff bigger before getting any feedback.
Note, this is a slightly breaking change for the github.com/google/godepq/pkg API, but according to godoc and sourcegraph there's no one else currently using it.

Thanks for fixing this! Overall approach is good, but I have some suggestions to make the change more library-oriented.

Hey @timstclair, please don't be offended, but in order to make most efficient use of everyone's time, I'll close this PR and let you take it.
I made this PR because issue #2 had the label "help wanted" and this issue was easy for me to fix (I'm very familiar with this subject), and I wanted to help out.
But as I mentioned in the PR description:

There are many, many variations of style for how to implement this. I don't think this implementation is final or ideal, but it resolves the issue.

Given how trivial it is to fix the issue and the many different ways it can be done (i.e., it's quite open ended and requires knowing how you want to evolve the API, etc.), I'd rather let you do it quickly yourself rather than incurring the overhead and latency of 2 way communication over github comments and code review. You can use this PR as a reference.
Thanks!

I'll reply to your inline comments here.

I don't think this API change is necessary. The graph construction already works with relative imports, it's just the querying (here) that's broken.

I agree it's not neccessary.
Let me elaborate on the options I think you have, and which ones I think would work better, and of course I'm happy to hear counter-arguments.
Essentially, you have a design decision with three choices here:

Make the library be able to work both with full import paths ONLY, and have the command always resolve relative import paths to full and pass those to the library (the general direction I went with).

The library is simpler and less capable, and the command has to do a little more pre-processing work.
Since the library can do fewer things/accept fewer types of input, it's also easier/simpler to use.

Make the library capable of dealing with both full import paths and relative import paths as input, but it always converts relative import paths right away and uses full import paths internally and for querying.

The library is more complex and powerful (i.e., able to accept a wider range of inputs).
The conceptual model of the library is still simple (each package is always represented as a full import path, which is guaranteed to be unique).

Make the library capable of dealing with both full import paths and relative import paths as input, but don't convert relative import paths, use them directly internally.

I think this can be more confusing as an API. Now, a single package can be potentially referred to by its full import path or some relative path when it comes to querying. Are both okay? Which one should be used?

So, I think it's better to go with either option 1 or 2, but not 3, because 3 leads to a more confusing and difficult to use API imo. What do you think?
Between 1 and 2, I'm more indifferent, but I still prefer 1 slightly because I think it's nicer to always resolve relative import paths right away and always use those for presentation. I think it's more clear to always see full import paths in the output. But this is more of a personal choice.

it's just the querying (here) that's broken.

It's also here (and any other code written by others that uses the library).

I'd rather let you do it quickly yourself rather than incurring the overhead and latency of 2 way communication over github comments and code review.

Sure, thanks for the starting point.
I was envisioning something like your second point.


Will Statemint be registered in BIP-0044 coin types?

Should parachains like statemint be registered in BIP-0044 coin types?
This question can be applied to any parachain, once non-standalone parachains/parathreads are ephemeral as it can lose an auction and consequently be "kicked" from polkadot shared security. Should parachains like Statemint be registered in BIP-0044? If they don't, how users will store assets in their third-part wallets and do transfers?
Links

BIP-44: Multi-Account Hierarchy for Deterministic Wallets
Registered coin types for BIP-0044

I'm not sure what you mean by "parachains like Statemint" or "non-standalone parachains".
Addresses on Statemint should be usable on other parachains in the network, especially as more common good parachains get rolled out, like Staking, Elections, and Governance. One address should be able to have state on multiple chains. Using unique BIP44 paths for each one would result in different key pairs for each one.
I do think that wallets/applications will need to re-evaluate some of their assumptions, as a multi-chain application paradigm will be different to a single chain paradigm. As applications/balances will be spread across multiple chains, it will make sense for addresses to exist in the context of multiple chains.

I'm not sure what you mean by "parachains like Statemint" or "non-standalone parachains".

@joepetrowski not sure if I'm using the right words, but by standalone chain I mean Bitcoin, Ethereum, etc, chains that will continue existing and operating independently of polkadot network, however I know they still need a bridge parachain/parathread like snowbridge to continue existing in polkadot network. By non-standalone I mean parachains like statemint that if don't have a reserved slot, stop to operate completely.

As applications/balances will be spread across multiple chains, it will make sense for addresses to exist in the context of multiple chains.

Currently Polkadot BIP-0044 coin id is 0x80000162 and Kusama is 0x800001b2, so do you think Statemint addresses will use the same base coin id as polkadot?

Addresses on Statemint should be usable on other parachains in the network

It maybe a quite challenge too, because if I understand correctly, Polkadot doesn't make any assumption about how a parachain operates, it means that one parachain can implement a UTxO based model to track balances, or even not follow Polkadot's address format.

I do think that wallets/applications will need to re-evaluate some of their assumptions, as a multi-chain application paradigm will be different to a single chain paradigm

I think this standard must be defined by the Polkadot community, not by wallets and applications, for example polkadot have its key derivation path documented here. But I didn't find any reference to parachain's accounts in this derivation standard, is there any discussion about that going on behind the scenes?

Is not clear for me if statemint mnemonic path derivate from <mnemonic phrase>//statemint or is this rule applied only to kusama/polkadot, which are different networks not parachains in the same network

We never enforce anything about key derivation on-chain, and soft derivations always bring related key attack worries, so..
We should prevent cross chain related key attacks by using context in the signature, not derivation, maybe good reasons exist for doing cross chain signatures, like airdrops, voting with staked dots on a parachain by providing a merkle proof into relay chain state, etc.
We've envision an extremely wide range of chains running on polkadot, so really nobody should feel too hemmed in by what others do.  In the short term, I suppose "standards" here serve mostly to help developers avoid thinking about details which they'd consider distracting.
I do however ask that folks proposing standards read about ZCash's unified addresses because they deal specifically with multiple wallets supporting only a subset of valid ZCash transactions.

https://github.com/zcash/zips/issues/470
https://github.com/zcash/zips/issues/478
https://github.com/zcash/zips/pull/485

The genesis hash is part of the signature payload so we don't have replay attack here (unless some parachain choose to not include it).
Requiring different key derivation path is 100% going to cause many UX burdens and make crosschain interaction complicated (e.g. user make tx on chain A to transfer asset on chain B to chain C, now we need a way to somehow proof the ownership on chain B)


fix: scalar and nan stacking

Goals:

Fixes #1085

Does this issue close any open ticket?

@Jackmin801 do we know if this helped with the TorchDataset performance?

Nope. It does not improve TorchDataset performance on Image DocumentArrays.
But it does show why stacking Image is so expensive. We stack nans uncecessarily.


Add JVM memory limit

The JVM doesn't respect the Kubernetes memory limit since the container and JVM cannot detect this limitation. When Jenkins exceeds its memory it will eventually get killed by Kubernetes. This can be prevented by setting a JVM memory maximum.
See issue #99

Thanks @jacobfederer!


refactor: move CountDocument logic into collection API

Description
What is changing?

Delete CountDocument operation class
Move logic into collection.countDocuments

Is there new documentation needed for these changes?
What is the motivation for this change?
The subclass is unneccessary and make typing difficult. See: https://github.com/mongodb/node-mongodb-native/pull/4112#discussion_r1630101527
Double check the following

[x] Ran npm run check:lint script
[x] Self-review completed using the steps outlined here
[x] PR title follows the correct format: type(NODE-xxxx)[!]: description

Example: feat(NODE-1234)!: rewriting everything in coffeescript

[x] Changes are covered by tests
[x] New TODOs have a related JIRA ticket

@baileympearson looks good to go?


"iat" validation failure - timestamp value in miliseconds but expects seconds

I got an issue with the iat value I get back from my Indentity Server (WSO2 in my case).
oidc_proto_validate_iat: "iat" validation failure<PHONE_NUMBER>311): JWT was issued more than 600 seconds in the future

As you can see the value is in miliseconds and I think mod_auth_openidc expects the value to be in seconds (not sure though). I already tried raising OIDCIDTokenIatSlack significantly, but the value overflows (too large) resulting in a negative number.
Is there a way to set it to miliseconds instead? Or disable the iat value check?

the spec requires iat to be specified in seconds, see: http://openid.net/specs/openid-connect-core-1_0.html#IDToken, so that's a bug on the OP side


Adding Section 6 Agreement Execution with two new images

1st Proposal for Agreement Execution (representing the processing layer)
Does not contain the DLT requirements (separate PR)

Have at it!

@Therecanbeonlyone1969 - Question on "[R32] Application/s providing (commercial) transaction functionality such as billing to (commercial) counterparties, and reare, therefore, external with respect to the BPI, MUST be independent of any BPI."
Does it mean that there could never be a baseline-native application ? For example -  a baseline-native CRM  where BPI and external application are in fact the same platform ?

@humbitious - What do you think of the following requirements:
[RXX] A workgroup MUST have at least one administrator.
[RXX] A workgroup MUST have at least one security policy.
Note that a security policy consists of authentication and authorization rules for the workgroup participants. Note also that one or more workgroup administrators define the workgroup security policy.

@Therecanbeonlyone1969 - Question on storage capability requirements in section 6.6 BPI Transaction Lifecycle.
Are those requirements for external applications' storage?

@Consianimis @kthomas ... any more comments/concerns or can we merge after tomorrow's meeting?


How could PresEx work "within" VPR?

Discussed last week at VC-API WG

Related issue: https://github.com/w3c-ccg/vp-request-spec/issues/7