vSphere: DAX not supported by filesystem

Hi folks, I'm trying to use PMEM-CSI with a 2 node OpenShift 4.6.9 cluster. I wasn't able to install PMEM-CSI using the operator. It seems caused by the OpenShift privilege issue mentioned in https://github.com/intel/pmem-csi/issues/652.
Then, I installed PMEM-CSI from the source(devel branch) by adding the OpenShift special permission to deploy/kubernetes-1.20/pmem-csi-lvm.yaml(mentioned in the example in #652).
During the installation, I cannot setup the certificates by following the install.md:
$ curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o _work/bin/cfssl --create-dirs
$ curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o _work/bin/cfssljson --create-dirs
$ chmod a+x _work/bin/cfssl _work/bin/cfssljson
$ KUBCONFIG="<<your cluster kubeconfig path>>" PATH="$PWD/_work/bin:$PATH" ./test/setup-ca-kubernetes.sh

The setup-ca-kubernetes.sh returns following error because the .pem certificate file does not exists:
[root@aep0 pmem-csi]# ./test/setup-ca-kubernetes.sh
realpath: /home/tomz/container/openshift/pmem-csi/pmem-csi/_work/pmem-ca/ca.pem: No such file or directory
realpath: /home/tomz/container/openshift/pmem-csi/pmem-csi/_work/pmem-ca/ca-key.pem: No such file or directory
Generating Certificate for 'pmem-registry'(NS=pmem-csi) ...
{
    "CN": "pmem-registry",
    "hosts": [
        "<IP_ADDRESS>",
"pmem-csi-scheduler", "pmem-csi-scheduler.pmem-csi", "pmem-csi-scheduler.pmem-csi.svc",
"pmem-csi-intel-com-scheduler", "pmem-csi-intel-com-scheduler.pmem-csi", "pmem-csi-intel-com-scheduler.pmem-csi.svc",
"pmem-csi-metrics", "pmem-csi-metrics.pmem-csi", "pmem-csi-metrics.pmem-csi.svc",
"pmem-csi-intel-com-metrics", "pmem-csi-intel-com-metrics.pmem-csi", "pmem-csi-intel-com-metrics.pmem-csi.svc",
        "pmem-registry"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    }
}
2021/02/04 16:19:11 [ERROR] need a CA certificate (provide one with -ca)
Failed to parse input: unexpected end of JSON input
Generating Certificate for 'pmem-node-controller'(NS=pmem-csi) ...
{
    "CN": "pmem-node-controller",
    "hosts": [

        "pmem-node-controller"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    }
}
2021/02/04 16:19:11 [ERROR] need a CA certificate (provide one with -ca)
Failed to parse input: unexpected end of JSON input

The _work directory only contains bin, I couldn't find any resource to get pmem-ca from the installation guide.
ls -l _work/
total 0
drwxr-x--- 2 root root 36 Feb  3 15:38 bin

So I comment out the StatefulSet pmem-csi-intel-com-controller-0 in pmem-csi-lvm.yaml.
Before the deployment, I use sudo vgcreate --force ndbus0region0fsdax /dev/pmem0 && sudo vgchange --addtag fsdax ndbus0region0fsdax to create VG since I deployed the OpenShift cluster on vSphere and ndctl doesn't support label storage area in ESXi. This is also mentioned in the PMEM-CSI's GCE example: https://github.com/intel/pmem-csi/blob/devel/examples/gce.md#installing-pmem-csi.
vgdisplay output on the CoreOS worker nodes:
$ sudo vgdisplay
  --- Volume group ---
  VG Name               ndbus0region0fsdax
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  10
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <50.00 GiB
  PE Size               4.00 MiB
  Total PE              12799
  Alloc PE / Size       0 / 0
  Free  PE / Size       12799 / <50.00 GiB
  VG UUID               drv3Pg-2cFx-IFMD-YY2h-kib3-BsyJ-LPxSG1

Then, I labeled all worker nodes and deployed PMEM-CSI and the following pods are running normally:
$ oc get nodes --show-labels
...
osc-nkmp7-worker-cn5f5   Ready    worker   47h   v1.19.0+7070803   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=osc-nkmp7-worker-cn5f5,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,pmem-csi.intel.com/node=osc-nkmp7-worker-cn5f5,storage=pmem
osc-nkmp7-worker-phzst   Ready    worker   46h   v1.19.0+7070803   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=osc-nkmp7-worker-phzst,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,pmem-csi.intel.com/node=osc-nkmp7-worker-phzst,storage=pmem
...

$ oc create -f kubernetes-1.20/pmem-csi-lvm.yaml
$ oc get pods -n pmem-csi
NAME                            READY   STATUS    RESTARTS   AGE
pmem-csi-intel-com-node-qpghw     3/3     Running   0          21h
pmem-csi-intel-com-node-zkksc     3/3     Running   0          21h

Create SC and PVC:
$ oc create -f https://github.com/intel/pmem-csi/raw/devel/deploy/common/pmem-storageclass-ext4.yaml
$ oc create -f https://github.com/intel/pmem-csi/raw/devel/deploy/common/pmem-storageclass-xfs.yaml
$ oc get sc
NAME               PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
pmem-csi-sc-ext4   pmem-csi.intel.com             Delete          Immediate           false                  26h
pmem-csi-sc-xfs    pmem-csi.intel.com             Delete          Immediate           false                  26h
thin (default)     kubernetes.io/vsphere-volume   Delete          Immediate           false                  47h

$ oc create -f ./pvc-hazelcast.yaml
$ oc get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pmem-csi-pvc-xfs-hazelcast   Bound    pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe   20Gi       RWO            pmem-csi-sc-xfs   21h

Create application pod with node affinity spec:
...
spec:
  affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: storage
                operator: In
                values:
                - pmem
...

but I can only see one of the worker nodes has /dev/ndbus0region0fsdax/ device after the application pod starts. I attached to my application containers and found that pmem_map_file() returns not PMEM although I can see the device mount through df. It seems my application pod is scheduled on the other node.
Do you have any clue what I missed? Or is this related to the controller StatefulSet? Thanks!

Thanks for the detailed report. @avalluri : can you look into this?

The setup-ca-kubernetes.sh returns following error because the .pem certificate file does not exists:

The certificate/secret creation scripts are more centric to out test cluster. #885 should fix this.

The certificate/secret creation scripts are more centric to out test cluster.

That depends. We still need them for creating certificates in advance for the scheduler extensions. Users can also do that differently of course, but those scripts are what we document.
@avalluri have you also looked into the issue of starting pods on the right nodes?

@avalluri have you also looked into the issue of starting pods on the right nodes?

Not really, I doubted the reason could be that the scheduler extension(controller) was not running that might have led to pick the wrong node.  Though. other possible reason could be the manual LVM volume group creation is missing in one node, but i presume not the case.

Thanks! I will try this fix and inform you the update.

@Tianyang-Zhang : regarding bringing up the statefulset pods on each node - are you sure that LVM was set up on all nodes that you have labeled for the driver?
And another, unrelated question: are you working for Intel? In that case we are also available on internal chat, if you prefer that.

As you can see above, the cn5f5 node has Alloc PE size 20.00 GB, which matches what I have in the PVC yaml file.

That matches what you quoted earlier about one volume being created:
$ oc get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pmem-csi-pvc-xfs-hazelcast   Bound    pvc-f5c8fc7d-93a4-49bb-9f46-9e9d72390e06   20Gi       RWO            pmem-csi-sc-xfs   14m

If you look at the full output for volumes (kubectl get -o yaml pv), then it should show that there is one which has cn5f5 as topology.

But for phzst node, the allocated PE is 0.

That looks fine to me. Volumes only get created on one node.

Additionally, I got these controller log:
... pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d ...  'Warning' reason: 'VolumeFailedDelete' not implemented

This is another persistent volume. The error looks a bit odd.
On which node does pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d exist (topology in -o yaml output) and is this error from the provisioner running on that node or some other node?

Unfortunately, I'm not working for Intel. Please let me know if you prefer any other chat, I can also show you the entire flow I did via screen-share/video if you need. Thanks!

I was just wondering. Let's stick with remote debugging for now and we switch to a life session if we get stuck.

Additionally, I got these controller log:
... pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d ... 'Warning' reason: 'VolumeFailedDelete' not implemented

This is another persistent volume. The error looks a bit odd.

Sorry, I forgot to mention that I reset the environment when applying the CA fix. I just updated the content in the issue description. The PVC I have right now is:
$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS      REASON   AGE
pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe   20Gi       RWO            Delete           Bound    hazelcast/pmem-csi-pvc-xfs-hazelcast   pmem-csi-sc-xfs            21h

But I cannot find where is pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d exist. Here is the out put of kubectl get pv and the full output of kubectl get -o yaml pv, I can only see pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe:
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS      REASON   AGE
pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe   20Gi       RWO            Delete           Bound    hazelcast/pmem-csi-pvc-xfs-hazelcast   pmem-csi-sc-xfs            21h

$ kubectl get -o yaml pv
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/provisioned-by: pmem-csi.intel.com
    creationTimestamp: "2021-02-08T19:52:45Z"
    finalizers:
    - kubernetes.io/pv-protection
    managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:pv.kubernetes.io/provisioned-by: {}
        f:spec:
          f:accessModes: {}
          f:capacity:
            .: {}
            f:storage: {}
          f:claimRef:
            .: {}
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
            f:namespace: {}
            f:resourceVersion: {}
            f:uid: {}
          f:csi:
            .: {}
            f:driver: {}
            f:fsType: {}
            f:volumeAttributes:
              .: {}
              f:storage.kubernetes.io/csiProvisionerIdentity: {}
            f:volumeHandle: {}
          f:nodeAffinity:
            .: {}
            f:required:
              .: {}
              f:nodeSelectorTerms: {}
          f:persistentVolumeReclaimPolicy: {}
          f:storageClassName: {}
          f:volumeMode: {}
      manager: csi-provisioner
      operation: Update
      time: "2021-02-08T19:52:45Z"
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:phase: {}
      manager: kube-controller-manager
      operation: Update
      time: "2021-02-08T19:52:45Z"
    name: pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe
    resourceVersion: "2295680"
    selfLink: /api/v1/persistentvolumes/pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe
    uid: d774ea9e-3ac2-4f51-a018-9f4949a0ade3
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 20Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: pmem-csi-pvc-xfs-hazelcast
      namespace: hazelcast
      resourceVersion: "2295676"
      uid: 4cafe87a-a435-43d7-806d-bcc5f7c99ebe
    csi:
      driver: pmem-csi.intel.com
      fsType: xfs
      volumeAttributes:
        storage.kubernetes.io/csiProvisionerIdentity:<PHONE_NUMBER>744-8081-pmem-csi.intel.com-osc-nkmp7-worker-cn5f5
      volumeHandle: pvc-4c-a929f81c172dbc4ffda9937437e400e68c3643f60f7c3cafc7505f31
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: pmem-csi.intel.com/node
            operator: In
            values:
            - osc-nkmp7-worker-cn5f5
    persistentVolumeReclaimPolicy: Delete
    storageClassName: pmem-csi-sc-xfs
    volumeMode: Filesystem
  status:
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

That all looks pretty normal.

But I cannot find where is pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d exist.

That's shown here:
$ kubectl get -o yaml pv
apiVersion: v1
items:
- apiVersion: v1
...
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: pmem-csi.intel.com/node
            operator: In
            values:
            - osc-nkmp7-worker-cn5f5
...

That all looks pretty normal.

But I cannot find where is pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d exist.

That's shown here:
$ kubectl get -o yaml pv
apiVersion: v1
items:
- apiVersion: v1
...
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: pmem-csi.intel.com/node
            operator: In
            values:
            - osc-nkmp7-worker-cn5f5
...

I think this node has the pvc-4cafe87a-a435-43d7-806d-bcc5f7c99ebe provisioned but not `pvc-62ee033f-b43e-43f9-a300-ec17767e9d8d,  isn't it?
Maybe I should try to remove the PMEM label on the node phzst and try if everything works well?

Now I see, we are talking about two different PVs. I missed that - these UUIDs are a bit hard to read...
The get pv output only shows one. pvc-...bcc5f7c99ebe doesn't exist anymore but somehow PVC pmem-csi-pvc-xfs-hazelcast is bound to it. How this happened isn't clear.
My suggestion is to force-delete all PVs and PVCs (see https://github.com/kubernetes-csi/external-provisioner/#deleting-local-volumes-after-a-node-failure-or-removal for PV - the same works for PVCs, if needed). Then also delete all logical LVM volumes that you have on one node. That restores a clean state. Kill driver pods to ensure that they also have no hidden state, but that shouldn't be necessary.
Then start again by creating one PVC and checking that this works. Are you using late binding or immediate binding in your storage class? Both should work, but late binding is recommended - we should have that documented somewhere, but I currently don't find it. I'll enhance that in the documentation.
BTW, thanks for trying out our master branch. Using the latest stable release might have been easier because you are running into issues with our recent changes, but we do appreciate the feedback before tagging 0.9.0.

Maybe I should try to remove the PMEM label on the node phzst and try if everything works well?

It should work with two nodes.

I tried the method you suggests several times but it still not working. I notice that there is a warning message when I creating the certificate, but it seems harmless.
2021/02/12 12:05:15 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

I will re-install the whole OpenShift cluster to make sure it is fresh and no hidden states. Thanks for following up, I will update after the re-installation.

Hi @pohly, I re-created the OpenShift cluster to make sure everything is fresh. Unfortunately, I still got the same is not PMEM error. The storage class I create is using late binding:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pmem-csi-sc-xfs
parameters:
  csi.storage.k8s.io/fstype: xfs
  eraseafter: "false"
provisioner: pmem-csi.intel.com
volumeBindingMode: WaitForFirstConsumer

I see you mentioned using allowedTopologies in the installation guide. Could you please provide more details of applying this flag? Thanks!

I attached to my application containers and found that pmem_map_file() returns not PMEM although I can see the device mount through df. It seems my application pod is scheduled on the other node.

I think this is where this went a bit astray: we ended up debugging the "scheduled on the other node" part and ignored the possibility that everything had been set up as expected, but then failed to provide a volume with DAX support.
Can you explain where the "not PMEM" error comes from? Does the application get that from the Linux kernel?
One way to test that is to run the pmem-csi-driver-test image itself as the application with "sleep infinity" as the command, with a PMEM volume mounted somewhere. Then log into that container and run /usr/local/bin/pmem-dax-check /<PMEM-VOLUME>/test-file.
The source code of that command is here:
https://github.com/intel/pmem-csi/blob/devel/test/cmd/pmem-dax-check/main.go
It does the mmap(MAP_SHARED_VALIDATE|MAP_SYNC) and will tell us whether the kernel supports it.
If that fails, then we need to double-check that the volume was mounted with -odax. If that is the case, then we'll have to consult with our kernel developers because I am running out of ideas. I thought that once the filesystem accepts a mount with -odax that then MAP_SYNC is supported for all files.
What is the version of the kernel?

I see you mentioned using allowedTopologies in the installation guide. Could you please provide more details of applying this flag?

Please ignore that. That sentence shouldn't have been merged, it was for an experiment that didn't work out.

The "not PMEM" error I got is from the pmem_map_file() function call of libpmem. The int *is_pmemp arg returns 0 which indicates the device is not PMEM.
The OpenShift 4.6.9 vShpere IPI deploys the worker nodes in CoreOS 4.6, with 4.18.0 kernel. Here are the details:
[core@osc-szm5p-worker-ql6x9 ~]$ cat /proc/version
Linux version 4.18.0-193.37.1.el8_2.x86_64<EMAIL_ADDRESS>(gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Sun Dec 6 19:59:00 EST 2020

[core@osc-szm5p-worker-ql6x9 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="46.82.202012151054-0"
VERSION_ID="4.6"
OPENSHIFT_VERSION="4.6"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.6"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.6"
OSTREE_VERSION='46.82.202012151054-0'

Below are the outputs I got from pmem-csi-driver-test image and /usr/local/bin/pmem-dax-check. I created a PVC, mount it to /mnt/pmem/, and ran a init container to fallocate a file in that directory, then attached to pmem-csi-driver-test container to run the test program.  Eventually, I got does not support MAP_SYNC: operation not supported error from /usr/local/bin/pmem-dax-check. I'm not sure if these steps are the same as you suggests.
$ oc get sc
NAME              PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
pmem-csi-sc-xfs   pmem-csi.intel.com             Delete          WaitForFirstConsumer   false                  22h

$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS      REASON   AGE
pvc-a65db8ac-56a6-4a08-b629-f8f4c7026d02   20Gi       RWO            Delete           Bound    default/pmem-csi-pvc-xfs-hazelcast   pmem-csi-sc-xfs            25m

$ oc get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pmem-csi-pvc-xfs-hazelcast   Bound    pvc-a65db8ac-56a6-4a08-b629-f8f4c7026d02   20Gi       RWO            pmem-csi-sc-xfs   12m

pmem-csi@test-pmem-csi:/$ df -h
Filesystem                                                                               Size  Used Avail Use% Mounted on
overlay                                                                                120G   12G  109G  10% /
tmpfs                                                                                64M     0   64M   0% /dev
tmpfs                                                                                3.9G     0  3.9G   0% /sys/fs/cgroup
shm                                                                                64M     0   64M   0% /dev/shm
tmpfs                                                                                3.9G   46M  3.9G   2% /etc/hostname
/dev/ndbus0region0fsdax/pvc-a6-e11e8fc4774109ee926ced273cca808beb924d51fb4ecf6e490bb055   20G   16G  5.0G  76% /mnt/pmem
/dev/mapper/coreos-luks-root-nocrypt                                                     120G   12G  109G  10% /etc/hosts
tmpfs                                                                                3.9G   28K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                3.9G     0  3.9G   0% /proc/acpi
tmpfs                                                                                3.9G     0  3.9G   0% /proc/scsi
tmpfs                                                                                3.9G     0  3.9G   0% /sys/firmware

pmem-csi@test-pmem-csi:/$ ls -lh /mnt/pmem/
total 15G
-rw-r--r--. 1 root root 15G Feb 17 22:58 mm

pmem-csi@test-pmem-csi:/$ /usr/local/bin/pmem-dax-check /mnt/pmem/mm
/mnt/pmem/mm: does not support MAP_SYNC: operation not supported

And here is the description of pmem-csi-driver-test container:
Containers:
  pmem-csi-driver-testr:
    Container ID:  cri-o://fdf32ce8f9a22cb0c103b9a5adc9accc41484940886cbf237f7c468875bbdbcf
    Image:         intel/pmem-csi-driver-test:canary
    Image ID:      docker.io/intel/pmem-csi-driver-test@sha256:b330003056697f4acfaed3d6cd77f150739cb89ebf7f3251b4495810030267fc
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      sleep infinity
    State:          Running
      Started:      Wed, 17 Feb 2021 14:59:03 -0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/pmem from pmem-pv-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rlwtz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  pmem-pv-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pmem-csi-pvc-xfs-hazelcast
    ReadOnly:   false

I think this is where this went a bit astray: we ended up debugging the "scheduled on the other node" part and ignored the possibility that everything had been set up as expected, but then failed to provide a volume with DAX support.

I think you are right, the title of this issue is probably not proper, please change the issue title if you think it's necessary, thanks!

It did not become clear for me from above, does the problem of missing DAX support happen also for fstype:ext4 ?
The logs and state view mostly show it for fstype:xfs, so the question arises: is this issue xfs-specific?

Eventually, I got does not support MAP_SYNC: operation not supported error from /usr/local/bin/pmem-dax-check. I'm not sure if these steps are the same as you suggests.

Yes, that's it.
Can you check with "mount" that the filesystem on /dev/ndbus0region0fsdax/pvc-a6-e11e8fc4774109ee926ced273cca808beb924d51fb4ecf6e490bb055 was mounted with -odax?

It did not become clear for me from above, does the problem of missing DAX support happen also for fstype:ext4 ?
The logs and state view mostly show it for fstype:xfs, so the question arises: is this issue xfs-specific?

It's definitely worth trying with ext4.

I don't see the dax mount option(I retried and still don't see dax):
mount | grep /mnt/pmem
/dev/ndbus0region0fsdax/pvc-a6-e11e8fc4774109ee926ced273cca808beb924d51fb4ecf6e490bb055 on /mnt/pmem type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

When I try ext4, it seems the kernel CoreOS uses is not supporting mount ext4 with -o dax option:
$ oc get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
pmem-csi-pvc-ext4-test   Bound    pvc-53018546-0168-4135-91f6-801fa7b783ba   20Gi       RWO            pmem-csi-sc-ext4   83s

Pod description:
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    25s               default-scheduler  Successfully assigned default/test-pmem-csi to osc-szm5p-worker-ql6x9
  Warning  FailedMount  9s (x6 over 25s)  kubelet            MountVolume.MountDevice failed for volume "pvc-53018546-0168-4135-91f6-801fa7b783ba" : rpc error: code = Internal desc = mount filesystem failed: command "/bin/mount -c -o dax /dev/ndbus0region0fsdax/pvc-53-9d1d7ec2f678ae5cd0176ec6d3af236c1c5fde37dcfa6864ef032f13 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-53018546-0168-4135-91f6-801fa7b783ba/globalmount" failed: exit status 32
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-53018546-0168-4135-91f6-801fa7b783ba/globalmount: wrong fs type, bad option, bad superblock on /dev/ndbus0region0fsdax/pvc-53-9d1d7ec2f678ae5cd0176ec6d3af236c1c5fde37dcfa6864ef032f13, missing codepage or helper program, or other error.

I don't see the dax mount option when I use xfs

That's a bit surprising. I'll double check whether that should be present tomorrow.
But either way, it looks like the kernel simply doesn't support PMEM. Here are some relevant kernel config options (from a Debian kernel):
$ grep DAX /boot/config-4.19.0-14-amd64
CONFIG_NVDIMM_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y

Can you investigate whether those are present in the kernel?

Yes, these configs present in the kernel config:
[core@osc-szm5p-worker-ql6x9 ~]$ cat /lib/modules/$(uname -r)/config | grep DAX
CONFIG_NVDIMM_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_DEV_DAX_PMEM_COMPAT=m
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y

Was the underlying PMEM set up in fsdax mode? I don't remember whether we discussed this.
This section has some instructions which may or may not be applicable to vSphere:
https://software.intel.com/content/www/us/en/develop/articles/provision-intel-optane-dc-persistent-memory-for-kvm-qemu-guests.html

I can confirm that mounting an XFS volume silently ignores the -odax:
[fedora@pmem-csi-pmem-govm-worker3 ~]$ sudo ndctl create-namespace -m raw
{
  "dev":"namespace0.0",
  "mode":"raw",
  "size":"63.98 GiB (68.70 GB)",
  "uuid":"1a7da011-e570-4706-ba59-bd292c06e453",
  "sector_size":512,
  "blockdev":"pmem0"
}
[fedora@pmem-csi-pmem-govm-worker3 ~]$ ls /dev/pmem0
/dev/pmem0
[fedora@pmem-csi-pmem-govm-worker3 ~]$ sudo mkdir /tmp/mnt
[fedora@pmem-csi-pmem-govm-worker3 ~]$ sudo mkfs.xfs -b size=4096 -m reflink=0 -f /dev/pmem0
meta-data=/dev/pmem0             isize=512    agcount=4, agsize=4193280 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=16773120, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=8190, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[fedora@pmem-csi-pmem-govm-worker3 ~]$ sudo mount -odax /dev/pmem0 /tmp/mnt
[fedora@pmem-csi-pmem-govm-worker3 ~]$ mount
...
/dev/pmem0 on /tmp/mnt type xfs (rw,relatime,seclabel,attr2,dax=never,inode64,logbufs=8,logbsize=32k,noquota)

I also do get the message from https://docs.pmem.io/ndctl-user-guide/troubleshooting:
$ dmesg
...
[ 3115.869604] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
[ 3115.870245] XFS (pmem0): DAX unsupported by block device. Turning off DAX.

We haven't had that issue before because normally, PMEM-CSI itself creates the namespace that it then uses for LVM, so we never run into this. I wonder whether we should add some code to detect it or just extend our documentation for those who do this initial setup manually - we should at least document it, because the manual setup is currently hard to find in the gce example.

because normally, PMEM-CSI itself creates the namespace

I think by design, PMEM-CSI should always operate on namespaces created by itself, in fsdax mode. Is the namespace in current case not created by PMEM-CSI? Is it not in fsdax mode?
but yes it is posisble to bypass "foreign" check by creating namespace manually and give it certain Name attribute,
in that sense the "foreign vs own detection" is not very secure.

/dev/ndbus0region0fsdax/pvc-53-9d1d7ec2f678ae5cd0176ec6d3af236c1c5fde37dcfa6864ef032f13

The naming of device however leaves impression it was created by PMEM-CSI, so it has to be in fsdax mode.

sudo ndctl create-namespace -m raw

In your example, by manually forcing mode=raw, of course it will not support DAX. But namespace created by PMEM-CSI should always come in fsdax mode, we don't have other modes supported.

I found related discussion and analysis in #23, where it was detected that in some combination of kernel/distro, dax capability was not propagating for LV devices. The kernel 4.18 was mentioned there BTW. So it may be here we hit the same?
Can we cross-checking using direct mode to exclude LV-specific potential troubles?

Is the namespace in current case not created by PMEM-CSI?

No, it was created manually.

Can we cross-check using direct mode to exclude LV-specific potential troubles?

In GCE example, it mentions that direct mode does not work on VMs since ndctl init-label doesn't work. I verified this with ndctl team that vSphere ESXi also doesn't support label storage area. Is there any workaround to run direct mode on vSphere VM without using LSA?

Is the namespace in current case not created by PMEM-CSI?

No, it was created manually.

In my case with OpenShift, the namespace is created by PMEM-CSI. I didn't install ndctl on the CoreOS worker nodes, and I had some trouble installing ndctl in privileged container(I will try again later). I only create the VG with name ndbus0region0fsdax and added --addtag fsdax flag as mentioned in GCE example.

Seems my wording "in current case" created some misunderstanding, I meant to ask about original case of that issue, not the specific test @pohly performed (I fully understood namespace was created manually there).
If namespace was created by PMEM-CSI, it for sure has been created in fsdax mode as PMEM-CSI can not do otherwise.
In that case I suspect similar effect as in #23: namespace supports DAX mode, but this support does not propagate to the LV volume, which remains without dax support and causes XFS failure.
But why this case can not be repeated in direct mode? it should not need ndctl on nodes (although that is useful for debugging)

In my case with OpenShift, the namespace is created by PMEM-CSI. I didn't install ndctl on the CoreOS worker nodes, and I had some trouble installing ndctl in privileged container(I will try again later). I only create the VG with name ndbus0region0fsdax and added --addtag fsdax flag as mentioned in GCE example.

If the namespace really was created by PMEM-CSI, then why did it not also create the volume group? Something doesn't add up here.
I think what really happened here is that the PMEM device showed up as one "raw" namespace when the kernel first booted up. That namespace then was ignored by the PMEM-CSI driver. After creating the VG with the expected name, PMEM-CSI started to manage it without validating that the underlying namespace actually is in "fsdax" mode which then leads to the "not pmem" error when later some application actually asks for DAX.
What I mentioned in https://github.com/intel/pmem-csi/issues/884#issuecomment-781892567 should address this. If it does, then we discuss whether we should add a feature to PMEM-CSI where it also takes over raw namespaces. So far, our position has been that preparing fsdax namespaces is something that the cluster admin needs to take care of when deploying nodes. The reason for that is that this is a one-time operation which is potentially destructive and may depend a lot on the hardware, so it's better to put it into hardware specific deployment scripts.
Regarding running ndctl: kubectl exec -ti pmem-csi-intel-com-node-XXX -c pmem-driver -- /bin/bash should allow running ndctl. It might be affected by https://github.com/intel/pmem-csi/issues/786, though, for which we haven't merged the workaround (https://github.com/intel/pmem-csi/pull/867) because we have not been able to verify it beforehand.

What I mentioned in #884 (comment) should address this.

I tried this on a CentOS 8 vSphere VM and I can successfully create a namespace with fsdax mode and mount it with -o dax:
mount | grep pmem0
/dev/pmem0 on /mnt/pmem0 type xfs (rw,relatime,seclabel,attr2,dax,inode64,sunit=4096,swidth=4096,noquota)

Also, with your suggestion attach to pmem-csi-intel-com-node-xxxxx pod, I can also create a namespace with fsdax mode and mount with -o dax with in the pmem-driver container. The dax options shows in the mount option correctly:
root@pmem-csi-intel-com-node-2ndnm:/# ndctl list
[
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"dev",
    "size":33820770304,
    "uuid":"6bb105e7-6e83-4afa-8956-04dcd20560f0",
    "blockdev":"pmem0"
  }
]

root@pmem-csi-intel-com-node-2ndnm:/# mkfs.xfs -f /dev/pmem0
meta-data=/dev/pmem0             isize=512    agcount=4, agsize=2064256 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=8257024, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=4031, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

root@pmem-csi-intel-com-node-2ndnm:/# mount -o dax /dev/pmem0 /mnt/pmem-test/

root@pmem-csi-intel-com-node-2ndnm:/# mount | grep pmem0
/dev/pmem0 on /mnt/pmem-test type xfs (rw,relatime,seclabel,attr2,dax,inode64,noquota)

I think your assumption makes sense, it seems PMEM-CSI directly managing the raw namespace. But the weird thing is that I tried the same before on a single node k8s cluster on top of vSphere VM(CentOS 8 worker node), and everything worked well(I don't think I manually force-create a namespace on that, only VG is created manually).

Good news is that after change the namepsace to fsdax in all nodes through pmem-csi-intel-com-node-xxx containers, I'm finally able to get rid of the not pmem error. PMEM-CSI is able to correctly provision PMEM for my pods through persistentVolumeClaim and CSI Ephemeral volume in this case.

So how do we take this further?
Can you imagine that your customers will be able to run ndctl to change the namespace mode before bringing up PMEM-CSI?
Or do we need to some kind of support for this scenario in PMEM-CSI itself? What could work is this:

in LVM mode (and only in that mode!), check for raw namespaces that are not in use
convert those to fsdax
enable LVM on top of it

This is a change of behavior that we probably have to make opt-in via a new driver parameter.
The "not in use" part is a bit tricky. Is it possible to convert with ndctl create-namespace -e namespace0.0 --mode fsdax, i.e. without -f = --force? I'm not comfortable with running something that requires explicit confirmation via that flag.

It would be great if PMEM-CSI can handle this. It's kind of inconvenient for customers to write a script to attach to every PMEM-CSI pod, reconfigure the namespace mode, and re-deploy PMEM-CSI.

Is it possible to convert with ndctl create-namespace -e namespace0.0 --mode fsdax

I deleted all PMEM-CSI and user pods in the cluster and tried it without -f, ndctl returns Device or resource busy error:
root@pmem-csi-intel-com-node-hz6xg:/# ndctl create-namespace -e namespace0.0 --mode fsdax
  Error: namespace0.0 is active, specify --force for re-configuration

failed to reconfigure namespace: Device or resource busy

Was that with a "fresh" VM or one where the namespace had already been used?

Was that with a "fresh" VM or one where the namespace had already been used?

I was not fresh, the namespace had been used.

I tried in a centos:8 container in a fresh OpenShift Cluster, the -f flag is still needed to reconfigure the namespace mode.

I have two proposals.
The first one is to add a new --force-convert-raw-namespaces parameter. When the driver starts up in LVM mode, it will then first do the ndctl create-namespace -e xxx --mode fsdax --force for all namespaces xxx which are currently in raw mode. Then it creates the volume group like it normally does. The downside is that it will do that on all nodes. We have to change our deployment files and operator API to make this behavior opt-in.
The second proposal is to use a pmem-csi.intel.com/convert-raw-namespaces=force label for nodes to trigger this behavior. We define a DaemonSet which runs PMEM-CSI on all nodes with that label. When it runs, it does:

change the namespaces
remove the pmem-csi.intel.com/convert-raw-namespaces=force label
add the labels that the "normal" DaemonSet expects
terminates

Once that is done, the normal DaemonSet starts the pod on the converted node.
I like the second proposal a bit better. We can unconditionally enable that second DaemonSet and it won't do anything by default. Starting up the driver will take a bit longer the first time, but IMHO that's okay.

A third proposal is to use just one DaemonSet (i.e. as it is now) and add code in PMEM-CSI which checks node labels. If pmem-csi.intel.com/convert-raw-namespaces=force is set when it starts, then this enables the namespace conversion in LVM mode.
The downside of this is worse integration with Node Feature Discovery (NFD): if PMEM-CSI is configured to run on nodes with the feature.node.kubernetes.io/memory-nv.dax=true label, then it won't even start on those nodes which currently only have a raw namespace.

For our current usage, all three methods sound good to me. I also like the second proposal better. It will be you and your team's call to decide how to do it. Thanks for keep tracking this issue!

The second proposal is to use a pmem-csi.intel.com/convert-raw-namespaces=force label for nodes to trigger this behavior. We define a DaemonSet which runs PMEM-CSI on all nodes with that label. When it runs, it does:

change the namespaces
remove the pmem-csi.intel.com/convert-raw-namespaces=force label
add the labels that the "normal" DaemonSet expects

How is the node labeling handled in case of NFD? Do we skip this step?

There is no automatic node labeling by NFD for raw namespaces, at least as far as I know. Even if there is one, we don't want to use that because converting an existing raw namespace is destructive and should require an explicit opt-in. So pmem-csi.intel.com/convert-raw-namespaces=force has to be set manually.
The "normal" label then can be feature.node.kubernetes.io/memory-nv.dax=true, i.e. the same one that NFD would also set. Setting it via the PMEM-CSI conversion step is just a shortcut because we don't need to wait for NFD to notice that there is now an fsdax namespace.

@avalluri so do you agree that option 2 is the right way to go?

@avalluri so do you agree that option 2 is the right way to go?

Yes,

There's a slight usability issue with the proposed solution: it is possible to inspect the daemonset which is responsible for converting namespaces, but then finding further information about what was done isn't possible. The logs of the pods cannot be retrieved because they get deleted immediately and it's also not visible where they ran:
$ kubectl describe daemonsets/pmem-csi-intel-com-node-setup -n pmem-csi
Name:           pmem-csi-intel-com-node-setup
Selector:       app.kubernetes.io/instance=pmem-csi.intel.com,app.kubernetes.io/name=pmem-csi-node-setup,pmem-csi.intel.com/deployment=lvm-testing
Node-Selector:  pmem-csi.intel.com/convert-raw-namespaces=force
Labels:         app.kubernetes.io/component=node-setup
                app.kubernetes.io/instance=pmem-csi.intel.com
                app.kubernetes.io/name=pmem-csi-node-setup
                app.kubernetes.io/part-of=pmem-csi
                pmem-csi.intel.com/deployment=lvm-testing
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=node-setup
                    app.kubernetes.io/instance=pmem-csi.intel.com
                    app.kubernetes.io/name=pmem-csi-node-setup
                    app.kubernetes.io/part-of=pmem-csi
                    pmem-csi.intel.com/deployment=lvm-testing
                    pmem-csi.intel.com/webhook=ignore
  Service Account:  pmem-csi-intel-com-node-setup
  Containers:
   pmem-driver:
    Image:      <IP_ADDRESS>:5001/pmem-csi-driver-test:canary
    Port:       <none>
    Host Port:  <none>
    Command:
      /usr/local/bin/pmem-csi-driver
      -v=3
      -logging-format=text
      -mode=force-convert-raw-namespaces
      -nodeSelector={"storage": "pmem"}
      -nodeid=$(KUBE_NODE_NAME)
    Environment:
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      TERMINATION_LOG_PATH:  /tmp/termination-log
    Mounts:
      /dev from dev-dir (rw)
      /sys from sys-dir (rw)
  Volumes:
   dev-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  DirectoryOrCreate
   sys-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  DirectoryOrCreate
Events:
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  2m20s  daemonset-controller  Created pod: pmem-csi-intel-com-node-setup-52f6k
  Normal  SuccessfulDelete  2m19s  daemonset-controller  Deleted pod: pmem-csi-intel-com-node-setup-52f6k
  Normal  SuccessfulCreate  5s     daemonset-controller  Created pod: pmem-csi-intel-com-node-setup-hb6v8
  Normal  SuccessfulDelete  4s     daemonset-controller  Deleted pod: pmem-csi-intel-com-node-setup-hb6v8

If the conversion doesn't do anything, as in this case, then it is possible to run the same command in a regular PMEM-CSI driver pod:
$ kubectl exec -ti -n pmem-csi pmem-csi-intel-com-node-nsfj7 -c pmem-driver -- /bin/bash
root@pmem-csi-intel-com-node-nsfj7:/# /usr/local/bin/pmem-csi-driver -v=5 -mode=force-convert-raw-namespaces -nodeSelector='{"storage": "pmem"}' -nodeid=pmem-csi-pmem-govm-worker3
I0317 15:45:34.044221      28 main.go:69] Version: v0.8.0-250-gf482cb69a
I0317 15:45:34.046261      28 convert.go:157] "raw-namespace-conversion: checking" namespace="{\"blockdev\":\"pmem0\",\"dev\":\"namespace0.0\",\"enabled\":true,\"id\":0,\"mode\":\"raw\",\"name\":\"\",\"size\":68719476736,\"uuid\":\"00000000-0000-0000-0000-000000000000\"}"
I0317 15:45:34.046339      28 convert.go:147] "raw-namespace-conversion: successful" converted=0
I0317 15:45:34.046358      28 convert.go:230] "relabel: node" patch="{\"metadata\":{\"labels\":{\"pmem-csi.intel.com/convert-raw-namespaces\": null, \"storage\": \"pmem\"}}}"
failed to run driver: relabel node pmem-csi-pmem-govm-worker3: failed to patch node: nodes "pmem-csi-pmem-govm-worker3" is forbidden: User "system:serviceaccount:pmem-csi:pmem-csi-intel-com-controller" cannot patch resource "nodes" in API group "" at the cluster scope:
PASS
coverage: 6.6% of statements in ./pkg/...

I need to add more logging, but at least it is possible to see it (albeit in a rather difficult way).

@Tianyang-Zhang : I implemented option 2 in PR https://github.com/intel/pmem-csi/pull/919.
What I don't know is whether it really works on vSphere. I was only able to test on QEMU. The PR has an E2E test that runs on that. Perhaps you can help with testing it on vSphere? Do we need to merge the PR for that first (in which case the "canary" images have the necessary support) or can you build from source?
While writing documentation, I realized that a design choice might not be suitable:

If the conversion finds no raw namespaces on a node, it will remove
convert-raw-namespaces label and exit without adding the normal
label, i.e. the node will not be used for PMEM-CSI. This is not
considered an error, which may be useful in clusters where it is not
certain in advance which nodes will have a raw namespace.
...
A log monitoring solution is needed to catch this output because these
pods are short-lived and kubectl logs does not work for pods that
were already deleted.

Would it be better to treat a missing raw namespace as error, i.e. exit with non-zero exit code and not remove the label? Then the pod does not get deleted and its log output can be inspected easily to figure out what it found (or rather, did not find) on the node.

Thanks for updating @pohly ! I built the image from your branch pohly:raw-namespace-conversion and tested on OpenShift 4 + vSphere environment. I added the new label to each node and I can see the daemonset is created as expected:
$ oc get daemonset
NAME                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                     AGE
pmem-csi-intel-com-node         3         3         0       3            0           storage=pmem                                      2s
pmem-csi-intel-com-node-setup   3         3         0       3            0           pmem-csi.intel.com/convert-raw-namespaces=force   2s

But the pods are not converting the raw PMEM device to fsdax mode, no matter I create VG on each node or not:
$ oc get pods -o wide -n pmem-csi
NAME                                  READY   STATUS             RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
pmem-csi-intel-com-controller-0       1/1     Running            0          20s   <IP_ADDRESS>   osc-5k68w-master-1       <none>           <none>
pmem-csi-intel-com-node-2rd8f         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-m6sj8         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-jvd4p   <none>           <none>
pmem-csi-intel-com-node-setup-tn6qz   0/1     CrashLoopBackOff   1          20s   <IP_ADDRESS>    osc-5k68w-worker-jvd4p   <none>           <none>
pmem-csi-intel-com-node-setup-z96p2   0/1     CrashLoopBackOff   1          20s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>
pmem-csi-intel-com-node-setup-zq4sx   0/1     Error              2          20s   <IP_ADDRESS>   osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-xmnc8         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>

The logs shows as following:
oc logs -f pmem-csi-intel-com-node-setup-zq4sx
I0323 00:24:49.123326       1 main.go:69] Version: v0.8.0-266-g1b8bb713
I0323 00:24:49.125085       1 convert.go:157] "raw-namespace-conversion: checking for namespaces"
I0323 00:24:49.125517       1 convert.go:159] "raw-namespace-conversion: checking" bus="{\"dev\":\"ndbus0\",\"dimms\":[{}],\"provider\":\"ACPI.NFIT\",\"region
s\":[{}]}"
failed to run driver: Failed to set namespace uid: No such file or directory
I0323 00:24:49.125926       1 convert.go:161] "raw-namespace-conversion: checking" region="{\"available_size\":0,\"dev\":\"region0\",\"mappings\":[{}],\"max_a
vailable_extent\":0,\"namespaces\":null,\"size\":34359738368,\"type\":\"pmem\"}"
I0323 00:24:49.126067       1 convert.go:178] "raw-namespace-conversion: converting raw namespace" namespace="{\"blockdev\":\"\",\"dev\":\"namespace0.0\",\"en
abled\":false,\"id\":0,\"mode\":\"raw\",\"name\":\"\",\"size\":34359738368,\"uuid\":\"00000000-0000-0000-0000-000000000000\"}"
E0323 00:24:49.126299       1 convert.go:151] "raw-namespace-conversion: failed" err="Failed to set namespace uid: No such file or directory" converted=0

After the daemonset ran, the /dev/pmem0 device is gone, and ndctl list doesn't show anything. I need to reboot all host to recover.
For your question

Would it be better to treat a missing raw namespace as error, i.e. exit with non-zero exit code and not remove the label

IMHO, maybe it is better to act differently in 2 cases:

No raw namespace and no fsdax namespace were found. In this case, we may treat it as an error and not remove the label.
No raw namespace found, but found fsdax namespace(s). In this case, I think we can treat it as no error and remove the label.

What do you think?

On the bright side, at least you were able to retrieve logs in this case :sweat_smile:
They just don't help a lot, unfortunately. Somehow my implementation of ndctl create-namespace --convert must be different enough that it worked for me under QEMU, but fails under vSphere. Replicating the exact logic of that particular operation was not straight-forward. I wonder whether it is worthwhile to even try it - calling the external ndctl would be simpler. The downside of that is that it provides less control (in theory, at least) and we introduce a dependency on the tool.
Regarding error handling: I like your proposal. The first case makes it easy to debug when something unexpected happened. The second case covers idempotency if namespace conversion worked, but then (intermittently) communication with the API server failed.

@avalluri what do you think about calling the ndctl binary in this case?
In general, what was the rationale for writing Go bindings for libndctl instead of just calling the binary and parsing the JSON output?

In general, what was the rationale for writing Go bindings for libndctl instead of just calling the binary and parsing the JSON output?

I recall we considered:

library call being lower overhead/faster than running binary (which ends up calling the same library anyway).
result as return value is easier to parse than writing (new) parser for text-form output which tends to change more often that library API

If we have case where calling binary is better, then of course this can be reconsidered.

@avalluri what do you think about calling the ndctl binary in this case?

convert.go:151] "raw-namespace-conversion: failed" err="Failed to set namespace uid: No such file or directory" converted=0
By seeing the error, I would propose to try to add a check if ns.Type() != IoNamespace{ } while calling ns.SetUUID().
If that won't help, maybe better to use ndctl binary.

In general, what was the rationale for writing Go bindings for libndctl instead of just calling the binary and parsing the JSON output?

At the initial prototyping stage, we used ndctl binary directly. Later when we were aware of libndctl, just moved to use that assuming that we get more flexibility/control.

https://github.com/pmem/ndctl/blob/ea014c0c9ec8d0ef945d072dcc52b306c7a686f9/ndctl/namespace.c#L493-L503 has such a check. But can we really skip that SetUUID, SetSize and SetAltName in this case? In particular the name is important. Without it, the LVM mode will not set up the volume group.
@Tianyang-Zhang : may I ask you to test whether at least the conversion now works? I suspect I'll have to add volume group setup to the conversion step instead of relying on the normal driver startup for that, but I can add that once I know that the conversion to fsdax works.

Sure. I rebuild the image and did the test again. This time I get one pod end up in complete state but the other 2 still
in CrashLoopBackOff state:
oc get pods -n pmem-csi -o wide
NAME                                  READY   STATUS             RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
pmem-csi-intel-com-controller-0       1/1     Running            0          20s   <IP_ADDRESS>   osc-5k68w-master-1       <none>           <none>
pmem-csi-intel-com-node-9zp78         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>
pmem-csi-intel-com-node-hg85f         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-setup-5wfdj   0/1     Completed          1          20s   <IP_ADDRESS>   osc-5k68w-worker-jvd4p   <none>           <none>
pmem-csi-intel-com-node-setup-ml8rb   0/1     CrashLoopBackOff   1          20s   <IP_ADDRESS>   osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-setup-w2bfw   0/1     CrashLoopBackOff   1          20s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>
pmem-csi-intel-com-node-tz56k         3/3     Running            0          20s   <IP_ADDRESS>   osc-5k68w-worker-jvd4p   <none>           <none>

Then the complete pod also turns to CrashLoopBackOff state. The log of three pods are the same:
oc logs pmem-csi-intel-com-node-setup-5wfdj
I0326 03:03:51.754546       1 main.go:69] Version: v0.9.0-19-g52dcd362-dirty
I0326 03:03:51.756486       1 convert.go:157] "raw-namespace-conversion: checking for namespaces"
I0326 03:03:51.756953       1 convert.go:159] "raw-namespace-conversion: checking" bus="{\"dev\":\"ndbus0\",\"dimms\":[{}],\"provider\":\"ACPI.NFIT\",\"regions\":[{}]}"
failed to run driver: set fsdax mode: Failed to set enforce mode: No such device or address
PASS
coverage: 8.0% of statements in ./pkg/...
I0326 03:03:51.757406       1 convert.go:161] "raw-namespace-conversion: checking" region="{\"available_size\":0,\"dev\":\"region0\",\"mappings\":[{}],\"max_available_extent\":0,\"namespaces\":null,\"size\":34359738368,\"type\":\"pmem\"}"
I0326 03:03:51.757554       1 convert.go:178] "raw-namespace-conversion: converting raw namespace" namespace="{\"blockdev\":\"\",\"dev\":\"namespace0.0\",\"enabled\":false,\"id\":0,\"mode\":\"raw\",\"name\":\"\",\"size\":34359738368,\"uuid\":\"00000000-0000-0000-0000-000000000000\"}"
E0326 03:03:51.757672       1 convert.go:151] "raw-namespace-conversion: failed" err="set fsdax mode: Failed to set enforce mode: No such device or address" converted=0

The PMEM device is gone as same as the last build, oc exec -it pmem-csi-intel-com-node-tz56k -c pmem-driver -- ndctl list returns nothing.
Then I rebooted all nodes and ran again. This time I got:
oc get pods -n pmem-csi -o wide
NAME                                  READY   STATUS      RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
pmem-csi-intel-com-controller-0       1/1     Running     0          25s   <IP_ADDRESS>   osc-5k68w-master-1       <none>           <none>
pmem-csi-intel-com-node-8dvcn         3/3     Running     0          25s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>
pmem-csi-intel-com-node-fdjtj         3/3     Running     0          25s   <IP_ADDRESS>    osc-5k68w-worker-jvd4p   <none>           <none>
pmem-csi-intel-com-node-kwznm         3/3     Running     0          25s   <IP_ADDRESS>    osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-setup-684f4   0/1     Completed   2          25s   <IP_ADDRESS>   osc-5k68w-worker-jvd4p   <none>           <none>
pmem-csi-intel-com-node-setup-njbkx   0/1     Completed   2          25s   <IP_ADDRESS>   osc-5k68w-worker-7c4g5   <none>           <none>
pmem-csi-intel-com-node-setup-pql6h   0/1     Completed   2          25s   <IP_ADDRESS>   osc-5k68w-worker-9c42p   <none>           <none>

But the logs seems still showing error:
oc logs pmem-csi-intel-com-node-setup-684f4
I0326 03:08:00.396148       1 main.go:69] Version: v0.9.0-19-g52dcd362-dirty
I0326 03:08:00.398067       1 convert.go:157] "raw-namespace-conversion: checking for namespaces"
I0326 03:08:00.398499       1 convert.go:159] "raw-namespace-conversion: checking" bus="{\"dev\":\"ndbus0\",\"dimms\":[{}],\"provider\":\"ACPI.NFIT\",\"regions\":[{}]}"
I0326 03:08:00.398869       1 convert.go:161] "raw-namespace-conversion: checking" region="{\"available_size\":0,\"dev\":\"region0\",\"mappings\":[{}],\"max_available_extent\":0,\"namespaces\":null,\"size\":34359738368,\"type\":\"pmem\"}"
failed to run driver: set fsdax mode: Failed to set enforce mode: No such device or address
PASS
coverage: 8.0% of statements in ./pkg/...
I0326 03:08:00.399509       1 convert.go:178] "raw-namespace-conversion: converting raw namespace" namespace="{\"blockdev\":\"\",\"dev\":\"namespace0.0\",\"enabled\":false,\"id\":0,\"mode\":\"raw\",\"name\":\"\",\"size\":34359738368,\"uuid\":\"00000000-0000-0000-0000-000000000000\"}"
E0326 03:08:00.399711       1 convert.go:151] "raw-namespace-conversion: failed" err="set fsdax mode: Failed to set enforce mode: No such device or address" converted=0

And ndctl list still returns nothing:
oc exec -it pmem-csi-intel-com-node-8dvcn -c pmem-driver -- /bin/bash
root@pmem-csi-intel-com-node-8dvcn:/# ls -l /dev/pmem*
ls: cannot access '/dev/pmem*': No such file or directory
root@pmem-csi-intel-com-node-8dvcn:/# ndctl list
root@pmem-csi-intel-com-node-8dvcn:/#

Okay, I'll switch to calling the ndctl binary. That seems easier than trying to reimplement it.
However, there is one open question: can we set the pmem-csi name under vSphere?
@Tianyang-Zhang can you perhaps run ndctl create-namespace -e namespace0.0 --mode fsdax --name pmem-csi --force on a machine which has a namespace0.0 in raw mode and then verify with ndctl list -NR that the name is set?
I'm starting to suspect that the name will be silently dropped by ndctl. In that case there's further work needed on setting up the volume group.

It seems the name is dropped by ndctl:
[root@pmem-fsdax-config-8b5wc /]# ndctl create-namespace -e namespace0.0 --mode fsdax --name pmem-csi --force
{
  "dev":"namespace0.0",
  "mode":"fsdax",
  "map":"dev",
  "size":"31.50 GiB (33.82 GB)",
  "uuid":"5cd0222e-64f0-4f5a-b9a0-36d418307bb4",
  "sector_size":512,
  "align":2097152,
  "blockdev":"pmem0"
}

The output of ndctl list -NR
[root@pmem-fsdax-config-8b5wc /]# ndctl list -NR
{
  "regions":[
    {
      "dev":"region0",
      "size":34359738368,
      "available_size":0,
      "max_available_extent":0,
      "type":"pmem",
      "persistence_domain":"memory_controller",
      "namespaces":[
        {
          "dev":"namespace0.0",
          "mode":"fsdax",
          "map":"dev",
          "size":33820770304,
          "uuid":"5cd0222e-64f0-4f5a-b9a0-36d418307bb4",
          "sector_size":512,
          "align":2097152,
          "blockdev":"pmem0"
        }
      ]
    }
  ]
}

Thanks for the confirmation. I'll work around that.

IMHO, maybe it is better to act differently in 2 cases:

No raw namespace and no fsdax namespace were found. In this case, we may treat it as an error and not remove the label.
No raw namespace found, but found fsdax namespace(s). In this case, I think we can treat it as no error and remove the label.

I've implemented that, and the idempotency aspect of it worked rather well even when the setup pod was started twice: it first did its work and must have removed labels, but then it was restarted by the daemonset controller before that noticed the label changes. The second run then didn't change anything and eventually the driver pod started up.
Having said that, the pod shouldn't have been started twice. It seems related to the operator and how it updated the DaemonSet (unnecessarily?):
<EMAIL_ADDRESS>I0406 18:15:26.412498       1 deployment_controller.go:126] "UPDATED" object="default/second-pmem-csi-intel-com-node-setup <*v1.DaemonSet>"
<EMAIL_ADDRESS>I0406 18:15:26.412976       1 deployment_controller.go:466] "newDeployment: load controller TLS secret" secret="default/second-pmem-csi-intel-com-controller-secret" CAlength=1086
<EMAIL_ADDRESS>I0406 18:15:26.413093       1 controller_driver.go:627] "deployment/event: start" object="default/second-pmem-csi-intel-com-node-setup <*v1.DaemonSet>" type="*v1.DaemonSet"
<EMAIL_ADDRESS>I0406 18:15:26.413289       1 controller_driver.go:641] "deployment/event: redeploying" name="node setup driver" object="default/second-pmem-csi-intel-com-node-setup <*v1.DaemonSet>"
<EMAIL_ADDRESS>I0406 18:15:26.413454       1 controller_driver.go:194] "redeploy/getSubObject: get" object="default/second-pmem-csi-intel-com-node-setup <*v1.DaemonSet>"
<EMAIL_ADDRESS>I0406 18:15:26.415331       1 controller_driver.go:280] "redeploy: update" object="default/second-pmem-csi-intel-com-node-setup"

The status of second-pmem-csi-intel-com-node-setup changed as part of the first pod termination, but that should have caused an updated for that DaemonSet.
This was for make test_e2e TEST_E2E_FOCUS=operator.*lvm-production.*raw-conversion.*works on my branch.

It's not clear to me from the log messages at -v=5 whether the operator actually does something:
<EMAIL_ADDRESS>I0406 18:53:19.837132       1 controller_driver.go:280] "redeploy: update" object="default/second-pmem-csi-intel-com-node"
<EMAIL_ADDRESS>I0406 18:53:19.851437       1 deployment_controller.go:67] "UPDATED" object="second.pmem-csi.intel.com <*v1beta1.PmemCSIDeployment>" generation=1

That's for TEST_OPERATOR_LOGLEVEL=5 -> -vmodule=*=5. Why is the -vmodule parameter used instead of the simpler -v?

Yes, line 280 does apply a patch, but without the patch it remains unclear what change it is making. I'll add that information.

The patch is a lot larger than I expected:
<EMAIL_ADDRESS>I0406 20:08:35.694748       1 controller_driver.go:280] "redeploy: update" object="default/second-pmem-csi-intel-com-node-setup" patch="{\"spec\":{\"revisionHistoryLimit\":null,\"template\":{\"spec\":{\"containers\":[{\"command\":[\"/usr/local/bin/pmem-csi-driver\",\"-v=3\",\"-logging-format=text\",\"-mode=force-convert-raw-namespaces\",\"-nodeSelector={\\\"feature.node.kubernetes.io/memory-nv.dax\\\":\\\"true\\\"}\",\"-nodeid=$(KUBE_NODE_NAME)\"],\"env\":[{\"name\":\"KUBE_NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"spec.nodeName\"}}},{\"name\":\"TERMINATION_LOG_PATH\",\"value\":\"/tmp/termination-log\"}],\"image\":\"<IP_ADDRESS>:5001/pmem-csi-driver:canary\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"pmem-driver\",\"resources\":{},\"securityContext\":{\"privileged\":true,\"runAsUser\":0},\"terminationMessagePath\":\"/tmp/termination-log\",\"volumeMounts\":[{\"mountPath\":\"/dev\",\"name\":\"dev-dir\"},{\"mountPath\":\"/sys\",\"name\":\"sys-dir\"},{\"mountPath\":\"/host-sys\",\"name\":\"sys-dir\"}]}],\"dnsPolicy\":null,\"restartPolicy\":null,\"schedulerName\":null,\"securityContext\":null,\"serviceAccount\":null,\"terminationGracePeriodSeconds\":null}},\"updateStrategy\":{\"rollingUpdate\":null,\"type\":null}}}"

@avalluri is that normal? Shouldn't it only patch the fields that really need to be changed? Is there ever a situation when the if string(data) != "{}" && len(data) >= 0 causes an update to be skipped?

That's for TEST_OPERATOR_LOGLEVEL=5 -> -vmodule=*=5. Why is the -vmodule parameter used instead of the simpler -v?

I had to use -vmodule when setting -v did not work. But I realized that there is a typo in sed regex. Fixing that could remove -vmodule usage. I figured this out while working on version skew tests.

I also found and fixed the -v sed regex issue. I have that together with -vmodule removal in my branch.

@avalluri is that normal? Shouldn't it only patch the fields that really need to be changed?

The diff supposed to contain only changed fields. But I am not sure what changed.

Is there ever a situation when the if string(data) != "{}" && len(data) >= 0 causes an update to be skipped?

I remember the patch data used to  have '{}' for empty diff, that means it used to skip those cases. But I am not sure what changed to give that long diff. I will check.

Restarting the "node-setup" pod seems related to the operator behavior. I don't see that when testing without it.

I think I have addressed all opens. The revised version that I just pushed creates the volume group during conversion without relying on the name and the error handling is different.
The operator patch issue can be investigated in parallel.
@Tianyang-Zhang can you perhaps give it one more try?
@avalluri can you review?

The operator patch issue can be investigated in parallel.

It's worth pointing out that it no longer causes the pod to restart: I changed the exit handling of the conversion pod so that it keeps running until terminated by Kubernetes, i.e. there is no longer a race between "pod terminates itself" and "Kubernetes restarts pod". That seems to help.

I tried the newest changes on our vSphere + OpenShift environment and it works! The raw namespaces are converted to fsdax, volume groups is created on each node, and I am able to claim PMEM through CSI ephemeral volume.
I also ran this on the cluster that has all node already has fsdax mode PMEM, and I don't see any issue.
Thanks again for keep updating and resolving this issue!


PuglEventText time in seconds

I'm guessing this was just a doc typo.

Whoops.  It got cherry-picked from an old branch with the wrong type, which @swesterfeld pointed out, but I forgot to update the comment when I fixed the type.
Merged as 6f39b67, thanks.


Page - Error Handling

The Error Handling (https://developer.nhs.uk/apis/ems-beta/explore_errors.html) page would benefit from the following changes to the specification:

The url for the page might be better being "error_handling" rather than "explore_errors".

There is a link to the spine core spec, but the link is to a non static version of the spine core specification. We should reference a static version of the specification so that when someone looks at this version of the NEMS spec they always see the spine core spec which was correct at the point where the specification was release. If we link to the current (latest) version of the spine core spec they could introduce a breaking change or additional changes we have not taken into consideration for the NEMS spec, which could cause problems for suppliers who build against that later core spec when they interact with the NEMS apis.

The "Event Message Response" link does not work and leads to a 404.

The url for the page might be better being "error_handling" rather than "explore_errors". - This change has been applied.
The "Event Message Response" link does not work and leads to a 404. - This reference has been removed as the Event Message Response has been replaced by the guidance in the Error Handling section.
Point 2 will require comment from @ahatherly or @fletch1975

Links to static version of spine core and other external specs will be raised as a seperate github issue. All other points on this issue have been fixed so I am closing as the changes are in gh-pages.


Updated 'main' value of package.json and bower.json

Updated main property of package.json to point to file angular-listview.js

@wtfribley Did you have a chance to review this PR? Let me know if I should include a package version bump. Thanks!


~1 second delay before processor picks up a task from a new queue

Note: This observation is not always consistent, but it happens often enough
It has been observed that when a new task is enqueued to asynq for a new queue, asynq seemingly takes roughly ~1 sec to pickup the task and begin processing it. Otherwise, normally asynq practically immediately picks up tasks and begins processing
Not necessarily a big deal in production, but the delay has adverse effects on content fabric tests

From @elv-gilles :
for the ‘one sec’ delay (#12 & #13): could you give a try changing the configured value of TaskQConfig.EmptyQSleep (the default is 1s) during the tests and see what it gives ?
Indeed, EmptyQSleep is the culprit, so closing this as this is expected behavior


Radeon VII mining rate increase with old ROCm 3.3

Previous system: Ubuntu 20.04

Previous ROCm version: 4.2

New System: Ubuntu 18.04

New ROCm version: 3.3

Ethereum mining rate increase from 80 MH/s to 86 MH/s.

Hi @valeriob01
Thanks for reaching out.
ROCm 3.3 is very old and the latest ROCm version is 4.3. Not sure why are you comparing 3.3 and 4.2.
Because between these 3.3 and 4.2, 9 more releases are there.
So to understand and solve the problem better, I request you to reduce the gap between 3.3 and 4.2 and share the better information.
And also request you to share me the exact steps to reproduce the problem.
Thank you.

How you concluded that there is a problem?
This is only a mere mining rate comparison !


[WIP][Dependency Update] Update libjpeg-turbo to 2.0.2

Description
please don't merge until I add publish test on CI for cuda part
Upgrade the libjpeg-turbo to 2.0.2 due to the issue at 1.5.90

divide by zero when processing a crafted BMP image

Version 2.0.2 also fixes the performance issue

Fixed a severe performance issue in the Loongson MMI SIMD extensions that occurred when compressing RGB images whose image rows were not 64-bit-aligned.

as stated in the release note
Checklist
Essentials

[ ] Test build with Ubuntu 14.04
[ ] Test build with Ubuntu 16.04

Changes
Comments

Thanks for your contributions @stu1130. The CI seems to be failing. Can you look into this ?
@mxnet-label-bot Add [Build, pr-awaiting-review]

Thanks for your contributions @stu1130. The CI seems to be failing. Can you look into this ?
@mxnet-label-bot Add [Build, pr-awaiting-review]

@stu1130  ping to take a look at CI. thanks!

@mxnet-label-bot update [Build, pr-awaiting-testing]

@mxnet-label-bot update [Build, pr-awaiting-review]


Translations

Some translations to Spanish

Ping @joecohens.

Another file left to translate, you'll have it tomorrow.

Thanks. We'll wait for that and then I'll squash and merge this once Joe is happy. ;)

Here it's.

Merged. Thank you. :)

:+1:


Form validation function

Hi @davidkpiano ,
I have done some work to get the form validation as a function somewhat working.
I was just wondering if you would like me to open a pull request for you to look at to see if it's something you are either currently working on or interested in.
The branch is here https://github.com/sl33kr/react-redux-form/tree/form_validation_function
It's rough at the moment but works for my case.
It isn't using invertValidators at the moment either.
Usage is
export function validateForm(model) {
    const cust = model && model.customers[0];

    if (cust && (required(cust.phoneNumber) || required(cust.email))) {
        return false;
    }

    return {
        'customers[0].phoneNumber': { 'hasPhoneOrEmail': true },
        'customers[0].email': { 'hasPhoneOrEmail': true }
    };
}

<Form model="appState.enquiry" validators={validateForm}>

			<Errors model=".customers[0].phoneNumber"
					messages={{ hasPhoneOrEmail: 'Please enter either a phone number or email.' }} />
			<Control.text model=".customers[0].phoneNumber" id="new-customer-phone" />
			Phone number *

			<Errors model=".customers[0].email"
					messages={{ typeMismatch: "Please enter a valid email.", hasPhoneOrEmail: 'Please enter either a phone number or email.' }} />
			<Control model=".customers[0].email" id="new-customer-email" />
			Email

</Form>

Hi @davidkpiano
Was just wondering if you have had a chance to consider this?

Hey @sl33kr, taking a look at it now! Sorry for the delay.

Would you mind merging in the latest from master and creating a pull request so I can further review? I definitely want to go forward with this.

Great! I'll tidy it up try to get this done over the next day or 2.


Will jsTree plugins get support?

Is there any timeline for support of these available plugins?

It could be interesting a simple example of the search plugin with ngJsTree.
Thx

can someone help with ngJsTree demo with search plugin ?

Search plugin don't work, anyone can help me?

I am using it in my projects, will add an example

Is there a working example with search that we can take a look at ?


Anti-feature: Promotes or depends on a non-free network service

This app promotes DNS servers provided by @Revertron
<string name="dns_server_info_revertron">The server supports resolving regular ICANN domains, ALFIS domains, OpenNIC domains.\n\nAlso, it blocks ads, analytics and malware websites.\n\nThe server is run by Revertron.</string>

https://github.com/yggdrasil-network/yggdrasil-android/blob/main/app/src/main/res/values/strings.xml#L9

The DNS servers are not configured by default for exactly this reason.

It promotes them all the same, and probably shouldn't.


Must use splat syntax to access X,  because it has "count" set

With the following module setup I'm getting a failure:
Warning: output "log_group_arn": must use splat syntax to access aws_cloudwatch_log_group.default attribute "arn", because it has "count" set; use aws_cloudwatch_log_group.default.*.arn to obtain a list of the attributes across all instances

Warning: output "vpc_flow_id": must use splat syntax to access aws_flow_log.vpc attribute "id", because it has "count" set; use aws_flow_log.vpc.*.id to obtain a list of the attributes across all instances

Warning: output "kinesis_id": must use splat syntax to access aws_kinesis_stream.default attribute "id", because it has "count" set; use aws_kinesis_stream.default.*.id to obtain a list of the attributes across all instances

Warning: output "kinesis_name": must use splat syntax to access aws_kinesis_stream.default attribute "name", because it has "count" set; use aws_kinesis_stream.default.*.name to obtain a list of the attributes across all instances

Warning: output "kinesis_shard_count": must use splat syntax to access aws_kinesis_stream.default attribute "shard_count", because it has "count" set; use aws_kinesis_stream.default.*.shard_count to obtain a list of the attributes across all instances

Warning: output "kinesis_arn": must use splat syntax to access aws_kinesis_stream.default attribute "arn", because it has "count" set; use aws_kinesis_stream.default.*.arn to obtain a list of the attributes across all instances

If I follow the suggestions for the above it then seems to work.
Here's my flow_logs module:
module "flow_logs" {
  source    									= "git::https://github.com/cloudposse/terraform-aws-cloudwatch-flow-logs.git?ref=master"
  vpc_id    									= "${var.vpc_id}"
  namespace 									= "${var.ns}"
  stage     									= "${var.env}"
}

If I update the outputs.tf for flog_logs like so it then appears to work:
output "log_group_arn" {
  value       = "${aws_cloudwatch_log_group.default.*.arn}"
  description = "ARN of the log group"
}

output "vpc_flow_id" {
  value       = "${aws_flow_log.vpc.*.id}"
  description = "Flow Log IDs of VPCs"
}

output "subnet_flow_ids" {
  value       = "${aws_flow_log.subnets.*.id}"
  description = "Flow Log IDs of subnets"
}

output "eni_flow_ids" {
  value       = "${aws_flow_log.eni.*.id}"
  description = "Flow Log IDs of ENIs"
}

output "kinesis_id" {
  value       = "${aws_kinesis_stream.default.*.id}"
  description = "Stream ID"
}

output "kinesis_name" {
  value       = "${aws_kinesis_stream.default.*.name}"
  description = "Stream name"
}

output "kinesis_shard_count" {
  value       = "${aws_kinesis_stream.default.*.shard_count}"
  description = "Count of Shards for Stream"
}

output "kinesis_arn" {
  value       = "${aws_kinesis_stream.default.*.arn}"
  description = "ARN of Stream"
}

Also "log_group_name": [DEPRECATED] use 'log_destination' argument instead

Any update on this? This is affecting us as well.

@ChrisMcKee @nema-sigsci
that was addressed in https://github.com/cloudposse/terraform-aws-cloudwatch-flow-logs/pull/7, but waiting on @slmingol  to address the comments.
If he could not get to it today, we'll open another PR with the fix tomorrow.
BTW, we should use splat+join pattern for all resources with counts.
So instead of
value       = "${aws_kinesis_stream.default.*.name}"
we should use
value       = "${join("", aws_kinesis_stream.default.*.name)}"
since when enabled=false, it should get an empty string (otherwise it could fail in some cases)

@nema-sigsci please ping me tomorrow so I don't forget to get to it

@nema-sigsci please ping me tomorrow so I don't forget to get to it

Ping :)


Add option to sign generated metadata

Description
As a follow on for #23, this PR adds the ability to sign generated metadata.   This is an added security measure when doing signing certificate rotations or other metadata sharing.
With this option enabled, IdPs that poll for metadata updates can now verify the metadata with the existing signing certificates it has configured before accepting new certs.  This prevents an attack vector from man-in-the-middle or compromised metadata hosting locations (assuming the signing cert private key was not also compromised).
Checklist:

Issue Addressed: [X]
Link to SAML spec: [ ]
Tests included? [X]
Documentation updated? [X]

@cjbarth could I get a maintainer to review this PR?  I would like to get this in and then I can get #30 ready as well.


TESTING DO NOT merge

BUILD on-tftp #12 : FAILURE

BUILD Archiving » tian » PR_GATE #1 : FAILURE

test this please

test this please

BUILD on-tftp #17 : FAILURE


Add design considerations for Dynamic updating, fix architecture diagram

Codecov Report

Merging #312 (eada720) into master (f598de3) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #312   +/-   ##
=========================================
  Coverage     34.87%   34.87%
  Complexity      348      348
=========================================
  Files           114      114
  Lines          2162     2162
  Branches        183      183
=========================================
  Hits            754      754
  Misses         1354     1354
  Partials         54       54

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f598de3...eada720. Read the comment docs.


Start a for loop after youtube-dl is finished

i want to check for new videos in a youtube channel every day, i use cron to do that. at the moment i start youtube-dl directly from the cron. i want to delete some strings from the file name and created a shell script thats checks the files in the downloader folder. the cron now should start a shell script that
1.) checks for new files and downloas them if available
2.) rename new files in the output folder
but when i start the shell script manually the for loop starts before youtube-dl starts. i execute the script and get the error that there are no files to rename, then it jumps back to the terminal prompt. after a few moments youtube-dl starts checking for new videos and download videos if available.
how can i ensure that the for loop only starts after youtube-dl is finished, no matter how long youtube-dl needs. a simple 'sleep' should not enough because i can't say how long the downloads needs.

You don't need any scripts for that. Use --download-archive and output template for naming.


Solution of "Merge two sorted Linkedlist" #1244

class Solution {
public:
ListNode* mergeTwoLists(ListNode* list1, ListNode* list2) {
if(list1 ==NULL) return list2;               //Check for empty linkedlists
if(list2 ==NULL) return list1;
    ListNode* ptr=list1;                           //initialise a pointer to list1

    if(list1->val < list2->val) list1=list1->next;      //check for first smallest element of two lists
    else{
        ptr=list2;
        list2=list2->next;

    }
    ListNode* curr=ptr;                                         //initialise another pointer
    while(list1 && list2){                                       //add other nodes accordingly
        if(list1-> val < list2->val){
            curr->next=list1;
            list1=list1->next;
        }
        else{
            curr->next=list2;
            list2=list2->next;
        }
        curr=curr->next;
    }
    if(!list1) curr->next=list2;
    else curr->next=list1;

    return ptr;                                             //head node is returned

}

};

I have added the code for problem "Merge two sorted Linkedlist" issue #1244

Hii, I am interested in contributing the code of Solution of "Merge two sorted Linkedlist" problem in cpp. Please assign me this issue

Hello, Can you please assign this issue to me?
Thanks in advance :)

Hy , I'm Prashant Varma and Please Assign me this because I'm a DSA Mentor at GFG , If you will Assign this project to me then , I will not solve your problem but i suggest you , how to solve these kind of problems .
If ,You Agree with me then kindly contact me on my linkedin-https://www.linkedin.com/in/prashant-varma-666735230/

Sir, I am interested in contributing to this problem please assign me this issue.


Add option to convert to multiple formats

Short description

Thanks for this project. It has made the process of using poetry whilst providing requirements.txt and setup.py that bit easier
Would it be possible to add the option to pass multiple output formats or files to? For example:
dephell deps convert --from=pyproject.toml --to=setup.py,requirements.txt

And support something similar in pyproject.toml. For example
[tool.dephell.main]
from = "pyproject.toml"
to = "setup.py,requirements.txtt"

Why?
Currently, in my projects I have to run the following to achieve this
dephell deps convert
dephell deps convert --from=pyproject.toml --to=setup.py

And this change would make for a small quality of life improvement
Thank you for your time

I think #188 will cover your case. What do you think?

Yeah, I guess so. If not feel free to re-open. Thank you for the request :)

It would. Thanks very much 😊


Added CellEditing samples,data and templates

I am currently playing around with the generated React sample from this configuration and several things seem to not work.
Some errors are coming from the webGridCellEditCellTemplate:

the call to this.grid1 as there is no grid1 generated but rather hierarchicalGrid1 in the sample.
use of attribute textContext in IgrSelectItem gives:

Type '{ selected: boolean; textContent: any; value: any; key: string; }' is not assignable to type 'IntrinsicAttributes & IntrinsicClassAttributes<IgrSelectItem> & Readonly<IIgrSelectItemProps>'.
  Property 'textContent' does not exist on type 'IntrinsicAttributes & IntrinsicClassAttributes<IgrSelectItem> & Readonly<IIgrSelectItemProps>'.

use of size property in IgrSelect gives:

Type '{ children: any; size: string; key: string; change: (x: any) => void; }' is not assignable to type 'IntrinsicAttributes & IntrinsicClassAttributes<IgrSelect> & Readonly<IIgrSelectProps>'.
  Property 'size' does not exist on type 'IntrinsicAttributes & IntrinsicClassAttributes<IgrSelect> & Readonly<IIgrSelectProps>'.

Finally, the row islands in the sample show no data for some reason.
I am using igniteui-react version 18.6.0 for testing.

Not really sure where size and textContext were coming from, they were not in the templates so, but  the templates were used in the flat grid as well, so might have been causing it. I added new templates so it should be running okay now.


Reformatting Control Statement Chapter

Reformatted this chapter to be earlier: no data visualization tools used, more intro to boolean arrays, deleted sampling from a DataFrame (moved to Empirical prob chapter). This chapter is titled: Randomness and Control Statements
Edited group, merge, pivot chapter to be three sections instead of two.

FYI: The textbook's table of contents in textbook/_toc.yml has to be updated to reflect the file name changes and additions here. These are the warnings I received when building the textbook locally:
textbook-datascience-1$ manage build --all

…

textbook/intro.md:: WARNING: toctree contains reference to nonexisting document '05/placeholder5' [etoc.ref]
textbook/05/1/Intro.ipynb: WARNING: document isn't included in any toctree
textbook/05/2/Control_Statements_conditionals.ipynb: WARNING: document isn't included in any toctree
textbook/05/3/Control_Statements_Iteration.ipynb: WARNING: document isn't included in any toctree
textbook/05/randomness-and-control-statement.md: WARNING: document isn't included in any toctree
textbook/07/1/Merge.ipynb: WARNING: document isn't included in any toctree
textbook/07/2/Groups.ipynb: WARNING: document isn't included in any toctree
textbook/07/3/pivots.ipynb: WARNING: document isn't included in any toctree
textbook/07/merge-group-pivot.md:: WARNING: toctree contains reference to nonexisting document '07/1/Merge_groups' [etoc.ref]
textbook/07/merge-group-pivot.md:: WARNING: toctree contains reference to nonexisting document '07/2/Groups_pivots' [etoc.ref]

Cool. I'm still getting an error. (BTW you can probably build the textbook with your changes as well.)
$ manage build --all

textbook/07/merge-group-pivot.md:: WARNING: toctree contains reference to nonexisting document '07/2/pivots' [etoc.ref]
textbook/07/3/pivots.ipynb: WARNING: document isn't included in any toctree

I can just fix this up.

OK! I'm guessing that change will not offend.
Looks good to me. I can merge if it's ready to go.

Thanks for fixing that for me! It's ready now.


What's new for 4.7.2

What's new for 4.7.2
Fixes #4793
//cc @preetikr

Thanks for the thorough review and the comments, @svick! I'll address them in the next round of revisions.

Thanks for the reviews, @mairaw and @svick. I've addressed your comments.


[upgrade]  MoveVM cache issue around upgrades, trigger reconfig

0L's Hot Upgrade mechanism can cause block prologue to abort around the Upgrade Tick (round 2 of each epoch). This does not halt the network, since the block just gets rejected. Any round 2 block with transactions will get rejected. This means that the upgrade may not complete. Previously we tested this by disabling transactions on the main fullnodes for Carpe around the epoch period. The issue is documented here: https://github.com/OLSF/libra/issues/941
The definitive solution thanks to @sblackshear, is to also trigger a reconfiguration so that all the services reset, and among other things clear the Move cache. However there was no way to trigger the reconfiguration without also incrementing an epoch.
This PR addresses that and creates a new type of reconfig: upgrade_recofig(), which is a private function that can be called from rust in the block lifecycle, in this case during the upgrade event.
Changes:

[x] Move: add upgrade_reconfig() which only emits a reconfig event, however with the exact same config. Just to signal to the services they need to restart.
[x] Rust: add a dispatch to Move during the upgrade flow.

This looks good to me and matches me mental model of why the upgrade was failing>
This looks like like it gets deployed by all the nodes upgrading their binaries before the next coordinated upgrade.


Add quiet

Add --quiet flag and tidies handling of validation logging

Pull Request Test Coverage Report for Build 170

10 of 10 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.04%) to 91.86%

Totals

Change from base Build 165:
-0.04%

Covered Lines:
1185

Relevant Lines:
1290

💛  - Coveralls


[WIP] Legends

fixes #1397. Implements the new legends.
External API to work with legends
var layer = vis.getLayer(1);

// Layers supports 4 types of legends, and can be accessed via:

layer.legends.bubble;
layer.legends.choropleth;
layer.legends.category;
layer.legends.custom;

// Legends support the following methods (using bubble here as an example):

layer.legends.bubble.show();
layer.legends.bubble.hide();
layer.legends.bubble.set({ title: 'My legend' });
layer.legends.bubble.isAvailable(); // returns true if Maps API returned some data for the legend

// Legends trigger a 'change' event every time they are updated with some new data from the Maps API.

// Here's an example:
layer.legends.bubble.on('change', function (legend) {
  if (legend.isAvailable()) {
    // do something (eg: disable the button to select bubble legends)
    legend.show();
  } else {
    legend.hide();
  }
});

@nobuti ☝. This is what the BUILDER should be using. What do you think? Is there anything else you think we'll be needing?
Changes to viz.json
In order for CartoDB.js to know what legends should be displayed for each of the layers, we need to change the viz.json a bit. Here's an initial proposal, that includes the legends that the user has enabled in the BUILDER (we'll be supporting multiple types of legends for each of the layers):
{
  "id": "fa6cf872-fffa-4301-9a60-849cedba7864",
  "type": "CartoDB",
  "visible": true,
  "infowindow": { ... },
  "tooltip": { ... },
  "options": { ... },
  "legends": [
    { type: 'bubble', title: 'Barrios Legend' },
    { type: 'category', title: 'Categories Legend' },
    {
      type: 'custom',
      items: [
        { name: 'Category 1', type: 'color', color: '#CACACA' },
        { name: 'Category 2', type: 'color', color: '#FABADA' }
      ]
    }
  ]
}

@juanignaciosl @jatorre @javitonino  @gfiorav ☝ . I'd love to get some feedback from you guys about this change. I can open a separate issue to keep track of this work too.

I think that you actually wanted to ping @javitonino ;-)
LGTM. A couple of questions that came up to my mind while I was reading this:
1.- Order: currently, it's implicit, isn't it? Are legends displayed depending on its type?
2.- Type: one layer can only have one legend of each type, isn't it? Could this be a limitation, specially considering custom type?

1.- Order: currently, it's implicit, isn't it? Are legends displayed depending on its type?
I haven't thought about this. So far, CartoDB.js decides, for each layer, in which order they should be displayed.

2.- Type: one layer can only have one legend of each type, isn't it? Could this be a limitation, specially considering custom type?

With the new legends, a layer might have more than one legend (eg: if a layer is showing features as colorised bubbles, user could choose to add a bubble legend and a category legend)...

LGTM only one concern, in the UI the title can be checked or unchecked, showing or hiding the title even when it is not empty, is this taking it into account? Also, for categories I guess the builder only has to replace ìtems` as the items collection in the order the user decides, right?

Looks sharp to me, well done!

in the UI the title can be checked or unchecked, showing or hiding the title even when it is not empty, is this taking it into account?

Good catch @buti! I didn't take this into account. I can think of two options here:

The UI always sends a title when checkbox is checked (eg: "Color" or "My Legend").
We add support for a flag (showTitle) so that CartoDB.js knows when to display the given title or a default one.

I like the first one better, but we can go with the second one if you feel strong about it.

Also, for categories I guess the builder only has to replace ìtems` as the items collection in the order the user decides, right?

Only for custom legends, the editor will have to set the "categories" explicitly. You'll be able to do something like:
layer.legends.custom.set('items', [
  { type: 'color', name: 'Item1', color: '#CACACA' },
  { type: 'color', name: 'Item2', color: '#FABADA'}
]);

Would this work? For other types of legends, data will come from the Maps API.

@alonsogarciapablo it sounds perfect, if checked, we set the title to user's choice or layer's name by default. If unchecked, we set title to empty. Perfect as well the categories part.
well done!

@nobuti from what I understand, the name of the layer will always be displayed and is not customisable. Users can only choose a title for each of the legends:

@rochoa and @xavijam, would love your feedback here too!

@gfiorav @javitonino @juanignaciosl one question, is the backend (rails) ready for managing more than one legend per layer? I mean, right now, in the editor, the legend is linked to the layer definition model, so it's using that endpoint. Is this still valid? What should we consider in order to save the data with the new implementation?

About API, not sure, but here it goes my 2 cents:
var legends = layer.getLegends(); // -> legends collection
var size = legends.size();
var bubbleLegends = legends.getBubbleLegends(); // -> bubbles collection (a filter by type from legends?)
var firstBubble = bubbleLegends.first();
var secondBubble = bubbleLegends.at(1);
firstBubble.show();
firstBubble.hide();
firstBubble.append('perico');
firstBubble.prepend('de los palotes');
firstBubble.update({ title: 'whatever' });
firstBubble.isAvailable();

firstBubble.on('change', function () {
  console.log('hello world');
});

@nobuti legend is currently stored inside options "free schema" json, so storing it is not a problem. In order to display it backend presenters should be aware of the new legends (plural) attribute, but storing it should* work right out of the box.
*: there is some filtering for some risky options, but I'm almost sure that legends would be stored without doing anything.

Thanks for your feedback @xavijam.
Guys, I was thinking about the need of exposing multiple legends of a given type for the given layers (eg:  legends.getBubbleLegends();). Here're some of my thoughts:

If we went with exposing multiple legends... how would the client of this API differentiate between the different legends and know which is the one he/she should work with?. eg:
var layer = vis.getLayer(1);
var bubbleLegends = layer.getBubbleLegends(); // returns 3 different layers

// does that mean CartoDB.js is rendering 3 legends?
// how do I know what legend is begin rendered?
// if I want to customise the title of the bubble legend for a layer... do I need to change it in all three models?

Based on ☝ , and regarding the use case that @rochoa mentioned offline (eg: different color ramps/category legends based on the zoom level), I think that this could be handled by cartodb.js internally. A single legend of each type could be exposed in the API for each layer so that users can customise it (and also rendered) and cartodb.js could take care of everything else internally... eg: as the map is zoomed in/out, it can select the right categories from the response of the Maps API...

To summarise, while we could/might need this in the future, AFAIK we don't need it right now, so I suggest everything working with our current needs and worrying about this when we get to this particular thing. We'll need to change things anyway (eg: taking the zoom into account). Agree @nobuti @xavijam @rochoa? thanks!

good for me.

I wouldn't worry about the per zoom legends use case yet. But if we go with an API like the one @xavijam is proposing I think we are covering those future use cases. Answering your questions:

var bubbleLegends = layer.getBubbleLegends(); // returns 3 different layers

// does that mean CartoDB.js is rendering 3 legends?
No, you just get a reference to the collection.

// how do I know what legend is begin rendered?
I see several options, I don't know much about how we usually expose collections, but it could be:

layer.getBubbleLegends().visible();
layer.getBubbleLegends({ visible: true });
layer.getBubbleLegends().filter({ visible: true });

// if I want to customise the title of the bubble legend for a layer... do I need to change it in all three models?
I don't understand this question. Can you provide us with an example?

I'm missing here the raw/html legend type. We will need it for migrating old legends.

Yeah, quite important ^^^.

About the legends.getBubbleLegends(), if we consider that there is no possibility to have more than one bubble (or other type) by layer, it is likely it doesn't make sense to "pluralize" that method.

Talked to @xavijam and @rochoa. API we'll finally expose one legend of each type per layer for now. Internally, CartoDB.js might need to handle cases were the Maps API returns multiple metadata for the same legend type, but that we'll be taken care of internally and it'll be transparent in terms of the API...

Fine by me!

@nobuti @gfiorav I have updated the "Changes to viz.json" section in the PR description with all the different legend types that I'm supporting in this PR.

Merged! Great job @piensaenpixel! And sorry for such a big PR.
I will open a follow-up PR with some feedback from @rochoa and some things that are changing in the Maps API.
cc: @xavijam @nobuti @rochoa

👏 👏 👏