Enable request & response validation in data server and fix resulting issues

There was a bug in index.ts on dataserver leading to validation not actually taking place -- this fixes it

 Design & implement 'Case' API entity


docs: fix URL for Zitadel k8s deployment guide

The current URL does not link to the docs; removing the /docs/ path fixes the URL.

Thanks again for the PR!
If you are interested with some stickers for this contrib contact<EMAIL_ADDRESS>


PY/R: StationFinder: Min and Max Dates differ

Py: YYYY-MM-DD
R: M/D/YYYY
My vote is for ISO format

Actually, given they are vending data as M/D/YYYY, let's go with that for now - I'll change


False positive OOM events

Description
Hi,
we use a Kubernetes DaemonSet (a node agent) to monitor containerd EventBus and look for pods OOM events.
Recently, we started to notice several false positives, logs are always like these
containerd[8442]: time="2022-06-24T12:59:17.225386941Z" level=info msg="TaskOOM event &TaskOOM{ContainerID:4c0b7bc1225d148f13492e3cd4061b28d803701404abddc3d80431ec92d91d93,XXX_unrecognized:[],}"
but in "dmesg" output on the host there isn't a corresponding OOM stack trace.
Steps to reproduce the issue
none
Describe the results you received and expected
No false positives
What version of containerd are you using?
containerd containerd.io 1.4.7 3194fb46e8311ae0eeae5a7a5843573adfebb16d
Any other relevant information
# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

# runc  --version
runc version 1.0.0
commit: v1.0.0-0-g84113ee
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.3.1

# crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "k8s-pod-network",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "calico",
                "ipam": {
                  "type": "calico-ipam"
                },
                "dns": {}
              },
              "Source": "{\"datastore_type\":\"kubernetes\",\"ipam\":{\"type\":\"calico-ipam\"},\"kubernetes\":{\"kubeconfig\":\"/etc/cni/net.d/calico-kubeconfig\"},\"log_file_path\":\"/var/log/calico/cni/cni.log\",\"log_level\":\"info\",\"mtu\":1480,\"nodename\":\"ip-10-159-37-68.eu-south-1.compute.internal\",\"policy\":{\"type\":\"k8s\"},\"type\":\"calico\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"snat\":true,\"type\":\"portmap\"}"
            },
            {
              "Network": {
                "type": "bandwidth",
                "capabilities": {
                  "bandwidth": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"bandwidth\":true},\"type\":\"bandwidth\"}"
            }
          ],
          "Source": "{\n  \"name\": \"k8s-pod-network\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"calico\",\n      \"log_level\": \"info\",\n      \"log_file_path\": \"/var/log/calico/cni/cni.log\",\n      \"datastore_type\": \"kubernetes\",\n      \"nodename\": \"ip-10-159-37-68.eu-south-1.compute.internal\",\n      \"mtu\": 1480,\n      \"ipam\": {\n          \"type\": \"calico-ipam\"\n      },\n      \"policy\": {\n          \"type\": \"k8s\"\n      },\n      \"kubernetes\": {\n          \"kubeconfig\": \"/etc/cni/net.d/calico-kubeconfig\"\n      }\n    },\n    {\n      \"type\": \"portmap\",\n      \"snat\": true,\n      \"capabilities\": {\"portMappings\": true}\n    },\n    {\n      \"type\": \"bandwidth\",\n      \"capabilities\": {\"bandwidth\": true}\n    }\n  ]\n}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "nvidia-container-runtime": {
          "runtimeType": "io.containerd.runc.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        },
        "runc": {
          "runtimeType": "io.containerd.runc.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": null,
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": {
        "registry-1.docker.io": {
          "auth": {
            "username": "c3rv3dgr0up",
            "password": "x7UHrVx7kraNpph",
            "auth": "",
            "identitytoken": ""
          },
          "tls": null
        }
      },
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": ""
    },
    "disableTCPService": true,
    "streamServerAddress": "<IP_ADDRESS>",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "k8s.gcr.io/pause:3.2",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "ignoreImageDefinedVolumes": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.15.14",
  "lastCNILoadStatus": "OK"
}

# uname -a
Linux ip-10-159-37-68.eu-south-1.compute.internal 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.
No response

hi, it depends on the kernel behavior.
In v3.10, kernel will notify subscriber (containerd) at first before kill 1
And if the oom_kill has been disabled or the kernel doesn't find the task to kill 2, I think there is no call trace in kern log.
IMO, you can check the target thread is alive and the cgroup setting.

Hi, @fuweid , sorry for delay but I was on holiday.
Could you elaborate your suggestion?
What is the best strategy to monitor OOM events from Kubernetes?
Thanks again

@mimmus Basically, the containerd will report the OOM_EVENT if there is any out of memory.
But it is related to kernel. I think you  can check the memory cgroup setting, like memory.oom_control,
and confirm the kernel behavior for your pod.
// https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
10. OOM Control

memory.oom_control file is for OOM notification and other controls.

Memory cgroup implements OOM notifier using the cgroup notification
API (See cgroups.txt). It allows to register multiple OOM notification
delivery and gets notification when OOM happens.

To register a notifier, an application must:
 - create an eventfd using eventfd(2)
 - open memory.oom_control file
 - write string like "<event_fd> <fd of memory.oom_control>" to
   cgroup.event_control

The application will be notified through eventfd when OOM happens.
OOM notification doesn't work for the root cgroup.

You can disable the OOM-killer by writing "1" to memory.oom_control file, as:

	#echo 1 > memory.oom_control

If OOM-killer is disabled, tasks under cgroup will hang/sleep
in memory cgroup's OOM-waitqueue when they request accountable memory.

For running them, you have to relax the memory cgroup's OOM status by
	* enlarge limit or reduce usage.
To reduce usage,
	* kill some tasks.
	* move some tasks to other group with account migration.
	* remove some files (on tmpfs?)

Then, stopped tasks will work again.

At reading, current status of OOM is shown.
	oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
	under_oom	 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
				 be stopped.)

Perhaps these info aren't really applicable to my use case.
I think that a consolidated OS like CentOS already has OOM killer enabled.
I'm trying to understand if containerd events are reliable and, if not, why.

Perhaps these info aren't really applicable to my use case.
I think that a consolidated OS like CentOS already has OOM killer enabled.
I'm trying to understand if containerd events are reliable and, if not, why.
Thanks again

I think it will be helpful if you can provide the cgroup status when you see OOM event and no OOM kill log.

We saw similar behavior in our GKE Kubernetes clusters:
containerd --version
containerd github.com/containerd/containerd 1.4.8 7eba5930496d9bbe375fdf71603e610ad737d2b2

We hit this during a rollout when containers were terminated. kube_pod_container_status_terminated_reason emitted by kube-state-metrics shows that pod was OOMKilled. IIUC this is basically from containerd.
We cannot find any related even in kernel logs that would suggest that OOM happened. The weird thing is that this happened to 24 different containers running on different nodes during the rollout (during ~10 minutes).
I checked the node setting regarding OOM and it is following.
$ cat /sys/fs/cgroup/memory/memory.oom_control
oom_kill_disable 0
under_oom 0
oom_kill 0

OOMkilled => Pod exited with code 137
(@dohnto strange to see this during a rollout)
But this is NOT true if container contains more than one process and the process with PID not 1 is killed.
Thus I was looking for a reliable way to monitor OOMs....

Thanks for the information. Will take a look and reproduce it.

@mimmus @dohnto
Currently, containerd is using cgroup memory event control to receive event with epoll. There are two events to trigger epoll_wait: 1) OOM event from page_fault 2) the memory cgroup has been deleted.
For the first one, I tried to produce your cases but there is always OOM related log in dmesg, even if the killed process is not init process, like the following yaml describes. I highly recommended to check the pod's cgroup oom setting instead of root one.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-ng-deployment
  labels:
    app: stress-ng
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stress-ng
  template:
    metadata:
      labels:
        app: stress-ng
    spec:
      # shareProcessNamespace: true
      containers:
      - name: stress
        image: alexeiled/stress-ng:0.11.19-ubuntu
        command:
          - /bin/bash
          - -c
          - |
            stress-ng --vm 1 --vm-bytes 1G --timeout 120s &
            wait
            echo "start to sleep"
            sleep 1d
        resources:
          requests:
            memory: "128Mi"
            cpu: "500m"
          limits:
            memory: "128Mi"
            cpu: "500m"

stress-ng is child process of bash and it also was killed by oom.
$ sudo ctr event
2022-08-06 10:14:27.774982503 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:27.944300703 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.066265159 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.189234386 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.370020494 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.544772029 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.677114941 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.848536314 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:28.978731866 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}
2022-08-06 10:14:29.158928893 +0000 UTC k8s.io /tasks/oom {"container_id":"7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0"}

$ sudo dmesg -Tw
[Tue Aug  9 10:58:59 2022] stress-ng-vm invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=1000
[Tue Aug  9 10:58:59 2022] CPU: 2 PID: 69012 Comm: stress-ng-vm Tainted: G           OE     5.18.9-arch1-1 #1 137f0035b2ece06cb65382579db27e9de66af504
[Tue Aug  9 10:58:59 2022] Hardware name: HP HP ZHAN 66 Pro 14 G4 Notebook PC/87E2, BIOS T71 Ver. 01.06.03 08/26/2021
[Tue Aug  9 10:58:59 2022] Call Trace:
[Tue Aug  9 10:58:59 2022]  <TASK>
[Tue Aug  9 10:58:59 2022]  dump_stack_lvl+0x48/0x5d
[Tue Aug  9 10:58:59 2022]  dump_header+0x4a/0x1fc
[Tue Aug  9 10:58:59 2022]  oom_kill_process.cold+0xb/0x10
[Tue Aug  9 10:58:59 2022]  out_of_memory+0x27e/0x520
[Tue Aug  9 10:58:59 2022]  mem_cgroup_out_of_memory+0x135/0x150
[Tue Aug  9 10:58:59 2022]  try_charge_memcg+0x6fd/0x7c0
[Tue Aug  9 10:58:59 2022]  ? __alloc_pages+0xea/0x230
[Tue Aug  9 10:58:59 2022]  charge_memcg+0x8e/0xf0
[Tue Aug  9 10:58:59 2022]  __mem_cgroup_charge+0x2c/0x80
[Tue Aug  9 10:58:59 2022]  __handle_mm_fault+0x744/0x14c0
[Tue Aug  9 10:58:59 2022]  ? generic_exec_single+0x51/0xb0
[Tue Aug  9 10:58:59 2022]  handle_mm_fault+0xb2/0x280
[Tue Aug  9 10:58:59 2022]  do_user_addr_fault+0x1be/0x680
[Tue Aug  9 10:58:59 2022]  exc_page_fault+0x74/0x170
[Tue Aug  9 10:58:59 2022]  ? asm_exc_page_fault+0xc/0x30
[Tue Aug  9 10:58:59 2022]  asm_exc_page_fault+0x22/0x30
[Tue Aug  9 10:58:59 2022] RIP: 0033:0x471bd4
[Tue Aug  9 10:58:59 2022] Code: 00 00 4d 39 fd 73 7a 48 85 ed 4d 89 ec 0f 95 44 24 08 0f b6 44 24 08 4c 89 6c 24 08 49 89 dd 4c 89 e3 41 89 c4 e8 5c f2 f9 ff <88> 03 d0 c8 49 83 c5 01 88 43 01 d0 c8 88 43 02 d0 c8 88 43 03 d0
[Tue Aug  9 10:58:59 2022] RSP: 002b:00007fff0ba748f0 EFLAGS: 00010282
[Tue Aug  9 10:58:59 2022] RAX: 00000000fe9926dc RBX: 00007f1d3386e000 RCX:<PHONE_NUMBER>72f870
[Tue Aug  9 10:58:59 2022] RDX:<PHONE_NUMBER>7326dc RSI: 000000008e27ed98 RDI: 00000000a468232a
[Tue Aug  9 10:58:59 2022] RBP:<PHONE_NUMBER>000000 R08:<PHONE_NUMBER>000000 R09: 00007fff0ba72197
[Tue Aug  9 10:58:59 2022] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000206 R12:<PHONE_NUMBER>000000
[Tue Aug  9 10:58:59 2022] R13:<PHONE_NUMBER>f43000 R14: 00007fff0ba74ae0 R15: 00007f1d6be56000
[Tue Aug  9 10:58:59 2022]  </TASK>
[Tue Aug  9 10:58:59 2022] memory: usage 131072kB, limit 131072kB, failcnt 43877
[Tue Aug  9 10:58:59 2022] swap: usage 0kB, limit<PHONE_NUMBER>740988kB, failcnt 0
[Tue Aug  9 10:58:59 2022] Memory cgroup stats for /kubepods/pod8eeae11a-6179-44d9-a7ab-f87aac75bc2e:
[Tue Aug  9 10:58:59 2022] anon 130641920
                           file 2207744
                           kernel 1298432
                           kernel_stack 81920
                           pagetables 507904
                           percpu 62600
                           sock 0
                           vmalloc 8192
                           shmem 2207744
                           file_mapped 2207744
                           file_dirty 0
                           file_writeback 0
                           swapcached 0
                           anon_thp 125829120
                           file_thp 0
                           shmem_thp 0
                           inactive_anon 132820992
                           active_anon 16384
                           inactive_file 0
                           active_file 0
                           unevictable 12288
                           slab_reclaimable 94856
                           slab_unreclaimable 327632
                           slab 422488
                           workingset_refault_anon 0
                           workingset_refault_file 0
                           workingset_activate_anon 0
                           workingset_activate_file 0
                           workingset_restore_anon 0
                           workingset_restore_file 0
                           workingset_nodereclaim 0
                           pgfault 513831
                           pgmajfault 0
                           pgrefill 1775
                           pgscan 1782
                           pgsteal 8
                           pgactivate 1775
                           pgdeactivate 1775
                           pglazyfree 0
                           pglazyfreed 0
                           thp_fault_alloc 47344
                           thp_collapse_alloc 0
[Tue Aug  9 10:58:59 2022] Tasks state (memory values in pages):
[Tue Aug  9 10:58:59 2022] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Tue Aug  9 10:58:59 2022] [  67509] 65535 67509      243        1    24576        0          -998 pause
[Tue Aug  9 10:58:59 2022] [  67541]     0 67541     4595      783    77824        0          -997 bash
[Tue Aug  9 10:58:59 2022] [  67554]     0 67554     6319     1552    69632        0          1000 stress-ng
[Tue Aug  9 10:58:59 2022] [  67555]     0 67555     6320      871    65536        0          1000 stress-ng-vm
[Tue Aug  9 10:58:59 2022] [  69012]     0 69012   268464    32105   323584        0          1000 stress-ng-vm
[Tue Aug  9 10:58:59 2022] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0,mems_allowed=0,oom_memcg=/kubepods/pod8eeae11a-6179-44d9-a7ab-f87aac75bc2e,task_memcg=/kubepods/pod8eeae11a-6179-44d9-a7ab-f87aac75bc2e/7afc8d42d0089b0c0880d2e5c55eb16a200c9f769100c22a399d8cb667f8b8b0,task=stress-ng-vm,pid=69012,uid=0
[Tue Aug  9 10:58:59 2022] Memory cgroup out of memory: Killed process 69012 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:58:59 2022] Memory cgroup out of memory: Killed process 69013 (stress-ng-vm) total-vm:1073856kB, anon-rss:126892kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:58:59 2022] Memory cgroup out of memory: Killed process 69015 (stress-ng-vm) total-vm:1073856kB, anon-rss:126892kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:58:59 2022] Memory cgroup out of memory: Killed process 69016 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:58:59 2022] Memory cgroup out of memory: Killed process 69021 (stress-ng-vm) total-vm:1073856kB, anon-rss:126892kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69022 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69023 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69024 (stress-ng-vm) total-vm:1073856kB, anon-rss:126892kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69025 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69027 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000
[Tue Aug  9 10:59:00 2022] Memory cgroup out of memory: Killed process 69030 (stress-ng-vm) total-vm:1073856kB, anon-rss:127156kB, file-rss:1260kB, shmem-rss:4kB, UID:0 pgtables:316kB oom_score_adj:1000

k8s-poc cat /sys/fs/cgroup/kubepods/pod4990c3db-4034-43b8-943c-9ded470497c6/memory.events
low 0
high 0
max 5688
oom 153
oom_kill 153
oom_group_kill 0

For the second one, I was thinking there is data race to check cgroup stat because @dohnto mentions it can happen during the rollout. But I still can't reproduce. Based on our codebase, if using cgroupv1, the cgroup stat always returns Delete if there is any subsystem cgroup has been deleted. If not, containerd shim will send the OOM event out.
// https://github.com/containerd/containerd/blob/release/1.6/metrics/cgroups/v1/oom.go#L142
func (o *oomCollector) process(fd uintptr, event uint32) {
	// make sure to always flush the fd
	flush(fd)

	o.mu.Lock()
	info, ok := o.set[fd]
	if !ok {
		o.mu.Unlock()
		return
	}
	o.mu.Unlock()
	// if we received an event but it was caused by the cgroup being deleted and the fd
	// being closed make sure we close our copy and remove the container from the set
	if info.c.State() == cgroups.Deleted {
		o.mu.Lock()
		delete(o.set, fd)
		o.mu.Unlock()
		unix.Close(int(fd))
		return
	}
	atomic.AddInt64(&info.count, 1)
	for _, t := range info.triggers {
		t(info.id, info.namespace, info.c)
	}
}

All the systems are using the oom control event to watch the OOM, which I think it is reliable.
I am still wondering there is something wrong about setting or data-race. More details will be appreciated.

@mimmus I'm going to close it because there is no any update from your side. Please feel free to reopen it if you have new clue.

@mimmus : I am observing a similar issue where Kube-scheduler got restarted due to the leadership election lost, but when we inspected the container using "crictl inspect" it shows that it got restarted due to reason as OOMKilled and Exit code 1.
And in the dmesg on the host, there is no corresponding trace of OOM kill. Even we are monitoring the metrics of this pod and overall it hasn;t exceeded the memory limits.
Please find the below details and let me know if you required any more additional details:
runc  --version
runc version 1.1.2
commit: v1.1.2-0-ga916309
spec: 1.0.2-dev
go: go1.17.11
libseccomp: 2.5.1
crictl info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
},
"cniconfig": {
"PluginDirs": [
"/opt/cni/bin"
],
"PluginConfDir": "/etc/cni/net.d",
"PluginMaxConfNum": 1,
"Prefix": "eth",
"Networks": [
{
"Config": {
"Name": "cni-loopback",
"CNIVersion": "0.3.1",
"Plugins": [
{
"Network": {
"type": "loopback",
"ipam": {},
"dns": {}
},
"Source": "{"type":"loopback"}"
}
],
"Source": "{\n"cniVersion": "0.3.1",\n"name": "cni-loopback",\n"plugins": [{\n  "type": "loopback"\n}]\n}"
},
"IFName": "lo"
},
{
"Config": {
"Name": "k8s-pod-network",
"CNIVersion": "0.3.1",
"Plugins": [
{
"Network": {
"type": "calico",
"ipam": {
"type": "calico-ipam"
},
"dns": {}
},
"Source": "{"datastore_type":"kubernetes","ipam":{"assign_ipv4":"true","assign_ipv6":"false","ipv4_pools":["<IP_ADDRESS>/16"],"type":"calico-ipam"},"kubernetes":{"kubeconfig":"/etc/cni/net.d/calico-kubeconfig"},"log_level":"info","mtu":1400,"nodename":"<IP_ADDRESS>","policy":{"type":"k8s"},"type":"calico"}"
},
{
"Network": {
"type": "portmap",
"capabilities": {
"portMappings": true
},
"ipam": {},
"dns": {}
},
"Source": "{"capabilities":{"portMappings":true},"snat":true,"type":"portmap"}"
}
],
"Source": "{\n  "name": "k8s-pod-network",\n  "cniVersion": "0.3.1",\n  "plugins": [\n    {\n      "type": "calico",\n      "log_level": "info",\n      "datastore_type": "kubernetes",\n      "mtu": 1400,\n      "nodename": "<IP_ADDRESS>",\n      "ipam": {\n        "assign_ipv6": "false",\n        "assign_ipv4": "true",\n        "ipv4_pools": ["<IP_ADDRESS>/16"],\n        "type": "calico-ipam"\n      },\n      "policy": {\n          "type": "k8s"\n      },\n      "kubernetes": {\n          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"\n      }\n    },\n    {\n      "type": "portmap",\n      "snat": true,\n      "capabilities": {"portMappings": true}\n    }\n  ]\n}"
},
"IFName": "eth0"
}
]
},
"config": {
"containerd": {
"snapshotter": "overlayfs",
"defaultRuntimeName": "runc",
"defaultRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": null,
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"untrustedWorkloadRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": null,
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"runtimes": {
"runc": {
"runtimeType": "io.containerd.runc.v2",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": null,
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
}
},
"noPivot": false,
"disableSnapshotAnnotations": true,
"discardUnpackedLayers": false,
"ignoreRdtNotEnabledErrors": false
},
"cni": {
"binDir": "/opt/cni/bin",
"confDir": "/etc/cni/net.d",
"maxConfNum": 1,
"confTemplate": "/etc/cni/net.d/10-calico.conflist",
"ipPref": ""
},
"registry": {
"configPath": "",
"mirrors": null,
"configs": null,
"auths": null,
"headers": null
},
"imageDecryption": {
"keyModel": "node"
},
"disableTCPService": true,
"streamServerAddress": "<IP_ADDRESS>",
"streamServerPort": "0",
"streamIdleTimeout": "4h0m0s",
"enableSelinux": false,
"selinuxCategoryRange": 1024,
"sandboxImage": "dockerhub.cisco.com/mag-docker/pause:3.6",
"statsCollectPeriod": 10,
"systemdCgroup": false,
"enableTLSStreaming": false,
"x509KeyPairStreaming": {
"tlsCertFile": "",
"tlsKeyFile": ""
},
"maxContainerLogSize": 262144,
"disableCgroup": false,
"disableApparmor": false,
"restrictOOMScoreAdj": false,
"maxConcurrentDownloads": 3,
"disableProcMount": false,
"unsetSeccompProfile": "",
"tolerateMissingHugetlbController": true,
"disableHugetlbController": true,
"device_ownership_from_security_context": false,
"ignoreImageDefinedVolumes": false,
"netnsMountsUnderStateDir": false,
"enableUnprivilegedPorts": false,
"enableUnprivilegedICMP": false,
"containerdRootDir": "/data/docker/containerd/daemon",
"containerdEndpoint": "/run/containerd/containerd.sock",
"rootDir": "/data/docker/containerd/daemon/io.containerd.grpc.v1.cri",
"stateDir": "/var/run/docker/containerd/daemon/io.containerd.grpc.v1.cri"
},
"golang": "go1.17.11",
"lastCNILoadStatus": "OK",
"lastCNILoadStatus.default": "OK"
}
uname -a
Linux mag-master-10-106-172-176 5.4.0-122-generic #138~18.04.1 SMP Sun Jul 3 05:55:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Attached the logs during timestamp Mar 15 16:41:xx
containerd_selected.log
Below containerd ID is not restarted due to OOM and there is no trace in kernel log for OOM
root@mag-master-10-106-172-176:~# cat /tmp/container.log | grep 088c6d9d15683  | grep OOM
Mar 15 16:41:51 mag-master-10-106-172-176 containerd[7377]: time="2023-03-15T16:41:51.218228855Z" level=info msg="TaskOOM event &TaskOOM{ContainerID:088c6d9d1568341c170176f4934a4fd95c8ddd67335151f1b768ef1f9bd8a7e7,XXX_unrecognized:[],}"
root@mag-master-10-106-172-176:~#


ModuleNotFoundError: No module named 'ldm'

After following the directions, the txt2img.py script itself doesn't seem to recognize the LDM we created with the .yaml, though it exists in the .\anaconda3\envs. Do I perhaps have the wrong version of python or something? (I'm using 3.10) I don't see anything specified.
txt2img.py", line 15, in <module>
    from ldm.util import instantiate_from_config
ModuleNotFoundError: No module named 'ldm'

Also I found manually running 'pip install ldm' would install the wrong package, then it will ask for ldm.utils if I go this route. ref: https://github.com/CompVis/latent-diffusion/issues/71 but this looks like it was using an online notebook

I fixed this by using sys.path.append('<my path to stable-diffusion parent directory>) before the ldm imports
And then with python ran scripts/txt2img.py from the parent directory

@kisenera is right.  This issue can be closed.
Another name instead of "ldm" is preferred still.


Possible client update?

Hey Travis,
Hope you have been well! I still use this plugin often when developing Shopify stores for clients and it always works wonderfully - thanks for all the effort put into this.
I am slowly moving over to Nuxt since it seems Gridsome hasn't been updated in a while (although its definitely not as easy to set up as Gridsome is), but in the meantime I was wondering if it would be possible to update the Api client to the latest 2022-7 version? There has been some parts added to the graphql api and it would be great to make use of them.
If not its totally cool. I get that this may no longer be in your "stack" since there hasn't been much Gridsome updates (although a small update did happen last week for Vue 2.7).
Also, if you haven't seen it, I updated that old Shopify/Sanity repo you helped me on a long time ago which is on my Github page!

@ziopod I am confused on the metafield change... is metafields no longer viable to use on the product query?

@ziopod @thetre97 I made a fork of this repo and updated everything for the current 2023-01 API → Here is the fork. I have tested this out and it is working just fine... you can use it by running yarn add https://github.com/rylanharper/gridsome-source-shopify
Of course I am still a bit confused about the best approach for the metafields tags, I currently have the following where you would simply put in a color metafield on the product variants:
metafields(identifiers: { namespace: "custom", key: "color" }){
  key
  value
  namespace
}

Again, not sure if this repo if still supported, but I can make a pull request if so. Keep me updated. If not you can just use the fork I made:)

@rylanharper As you need to specify specific identifiers (which can be an array of objects containing namespace + key, as above), I guess it would need a plugin config option where you specify the metafields you want to fetch?

It would, yes.. Haha I have no idea how to do that though unfortunately 😖 The fork I made just uses the current 2023-01 client with updated queries (articles, shop, product, etc.). I also changed the transformedSrc resolver to receive the new image url tag instead of the deprecated originalSrc tag. On top of this, I removed tags that are no longer used.
The metafields tag I made above (which is in the fork I made) just takes in a basic namespace & key I was using (in this case color), but you totally correct in that it would be an array of unique objects.
It seems like a lot of work to add that within the plugin config options though.. Its not an issue if that can't be added in.


How to change resolution when Broadcasting??

Users can convert the video quality when a slow network access when it is live.
Please help me!!!

Firefox supports applyConstraints  API:
var width = 1280;
var height = 720;

var supports = navigator.mediaDevices.getSupportedConstraints();

var constraints = {};
if (supports.width && supports.height) {
    constraints = {
        width: width,
        height: height
    };
}

connection.applyConstraints({
    video: constraints
});

Chrome, on the other hand, requires renegotiation:
connection.getAllParticipants().forEach(function(uid) {
    var user = connection.peers[uid];

    user.peer.getLocalStreams().forEach(function(localStream) {
        user.peer.removeStream(localStream);
    });
});

connection.attacchStreams = [];

// or:
// connection.attachStreams.forEach(function(stream) { stream.stop(); });

connection.mediaConstraints = {
    audio: true,
    video: {
        mandatory: {
            maxWidth: 1920,
            maxHeight: 1080
        },
        optional: []
    }
};

connection.addStream({
    audio: true,
    video: true,
    oneway: true
});

Thank you!!!

Currently at version 2.2, then there is no applyConstraints function. So how to update it. Many thanks.

For v2.2.2:
function applyConstraints(stream) {
    var width = 1280;
    var height = 720;

    var supports = navigator.mediaDevices.getSupportedConstraints();

    var constraints = {};
    if (supports.width && supports.height) {
        constraints = {
            width: width,
            height: height
        };
    }

    stream.getVideoTracks().forEach(function(track) {
        track.applyConstraints(constraints);
    });
}

applyConstraints(connection.attachStreams[0]);

Thanks! I am most grateful!
But can you explain to me this:
When the Open Room, you can change the resolution but Join Room then don't change it. Here I want the clients can change the resolution ... Is there any way to help me.

Code here!!
<pre> connection.applyConstraints({                             video: constraints                         },_obj_live.streamid); </pre>

Hi Muaz-Khan!! When I change resolution client side then get the message:
OverconstrainedError: Constraints could be not satisfied.

@muaz-khan Can you please provide an example how to change video resolution in v3 for Chrome and Firefox?

Added new demo:

DemoTitle
TestLive
ViewSource

Change Video Resolutions in your Live Sessions
Demo
Source

Thanks!!  But :
On FireFox tested:
A broadcasting to B.. When B change resolution then A also change resolution...
P/s : Here I want to when B changes the resolution, only B changes, then The A keep the resolution
@muaz-khan Can you please provide an example case this.
On Chrome : Change incorrect... . Tested on Version 49.0.2623.110 (64-bit). MacOS

@muaz-khan : Please help me!


[Bug][Lake] repos table is not getting cleared when Project/Connection is deleted

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
After deleting Project/Connection of my github datasource, the repos table was not cleared and the dropdown showed the deleted Project's/Connection's repos as well. Refer screenshot.

What do you expect to happen
repos of deleted Project/Connection should also get deleted from repos table.
How to reproduce

Create a github connection and add to a project.
Sync data from github including repositories
Delete Project and Connection
View repos table. The repos from the deleted Project / Connection is still visible.
View the default github dashboard as shown in below screenshot. The repos from deleted Project/Connection are still visible in the dropdown.

Anything else
Problem happens always.
Version
0.21.0-beta5
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@sayeedhussain Have you deleted the data scope and connection?
After I deleted the project, I pressed 1 and 2 to delete the data scope and connection respectively. The data in the repos table was indeed deleted.

@abeizn Will re test this in upcoming beta7 and get back.

@abeizn Issue not happening in 0.21.0-beta7. Closing this card.


PomBase2UniProt.tsv  replace by GP2protein....

PomBase2UniProt.tsv  replace by GP2protein....

Done. The file name has not changed, but the content is now up to date.

I've changed the Chado load configuration to point to that new location rather than the getting the file from the EBI FTP site.


feat(filename): generate filename from abs path

Update build step to use c++20 and add libc++ so we can use the string helper methods and pass project root so we can determine a project root. Not 100% bulletproof, but something we can keep iterating/improving in the future as most use-cases show up.

fixes https://github.com/getsentry/profiling-node/issues/73


Rounded corners?

Any chance you've played with rounded corners and would be interested in adding it?
Thanks for the plugin!

Soreau made https://github.com/soreau/pixdecor that supports rounded corners with shadows. Its shadows look a bit different though.
I am not planning to add this right now, but for reference: The website that provided the source for core of the gaussian shadow also has a version for rounded rectangles. https://madebyevan.com/shaders/fast-rounded-rectangle-shadows/
However I derived the glow shader directly with a CAS integral solver, it would require a bit more thinking to make it work with rounded corners.

I have a formula for arcs too now, so it would be possible to implement rounded corners for glow too.
https://timgott.github.io/neon/
If I feel like working on this again some day I might add it.


Over resource limits on Streamlit Cloud

Hey there :wave: Just wanted to let you know that your app on Streamlit Cloud deployed from this repo has gone over its resource limits. Access to the app is temporarily limited. Visit the app to see more details and possible solutions.

Hello There 👋
Yes, the problem you mentioned occurs when a large number of visitors visit the page. I didn't think this would be a problem as there was more demand than I expected. The situation you mentioned has been resolved. I might try other solutions in case it happens again in the future. I'm closing the issue as the issue you mentioned has been resolved.
Thank you.


[json] problem with comment after literal

Issue Type: Bug
In my user settings i set
{
    "workbench.sideBar.location": "left"//note
}

when i change the value of something set from interface, the result values is that
{
    "workbench.sideBar.location": "left",
    "php.suggest.basic": false//note
}

and give me a error ("[jsonc] Value expected")
VS Code version: Code 1.24.1 (24f62626b222e9a8313213fb64b10d741a326288, 2018-06-13T17:51:32.889Z)
OS version: Windows_NT x64 6.1.7601

Verified that I no longer see the bug. The comment still ends up the second line which seems unexpected


[Feature Request] Add visual status rendering to Logic Gate

Concept:

A small amount of visual feedback would help troubleshoot designs involving the Logic gate. (Colors used & design were chosen for clarity. Use what you like, ignore the rest)

Visual feedback indicating a valid redstone connection is "found". Indicated by two bright green parallel lines in the concept image.
Visual feedback indicating an active redstone signal. Indicated by a red square nestled inside the "arrow" in the concept image.

Thanks!

The logic blocks avoid that kind of thing for full efficiency. Visual changes like that is what can cause performance issues for high speed redstone logic. Also The One Probe gives you more information when looking at a logic block.


Better control of caching logic by following headers and a manual option to turn of caching for a request

This PR adds support for better control of what is cached and when - first of, using HttpRequestMessage.Properties/Options with a specific key allows one to disable checking the cache for a request. It also follows the Cache-Control headers from the server, including the max-age if it's lower than the specified cache time.
Issue: https://github.com/thomasgalliker/HttpClient.Caching/issues/18

I‘ll have a look at it. Thanks for your effort.


WIP: STORM-2648/STORM-2357: Add storm-kafka-client support for at-most-onc…

…e processing and a toggle for whether messages should be emitted with a message id when not using at-least-once
See https://issues.apache.org/jira/browse/STORM-2357 and https://issues.apache.org/jira/browse/STORM-2648.
I'd like to get some opinions on whether this approach is a good idea, or whether I've overlooked a better option, before finishing this up with some tests. I don't love that we'll end up with 3 different committing behaviors.
In 2357 it was noted that the spout doesn't currently support true at-most-once, because using Kafka's auto commit option leaves the possibility that the spout receives a tuple, emits it to the topology, crashes and recovers, and then receives and emits the same tuple. The linked issue suggests solving this by committing polled offsets before emitting them to the topology, which is an option added here.
2648 notes that there is currently no way to make Storm track messages when using auto commit with this spout. This prevents Storm UI from showing the complete latency for the spout, and I would assume also prevents max spout pending from having an effect. I've added a toggle to KafkaSpoutConfig to force the spout to emit messages with message ids, even when using auto commit or the at-most-once option. The spout does nothing on ack or fail when not doing at-least-once.
I'd like to keep the spout config simple for the user, so I've added a processing guarantee setting corresponding to the standard at-least-once code path, the path that uses auto commit, and the path that commits offsets before emitting any tuples.

@srdo can you please assign JIRA's to you and mark them as in progress as you work on them and/or submit a pull request. Thanks.

Yes, I forgot.

Conceptually the changes look good to me.  I have not dug into it in great detail yet, but I do like the direction of the change.
I would also like to see the documentation and examples updated to reflect the new change.

+1

@srdo reviewing it

Added some tests and updated the docs.

@hmcl Are you reviewing this, or are you satisfied with it?

Still +1

@hmcl I don't mean to try to rush you, but please let me know if you're still reviewing. If not I'll probably merge in the next few days.

@srdo apologies for the delay. I will finish today.

Thanks :)

@hmcl Are you still reviewing this? It's fine if you don't have time to look at this, but please say so. I'd like to not keep holding this up.

This PR has been waiting for about 2 months, and once it gets +1 and no -1, it can be merged.
I'm +1 and will just merge.
@hmcl Please vote -1 later and rollback the merge if you have concern about the patch and would want to vote -1.

Merged via https://github.com/apache/storm/commit/48f6969027e7b02a5b9220577189d3911aa2226d
@srdo
Sorry I forgot to add auto close message while squashing commits. Could you close this?
And please craft the patch for 1.x branch since it doesn't looks like a clean cherry-pick.
Thanks in advance!

Thanks for reviews. Opened the 1.x version here https://github.com/apache/storm/pull/2353.


fix(postgres): cast list of enums to their correct type (#417)

This adds an explicit type cast to enum arrays

Now using sql.identifier. Also changed it to work not only with enums but custom classes, too.

This looks great @danielbuechele! Are you able to add any tests for it?

Added test for the function I modified

@benjie @danielbuechele What is the status of this?  Looks like it's just blocked by some tests.  I can add these if that'd be helpful.

@danscan I think we just need CI to pass before we can merge.

It seems to just be lint issues that need resolving.

I'll fix them tomorrow! (and get my linter working)

Huh, linter is now passing, but a lot of other test failing. I don't really know where this is coming from.

The errors centre around:
GraphQLError: function c.types_mutation(unknown, unknown, unknown, text[], unknown, numrange) does not exist
GraphQLError: function c.types_query(unknown, unknown, unknown, text[], unknown, numrange) does not exist
Here's the relevant definitions:
https://github.com/postgraphql/postgraphql/blob/886f8752f03d3fa05bdbdd97eeabb153a4d0343e/examples/kitchen-sink/schema.sql#L155-L156
Seems the integer array is being interpreted as a text array instead.


Remove leading spaces from code tag code (minor)

I forgot when transferring the html that the code tag leaves the leading spaces in. Small fix.

:star: ⬅


Evaluate the MCCI 4610 board

A better alternative is the P-NUCLEO-LRWAN2 which I'm testing at the moment.


Looking for a Mask object 'M' which derives from .evl files

Your library provides excellent functionality and I have been utilizing the Mask object M in particular which I derive from your regions object to perform clustering operations. It works well however I am looking for an equivalent Mask 'M' object which derives from the line file (.evl) . Could you create this or give e a hint on how you may create it. Thanks.

Thank-you for sending the feature. It works great!


Added counting speed game

Hey @arnavmenon, Hope you have read the guidelines.
Please add your program in directory CountSpeed/USERNAME and give you PR again.
Thank You. :)


GH-37485: [C++][Skyhook] Don't use deprecated BufferReader API

Rationale for this change
GH-37360 deprecated BufferReader(const uint8_t*, int64_t).
What changes are included in this PR?
Use BufferReader(std::shared_ptr<Buffer>) instead.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.

Closes: #37485

@github-actions crossbow submit test-skyhook-integration

CI looks unrelated, will merge. Thanks @kou !


List of all Regions and Zones

Dear team,
I'm trying to choose proper region for my instance following "some" criteria I have, but can't seem to find a way to list all available regions and zones... Especially I need the country or city information about each region as it is in the web interface.
Any samples how to complete that will be much appreciated!
Thanks

Hi @sniffski, hope this helps - feel free to reopen if not.

Hi @sniffski, hope this helps - feel free to reopen if not.

In fact that's exactly what I'm using, but currently parsing the table in a JSON and then using that JSON for further processing... I was hoping there is an API call I can use that will give me the same filtering experience as per this page...


Add examples for SKLearn regression

Add regression example script/notebook to "examples/sklearn_examples/"
Format should mirror that of examples in other subdirs

Hi,
Wondering if this will be added anytime soon? Thanks

Thanks for asking, @saadjavaid. I'm sorry, but this really hasn't seemed like a priority, since there are already classification examples for SKLearn, as well as regression examples for most of the other libraries. Although I won't be adding formal examples anytime soon, I'm happy to help you if you're having trouble getting something working.


Success / error scenarios to check we've covered most behaviours

Here's a list of things that could happen during a restore operation.  In the interests of thoroughness, I've listed things that should be normal in addition to error conditions.  I think we should look at all of them to ensure we haven't missed anything.
To save typing I've used BDS to mean Backup Data Set and DM for Device's Device Model (as expressed by NcClassManager).
(Side question: Does the BDS include the NcClassManager and if so, what trait does it have?)
(Unlike IS-12, it seems that the primary way of identifying an object is via its role path but this does require use of oids.  Therefore it seems like there should be a mapping process between the BDS set of oids and those in use on the device.  If so, we need to describe that.)
The scenarios: What do we set in NcMethodResultObjectPropertiesSetValidation and/or NcObjectPropertiesSetValidation when...

The BDS Vendor fingerprint is not compatible with the Device?

The BDS includes a property in an NcObject the Device doesn't know about?

The BDS is missing an optional DM property?

The BDS is missing a mandatory DM property?

The BDS has a property that is read-only?

The type of a BDS property differs from the type in the DM?

The traits of a BDS property differs from what the device thinks they should be?

The value in the BDS is out of range?

A property is a struct and the BDS has a field with a name not in the DM?

A property is a struct and the BDS has a field with type that is different to that in the DM?

A property is a struct and the BDS is missing a field that is optional?

A property is a struct and the BDS is missing a field that is mandatory?

The BDS has an NcObject with a role path that isn't listed in any NcBlock in the BDS? (Maybe a result of editing)

The BDS lists a role in an NcBlock but there is no NcObject with that role path in the BDS or on the Device?

The BDS lists a role in an NcBlock but there is no NcObject with that role path in the BDS but there is one on the Device?

The BDS lists a role in an NcBlock but there is no NcObject with that role path on the Device but there is one in the BDS?

The BDS lists more NcWorkers in a block than the Device can support?

The BDS lists an oid that the Device has but the role names differ?

Two NcBlocks list the same oid?

Two NcObjects have the same oid?

Two instances of NcObjectPropertiesHolder have the same role path?

An NcBlock in the BDS lists an oid that is also listed in a NcBlock on the Device that has a different purpose?

Multiple instances of the same manager are present in the BDS?

Other comments:

It would be nice if NcMethodResultObjectPropertiesSetValidation included an overallStatus field

Do attempts to write to read-only properties create an NcPropertyRestoreException?

Should NcPropertyRestoreException have a 'severity' field (e.g. 'info' vs 'error')

NcRestoreValidationStatus is per NcObject - How do we deal with properties that are OK

isReadOnly in NcPropertyValueHolder shouldn't be nullable (unless null can be represented as absent)

The new "notices" concept covers most of these feedback interactions.


dataset.py

Excuse me, dataset Is py for NTU RBG raw data processing? Why should Motion write specific RGB and Depth paths after selecting Motion for video clip processing
报错
笔记

The raw RGB-D images and dynamic images are in different directories on my server, and my train.txt records dynamic images' paths. So when training with RGB&depth raw images, I need to change the data path to find the corresponding action samples, e.g, S001C001P001R001A004 of NTU RGB+D 120

The code of generated dynamic images is not allowed to be exposed for some reason, but I can share with you another simple alternative code for video representation I wrote before. Although this video representation is inferior to rank pooling, I hope it will be helpful to you.
import os
import numpy as np
import cv2
import time
import datetime
import imageio
np.seterr(divide='ignore', invalid='ignore')

def min_max_normalization(image):
    max_val = np.max(image)
    min_val = np.min(image)
    norm_image = (image - min_val) / (max_val - min_val)
    return norm_image

def save_motion_images(augs):
    save_path, image, seg = augs

    save_image_path = os.path.join(save_path, 'mi_%003d.jpg' % (seg))
    mo_img = np.round(image * 255)		# keep this operation
    imageio.imwrite(save_image_path, mo_img.astype(np.uint8), quality=100)  # default: 75
    return 0

def motion_image_seg(augs):
    sub_path, save_sub_dir, segment = augs
    # sub_path = "/home/liuz/ntu_rgb_frames/S001C001P001R001A008/" # for testing
    frames_list = os.listdir(sub_path)
    frames_list.sort()
    num_frames = len(frames_list)

    seg_len = num_frames // segment

    for seg in range(0, segment):

        dynamic_image = np.zeros((311, 384, 3), dtype='float')
        for i in range(0, seg_len):
            frame_path = os.path.join(sub_path, frames_list[i + seg * seg_len])
            frame_image = cv2.imread(frame_path, cv2.IMREAD_COLOR)
            # frame_image = cv2.resize(frame_image, (384, 311))
            dynamic_image = dynamic_image + frame_image

        dy_image = dynamic_image // seg_len
        dy_image = min_max_normalization(dy_image)
        save_motion_images((save_sub_dir, dy_image, seg + 1))

rgb_frames_path = "/home/xxx/datapathname/"     # source
save_dir = "/home/xxx/dataset_motion_img/"    # destination

dir_list = os.listdir(rgb_frames_path)
dir_list.sort()

if __name__ == '__main__':

    segment = 1

    print('Starting to motion image construction...')
    proc_start_time = time.time()
    for i in range(0, len(dir_list)):
        sub_path = os.path.join(rgb_frames_path, dir_list[i])

        sub_dir_f = os.path.join(save_dir, dir_list[i])
        if not os.path.exists(sub_dir_f):

            os.makedirs(sub_dir_f)
        motion_image_seg((sub_path, sub_dir_f, segment))

        cnt_time = time.time() - proc_start_time
        nowTime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        print('video {} / {} done,  spend time: {} sec,  avg time: {} sec/video,'.format(i + 1, len(dir_list), cnt_time,
                                                                                float(cnt_time) / (
                                                                                i + 1)) + ' now time: ' + nowTime)
    print('========================')
    print('Done all videos: {}, segment: {}'.format(i + 1, segment))


Configure App Environments

There's not much to add here until these issues (#22 and #23 are done)
Context
In order to deploy our application and work with containerization tools like Docker, we'll need some extra configuration to use certain Environment variables. The easiest option for making this work is via a secret .env file that will not be checked into git version history.
Definition of Done

environment configuration is complete for front and back end with the app now pulling key variables like "Environment", "Database URL", etc. from an "ENV" file (.env)
env configuration has appropriate defaults for working with the app in development
.gitignore is updated to not check the env file into git history
README is updated with details on how this config is setup and what variables may be useful (if any at this point) to add to your local .env

Engineering Details

see this doc for details on completing the setup in FastAPI.

Also, take a look at how the configuration is done in this template app

it appears NEXT has native support for .env files... so yay!

Done

Done


Remove metadata from session

What is the context of this PR?
Removes metadata from the session in order to reduce the amount of data stored as session data. This will help to reduce the overall datastore payload size, and will also prevent duplication as metadata is stored and available in our questionnaire state.
The following attributes will still be kept in our session data as they are updated as part of the session: language_code, confirmation_email_count, feedback_count,.
All other existing session_data values will be set to None to allow runner to be temporarily backwards compatible, and are to be removed in the future.
How to review
Check nothing is broken by this change.
Have any items been missed.
Checklist

[ ] New static content marked up for translation
[ ] Newly defined schema content included in eq-translations repo

 Remove metadata from session (5 SP)


bad_dtls_test.c: fix mem leaks - Coverity CID 1474468

SSIA

This was fixed by #14716 which also added some additional TEST_() checks for the error handling.


[docs] Proof the Styles section

(with apologies to our translators for the overhead :sweat_smile:)

[x] I have followed (at least) the PR section of the contributing guide.

Also, "stress test"?

Also, "stress test"?

?

Also, "stress test"?

?

It's referring to https://next--material-ui.netlify.com/css-in-js/basics/#stress-test.
I agree that this is a misnomer currently. That section isn't testing the solution under heavy load nor is it asserting when the solution should fail.
It is showcasing the combination of theme and props based styles. Maybe "Combine theme and props based styles"?
That section could also be improved from a usability standpoint. The dialog is blocking everything (including tab switch) until I pressed the tiny "X" to close it.

That section isn't testing the solution under heavy load nor is it asserting when the solution should fail.

To defend my church: This is a demo that people can use to "stress test" the solution. The values can update at 60 FPS. I'm happy to remove/revisit it.

The values can update at 60 FPS.

How? I for one can't change the values with the current solution 60 times per second with my mouse.

@eps1lon Like this:


Can't compile with gcc 4.*

Code from ImathConfig.h(generated from ImathConfig.h.in) fails to compile with gcc 4.(tried 4.8 and 4.9). on 5. it compiles fine:
#define IMATH_ENABLE_API_VISIBILITY

// MSVC does not do the same visibility attributes, and when we are
// compiling a static library we won't be in DLL mode, but just don't
// define these and the export headers will work out
#if ! defined(_MSC_VER) && defined(IMATH_ENABLE_API_VISIBILITY)
#  define IMATH_PUBLIC_SYMBOL_ATTRIBUTE __attribute__ ((__visibility__ ("default")))
#  define IMATH_PRIVATE_SYMBOL_ATTRIBUTE __attribute__ ((__visibility__ ("hidden")))
// clang differs from gcc and has type visibility which is needed for enums and templates
#  if __has_attribute(__type_visibility__)
#    define IMATH_PUBLIC_TYPE_VISIBILITY_ATTRIBUTE __attribute__ ((__type_visibility__ ("default")))
#  endif
#endif

Is it possible to support gcc 4.*?

We dropped support for gcc 4 with the 3.0 release.  We are primarily driven by the VFX reference platform, which moved beyond gcc 4 a number of years ago.
Specifically what error are you getting? You may be able to work around it by disabling some of the optional features or controls.

Did we really? Are we C++14 minimum now? I wouldn't object to that, but I'm not sure I had understood that.
Is this just a case of the __has_attribute not being defined for that version?

The issue is with gcc 4, not with C++11.  I ran into problems trying to compile the recent Imath changes with gcc 4.8.5, where legal C++11 code gives errors.

Currently, Imath 3.0's INSTALL.md file is clear that we expect any compiler supporting C++11 to be fine.
What were the errors? I only last month changed OIIO master to C++14/gcc6 minimum, and am scheduled to do so for OSL this coming week. I'm pretty confident that imath is not doing anything fancy enough that it shouldn't work fine with gcc 4.8 with at most VERY minor workarounds.

It looks like preprocessor parsing error, not related to c++ standard
OpenEXRConfig.h:100:22: error: missing binary operator before token "("
 #  if __has_attribute(__type_visibility__)
                              ^
gmake[2]: *** [src/lib/Iex/CMakeFiles/Iex.dir/IexBaseExc.cpp.o] Error 1
gmake[1]: *** [src/lib/Iex/CMakeFiles/Iex.dir/all] Error 2
gmake: *** [all] Error 2

Hello, any news about that since 2 and half month ? I have fallen in the same issue, and I would be interested in compiling it in gcc 4.8.5 too (and alembic 1.8.2).

Can you try applying this patch by hand and see if it fixes? https://github.com/AcademySoftwareFoundation/Imath/pull/199 (still under review)

Thank you for the answer. Unfortunately that still does not work under gcc 4.8.5. I would suggest :
# if !defined(__GNUC__) || __GNUC__ >= 5
# if __has_attribute(__type_visibility__)
#    define IMATH_PUBLIC_TYPE_VISIBILITY_ATTRIBUTE __attribute__ ((__type_visibility__ ("default")))
#  endif
# endif

That would change the behavior to ignore this part for gcc < 5, and would not modify the behavior for clang or others.

I'm nervous about your first proposal, since icc and clang define __GNUC__, but not sure what value it has.
I like the second solution, though, and I will use that and update the PR.

I updated the Imath PR #199 , can you try it out?

All good for me, thank you !


Date is off by one day

Something in the DateHelper class is not returning the correct date.
After editing the batch date, it always displays as one day earlier than what was set.

Good catch, you want to take this on @tec7z7 ?

Yeah, I'll take a look at it.


Workflow for the news (do you really want to publish?)

Acceptance criteria

Editor has to confirm that he/she wants to publish a news article or publish updates to an existing one

@wenche @fernandolucchesi @padms
I have pushed a custom document action that replaces the built in Publish action: https://github.com/equinor/energyvision/commit/54b30fa79e360851605c686d92818113f37fddd9
This action will pop up a confirmation dialog for the news and localNews document types, for the rest it should act like before (with the exception that the label on the button isn't updated by state as the built in one).
The code and comments are I think pretty self explanatory, but a few comments:
We need to use the publish action from the useDocumentOperation hook to manually publish the document. This command does not care about drafts or validation, so we need to manually check the validation status before running that. To that end I made a function that checks the validation markers passed by the useValidationStatus hook for any errors.
Similarly we check for the existence of a draft document (see: https://www.sanity.io/docs/document-actions-api#draft-f1269efe5ab2) which is used to conditionally disable/enable the publish button along with the validation check.
What is important to check here is whether the validation stuff works; users should not be able to publish a document if there are validation errors, nor should they be able to publish if there are no changes made to the document (no draft). We also need to check that the confirm dialog works for news and local news, and does not appear for other documents.

LGTM :thumbsup:

@meols Please take a look at what Sven did. I have added some suggestions for testing in the issue description

@SvSven - works perfectly :thumbsup:
@wenche - thanks for great test cases :smile:


Remove java notebooks

This code has been moved to a different repository.

I'll probably also need to update the tests to check those repos out before running the tests, eh?


#521 Support a collections of roles in the MongoDB AuthorizationProvider

Motivation:
This is a first proposal.
I have modified the MongoAuthorizationTest to create and initialize a roles collection with a superadmin role having a sudo permission.
I have added a new testAuthoriseWithRolePermission method to MongoAuthorizationTest showing that the "read roles" behavior is disabled by default: tim does not have the sudo permission event if he has the superadmin role.
The new MongoAuthorizationWithRolesTest test extends MongoAuthorizationTest, enables the "read roles" behavior, overrides the testAuthoriseWithRolePermission and modifies it expectations:  tim now has the sudo permission.

I forgot to say that I modified some javadoc comments, hoping that they will be clearer. I can revert to the original comments if you don't like the new ones.

Wait... the PR is containing a line that should not be here:
user.authorizations().clear(providerId);

That is linked with the issue described here:
https://groups.google.com/g/vertx/c/vTCLEQsv1N0
I will remove it for the moment.

This PR isn't applicable to this repo anymore. It should go on #vert-x3/vertx-auth


Add Mentor birthday

We need a way for mentors to enter their birthday. This is going to be used to check to see if they require a background check (under 18 do not), for us to have demographic data to use when promoting the company/business, and much more.
Requirements

[ ] Needs to be mandatory.
[ ] Needs to add a default birthday for all mentors that are already in our system (db).
[ ] Require anyone who has a default birthday to update it with a real birthday.
[ ] Visually update the frontend to ask for this information. Look at the guardian flow for how we did that already.

Related to #553 and #386

Do we need to do anything with the existing "background_check" field based on the value of the mentor birthday field?  Like if they are under 18 at time of signup/record creation, should we just set background_check to true?
Also, what happens to an under 18 mentor once they turn 18?  Do they then need a background check?  Or are they grandfathered in?

Do we need to do anything with the existing "background_check" field based on the value of the mentor birthday field? Like if they are under 18 at time of signup/record creation, should we just set background_check to true?

The background issue is #553. But to answer your questions. I think the easiest way to update this code is to set the background_check field to True if they are under 18.

Also, what happens to an under 18 mentor once they turn 18? Do they then need a background check? Or are they grandfathered in?

They are grandfathered in for now. Version 2 of our website will have a new background check service.

OK, so will this workflow work?
New mentor : check mandatory birthday field against current date.  If under 18, set background_check to true.  If over 18, set to false.
Existing mentor : set birthday field to some arbitrary date and make them change it on next log in.  When field is updated, check current age against current date, if under 18, set background_check to true.  If over 18, set background_check to false.
Or for the existing mentor, should we use create date when checking the age so that mentors who signed up when they were younger than 18 are grandfathered in?   Or is this an edge case we don't even need to worry about?

New mentor : check mandatory birthday field against current date. If under 18, set background_check to true. If over 18, set to false.

Yup. Copy over get_age() that's under the Student model. Can use a_mentor.get_age() in the view.

Existing mentor : set birthday field to some arbitrary date and make them change it on next log in.

Yes

When field is updated, check current age against current date, if under 18, set background_check to true. If over 18, set background_check to false.

Almost.

If the background_check is already true, don't touch it. We don't want to flip them to false if they are over 18 and have already done a background check.
If the background_check is already false and the now we know the mentor is under 18, set it to true.

Or for the existing mentor, should we use create date when checking the age so that mentors who signed up when they were younger than 18 are grandfathered in? Or is this an edge case we don't even need to worry about?

Don't worry about this.

@davidkam any updates?


Fix network status name/namespace to compliant with multi-net-spec

This fix changes network status name to net-attach-def name with
namespace, to compliant with k8s npwg's multi-net-spec.

Pull Request Test Coverage Report for Build 129341943

3 of 3 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 71.525%

Totals

Change from base Build 128626313:
0.02%

Covered Lines:
1065

Relevant Lines:
1489

💛  - Coveralls


Give sbt more memory?

https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/4720

SUCCESS 193 FAILED?! zinc,mima
BLOCKERS: 1 zinc

the same failures occur in the main build, so you should re-run this after I get that sorted out

optimistically merging


Add a way to reset the LDK router

Sometimes the LDK router gets stuck and we want to reset it as a last resort.
Consume https://github.com/getAlby/ldk-node/pull/10 and put a button to execute the function on a very simple settings page (don't worry about styling or linking to the page from anywhere for now)

I think we have that by now, don't we? Feel free to close it @rolznz


move to expressjs org

Why is this under strongloop? I disagree with with move, we have an org to manage the express stuff unless someone does not trust the org?

+1

As announced by @jasnell in today's Express TC meeting, this repo will be transferred to the expressjs org and and the expressjs.com domain name will be transferred to the foundation.   It may take some time, but the process is now underway and will happen. :-)

Repo finally got moved, so can close this.  Thanks @jasnell.


Fix gpg key checking warings after f588f26 in puppetlabs-apt

Use the full fingerprint for the apt key to silence the warning.

Failing tests should pass when PR #120 is merged

I have tried this fix and it breaks npm installation. When installing npm modules it throws: "Error: Command npm is missing".
Reverting to the short key installed npm and the modules, but gave the warning again obviously.

@xtreamwayz
Verified as failing on puppetlabs/ubuntu-14.04-64-nocm 10.00 with puppetlabs-apt (v1.8.0) using Puppet 3.7.4. The problem appears to be related to dependency ordering and not specifically this PR though. (You're seeing this error because the native nodejs package got installed instead, which doesn't include npm)

@xtreamwayz The class containment issue should hopefully be fixed in #121

Thanks @paroga !


Rename FullscreenSettingsScreen tests

Codecov Report

Merging #733 (272f8d5) into main (bfbe498) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #733   +/-   ##
=======================================
  Coverage   69.83%   69.83%
=======================================
  Files         113      113
  Lines        5562     5562
  Branches     1110     1110
=======================================
  Hits         3884     3884
  Misses        917      917
  Partials      761      761

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Autocomplete in StringEditor

Hi @volkanceylan ,
Is it possible to use Autocomplete in StringEditor? Did you have any example?

No, i don't have any example.


Follow 3xx redirects

What happened?
An exception was thrown for a 301 response with a valid location header:
Caused by: feign.RetryableException: Error 301. (Failed to parse response body as SerializableError.) Body:

<title>301 Moved Permanently</title>

301 Moved Permanently

nginx

 executing POST https://<URL>
	at feign.FeignException.errorExecuting(FeignException.java:67)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:102)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)

What did you want to happen?
Follow the redirect instead of throwing

http-remoting supports 308 for permanent redirect, but not 301. https://github.com/palantir/http-remoting#quality-of-service-retry-failover-throttling

OK luckily we control the server so we can change to 308. Is there an intentional reason that 301s are errors? It's a pretty common response code

http-remoting does intentionally not support all of HTTP. Codes with explicitly defined behavior are 308, 429, 503, see README.


Add test for VPP images

Description
Add post release checks for VPP.
Related issues/PRs

Todos

[ ] figure out where to get VPP release number from
[x] test images
[ ] test tags and tars

Release Note

None required

@caseydavenport Am I on the right lines with this PR?

Tried testing this by checking out the v3.22.1 tag and bringing these changes over to there.
I get this failure:
FAIL: test_images.test_vpp_branch
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lance/.local/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/lance/go/src/github.com/projectcalico/calico/calico/release-scripts/tests/test_images.py", line 46, in test_vpp_branch
    assert VPP_RELEASE != "master", "vppbranch cannot be 'master' for a release"
AssertionError: vppbranch cannot be 'master' for a release

Which I think is actually correct?


Roadrunner export option not available on demo version !

Hello all of you
I would like to make my own map , so I follow  the instructions  from the Documentation.
https://carla.readthedocs.io/en/latest/how_to_make_a_new_map/
I downloaded and installed the Roadrunner on Ubuntu , but  I  was not able to export any files from  the program, no odr ,  no fbx file .
I got from the program that I have to purchase Roadrunner for this option   .....

Is there any trick for  trying exporting files   ?  Or did I miss something ?
Thank you in advance .....

Demo version does not come with the exporting option...
You have to buy a license.


Update list-onedrive-urls.md

Incorporate fix #1727.

@nenonix @JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@kaarins PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Improve Runner report with links and less jumping

I have checked the following:

[X] I've searched existing issues and found nothing related to my issue.

Describe the feature you want to add
When I see the Runner report, and a request/test/etc fails, it would be useful to click in the title of that test and be redirected directly to it.
One of the main problems right now, it that every time I make any change, like open a folder, a request (essentially any click) the runner report scrolls all the way down. If that stopped happening (second suggestion) it would make it much easier to go over all the failing ones.
Some filtering (only show failing, passing, etc) would also be great.
Thanks!
Mockups or Images of the feature

Hey @robs-nice99
Thanks for the feedback! I agree, the runner scrolling down automatically can be quite annoying. I also think the idea of adding filters is a good one. We'll take a look into this once we've wrapped up addressing some key bugs and feature requests.


Failing tests on macos, py 3.9, numpy 1.21.5

See https://github.com/NeuralEnsemble/python-neo/actions/runs/1962231833

Seems to be fixed again in CI.


Added CSS3 Pie polyfill

Thanks! But we do not recommend a polyfill for box-shadow deliberately. It does not impact the readability or the design significantly enough to require a polyfill.

Beat me to it :) Also, just for reference: https://github.com/h5bp/html5please/issues/29

Is this because CSS3Pie is 40kb? For the sake of argument, what if someone's design relied heavily on box-shadows? Shouldn't HTML5Please at least notify them of the option of a polyfill? Maybe even with a warning "...but CSS3Pie is huge, so use it at your own risk"?

Any polyfill that occurs in HTML5 Please are those that we think are worth gaining benefits from. Not those that we think are 'use at your own risk'. Just generic polyfills for any of these features can be found on modernizr polyfill wiki

Gotcha.
So if the file size was smaller CSS3Pie would be supported? What if I were to break up CSS3 Pie's different features into their own little packages? Filesize for each polyfill would be around 5-10kb, and then people wouldn't have to import 40-friggin-kb to have box-shadow in IE7-.


coo_matrix file

Hi @acycliq . Thanks for your great work on pciSeq! I'd like to know how to get the labeled image as a coo_matrix file? Which software is usually used?

Hi there,
Thanks for your interest. Assuming that the label image is the 2D array returned by your segmentation you can do:
from scipy.sparse import coo_matrix

coo = coo_matrix(label_image)

There are might be cases (not very often) where your segmentation spits out a label image where the labels arent a sequence of consecutive integers (ie without any gaps). Or they are alphanumeric strings
Hence, I would say, first relabel your label image (see function below) and then convert it to a sparse matrix
def reorder_labels(label_image):
    """
    rearranges the labels so that they are a sequence of integers
    """
    _, idx = np.unique(label_image.flatten(), return_inverse=True)
    out = idx.reshape(label_image.shape)
    return out

To recap, the full operation would be something like
from scipy.sparse import coo_matrix

label_img =  reorder_labels(label_image)
coo = coo_matrix(label_img)

Then use that coo as your input to pciSeq. To be fair, I need to include that reorder_labels function into my code so that it is getting called if needed without the end-user having to do it on his own, beforehand.
Let me know if you have any problems please

Hi there,
Thanks for your interest. Assuming that the label image is the 2D array returned by your segmentation you can do:
from scipy.sparse import coo_matrix

coo = coo_matrix(label_image)

There are might be cases (not very often) where your segmentation spits out a label image where the labels arent a sequence of consecutive integers (ie without any gaps). Or they are alphanumeric strings Hence, I would say, first relabel your label image (see function below) and then convert it to a sparse matrix
def reorder_labels(label_image):
    """
    rearranges the labels so that they are a sequence of integers
    """
    _, idx = np.unique(label_image.flatten(), return_inverse=True)
    out = idx.reshape(label_image.shape)
    return out

To recap, the full operation would be something like
from scipy.sparse import coo_matrix

label_img =  reorder_labels(label_image)
coo = coo_matrix(label_img)

Then use that coo as your input to pciSeq. To be fair, I need to include that reorder_labels function into my code so that it is getting called if needed without the end-user having to do it on his own, beforehand.
Let me know if you have any problems please

Thanks for your quick reply! I'll try it out today and go back to you later. In addition, why should we use scRNA-seq marker expression matrix as one of the pciSeq inputs? Finally, can I visualize my result in URL like https://acycliq.github.io/ca1/ ?

you need single cell data because this is where the cell types come from. The cell calling algorithm assigns spots to cells while at the same time, tries to find which cell type matches the best possible way the pattern that emerges from these assignments
Yes, you can have the viewer work with your own data, let me know when you get there....
Just remember to run pciSeq with the save_data option set to True, something like:
opts={ 'save_data': True}
res = pciSeq.fit(spots_df, label_image, scRNA_df, opts)

That should save the output at tsv files to your temp dir. The files are needed for the viewer

I've got all the output files, including cellBoundaries.tsv, cellData.tsv and geneData.tsv. How can I proceed to visualize these results in URL?

Hi there
Had to do a few changes...
Can you follow the readme of this repo https://github.com/acycliq/visage  and tell me if it works for you?

Thats weird, Since you are fetching the original demo data i would say it has to do with your config.js file. Have you saved your data under ROOT\viewer\data, just replace the three demo tsvs with yours.
Let me know how it goes, if your project is on some public repo, I can have a look too

ok, I am assuming then you are trying to open in your browser E:\Downloads\visage-main\visage-main\index.html. In other words you cd into your E:\Downloads\visage-main\visage-main and then you create your local server.
Can you try using these in your config.js please
cellData: [{mediaLink: './data/cellData.tsv', size: "21267078"}],
geneData: [{mediaLink: './data/geneData.tsv' size: "74721272"}],
cellBoundaries: [{mediaLink: './data/cellBoundaries.tsv', size: "10394177"}],

Thanks for your patience! And I'd like to fix this problem today. I've edited my config.js file. However, the problem still exists. In addition, I find that no matter where I cd to, I can always create a local server that is related to your pciSeq data. Why?

Can you make a public repo so I can have a look?

Can you share the javascript code by any chance, the viewer code that loads up the demo data instead of yours?

I didn't use javascript. I visualized my data following your instruction here,
https://github.com/acycliq/visage
which means I used totally the same scripts as in this repo, except for the configuration files. As for some configuration parameters, such as roi, for now it's ok for you to use the default value.
The size of my tsv files are as follows,
1.cellBoundaries.tsv: 4,823,294 bytes
2.cellData.tsv: 5,059,483 bytes
3.geneData.tsv: 90,705,878 bytes
Two questions:

Why should we declare the size of tsv files in config.js?
Is it possible to do cell-type annotation without single-cell RNA data? In my opinion, we can speculate on cell type easily by calculating the proportion of each marker gene in each cell.

You need to clone/download the viewer code (ie the visage repo), do the edits necessary (it is three files you need to edit, config.js and two more for the colour scheme) and then open in the browser your local index.html.
No, there is no code that works without single cell data, maybe will come in the future.

For now, I can see my data has been successfully imported into the web page. But I'm stuck in the below snapshot for 10 minutes. Any recommendation?

Most likely you missed to set the colours for you genes or cell classes. It's the glyphConfig.js you have to edit. Every single gene in your gene panel should be described there. If you miss one, the viewer will not load.
The same applies for the cell classes. Edit your classColours.js to set the colours for all your classes.

I've assigned each gene and each cell type a unique color, but the problem remains. All the three configuration files I used are uploaded to my repo. Could you help me check it out?

Yes x1 and y1 are the width and height of your tiff in pixels
Can you send your browser log please? Typically in your Chrome in the dev-options of the browser (to go there press F12, or Ctrl+Shift+I or under the three vertical dots at the top right -> more tools -> developer tools)
Then select "Console" from the menu bar at the very top and send me a screen shot of that

Yes probably one of your classes is missing from your classColors.js.  Add this line at the bottom
{className: 'Zero', IdentifiedType: 'Zero', color: '#000000'},
The Zero is the zero expressing class, if something doesnt fit the point pattern observed in your single cell data, it will be classified as Zero class. The Zero class doesnt exist in the single cell data, but if pciSeq cannot find a good match then it will be assigned to this "extra" class, the Zero class

Great! Yes get back to me if you have any questions

Hi @acycliq , I'd like to create a gene-by-cell expression matrix based on pciSeq result. But I don't know how to assign each spot to each cell. Any recommendation?

You can use the data from cellData.tsv, in particular the columns Genenames and CellGeneCount. The former (ie  Genenames ) describes the genes that have been assigned to the cell and the latter (ie CellGeneCount) the corresponding gene counts.
Is this what you need?
If you are wondering how come we have floats and not integers for the gene counts the answer is because this is a "soft clustering" method. It assigns the spot to a cell with some probability. For example lets say you had one spot and only two cells. Assume also, there was 50% prob for the spot to get assigned to the first and another 50% to the second cell. Then you would register 0.5 counts on each of the two cells. Under a hard clustering scheme where a spot is assigned to only cell, the you would register 1 count to one of the cells and 0 counts to the other.

That's what I want. How can I transform it into a matrix which has genes as rows and cells as columns? In addition, can I perform standard scRNA-seq data preprocessing, such as normalization and scaling, in this matrix? What's more, how can I export this .html file so that I can share it with other people without making a local server?

Can you get the latest version,  v:0.0.44. Then if you call the fit method with save_data set to True, ie like
pciSeq.fit(_iss_spots, _coo, _scRNAseq, {'save_data': True})
it should also save a pickle file along the three tsv flatfiles. You can read the pickle by:
import pickle

infile = open(r'path\to\pciSeq.pickle', 'rb')
pciSeq_data = pickle.load(infile)
infile.close()

To get the array of the number of counts per cell you can do
pciSeq_data.cells.geneCount
That will return an array of shape Num_of_cells+1-by-genes. The gene labels are sorted alphabetically but you can get them from pciSeq_data.genes.gene_panel
Also the top row of pciSeq_data.cells.geneCount corresponds to the background and it is filled with zeros, ie pciSeq_data.cells.geneCount[0, :].sum()  = 0  The background counts are kept elsewhere, here: pciSeq_data.cells.background_counts
Since you do not care about the background, to get the array you want do pciSeq_data.cells.geneCount[1:, :].
Regarding your other questions, I do not know if you can use it to perform standard scRNA-seq data preprocessing.
For hosting the website somewhere else and not on your personal pc, If I have time I will write some notes duriing the weekend

Can you send me your single cell data?

I've sent it to<EMAIL_ADDRESS>

Thanks and sorry! Can you try with the latest build please, v 0.0.45. Let me know if you have any problems

Solved it, thank you! However, I find out the cell typing results are partially different before and after the release of pciSeq(version=0.0.45). For example, a large number of cells previously annotated as TypeA are annotated as TypeB now. Are there any changes in the cell typing process of pciSeq(version=0.0.45)?

Yes results will not be exactly the same but they should be quite similar where differences exist. Hopefully, in these cases, when you have differences results with the latest code will make more sense in general (or at least they wont be worse generally speaking)
From: Jiayu @.>
Sent: 23 June 2022 10:13
To: acycliq/pciSeq @.>
Cc: Dimitris Nicoloutsopoulos @.>; Mention @.>
Subject: Re: [acycliq/pciSeq] coo_matrix file (Issue #4)
Solved it, thank you! However, I find out the cell typing results are partially different before and after the release of pciSeq(version=0.0.45). For example, a large number of cells previously annotated as TypeA are annotated as TypeB now. Are there any changes in the cell typing process of pciSeq(version=0.0.45)?
—
Reply to this email directly, view it on GitHub https://github.com/acycliq/pciSeq/issues/4#issuecomment-1164157538 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADBLTTSJRP2KZLTE2ZBCPETVQQTBXANCNFSM5XAIUR4Q .
You are receiving this because you were mentioned.  https://github.com/notifications/beacon/ADBLTTW6H65OZ7BG3F6RHW3VQQTBXA5CNFSM5XAIUR42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIVR2EYQ.gif Message ID: @.*** @.***> >


add-umar-faruq

Issue Addressed

Proposed Changes and Benefits

Screenshots

Notes to Reviewers

Checklist

[x] I have thoroughly read and understand The Contribution Guidelines.
[x] The title of this PR follows the keyword: brief description of change format, using one of the following keywords:

Feature - adds new or amends existing user-facing behavior.
Chore - changes that have no user-facing value, refactors, dependency bumps, etc.
Fix - bug fixes.
Docs - fixing typos and adding new content.

[ ] I have listed my proposed changes and their benefits.
[ ] I have linked the corresponding issue in the Issues section.

@Luwa-Tech  Your pull request has been successfully merged!!! Congratulations .
If you haven't already, please go ahead and star this repo and you are more than welcome to continually contribute to this project by assisting with our open issues, or adding your own new open issues, and additionally you can get an idea of ideas we have to continually grow this project further as technology and tech stack continually evolve by looking at the ROADMAP for the project. Have a great day, and thank you for taking the time to contribute,
Yusuf! You can view your contribution now on the live link. Have a great rest of your day, and I will be sending you an invite to be a member of the Before I Die Code now.


try working on tests to abort

Scope & Purpose
the cluster recovery tests sometimes block. These are starting efforts to find a way to circumvent this.

[ ] :hankey: Bugfix (requires CHANGELOG entry)
[ ] :pizza: New feature (requires CHANGELOG entry, feature documentation and release notes)
[ ] :fire: Performance improvement
[ ] :hammer: Refactoring/simplification
[ ] :book: CHANGELOG entry made

Backports:

[ ] No backports required
[ ] Backports required for: (Please specify versions and link PRs)

Related Information
(Please reference tickets / specification / other PRs etc)

[ ] Docs PR:
[ ] Enterprise PR:
[ ] GitHub issue / Jira ticket number:
[ ] Design document:

Testing & Verification
(Please pick either of the following options)

[ ] This change is a trivial rework / code cleanup without any test coverage.
[ ] The behavior in this PR was manually tested
[ ] This change is already covered by existing tests, such as (please describe tests).
[ ] This PR adds tests that were used to verify all changes:

[ ] Added new C++ Unit tests
[ ] Added new integration tests (e.g. in shell_server / shell_server_aql)
[ ] Added new resilience tests (only if the feature is impacted by failovers)

[ ] There are tests in an external testing repository:
[ ] I ensured this code runs with ASan / TSan or other static verification tools

Link to Jenkins PR run:
Documentation

All new features should be accompanied by corresponding documentation.
Bugs and features should furthermore be documented in the CHANGELOG so that
support, end users, and other developers have a concise overview.

[ ] Added entry to Release Notes
[ ] Added a new section in the Manual
[ ] Added a new section in the HTTP API
[ ] Added Swagger examples for the HTTP API
[ ] Updated license information in LICENSES-OTHER-COMPONENTS.md for 3rd party libraries

External contributors / CLA Note
Please note that for legal reasons we require you to sign the Contributor Agreement
before we can accept your pull requests.

Work on this using:
./scripts/unittest  recovery_cluster  --cluster true --test arangosearch   2>&1 |less


Share task details with other apps. #172

(Continuation of https://github.com/dmfs/opentasks/pull/282)
TODOs

[x] add German transaltions of new text: https://github.com/dmfs/opentasks/issues/172#issuecomment-312724705
[x] use template engine for creating the shared text
https://github.com/dmfs/opentasks/pull/355#issuecomment-312869908
https://codeka.github.io/carrot/
-> create separate task https://github.com/dmfs/opentasks/issues/372

The carrot library has Java8 codebase (lambda, Java8 JDK classes and methods), so we can't use that.
This is blocked until we decide what solution to use.

Ok, please open another story for the template engine. I'd be willing to contribute a patch to carrot to make it Android compliant (see codeka/carrot#1). After all it's probably not that much work.
For the time being, please rebase on master and squash.

Story created for the template engine: https://github.com/dmfs/opentasks/issues/372
Squashed and rebased, ready for review.

Moved this back to "In Progress" reassigned to myself to attend to it later.

Ready for check again.

Squashed and rebased onto master.

btw, please also rebase

Rebased to master, ready for review again.

Please note the extra commit: I've renamed CachingCharSequence to LazyCharSequence and corrected the template comment in SharedTaskText.

@lemonboston please rebase

Squashed and rebased, and created ticket to replace some classes: https://github.com/dmfs/opentasks/issues/436


Missing info

[type in CLI field list not defined]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e0091902-7cb0-8dc3-2a7a-dc644f39a961
Version Independent ID: d2757c88-1bad-8d27-b548-9e04a8ec7fe1
Content: Understand Azure role definitions - Azure RBAC
Content Source: articles/role-based-access-control/role-definitions.md
Service: role-based-access-control
GitHub Login: @rolyon
Microsoft Alias: rolyon

@CodingMaximus
Thanks for your feedback! We will investigate and update as appropriate.

Hello @CodingMaximus
It would be great if you can please elaborate your query for better understanding. This would help us redirect the issue to the appropriate team. Thanks!

@CodingMaximus
Thank you for bringing this to our attention.
I've delegated this to content author @rolyon, who will review it and offer their insightful opinions.

Hi @CodingMaximus
Thanks for your feedback. We made some updates to this doc.
https://learn.microsoft.com/en-us/azure/role-based-access-control/role-definitions
thanks

#please-close


Add RBI for GlobalID

Type of Change

[x] Add RBI for a new gem
[ ] Modify RBI for an existing gem
[ ] Other:

Changes

Gem name: GlobalID
Gem source: https://github.com/rails/globalid

Thanks Bart. I'm merging this with a CI failure because it's unrelated to the annotations and requires a new tapioca version.
An alternative is fixing the Sorbet version but I like that we use the latest one. It allows us to know about any compatibility issues early.


wrong endpoints in simulator packages

urls should be in the format of $packageauthor/$packagename/$morevars...

probably going to need a change in the admin dashboard as well for app inclusion.

bd432fb880d3ad086991543483ebf777d0cdff40


Backport: cluster state saved/fetched incorrectly in clusters with prefix path

Backport https://github.com/rancher/rancher/issues/17965

Can be tested in rc8

Tested with the steps mentioned in Issue #17965 using Rancher OS 1.5.1 nodes
Create ros machine
Run rke v0.1.17 up on it
Break the kubeconfig by changing the client-certificate-data or client-key-data in the kubeconfig file
rke up fails as below:
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [state] Possible legacy cluster detected, trying to upgrade
INFO[0000] [reconcile] Rebuilding and updating local kube config
INFO[0000] [state] Fetching cluster state from Kubernetes
WARN[0000] Failed to fetch state from kubernetes: Failed to create Kubernetes Client: Error loading config file "./kube_config_cluster.yml": v1.Config: AuthInfos: []v1.NamedAuthInfo: v1.NamedAuthInfo: AuthInfo: v1.AuthInfo: ClientKeyData: ClientCertificateData: decode base64: illegal base64 data at input byte 1428, error found in #10 byte of ...|tLQo=ABCD","client-k|..., bigger context ...|veTJDZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=ABCD","client-key-data":"LS0tLS1CRUdJTiBSU0EgUFJJVkFURS|...
There is no attempt to get the correct state from the nodes.
4. Download latest rke 0.2.2.-rc1
5. Ensure cluster-state.yml file exists on the nodes in /opt/rke/etc/kubernetes/.tmp
If it doesn't exist, then create it with content of the same cluster.yml that you have
7. remove the broken rkestate file in your local directory
8. run rke up
Cluster comes up correctly.