2 single node microk8s on different servers suddenly do not create pods after working perfectly for months...

Summary

2 single node microk8s on different servers suddenly do not create pods after working perfectly for months! guess happened after rebooting of server:
looking at the "describe pods" & "pod event" and "jurnalctl" and "microk8s inspect" the major error I encountered is:
Warning  FailedCreatePodSandBox  7m46s (x3 over 7m59s)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown              desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown
What Should Happen Instead?

Pod should be running in kube-system namespace like calico pods for CNI and my application pods in other namespaces.
[root@localhost containerd]# kubectl get all --namespace kube-system
NAME                                          READY   STATUS    RESTARTS   AGE
pod/calico-node-z2npr                         0/1     Unknown   13         308d
pod/hostpath-provisioner-5c65fbdb4f-t8hz5     0/1     Unknown   13         308d
pod/coredns-7f9c69c78c-kcsxz                  0/1     Unknown   11         307d
pod/calico-kube-controllers-c85f8f74b-stth6   0/1     Unknown   13         308d
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   <IP_ADDRESS>           53/UDP,53/TCP,9153/TCP   321d
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/calico-node   1         1         0       1            0           kubernetes.io/os=linux   321d
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hostpath-provisioner      0/1     1            0           321d
deployment.apps/calico-kube-controllers   0/1     1            0           321d
deployment.apps/coredns                   0/1     1            0           321d
NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-f7868dd95    0         0         0       321d
replicaset.apps/calico-kube-controllers-59b46f8b57   0         0         0       321d
replicaset.apps/hostpath-provisioner-5c65fbdb4f      1         1         0       321d
replicaset.apps/calico-kube-controllers-c85f8f74b    1         1         0       312d
replicaset.apps/coredns-7f9c69c78c                   1         1         0       321d
Reproduction Steps

1.installed microk8s offline through this page: https://microk8s.io/docs/install-offline
2.# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
3.# uname -sr
Linux 3.10.0-1160.49.1.el7.x86_64
5. # microk8s version
MicroK8s v1.25.3 revision 4094
6.# microk8s kubectl version --short
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.25.3
7.# docker version
Client:
Version:           18.09.0
API version:       1.39
Go version:        go1.10.4
Git commit:        4d60db4
Built:             Wed Nov  7 00:48:22 2018
OS/Arch:           linux/amd64
Experimental:      false
Server: Docker Engine - Community
Engine:
Version:          18.09.0
API version:      1.39 (minimum version 1.12)
Go version:       go1.10.4
Git commit:       4d60db4
Built:            Wed Nov  7 00:19:08 2018
OS/Arch:          linux/amd64
Experimental:     false
8.# containerd
WARN[0000] containerd config version 1 has been deprecated and will be removed in containerd v2.0, please switch to version 2, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header
INFO[2022-11-07T16:20:26.025791559+03:30] starting containerd                           revision=1c90a442489720eec95342e1789ee8a5e1b9536f version=1.6.9
INFO[2022-11-07T16:20:26.050331270+03:30] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-11-07T16:20:26.050387334+03:30] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.060427872+03:30] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 "modprobe: FATAL: Module aufs not found.\n"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.060881302+03:30] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.061799698+03:30] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.061847774+03:30] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-11-07T16:20:26.061879368+03:30] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-11-07T16:20:26.061942871+03:30] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.062041394+03:30] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.062490054+03:30] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.063331978+03:30] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-11-07T16:20:26.063379114+03:30] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-11-07T16:20:26.063424130+03:30] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-11-07T16:20:26.063448575+03:30] metadata content store policy set             policy=shared
WARN[2022-11-07T16:20:36.064254925+03:30] waiting for response from boltdb open         plugin=bolt
9.# yum info installed containerd.io
Loaded plugins: copr, fastestmirror, remove-with-leaves
Installed Packages
Name        : containerd.io
Arch        : x86_64
Version     : 1.6.9
Release     : 3.1.el7
Size        : 112 M
Repo        : installed
From repo   : docker-ce-stable
Summary     : An industry-standard container runtime
URL         : https://containerd.io
License     : ASL 2.0
Description : containerd is an industry-standard container runtime with an emphasis on
: simplicity, robustness and portability. It is available as a daemon for Linux
: and Windows, which can manage the complete container lifecycle of its host
: system: image transfer and storage, container execution and supervision,
: low-level storage and network attachments, etc.
Introspection Report

inspection-report-20221107_162301.tar.gz
Can you suggest a fix?

I searched formentioned above error and found this solution that due to runc vulnerability kernel version should be greater than 3.XX,
or for workaround can downgrade to old docker version (docker-ce-18.09.0-3.el7, docker-ce-cli-18.09.0-3.el7, containerd.io-1.2.0-3. el7), I did downgrade to docker version 18.09.0 (yum downgrade docker-ce-cli -y, yum downgrade docker-ce -y) but still had the same problem. (for i in {1..30}; do yum downgrade containerd.io -y; done)
pages talked about this issue: (some are in chinese pls use google translate)
<EMAIL_ADDRESS>https://github.com/canonical/microk8s/issues/2223
https://cloud.tencent.com/developer/article/1411527
https://www-cnblogs-com.translate.goog/mrnx2004/p/10601490.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp
Are you interested in contributing with a fix?

yes @timgreen
this error is mentioned in these links too:
https://github.com/canonical/microk8s/issues/2223
https://github.com/canonical/microk8s/issues/3221

Hi @javad87, MicroK8s packages its own containerd so it ignores any containerd/docker installed on the host. Since you are hitting this error and you have identified a potential fix you may want to edit the kubelet arguments [1] found in /var/snap/microk8s/current/args/kubelet and configure MicroK8s to use the containerd you have manually installed. Unfortunately I have no experience in that setup but it should be possible. Of course a safer choice is to get an Ubuntu LTS VM but I suspect this might not be possible.
Another point you should be aware of is that MicroK8s (in its default containerd setup) consumes storage from /var/snap/microk8s/common. This should explain the pod evictions you are seeing as your system seems to be running out of disk even though your /home partition is almost empty.
[1] https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

hi @ktsakalozos,  again tnx for reply.
about storage, microk8s uses  tmpfs for /var/snap/microk8s/common/var/lib/kubelet/* which is volatile and of course will be erased with reboot however containerd data is stored in this path: /var/snap/microk8s/common/var/lib/containerd and for me mounted on /  and I freed up more space and has now 18G free for containerd (and microk8s), anyway still has the same problem, I think we can safely eliminate lack of resource problem!
#df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                  30G     0   30G   0% /dev
tmpfs                     30G     0   30G   0% /dev/shm
tmpfs                     30G  492M   29G   2% /run
tmpfs                     30G     0   30G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   33G   18G  65% /
/dev/loop4                56M   56M     0 100% /var/lib/snapd/snap/core18/2620
/dev/loop0               210M  210M     0 100% /var/lib/snapd/snap/microk8s/2848
/dev/loop2               100M  100M     0 100% /var/lib/snapd/snap/core/11993
/dev/loop1               115M  115M     0 100% /var/lib/snapd/snap/core/13886
/dev/loop3               167M  167M     0 100% /var/lib/snapd/snap/microk8s/4094
/dev/loop5                56M   56M     0 100% /var/lib/snapd/snap/core18/2284
/dev/sda1               1014M  194M  821M  20% /boot
/dev/mapper/centos-home  952G   27G  926G   3% /home
tmpfs                    5.9G     0  5.9G   0% /run/user/0
tmpfs                     59G   24K   59G   1% /var/snap/microk8s/common/var/lib/kubelet/pods/5b91a2a7-6f05-4818-bacd-1f399f332ffa/volumes/kubernetes.io~secret/secret
tmpfs                     59G   12K   59G   1% /var/snap/microk8s/common/var/lib/kubelet/pods/0f8dc418-19c9-4a18-a4ef-8278f203dc
#du -sch /var/snap/microk8s/common/var/lib/*
21G     /var/snap/microk8s/common/var/lib/containerd
712K    /var/snap/microk8s/common/var/lib/kubelet
21G     total
tried to use containerd on my host or even using docker for cotainer runtime by editing file as follows in path: /var/snap/microk8s/4094/args/kubelet and then microk8s stop & microk8s start but containerd has not changed to my cotainerd version running on my host!
#kubectl describe node
still the microk8s version
#journalctl -u snap.microk8s.daemon-containerd --since 17:20 -r -o json-pretty
"_CMDLINE" : "/snap/microk8s/4094/bin/containerd --config /var/snap/microk8s/4094/args/containerd.toml --root
still running containerd from microk8s!!
#cat /var/snap/microk8s/4094/args/kubelet
--kubeconfig=${SNAP_DATA}/credentials/kubelet.config
--cert-dir=${SNAP_DATA}/certs
--client-ca-file=${SNAP_DATA}/certs/ca.crt
--anonymous-auth=false
--root-dir=${SNAP_COMMON}/var/lib/kubelet
--fail-swap-on=false
--feature-gates=DevicePlugins=true
--eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
--container-runtime=remote
--container-runtime-endpoint=unix:///run/containerd/containerd.sock      <--------> change it to host containerd
--containerd=unix:///run/containerd/containerd.sock         <--------> change it to host containerd
--node-labels="microk8s.io/cluster=true"
--authentication-token-webhook=true
--cluster-domain=cluster.local
--cluster-dns=<IP_ADDRESS>
and did not change "/var/snap/microk8s/4094/args/containerd.toml" config file:
need a guide to walk us through parameters and files need to be changed so that container runtime be different from what is running by microk8s. like:
https://dev.to/stack-labs/how-to-switch-container-runtime-in-a-kubernetes-cluster-1628
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd
Also tried useing docker as container runtime in following way with no success:
https://stackoverflow.com/questions/61119975/is-it-possible-to-set-microk8s-to-use-docker-engine-instead-of-containerd
https://github.com/canonical/microk8s/blob/1.13/microk8s-resources/default-args/kubelet
I will post an issue on how to make microk8s use host docker engine or containerd for container runtime? or even for using different version of containerd, Can I just change binary file in this path: /snap/microk8s/4094/bin/?
also checked my kernel for container compatibility and result is ok:
https://blog.hypriot.com/post/verify-kernel-container-compatibility/
https://github.com/containerd/containerd/releases?after=v1.0.0-alpha1
https://containerd.io/releases/
the other option left for me is upgrading my kernel, still have doubt to do it since my docker engine and before reboot microk8s working well with kernel 3.XX

Unfortunatly problem is not solved with kernel update either, I updated kernel version to kernel-ml which is at this time version: 6.0.7-1.el7.elrepo.x86_64
kubectl get nodes -o wide
NAME                    STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
localhost.localdomain   Ready       323d   v1.25.3   <IP_ADDRESS>           CentOS Linux 7 (Core)   6.0.7-1.el7.elrepo.x86_64   containerd://1.6.6
first stopped microk8s then updated kernel and again start it.
also did stop and disable , containerd and docker (#systemctl disable docker) on host machine so that no conflict for shim happens. the only containerd running on my server is from microk8s:
ps -aux|grep containerd
root       2547  4.1  0.1 2795668 73204 ?       Ssl  11:23   0:13 /snap/microk8s/4094/bin/containerd --config /var/snap/microk8s/4094/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
I ran out of option, seams this bug is related to containerd:
https://github.com/containerd/containerd/issues/4068
how can I update version of containerd in microk8s? new version is 1.6.9 and microk8s uses 1.6.6...
Is it possible just replace containerd bin files in path: /snap/microk8s/4094/bin/   ?
https://github.com/canonical/microk8s/pull/3536

I examined result of "microk8s inspect" accuratly for both of my servers which got this problem after reboot, in both files comming errors (in folder snap.microk8s.daemon-containerd) are common (bold & italic):
server 1:
inspection-report-20221109_141931.tar.gz
server 2:
inspection-report-20221109_142214.tar.gz
level=error msg="failed to delete" cmd="/snap/microk8s/4055/bin/containerd-shim-runc-v1 -namespace k8s.io -address /var/snap/microk8s/common/run/containerd.sock -publish-binary /var/lib/snapd/snap/microk8s/4055/bin/containerd -id 1b951cd56c001742864cd6c1d8843b5c3b3fbfa0042064d228c052417f016ef0 -bundle /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/1b951cd56c001742864cd6c1d8843b5c3b3fbfa0042064d228c052417f016ef0 delete" error="exit status 1"

level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:openebs-ndm-w2hl4,Uid:dd17245f-dbf8-4928-8179-2dc878e29a9e,Namespace:openebs,Attempt:21,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"

level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed"
[ERROR][6879] customresource.go 136: Error updating resource Key=IPAMBlock(10-1-103-128-26) Name="10-1-103-128-26" Resource="IPAMBlocks" Value=&v3.IPAMBlock{TypeMeta:v1.TypeMeta{Kind:"IPAMBlock", APIVersion:"crd.projectcalico.org/v1"}, ObjectMeta:v1.ObjectMeta{Name:"10-1-103-128-26", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"45540869", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.IPAMBlockSpec{CIDR:"<IP_ADDRESS>/26", Affinity:(*string)(0xc00028c410), StrictAffinity:false, Allocations:[]*int{(*int)(0xc0003cfe28), (*int)(0xc0003cfd08), (*int)(0xc0003cfef8), (*int)(0xc0003cfe88), (*int)(0xc0003cffc8), (*int)(0xc0003cfe98), (*int)(0xc0003cff38), (*int)(0xc0003cfea8), (*int)(0xc0003cff58), (*int)(0xc0003cfff8), (*int)(0xc0003cfd98), (*int)(0xc0003cfdd8), (*int)(0xc0003cfc38), (*int)(0xc0003cfc48), (*int)(0xc0003f6138), (*int)(0xc0003cfde8), (*int)(0xc

My problem solved: LONG STORY SHORT ----------> this problem exactly created after reboot, the reason is snap will automaticaly update installed packages, and there is a discution about this forceful behaviour:(https://forum.snapcraft.io/t/disabling-automatic-refresh-for-snap-from-store/707/322), fortunatly I could revert back to previous vrsion (https://askubuntu.com/questions/1198022/how-to-run-a-previous-version-of-a-snap-package):
#snap list "microk8s" --all
Name      Version  Rev   Tracking  Publisher   Notes
microk8s  v1.23.1  2848  -         canonical✓  classic
microk8s  v1.25.3  4094  -         canonical✓  disabled,classic
#sudo snap revert microk8s
you can see after reboot snap changed microk8s from v1.23.1 to v1.25.3 and then microk8s containerd does not work on my system it is not also related to kernel version (as I explained above I update my kernel to 6.xx and error still remained) I again moved to kernel 3.xx and all my containers created after reverting back to previous microk8s version! @ktsakalozos So could you please ask contributors to this project see whats going on with new version that containerd does not work correctly? and also add to installation guide that if you want ur cluster remain alive after reboot first stop (hold) snap to automaticly update (refresh) microk8s and ruining everything....
this is some work around to hold damn auto update of snap:
https://askubuntu.com/questions/1131182/how-can-i-disable-automatic-update-for-a-single-snap-in-ubuntu
https://snapcraft.io/docs/keeping-snaps-up-to-date#:~:text=Snaps update automatically%2C and by,with the snap refresh command.
https://snapcraft.io/blog/how-to-manage-snap-updates
containerd has a great tool named ctr which you can test if it is working or not and running container:
https://iximiuz.com/en/posts/containerd-command-line-clients/
https://github.com/containerd/containerd/blob/main/docs/getting-started.md
#microk8s  ctr run docker.io/library/busybox:latest javad87
#microk8s ctr container ls
f82c42a040ce4594515c2e0ee5cbad0b114129a1364d58023562e19666d60684    docker.io/openebs/provisioner-localpv:2.12.1               io.containerd.runc.v1
f9c2ea2b97af5100b82887b7b45843b3331099a258e69b3c72249797ec741da6    docker.io/jupyterhub/configurable-http-proxy:4.5.0         io.containerd.runc.v1
fb9914a1ad8be600f9bc2b62ea5f0d2cd3a9bbc35b2d50a07bffe490f7e672fc    k8s.gcr.io/pause:3.1                                       io.containerd.runc.v1
fcc29dbd6631a4b3aa5c301380ae1738d424071ccac8df4e9ff662e7cea0737f    docker.io/calico/cni:v3.13.2                               io.containerd.runc.v1
fd1a09f971b3af5a68b0ccf87a867f391e575431109546752ee71bb10a09d844    k8s.gcr.io/pause:3.1                                       io.containerd.runc.v1
fe7497b3530622571383ee7e167929f6cb8ba312e93dc494c24a51b2bba0a533    k8s.gcr.io/pause:3.1                                       io.containerd.runc.v1
gorge                                                               docker.io/library/busybox:latest                           io.containerd.runc.v2
javad                                                               docker.io/library/busybox:latest                           io.containerd.runc.v2
javad87                                                             docker.io/library/busybox:latest                           io.containerd.runc.v2
v1                                                                  docker.io/library/busybox:latest                           io.containerd.runc.v2
v2                                                                  docker.io/library/busybox:latest                           io.containerd.runc.v2
with new version of microk8s it makes the error:
error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
but still creates container with io.containerd.runc.v2, there is a problem with "shim" creation for stdout, stderr to container, I don't know nuts and bolts of container creation, this guy explained what is going on under the hood:
https://iximiuz.com/en/posts/journey-from-containerization-to-orchestration-and-beyond/#containerd

On one hand, containerd will not auto-update system packages. Instead, MicroK8s comes with the required packages and binaries included in the snap bundle. Instead, I noticed that:
#snap list "microk8s" --all
Name Version Rev Tracking Publisher Notes
microk8s v1.23.1 2848 - canonical✓ classic
microk8s v1.25.3 4094 - canonical✓ disabled,classic

you can see after reboot snap changed microk8s from v1.23.1 to v1.25.3

This should not be the case in general. By default, MicroK8s installs and pins to a minor version (e.g. 1.23). You can see this by running:
$ sudo snap info microk8s
....
tracking:    1.2X/stable

Minor version upgrades are not automatic, and would only happen if (a) you changed the channel with a snap refresh microk8s --channel 1.25 or (b) you installed microk8s with snap install microk8s --channel latest, which is not recommended for production setups.
In any case, mind the following:

MicroK8s upgrades should not skip minor versions. If you are on 1.23 and want to move to 1.25, you should first upgrade to 1.24, ensure all workloads are fine, and only then move to 1.25. This is a requirement of the upstream Kubernetes project, and not a peculiarity of MicroK8s itself.
MicroK8s comes with all packages and binaries it needs, no packages are installed/updated in the underlying system.
For production setups, make sure to install from a specific minor version snap install microk8s --channel 1.2X/stable, so that minor version upgrades are managed properly. See the documentation for more details on upgrading a production cluster.

Hope this helps the OP, and any future onlookers.


Fix: README.md has wrong unstructured install command

As in https://pypi.org/project/unstructured/ described, it should be:
pip install "unstructured[all-docs]"

Oof, thanks @cutmasta-kun!


Generating diff between two large C files takes a long time & uses a lot of memory

Generating a diff between two big C files takes a long time and uses a lot of memory.
$ difftastic p1.c p2.c

In fact I aborted this after ~30min at then 12GB of RAM usage.
diff tells me that roughly 8,800 out of 23,000 lines changed here. These are parser.c files generated by tree-sitter.
While I wouldn't expect an instantaneous response from difftastic, not being able to complete in a couple minutes within reasonable RAM consumption feels off to me.
Files:

p1.c.gz
p2.c.gz

Could you confirm what version of difftastic you're using? 0.20 has DFT_NODE_LIMIT and DFT_BYTE_LIMIT which should prevent runaway memory usage.

Hmm, I can reproduce this. It should have hit DFT_BYTE_LIMIT but it doesn't seem to.

Could you confirm what version of difftastic you're using?

I missed the rename of the binary so difftastic in my reproducer above referred to a 0.11.0 installation I still had sitting around.
With 0.21.0 I still see this.

OK, something in the line-based differ is using an absurd amount of memory in these files. I'm gonna release 0.22 soon, but I'll follow up on this after.

It looks like the textual diffing is struggling with a file of this size: it's 140,000 lines and the first unchanged line is around line 11,000. The GNU diff docs mention that it strips leading/trailing lines that are unchanged, and difftastic should do this too (for text, it already does this for syntax).
That might not be enough though. There's several LCS diffing libraries on crates.io, but most of them seem unmaintained. I know git has switched diffing algorithms over time, it might be worth looking at its current default algorithm.

Not sure this is just the diffing since I also see e.g., difft --dump-ts taking a long time with these files (memory seems to be under control there though). Similarly for --dump-syntax. I checked tree-sitter parse with the vendored grammar and it seems to return rather quickly.

Looks like --dump-ts was also computing syntax highlighting, which is expensive in files of this size. I've improved this in dba68d1d2ac80db162a051988a10c7f249557ee3, although normal usage still need to compute highlighting.

OK, performance for this file is now acceptable (3 seconds my machine). There's more optimisations possible, but I'm going to consider this fixed.


Max, how to optimization objective function trace?

I'm using GPy, and I want to plot a figure like this, how to do this under paramz?
https://ibb.co/k5fUWT

You will need to trace the objective and parameters manually. There is two ways of achieving this.

You inherit from the model you want to record and overwrite parameters_changed function. This function gets called once per iteration. In there you can record the parameters you want to record in an array of sorts. Make sure you call the super parameters_changed!
You write a little wrapper around the optimizer, which records the optimization process and then subsequently does the optimization. You will be exposed to the raw optimization array though, so you need to map back to the parameter names.

Than you, Max! By the way, is there any framework like paramz that could do optimization such as ADMM and SGD? Because I want to implement RPCA, I don't known if paramz's Quasi Newton able to optimize RPCA's objective function

You can use adadelta or rprop from the climin package in paramz :)

On 22 Jun 2018, at 16:34, Jeff Li<EMAIL_ADDRESS>wrote:
Than you, Max! By the way, is there any framework like paramz that could do optimization such as ADMM and SGD? Because I want to implement RPCA, I don't known if paramz's Quasi Newton able to optimize RPCA's objective function
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

Max, I have a question about optimization, I don't know if it's suitable to ask you here, but I really want to get some advice from an expert.
You implement Lasso regression using gradient based optimization algorithm, but I have been told that LASSO should be optimized using ADMM, because L1 regularizer is not differentiable. But actually LASSO can be optimized using gradient based algorithm, that is conflict in my knowledge. Why? What I'm missing? Is L1 regularizer actually differentiable or gradient descent or quasi-newton can optimize objective function with L1 regularizer event if it is not differentiable?


Add new training license

This change is

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/e41900a392da79f0d8e4ccb5fd856204331b886f/


make api rate limit test a little more robust

Description
This PR...

Fixes: #6975

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[x] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

unit test

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian :heavy_multiplication_x: suse15. SL-JID 4911

@blueorangutan package

@davidjumani a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_multiplication_x: debian :heavy_check_mark: suse15. SL-JID 4919

@blueorangutan package

Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian :heavy_multiplication_x: suse15. SL-JID 4941

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_multiplication_x: debian :heavy_check_mark: suse15. SL-JID 4942

@blueorangutan package

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_multiplication_x: debian :heavy_check_mark: suse15. SL-JID 4944

test succeed after 12 seconds, looks good.
20:39:01 [INFO] Running org.apache.cloudstack.ratelimit.ApiRateLimitTest
20:39:06 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
20:39:06 SLF4J: Defaulting to no-operation (NOP) logger implementation
20:39:06 SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
20:39:06 log4j:WARN No appenders could be found for logger (org.apache.cloudstack.ratelimit.ApiRateLimitServiceImpl).
20:39:06 log4j:WARN Please initialize the log4j system properly.
20:39:06 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
20:39:10 time elapsed 3347 ms
20:39:13 [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.544 s - in org.apache.cloudstack.ratelimit.ApiRateLimitTest


Permission Denied on /opt/factorio/saves

I could use some help while setting up the docker image. I did use your docker image for 0.16 and i didnt have those problems but i cant get it to work for 0.17.
I get a permission denied whenever I start the docker container for factorio.
I crash with the error Util.cpp:75: filesystem error: status: Permission denied [/opt/factorio/saves]
I have set up the client as follows:
`sudo docker pull dtandersen/factorio:0.17
sudo docker run -d
-p 34198:34197/udp
-p 27016:27015/tcp
-v /volume1/GameInstances/factorio_map017:/factorio:Z
--name factorio2
dtandersen/factorio:0.17`
I changed the chown of the /volume1/GameInstances/factorio_map017 to 845:845
I let the container run in high privilege mode.
I tried the Z at the end.
Nothing worked so far. Has someone an idea where I might have done something wrong?
errorlog_factorio2.txt

You can try 0.17.2-2. I think maybe the map generator created some files as root.
https://github.com/dtandersen/docker_factorio_server/commit/8e7bca222ae57333afcee7390ffaad683e537f27

just tried it, its still the same.
i have no idea from where the container gets the /opt/factorio/saves...

You can use this to see:
[vagrant@docker 0.17]$ docker run --rm -it --entrypoint /bin/sh dtandersen/factorio:0.17.2-2
/ # cd /opt/factorio/
/opt/factorio # ls -al
total 20
drwxrwxrwx    4 factorio factorio      4096 Feb 27 19:57 .
drwxr-xr-x    1 root     root          4096 Feb 27 19:57 ..
drwxr-xr-x    3 factorio factorio      4096 Feb 27 19:57 bin
-rw-r--r--    1 factorio factorio       998 Jan 16  2018 config-path.cfg
drwxr-xr-x    4 factorio factorio      4096 Feb 27 19:57 data
lrwxrwxrwx    1 factorio factorio        14 Feb 27 19:57 mods -> /factorio/mods
lrwxrwxrwx    1 factorio factorio        15 Feb 27 19:57 saves -> /factorio/saves
lrwxrwxrwx    1 factorio factorio        19 Feb 27 19:57 scenarios -> /factorio/scenarios
lrwxrwxrwx    1 factorio factorio        23 Feb 27 19:57 script-output -> /factorio/script-output

duplicate of #173


Rewrite build scripts

I have rewritten and perhaps slightly overengineered the build scripts to suit my own needs, but I hope they can be useful for the project anyway since there is no real build system currently and I feel that the current build scripts could have been in a better condition when it comes to which features are supported in the scripts and how they work.
I oftentimes find myself needing to disable some part of one or both of the scripts when I run them locally because I don't always want to go through the full cycle.
I fully understand if this is not something you would like to merge because the scripts are now considerably longer and more complex, but still thank you for your consideration.
Although I've done a fair bit of testing and will keep doing so, it's a good idea if more people to try the new scripts. If this is something that can be useful then I think it is a good time to discuss command line options and default values. For the time being it may even be a good idea to keep the old scripts as fallback.
If you want to give the scripts a go, try this:
./script/build.sh --reformat --build-examples --test --go-test

Use script\build.bat on Windows with the same options.
Help text (sh):
This is the build script for the webview library.

Usage:
    program --help
    program [--clean] [--build] [--build-examples] [--build-tests]
            [--test] [--target-arch=ARCH] [--reformat] [--lint[=lax]]
            [--cc=PATH] [--cxx=PATH] [--go-build-examples] [--go-test]

Options:
    --help                      Display this help text.
    --clean                     Clean the build directory.
    --build                     Build library.
    --build-examples            Build examples (implies --build).
    --build-tests               Build tests (implies --build).
    --test                      Run tests (implies --build-tests).
    --target-arch=ARCH          Build for the target CPU architecture.
                                Choices: all, x86, x64
    --reformat                  Reformat code (requires clang-format).
    --lint                      Run lint checks (requires clang-tidy).
    --lint=lax                  Run lint checks in lax mode.
    --cc                        C compiler binary, e.g. cc, gcc or clang.
                                Can be set by the CC environment variable.
    --cxx                       C++ compiler binary, e.g. c++, g++ or clang++.

Cross-compilation
=================

Your toolchain must have multilib support. In addition you must install
dependencies for your target architecture onto your system.

Help text (bat):
This is the build script for the webview library.

Usage:
    program --help
    program [--clean] [--build] [--build-examples] [--build-tests]
            [--test] [--target-arch=ARCH] [--reformat] [--lint[=lax]]
            [--cc=PATH] [--cxx=PATH] [--go-build-examples] [--go-test]
            [--fetch-deps] [--toolchain=TOOLCHAIN] [--webview2-version=VERSION]

Options:
    --help                      Display this help text.
    --clean                     Clean the build directory.
    --build                     Build library.
    --build-examples            Build examples (implies --build).
    --build-tests               Build tests (implies --build).
    --test                      Run tests (implies --build-tests).
    --target-arch=ARCH          Build for the target CPU architecture.
                                Choices: all, x86, x64
    --reformat                  Reformat code (requires clang-format).
    --lint                      Run lint checks (requires clang-tidy).
    --lint=lax                  Run lint checks in lax mode.
    --cc                        C compiler binary, e.g. cc, gcc or clang.
                                Can be set by the CC environment variable.
                                Currently only supported with MinGW.
    --cxx                       C++ compiler binary, e.g. c++, g++ or clang++.
                                Currently only supported with MinGW.

                                Implied when building and linting.
    --toolchain=TOOLCHAIN       C/C++ toolchain.
                                Choices: msvc, mingw.
    --webview2-version=VERSION  WebView2 version to use.

Compilation with MinGW-w64
==========================

Unless your MinGW-w64 toolchain has multilib support then you need to
install both the 64-bit- and 32-bit toolchains for cross-compilation.
MinGW-w64 is expected to be found in one of the following locations:
 - C:\mingw64 or C:\mingw32
 - C:\msys64\mingw64 or C:\msys64\mingw32
 - Chocolatey

Changes
Scripts:

They have the same features as before with features that were missing in one script being implemented in the other.
They now support command line options and have proper help text.
The Windows script now supports reformatting and linting. The clang-tidy rules seem to need an overhaul but that's for another time (#767).
Build output now goes into the build on all platforms.
Supports cross-compilation as long as the platform supports it.

On Linux you need multilib support when using GCC.
On Windows you need both 64-bit and 32-bit MinGW-w64 and the script will find the appropriate installation.
Modern macOS does not support 32-bit.

A different version of Microsoft WebView2 can now be specified as a command line option.
Variables coming from the Visual Studio/VC++ toolchain no longer pollute the environment outside the scope of where it is being used. Previously there was a problem where running the script too many times resulted in an error because vsdevcmd keeps variables growing until command lines are too long.
They always overwrite existing files now on Windows. The old Windows script skipped building files that had already been built but that made it inconvenient to make changes to the library and rebuild it. I would like to implement some state management but for now that is not necessary, and having a slower but correct build is better than a fast and error-prone build.
Searches for source files and will therefore find sources that were previously not touched in some way (e.g. one of the examples not being reformatted).
nuget.exe is not a requirement anymore on Windows.
A toolchain (msvc, mingw) can now be specified on Windows which will allow the library and examples to be built using the specified toolchain.
C/C++ compiler binary can now be specified for Unix platforms and Windows when using MinGW-w64 via CLI options or environment variables.

GHA workflow:

Added a new job purely for linting.
Updated the version of the checkout action.
Installs a specific version of MinGW-w64 from WinLibs for improved control and compatibility. The Chocolatey distribution has a few issues:

Does not have multilib support and one cannot have 32/64-bit installed side by side.
Uses the UCRT runtime with no option for MSVCRT. I chose the MSVCRT runtime over UCRT because MSVCRT is available on old versions of Windows while UCRT has only been shipped with Windows since Windows 10.

Builds and tests with combinations of:

MSVC and MinGW-w64 (GCC and Clang)
x86 and x64

Overall, this solves many problems with our current build system. I really like it. It makes building and testing a whole lot more straightforward.
I'm still hesitant about maintaining two copies of essentially the same thing for our build system, especially now that the scripts are far more complex than they were before.
I'd also like @dandeto's opinion on this, since he did some work on the build script.

The scripts now support overriding dependent or implied options from the command line.
This means that by default e.g. --test implies --build-tests which implies --build every time except when you override options:

--test --build-tests=false --build=false skips building the library, skips building tests but runs tests.
--test --build=false skips building the library, builds the tests (implied by --test) and then runs tests.

Dang, dude! This looks like a ton of work! This is a great solution to all of the problems I saw with the old script. Thanks for doing this.
Were you able to remove all those entries in the .gitignore because a developer can run the clean routine before commits? I think the webview nuget should still be gitignored unless you did something I missed.
As to @justjosias's comment... He brings up a good point about maintaining 2 copies. That's necessary unless we commit to a 'proper build system' which comes with its own set of challenges and setup work. On one hand I think your code is so clean that it is maintainable by others, but on the other hand, it may look daunting and comes with a learning curve.
I'll say this: this PR is a huge improvement and adds all the functionality that we have needed forever. I would say we should try to convert this into a makefile, but that's not compatible with the windows compiler as far as I know.

Thanks for taking a look at it. It took a few sleepless nights. :)
Build output goes into the build directory which is still in the .gitignore file. Nuget stuff goes into build/deps so it's ignored as well.
To be frank I do not love the idea of having two scripts that essentially do the same job but I don't quite see how to do it without settling for a proper build system. Well, we could rewrite the whole thing into another language like Python but that doesn't solve the whole problem, plus it adds another dependency.
I don't know how established makefiles are on Windows but it's certainly not something I am used to seeing there. I see more and more projects use something like CMake and Meson across platforms. In my own fork of the library I went with CMake because it's established and flexible, and I just didn't find Meson to be flexible enough.
In any case the solution must work well across platforms, and it must do its best to support both development of the library as well as consumption of it. The way to do that in my opinion is to settle for an established build system that has good platform support, but this PR is an attempt to meet somewhere in the middle.

I like the idea of pulling your work in since you've already done it... Then we can focus on investing in a build system as our long-term solution that we can better maintain. I've heard good things about CMake but never used it. I'll research make since I brought it up (and I like it), but I don't think its going to work for this project.

I agree with @dandeto. I think we should review this and then merge it. It's an amazing contribution to the project. It doesn't appear to break anything, and makes things like autogenerating DLLs on releases a whole lot simpler.
We should still look to setting up a complete build system in the future. Probably CMake or Meson. Both were mentioned by zserge and the former used to be used in Webview years ago. I also agree that using Python isn't a good idea.
Is there anything else you want to address in this PR before merging @SteffenL?
Also, discussion of the command line options and behavior should be done now, before we merge it, to avoid breaking the scripts in the future.

I would love to participate in a future discussion about the direction of the build system, CMake, etc.
I think the only thing I would like to address in this PR is updating the readme file.
Since you seem positive to review and merge this then yes, I think it's a good time to discuss the command line options and behavior.

As for breaking changes, one that is going to be immediately apparent is that that the scripts do not by default do anything now except display the help text.

I feel that it would make sense to decouple the library's build scripts from end users.
I just found out that the tar command is available on Windows 10. That means that we can stop depending on the nuget executable and simply use curl to download the WebView2 package and then use tar to extract it.

Now that I have confirmed that the library can be built on Windows entirely using MinGW-w64, it could make sense to be able to select which toolchain to use in the build script.

Because the CI job for Windows takes longer now, I have attempted to split it up into x64 and x86. Linting takes a good chunk of time since it's done for both target architectures. Actually this is something that could be done later so I don't mind reverting that. I'm experimenting a bit with that now.

I think I'm happy with how the build has been split now. It will run lint checks while the main build is on-going  to get feedback sooner. I also disabled the code reformatting in the CI job because reformatting has never given any feedback in case of incorrect code formatting.

With #773 being merged, the user-facing instructions for using the library ("getting started") have been decoupled from the build scripts. Does that in any way change the stance on whether to rewrite the scripts into another common language like Python, or whether to keep only the shell script? I was reluctant to do so before the merge but I think the main purpose of the scripts now is to support development.

This actually seems reasonable now. It's not a proper build system by any means, but for dev scripts it doesn't need to be. I think having Python installed is an acceptable prerequisite for running the scripts. It would definitely be easier to maintain.
How would Python build scripts look? Is there a Python interface to mingw, gcc, and other build systems?
What do you think, @dandeto?

I am not sure if it would be much different from running commands manually but I can imagine that Python has a lot of third party packages available through pypi that can help. It would be great if we could have incremental builds and what-not but I think it would be better to have a real build system underneath to get all those goodies even though we could have a CLI on top made with Python.

One more thing is debugging the library. That's one thing I would like to address properly.

If the script could also generate a compile_commands.json file then perhaps code editors/IDEs would be able to use that at least for C/C++.

Throwing an idea out there: Ninja build system. Visual Studio includes Ninja. I have never written Ninja build scripts by hand before but I usually pair Ninja with other build systems like CMake.

Ninja would be interesting. I've mostly seen it used with Meson. We might want to consider that as well. I don't have extensive experience with build systems beyond make and some CMake.
Simplicity and interoperability are key to whatever is decided.

I also have experience with Make, but none of the other tools mentioned. I'm not afraid of learning, though.
A rewrite in python of the build script would make it cross-platform, but don't we all have mingw? We should be able to run the .sh script and get rid of the .bat script as long as there is some form of OS detection so that the different dependencies can be installed. A rewrite in python just seems like extra work at this point. If we want a rewrite, I think changing it to a 'proper' build system would be more future-proof.

Respect++

I had some time to do more work on these scripts. Still not ideal to have long build scripts, and to have two of them... Is there anything here that you feel is worth using? Any other wishes?
Personally I would probably not have merged this because nearly the whole point with these scripts disappear if and when we start using a better build system.

At this point, your scripts are a work of art! It's definitely an improvement on the script, so I'm not opposed to merging it, even if we decide to go another direction later on. I think webview is a great project for a build system that doesn't require a bunch of config files, because at the moment it is only a single-header with tests and examples. The only downside I see is that you had to write one sh and one bat file.
I previously suggested using make, but I did some reading on it recently and it has some platform compatibility issues.

I rewrote the scripts again into Python. The result is even better and cleaner than the scripts in this PR but again the system is complex because I wanted to take into account different use cases that are important to me such as the following:

header-only library
static library
shared library
multiple target operating systems and architectures for C/C++ as well as Go
compiler detection
multiple toolchain environments (MSVC, variants of MinGW-w64)
multiple compilers and versions (minimum GCC, Clang, MSVC)
building proper shared library, i.e. proper so/dylib/dll with rpath taken into account as well.

After all this work I am again left with the same question as before and that is who is willing to maintain such complex scripts.
While I am proud of the Python rewrite, I would again lean toward not merging even that.
For those interested, here is a preview with a minimum amount of options.

As for build systems, I would lean toward something that is both flexible and helps us move on with our day. Something that is widely adopted and well supported.
I am not sure I can imagine Make/NMake offering the kind of quality of life I would want for core developers and library integrators.
I would personally go with CMake because I'm familiar with it. It's widely adopted, flexible and also a bit annoying to set up, but far better than nothing.
CMake and Meson have been brought up before with no conclusion, so I am hesitant to dedicate time on it.

Closing this for now because I don't think anyone will have an easy time maintaining the scripts.


Fail fast without forwarders

Seeing as everyone else is creating PRs.
Fixes #13

Awesome, thanks! :)


MM-22064: Update verbage for domain restrictions

Summary
Update verbage on domain restrictions.
Ticket Link
https://mattermost.atlassian.net/browse/MM-22064
Related Pull Requests

Has server changes https://github.com/mattermost/mattermost-server/pull/14563

Screenshots

@thefactremains I feel that the helper text displayed under this field needs an update too. Currently it only talks about the account creation and indicates this setting only affects email login for users.
User accounts can only be created from a specific domain (e.g. "mattermost.org") or list of comma-separated domains (e.g. "corp.mattermost.com, mattermost.org"). This setting only affects email login for users. For Guest users, please add domains under Signup > Guest Access.

Can you please confirm the new helper text if this needs to be changed?

@thefactremains I feel that the helper text displayed under this field needs an update too. Currently it only talks about the account creation and indicates this setting only affects email login for users.
User accounts can only be created from a specific domain (e.g. "mattermost.org") or list of comma-separated domains (e.g. "corp.mattermost.com, mattermost.org"). This setting only affects email login for users. For Guest users, please add domains under Signup > Guest Access.

Can you please confirm the new helper text if this needs to be changed?

yes, thank you @srkgupta
I suggest the following change:
New user accounts are restricted to the above specified email domain (e.g. "mattermost.org") or list of comma-separated domains (e.g. "corp.mattermost.com, mattermost.org"). New teams can only be created by users from the above domain(s). This setting affects email login for users. For Guest users, please add domains under Signup > Guest Access.

/update-branch


accept ts-files and push --loader tsx

I use the node test runner with typescript files without explicitly translating the files to js.
The files are only translated when executed.
I think your plugin is great. So it would be nice if it could also process ts files without having to translate them to js.
That's why I created the pull request.
The changes have no effect when the plugin processes js files

Note: I couldn't work with the "nodejs-testing.nodejsParameters" parameter, because these parameters are not passed to the spawn, that runs the unit test.

Thank you for the feedback. I have added Prettier as a script in the package.json, as well as in the dev dependencies. Now the package.lock also matches the version.

Thanks for the followup! With the last change to add that check this should be good to go 🙂

thanks:-)


Spec is missing

Is there a spec somewhere that is being worked on? So far I see a list of parameters, but there is no details or document describing how/where to use them. For example, there are parameters listed to, presumably an HTTP POST endpoint at /lsp/channel, but there's no indication of where to put them - is the body expected to be JSON or something else? There is no human-readable text explaining what a server responding to such a request has to do, only some fields they should (must?) include in the reply. Amounts are numbers, but..in what denomination, and how are precision errors handled? More importantly, what is the "fee", is it something paid explicitly? Is it something taken out of the first HTLC? When can I expect the channel to be opened, and do I get an option for how the fee is paid, etc.

Given that we have decent LSPS0, LSPS1, and LSPS2 specs already merged in, can we close this issue?


Graduated zeroes error

When the value on graduated theme is zero the style is never set, even if there is a zero category.
This PR will set the style if one has not been set.
Using graduated_theme in bugs_testing, the polygon over birmingham will be blank on main, and will fill with green correctly on this PR.

To see the specific tasks where the Asana app for GitHub is being used, see below:

https://app.asana.com/0/0/1206744686047675

@AlexanderGeere Please check my last commit. I think this can be done with array methods.
  if (!isNaN(catValue) && catValue !== null) {

    const isLassThan = cat => catValue <= cat.value;

    let index = theme.cat_arr.findIndex(isLassThan)

    let cat = theme.cat_arr.at(index)

    let catStyle = structuredClone(cat.style || cat)

    delete catStyle.label

    mapp.utils.merge(feature.style, catStyle)
  }

@dbauszus-glx  Works in the graduated_theme work space as well, looks a lot cleaner too. Nice!


Fix misleading figure in descriptor tables doc

I remember trying to create a texture in an upload heap and the debug layer yelling at me.
I've changed the relevant parts of the doc. I also provided a new diagram.
Not sure what the conventions are for storing image for diagrams so do let me know if it should be changed.
I've made an issue here https://github.com/MicrosoftDocs/feedback/issues/3669
also lmk if im just plain wrong I'll close both items :)

@FreakingBarbarians : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Thanks for the PR and the GitHub issue. I'll run with the issue, do some investigation, and if we need to make a change to the topic then I'll make that in a separate PR. I understand the changes you're proposing in this PR (change "an upload" to "the default", in both the topic and the diagram).
So I'll go ahead and close this PR, and we'll pick up from here in the GitHub issue.
Thanks again!
Steve


column index out of range when reading from db

this is the code I have to read from my database
new Promise((resolve, reject) => {
    db.each(
        'SELECT * from moderation WHERE userID = ?',
        [user.id],
        (err, row) => {
            if (err) {
                console.log(err)
                reject(err)
            }
            else {
                console.log(row)
            }
        },
        (err, _) => {
            if (err) {
                reject(err)
            }
            else {
                resolve()
            }
        }
    )
}).then(
    message.channel.send(embed)
)

whenever I run it it gives the error
  errno: 25,
  code: 'SQLITE_RANGE'
}

How do I stop this error and get it to return the row correctly

If you run the query via db.all() or .row(), do you get the same error? The code looks right to me (our unit tests show that a like-query passes).
I also noticed there's this issue too on the sqlite3 repo:
https://github.com/mapbox/node-sqlite3/issues/999

I think I know what's wrong. Your fourth parameter / second callback. It doesn't exist in this library.
The second callback is actually the result of the call to each(). See the readme:
https://github.com/kriasoft/node-sqlite#each
And the impl for this:
https://github.com/kriasoft/node-sqlite/blob/master/src/Database.ts#L186
If you are still having an issue, I would check that user.id actually has a value. If not that, then this might be a bug with sqlite as reported by:
https://github.com/mapbox/node-sqlite3/issues/999

working now, I've used sqlite3 lots so must have done that accidentally

I'll add some kind of guard for this, like throw an error of two callbacks are found. Thanks for your help!

Addressed in https://github.com/kriasoft/node-sqlite/commit/ee59e5e868105819b2d00e235d19c0a20031c6f9


zfsUnstable: 2.0.0-rc4 -> 2.0.0-rc5

Motivation for this change
latest zfsUnstable release
https://github.com/openzfs/zfs/releases/tag/zfs-2.0.0-rc5
some linux  5.10 support, and other bug fixes
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg test zfs.unstable

builds were already failing in master
https://github.com/NixOS/nixpkgs/pull/102813
2 packages failed to build:
linuxPackages_latest_hardened.zfsUnstable linuxPackages_latest_xen_dom0_hardened.zfsUnstable

16 packages built:
linuxPackages-libre.zfsUnstable linuxPackages.zfsUnstable linuxPackages_4_14.zfsUnstable linuxPackages_4_19.zfsUnstable linuxPackages_4_4.zfsUnstable linuxPackages_4_9.zfsUnstable linuxPackages_5_8.zfsUnstable linuxPackages_5_9.zfsUnstable linuxPackages_hardened.zfsUnstable linuxPackages_latest-libre.zfsUnstable linuxPackages_latest_xen_dom0.zfsUnstable linuxPackages_testing_bcachefs.zfsUnstable linuxPackages_xen_dom0.zfsUnstable linuxPackages_xen_dom0_hardened.zfsUnstable linuxPackages_zen.zfsUnstable zfsUnstable

Result of nixpkgs-review pr 102813 1

2 packages marked as broken and skipped:

linuxPackages_hardkernel_4_14.zfsUnstable
linuxPackages_hardkernel_latest.zfsUnstable

2 packages failed to build:

linuxPackages_latest_hardened.zfsUnstable
linuxPackages_latest_xen_dom0_hardened.zfsUnstable

16 packages built:

linuxPackages-libre.zfsUnstable
linuxPackages.zfsUnstable (linuxPackages_5_4.zfsUnstable)
linuxPackages_4_14.zfsUnstable
linuxPackages_4_19.zfsUnstable
linuxPackages_4_4.zfsUnstable
linuxPackages_4_9.zfsUnstable
linuxPackages_5_8.zfsUnstable
linuxPackages_5_9.zfsUnstable (linuxPackages_latest.zfsUnstable)
linuxPackages_hardened.zfsUnstable
linuxPackages_latest-libre.zfsUnstable
linuxPackages_latest_xen_dom0.zfsUnstable
linuxPackages_testing_bcachefs.zfsUnstable
linuxPackages_xen_dom0.zfsUnstable
linuxPackages_xen_dom0_hardened.zfsUnstable
linuxPackages_zen.zfsUnstable
zfsUnstable


Extract command polling logic to it's own crate

This patch extracts the command polling logic to it's own crate.
It doesn't use any feature flags, and does not duplicate the polling
logic.
The internal behavior is delegated to a mpsc channel, where the caller
can decide how to process completions and updates.
Signed-off-by: Joe Grund<EMAIL_ADDRESS>

This change is

We need wait_for_cmds_success here as well.

Looks like it's been added in your patch.


Tactics ogre blending not working with windows

is what it should look like. I don't properly blend the contents of the circular window in the center of the screen. I have a feeling I don't really do anything with window blending information either in or out

Notably this still doesn't work after implementing obj windows

metroid fusion during intro also hits some issues with blending


add config for huzza32 v2 with original featherwing

This is simply to add a config for using the adafruit huzzah32 v2 with the original featherwing 3.5"

Thank you


Not able to use package! for package zh-align

What did you expect to happen?
with the command
(package! zh-align :recipe
  (:host github
   :repo "chen-chao/zh-align.el"))

placed in $DOOMDIR/packages.el, I expected to get the package installed in my system.
Package at hand: https://github.com/chen-chao/zh-align.el
What actually happened?
After putting in place the code mentioned above, in the packages file, I got the following from doom sync, I got
      > Cloning zh-align.el.git (for zh-align)......
      x There was an unexpected error
        Message: error
        Error: (error "Malformed protocol detected: (:host github :repo \"https://github.com/chen-chao/zh-align.el.git\")")
        Backtrace:
          (signal error ("Malformed protocol detected: (:host github :repo \"https:/...
          (error "Malformed protocol detected: (:host %S :repo %S)" github "https://...
          (progn (error "Malformed protocol detected: (:host %S :repo %S)" host repo...
          (if (string-match-p ":" repo) (progn (error "Malformed protocol detected: ...
          (cond ((null host) repo) ((memq host '(bitbucket gitlab github)) (if (stri...
          (straight-vc-git--encode-url "https://github.com/chen-chao/zh-align.el.git...
          (let ((success nil) (repo-dir (straight--repos-dir local-repo)) (url (stra...
          (let ((package (plist-get --recipe-- ':package)) (local-repo (plist-get --...
          (let ((--recipe-- recipe)) (let* ((--cl-rest-- --recipe--) (upstream (car ...
          (straight-vc-git-clone (:host github :repo "https://github.com/chen-chao/z...
      ! Extended backtrace logged to ~/.edoom/.local/doom.error.log

Describe your attempts to resolve the issue
I have tried all forms of :repo url assignment possible. That is, with the url from the page, https://github.com/chen-chao/zh-align.el, with the git code url, https://github.com/chen-chao/zh-align.el.git, finally, with the user-name/repo-name, chen-chao/zh-align.el.
I got the same error, everytime.
Steps to reproduce

Put any of the three snippets on your $DOOMDIR/packages.el,

(package! zh-align :recipe
  (:host github
   :repo "https://github.com/chen-chao/zh-align.el.git"))

(package! zh-align :recipe
  (:host github
   :repo "https://github.com/chen-chao/zh-align.el"))

(package! zh-align :recipe
  (:host github
   :repo "chen-chao/zh-align.el"))

Hit doom sync in the terminal.

See an error.

System Information
https://pastebin.com/3R6wWtyX

I can't reproduce this. Check your gitconfig. Perhaps you have something like this in it:
[url<EMAIL_ADDRESS>	insteadOf = https://github.com/

Oh, sorry, here is the doom/info,
https://pastebin.com/XkbEUidu
My system install has less than 1 month, so it's pretty fresh. I don't have much incidental-configs etc.
This is my .gitconfig.
[user]
	name = BuddhiLW
	email =<EMAIL_ADDRESS>[credential]
	helper = cache

I don't see any issues with your config, but again, I am unable to reproduce this on any of my test environments, so I cannot look into this further. I'm inclined to suspect this is a local issue.
Please try my package! declaration:
(package! zh-align
  :recipe (:host github
           :repo "chen-chao/zh-align.el"))

And post the contents of ~/.emacs.d/.local/doom.error.log on pastebin afterwards (assuming it fails).

It worked with your snippet. Thanks!


I can't view build dashboard logs

I'm trying to debug https://github.com/flutter/flutter/issues/65775
I go to https://flutter-dashboard.appspot.com/#/build
I click on one of the yellow squares.
If I click OPEN LOG FOR ATTEMPT #1, it eventually says I'm not allowed to see logs.

If I click LOGIN it doesn't do anything.
I'm not really sure what I'm doing wrong? I'm pretty sure my account<EMAIL_ADDRESS>is allow-listed.

This is a problem with the authentication plugin. You'll need to click login and refresh when your name appears in the top right part of the dashboard you'll be able to download the logs.

@CaseyHillers can you please link to the auth plugin bug?

I click "SIGN IN". A new window pops up, shows a progress indicator, and then goes away. I refresh the dashboard (with or without shift). It still says "SIGN IN" in the top right, not my name.

Thanks for filing this @Hixie
I think this is similar to https://github.com/flutter/flutter/issues/59086
The Google sign in plugin has a race condition on web that causes it to crash https://github.com/flutter/flutter/issues/52338, and it was prevalent when auto sign in was on. Now the dashboard's auto sign in is disabled to mitigate the sign in crashes.
Can you share the error that is showing up in your browser's dev tools when you click the sign in button?

I don't get any errors in the console. There's a couple of warnings about cross-site cookies being blocked.

According to https://github.com/flutter/flutter/issues/59086#issuecomment-690735904 blocking third party cookies causes issues for the sign in plugin.
\cc @ditman was there a new version of the google sign in web plugin that surfaces errors like this now?

@CaseyHillers that is correct, the latest version of the plugin should properly throw exceptions when things are breaking on the JS SDK. The changes were:

almost all operations, (normally silentSignIn): may throw a PlatformException with code idpiframe_initialization_failed (see this) if the init method failed.
signIn(): may throw a PlatformException with different codes, depending on how the sign in failed. Mainly: popup_closed_by_user or access_denied (see this).

It depends on how the app decides to handle those.
@Hixie what browser, version and OS are you using? I remember seeing warnings about cross-site cookies being blocked (in the future)
Another thing, when the dashboard enters this weird state, can you please run in the JS console?
gapi.auth2.getAuthInstance()
(That should not be null.)

This is great @ditman! I'll follow up today in Cocoon to update to the latest version and catch these errors to mitigate these issues.

Apologies for the breakage, @Hixie, @CaseyHillers, and thanks for your patience! This bug is being kind of hard to track down (especially because I haven't been able to reproduce it locally so far :/)

@Hixie the dashboard should now be logging all Google Sign in exceptions. Can you log them here?

@Hixie I believe the issues you saw with Google Sign In should be fixed with the try catch.
To follow up on the other issues:
https://github.com/flutter/flutter/issues/54509 is tracking stack driver logs being viewable by the Flutter community. With the migration to LUCI, this will no longer be needed.
https://github.com/flutter/flutter/issues/59086 is tracking the build dashboard sign in issues. I'm going to increase the priority as now it's possible to catch these exceptions.
Feel free to reopen if I missed anything.

Sorry for the delay answering, the updates got buried in my inbox since I get too many notifications.
This still seems to happen as described above, and I still can't see devicelab logs.

@Hixie what browser, version and OS are you using? I remember seeing warnings about cross-site cookies being blocked (in the future)

Chrome 81.0.4044.113 (Official Build) (64-bit)
Revision | e3225dafb0475864a1812a374d73a92e391635ac-refs/branch-heads/4044@{#936}
OS | Linux
JavaScript | V8 8.1.307.28
Flash | <IP_ADDRESS>4 /home/ianh/.config/google-chrome/PepperFlash/<IP_ADDRESS>4/libpepflashplayer.so
User Agent | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36

(I don't see any exceptions.)

Full logs in the console from loading the build dashboard, clicking SIGN IN (which shows a spinner briefly):
A cookie associated with a cross-site resource at http://google.com/ was set without the `SameSite` attribute. It has been blocked, as Chrome now only delivers cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/#/build:1 A cookie associated with a cross-site resource at https://google.com/ was set without the `SameSite` attribute. It has been blocked, as Chrome now only delivers cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
canvaskit.js:311 Fetch finished loading: GET<EMAIL_ADDRESS>html_dart2js.dart:18175 XHR finished loading: GET "https://flutter-dashboard.appspot.com/assets/FontManifest.json".
html_dart2js.dart:32857 Fetch finished loading: GET "https://fonts.gstatic.com/s/roboto/v20/KFOmCnqEu92Fr1Me5WZLCzYlKw.ttf".
cb=gapi.loaded_0:228 XHR finished loading: GET "https://content.googleapis.com/discovery/v1/apis/analytics/v3/rest?fields=k…Url%2CservicePath%2Cresources%2Cparameters%2Cmethods%2CbatchPath%2Cid&pp=0".
html_dart2js.dart:32857 Fetch finished loading: GET "https://flutter-dashboard.appspot.com/assets/packages/cupertino_icons/assets/CupertinoIcons.ttf".
html_dart2js.dart:32857 Fetch finished loading: GET "https://flutter-dashboard.appspot.com/assets/fonts/MaterialIcons-Regular.otf".
js_helper.dart:1857 [Violation] 'setTimeout' handler took 100ms
js_helper.dart:1857 [Violation] 'setTimeout' handler took 121ms
/#/build:1 A cookie associated with a cross-site resource at https://accounts.google.com/ was set without the `SameSite` attribute. It has been blocked, as Chrome now only delivers cookies with cross-site requests if they are set with `SameSite=None` and `Secure`.

One last test, @Hixie, can you run the following in the JS console, when your dashboard is in the weird state?
gapi.auth2.getAuthInstance()
(That should not be null.)
Thanks!

I get back:
gapi.auth2.getAuthInstance()
Iw {fm: dx, mY: {…}, Te: ƒ, tj: ƒ, Fq: ƒ, …}
Fq: ƒ ()
LR: ƒ ()
Te: ƒ ()
YC: ƒ ()
attachClickHandler: ƒ ()
currentUser: sw {le: Cw, P1: rw, Yb: pw}
disconnect: ƒ ()
fm: dx {Ba: {…}, uS: "single_host_origin", uW: true, kv: true, Pq: null, …}
getInitialScopes: ƒ ()
grantOfflineAccess: ƒ ()
isSignedIn: sw {le: false, P1: rw, Yb: pw}
mY: {Te: ƒ, tj: ƒ, Fq: ƒ, uj: ƒ, uI: ƒ, …}
signIn: ƒ ()
signOut: ƒ ()
tj: ƒ ()
uI: ƒ ()
uj: ƒ ()
__proto__: Object

...but nothing particularly notable happens otherwise.

@ditman is there anything the dashboard app can do in this case?

Not sure, I'll take a look and see if I can repro the issue, then see where the breakage is (maybe it's some captured exception that we're not seeing anywhere)

@ditman i'm going to unassign myself for now as I'm not sure what more is needed on the cocoon side.

Moving out of the infra ticket queue as this may take some time to fix and infra depends on fixes to the plugin.

Can we remove the authentication requirement then? I have to be able to see logs to do my work, as do plenty of other non-Googlers.

See also: https://github.com/flutter/flutter/issues/49484

what needs to happen to get this closed?

what needs to happen to get this closed?

what needs to happen to get this closed?

After the DeviceLab migration to LUCI, this will be obsolete. Logs on LUCI are public. Tracked in https://github.com/flutter/flutter/projects/151#card-51747372

what needs to happen to get this closed?

After the DeviceLab migration to LUCI, this will be obsolete. Logs on LUCI are public. Tracked in https://github.com/flutter/flutter/projects/151#card-51747372

Moving as P3, this bug is waiting on getting the devicelab migration to luci to get it closed.

Moving as P3, this bug is waiting on getting the devicelab migration to luci to get it closed.


Do not remove table aliases from SQL queries

We're using dd-trace-js for postgres on a postgraphile project, which makes heavy use of aliases in its generated queries to avoid collisions.
When running a query such as
select
 to_json((__local_0__."id")) as "id",
 to_json((__local_0__."name")) as "name",
 to_json((__local_0__."description")) as "description"
from "public"."my_example_entities" as __local_0__
where (__local_0__."id" = $1)

the datadog-agent will quantize the query and remove all aliases, becoming
select to_json ( ( __local_0__. id ) ), to_json ( ( __local_0__. name ) ), to_json ( ( __local_0__. description ) ) from public . my_example_entities where ( __local_0__. id = ? )

and for some reason the as __local_0__ alias is removed from the table in the FROM clause but kept in the column selection expressions. This is pretty confusing, especially when looking into deeply queries where it is not clear which alias refers to which table.
I would suggest to either keep the alias everywhere or to remove it everywhere.
I suspect this is related to https://github.com/DataDog/datadog-trace-agent/pull/225 and https://github.com/DataDog/datadog-agent/pull/5664?

It would be great if this was at least available as an option


Upgrade aerospike dependency

Motivation
Fixes/features

LGTM ! Any reason why we didn't include this version directly when introducing the dependency ?

Didn't want to change too much initially from how it was in -extras


Fix syntax error issue while using [primary] column name in DML/DDL statements

Description
Signed-off-by: Sumit Jaiswal<EMAIL_ADDRESS>Issues Resolved
Task: BABEL-5422
Issue: https://github.com/babelfish-for-postgresql/babelfish_extensions/issues/3127
Test Scenarios Covered

Use case based - Yes

Boundary conditions - Yes

Arbitrary inputs - NA

Negative test cases - NA

Minor version upgrade tests - NA

Major version upgrade tests - NA

Performance tests - NA

Tooling impact - NA

Client tests - NA

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 74.744%

Totals

Change from base Build<PHONE_NUMBER>1:
0.0%

Covered Lines:
45720

Relevant Lines:
61169

💛  - Coveralls


Fix: Bail flag causes before() hooks to be run even after a failure

Fixes #1812
Added a bail check to the runSuite method in runner.js. There was already a commented out check for a subsequent before block in the bail fixture so I just uncommented that for the test.

Good catch :) That's been an outstanding TODO for a while now

Thanks! LGTM


Update to rails 6.0.0.rc2

FYI. Rails 6 has been released.
Mind changing this PR to test against Rails 6.0.0?


Sklearn Evaluation Metric compability

Hello,
I am training several models using Sklearn classifiers and XGBoost. I need the last version of Sklearn because I am using other package which depends on that version. I would like to include XGBoost, but I am getting the next warning:
   WARNING: /home/conda/feedstock_root/build_artifacts/xgboost_1607604574104/work/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
I am working in a conda environment with the next configuration:

active environment : DMP
active env location : /home/arritmic/anaconda3/envs/DMP
shell level : 1
user config file : /home/arritmic/.condarc
populated config files : /home/arritmic/.condarc
conda version : 4.9.2
conda-build version : 3.18.11
python version : 3.8.3.final.0
virtual packages : __cuda=11.2=0
__glibc=2.31=0
__unix=0=0
__archspec=1=x86_64
base environment : /home/arritmic/anaconda3  (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/arritmic/anaconda3/pkgs
/home/arritmic/.conda/pkgs
envs directories : /home/arritmic/anaconda3/envs
/home/arritmic/.conda/envs
platform : linux-64
user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.3 Linux/5.8.0-43-generic ubuntu/20.04.2 glibc/2.31

Some of the libraries installed and related to XGBoost:

_libgcc_mutex                0.1                      conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
_py-xgboost-mutex          2.0                      cpu_0    conda-forge
scikit-image                   0.17.2                 py36h284efc9_4    conda-forge
scikit-kinematics           0.8.5                    pypi_0    pypi
scikit-learn                    0.23.1                  pypi_0    pypi
scipy                             1.5.4                    pypi_0    pypi
xgboost                        1.1.1                     pypi_0    pypi

But when I have installed it, it shown this:
Requirement already satisfied: xgboost in /home/arritmic/anaconda3/envs/DMP/lib/python3.6/site-packages (1.3.0) Requirement already satisfied: numpy in /home/arritmic/anaconda3/envs/DMP/lib/python3.6/site-packages (from xgboost) (1.19.5) Requirement already satisfied: scipy in /home/arritmic/anaconda3/envs/DMP/lib/python3.6/site-packages (from xgboost) (1.5.4)
Thanks in advance.
A

Hi @Arritmic , it's just a warning for users updating xgboost.  You can safely ignore it or do either one of:

specify the evaluation metric you want.
provide disable_default_eval_metric as parameter.

Thannk you @trivialfis . I guessed it, and it is not disturbing finally my results, but I wanted to share it just in case it could be useful.
And thanks for the extra information which I would take into account.
:)

I'm trying to use cohen_kappa_score as the eval_metric but I'm receiving an error
xgb_clf = xgb.XGBClassifier()

eval_set = [(train_sorted_embeddings, train_sorted_labels), (validation_sorted_embeddings, validation_sorted_labels)]
xgb_clf.fit(train_sorted_embeddings, train_sorted_labels, eval_set=eval_set, eval_metric=cohen_kappa_score)

TypeError                                 Traceback (most recent call last)
 in
2
3 eval_set = [(train_sorted_embeddings, train_sorted_labels), (validation_sorted_embeddings, validation_sorted_labels)]
----> 4 xgb_clf.fit(train_sorted_embeddings, train_sorted_labels, eval_set=eval_set, eval_metric=cohen_kappa_score)
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/core.py in inner_f(*args, **kwargs)
504         for k, arg in zip(sig.parameters, args):
505             kwargs[k] = arg
--> 506         return f(**kwargs)
507
508     return inner_f
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/sklearn.py in fit(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights, callbacks)
1259             verbose_eval=verbose,
1260             xgb_model=model,
-> 1261             callbacks=callbacks,
1262         )
1263
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/training.py in train(params, dtrain, num_boost_round, evals, obj, feval, maximize, early_stopping_rounds, evals_result, verbose_eval, xgb_model, callbacks)
194                           evals_result=evals_result,
195                           maximize=maximize,
--> 196                           early_stopping_rounds=early_stopping_rounds)
197     return bst
198
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/training.py in _train_internal(params, dtrain, num_boost_round, evals, obj, feval, xgb_model, callbacks, evals_result, maximize, verbose_eval, early_stopping_rounds)
80             break
81         bst.update(dtrain, i, obj)
---> 82         if callbacks.after_iteration(bst, i, dtrain, evals):
83             break
84
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/callback.py in after_iteration(self, model, epoch, dtrain, evals)
432             for _, name in evals:
433                 assert name.find('-') == -1, 'Dataset name should not contain -'
--> 434             score = model.eval_set(evals, epoch, self.metric)
435             score = score.split()[1:]  # into datasets
436             # split up test-error:0.1234
~/environments/behold-explain/lib/python3.6/site-packages/xgboost/core.py in eval_set(self, evals, iteration, feval)
1751             for dmat, evname in evals:
1752                 feval_ret = feval(self.predict(dmat, training=False,
-> 1753                                                output_margin=True), dmat)
1754                 if isinstance(feval_ret, list):
1755                     for name, val in feval_ret:
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
61
62
---> 63 def assert_all_finite(X, allow_nan=False):
64     """Throw a ValueError if X contains NaN or infinity.
65
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/metrics/_classification.py in cohen_kappa_score(y1, y2, labels, weights, sample_weight)
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
61
62
---> 63 def assert_all_finite(X, allow_nan=False):
64     """Throw a ValueError if X contains NaN or infinity.
65
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/metrics/_classification.py in confusion_matrix(y_true, y_pred, labels, sample_weight, normalize)
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/metrics/_classification.py in _check_targets(y_true, y_pred)
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/utils/validation.py in check_consistent_length(*arrays)
314     if force_all_finite:
315         if not hasattr(spmatrix, "data"):
--> 316             warnings.warn("Can't check %s sparse matrix for nan or inf."
317                           % spmatrix.format)
318         else:
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/utils/validation.py in (.0)
314     if force_all_finite:
315         if not hasattr(spmatrix, "data"):
--> 316             warnings.warn("Can't check %s sparse matrix for nan or inf."
317                           % spmatrix.format)
318         else:
~/environments/behold-explain/lib/python3.6/site-packages/sklearn/utils/validation.py in _num_samples(x)
253         Data type of result. If None, the dtype of the input is preserved.
254
--> 255     copy : boolean
256         Whether a forced copy will be triggered. If copy=False, a copy might
257         be triggered by a conversion.
TypeError: Expected sequence or array-like, got <class 'xgboost.core.DMatrix'>

^ Turned out that was actually well documented. eval_metric has moved from .fit to the initialization.
xgb_clf = xgb.XGBClassifier(eval_metric=cohen_kappa_score)

eval_set = [(train_sorted_embeddings, train_sorted_labels), (validation_sorted_embeddings, validation_sorted_labels)]
xgb_clf.fit(train_sorted_embeddings, train_sorted_labels, eval_set=eval_set)

It was moved after 1.5 and is pending for 1.5.1 release, so I guess you are compiling from the master branch?

Moving the parameter was to have better compatibility with scikit-learn utilities.  Otherwise, grid search might split up the string "mlogloss" as input data ...


Census builder failing due to Portal REST API response

The latest main fails when requesting the Portal REST API (https://api.cellxgene.cziscience.com/curation/v1/collections) list of datasets:
2023-02-11 02:18:20 10969   INFO     Loading manifest from CELLxGENE data portal...
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/bruce/cell-census/tools/cell_census_builder/__main__.py", line 340, in <module>
    sys.exit(main())
  File "/home/bruce/cell-census/tools/cell_census_builder/__main__.py", line 83, in main
    cc = build(args, soma_path, assets_path, experiment_builders)
  File "/home/bruce/cell-census/tools/cell_census_builder/__main__.py", line 137, in build
    datasets = build_step1_get_source_assets(args, assets_path)
  File "/home/bruce/cell-census/tools/cell_census_builder/__main__.py", line 196, in build_step1_get_source_assets
    datasets = load_manifest(args.manifest)
  File "/home/bruce/cell-census/tools/cell_census_builder/manifest.py", line 145, in load_manifest
    datasets = load_manifest_from_CxG()
  File "/home/bruce/cell-census/tools/cell_census_builder/manifest.py", line 59, in load_manifest_from_CxG
    datasets = {
  File "/home/bruce/cell-census/tools/cell_census_builder/manifest.py", line 64, in <dictcomp>
    "dataset_title": dataset["title"],
KeyError: 'title'

This seems related to #126 but I'm not sure if the bug is in the builder or in the REST API (#126 suggests it is a Portal REST API bug).
CC: @brianraymor @Bento007

The dataset_title is returned by the dataset API call, not the collections API call. Turns out the builder manifest code looked for it in both places. The collection API response would never have this data, but it would later successfully be obtained from the dataset API call. We simply need to remove that line of code.


Strange Behavior

Anybody notice browsers losing the connection to webui?  I have tried on Chromium, Firefox, I even tried the official binaries for 2.4.2 and they all still lose connection.  This is on Arch and on Linux Mint, both sets of browsers on both distros (just in case it was the Arch install needing some love).
I would like to think my skills aren't that lacking, but I just don't know why it also affects the older stable build as well, so I'm wondering if something went on with the latest versions of those browsers.
Thoughts?

Mine is archlinux, using the main branch, everything is normal

That's what I would get with the minimal example using the current main branch and using 2.4.2 official archive files after it would be connected for maybe 20-30 seconds.   I have used it with zig build and regular gcc as well.

I used an application that I was working on, with known working links to open up new windows and file handling and the file handling was not displaying properly, nor would it actually open windows when buttons were pressed.  So even if I am wanting to get the server from timing out, it seems to be blocked.  Really strange.
log.txt

No.  It turns out that I had an id="webui" attribute and I changed it and boom it worked.


Dev

exclude_none and exclude_unset serialization flags support added. See https://github.com/dapper91/pydantic-xml/pull/204.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.

Project coverage is 91.83%. Comparing base (e928d4b) to head (3b81aeb).
Report is 34 commits behind head on master.

Files
Patch %
Lines

pydantic_xml/serializers/factories/model.py
90.90%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #205      +/-   ##
==========================================
- Coverage   91.86%   91.83%   -0.03%
==========================================
  Files          29       29
  Lines        1610     1617       +7
==========================================
+ Hits         1479     1485       +6
- Misses        131      132       +1

Flag
Coverage Δ

unittests
91.83% <96.15%> (-0.03%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Broken deps as Ecto was upgraded

Hi,
As Ecto was updated the module Apartmentex.MigrationsPathBuilder no longer works since build_repo_priv/1 was removed/rename/made private from the project:
defmodule Apartmentex.MigrationsPathBuilder do
  import Mix.Ecto, only: [build_repo_priv: 1]
  @migrations_folder Application.get_env(:apartmentex, :migrations_folder) || "tenant_migrations"

  def tenant_migrations_path(repo) do
    Path.join(build_repo_priv(repo), @migrations_folder)
  end
end

Any clue of the new method/fix for this?
Reference at:
https://github.com/elixir-ecto/ecto/issues/2278

@Rovel how did you fix this for you?

Hey @fr1zle
Actually the hex package / github is fixed but for some weird reason since there is no version change its doesn't upgrade, even when you tells it too. (mix deps.get or upgrade) so there is 2 options:

Get the code from github and update your apartmetex lib inside the project via code editor, since this is a small lib its easy, next time you make mix deps.get it will update properly
Point the dep to the github repo, so you will ben in sync with the latest commits from this repo

I went with the first one, but I think the second is more suitable.

@Rovel: Thanks for the super detailed answer! Totally forgot about the option 2!
Still would be great if the version in hex could be updated.

Whoops, ill get this updated tomorrow.  Thank you for mentioning it!


Mailing list

It should be possible to register for status updates sent via email, so users don't have to always check the status page. Should be fairly straightforward, unless there are some new requirements introduced by General Data Protection Regulation (GDPR) I don't know of.

I've implemented this feature in PR #37 but with telegram instead of mail. Regarding GDPR you need to secure the personal data. Also you need to have a privacy policy wich contains who you are, what data you collect and why. Here is some more information: https://ec.europa.eu/justice/smedataprotect/index_en.htm

With GDPR you also need double opt in mailing list and also it needs to be possible to remove subscription easily :slightly_smiling_face: Some kind of privacy policy page should be added either way

Correct. Should be something like

sign up for mail alerts
mail with register link is sent
user clicks on the link
subscription is registred
mails can be sent

opt out could be a link in the profile page and/or a unsubscribe link in the email to remove yourself from the DB.

I believe all of the requirements listed by @jhuesser are handled by #71


Problem installing when no GPU present (in docker build step for example)

Doesn't install during docker build phase (that does not have GPUs configured).
Get error:  /root/miniconda/lib/python3.8/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/c10/cuda/CUDAFunctions.cpp:100.)
return torch._C._cuda_getDeviceCount() > 0
If I install on the same image after running it with GPUs enabled it installs fine.

Hmm, setup.py currently checks if nvcc is installed before attempting to build the CUDA extension. Perhaps the docker image you are using has nvcc installed, but no CUDA driver, which is causing the failure. In that case maybe it should also check if torch.cuda.is_available().
I have added this to the docker-fix branch. Let me know if the following works and I'll get it merged asap.
pip install git+https://github.com/teddykoker/torchsort.git@docker-fix

Thanks for the quick fix. Confirming indeed that this fis works.
And yes I have nvcc installed but no driver (driver is not in the docker container, but part of the host).

Great! I'll put this in 0.1.1 now :)

Sorry but I replied too quickly.
With the fix, it indeed installs and I even tried the simple CPU example successfully. However running a GPU example from the main page:
x = torch.tensor([[8., 0., 5., 3., 2., 1., 6., 7., 9.]], requires_grad=True).cuda()
y = torchsort.soft_sort(x)
fails with:
~/miniconda/lib/python3.8/site-packages/torchsort/ops.py in forward(ctx, tensor, regularization, regularization_strength)
126         # note reverse order of args
127         if ctx.regularization == "l2":
--> 128             sol = isotonic_l2[s.device.type](w - s)
129         else:
130             sol = isotonic_kl[s.device.type](w, s)
TypeError: 'NoneType' object is not callable

I think this is related to this issue. It will skip building the CUDA extension if cuda is not available, so it will need to be installed while cuda is available.
I think you should be able to solve this by exporting environmental variable TORCH_CUDA_ARCH_LIST=All, which will build the extension for all of the architectures (this may result in a large binary file size but should work). This of course won't work with the most recent version (0.1.1), so perhaps try this with 0.1.0.
Let me know if this works and I can add a note to the readme. Additionally, if you could share a Dockerfile and steps to reproduce that would be very helpful!

Here is relatively small dockerfile that reproduces it.

Ah my bad,  TORCH_CUDA_ARCH_LIST="Pascal;Volta;Turing" is what you need. You should be able to use just one of those if you know what architecture your GPU is.

Thanks, I can verify that that works with 'Ampere' added for my system.


New tag

Things that depend on this such as gophercloud do not compile against 0.0.0 tag. They do compile against current sources. It would be nice if there could be a new tag 0.1.0 maybe. That way we can specify a version rather than having to pull sources. Its better for packaging, such as gentoo ebuilds, where I presently have to use the 9999/live source version instead of a tagged version. Much rather use a tagged version, so I can depend on that rather than 9999 ebuild. That you have to re-emerge to re-fetch current source. Good for development, not good for a package that others will use. Thanks!
https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-go/gophercloud/gophercloud-1.0.0.ebuild

It is probably too late, but I've just cut a v0.1.0 tag for HEAD. Sorry this ticket got lost in the shuffle, @wltjr.
https://github.com/racker/perigee/releases/tag/v0.1.0

Oh no problem, thanks for the tag. 👍  I was closing more because some do not like to version Go projects because Go is anti versioning.

I feel bad; I had no idea this issue was open, and to think it's been open for almost a year!  Thanks @mburns for cutting the tag.  Notification fail.  :(

No need to feel bad, it was VERY minor. Plus I opened another similar request after this one and was denied mitchellh/mapstructure/issues/39. These are just dependencies of Rackspace rack command, which I was packaging for Gentoo. ebuild. The slots were so I could pull in the right stuff and have it build. I had some issues where one required a slotted version and others live or vice versa. But I haven't been using rack, so its pretty moot. I have other things ingored in Gentoo for much longer that are actual fixes and other things. This was just over tag, but clearly you all are on it normally thus taking it to heart. Its all good!


chore(chore): add github stale bot

Added stale.yml file to configure stale options
GH-55
Description
Adding stale bot app and yml file to configure issue and PR stale
Issue - 3 months
PR - 2 months
Fixes #55
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Intermediate change (work in progress)

Checklist:

[x] Performed a self-review of my own code
[x] npm test passes on your machine
[ ] New tests added or existing tests modified to cover all changes
[ ] Code conforms with the style guide
[ ] API Documentation in code was updated
[ ] Any dependent changes have been merged and published in downstream modules

:tada: This PR is included in version 4.2.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Update README.md

This is great!
Made some small changes and moved it after the community section.

Thanks, much better!
It's also really easy to edit on the PR "files" section, but probably easiest to see it f2(v)f (face to (virtual) face).  Let's do it sometime and shorten some rounds.

I tried! This is all new to me :)
Will get it right next time - and will maybe find some really easy non
technical issues at hacktoberfest to practice on.
Need more t-shirts.
On Mon, Sep 21, 2020 at 3:23 PM arielshaqed<EMAIL_ADDRESS>wrote:

Thanks, much better!
It's also really easy to edit on the PR "files" section, but probably
easiest to see it f2(v)f (face to (virtual) face). Let's do it sometime and
shorten some rounds.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/treeverse/lakeFS/pull/630#issuecomment-696149557, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AQBF5L24FKCELDARMCZK6HDSG5OWJANCNFSM4RUSXBMQ
.