Having issues with gk-deploy script

Hi I am running kubernetes 1.8.5 on centos7.
I tried the script, but I get the glusterd not running. I seem to be having issues enabling the service, I checked some of the other issues on here, noticed that some other people were having similar problems so I went to abort my gk-deploy and the gluster pods did not delete.
Question 1
Is that an issue or can I keep trying to now re run the gk-deploy script even though my gluster pods are still up?
If it is an issue how can I delete my gluster deamonset without causing and possible kubernetes caching problems?
Question 2
Why is it complaining about glusterd service not being started? Is that why I read that you needed to get rid of everything yum installed except for the gluster client?
Question 3
@jarrpa halp me. I see you answer a lot of these questions, getting exact error logs is tough because I am in an offline environment, but I will transcribe the important looking parts

3rd one is admittedly not a question.

Hey there! Sorry to hear about your troubles. Let's see...

This is definitely an issue. While the abort function is somewhat incomplete, it should be getting rid of the GlusterFS Pods and DaemonSet. Just do a kubectl delete ds on the DaemonSet, that should also remove the pods. There shouldn't be any Kube caching problems. Be sure to delete any remaining GlusterFS configuration directories (e.g. /var/lib/glusterfs) on the host. You can get a list of these directories from the hostPath mounts in the GlusterFS DaemonSet template.
Could be any number of things. To start, make sure you have the necessary firewall ports open, the required kernel modules loaded, and that the glusterfs-server RPM is not installed.
Still a valid enough point. :) Can you take screenshots or photos of a terminal and post those? If not we'll have to make do.

Try a few of my recommendations above. If nothing works, then I'm going to want to look at:

The output of gk-deploy with -v around when it fails
Any relevant events in the output of kubectl describe po on one of the GlusterFS pods
Any relevant-looking error (marked E) lines from the tail-end of /var/log/glusterfs/glusterd.log on any of the nodes running GlusterFS pods.

How can I check that the correct kernels modules exist?

I will not be able to take screenshots, but I can do my best to get you the most info that I can.

I found the host path mounts, so I will clear those out. For some weird reason the script deployed two identical deamonsets deployments to kubernetes. It still only deployed a single set of pods, but that may have been why the script failed to take down the Deamonset on --abort.

If you ran the script more than once that would account for why there are more than one DaemonSets... that's the only thing I can think of, anyway. But the presence of another set of GlusterFS pods would account for some of these issues. Make sure the DaemonSets are removed and any associated node labels are also removed.
For kernel modules, lsmod | grep <name> will show you if a given module is present, and modprobe <name> will load a given module. You'll want to create a file in /etc/modprobe.d/ to load those modules at startup would also be a good idea.

Sorry I am looking at the setup guide and I don't see the modules that I need there?
For the host paths, should I delete the folder as well. For example should I remove the whole heketi folder?
I noticed in /run/lvm/ there are two files in there, I went to look at them and I got permission denied even as root. Is that lvm folder created by the gluster deployment (and therefor I can delete) or could it be created by something else and I shouldn't just recursively delete that folder?
In the folder I see a lvmetad.socket and a lvmpolld.socket file

NOnono, do not delete any system directories! Only delete the ones specific to GlusterFS and heketi (they should be obvious by name). And yes, you can remove the directories themselves.
If you don't skip it, the intro text to gk-deploy tells you what you need to have available: https://github.com/gluster/gluster-kubernetes/blob/master/deploy/gk-deploy#L528-L560

I already did... just kidding ( :
So I plan to delete these folders:
/var/lib/heketi
/etc/glusterfs
/var/log/glusterfs (though I cant see why logs would matter)
/var/lib/glusterd
/var/lib/misc/glusterfsd
and not delete:
/run/lvm
/dev (obviously)
/sys/fs/cgroup
/etc/ssl
Ah I see in the gk-ddeploy skip the kernel modules. I can look at those next. Let me know if proposed delete/donotdelete list looks reasonable. Thanks!

A reasonable list. Go ahead!

Okay I got my modules loaded up, I had seen that before but for some reason choose not to explore it more. Those are all loaded up now.
how can I check that mount.glusterfs is available?

Can I just sudo yum install the glusterfs-fuse.x86_64 package on centos7 and assume it is there?

list of packages proposed to yum install on worker nodes:
centos-release-gluster313 (or 312)
glusterfs-cli.x86_64
glusterfs-fuse.x86_64

I decided for now I would only get the cli and fuse packages I listed above and give it a try.
The deployment was not successful
Error:
unable to create new node: New Node doesn't have glusterd running
How do I turn on glusterd?
I saw that you can enable glusterd.service , but when I tried that I got no such file or directory.

If you're running GlusterFS in pods, heketi is looking for clusterd in the contaiers and there must be NO glusterd present on the host. You do not "turn on" glusterd in this case, it should be running when the containers come up. The error you're getting from heketi could mean a number of things, but fundamentally it means it can't talk to a glusterd process where it expects it should. Check the outputs of kubectl logs <heketi_pod> and kubectl describe po <gluster_pod>. If nothing shows there, check the GlusterFS logs on the hosts as I mentioned above.
glusterfs-fuse is the only RPM you need to install to mount GlusterFS volumes.

Okay got it, I will remove the gluster-cli from the nodes then

By the way, the gk-deploy is still creating 2 deamonset deployments in kubernetes, but only one set of pods (so just when I go to tear it down I have to delete the deamonset manually and clear out the host file folders, which isn't a huge deal)
So things running:
the three modules
gluster-fuse.x86_64
Still getting the unable to create node from heketi, so I'll grab the logs and transcribe them.
kubectl logs heketipod:
Error  /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:301: Get https://<IP_ADDRESS>:443/api/v1/namespace/default/pods?labelSelector=glusterfs-node: dial tcp <IP_ADDRESS>:443 i/o timeout
And then goes on to say that it fails to get list of pods
kubectl describe po gluster pod:
I don't know what you would be interested in here?
I see a Tolerations section:
node.aplha.kubernetes.io/notReady:NoExecute
node.aplha.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule

When I describe the gluster pod it says that it is initialized, ready and scheduled as True

From the heketi logs I also see:
Started GET /clusters/
Completed 200 OK in
Started delete /clusters/

ERROR  /src/github.com/heketi/heketi/apps/glusterfs/allocator_simple.go: Unknown cluster id requested

Please don't spam multiple messages if you can avoid it.
On the describe I'm looking for Events towards the bottom to see if any of them indicate an error.
Are you sure you have no GlusterFS DaemonSets or Pods running after you clean up a failed deployment? Are you using something other than the default namespace and could there be something conflicting in another namespace?
You never answered an earlier question: do you have the necessary firewall ports open on all GlusterFS nodes?

Please don't spam multiple messages if you can avoid it.
Sure not a problem

For firewall ports I have a security group set up that allows:
TCP over ports 2222, 24007, 24008, 49149-49251
So that shouldn't be an issue. Also I think with our setup if nodes are in the same region then communication within the region should not be restricted... I just set up the security group to be extra safe that ports would not be an issue.
I am sure that I clean up all the pods, first I delete the deamonset with:
kubectl delete ds/glusterfs
by the way on that deamonset note, its still doing the weird thing where it creates two deamonsets but deploys one set of pods. I have seen that before on other issues, but I don't know if it was ever resolved.
Then I do a kubectl get all to make sure nothing is there and I can see only the svc/kubernetes service running.
I went to the host machine and noticed that when I did a sudo yum list installed | grep gluster
that a few packages were installed in addition to fuse as dependecies:
glusterfs.x86_64                                                             3.8.4-18.4.e17.centos    @os_centos_base
glusterfs-client-xlators.x86_64                                       3.8.4-18.4.e17.centos    @os_centos_base
the one I wanted-> glusterfs-fuse.x86_64                     3.8.4-18.4.e17.centos    @os_centos_base
glusterfs-libs.x86_64                                                      3.8.4-18.4.e17.centos    @os_centos_base
Is this an issue?

Hmm... yes, the networkign you described should be working.
I don't recall other issues with multiple DaemonSets... what are the names of the DaemonSets and are they in the same Namespace?
"all" may not get everything we're interested in. Try kubectl get deployment,ds,po,job,secret,ep,svc,sa --all-namespaces and see if there's anything with the words "glusterfs" or "heketi" in it.
Those packages are the correct ones. The reason I only specify glusterfs-fuse is that it takes care of installing all needed dependencies.

Hmmmm insteresting, so after the abort I did a kubectl get all (in just my namespace) and I see the deamonset twice, however when I do a kubectl get ds (Just in my namespace at first) I only see the one deamonset. So this may be an issue with the kubectl get all. As a result, I think that can be ignored for now.
kubectl get deployment,ds,po,job,secret,ep,svc,sa --all-namespaces | grep gluster
and only saw the pods and the daemonset deployment running, then I deleted the deamonset with
kubectl delete ds glusterfs and that deleted out just fine
kubectl get deployment,ds,po,job,secret,ep,svc,sa --all-namespaces | grep gluster and found nothing
kubectl get deployment,ds,po,job,secret,ep,svc,sa --all-namespaces | grep heketi also found nothing
Other things I can check? Oh by the way the names of the deamonsets when I had two of them (but again only one set of pods) was ds/glusterfs and ds/glusterfs, so they were named the same.

OKay yeah something is screwy with your cluster. You can't have two resources of the same name and type in the same namespace.
The command you listed is definitely fine.
If you're at a point where the heketi and GlusterFS pods are up and running but heketi can't communicate to GlusterFS, check that all pods are showing 1/1 Ready. Next try going into a GlusterFS container and running systemctl status glusterd to see if the process is running. If all this is fine, you may have something weird in your networking. Note that the GlusterFS pods use host networking, meaning that they assume the network identity of the hosts they're running on. You would have to verify that processes in containers in your cluster can find routes to the host IP addresses.

glusterd is running in the container
so that is good.
I ran hostname -i from inside the container and that worked and I got the correct IP back from the command.
Any test commands I can run from inside the gluster container that will let me know "yes the cluster is connected"? I did a gluster peer probe to one of the other containers and that worked. Then I did a gluster peer status and saw there was only one peer (the one I had just connected to) so it seems like that isn't quite working correctly?
I went into the heketi container as well and did a hostname -i and that returned command not found. What can I do from that container to see if things are working correctly or not as I debug more?

I have a 5 node cluster though.
What I am saying is that the gluster pods are not connecting to each other via the script because when I go into a gluster container origonally the container has 0 peers, however when I manually add a peer in the container that works and it connects to a different host ip.
busybox commands and values:
hostname = busybox-
hostname -i = <IP_ADDRESS> (this is an ip of the container)
ping <IP_ADDRESS> (one of the kubernetes workers) = no packets transmitted or returned from busybox container
ping <IP_ADDRESS> (kubernetes service ip for heketi) = nothing
ping <IP_ADDRESS>:8080 (same ip but now over port 8080) = nothing

So it looks like my busybox can't ping anything but my gluster container can ping anyt machine in the cluster

I was looking at the gluster yaml file and noticed that there are not any ports specified on there? Is that expected? How does communication happen if there are no ports?

Update, mostly so people who come after me can follow some sort of documentation
I was able to curl heketi and get heketi to say hello to me from inside the heketi pod going through the kubernetes heketi deployment service ip. the command is curl <service_ip for heketi-deploy>:8080/hello
I was also able to curl hello heketi from a single gluster container that is on the same node. So there are some networking issues going on.
deploy-heketi's ip is <IP_ADDRESS> with port 8080 open
I think if port 8080 was not open I wouldn't be able to do this... the question is, why does this service ip only work on the single node...

A few things:

When you're in a container that is not using host networking, you want to verify two things:

Can I access other pods via their service IPs? This is typically done via curl or telnet. This will tell you if cluster networking is functional.
Can I access Kubernetes nodes via their host IPs? ping will suffice here. This will tell you if the cluster network has access to the network of the hosts.

Since GlusterFS runs with host networking, both of these must be true in order for it to function. The GlusterFS pods will all automatically be able to communicate with each other because they are on the same host network. The heketi pod is in the cluster network, and thus will need a route to access the host network.

The script is failing because it is trying to communicate to the GlusterFS pods from the heketi pod.
Since heketi is not using host networking, port 8080 does not have to be open on the host. Indeed, there is nothing needed in Kubernetes for a service to be able to receive traffic on any arbitrary port.
Specifying network ports for a pod is not required, and in fact offers no real utility other than letting someone reading the YAML know what ports may be in use. Kubernetes does no checking or enforcement of network port usage.
Given the way Kubernetes handles routing tables on the nodes, it makes sense that a pod on a given node would be able to ping the IP of that node.
Documentation for interacting with heketi can be found at the project's Github repo. For this specific case, see here: https://github.com/heketi/heketi/blob/master/doc/admin/server.md#testing-the-configuration

@jarrpa I'm having a similar issue to mjschmidt. You mentioned making a file in /etc/modprobe.d/ if I need to use modprobe <name> to load a module at start up; how do set that file up? Is there a example somewhere I can reference?

@jarrpa Next question, which yaml file sets up a pod that gets labelled "heketi-storage-copy-job"?

An update I am still working on this. I have learned more about iptables and networking than I anticipated. For anyone interested in a brief overview of IP tables role in Kubernetes the general gist is:
Kubernetes does a ton with IPtables...

By the way it looks like it was issues with the way I was deploying my kubernetes cluster and flannel or docker networking issues. So the next question is now that my heketi has hooked up with my gluster containers I see it is trying to do the setup-openshift-heketi-storage.
I am not running on Openshift and don't need this. I am running on centos7 machines. How can I exclude openshift stuff from my deployment?

On Fedora/CentOS/RHEL systems you can drop an executabl file under
/etc/sysconfig/modules/ . An example would be:
#!/bin/sh
#
# Force loading some modules

grep -q -w dm_thin_pool /proc/modules || modprobe dm_thin_pool

Save that file as /etc/sysconfig/modules/gluster-bricks.modules and make
it executable. Upon a reboot the script should get executed and the
modules get loaded.
The configuration files in /etc/modprobe.d/ can be used to modify the
parameters while the module gets loaded. I dont think there is a way to
mark modules to get loaded automatically there.

@nixpanic  So I am not very familiar with the modules, or etc/sysconfig.
Does this look like it would be right?
These are the modules I need:
dm_snapshot
dm_mirror
dm_thin_pool
Saving a file here /etc/sysconfig/modules/gluster-bricks.modules that contains:
#!/bin/sh
#
# Force loading some modules

grep -q -w dm_thin_pool /proc/modules || modprobe dm_thin_pool
grep -q -w dm_thin_pool /proc/modules || modprobe dm_mirror
grep -q -w dm_thin_pool /proc/modules || modprobe dm_snapshot

^End file
Then I will do a chmod +x /etc/sysconfig/modules/gluster-bricks.modules
Then reboot all the machines that I do this to (in this case the 3 kubernetes nodes that I plan to get this gluster running on).
Then run gk-deploy and profit?

@mjschmidt yes, that looks correct. But you need to update the grep commands in the script ;-)
But, maybe there is a better way. Sometimes kernels compile certain modules in their main executable, and do not offer to load them. In that case, /proc/modules will not list them either. To make it less dependent on the kernel/distribution, I should have suggested:
#!/bin/sh
#
# Force loading some modules

[ -d /sys/module/dm_thin_pool ] || modprobe dm_thin_pool
[ -d /sys/module/dm_mirror ] || modprobe dm_mirror
[ -d /sys/module/dm_snapshot ] || modprobe dm_snapshot

Try the script+reboot on one of your nodes, and verify that the modules are loaded (check in /proc/module or use lsmod).

I will use the lsmod and grep for the desired modules ( : I'll get back to you in a bit.

Okay cool it looks like that works ( : and that wasn't too bad, but what does that have to do with the heketi openshift stuff I was referring to earlier?
I don't think I am understanding that yet.

The setup-openshift-heketi-storage command executed by Heketi will create a Gluster Volume to store the heketi database on. Creating the LVM/LVs can fail if the dm_* modules are not loaded. These should get loaded automatically, but that only works if the container can load them (which might not be the case because of several reasons).

@mjschmidt @nixpanic I'll note that the setup-openshift-heketi-storage command is mislabeled. It applied to both Kubernetes and OpenShift. It is currently needed by the configuration used in gk-deploy, as it allows for robust persistent storage for the heketi database.
Supposedly there is a way that you can just specify module names in a flat file so that they get loaded on system startup. See here: https://github.com/mjudeikis/openshift-ansible/blob/7d67afffe217e29c7bb8941343bd017907ffc9df/roles/openshift_storage_glusterfs/tasks/kernel_modules.yml and https://github.com/mjudeikis/openshift-ansible/blob/7d67afffe217e29c7bb8941343bd017907ffc9df/roles/openshift_storage_glusterfs/templates/glusterfs.conf.j2
The format is just the names of modules, one per line.

@mjschmidt Yes, the setup-openshift-heketi-storage command by default will always create a YAML manifest for the copy job that specifies heketi/heketi:dev as its image. You can change that by using the --image option into the command. So you have two options:

Edit your gk-deploy script to add --image <image_name> here: https://github.com/gluster/gluster-kubernetes/blob/master/deploy/gk-deploy#L859
Pull the heketi/heketi:dev image or tag your existing image with that name.

Hmmmm okay, I am going to try that and if it works tomorrow. Thank you.

@jarrpa @nixpanic WHOOP
So good news after the hard code into the open-shift command (a change I very much want to open a new issue for/help out with when this is all said and done)  and countless cluster deletes and re-provisions, I was able to have the gk-deploy script run successfully.
NOW how do I use this storage?
Upon completion the script says "heketi is now running and accessible via http://<IP_here>:8080 to run admin command you can install heketi-cli and use it as follows:
etc
etc
etc
I am trying to have the storage I just set up serve the kubernetes cluster it is currently running on. Is that where that yaml file comes into play? So Throw the following into a yaml file then kubectl apply -f whatever_I_call_this file.yaml:

apiversion: storage.k8s.io/v1beta
kind: StorageClass
metadata:
name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http:<IP_HERE>:8080"
By the way is this IP generated or statically written and I have to fill in my IP? It looks like the later but I want to be sure.

Never mind I just had to rename the volume mount point to the storage class I made. WOOOOOOO IT WORKS! 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉
I'll have to find some time to create a troubleshooting page this weekend for Kubernetes deployment
So can I do the heketi deployment and scale that out if I need to and have requests come into the hekeiti in say a round robin fashion? Or would that start to break things?

Not to spam the chat, but how do I expand out to more nodes?

WOO!! Great to hear! :D
All right, so, at this point you don't want to scale the current heketi deployment. heketi expects exclusive access to its metadata DB. But there are several ways you can expand this setup:

Add more nodes to the current clusters. This can be done via the heketi-cli by executing heketi-cli node add with the node info, then add the devices on that node.
Add more clusters to the current heketi instance. heketi can manage multiple clusters from the same instance. I believe this is done by specifying a new clusterID in the node add command. By default heketi will just use that additional cluster in its storage allocation algorithm, but you can now also specify one or more clusterIDs in your StorageClasses to direct volume requests to specific clusters.
You can spin up a new heketi instance, either in a different namespace (easiest) or with a different name. You would also need a new GlusterFS DaemonSet. It would basically be running the gk-deploy process again just with different names.

So adding gluster nodes to the deamon set is a manual process? I know if I label a node with the gluster tag it will spin up another gluster pod, but I guess there isn't anything that will trigger heketi to work with that new gluster pod?

Correct. We are discussing ways to do that, but it's still some time out.

Okay can you give an example of the heketi node add command?
Is there a way to hook my terminal to the heketi container if I install the heketi-cli on it, or do I have to bash into the container in order to do these commands?

You need to open a shell in the container to use the built-in heketi-cli, yes. You can also install heketi-cli on any machine that can send http requests to the heketi service. So if curl://<heketi_url>/hello works you can use heketi-cli from there. Regardless, you can run heketi-cli -s http://localhost:8080 --user admin [--key <SECRET>] node add --help to get the usage message. Note that the SECRET is only required if you configured one.

Okay so I have the issue where a node contains devices, but the node is dead so I can't delete if from the heketi-cli but I also cant clean up any of my pvs that are deleted because one of the peers are missing.
How do I deal with this?

When I try to disable and remove the node that fails because now the node is dead/gone

@nixpanic any idea? Any idea of who I can ask?

last try (and tomorrow morn) to reach @jarrpa or @nixpanic or anyone else before I try opening a new ticket (this could potentially be a new ticket anyway).
I have glusterfs pretty much up and running great, and I have learned a lot about trouble shooting kubernetes nodes to bring them back to life. However, I want to know how deal with node death in this setup because heketi doesn't want to remove nodes that it thinks has bricks on it, but kubernetes doesn't want to clean up pvcs unless the whole gluster cluster is present, in turn causing the k8s cluster to run out of storage. What do I do if I am running gluster in k8s and a node literally blows up and is disconnected from the cluster with no hope of ever coming back. Lets also say in this hypothetical instance that I have 12 gluster nodes so I would still be above the 3 min requirement.

Looks like this conversation continued in https://github.com/heketi/heketi/issues/1120.

It is I am still waiting on an answer from them on some stuff, is there a command I can run on gluster itself to see if it is out of space or not? My total pvcs do not take up all the space I provided, so I think it may be an issue with stuff not being cleaned up correctly?)

From gluster, no. From the GlusterFS pods you can run standard LVM command to inspect the configuration and usage. Ones to be likely be most relevant are pvs, vgs, pvdisplay, and vgdisplay.

I did the lvm vgdisplay on the gluster node and I see what I think is the relevant volume (~100Gig)
I see
VG size 99.87 gib
PE size 4.0 mib
total PE 225567
alloc pe / siz 23437 / 91.55 gib
Free PE 2130 / 8.32 gib
VG UUID AcF6yadayadayada

Okay something between heketi and gluster isn't working right. Heketi I show my 14 volumes but when I look at the available space on the node I see no free space. This happens when ever I have gluster and heketi up for an extended period of time making believe that heketi isn't actually cleaning up the space it says it is. How do I ask gluster these questions, for example. what are all the volume id's you have? Because then I could compare volume id's in gluster with volume ids in heketi and see which volumes in gluster need destroying.

You may be hitting another heketi issue. You want to compare against the output of gluster volume list. gluster volume status will also tell you the names of each volume's bricks.

great thanks @jarrpa

I NOW KNOW WHAT THE PROBLEM IS! Wow that was a long road. Long story short this is a bug between gluster and heketi running in containers. My second pvc claim was for 1 brick with 3 replication according to heketi-cli, but somewhere in translation gluster is creating 8 bricks x 3 replication for a total of 24 bricks. At 11 gib each that is what is eating all my space.
Here is the long story if interested:
Okay so I found something very interesting that probably explains my space problems...
So I started out by checking out the gluster list and then checking out the volume info on each other the volumes. I found one that was creating 24 bricks! This pvc happened to be 11 gib each in size. That would explain my space problem. It also sees a second set pvc associated with the same stateful set deployment, but this set of pvcs do not have any problems associated with them so we ignore them for now and just know that I should be taking up and regaining a total of (11 gib for first claim per container * 3 for replication * 3 containers each making a pvc claim + 6 gib for first claim per container * 3 for replication * 3 containers each making a pvc claim = 153 total gib for the stateful set) Since in this stateful set deployment only had two containers deploy/attach pvcs I only had 101 gib that should be returning to a heketi that believes it is full.
So there is a disconnect between heketi and gluster somewhere. When I look at the volume in heketi using heketi-cli volume info  here are the important rows:
Size: 11
Mount: :vol_<vol_blah851b>
Durability type: replicate
Distributed+Replica: 3
When I bash into a gluster container and do gluster volume info <vol_blah851b> I see:
Volume ID: Distributed-Replicate
Volume ID: 6ba8blah-blah-blah-blah12aab7a3
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x 3 = 24
This is where I believe the problem lies, a replica pod that has an identical pvc claim is only getting 1 x 3
I cleared out those pvcs from kubernetes to see what would happen and to make sure my stuff would get cleaned up.
Here is what I found:
I take down the statefulset that had the associated pvc claims
After the stateful set is gone I clean up the PVCs with kubectl delete pvc <pvc 1> <pvc 2> <pvc 3>
I check the heketi-cli to make sure the 11 gib volumes I expect to be deleted are deleted:
I check gluster to make sure that the associated volumes are deleted and they were in fact deleted
I then went back to heketi-cli and did some volume node info to make sure I gained back my space, but since there is a disconnect between what gluster thinks is happening and what heketi thinks is happening I only have 108 gib of space total even though I should have gotten back 264 gigs of space for the one volume that . Heketi believes that when it cleared out the pvcs it regained

https://github.com/heketi/heketi/issues/1127
also posted here

....dang. That's really weird, I've never heard of heketi creating an 8x3 volume from a dynamic provisioning request. Are you able to reproduce this behavior consistently?
We have also been aware of heketi's tendency to get out of sync with the state of Gluster underneath it. The most recent release of heketi does a lot to address these issue. Check if you're running heketi v6.

I was not before, I just switched to heketi:6 today on a seperate cluster for testing, but now I am having deploy issues I am working through for that. I will update on that front when I figure out the nitty gritty details. My question on that front is, can I run heketi 6 with older gluster containers (what ever the latest was when we first started this thread and weren't best friends)
Yes the heketi database seems to be getting out of sync with gluster underneath I saw the stuff about heketi 6. I really hope that fixes things
In general, am I able to resync the database with gluster? How do I do that?

You should be able to run heketi:6 with older GlusterFS images, yes. And yes, it is possible with some new DB management tools to tyr and repair an out-of-sync DB. Ask in the heketi project for more info on that.

I am putting this at the top, but its really the last thing you should read:
OH MY GOD I DID ALL THIS TRANSCRIBING TO FIND OUT GLUSTERFS DELETED A VOLUME AND HEKETI DIDN'T KEEP TRACK (I did heketi-cli volume list and got that the volume was not cleaned up then did a gluster volume list and saw gluster did in fact clean it up... fuq...)
Okay I deployed heketi:6 to our old cluster, the new cluster I was setting up was being done with puppet and there was no way to see if it was a puppet issue or a heketi issue. I have confirmed it is a puppet issue, so lets ignore that for now and we are staying consistent by staying with the old cluster. I deployed heketi:6 successfully with "old" (its not that old) gluster and successfully created a dynamic pvc claim.
However now when I delete the pvc claim the pv is not cleaning up. I bashed into the hekti container and
Please keep in mind I have to hand transcribe these logs, so I am getting the relevant stuff. and ignoring stuff like ids and times
I see the following relevant information from the logs:
[negroni] Started DELETE /volumes/4cdaa...
[heketi] INFO 2018/04/25 Loaded simple allocator
[negroni] Completed 202 Accepted in 23.03ms
[asynchttp] Info  Start job d6444r8...
[heketi] INFO 2018/04/25 Started asymc operation: Delete Volume
[negroni] Started GET /queue/d6444r8...
[negroni] Completed 200 OK in 85.9
[kubeexec] DEBUG 2018/04/25 /src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: glusterfs-vnzql --mode=script snapshot list vol_4cdaa... --xml
Result:

0
0

0

[kubeexec] DEBUG 2018/04/25 /src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: glusterfs-vnzql --Command lvs --options=lv_name, thin_count --separator=:
Result:      LV:#Thins
vol_data:
brick_552c...:
tp_552c...:1
lv_home:
lv_home:
lv_root:
lv_swap:
lv_temp:
lv_usr:
lv_usr:
lv_var:
lv_var:
[kubeexec] DEBUG 2018/04/25 /src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker5 Pod: glusterfs-2g16b Command: lvs --options=lv_name, thin_count --separator=:
Result:      LV:#Thins
vol_data:
brick_b9c6...:
tp_b9c6..:1
lv_home:
lv_home:
lv_root:
lv_swap:
lv_temp:
lv_usr:
lv_usr:
lv_var:
lv_var:
[kubeexec] DEBUG 2018/04/25 /src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker4 Pod: glusterfs-pg6ct Command: lvs --options=lv_name, thin_count --separator=:
Result:      LV:#Thins
vol_data:
brick_d0d4...:
tp_d0d4..:1
lv_home:
lv_home:
lv_root:
lv_swap:
lv_temp:
lv_usr:
lv_usr:
lv_var:
lv_var:
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK in 181ms
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK in 73ms
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK in 73ms
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK in 73ms
[kubeexec] DEBUG 2018/04/25 /src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host:
worker7 Pod: glusterfs-vnzql  Command: gluster --mode=script volume stop vol_4cdaa... force
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK in 73ms
Result volume delete: vol_4cdaa...: success
[negroni] Started GET /queue/d644...
[negroni] Completed 200 OK
[kubeexec] DEBUG src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: glusterfs-vnzql Command: umount /var/lib/heketi/mounts/vg_7a90.../brick_d0d4...
Result:
[kubeexec] DEBUG src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker5 Pod: glusterfs-vnzql Command: umount /var/lib/heketi/mounts/vg_548f.../brick_552c...
Result:
[kubeexec] DEBUG src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker4 Pod: glusterfs-vnzql Command: umount /var/lib/heketi/mounts/vg_357e.../brick_b9c6...

I am as a result unable to delete my volume because it is still in use by another device.
A lvc --segments reveals
LV                                               VG                                       Attr                   #str Type           ssize
vol_data                                    vg01                                    -wi-ao----             1 liniar              297g
brick_39                             vg_ee..

Okay I already deleted my volumes but if I encounter this again its good to know how to handle it (:

@jarrpa have you seen this issue with heketi:6 container? From the logs it looks like heketi is starting to do all the right things then it gets stuck then never retries to continue onward? Thoughts? I don't see any gluster issues from the heketi logs.

I'm not sure, I don't track exact heketi versions as well as I should. I know in the 6 there was a lot of work put into improving stability especially with regards to volume creation and deletion. Is the current issue that you can't get the volumes to delete? Is this via kubectl, heketi-cli, or both?

Do you guys run containerized gluster and heketi? Or is this something that is not really being done?
The problem is that it seems like heketi is getting stuck provisioning volumes in the middle of creation then freezing up.
I see this in the heketi logs
[kubeexec] DEBUG src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: glusterfs-vnzql Command: lvcreate --poolmetadatasize 16384K -c 256K -L 3145728K -T vg_548f.../tp8f08... -V 31457287K -n brick_8f08...
Result: Using defult stripesize 64.00 KiB
Thin pull volume with chunk size 256 Kib can address at most 63.25 Tib of data
Logical volume "brick_8f08..." created
I am tired of typing, but the point is that it gets to this point
[kubeexec] src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: gluster-vnzql Command: mount -o rw,inode64,nouuid /dev/mapper/vg_548f...
Result:
[kubeexec] src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: gluster-vnzql Command: mkdir /var/lib/heketi/mounts/vg_548f/brick_8f08
Result:
[kubeexec] src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: gluster-vnzql Command: chown :2001 /var/lib/heketi/mounts/vg_548f/brick_8f08
[kubeexec] src/github.com/heketi/heketi/executors/kubeexec/kubexec.go:244: Host: worker7 Pod: gluster-vnzql Command: chmod 2775 /var/lib/heketi/mounts/vg_548f/brick_8f08

And I would expect to see these steps next:
[cmdexe] INFO Creating volume vol_<id> replica 3
[kubeexec] Debug <statement with the stuff that does the gluster command> to "gluster --mode=script volume create vol_<id> replica 3 <hosts of bricks it is mounting>
Result: volume create: vol_<id> success: please start volume to access the data"
[kubeexec] <debug statement to the host that has the started volume on it> gluster --mode=script volume start vol_<id>
Result: volume start: vol_<id>: success
[asynchttp] INFO <stuff here> Completed job <job_id> in ~7 seconds

But they are never happening and heketi just stops doing stuff, no error messages to be seen, no retries but it continues to heartbeat the volumes it has already created.

As of version 6 Heketi generally operates in the following manner:

The request is processed, the needed elements (devices, etc) are determined, and that is all logged in the db
Heketi creates (or deletes, etc) the items from the gluster nodes.
If no errors occurred the db entries are marked as no longer pending, otherwise attempts are made to undo the operation

Because of this there are a few common ways it can get out sync with the underlying systems.

Heketi process is terminated while it is performing an operation. This can be triggered by a node restart, pod move, etc.
Heketi is executing a command on a node and the communication channel fails in a way that the operation is unable to fail (see below). Typically this would eventually lead into the first item because the server with a blocked operation is eventually going to get restarted.
The gluster nodes that Heketi is working with are being restarted in such a way that the rollback can not complete. An example is a gluster node being restarted between creating a volume and starting the volume. This will fail the create operation but if there are too few nodes to allow deleting the incomplete volume heketi can not do anything more about the situation.

The 2nd point has been discussed recently among the Heketi developers. There is some concern that the kubernetes executor is not as stable as we need and that communicating with the gluster pods through that channel could lead to server hangs.
In cases 1 & 2 the things that heketi is going to process is logged in the db. These items can be viewed from a db dump (heketi-cli db dump) and can be manipulated via the offline command heketi db delete-pending-entries (please review the help prior to use).
You can also edit and reload the db dump manually but beware that this can lead to internal inconsistencies between the device storage counts if you add/remove bricks without making the correct changes to the device storage counters.
Case #3 was discovered after the version 6 release, AFAIR. There should be a fix in master for that condition. It will make #3 behave like 1 & 2 in that if the volume can not be cleaned up it will remain in the db for manual cleanup later.
If your shell sessions into the heketi container are being interrupted please verify that the pod/containter is being restarted. It may be failing liveness checks (due the root issue) and then being restarted by kubernetes. I'm not entirely sure about that part myself.
To wrap up, I think you are running into the following situation: something is preventing heketi from running operations to successful completion. This then may be causing heketi to be restarted. Restarts lead to stale operations and apparent de-syncs between heketi and the underlying system. While I can't guarantee this is the case it seems like a good place to start.
I would:

Keep track of when / how often the heketi container get restarted
Keep track of when / how often the gluster containers get restarted
Monitor how busy the master node(s) are, I've been told that the kubernetes executor has to go through the master
Use the heketi-cli db dump and heketi db export commands to check for pending operations in the db (the latter is better but must be done when heketi is stopped)
Read up on the heketi db delete-pending-entries command to see if will help you resolve sync issues
Attempt to root-cause the freezes you seem to experiencing. This may require you to disable liveness checks on the pod.

I know this is a lot to digest but without specifics all I can do it outline the general areas I would look into if I were debugging this issue.

I have seen gluster node restarts that think may have coincided with heketi failure. I am not sure if this also causes heketi restarts or not. I will look into this more.
So I tore it down again for more testing. But I will try to figure out if it is 1 2 or 3 and get back to you all at some point. I appreciate the help.

So my current way forward is to deploy gluster underneath kubernetes rather than in Kubernetes, and only run the heketi container inside of kubernetes since it seems like gluster container restarts are the issue I am facing. I want to see if deploying gluster not in containers solves my problems or not and debug from there, gluster is already distributed so I don't really need it running in a container.
@jarrpa if I am only running the heketi in the container for the deployment. What steps with gluster do I need to take ahead of time. Right now I have gluster installed (4.0 for now) on centos7 machines
yum list installed | grep gluster gives me
centos-release-gluuster40..x86_64
glusterfs.x86_64
glusterfs-api.x86_64
glusterfs-cli.x86_64
glusterfs-client-xlators.x86_64
glusterfs-fuse.x86_64
glusterfs-libs.x86_64
glusterfs-server.x86_64
userspace-rcu.x86_64
The gluster nodes are not connected together yet with peer probe yet. And they have no volumes attached to them.

Also I spread the cluster out (less space on each node, but more nodes for same total amount of space)  to try to eliminate issue #3 that @phlogistonjohn pointed out.

You do not want to do a peer probe. You only want the glusterd server running, no configuration done. Also make sure you have the requisite kernel modules loaded, the needed firewall ports opened, and the target devices are all bare (wipefs -a <device> should do it).

Okay that is easy (: I can uninstall and reinstall just to be safe on that front

Ugh still working this. Had fips mode on... I don't think this is fully fixed in gluster 4

Agreed, that is a very sore point in the user experience. :(
What's the status at the moment? Have we really not gotten past initial deployment?

So I have gotten past initial deployment both in containers and outside of containers, the issue is that I haven't been able to get any semblance of stability from Heketi and Gluster thus far.
Here my user evaluation thus far. If this is due to gaps in my understanding I welcome additional feedback and knowledge that can help me achieve what I am going for, which ultimately would be "cloud native storage for my k8s applications", though I would very much settle for performant/stable storage for my cloud native applications. This is why I wanted to go the Heketi/Gluster route in the first place.
I have been trying to test further with hard installed Heketi and Gluster providing storage up to Kubernetes on bare metal  servers due to @phlogistonjohn concerns with the kube-system pods not being stable enough for Heketi's purposes running in a container on Kubernetes. I was hoping the more external pieces I cut out the closer I would get to the root of the problems I am having and ultimately a final solution. So with hard install setup, I am seeing this crazy thing happen with Gluster where it crashes the machines and containers because for some unknown reason it is spinning up hundreds or thousands of Gluster deamons eventually crashing the machine. I can only assume that this same thing is happening in containerized gluster, though I have no hard evidence of this. This is when the real problems start as it causes to get Heketi out of sync with the cluster, its extremely difficult to clean up Gluster volumes, I also can't remove and re-provision problem nodes and get that back in sync with Heketi due to stuff discussed above, I can't force Heketi to remove volumes or nodes if dead nodes aren't working, and even if I could, I don't think I have the ability to say "okay as a system since I know that force removed node a had volumes 1 and 2 on it, I now need to redistribute that workload to other parts of the cluster". This functionality works, but only if the cluster is working perfectly and healthy, which I would argue that this functionality is desired mostly when the cluster is unhealthy and unhappy and a sys admin is attempting to get the cluster back to operating status.
I understand that these are all extremely difficult things to implement and there is a large amount of complexity with this sort of setup.  I definitely am not meaning to bash you guys in any way shape or form.

Thank you for the thorough feedback! Indeed, the failure scenario user experience leaves much to be desired. Howeverm I'll mention that the latest versions of heketi have the ability to directly edit their DB. While I don't know much about it personally, hopefully @phlogistonjohn can chime in with more.

I mean that would be great because then I could go back and at least edit the db in heketi and have it working.
But that would basically make the memory leak issues that heketi did so much to fix gluster's problem because if I can't go clean up the gluster volumes (since a node is down) then still have a hard time figuring out how to use this in a prod setting? Suggestions on that front?

So you guys over there really don't see any of these issues running gluster and heketi in contianers?
What OS are you guys running? What does your k8s cluster look like? I think you all mentioned you were bare metal? What is doing the install of Kubernetes? What version of K8s? So many questions.

Sorry for the delay, I'm been traveling and later swamped because of it.
Unfortunately, yes, we have really not seen issue like the ones you're running into. This is across the Red Hat family of distributions (Fedora, RHEL, etc.) in multiple environments including baremetal, VMs, and cloud. I myself have been running this since Kubernetes 1.7 on through 1.10, installed via kubeadm, on CentOS VMs.

I think at this point we can close this issue. We have decided where we are to go the local persistent volume route and deal with managing the applications that need volumes manually until a better solution for local pvcs come out. A lot of applications in our use case are already distributed so we don't really use the gluster replication. I really appreciate the help and I learned a ton about volume management, gluster, heketi, containers and kubernetes.

No problem! It's been a wild ride, sorry that we couldn't make it work out for you, but I hope the local storage solution proves sufficient!


[Question] Is it possible to set rootUrl in runtime?

Gemini is a great help for our project so far, but one thing makes its usage a little bit inconvenient: I'm speaking about setting rootUrl before test runs.
We have remote Selenoid server and we want to be able to start tests not only from CI-agents, but also from local developer environment that sometimes is placed in various networks. To solve an access problem (NAT/firewalls) we've developed a plugin that creates a public tunnel from URL like https://alexey.localtunnel.me to local web-server and it works perfectly, but since we aren't able to change rootUrl in the plugin we have to do something like this:
package.json
"scripts": {
    "gemini": "gemini_root_url=\"http://$(hostname -s).localtunnel.me\" gemini test",
  },

This step would be completly unnecessary if we could do something like:
gemini.on(gemini.events.START_RUNNER, () => {
   const URL = getTunnel();
   gemini.config.rootUrl = URL;
});

But it doesn't work and I'd like to know if there is a way to set rootUrl somehow from plugin API. And if there isn't would you accept a pull request than enables this functionality?

Hi you should change rootUrl for each browser:
gemini.on(events.START_RUNNER, async () => {
    const URL = getTunnel();

    gemini.config.getBrowserIds().forEach((id) => {
        const rootUrl = url.parse(gemini.config.forBrowser(id).rootUrl);

        gemini.config.forBrowser(id).rootUrl = url.format({
            host: URL,
            pathname: rootUrl && rootUrl.path || ''
        });
    });
});

All data in gemini.config - are just the default value for each browser config.

@alexeyoganezov and I'd suggest you to migrate to hermione. gemini will be deprecated soon

Alright, we'll try it. Thank you for the fast response.


fix: Fix click handling on 'Details' tab in Managed Connectors

Description

Nothing happened whenever the user clicked on Managed Connector's "Details" CTA. It should open a detailed model like Owned Connectors.

Steps To Reproduce

Goto User Menu & Click on "Connectors Management"
Switch to “Managed Connectors“
Click on the “Details“ link for any existing connector under Managed Connectors

Why

When the user clicks on the "Details" CTA it should work like Owned Connectors.

Issue
Link to Github issue.

Checklist
Please delete options that are not relevant.

[x] I have followed the contributing guidelines
[x] I have created and linked IP issues or requested their creation by a committer
[x] I have performed a self-review of my own code
[x] I have successfully tested my changes locally
[x] I have checked that new and existing tests pass locally with my changes
[x] I have commented my code, particularly in hard-to-understand areas

topic will be solved by

https://github.com/eclipse-tractusx/portal-backend/issues/945
https://github.com/eclipse-tractusx/portal-backend/issues/944

working on it already.
topic will be solved by

https://github.com/eclipse-tractusx/portal-backend/issues/945
https://github.com/eclipse-tractusx/portal-backend/issues/944

Hi @MaximilianHauer,

https://github.com/eclipse-tractusx/portal-backend/issues/945
https://github.com/eclipse-tractusx/portal-backend/issues/944

Above both PR are merged, but this issue is still their. Can you please review it again.
Thank you!

@MaximilianHauer It seems this topic hasn't been solved. It is important for the dataspace participants to see the details of the managed connectors they subscribed to through the marketplace. Can you look into this?


Where to donwload MSVD-QA dataset for evaluation?

Hello,
Thanks a lot for your great work. I would like to ask how can I download LLaMA-VID-Eval/MSVD-QA/video and LLaMA-VID-Eval/MSVD-QA/test_qa.json for evaluation?

Hello. You can download these files from here

I really appreciate it.


Render errors in KDoc

Especially for the intellij plugin, it would be great if the exception was rendered inside the kdoc instead of throwing an exception

Done in https://github.com/Jolanrensen/docProcessorGradlePlugin/commit/a3d99a6b8108b9de2134ab3a202959f1c074c303


help me download songs from spotify

What is the purpose of your issue?

[x ] Bug
[ ] Feature Request
[ ] Question
[ ] Other

Logroot@mint:~/git/spotify-downloader# spotdl --song https://open.spotify.com/track/1DhJgI2oAsDeDVSD5TNPCF
ERROR: 6IPXpHEV4JM: YouTube said: This video is unavailable.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 785, in extract_info
ie_result = ie.extract(url)
File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 440, in extract
ie_result = self._real_extract(url)
File "/usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py", line 1623, in _real_extract
expected=True, video_id=video_id)
youtube_dl.utils.ExtractorError: 6IPXpHEV4JM: YouTube said: This video is unavailable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pafy/backend_youtube_dl.py", line 40, in _fetch_basic
self._ydl_info = ydl.extract_info(self.videoid, download=False)
File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 808, in extract_info
self.report_error(compat_str(e), e.format_traceback())
File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 613, in report_error
self.trouble(error_message, tb)
File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 583, in trouble
raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: 6IPXpHEV4JM: YouTube said: This video is unavailable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/spotdl", line 11, in
load_entry_point('spotdl==1.2.4', 'console_scripts', 'spotdl')()
File "/usr/local/lib/python3.6/dist-packages/spotdl-1.2.4-py3.6.egg/spotdl/spotdl.py", line 68, in main
File "/usr/local/lib/python3.6/dist-packages/spotdl-1.2.4-py3.6.egg/spotdl/spotdl.py", line 27, in match_args
File "/usr/local/lib/python3.6/dist-packages/spotdl-1.2.4-py3.6.egg/spotdl/downloader.py", line 98, in init
File "/usr/local/lib/python3.6/dist-packages/spotdl-1.2.4-py3.6.egg/spotdl/youtube_tools.py", line 83, in match_video_and_metadata
File "/usr/local/lib/python3.6/dist-packages/spotdl-1.2.4-py3.6.egg/spotdl/youtube_tools.py", line 45, in go_pafy
File "/usr/local/lib/python3.6/dist-packages/pafy/pafy.py", line 124, in new
return Pafy(url, basic, gdata, size, callback, ydl_opts=ydl_opts)
File "/usr/local/lib/python3.6/dist-packages/pafy/backend_youtube_dl.py", line 31, in init
super(YtdlPafy, self).init(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pafy/backend_shared.py", line 97, in init
self._fetch_basic()
File "/usr/local/lib/python3.6/dist-packages/pafy/backend_youtube_dl.py", line 43, in _fetch_basic
raise IOError(str(e).replace('YouTube said', 'Youtube says'))
OSError: ERROR: 6IPXpHEV4JM: Youtube says: This video is unavailable.
root@mint:~/git/spotify-downloader#

Testing the song download with my software and worked for me, so the error is on your end. I'll paste here what I told someone else with errors. Always try this method to troubleshoot download errors.

Run the following commands in cmd. What versions do you have installed of the following programs? (this is assuming all items are in the same folder as is my case for spotdl, youtube-dl, ffmpeg, and python)
C:\Users\ ***********\AppData\Local\Programs\Python\Python38-32\Scripts>python --version
Python 3.8.1
C:\Users\ ***********\AppData\Local\Programs\Python\Python38-32\Scripts>spotdl --version
spotdl 1.2.4
C:\Users\ ***********\AppData\Local\Programs\Python\Python38-32\Scripts>youtube-dl --version
2020.02.16
C:\Users\ ***********\AppData\Local\Programs\Python\Python38-32\Scripts>ffmpeg --version
ffmpeg version git-2020-02-16-8578433 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.2.1 (GCC) 20200122
If you get anything other than this run pip install -U youtube-dl (or pip3 install -U youtube-dl). Substitute youtube-dl for whatever program is out of date

Thanks @xelam93 running pip install -U youtube-dl solved this issue for me

@xelam93 Thank you for the help! Y'all need to update youtube-dl as mentioned.


npm refusing to install eddystone-beacon-scanner as a dependency of itself

got above message while trying to install node-eddystone-beacon-scanner
on ubuntu 16.04 under proxy (proxy has been set)
can anyone help me with above

Hi @RajPratim21,
Maybe you are trying to run npm install in a folder name eddystone-beacon-scanner?
I'd suggest to make a ~/test directory and try to install in there. Let me know if this doesn't work.


Have StyleButton use map as a property

Changing NavigationToolbarExample to use mainMap caused the buttons to lose the reference to the map. I propose changing StyleButton to take the map as a parameter (property) instead of relying on it being a specific id

@jf990 do any of the examples that use navigation toolbar need to get updated to set the map property?

yes, i missed it because running the sample take the toolkit from one location and the examples from a different, so the change missed the repo. I'll add it.

I think StyleTemplateExample might also use this. Can you update any other instances of it in the example?

I made the correction to StyleTemplateExample and test it. I didn't find any other samples that reference it but if you notice anything else I missed LMK.

looks good, thanks @jf990


Issue #55 - Add error checking to JSON unmarshals

Add error check to all the unmarshal attempts in tests.

I've manually rerun the tests and everything looks good.
Thanks for the PR - the package is better already!


Deprecation of MCMT in favor of MCMTGate

The Multiple-Control-Multiple-Target in qiskit.circuit.library.generalized_gates.MCMT was pending for deprecation in https://github.com/Qiskit/qiskit/pull/13150 . This PR moves it to deprecated.

I think we want to deprecate the circuit library objects for 1.4 and 2.0, but might not want to remove them for 2.0. Even though this PR only changes from PendingDeprecationWarning to DeprecationWarning, we probably should discuss what the exact plan is 🙂


Add photo credits

https://www.flickr.com/photos/cifor/32778603930; https://www.flickr.com/photos/james_wheeler/9909142064; https://www.flickr.com/photos/cgiarclimate/14104673257/in/gallery-ilri-72157651071496591; https://www.flickr.com/photos/dbooster/15900757710; and https://www.flickr.com/photos/postflowersyndroms/7920883928 or https://www.flickr.com/photos/cifor/12184243296 depending on vote.

Second image for Sustainable Tropical Forestry wins vote, so also change this symposium's image to that one.

is this okay?

Symposia images by Juan Carlos Huayllapuma/CIFOR, James Wheeler, C. Schubert (CCAFS), David A. LaSpina, and Ricky Martin for Center for International Forestry Research

?Is that okay

Forestry image have been replaced #23

All good. Thanks, @w1lldone.


Show Rerun builds regardless the results

resolved: https://github.com/adoptium/aqa-test-tools/issues/814

@smlambert could you review the PR? Thanks


Fix pruning elements from SimpleCache in python3

Hi, I noticed that an exception was being thrown when using the SimpleCache in python 3
File "werkzeug/contrib/cache.py", line 262, in _prune
for idx, (key, (expires, _)) in enumerate(self._cache.items()):
RuntimeError: dictionary changed size during iteration
when running a sample test
from werkzeug.contrib import cache
c = cache.SimpleCache(threshold=2)
c.set("1", "1")
c.set("2", "2")
c.set("3", "3")
c.set("4", "4")
since in python3, the iterator is now a view into the original dict rather than a copy (unlike python2), so an error is thrown when trying to remove keys in the _prune method.
I've modified it to keep a list of elements to remove instead, and remove them after iteration; is that okay?
Thanks,
Jin

Could you add a changelog entry? Thanjs!

Okay, I've added a description to the CHANGES file in 0.9.7; hopefully that's correct. Thank you!

Unfortunately that's incorrect... the next release for master is 0.10. If you want to include this in 0.9.7 too, you'll have to refile this PR against the 0.9 maintenance branch.
Also, I forgot: I am not sure if this is easy, but if it is, please also add a testcase for this.

Okay, thanks; I've created a new pull request https://github.com/mitsuhiko/werkzeug/pull/651 based on 0.9 instead with a test attached.


File does not exist at path

"message": "File does not exist at path /var/my-work/bsamply/bsamply_backend/Modules/Agreements/module.json.",

Does your agreement module have a module.json?

all files exists


Update to 3.4.1

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Trying this out on the off chance something has changed and the issue has resolved itself :/

I added some commits to this PR: https://github.com/conda-forge/podman-feedstock/pull/14
It's building now, but it requires a newer version of conmon than is currently available.


Use spring cloud alibaba sentinel.How to set the jvm properties in project app

Use spring cloud alibaba sentinel.How to set the jvm properties in project application.yml?
eg.
csp.sentinel.metric.file.single.size
csp.sentinel.statistic.max.rt
I found that I can only modify it in jvm -D or propertiesFile .But I want to modify it in application.yml or bootstrap.yml.Can you give me some suggest.
(我想不通过jvm或者properties文件在springcloud的模块中，设置sentinel-core的值.)

Refer Spring Cloud Alibaba Sentinel Wiki.


Report ConstraintException when using guidance GPT-4 with ThreadPool.

The bug
A clear and concise description of what the bug is.
Report ConstraintException when using guidance GPT-4 with ThreadPool.
Traceback (most recent call last):
  File "/home/longxiang/lx/annotations_2_4b/batch_run_logi_qa_explaination.py", line 78, in <module>
    result = future.result()
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/longxiang/lx/annotations_2_4b/batch_run_logi_qa_explaination.py", line 56, in get_explaination_helper
    lm = lm + get_explaination(text, output)
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/models/_model.py", line 306, in __add__
    out = value(lm)
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/_grammar.py", line 45, in __call__
    return self.f(model, *self.args, **self.kwargs)
  File "/home/longxiang/lx/annotations_2_4b/batch_run_logi_qa_explaination.py", line 45, in get_explaination
    lm += gen("explaination", max_tokens=500, temperature=0.8, stop=["Question:", "<|im_end|>"])
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/models/_model.py", line 302, in __add__
    out = lm._run_stateless(value)
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/models/_model.py", line 465, in _run_stateless
    for new_bytes, is_generated, new_bytes_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/models/_model.py", line 785, in __call__
    logits = self._get_logits(token_ids, parser.bytes[start_pos:forced_pos], current_temp)
  File "/home/longxiang/miniconda3/envs/lx_py310/lib/python3.10/site-packages/guidance/models/_remote.py", line 216, in _get_logits
    raise self._report_failed_match(prompt)
guidance.models._model.ConstraintException: The model attempted to generate b' love chili\nThis option tells us aboutTh...' after the prompt `b'...anation:<|im_end|><|im_start|>assistant\n'`, but that does
not match the given grammar constraints! Since your model is a remote API that does not support full guidance
integration we cannot force the model to follow the grammar, only flag an error when it fails to match.
You can try to address this by improving the prompt, making your grammar more flexible, rerunning with
a non-zero temperature, or using a model that supports full guidance grammar constraints.

To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
# put your code snippet here
import os
import json
import guidance
from guidance import models, gen, system, user, assistant
import tiktoken
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor, as_completed
from copy import deepcopy
import threading
import time

MAX_THREADS = 2
SEM_FACTOR = 4
semaphore = threading.Semaphore(MAX_THREADS * SEM_FACTOR)
ST = 0
ED = 20

base_dir = "data/"
with open(os.path.join(base_dir, "logiqa_train.jsonl"), "r") as f:
    data_group_list = json.load(f)

tokenizer = tiktoken.encoding_for_model("gpt-4")
gpt4 = models.OpenAI(
    model="gpt-4-1106-preview",
    tokenizer=tokenizer,
    api_key="<a secret key>",
    echo=False
)

@guidance
def get_explaination(lm, question, answer):
    with system():
        lm = (
            lm
            + "You are an expert in creating logic and common sense reasoning questions. Now, there are some questions and answers that require your detailed explanation. Please ensure that each explanation is clear and precise, and presents the reasoning process in a step-by-step manner. For choice questions, please explain why the correct answer is correct and why the other choices are incorrect. For open-ended questions, please explain the reasoning process of the answer."
        )

    with user():
        lm += f"Question: {question}\nAnswer: {answer}\nExplanation:"

    with assistant():
        lm += gen("explaination", max_tokens=500, temperature=0.8, stop=["Question:", "<|im_end|>"])

    return lm

def get_explaination_helper(lm, data_group):

    options_text = "\n".join(data_group["choices"])
    text = f"{data_group['question']}\n{options_text}"
    output = f"{data_group['ans'].upper()}."
    with semaphore:
        lm = lm + get_explaination(text, output)
        explaination = lm.get("explaination")
        del lm
    result = {
        "input": f"<用户>Q:{text}\nA:<AI>",
        "output": f"{explaination}.\n\nThe answer is {output}",
    }

    return result

_data_group_list = data_group_list[ST:ED]
with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    futures = []
    for data_group in tqdm(
        _data_group_list, total=len(_data_group_list), desc="submit job"
    ):
        futures.append(executor.submit(get_explaination_helper, gpt4, data_group))
        time.sleep(0.1)
    results = []
    for future in tqdm(as_completed(futures), total=len(futures), desc="get result"):
        with open(os.path.join(base_dir, "logiqa_explaination.jsonl"), "a") as f:
            result = future.result()
            f.write(json.dumps(result) + "\n")

System info (please complete the following information):

OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Ubuntu
Guidance Version (guidance.__version__): 0.1.10

Hmm. I ran this and got no error when I made up the missing data. (I used the branch in #586) If you still see issues perhaps post a smaller example. Thanks!


[fix](group commit) Fix p2 regression-test

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


AssetType can't be changed once we have presented the picker controller.

Let i have set the the assetType to .allPhotos and then presented the controller then i press cancel. Now i set the assetType to .allVideos but it won't change now. Is there any workaround for that?

You can present a different instance.
let photoPicker = DKImagePickerController()
photoPicker.assetType = .allPhotos

let videoPicker = DKImagePickerController()
videoPicker.assetType = .allVideos

Is it possible with just one pickerController

Why do you need it to be the same instance?

Your above solution also didn't work. Setting assetType to later controller affects first controller

Ah, I was creating each instance just before presenting (which works), rather than keeping references.

Yes that was the only solution that worked by creating instance every time before presenting. Also i have to save the selectedAsset and set the selectedAsset to pickerController before presenting.
But why this happens?
Why can't we create two instances?

This problem occurs due to its shared instance. I have found a solution to this problem. Whenever you call DKImagePickerController() , dispose its previous instance before that.
Paste this method in DKImageManager class :
    func disposeInstance()
    {
        DKImageManager.Static.instance = nil
    }

And Call it before creating any object of DKImagePickerController(). Like this:
DKImageManager.sharedInstance.disposeInstance()
let photoPicker = DKImagePickerController()
photoPicker.assetType = .allPhotos

OR
DKImageManager.sharedInstance.disposeInstance()
let videoPicker = DKImagePickerController()
videoPicker.assetType = .allVideos

Hope It helps!
Cheers !
Happy Coding :)

Hi @ignivaankitchauhan , I'm facing the same issue. How can I fix this ?
Tried
DKImageManager.Static.instance = nil
But gets an error
Type 'DKImageManager' has no member 'Static'
I am using DKImagePickerController - version 3.6 (Pods)


Weather modules dont show up anymore

After https://github.com/MichMich/MagicMirror/pull/2161 got merged into develop my weather modules (current & forecast) dont show up anymore.
Not sure what happens but IMHO that PR seems to be incomplete and breaks existing configs and should therefor should be reworked.

Hmm... Sorry, don't know why its not working for you, all is good here, my weather modules keeps showing them self even if I have hidden them with another module. Do you not experience this?

I dont hide my weather modules from other modules. How do you do that?

@rejas
MMM-Voice will allow you to hide individual modules on command
my MMM-SleepWake provides an option to hide all modules instead of turning off the screen (where not possible)
there are others

@rejas I'm using my own MMM-Modulebar to hide and show modules. And also I'm using "profiles" MMM-ProfileSwitcher to show different modules for different users (@ different times of the day). But the problem is that both the weather modules do not "respect" that they have been hidden. They keep showing up when they update (default every 10 minutes).

their getDom() routines should become aware of the state and not draw when hidden
they should have suspend() and resume() methods that track the state

This might be an issue in the core itself. getDom should not be called when they are hidden.

@MichMich yep. updateDom() does not check the visibility state before calling getDom()

@Snille @rejas Does https://github.com/MichMich/MagicMirror/pull/2166 solve the issue?

No, I only see a big "undefined" in the place of the module. See my comment in the PR for possible reason.

I added a new commit returning an empty promise. Could you give it another try, @rejas?

The undefined was an error on my side. Still, returning an empty promise is the better way I think. So: thumbs up from me for this.

looks good..  i  think the real bug was in updateDomWithContent,
but this approach is cleaner
var updateDomWithContent = function (module, speed, newHeader, newContent) {
		return new Promise(function (resolve) {
			if (module.hidden || !speed) {                       // <--- maybe ig(!module.hidden && !speed)// speed handled later

Hmm.. It's not working for me, I think, if it's in the branch "develop-hidden-module-update-restriction", the weather modules still comes back on refresh and also the "clock" stops updating if it's been hidden and then shown again... :)

@Snille

the weather modules still comes back on refresh
what does that mean?

@sdetweil Oh, sorry for being unclear. :) I mean when the module "refreshes" it's content (default every 10 minutes)...

can u examine the file ~/MagicMirror/js/main.js
search for updateDom
should start like this
var updateDom = function (module, speed) {
		if (module.hidden) {
			return new Promise();
		}

@sdetweil Yepp, that is correct, I'm on "develop-hidden-module-update-restriction", there is is as you show above. When I hide and show the clock, it stops updating (seconds stops counting) and the weather modules shows up again (I have set the update interval to 1 minute) when they refresh. I'm using my "MMM-Modulebar" to hide and show modules... Is there something I can have missed?

what weather provider are you using?

Is that your module? Or someone else's?

How does your module show/hide modules? Do you use the MagicMirror² built in functionality for this?

Is that your module? Or someone else's?

To hide and show modules I'm using my own module MMM-Modulebar.

How does your module show/hide modules? Do you use the MagicMirror² built in functionality for this?

That's what I think I do... :) (https://github.com/Snille/MMM-Modulebar/blob/master/MMM-Modulebar.js)

what weather provider are you using?

openweathermap.org

Not sure what triggers the showing of the module. But to prevent this, you might want to look into the lockString option during hiding/showing. This prevents modules from showing when your module hides it.

can u open the developers console. ctrl-shift-i, and select console, add weather to the filter field and then dock the console on the bottom of the screen (you can do this on your pc if u have address:"<IP_ADDRESS>", and ipWhiltelist:[], in config.js to allow your pc to connect to pi
the code in weather.js does this
	// What to do when the weather provider has new information available?
	updateAvailable: function () {
		Log.log("New weather information available.");
		this.updateDom(0);
		this.scheduleUpdate();
	},

when the providers signal updateAvailable()
so, you should see a message on the console before every update.
you can also add this to the bottom of weather,js
	},  // < notice the comma
	resume() {                         // so it looks like thiss
		Log.log('weather resume called')
	}            <  last lien of add
});

Not sure what triggers the showing of the module. But to prevent this, you might want to look into the lockString option during hiding/showing. This prevents modules from showing when your module hides it.

Well, I don't want to lock it, because then other modules (like IFTTT och even MMM-Remote-Control) cant un-hide them. Also I'm using "profiles" that switches at different times, there also modules are shown and hidden... :/

Also I'm using "profiles" that switches at different times

@sdetweil Ok, done it all...
MMM-Modulebar.js:117 Hiding weatherforecast ID: 5
module.js:199 weatherforecast is suspended.
module.js:206 currentweather is resumed.
module.js:206 weatherforecast is resumed.
module.js:206 currentweather is resumed.
module.js:206 weatherforecast is resumed.
`

It's not the weather provider that cases the resume... As far as I can understand?...

so, I would look at profiles, and add a logging entry around its module.show()
sadly show/hide can't tell us who is invoking those calls.
no, all good how u are running.. nuc is 'pi'.. i do all my dev on ubuntu desktop, and have 5 SBCs and VMs and local machine all running instances..

@sdetweil Do you have this problem at all on your setup? I have had it from start (over 2 years)... That's why I started to "dig" in to it...

i do not.. my webcam based motion detector (MMM-SleepWake)  hides modules (as my TV doesn't support turn off/on)  and I never see anything pop up
used on pi, odroid, jetson nano. linux, catalina,

so, I would look at profiles, and add a logging entry around its module.show()
sadly show/hide can't tell us who is invoking those calls.
no, all good how u are running.. nuc is 'pi'.. i do all my dev on ubuntu desktop, and have 5 SBCs and VMs and local machine all running instances..

Hmm... Do you think it can have to do with the "classes" (that wrongly I call profiles)?

Do you think it can have to do with the "classes" (that wrongly I call profiles)?

i don't think there are very many places that hide/show will be called.. if u have access to the class name at the time..  u can log it..

hm.. but I see an error with the new code
main.js:115 Uncaught TypeError: Promise resolver undefined is not a function
    at new Promise (<anonymous>)
    at updateDom (main.js:115)
    at Object.updateDom (main.js:560)
    at Class.updateDom (module.js:364)
    at Class.socketNotificationReceived (calendar.js:153)
    at module.js:250
    at r.<anonymous> (socketclient.js:39)
    at r.emit (index.js:83)
    at r.onevent (index.js:83)
    at r.MMSocket.self.socket.onevent (socketclient.js:33)

as the promise doesn't have a resolve method

Do you think it can have to do with the "classes" (that wrongly I call profiles)?

i don't think there are very many places that hide/show will be called.. if u have access to the class name at the time.. u can log it..

I just removed all the classes from all modules... Same thing... Well, I got to get some sleep... :)
My config now looks like this, and it still have the same problems.. :)

/* Magic Mirror Config Sample
 *
 * By Michael Teeuw http://michaelteeuw.nl
 * MIT Licensed.
 *
 * For more information how you can configurate this file
 * See https://github.com/MichMich/MagicMirror#configuration
 *
 */

var config = {
	port: 8080,
	ipWhitelist: [],
	address: "<IP_ADDRESS>",
	language: "sv",
//	language: "en",
	timeFormat: 24,
	units: "metric",

	modules: [
		// 00
		{
			module: "alert",
		},
		// 01
		{
			module: "updatenotification",
			position: "top_bar"
		},
		// 02
		{
			module: "clock",
			position: "top_left"
		},

// Test here!

		// 03
		{
			disabled: false,
			module: "MMM-Modulebar",
			position: "bottom_bar",
			config: {
				showBorder: false,
				allowForce: true,
				buttons: {
				"1": {
						module: "clock",
						symbol: "clock"
					},
				"2": {
						module: "currentweather",
						symbol: "cloud"
					},
				"3": {
						module: "weatherforecast",
						symbol: "moon"
					}
				}
			}
		},
		// 04
		{
			disabled: false,
			module: "currentweather",
			position: "top_right",
			header: "Just nu Åstorp",
			config: {
				updateInterval: 60*1000,
				location: "Astorp",
				appendLocationNameToHeader: false,
				locationID: "2724435",
				appid: "--SeCreT--",
				showPeriodUpper: true,
				showHumidity: true
			}
		},
		// 05
		{
			disabled: false,
			module: "weatherforecast",
			position: "top_right",
			header: "Väderprognos för Åstorp",
			config: {
				updateInterval: 60*1000,
				location: "Astorp",
				appendLocationNameToHeader: false,
				locationID: "2724435",
				appid: "--SeCreT--",
				showRainAmount: true
			}
		},

// End Here!

		// 06
		{
			disabled: false,
			module: "MMM-Remote-Control",
			position: "top_left",
			config: {
			}
		}
	]

};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {module.exports = config;}

hm.. but I see an error with the new code
main.js:115 Uncaught TypeError: Promise resolver undefined is not a function
    at new Promise (<anonymous>)
    at updateDom (main.js:115)
    at Object.updateDom (main.js:560)
    at Class.updateDom (module.js:364)
    at Class.socketNotificationReceived (calendar.js:153)
    at module.js:250
    at r.<anonymous> (socketclient.js:39)
    at r.emit (index.js:83)
    at r.onevent (index.js:83)
    at r.MMSocket.self.socket.onevent (socketclient.js:33)

as the promise doesn't have a resolve method

I don't get that error... ?

ok, looks like I was always running forecast, not current.. I see the show() problem  with my MMM-SleepWake.
doah, its right there in currentWeather.
right at the end of updateWeather
		this.show(this.config.animationSpeed, { lockString: this.identifier });  < ---- shows itself on every update
		this.loaded = true;
		this.updateDom(this.config.animationSpeed);
		this.sendNotification("CURRENTWEATHER_DATA", { data: data });
	},

change to this
		if(!this.loaded){   // if data not yet loaded once
                        // unhide
			this.show(this.config.animationSpeed, { lockString: this.identifier });
                        // indicate data loaded
			this.loaded = true;
		}

well, forecast does the same.

@rejas & @Snille: does https://github.com/MichMich/MagicMirror/pull/2168 solve the issue for you?

In what branch has it been merged? :)

git fetch origin pull/2168/head:develop-hidden-module-update-restriction
git checkout develop-hidden-module-update-restriction

i think he already has that branch  @Snille  just change the branch name, add a 1 or something
git fetch origin pull/2168/head:develop-hidden-module-update-restriction1
git checkout develop-hidden-module-update-restriction1

:+1: looks good on my mirror!

@MichMich @sdetweil Looks good! Thank you! It's working. :)

Merged!

@MichMich @sdetweil I'm running into the same issue, although the proposed soln. doesn't work for me.
Also why was the change reverted in 2165?

Because it caused an issue. Please supply more info about your issue so we can see if this is the same issue.


CB-11714: (windows) added check for encoding in savePhoto() without height/width

Platforms affected
Windows 10
What does this PR do?
Adds extra content-type check when capturing photos without specifying targetWidth and/or targetHeight in options to preserve JPEG file extension.
What testing has been done on this change?
ditto
Checklist

[x] Reported an issue in the JIRA database
[x] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
[ ] Added automated test coverage as appropriate for this change.

Cordova CI Build has completed successfully.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Hey @DisruptiveMind , there seems to be a merge conflict now. Could you maybe fix this? Thanks.

Thanks @DisruptiveMind.
Tests are now failing because of
/Users/travis/build/apache/cordova-plugin-camera/src/windows/CameraProxy.js
  790:67  error  Strings must use singlequote  quotes
✖ 1 problem (1 error, 0 warnings)
  1 error, 0 warnings potentially fixable with the `--fix` option.

This doesn't make too much sense to me, especially as line 790 contains no quotes at all.
Can you maybe rebase on master? I hope this will go away with more recent test configuration. Thanks!

Closing and re-opening to trigger a new CI/test run with new PR merge.

Looks OK to me, merging


Memory issues when constructing a ListDataset

Hello
since upgrading to gluonts, I'm running into some memory management issues when trying to construct a ListDataset for training or inference.
Here's the script :
import pandas as pd
from gluonts.dataset.field_names import FieldName
from gluonts.dataset.common import ListDataset
import tqdm

start_ts = pd.Timestamp(df.BUSINESS_DATE.min(), freq="D")
train = []
test = []
df_groupby = df.groupby(["A", "B"], sort=False)
for _, df_subset in tqdm.tqdm(df_groupby):
    if len(df_subset.A) >= (3 * 90 * prediction_length):
        for lt in np.arange(0, 91):
            tseries_train = {
                FieldName.TARGET: df_subset.loc[
                    (df_subset.LT == 0), col_target
                ]
                .values[:-prediction_length]
                .tolist(),
                FieldName.START: start_ts,
                FieldName.FEAT_DYNAMIC_REAL: get_dynamic_feat_real_ratecat_dataframe(
                    df_subset, col_target, lt
                )
                .values.T[:, :-prediction_length]
                .tolist(),
                FieldName.FEAT_DYNAMIC_CAT: df_subset.loc[
                    (df_subset.LT == lt), COLS_DYN_CAT
                ]
                .values.T[:, :-prediction_length]
                .tolist(),
                FieldName.FEAT_STATIC_CAT: df_subset.loc[
                    (df_subset.LT == 0), COLS_STATIC_CAT_RATECAT
                ]
                .iloc[0]
                .values.tolist(),
                FieldName.FEAT_STATIC_REAL: df_subset.loc[
                    (df_subset.LT == lt), COLS_STATIC_REAL
                ]
                .iloc[0]
                .values.tolist(),
            }
            train.append(tseries_train)
            tseries_test = {
                FieldName.TARGET: df_subset.loc[
                    (df_subset.LT == 0), col_target
                ].values.tolist(),
                FieldName.START: start_ts,
                FieldName.FEAT_DYNAMIC_REAL: get_dynamic_feat_real_ratecat_dataframe(
                    df_subset, col_target, lt
                ).values.T.tolist(),
                FieldName.FEAT_DYNAMIC_CAT: df_subset.loc[
                    (df_subset.LT == lt), COLS_DYN_CAT
                ].values.T.tolist(),
                FieldName.FEAT_STATIC_CAT: df_subset.loc[
                    (df_subset.LT == 0), COLS_STATIC_CAT_RATECAT
                ]
                .iloc[0]
                .values.tolist(),
                FieldName.FEAT_STATIC_REAL: df_subset.loc[
                    (df_subset.LT == lt), COLS_STATIC_REAL
                ]
                .iloc[0]
                .values.tolist()
            }
            test.append(tseries_test)
return (
    ListDataset(train, freq="D"),
    ListDataset(test, freq="D"),
)

Before creating the individual dictionaries, my dataframe is around 16G in memory and while constructing the gluonts dataset memory ballons to 116 GB which is insane. I understand memory was bound to increase, but I wasn't expecting anything this high.
I don't recall having such issues with my previous version (0.3.4.dev26+g29ab758). I'm now on gluonts 0.4.0. Of course, this could purely be an mxnet thing.
Am I doing anything blatantly wrong ? I haven't changed my script in between version updates.

Never mind i found the culprit !


Update Home.md to fix grammar and add quote formatting

Clarify sentences and add quote notation.
This is a change you might not want a PR for, since it's just a minor change to fix grammar and add some formatting.  But just as an example, if you were making a bigger change, you can use this process to have someone review your changes.

I'm just going to merge this, it changes how the markdown is rendered from this:

To this:

Which I think is easier to read.


Add operation JsonPatch => dynamic support

(after rebase & including fix for NotNull on path as per Kirthi's comments)
Most important changes:

Added ObjectTreeAnalysisResult: this class analyzes the object tree and results in an object that's used by the Add method to know if it should use "dynamic logic" (ie: add to the Dictionary) or "regular logic" (staticly typed). JsonContract checks moved to this class instead of in the Add method
Changes in the Add method: split-up between regular & dynamic
Addition of GetNumericEndResult: used in Add to act differently when a numeric end is found vs when it's found but is invalid

Note that this is just the Add operation. Once this is approved & suggestions/comments are incorporated, the other methods will be changed as well - and that's when I can clean up the code a bit more (for example: GetNumericEnd can be removed, GetNumericEndResult can be renamed, helpers to find properties with JsonContract can be removed as they now are in ObjectTreeAnalyisResult, ...)

I think I get what's going on here, still have some questions about why JsonDynamicContract hasn't already solved this for us.

:shipit: - looking pretty good. I'm not in the office today, but I'm going to do a bit of style cleanup later and then merge it.

@KevinDockx sorry to bother you with CLAs, but I think that maybe you never signed the updated CLA that we started using a few months ago. You can find it here: https://cla2.dotnetfoundation.org/
This new one is even simpler than the old one. Sorry to trouble you with this!

@KevinDockx, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.
Thanks, DNFBOT;

@Eilon No problem, just signed it :-)

Thanks!

Thanks Kevin! Now I'll take care of merging this.

@KevinDockx - I've merged this to dev.
I ended up tweaking a bunch of the error handling to make it more readable. Please take a look and try to do the next parts in a more similar style.


Delete endpoint does not have URL filtering parameters version and spec_version

Raised here: https://community.dogesec.com/t/deleting-objects-using-the-arango-taxii-server/84

The behaviour should be
If not parameters passed, ALL objects with the id specified deleted.
If parameters passed, only matching object deleted.

cannot reproduce on latest version

@fqrious can you make sure these body parameters are also shown in swagger spec


How to use AzureServiceTokenProvider.GetAccessTokenAsync to obtain an access token for an Azure Function http trigger protected by Azure AD authentication?

I have an Azure Function app that uses the integrated Azure AD Authentication configured via the Azure portal.
Via the "Express" mode setup, this creates an Azure App Registration along with Enterprise Application.
Now when my app needs to invoke the function app HTTP triggers protected by this AAD auth, I need to set a bearer token.
I assume AzureServiceTokenProvider.GetAccessTokenAsync can be used to obtain such a token either as VS/Azure CLI (when in local dev) or as the managed identity (when deployed in Azure).
But I can't figure out which resource URL to use to obtain the token.
Is this a supported scenario?

//cc: @nonik0,  @anaismiller

Thank you for your feedback.  Tagging and routing to the team members best able to assist.

@felinepc
These links may be helpful for you Azure Function Authentication and allowed ResourceUrls .
Your understanding of what authentication method to use is correct. The resourceUrl would be for whatever service you are trying to access/authenticate to next. You should already be authenticated to the Azure Function with the initial integrated authentication.

I figured it out. The resource URL is the function app instance, i.e.  https://appname.azurewebsites.net.
HOWEVER, there is an additional step that is crucial: Under the App Registration of the service principal created by Azure Functions when AAD Auth was enabled, one must add any "non-interactive" clients (such as Azure CLI) to the "Authorized Client Applications" list under "Expose and API" (enter their Azure AD client ID). Otherwise these won't work when they attempt to access the app on behalf of a user, because they can't "consent" to the delegate permissions.


Don't require RBAC for confluent secret v3

Description
With version 3 of the CLI confluent secrets command, the RBAC login requirement was removed. The playbooks have not been updated to reflect this change and secrets management is getting disabled.
Fixes issues https://confluent.slack.com/archives/CM80GTNUQ/p1656981267820799
Type of change

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
These changes have been tested against EC2 instances running in AWS on a n mTLS non RBAC cluster with secret management and health checks enabled.
Test Configuration:
Checklist:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules
[ ] Any variable changes have been validated to be backwards compatible

the condition to check the cli version is missing on MDS login task in roles/common/secrets_protection, please add it as well


[DEV-2480] Mail Refactoring

Release Checklist
Pre-Release

[ ] Check migrations

No database related infos (sqldb-xxx)
Impact on GS (new/removed columns)

[ ] Check for linter errors (in PR)
[ ] Test basic user operations (on DFX services)

Login/logout
Buy/sell payment request
KYC page

Post-Release

Test basic user operations
Monitor application insights log

Die Ansicht im Outlook passt noch nicht


cloudflare-wrangler: 1.7.0 -> 1.8.1

Semi-automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/cloudflare/wrangler/releases.
meta.description for cloudflare-wrangler is: "A CLI tool designed for folks who are interested in using Cloudflare Workers."
meta.homepage for cloudflare-wrangler is: "https://github.com/cloudflare/wrangler"
Updates performed:

Quoted meta.homepage for RFC 45
Rust version update

Release on GitHub
Compare changes on GitHub

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.8.1 with grep in /nix/store/qiklxbn9y8gi03pqkd75n2xx6qrl9fsz-cloudflare-wrangler-1.8.1

directory tree listing: https://gist.github.com/5e2ba97ac17ae873e7e0999f46243e21

du listing: https://gist.github.com/df3037b54045709d4e422ca925644d1e

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)
1 package rebuild(s)
1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
cloudflare-wrangler

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/qiklxbn9y8gi03pqkd75n2xx6qrl9fsz-cloudflare-wrangler-1.8.1 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cloudflare-wrangler https://github.com/r-ryantm/nixpkgs/archive/39bdf9eb679a041e103851a5b47679bdf97b8b01.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/qiklxbn9y8gi03pqkd75n2xx6qrl9fsz-cloudflare-wrangler-1.8.1
ls -la /nix/store/qiklxbn9y8gi03pqkd75n2xx6qrl9fsz-cloudflare-wrangler-1.8.1/bin

Woohoo automated Rust updates!!!


Feature/idea: Color coding in console log

Node Inspector v0.9.1
Node v0.12.0
Windows 7 64bit
For some outputs I am using console.log with '[31;1m' and various other ANSI escape sequences for formatting color.
console.log('[31;1mThis should be red:')

It would be interesting if these could be used in the console on node-debug rather than outputing [31;1m as part of the string.
Could be done with span tags and inline css or a class.
I am tempted to do just the foreground and background colors.
Any advice/comments/objections?

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/10150694-feature-idea-color-coding-in-console-log?utm_campaign=plugin&utm_content=tracker%2F195817&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F195817&utm_medium=issues&utm_source=github).

AFAIK, debug supports color output in browser console too, it may be a good source of inspiration.

Duplicates #502

Sure im happy for it to be a plugin feature :)

Closing as a duplicate then.


Hi, I use python 3.5 and tensorflow 1.0.1 . and when I download and load the inception model in tutorial no. 7 my python program gives the error " stop working" . can you please help me?

STOP!
Most of the problems people are having are already described in the installation instructions.
Python 3.5
These tutorials were developed in Python 3.5 and may give strange errors in Python 2.7
Missing Files
You need to download the whole repository, either using git clone or as a zip-file. See the installation instructions.
Questions about TensorFlow
General questions about TensorFlow should either be asked on StackOverflow or GitHub.
Suggestions for Changes
The tutorials cannot change too much because it would make the YouTube videos too different from the source-code.
Requests for New Tutorials
These tutorials were made by a single person on his own time. It took a very long time to
research and produce the tutorials. If a topic is not covered then the best thing is to make
a new tutorial by yourself. All you need is a decent microphone, a screen-grabbing tool, and a
video editor. I used the free version of DaVinci Resolve.
Other Issues?
Please carefully read the installation instructions and only open an issue if you are still having problems.

How am I supposed to help you when you just write a subject-line that says the program has stopped working? You need to give more details and print-outs of the actual errors you are getting. Have you tried googling for the errors?


delayed_writer: change it for common values

Make this effective for resolution starting at 40min. In
the future the delay should probably be a function of the precision
with a fixed max.

This change is

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion.

biggraphite/drivers/_delayed_writer.py, line 62 at r1 (raw file):

        Args:
          mtric: the metric associated with these points.

typo

Comments from Reviewable

just the typo and

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable


Sync property model with chart model

If you call updateOptions or updateSeries, the property model should stay in sync.

Completed in 1.0.22


Document "Config Vars"

The Config Vars section in Dashboard is a generic interface to set special variables from. It currently lacks docs describing its use.

this is a duplicate of #338