[QUESTION] Applying Deep Convolutional GAN to other datasets

Describe what is unclear to you
I am trying to apply the Deep Convolutional GAN example from Chapter 17 p. 599 to the fruits-360 dataset on Kaggle:
Dataset properties
The total number of images: 90483.
Training set size: 67692 images (one fruit or vegetable per image).
Test set size: 22688 images (one fruit or vegetable per image).
Image size: 100x100 pixels.
I am getting outputs from the following code, but they don't look like fruit at all. My thought process was that in the generator in the example we set the first Dense layer to 7 * 7 * 128 because we have downsampled our 28 x 28 images to 7 x 7 with 128 channels, so I went with 25 x 25 x 128 as the Fruits images are 100 x 100 pixels. My first question is why do we use 128 channels instead of just 1 for greyscale?
We then go down to 64 channels for the Conv2DTranspose to reduce the depth and the stride of 2 upsamples to 14 x 14 (50 x 50 for the fruits images), before having the depth reduced to 1 and being upsampled again to 28 x 28 (100 x 100 for fruits). My second question is why did we settle on a kernel_size of 5?
My next question is why, in the following code, do we multiply by 2 and subtract 1?
X_train_dcgan = x_train.reshape(-1, 100, 100, 1) * 2. - 1.

And how do we pick the codings_size and batch_size?
To Reproduce
tf.random.set_seed(42)
np.random.seed(42)

codings_size = 100

generator = keras.models.Sequential([
    keras.layers.Dense(25 * 25 * 128, input_shape=[codings_size]),
    keras.layers.Reshape([25, 25, 128]),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2DTranspose(64, kernel_size=5, strides=2, padding="SAME", activation="selu"),
    keras.layers.BatchNormalization(),
    keras.layers.Conv2DTranspose(1, kernel_size=5, strides=2, padding="SAME", activation="tanh"),
])
discriminator = keras.models.Sequential([
    keras.layers.Conv2D(64, kernel_size=5, strides=2, padding="SAME",
                       activation=keras.layers.LeakyReLU(0.2), input_shape=[100, 100, 1]),
    keras.layers.Dropout(0.4),
    keras.layers.Conv2D(128, kernel_size=5, strides=2, padding="SAME",
                       activation=keras.layers.LeakyReLU(0.2)),
    keras.layers.Dropout(0.4),
    keras.layers.Flatten(),
    keras.layers.Dense(1, activation="sigmoid")
])
gan = keras.models.Sequential([generator, discriminator])

X_train_dcgan = x_train.reshape(-1, 100, 100, 1) * 2. - 1. # reshape and rescale

batch_size = 32
dataset = tf.data.Dataset.from_tensor_slices(X_train_dcgan)
dataset = dataset.shuffle(1000)
dataset = dataset.batch(batch_size, drop_remainder=True).prefetch(1)

train_gan(gan, dataset, batch_size, codings_size)

Sorry for all the questions, I am trying to make sure I understand what all of the hyperparameters do and whether or not I have made mistakes in my code or just need to tune it!
Thanks

Hi @Reisa14,
Thanks for your feedback.

My first question is why do we use 128 channels instead of just 1 for greyscale?

The generator's input is not an image, it's a dense vector (codings), and the generator's role is to turn it into an image. It's similar to the role of a decoder in an Autoencoder: it must decode some latent representation into an image. The codings have fairly few dimensions (e.g., 100), so they can't store information about every detail in the image, just the general aspect. Since the generator will need to build an image, we'll want to use Conv2D layers. But these take images as inputs, and all we have is a vector. So the first thing we need to do is to project this vector into 3D space (height, width, channels). But it's probably best to start by projecting them to a fairly low-resolution image (e.g., 7 x 7), since the codings can't hold much info. The generator's role will be to add the necessary details to make the image realistic. But we have more channels if we want: this way, all of the coding's information will be mapped to "semantic" information for each part of the image. For example, it may encode the fact that the top-left part of the image contains a leaf. It's fairly high level information. Then the Conv2DTranspose layers can increase the spatial resolution and gradually convert the high-level info into low-level information (i.e., pixel colors). If the images are monochrome, you want the last layer to have just 1 filter. If it's RGB, it should be 3.

My second question is why did we settle on a kernel_size of 5?

I usually use kernel size of 3, except sometimes in the very first layer of a CNN. I may have just tried out various combinations and found that this worked for me. I honestly can't remember. You can just try different kernel sizes and see what works best.

My next question is why, in the following code, do we multiply by 2 and subtract 1?
The inputs had values ranging from 0 to 1, and I wanted them to range from -1 to +1. Multiplying by 2 and subtracting 1 will do that. This is because neural nets tend to prefer when their inputs have a mean close to 0.

Hope this helps.

Thanks @ageron , that helps a lot!


Automated Package Upgrade

:tada: This PR is included in version 25.4.122 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


allow kubevirt VM work under container

What this PR does / why we need it:
We have EVE open-source project, https://github.com/lf-edge/eve, needs
to run kubernetes(k3s for example) inside a container. Kubevirt is part
of that setup. Although this patch should be needed for running k8s/kubevirt
inside a container in general.
This patch addresses two issues:

error: failed to get emulator capabilities, and
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing:

launch a VM in kubevirt, can not mount the disk
mount target invalid: failed opening kubelet for path root: /, relative: /var/lib/kubelet/pods/91a750fc-c0e5-4ea4-923a-7e86e831f599/volumes/kubernetes.io~empty-dir/container-disks/disk_0.img: no such file or directory

For the first issue, it seems it can be resolved by set capability
inside the 'node-labeller.sh' script. The capability is required, and
it will not make a difference if the bare-metal/VM has already enabled it.
For the 2nd issue, it seems rather than the normal bare-metal or VM cases,
just statically using '/proc/1/ns/mnt' for MountChroot won't work in
the case running kubernetes and kubevirt inside a container.
This patch is to get the pod's current process, and finds its parent processes.
If the parent process's name is 'containerd-shim', we then found the
mountNamespace and this '/proc//ns/mnt' instead to be used,
which is the namespace of the container running the kubernetes and kubevirt.
BTW, the similar issue also is addressed in the longhorn project, see the
PR: https://github.com/longhorn/go-iscsi-helper/pull/63
Special notes for your reviewer:
Release note:
None

@naiming-zededa great! Many thanks for your contribution, it will be great to be able to run kubevirt also in a containerized environment. I left a couple of comments.
Please consider also adding some unit tests.
/ok-to-test

@alicefr the PR is updated. thanks.
/retest

Hi @xpivarc @alicefr , anything else you can think of I need to do for this PR? thanks.

It looks good to me, just want to know @xpivarc's opinion about https://github.com/kubevirt/kubevirt/pull/9668/files#r1182161268
/lgtm

/retest-requiered

@xpivarc are you ok with the explanation above?
We had a similar PR/upstream commit in Longhorn: https://github.com/longhorn/go-iscsi-helper/pull/63,
which merged in longhorn release v1.4.2
Also some examples on a real host and in a container, running kubevirt with this patch,
both cases running kubevirt fine, all the pods running:

On Equinix bare-metal host:
root@pramod-k3s-node1:~# /usr/local/bin/kubectl get pod -n kubevirt
NAME                               READY   STATUS    RESTARTS   AGE
virt-api-57858d944b-cskpm          1/1     Running   0          2m36s
virt-controller-57d5466497-j4w77   1/1     Running   0          2m10s
virt-controller-57d5466497-tjlkb   1/1     Running   0          2m10s
virt-handler-n8xvd                 1/1     Running   0          2m10s
virt-operator-689d7d567f-6pfr4     1/1     Running   0          3m4s
virt-operator-689d7d567f-7rqw9     1/1     Running   0          3m4s

Inside a container 'eve-k3s' of an Alpine linux:
07ab2c40-408a-4bc2-b9f2-8ca94235074f:~# ps aux | grep "eve-k3s"
1792 root      0:00 /usr/bin/containerd-shim-runc-v2 -namespace services.linuxkit -id eve-k3s -address /run/containerd/containerd.sock

07ab2c40-408a-4bc2-b9f2-8ca94235074f:/# kubectl get pod -n kubevirt
NAME                               READY   STATUS    RESTARTS         AGE
virt-api-57858d944b-bzclm          1/1     Running   14 (16m ago)     2d5h
virt-controller-57d5466497-bdgx4   1/1     Running   15 (4h37m ago)   2d5h
virt-controller-57d5466497-x7hzv   1/1     Running   15 (4h37m ago)   2d5h
virt-handler-b6x52                 1/1     Running   9 (4h37m ago)    2d5h
virt-operator-689d7d567f-4r6vk     1/1     Running   26 (16m ago)     2d5h
virt-operator-689d7d567f-5dbqd     1/1     Running   25 (16m ago)     2d5h
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/#

hi, @alicefr @xpivarc , i'm not sure how to move this forward. what do you think we need to do here? thanks.

@naiming-zededa sorry for the late reply. Well, we need a green CI
/retest-required

/retest-required

@naiming-zededa sorry for the late reply. Well, we need a green CI /retest-required

Sorry @alicefr , I'm not familiar with how this retest works. I entered '/retest-required' and it does not seem to do anything. Do I miss something to trigger the re-test? thanks.

/retest

/test pull-kubevirt-fossa

Hi @alicefr , actually I was able to /retest the above filed 'pull-kubervirt-fossa', and found it repeatedly failed. but my puzzle is that, it complains about the error of git 'origin' remote missing, and also can not find executable 'svn', which seems to be is testing infra related. Although I don't understand why it happens only on my PR commit. Any help in this is appreciated. thanks.

rebased to the main again to see if the test can pass.

/retest

rebased to the main again to see if the test can pass.

Yes, usually this helps when there are issues unrelated to the PR like these.
Now, you only have the 2 storage lanes, but they are just flaky tests.

/retest-required

@naiming-zededa I'm fine with the part that finds the right path for chroot. Maybe you could remove the part that overwrites the capabilities. I do understand that doesn't solve your problem, but probably needs a more proper solution

@naiming-zededa I'm fine with the part that finds the right path for chroot. Maybe you could remove the part that overwrites the capabilities? I do understand that doesn't solve your problem but probably needs a proper solution

@alicefr , I have tried setting securityContext with 'privileged: true', 'capabilities: add: ["ALL"]', still can not get 'virt-handler' pod to run. This is with the environment of running inside a container with Alpine Linux. I have added the check in the 'node-labeler.sh' to do this setcap only if user explicitly defines a file '/var/lib/kubevirt-node-labeller/setcap-dac-override-enable', so that normal kubevirt users will not run into any issues, while if anyone runs kubevirt into a similar problem as we have found, there is at least a workaround, until of course if better solution can be found later.

I still think it is dangerous, if you manage to create that file then you basically activate the ability to bypass the permission checks.
This just hides the problem and doesn't solve it.

~This issue with dac_override looks very similar to: #8615 (comment). Though there it was Alpine on Equinix bare metal server.~
UPD: ah, yeah, I missed that the issue has been mentioned here already:

This is related to the reported issue: #8615

yes, we are working on the same EVE system. We opened the issue there, and our workaround PR is here.

Do I understand correctly that here we also have Alpine, though it is a container that runs k3s (single node) along with KubeVirt?

Yes, this is correct.

What is the kernel version on the host OS (refering to the comment #8615 (comment))?

the 'uname -a' output:
Linux 07ab2c40-408a-4bc2-b9f2-8ca94235074f 5.10.131-linuxkit #1 SMP Tue Jun 6 20:38:06 2023 x86_64 Linux

I still think it is dangerous, if you manage to create that file then you basically activate the ability to bypass the permission checks.
This just hides the problem and doesn't solve it.

@alicefr Thanks for looking into this issue. I spent long time on this bug as you can see in reported issue  #8615.
Please look at this code
https://github.com/libvirt/libvirt/blob/00cd7d2b1a9bd5a0d2f7555b7f54a6197af3c00f/src/qemu/qemu_process.c#L9402
Basically libvirtd before launching qemu clears all capabilities and sets DAC_OVERRIDE capability. For some reason that is not getting set in alpine systems and hence we explicitly added that in virt-handler as a reasonable workaround.
Please advice further on how to proceed.

/retest

@alicefr when QEMU runs into permission issues, the solution in 'libvirtd' code is also setting the CAP_DAC_OVERRIDE to get around this issue. Our patch here is following the similar approach from security point of view. With the file exist check in the latest patch, even if kubevirt users explicitly enables this, it is no worse than original code's intention I would think. thanks.

I still think it is dangerous, if you manage to create that file then you basically activate the ability to bypass the permission checks.
This just hides the problem and doesn't solve it.

@alicefr Thanks for looking into this issue. I spent long time on this bug as you can see in reported issue #8615.
Please look at this code https://github.com/libvirt/libvirt/blob/00cd7d2b1a9bd5a0d2f7555b7f54a6197af3c00f/src/qemu/qemu_process.c#L9402
Basically libvirtd before launching qemu clears all capabilities and sets DAC_OVERRIDE capability. For some reason that is not getting set in alpine systems and hence we explicitly added that in virt-handler as a reasonable workaround.
Please advice further on how to proceed.

Ah, now I understand! Many thanks for the explanation :)
This is run in a virt-handler that has the capability to set it, if it was in virt-launcher will fail. I also agree that it needs to be done in the script for the probing.
Please, a comment with the short explaination and the reference to libvirt code.
@xpivarc is it fine for you?

I am now trying to recap the issue https://github.com/kubevirt/kubevirt/issues/8615, as I was also trying to reproduce it (but without any luck, unfortunately), and take into account the recent findings (e.g. about cap_dac_override).
I remember that when working on SEV PR, I faced a permission issue when qemu was trying to access /dev/sev device for capabilities checking. To fix that, I added chmod o+rw /dev/sev, similar to what had been done previously for /dev/kvm:
if [ -e /dev/kvm ]; then
    chmod o+rw /dev/kvm
    VIRTTYPE=kvm
fi
if [ -e /dev/sev ]; then
  # QEMU requires RW access to query SEV capabilities
  chmod o+rw /dev/sev
fi

Now that I look at the mentioned libvirt code and see that according to the comment the capability is added to allow qemu accessing devices without permissions checking (e.g. /dev/sev which is 0600 by default), I think: might it be just the case that in Alpine image we get some other devices whose permissions prevent from probing it?
Apart from /dev/kvm and /dev/sev, I guess qemu also probes for (at least) /dev/vhost-net and /dev/net/tun. Do we have proper permissions on those when running on Alpine?

@vasiliy-ul here is my 'ls' inside the virt-handler:
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/# kubectl exec -it virt-handler-nx96l -n kubevirt -- /bin/sh
Defaulted container "virt-handler" out of: virt-handler, virt-launcher (init)
sh-5.1# ls -la /dev/kvm
crw-rw---- 1 root root 10, 232 Jul  7 06:50 /dev/kvm
sh-5.1# ls -la /dev/sev
ls: cannot access '/dev/sev': No such file or directory
sh-5.1# ls -la /dev/vhost-net
crw-rw---- 1 root root 10, 238 Jul  7 06:50 /dev/vhost-net
sh-5.1# ls -la /dev/net/tun
crw-rw---- 1 root root 10, 200 Jul  7 06:50 /dev/net/tun
sh-5.1#

@naiming-zededa, instead of doing setcap cap_dac_override=ep /usr/libexec/qemu-kvm, could you try just setting the proper access bits to the device files in the cmd/virt-launcher/node-labeller/node-labeller.sh:
diff --git a/cmd/virt-launcher/node-labeller/node-labeller.sh b/cmd/virt-launcher/node-labeller/node-labeller.sh
index 51b02546d..4f40983a3 100755
--- a/cmd/virt-launcher/node-labeller/node-labeller.sh
+++ b/cmd/virt-launcher/node-labeller/node-labeller.sh
@@ -32,6 +32,14 @@ if [ -e /dev/sev ]; then
   chmod o+rw /dev/sev
 fi

+if [ -e /dev/vhost-net ]; then
+    chmod o+rw /dev/vhost-net
+fi
+
+if [ -e /dev/net/tun ]; then
+  chmod o+rw /dev/net/tun
+fi
+
 virtqemud -d

 virsh domcapabilities --machine $MACHINE --arch $ARCH --virttype $VIRTTYPE > /var/lib/kubevirt-node-labeller/virsh_domcapabilities.xml

@vasiliy-ul thanks for the above diff. Actually I tried the exact diff yesterday on my setup, and it does not help and the error is the same unfortunately.
I did more debug into this area.

the error is due to this line in node-labeller.sh when launching 'virsh': virsh domcapabilities --machine ...
error: failed to get emulator capabilities
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing:

due to this line return error: https://github.com/libvirt/libvirt/blob/3bf02acdc5446b2c4a3078f99d8f5232acff9043/src/qemu/qemu_process.c#L9412

that was the reason above several lines has a check on 'WITH_CAPNG' and set 'CAP_DAC_OVERRIDE' to get around.
I believe that, if the libvert folks know the exact problem, they would not have put this there, so this issue seems not
so easy to debug

I believe that, when 'kubevirt' includes the libvirtd, it does not compile it with the '-Dcapng=enabled' flag, thus it won't
in the code above set this CAP_DAC_OVERRIDE. since the intention of the code is to allow user to set this if they see
the permission issues, but in this case we are using kubevirt public version, we don't get chance to modify this inside
the libvirtd outselves.

can i suggest that, to keep the same spirit as the original code of libvirt, we first let user in /var/lib/kubevirt-node-labeller/ to define a file WITH_CAPNG, and we check in the node-labeller.sh that
uid=$(id -u)
if [ -e "/var/lib/kubevirt-node-labeller/WITH_CAPNG" ] && [ "$uid" -eq 0 ]; then
setcap cap_dac_override=ep /usr/libexec/qemu-kvm
fi

will this be ok with all? thanks!

when 'kubevirt' includes the libvirtd, it does not compile with the '-Dcapng=enabled' flag

Not sure if this is the case as capng should be enabled in libvirt rpm for centos 9 stream:
https://gitlab.com/redhat/centos-stream/rpms/libvirt/-/blob/ff5db98bff1379f839f0f68700127534f11d3018/libvirt.spec#L1201

Actually I tried the exact diff yesterday on my setup, and it does not help and the error is the same unfortunately.

I see. Thanks for giving it a try. I think ideally it would be nice to strace qemu in order to see the exact failure. Though it might be tricky to do, and unfortunately I do not have a way to reproduce the issue. Still, I do not understand why I was not seeing the same error when I tried running KubeVirt on Alpine VM.

Hi @naiming-zededa ,
Few questions, are you using upstream Kubevirt images? Do they work if you do?
Would you be able to check device rules in the cgroup of the init container? Do you use v1 or v2?

Hi @naiming-zededa , Few questions, are you using upstream Kubevirt images? Do they work if you do?

I only run the upstream kubevirt when I debug with you. No, it does not work, thus we put in this PR.
check this: https://github.com/lf-edge/eve/blob/master/pkg/kube/kubevirt-operator.yaml#L6912
we are running my private kubevirt images temporarily.

Would you be able to check device rules in the cgroup of the init container?

☁  ~  k exec -it virt-handler-cvx6p -n kubevirt -- /bin/sh
Defaulted container "virt-handler" out of: virt-handler, virt-launcher (init)
sh-5.1# cat /proc/self/cgroup
13:name=systemd:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
12:pids:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
11:hugetlb:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
10:net_prio:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
9:perf_event:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
8:net_cls:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
7:freezer:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
6:devices:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
5:memory:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
4:blkio:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
3:cpuacct:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
2:cpu:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a
1:cpuset:/kubepods/burstable/poda472a805-f345-4984-bc7c-d3a0e6126093/3156de467e902e3971c4b84914a306afba5c5dc6a601f3b9959115d414086f3a

Do you use v1 or v2?

what is this v1 or v2?

@vasiliy-ul thanks for the above diff. Actually I tried the exact diff yesterday on my setup, and it does not help and the error is the same unfortunately.
I did more debug into this area.

the error is due to this line in node-labeller.sh when launching 'virsh': virsh domcapabilities --machine ...
error: failed to get emulator capabilities
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing:

due to this line return error: https://github.com/libvirt/libvirt/blob/3bf02acdc5446b2c4a3078f99d8f5232acff9043/src/qemu/qemu_process.c#L9412

that was the reason above several lines has a check on 'WITH_CAPNG' and set 'CAP_DAC_OVERRIDE' to get around.
I suspect that, if the libvert folks knew the exact problem, they would not have put this code there, so this issue seems
not so easy to debug
as I understand, when 'kubevirt' includes the libvirtd, it does not compile with the '-Dcapng=enabled' flag,
thus it won't be in the code above to set this CAP_DAC_OVERRIDE. Since the intention of the code is to allow user
to set this CAP if they see the permission issues, but in this case we are using kubevirt public version,
we don't get chance to modify this inside the libvirtd outselves. If kubevirt can maintain another set of images
in which the 'libvirtd' is compiled with the above flag, that should work too for users like us.
Can i suggest that, to keep the same spirit as the original code of libvirt, we first let user in
/var/lib/kubevirt-node-labeller/ to define a file WITH_CAPNG, and we check in the node-labeller.sh that

uid=$(id -u) if [ -e "/var/lib/kubevirt-node-labeller/WITH_CAPNG" ] && [ "$uid" -eq 0 ]; then setcap cap_dac_override=ep /usr/libexec/qemu-kvm fi
will this be ok with all? thanks!

@andreabolognani @zippy2 do you have an idea what is the reason for this failure?
I agree with Vasily, strace could help a lot in this case. @naiming-zededa can you build kubevirt from source code?

I see. Thanks for giving it a try. I think ideally it would be nice to strace qemu in order to see the exact failure. Though it might be tricky to do, and unfortunately I do not have a way to reproduce the issue. Still, I do not understand why I was not seeing the same error when I tried running KubeVirt on Alpine VM.

@vasiliy-ul We operate this with Alpine OS and inside also the container. As Pramodh reported that Alpine on bare-metail
for some version works, but others having the same permission issues. With this kubevirt running inside the
container and in Alpine, we see this permission problem all the time.

I agree with Vasily, strace could help a lot in this case. @naiming-zededa can you build kubevirt from source code?

@alicefr I do build kubevirt with the patch of this PR for our system to work. and also for debugging.
but I've never tried strace into this init-container thing, not sure how to debug with that.

I think something like the diff bellow can do strace of virtqemuq / qemu-kvm:
diff --git a/cmd/virt-launcher/node-labeller/node-labeller.sh b/cmd/virt-launcher/node-labeller/node-labeller.sh
index 51b02546d..d05589a62 100755
--- a/cmd/virt-launcher/node-labeller/node-labeller.sh
+++ b/cmd/virt-launcher/node-labeller/node-labeller.sh
@@ -32,7 +32,9 @@ if [ -e /dev/sev ]; then
   chmod o+rw /dev/sev
 fi

-virtqemud -d
+virtqemud -d -p virtqemud.pid
+
+strace -ff --output=/var/lib/kubevirt-node-labeller/trace -p $(cat virtqemud.pid) &

 virsh domcapabilities --machine $MACHINE --arch $ARCH --virttype $VIRTTYPE > /var/lib/kubevirt-node-labeller/virsh_domcapabilities.xml

diff --git a/hack/rpm-deps.sh b/hack/rpm-deps.sh
index 5a89c457d..efd263daa 100755
--- a/hack/rpm-deps.sh
+++ b/hack/rpm-deps.sh
@@ -111,6 +111,7 @@ launcherbase_extra="
   procps-ng
   selinux-policy
   selinux-policy-targeted
+  strace
   tar
   virtiofsd-${VIRTIOFSD_VERSION}
   xorriso

Though it requires running make rpm-deps and this is currently broken
Error: Package qemu-img-17:7.2.0-14.el9 does not exist

Any thought on how to quickly add strace to virt-launcher image? As an option, there is a doc https://github.com/kubevirt/kubevirt/blob/main/docs/custom-rpms.md

Any thought on how to quickly add strace to virt-launcher image? As an option, there is a doc https://github.com/kubevirt/kubevirt/blob/main/docs/custom-rpms.md

We could change QEMU version (using 8.0.0 as in this PR). You can customize the version setting the variable QEMU_VERSION with make QEMU_VERSION=17:8.0.0-6.el9 rpm-deps

We could change QEMU version (using 8.0.0 as in this PR). You can customize the version by setting the variable QEMU_VERSION with make QEMU_VERSION=17:8.0.0-6.el9 rpm-deps

Yep, with the updated qemu make rpm-deps now works :+1: Let's hope this will not somehow affect this usecase in a strange way.
@naiming-zededa, could you cherry-pick the commit https://github.com/vasiliy-ul/kubevirt/commit/af3a1cb404b16beb6ad2197313f61f9c7b400cba and try to reproduce the issue? With this change, the strace logs for virtqemud and its derived tasks will be collected in a tmp directory under /var/lib/kubevirt-node-labeller/tmp.*. One of the files should call execve(/usr/libexec/qemu-kvm) and hopefully it will contain some useful info.

@naiming-zededa, could you cherry-pick the commit vasiliy-ul@af3a1cb and try to reproduce the issue? With this change, the strace logs for virtqemud and its derived tasks will be collected in a tmp directory under /var/lib/kubevirt-node-labeller/tmp.*. One of the files should include execve(/usr/libexec/qemu-kvm) and hopefully it will contain some useful info.

@vasiliy-ul will try the patch to see.

@naiming-zededa, you might be running some old builder containers. Try to check with docker ps -a or podman ps -a and remove those. If that does not help, sometimes it helps to prune the volume kubevirt.
Another command (pid=15) is running. Waiting for it to complete on the server (server_pid=19)...
INFO: Analyzed target //:gazelle (91 packages loaded, 9141 targets configured).
INFO: Found 1 target...
ERROR: : BazelWorkspaceStatusAction stable-status.txt failed: Failed to determine workspace status: Process exited with status 1
hack/print-workspace-status.sh: line 36: KUBEVIRT_GIT_VERSION: unbound variable
Target //:gazelle failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 12.327s, Critical Path: 0.13s
INFO: 1 process: 1 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
make: *** [Makefile:199: format] Error 1

@vasiliy-ul BTW, since i could not build the latest with your patch, and I ported your patch in node-labeller.sh, and hack/rpm-deps.sh into 0.59.0 based code. did a 'make rpm-deps', build the image and applied the operator.yaml to
my system. when i looked at the node-labeller directory, at the time it contained 4 tmp.xxx sub-directories. But all of
them are empy, no trace file in them. I suspect the 'strace' in the patch failed to run for some reason. I tar'ed the
directory:
node-labeller-debug.tar.gz

Right, this can also work. You need to take the content node-labeller.sh and hack/rpm-deps.sh and then run make rpm-deps. Otherwise, strace will not be pulled into the image.

@naiming-zededa, if make rpm-deps fails with an error about EDK2 then try to change the following variable:
EDK2_VERSION=${EDK2_VERSION:-0:20230301gitf80f052277c8-2.el9}

Right, this can also work. You need to take the content node-labeller.sh and hack/rpm-deps.sh and then run make rpm-deps. Otherwise, strace will not be pulled into the image.

I did the above, the the trace file is not there in those tmp.xxx dirs.

Did you run make rpm-deps? Has it succeeded?

Could you check kubectl logs -n kubevirt virt-handler-... virt-launcher?

Did you run make rpm-deps? Has it succeeded?

yes, i did. here is my diff for the 0.59.0
patch-v0.59.0.diff.txt

The diff looks correct. So strace should be installed in the images. Yesterday I checked the patch and the traces were collected. We should probably look at the logs of the virt-launcher init container. Maybe there is some error there kubectl logs -n kubevirt virt-handler-... virt-launcher.

yeh, the 'strace' not found:
☁  Desktop  k logs -n kubevirt virt-handler-gfcrg virt-launcher
/usr/bin/node-labeller.sh: line 25: strace: command not found
error: failed to get emulator capabilities
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing:

In your diff for 0.59.0: I think you added strace to handlerbase_extra. It should be launcherbase_extra

@vasiliy-ul is it possible you can pass me the kubevirt-operator.yaml which you built (assume they can be downloaded) and i can try that.

In your diff for 0.59.0: I think you added strace to handlerbase_extra. It should be launcherbase_extra

let me check.

thanks for spot that. rebuilding now.

Well, I am a bit surprised, but I see
openat(AT_FDCWD, "/dev/null", O_RDWR|O_CLOEXEC) = -1 EACCES (Permission denied)

@naiming-zededa, could you check the access permissions of /dev/null?

@vasiliy-ul here is the one in the container runs k3s:
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/# ls -lt /dev/null
crw-rw----    1 1001     root        1,   3 Jul 10 20:28 /dev/null
and inside the virt-launcher:
k exec -it virt-launcher-my-pvc-vm2-snpjq -n eve-kube-app -- /bin/sh
sh-5.1$ ls -l /dev/null
crw-rw-rw- 1 qemu root 1, 3 Jul 10 20:43 /dev/null
sh-5.1$

and inside the virt-launcher:
k exec -it virt-launcher-my-pvc-vm2-snpjq -n eve-kube-app -- /bin/sh
sh-5.1$ ls -l /dev/null
crw-rw-rw- 1 qemu root 1, 3 Jul 10 20:43 /dev/null

I wonder, how you get /dev/null owned by qemu user.... Do you have your container runtime configured with smth like device_ownership_from_security_context = true? Is it containerd?

virt-handler inside is this:
k exec -it virt-handler-qgpsj -n kubevirt -- /bin/sh
Defaulted container "virt-handler" out of: virt-handler, virt-launcher (init)
sh-5.1# ls -lt /dev/null
crw-rw---- 1 root root 1, 3 Jul 12 07:19 /dev/null
sh-5.1#
I'll try the chmod o+rw on this /dev/null in the node-label.sh, and see what happens.

and inside the virt-launcher:
k exec -it virt-launcher-my-pvc-vm2-snpjq -n eve-kube-app -- /bin/sh
sh-5.1$ ls -l /dev/null
crw-rw-rw- 1 qemu root 1, 3 Jul 10 20:43 /dev/null

I wonder, how you get /dev/null owned by qemu user.... Do you have your container runtime configured with smth like device_ownership_from_security_context = true?

From what i know of this, we don't do special things for the container, the build yaml is like:
org: lfedge
image: eve-kube
config:
binds:
- /lib/modules:/lib/modules
- /dev:/dev
- /etc/resolv.conf:/etc/resolv.conf
- /run:/run
- /config:/config
- /:/hostfs
- /persist:/persist:rshared,rbind
- /sys/fs/cgroup:/sys/fs/cgroup
capabilities:
- all
pid: host
rootfsPropagation: shared
devices:
- path: all
type: a

I'll try the chmod o+rw on this /dev/null in the node-label.sh, and see what happens.

Ok, thanks. I think it may still fail, but I expect that it will be a different error. So in such case the traces should help us make a step forward.

From what i know of this, we don't do special things for the container, the build yaml is like:

Well, frankly speaking, the permissions that you have for /dev/null on your node do not seem right to me. Usually /dev/null is globally accessible, while in your case it's RW only for the user and group. IMHO, in general, I do not believe the permissions of /dev/null should be tweaked by KubeVirt in any way. Probably that needs to be handled on the host system.

with this diff, it works now:
~/go/src/github.com/kubevirt-v0.59.0$ git diff cmd/virt-launcher/node-labeller/node-labeller.sh
diff --git a/cmd/virt-launcher/node-labeller/node-labeller.sh b/cmd/virt-launcher/node-labeller/node-labeller.sh
index e3888dcd9..81d7d91ab 100755
--- a/cmd/virt-launcher/node-labeller/node-labeller.sh
+++ b/cmd/virt-launcher/node-labeller/node-labeller.sh
@@ -19,7 +19,17 @@ if [ -e /dev/sev ]; then
   chmod o+rw /dev/sev
:...skipping...
diff --git a/cmd/virt-launcher/node-labeller/node-labeller.sh b/cmd/virt-launcher/node-labeller/node-labeller.sh
index e3888dcd9..81d7d91ab 100755
--- a/cmd/virt-launcher/node-labeller/node-labeller.sh
+++ b/cmd/virt-launcher/node-labeller/node-labeller.sh
@@ -19,7 +19,17 @@ if [ -e /dev/sev ]; then
   chmod o+rw /dev/sev
 fi

-virtqemud -d
+if [ -e /dev/null ]; then
+  # QEMU requires RW access to query /dev/null capabilities
+  chmod o+rw /dev/null
+fi
+
+
+virtqemud -d -p virtqemud.pid
+# if it goes into a crash loop, use separate tmp dirs to store the traces
+OUTPUT_DIR=$(mktemp -d -p /var/lib/kubevirt-node-labeller)
+strace -ff --output=${OUTPUT_DIR}/trace -p $(cat virtqemud.pid) &
+

 virsh domcapabilities --machine q35 --arch x86_64 --virttype $VIRTTYPE > /var/lib/kubevirt-node-labeller/virsh_domcapabilities.xml

so, is it ok to modify this PR with this /dev/null change?

Good, seems that we managed to figure out what is causing the failure.

so, is it ok to modify this PR with this /dev/null change?

Please refer to my message above:

Well, frankly speaking, the permissions that you have for /dev/null on your node do not seem right to me. Usually /dev/null is globally accessible, while in your case it's RW only for the user and group. IMHO, in general, I do not believe the permissions of /dev/null should be tweaked by KubeVirt in any way. Probably that needs to be handled on the host system.

After a quick googling, found this https://github.com/opencontainers/runc/pull/3707 (may also affect this issue, but needs a closer look).

After a quick googling, found this opencontainers/runc#3707 (may also affect this issue, but needs a closer look).

Thanks, this seems possible. the system we are running has Linuxkit with runc of last years date. But I have
verified that, setting o+rw in the host side works also. I'll just remove the setcap in the kubevirt-node-labeller.sh.

setting o+rw in the host side works also

Yes, that would be my assumption as well: the k3s node inherits permissions of /dev/null from the host and KubeVirt - from the node container. So if it's the case that the permissions on the host are not set properly, likely those will be propagated to all the containers.
In general, I would suggest taking a closer look at this issue from the host side. There might be other devices that do not have proper access bits and that could potentially hit again somewhere in the future.

/retest

@vasiliy-ul @naiming-zededa great investigation!
@naiming-zededa you might take a look to https://github.com/kubevirt/kubevirt/blob/main/cmd/virt-handler/virt-handler.go. This file should give you a hint which devices to check.
@vasiliy-ul we correct the relabeling for the devices including /dev/null, do you think we should also set the permissions we need?

@vasiliy-ul we correct the relabeling for the devices including /dev/null, do you think we should also set the permissions we need?

@alicefr, in this particular case, /dev/null is world-inaccessible, and I believe the issue is in fact on the host side. I am not sure that it would be a good way forward to accumulate workarounds for external bugs in KubeVirt if those can be fixed properly in an inexpensive manner (i.e. the fix is straightforward and relatively easy/quick to apply). I am not strictly opposed to setting o+rw on /dev/null 'just in case, to be on the safe side', but that way at some point we may end up in tweaking most of the /dev/* which I do not see smth that KubeVirt should do.

@vasiliy-ul we correct the relabeling for the devices including /dev/null, do you think we should also set the permissions we need?

@alicefr, in this particular case, /dev/null is world-inaccessible, and I believe the issue is in fact on the host side. I am not sure that it would be a good way forward to accumulate workarounds for external bugs in KubeVirt if those can be fixed properly in an inexpensive manner (i.e. the fix is straightforward and relatively easy/quick to apply). I am not strictly opposed to setting o+rw on /dev/null 'just in case, to be on the safe side', but that way at some point we may end up in tweaking most of the /dev/* which I do not see smth that KubeVirt should do.

To add any container workload trying to access /dev/null would fail as well. In other words, this is not really a Kubevirt issue.

@vasiliy-ul @naiming-zededa great investigation! @naiming-zededa you might take a look to https://github.com/kubevirt/kubevirt/blob/main/cmd/virt-handler/virt-handler.go and https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-handler/device-manager/device_controller.go. These files should give you a hint which devices to check.

yes @alicefr , I checked those device files, and tried also 'o+rw' on them just like for /dev/kvm,
but they don't seem to cause any issues as the 'dev/null'. I think from our system point of view, we are
ok on this. we'll certainly double check them if we run into other issues in the future. thanks.

@alicefr coming in late as I was away for a few days. Do I understand correctly that the issue has ultimately been determined to be a host-side misconfiguration and changes or clarifications are no longer needed from my side?

@vasiliy-ul @alicefr @xpivarc PR is updated, please see if this is ok now. thanks.

/retest-required
@naiming-zededa, there are some failing unit-tests (related to this change). Also, the sig-compute job looks suspicious:
Unexpected Warning event received: vmi-kernel-boot-4hqm4,30fd328c-48c8-4481-8d4b-5df8bbfa4ff0: error mounting kernel artifacts: failed to find socket path for kernel artifacts: kernel boot socket path not found

but it may be a flaky test either. Let's see if it passes on the second run.

/retest

/retest

Please check the inline comments. Also, you have some changes in your working source tree:
ERROR: git tree state is not clean!

On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   pkg/virt-handler/container-disk/BUILD.bazel

You need to commit those and resubmit, otherwise the checker will fail.

Yes, i'm struggling with the compiling. my system can not 'make' the latest, i had to port it to v0.59.0 for it to compile.
thus missed that.

/test pull-kubevirt-unit-test

/approve
/retest-required
The failures in the CI do not seem related to the changes here. From my perspective, the PR now looks okay, and I do not see potential issues with the chosen approach. @alicefr, @xpivarc WDYT?

/retest-required

/retest pull-kubevirt-e2d-k8s-1.26-sig-storage

/retest-required

/approve /retest-required
The failures in the CI do not seem related to the changes here. From my perspective, the PR now looks okay, and I do not see potential issues with the chosen approach. @alicefr, @xpivarc WDYT?

Hi @vasiliy-ul , @alicefr , @xpivarc can we move this forward if there are no other issues of this PR? thanks.

Ping @xpivarc, @alicefr. So, what do you guys think about the latest update here? The CI is green, and I think this should be safe to get in. Any potential problem with the current approach you can think of?

/retest pull-kubevirt-manifests
/retest pull-kubevirt-build
/retest pull-kubevirt-e2e-k8s-1.28-sig-compute-migrations
/retest pull-kubevirt-e2e-k8s-1.28-sig-compute

/test pull-kubevirt-manifests
/test pull-kubevirt-build
/test pull-kubevirt-e2e-k8s-1.28-sig-compute-migrations
/test pull-kubevirt-e2e-k8s-1.28-sig-compute

/cc @vladikr

/cc @vladikr

@naiming-zededa Formatting needs to be changed for this code to be merged.
I was also told this in the community meeting, "Litte conservative about ldtm, need to look this PR with fresh eyes"

/remove-lifecycle rotten

The code looks good to me and I don't see any special drawback in getting the parent pid
/lgtm
/hold
to let the other review it

/release-note none

/release-note-none

@alicefr @xpivarc In general, I don't see any obvious issues.
I do wonder why the same is working with Kind without any required changes, already?

@alicefr @xpivarc In general, I don't see any obvious issues. I do wonder why the same is working with Kind without any required changes, already?

My 2cents. We are running OCI containers, and i went to host's '/proc/1/ns' and inside 'kube' container's /proc/1/ns, the display is almost identical:
on host:
f5677edf-a1d8-45bf-b72b-9124e80ff4ba:/proc/1/ns# ls -lt
total 0
lrwxrwxrwx    1 root     root             0 Sep 20 01:36 cgroup -> cgroup:[4026531835]
lrwxrwxrwx    1 root     root             0 Sep 20 01:36 ipc -> ipc:[4026531839]
lrwxrwxrwx    1 root     root             0 Sep 20 01:36 mnt -> mnt:[4026531841]
lrwxrwxrwx    1 root     root             0 Sep 20 01:36 net -> net:[4026531840]

in 'kube' container:
[kube] root@f5677edf-a1d8-45bf-b72b-9124e80ff4ba:/proc/1/ns$ ls -lt
total 0
lrwxrwxrwx    1 root     root             0 Sep 20 01:35 cgroup -> cgroup:[4026531835]
lrwxrwxrwx    1 root     root             0 Sep 20 01:35 ipc -> ipc:[4026531839]
lrwxrwxrwx    1 root     root             0 Sep 20 01:35 mnt -> mnt:[4026531841]
lrwxrwxrwx    1 root     root             0 Sep 20 01:35 net -> net:[4026531840]

I also launched a KIND on a bare-metal, run kubevirt also, it works as you said. When i compare
their /proc/1/ns, they are totally different
on host:
naiming@naiming-test-c3small:/proc/1/ns$ sudo ls -l
total 0
lrwxrwxrwx 1 root root 0 Sep 20 00:34 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Sep 20 01:18 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 root root 0 Sep 20 00:34 mnt -> 'mnt:[4026531841]'
lrwxrwxrwx 1 root root 0 Sep 20 01:18 net -> 'net:[4026531840]'

in KIND container:
root@naiming-kind-control-plane:/proc/1/ns# ls -l
total 0
lrwxrwxrwx 1 root root 0 Sep 20 01:36 cgroup -> 'cgroup:[4026532665]'
lrwxrwxrwx 1 root root 0 Sep 20 01:36 ipc -> 'ipc:[4026532603]'
lrwxrwxrwx 1 root root 0 Sep 20 01:36 mnt -> 'mnt:[4026532601]'
lrwxrwxrwx 1 root root 0 Sep 20 01:36 net -> 'net:[4026532605]'

so i suspect that in KIND, has it's own init running, when the kubevirt code trying to get to the /proc/1/ns/mnt, that is the correct one.

/retest-required

/test pull-kubevirt-e2e-k8s-1.30-sig-compute-serial

above test failed, seems not available for version 1.30:
pull-kubevirt-e2e-k8s-1.30-sig-compute-serial

Hi @xpivarc @alicefr @vasiliy-ul this PR is updated, please see if this is fine now. thanks.


Use EarlyPassLints from rustc (and clippy)

Some lints in rustc work via implementing the EarlyPassLint trait. This trait only needs ast level data and not types or hir level data, so it should be easy to port it, I think.
If this kind of lint implementation (each lint provides some functions, for example check_expr, check_block, check_ident, ... function, that RA should call it for each changed expression, block, ...) is ok performance-wise for ide usage (I think a small change leads to small expressions become invalidated, but I'm not sure), reusing them from rustc and clippy would save some effort and bring some consistency between rustc and RA diagnostics. With future unification of RA and rustc, we can think about porting other lints as well.

list of rustc lints that implements EarlyPassLint: link
list of clippy lints that implements EarlyPassLint: link (Some of these lints are already implemented in RA, like redundant field name, we may want to select a few others)

The AST in rust-analyzer allows partial or malformed code. The AST in rustc requires the code to parse.

What rustc calls AST is already name-resolved and macro-expanded, isn't it? That's very different from our AST. Doesn't mean that we couldn't provide it, we probably do have all the information to do so.

What rustc calls AST is already name-resolved and macro-expanded, isn't it?

Name-resolution for regular things happens during HIR lowering. Name-resolution for macros happens during macro expansion. Early lint passes happen after macro-expansion at https://github.com/rust-lang/rust/blob/3b263ceb5cb89b6d53b5a03b47ec447c3a7f7765/compiler/rustc_interface/src/passes.rs#L502-L510

Ah, right, so it is already macro-expanded? I got a bit confused because of https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/ast/struct.MacCall.html.

The AST may or may not be macro-expanded. The parser returns non-macro-expanded AST. EarlyPassLints however run after macro expansion has already happened.

The AST in rust-analyzer allows partial or malformed code. The AST in rustc requires the code to parse.

There should be a translate step between RA's ast and rustc's one, since they are too different. There is an error variant on rustc, so translation can be one to one, but we probably don't want to call lint methods with partial expressions (or even in presence of partial code).
Types of AST in rustc are some recursive structs, with P<T> as an owned ptr for indirection. Since fields of those structs are public, how we can prevent the translation to become deep and costly? Can we hijack P to provide lazy result?


Failed to create more than one volume in a batch

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
Failed to create more than one volume in a batch.
Some volumes get error status.
What you expected to happen:
Volumes should be able to create in a batch.
How to reproduce it (as minimally and precisely as possible):
Run multi "osdsctl create volume 1 --name=test" at once, or create multi pvc in a single yaml file.
Then check the status by "osdsctl volume list" or "kubectl get pvc"
Anything else we need to know?:
Root cause:
in pkg/dock/client/client.go
func (c *DockClient) Connect(edp string) error {
	// Set up a connection to the Dock server.
	conn, err := grpc.Dial(edp, grpc.WithInsecure())
	if err != nil {
		log.Errorf("did not connect: %+v\n", err)
		return err
	}
	// Create dock client via the connection.
	c.FileShareDockClient = pb.NewFileShareDockClient(conn)
	c.ProvisionDockClient = pb.NewProvisionDockClient(conn)
	c.AttachDockClient = pb.NewAttachDockClient(conn)
	/****************Root Cause******************/
	/* conn was saved to c.ClientConn */
	/* For example, if we have two request, we will have two conn, and c.ClientConn always saved the last one*/
	/* When we want to close the first conn, we call DockClient.Close, whiech call c.ClientConn.Close, which would close the second one*/
	c.ClientConn = conn

	return nil
}

func (c *DockClient) Close() {
	c.ClientConn.Close()
}

Environment:

Hotpot(release/branch) version: latest
OS (e.g. from /etc/os-release): Oracle Linux 7.7
Kernel (e.g. uname -a):
Install tools:
Others:

Fixed in PR #1177


Add salesforce docs

@odinuv Can you please merge? Also is it ok if I keep adding you to the pull request or should I ask someone new each time?

Ideally mention it together with the publication request as part of the checklist https://developers.keboola.com/extend/publish/checklist/

That way it goes out at the same time with the component and leaves me out of the loop :)


Send params as base64 encoded string

Removes all the StateDecodeParams calls to Lotus and send the parameters as base64 encoded string.
Let's discuss how to roll this out, but would be great to have this on calibrationnet so I can do some tests with the front-end.

I'm assuming this was tested locally, but my one question would be about base64 encoding params - are we double base64

I haven't been able to run graph successfully. I can build the Docker image, but when I run the container it fails to connect to the database. I'm not sure where I can find the complete instructions to run the whole project locally, but since everything compiles properly I assume that there aren't any issues.
The params are byte arrays in Golang, so they're encoded to base64 strings.

I don't think we should remove unused properties. If you don't need them, don't ask for them client side. The graph will only resolve the gascost, receipt...etc if you ask for them.

The reason for this was to improve usability of the TypeScript types that are generated from the GraphQL schema.
For example, TypeScript creates a Message type based on the full Message object as defined in the schema. When selecting a subset of the Message properties, the query result does not pass as a valid Message object in TypeScript because it is missing some properties.
We can work around this by manually defining the query result types, which is not that bad either. For example:
type GqlMessage = MessageQuery['message']

The GqlMessage type will then just contain the properties that were requested in our message query and we can use that type instead of the auto-generated Message type.
My initial feeling is that we should keep it as simple as possible for ourselves because I think we are the only clients of this server. So if we don't ever request the version, gasCost and receipt properties in Message, my intuition would be to remove them. But I'm fine with the other approach as well, so happy to revert the last commit if you feel strongly for keeping all the properties.
@Schwartz10 any thoughts on this?

@navFooh i spoke with ganzai about it, and he made a good point about not having typescript warrant the flexibility of our graphql backend. Graphql is built so that we can just request any data we want - if we don't need it, simply don't request it, but it is conceivable that one time in the futuer we would want to execute the message query with those other queries included, and it would just be annoying to have to add that back and then fix all the types in a different way again in the future.
I'd vote to go with your second approach, defining the message type itself


invalid

everytime it showing coockie invalid...

Use fresh cookies
On Mon, 06 Mar 2023, 10:31 AM mortaza01, @.***> wrote:

everytime it showing coockie invalid...
—
Reply to this email directly, view it on GitHub
https://github.com/RED-DEMON-ANNOS/Beast-Dump/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZDF4AVVW5EVPZBRG7FSOC3W2VZEZANCNFSM6AAAAAAVQVDV5Q
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

used fresh cookies from 2 account after login with id password.... not working....

Ok i will solve this problem


nodejs runtime out of support

The nodejs runtime for many of the examples are out of date.  They are based on nodejs 6.10 and they must be updated to nodejs 8.10 or 10.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Thank you for raising this issue. The nodejs based Cloud Formation templates have been updated.


alt.Vector3 is not working inside worker thread

Description of the problem
Some methods do not seem to work on Vector3 within a worker thread. In my example it is the distanceTo method.
Reproduction steps

create a clientside worker
Try to use Vector3 methods:

console.log('A ', currentVector instanceof alt.Vector3, distance);
console.log('B ', currentPlayerPosition instanceof alt.Vector3, distance);

[20:32:45] A  true NaN
[20:32:45] B  true NaN

Expected behaviour
DistanceTo should return a number
Additional context
No response
Operating system
Windows 11
Version
alt:V 12.13
Scope
client

I think we can close this, since workers are deprecated with alt:V 13.0


Ask question without passing context passage

I am using BERT QnA (run_squad.py) . I passed 1 passage in context and 1 question. On a TPU it took 22 seconds.
In my use case the context passage is fixed and questions are coming one after another. 22 seconds (on TPU and 66 seconds on CPU) is very high.
Is it possible that I just ask question on a predefined context passage, so that prediction time is reduced.

I have splitted th script at if FLAGS.do_predict
Time has been reduced to 7 seconds on TPU.. it is better but still high

I am not sure why I am not getting reply. However, till now I have figured out that running local_init_op is taking most of time.  Is there anybody to help?

If you are using an estimator it will be loading the model each time.

Wondering if there is a non estimator type code available for tensorflow, which we can tweak. Tensorflow is a learning curve, unlike pytorch.

You can use Tensorflow Serving for quick inferring.. and it works reasonably good with CPU only machine as well.


[R] Add link to ?acero from ?list_compute_functions

Describe the enhancement requested
As shown here, users may end up in list_compute_functions() docs when they actually would find the docs that come up calling ?acero more helpful.  Let's update the former to add a link to the latter.
Component(s)
R

Issue resolved by pull request 44210
https://github.com/apache/arrow/pull/44210


DCGS-A Military Overlay assistance with portal publishing workflow

Yesterday, I recieved an email from an account manager (DM) asking to assist contractors working with his customer on getting our Military Symbology solution working. It looks like they are having trouble publishing feature services that are using the dictionary renderer to display 2525B Change 2 symbols from ArcGIS Pro 1.2 to their instance of Portal 10.4.1. Here is the specific text from that email:
I wanted to follow up on the walk-through of the 2525b symbology the other week with an issue I am having regarding publishing/sharing feature layers to Portal when using the military templates.  When I am trying to share the overlay map as a web layer, it appears that nothing is happening.  I don’t get a progress bar or any errors, or any other indication of success or failure, the cursor just spins for a couple of minutes and nothing happens.  I have copied the features into a EGDB and changed the data source, removed all group layers and have no other errors or warnings from the share pane.
Do you have an idea of what could be causing this?
It sounds like it might just be an issue with their configuration of portal, not necessarily with our template. Regardless, we should figure out how to help them. We've set up a meeting tomorrow to help them out and I could use someone who is good at debugging portal configurations to join me in this. Perhaps @elinz or @ACueva. I'll update this issue after our meeting tomorrow.
@lfunkhouser @nfeuerstein FYI I'm putting this issue here so that the team can be made aware of our continuing support in getting our military symbology solutions to work with his customer.

Thank you @Dbarnes1 for making this issue visible!
It sounds like, at this point, it is still a conversation in order to identify the root cause of the issue.  If a BUG is identified, after the conversation tomorrow, then please log it here and add it to the backlog.  I'll prioritize....and it will be a high priority.

@Dbarnes1 Who is "we"?  First thing, ArcGIS Server logs need to be viewed.  We should be able to get clearer indication of what exactly is going on.
@nfeuerstein @lfunkhouser, my initial thoughts are we should involve support.  Let them be the first line of defense.  If they identify the issue is with our solutions then we can hop in and assist.  At this point it could just be an issue with their ArcGIS Server/Portal.
Either way, @Dbarnes1 if you send me a meeting invite I can try to make myself available.

@ACueva I'm currently working with @elinz on the publishing workflow. Looks like the meeting may get pushed back anyway, but I"ll send you an invite.
The reason I didn't redirect to support is because this template has been released, but the most recent version (2525Bc2 support in Pro 1.2) is the one that we handed off to the customer directly outside of our normal release cycle. One of our remaining PBI's for the November/December release is to release this template on the solutions site, at which point it will become tech support's domain.

@Dbarnes1 No worries on the invite.  I'm sure you're in good hands with @elinz.
The question was more for @nfeuerstein and @lfunkhouser to chime in.  In my opinion we need to figure out whether or not the customer has a functioning ArcGIS Server/Portal configuration prior to us being called in.  The workflow of publishing a service from ArcGIS Pro may actually be what's failing.  We risk getting tied up on an issue, troubleshooting it, and then realizing they have other underlying issues with their environment.  This is support's role.  I'm not saying we won't help, I'm simply saying not yet.

@nfeuerstein I had discussed this earlier with @Dbarnes1 and this template has not been released.

And that's our answer @ACueva - in order to send our customers to support first, the template must be released.

@nfeuerstein I just finished testing the same template the customer has and I've been able to successfully publish a feature/map service using 2525D symbology to 10.4.1 final server/portal. At this point, I don't believe it's an issue with the template, but possibly an issue with their server/portal environment. The issue that the customer is having is with running the out of the box "Share as Web Layer" publishing function; all the template is doing is opening this function for the user. Issues with publishing can be handled by tech support, and the account manager can certainly initiate this with tech support and elevate the priority of this issue. That's not to say that @ACueva or myself aren't willing to help. We just want to make sure that we are responding to this in the most efficient and effective way.
I would suggest to the customer to test to see if it's a general publishing issue, by creating a new ArcGIS Pro project (not use this template), add one of the enterprise geodatabase feature classes that was part of the Pro template and try to publish. The results of this test will help move this forward. I can make this suggestion to the account manager.

@elinz @Dbarnes1 @ACueva - thank you for looking into it, and I think your suggestion is spot on.


[In-Person Event] SOSS Community Day Europe 2024

In-Person Event Submission
Requirements

Event Name: SOSS Community Day Europe 2024
Link to the Event Website: https://events.linuxfoundation.org/soss-community-day-europe/
Are you an organizer of this event? Yes, Event Planner

Event Demographics

[x] This event commits to Event Diversity and Inclusion.

Q Detail the process for measuring Event Demographics.
A In our registration process to better cater to our attendees, we ask the following questions: Do you have a disability that we should be mindful of as we try to accommodate everyone for this event? Please specify dietary needs (if any). If you have a dietary need that isn’t listed, email us at<EMAIL_ADDRESS>and we will make every reasonable effort to accommodate your need(s). Please note, the questions are not required.  For speakers in the CFP process, we ask several questions related to speaker demographics such as gender identity, if the speaker identifies as a person of color, or identifies with any other underrepresented groups. All LF events commit to speaker diversity and we do not allow all male speaker lineups or panels. Please note, the questions are not required and we provide the option “prefer not to answer” in some instances.
Q Provide an example of an opt-out option on the Event registration page if available.
A All demographic questions on our registration are optional.
Q Provide an example of a demographics text input box on the Event registration page if available.
A Not available

Inclusive Experience at Event

[x] This event commits to the Code of Conduct at Event.

Q Provide an example of the Event Feedback page if available.
A We send out a post-event survey to all speakers and attendees. We also encourage all attendees to share any ideas with us during the event, and have the contact information for the SVP & GM of Events as well as the specific Event Manager made easily accessible should anyone want to share their feedback prior, during, or after the event.
Q Is the event team using feedback from previous event's attendees, speakers, and volunteers to improve DEI at this event?
A Yes, the information gathered from the previous project's post-event survey was considered when planning 2024.
Q Does the event team plan to use feedback from this event's attendees, speakers, and volunteers to improve DEI at future events?
A Yes, we continue to use our post-event survey to collect feedback to be considered for future events.
Q How can attendees learn more about accessibility at the event?
A Under the Diversity and Inclusion page on the event website there is a section that speaks to venue accessibility.
Q Does the event platform allow attendees to suggest future accomodations for the event?
A We send out a post-event survey to all attendees post event.
Q Will the event platform be accessible to attendees and speakers after the event?
A Yes, it will be made available at the conclusion of the event via YouTube to all at no charge.

Code of Conduct at Event

[x] This event commits to the Code of Conduct at Event.

Q Is the code of conduct posted at Event venue?
A Yes, we have the code of conduct printed onsite and at all off-site venues. We have the code of conduct linked in all event emails, as well as on the onsite Diversity & Inclusion resources flyer. This flier has all DEI-related resources, as well as event team members they can contact should they need anything onsite.
Q Provide a link for the Event Code of Conduct.
A https://events.linuxfoundation.org/soss-community-day-europe/attend/code-of-conduct/

Diversity Access Tickets

[x] This event commits to the Diversity Access Tickets.

Q How many different types of diversity access tickets are available for the event?
A There is no cost to attend this event. Registration is free.
Q What are the criteria for qualifying for a diversity access ticket?
A There is no cost to attend this event. Registration is free.
Q Provide a link to the page containing information about Diversity Access Tickets.
A https://events.linuxfoundation.org/soss-community-day-europe/register/

Family Friendliness

[x] This event commits to Family Friendliness.

Q Does the Event provide childcare facilities for its attendees and speakers?
A Yes! As a colocated event with Open Source Summit Europe we are able to offer complimentary onsite child care.
Q What are the other ways that a family-friendly environment is being created in the Event?
A We offer our attendees a private nursing room that they can access throughout the event and gender-neutral restrooms.
Q Provide relevant links related to family friendliness at the Event.
A https://events.linuxfoundation.org/soss-community-day-europe/attend/diversity-inclusion/

/result

/result

/end


Undefined 'extern' symbol when linking SDK statically

Required Info

Camera Model
D435i

Operating System & Version
Ubuntu 18.04 / GCC-9.4

Kernel Version (Linux Only)
4.9.201-tegra

Platform
NVIDIA Jetson

SDK Version
2.48.0

Language
C++

Segment
VR/AR

Issue Description
I'm compiling and linking SDK statically with shared object. When this shared object is loaded by executable, I see the following error: "dlerror system message: /home/mrs/repo/cv-core/build/bin/libcvcore.so: undefined symbol: _ZN14rs2rosinternal8TIME_MINE". Static libraries libfw.a, librealsense-file.a, librealsense2.a are linked to 'libcvcore.so'.
I tried linking SDK to 'libcvcore.so' with option '--no-as-needed' but it didn't help. Some undesirable workaround I found for this problem is declaring 'TIME_MIN' in time.h as 'static' instead of 'extern' to avoid implicit dependency of one static library (librealsense-file.a) from another (librealsense2.a) when linking both to shared object.

Hi @MikhailUskin  There are few information resources available about building librealsense as a static library, and I have not previously encountered the error that you quoted.
It's good that you found a workaround.  If you would prefer to use a more 'correct' method though, the best reference that I know of for building librealsense statically by setting BUILD_SHARED_LIBS to False is at the link below.
https://github.com/IntelRealSense/librealsense/issues/4709
If the guidance in that link does not resolve your problem then I will be happy to continue working through the problem with you.

Thanks, I saw issue #4709:

BUILD_SHARED_LIBS is set to False;
Also I tried link the libraries in different order.

Nothing helped me

Wrapping SDK linked libraries so that "--whole-archive <sdk_libraries> --no-whole-archive" solved the problem

Excellent news @MikhailUskin that you were able to develop a solution - thanks so much for sharing the solution with the RealSense community!

Case closed due to solution achieved and no further comments received.


feature request: add OnClose hook to allow writing customized load balance dialer

I have 10 redis slaves mapped to the same dns name.
Right now it is possible to use Dialer to ensure that new connections are evenly distributed to slave servers. However, we don't know which connections are get idle and closed. Thus we don't know which server has less connections after a while.
Is it possible to add OnClose hook so we know which connection(especially the IP and port) will be closed? So we can write a customized dialer to ensure new connections are connected to less stressed server. Also we could write customized stats to export how many connections are connected to a specific server.

Are we talking about cluster client? If so why not to use RouteRandomly or RouteByLatency that should load balance read-only command to the slaves?

Can't use cluster here because there are too many reads to same keys. Just normal slaves, each holds all keys.
We decide to assign a dedicated port for each Redis slave, and run netstat regularly to capture connections made to each slave.
Another option is create one client for each slave, but we may sometimes add new slave or destroy old slaves, so that's not working easily here.
Thanks for your reply.

I still don't understand why cluster client does not work for you. As I said go-redis should be able to load balance read-only traffic. Is the problem in go-redis client implementation or Redis Cluster itself?
I also don't understand your setup. I guess you have 2 redis client instances:

first is master which you use to set keys
second is many slaves which you use to get keys

Am I right? Even without using Redis Cluster you should be able to do something like:
	loadClusterSlots := func() ([]redis.ClusterSlot, error) {
		slots := []redis.ClusterSlot{
			// First node with 1 master and 1 slave.
			{
				Start: 0,
				End:   16383,
				Nodes: []redis.ClusterNode{{
					Addr: ":7000", // master
				}, {
					Addr: ":8000", // 1st slave
				}, {
					Addr: ":8001", // 2nd slave
				}, {
					Addr: ":8002", // 3rd slave
				}},
			},
		}
		return slots, nil
	}

	client := redis.NewClusterClient(&redis.ClusterOptions{
		ClusterSlots:  loadClusterSlots,
		RouteRandomly: true,
	})

	client.Set() // automatically routed to master
	client.Get() // automatically routed to random slave

See https://godoc.org/github.com/go-redis/redis#example-NewClusterClient--ManualSetup

Thanks for the cluster client usage.
All I have is a dns name read-only.redis.mycompany.com.
Behind that dns name are a lot of slave Redis, whose IPs may change occasionally.
RouterRandomly randomly chooses a node, but in my case all connections are in the same node (have to use the dns name, can't use IPs directly)
I've tried cluster client but got this err ERR This instance has cluster support disabled.
Maybe we need master node here but I don't have that node.
Currently I'm trying to collect more information to see if I really need to build a load balancer in client code.


Bad error handling for oauth errors

I got the following error on calls to tableService.InsertAll().Do():
Post https://www.googleapis.com/bigquery/v2/projects/wf-aaronkavlie/datasets/test/tables/blank2/insertAll?alt=json: asn1: structure error: length too large

I eventually discovered that this was due to a .pem file I generated with the wrong openssl options.
The error message doesn't hint at the problem at all. It should be clear what failed, and why (even indicating that the problem was in authentication would help a lot).

I believe I have seen this before with other APIs... and if I remember correctly, some payload wasn't base64 encoded when it should have been.  Also, the error message is coming from the JSON API endpoint server... not from google-api-go-client.  I'm not sure a whole lot can be done to help with this in the auto-generated library.  Leaving open for further investigation.
If you have a small, simple test case that reproduces the problem, that will dramatically help the speed with which I can investigate and debug this problem.

Steps to reproduce:

Set up an OAuth service account in Google dev console.
Download the .p12 file
Generate the .pem file as follows (this is not the right way to do it, and the source of the problem):

openssl pkcs12 -in wf-aaronkavlie-97eb85083073.p12 -nocerts -out key.pem

Instantiate an oauth client for this Oauth account, using "golang.org/x/oauth2/jwt".
Use the BigQuery client ("google.golang.org/api/bigquery/v2") to attempt insertion into a table roughly as follows:

bqClient, err := bigquery.New(oauthClient)
[put together jsonRows...]
request := &bigquery.TableDataInsertAllRequest{Rows: jsonRows}

tableService := bigquery.NewTabledataService(bq.client)
return tableService.InsertAll([project], [dataset], [table], request).Do()

@gmlewis let me know if you have any trouble reproducing based on the steps to reproduce above (which admittedly gloss over some details).
It may very well be that this can't be improved in the auto-generated library... in which case, it's one reason that auto-generated code isn't ideal (especially as the only option) for developers. I just reported here because this is where the issue surfaced.

It looks like the error message from the BigQuery API has improved since the time that this bug was written.
Here is the error I get with bad pem data:
2015/07/29 08:41:13 GML: res=(*bigquery.TableDataInsertAllResponse)(nil), err=Post https://www.googleapis.com/bigquery/v2/projects/vm-gmlewis/datasets/mydataset/tables/mytable/insertAll?alt=json:
private key should be a PEM or plain PKSC1 or PKCS8;
parse error: asn1: structure error: length too large

For reference, here is the test program that I created:
package main

import (
	"log"

	"golang.org/x/oauth2"
	"golang.org/x/oauth2/google"
	"golang.org/x/oauth2/jwt"
	bigquery "google.golang.org/api/bigquery/v2"
)

func main() {
	// Your credentials should be obtained from the Google
	// Developer Console (https://console.developers.google.com).
	conf := &jwt.Config{
		Email<EMAIL_ADDRESS>		// The contents of your RSA private key or your PEM file
		// that contains a private key.
		// If you have a p12 file instead, you
		// can use `openssl` to export the private key into a pem file.
		//
		//    $ openssl pkcs12 -in key.p12 -passin pass:notasecret -out key.pem -nodes
		//
		// The field only supports PEM containers with no passphrase.
		// The openssl command will convert p12 keys to passphrase-less PEM containers.
		PrivateKey: []byte(`-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----`),
		Scopes: []string{
			"https://www.googleapis.com/auth/bigquery",
		},
		TokenURL: google.JWTTokenURL,
		// If you would like to impersonate a user, you can
		// create a transport with a subject. The following GET
		// request will be made on the behalf of<EMAIL_ADDRESS>		// Optional.
		// Subject<EMAIL_ADDRESS>	}
	// Initiate an http.Client, the following GET request will be
	// authorized and authenticated on the behalf of<EMAIL_ADDRESS>	client := conf.Client(oauth2.NoContext)
	// client.Get("...")

	bqClient, err := bigquery.New(client)
	// [put together jsonRows...]
	jsonRows := []*bigquery.TableDataInsertAllRequestRows{
		&bigquery.TableDataInsertAllRequestRows{
			Json: map[string]bigquery.JsonValue{
				"myfield": "value1",
			},
		},
	}
	request := &bigquery.TableDataInsertAllRequest{Rows: jsonRows}

	tableService := bigquery.NewTabledataService(bqClient)
	res, err := tableService.InsertAll("myproject", "mydataset", "mytable", request).Do()
	log.Printf("GML: res=%#v, err=%v", res, err)
	if res != nil {
		for _, v := range res.InsertErrors {
			log.Printf("GML: v=%#v", v)
			for _, e := range v.Errors {
				log.Printf("GML: e=%#v", e)
			}
		}
	}
}

Closing bug since it appears that the error message has improved.


[Perf] Avoid Linq method and save its related allocations in TreeRouter

TreeRouter.GenerateVirtualPath() has the following expression
var parameter = entry.Template.Parameters
.FirstOrDefault(p => string.Equals(p.Name, kvp.Key, StringComparison.OrdinalIgnoreCase));
that is resulting in 3 allocations

Delegate for FirstOrDefault (System.Func)
Closure for TreeRouter
Enumerator for Parameters
This expression is used in a loop for each of the Route values.

By removing this Linq expression, there is 3.5% saving in total number of allocations and 2.5% in total number of bytes allocated.
The measurements are against Performance/BasicViews test app with the View having 6 action links . Each of these actions href is generated using TagHelpers with attribute routing. And loadtest is run against the app with the following setting
loadtest --rps 100 -n 3000 -k

Hi @mnltejaswini, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.
TTYL, DNFBOT;

:shipit: after fixing doc comments

:shipit:

:shipit:


Fix running the role in check mode from the scratch

In case check mode is running from the scratch, it'll fail due to absent apt-repo / systemd service. The tasks requiring repo / systemd service now contain custom failure/changed rules allowing the role to be executed in check mode.

Now, can you also merge tests (#21) into your PR and see if this will work

Thank you!


KAS Fleet Manager doesn't meet validator rules

https://github.com/redhat-developer/app-services-sdk-go/runs/3251943870?check_suite_focus=true
CC @craicoverflow what is the process for this?

Reach out to the team for a fix.
In the mean time I have created an issue to add a rule for this: https://github.com/redhat-developer/app-services-api-guidelines/issues/8

Done.


Implement team reports in the computed digests

Team reports should be designed to make it easier for teams to understand what's going on.
Key aspects:

Monthly report; weekly diff/changes report if technology exists to improve the time series lookup
Show team maintainers and members
Show any unlinked team members as a warning
List repositories and access levels

Now that nested teams have shipped, the design will need to take into consideration what to show in that case.

Partially implemented now.


chore(battery_plus): Add iOS Privacy Info

Description

Add PrivacyInfo to podspec of Battery Plus

Related Issues

Part of #2447

Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I titled the PR using Conventional Commits.
[x] I did not modify the CHANGELOG.md nor the plugin version in pubspec.yaml files.
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze) does not report any problems on my PR.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate that with a ! in the title as explained in Conventional Commits).
[x] No, this is not a breaking change.

Code compiles and builds as it did before. I followed the PR from the Flutter team as example and added it to the pod file:

Let's mark it not as chore because I believe such changes should be included in release notes generated by Melos.

Good catch!

Merged, I will be preparing the rest


Not working on iOS.

Version 1.5.7 is working, newer doesn't respond to tapping.

Hold the cell for 1 second and try again...


Get list of methods for particular class

Hi! Is it possible to get list of methods for particular class/object? Something like Object#methods in ruby.
Or is there way to do it in static way?
Or in anyway?:)
Thanks!

See #403

Closing as a duplicate of #403.


MacOS requires libmemcached

To make the project down loaded from github, you need to install libmemcached:
brew install libmemcached
Please add libmemcached to the requirements.

Thank you for your feedback. I covered this topic in README.


Achtung! Cthulhu 2d20 Bugfix (add missing field)

Add missing Armor field.

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x]  The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

New Sheet Details

The name of this game is: <   >  (i.e. Dungeons & Dragons 5th Edition, The Dresden Files RPG)

The publisher of this game is: <   > (i.e. Wizards of the Coast, Evil Hat)

The name of this game system/family is: <   > (i.e. Dungeons & Dragons, FATE)

[ ] I have followed the Character Sheets Standards when building this sheet.

[ ] This sheet has been made on behalf of, or by, the game's publisher.
[ ] This game is not a traditionally published game, but a copy of the game rules can be purchased/downloaded/found at: <   >
[ ] This sheet is for an unofficial fan game, modification to an existing game, or a homebrew system.

Changes / Description

Added missing Armor field to the Vehicle section of the character sheet.

Character Sheet Info Roll20 Internal Use only.


multires_image with transparent regions

Hello,
When I try to overlay an image on a tile map in mapviz, I notice the transparent regions of the image get colored black(I am using mapproxy with google map layer). If I put the same image on the google-earth application (with kml file), the transparent region is displayed correctly(I see the google map layer). Are there any ideas on how I can resolve this?

Are you drawing the image with Mapviz's image plugin?  I suspect that alpha blending may not be enabled when it's drawing the image; we probably haven't tested it with images that have alpha channels.  I'll do that sometime when I have the time for it, unless somebody else gets there first...

Ah sorry I think I wasn't specific in my description. I'm not using the Image plugin(sensor_msgs::Image), but the Multi-res Image plugin. I created a geo file giving the path to my image that I want to overlay onto a tile_map plugin. When I run mapviz and choose the multi-res image plugin, I see the image being overlaid correctly onto the tile_map plugin, but the transparent regions of my multi-res image turn out black.
If I use the same image on google earth the transparency is displayed correctly(or I can see the map layer underneath where the image is transparent)

Ok, I took a look at this and was able to reproduce it, and I think it's because the multires_image plugin is internally storing GL textures without an alpha channel.  Should be easy to fix.

Great thank you.


Error connection

I have tried to connect to an ODBC data source called RD32, based on "SQL Server" driver.
c: open ODBC://RD32

I receive the error: .... access failed for user ''  then the consolle closes
EDIT: Here is the output from RED console.exe
>> c: open odbc://rd32
28000 18456 [Microsoft][ODBC SQL Server Driver][SQL Server]Accesso non riuscito
per l'utente ''.
*** Script Error: ODBC error: ["28000" 18456 {[Microsoft][ODBC SQL Server Driver
][SQL Server]Accesso non riusc
*** Where: ctx||440~open-connection
*** Near : entity/spec/host
*** Stack:

Note that the USER DNS Origin has configured user and password inside
Also, I have tried the following:
connection: open make port! [
    scheme: 'odbc
    target: "driver={SQL Server};server=192.168.x.x;port=1433;database=ADB_RD32;uid=myuser;pwd=mypass"
]

(Server IP has been blanked)
The console hangs and I have to force close it.
SQL Server 2014 on Windows Server 2012 here
NOTE: One of the message I receive is "Invalid String Attribute" from the ODBC Driver. Removing the port=1433 has solved the problem

UPDATE:
I have had to:
change the server address to the textual one.
Remove the port configuration
Also I have moved from RED gui-console to console
This is the error I have:

do %test.red
01000 5701 [Microsoft][ODBC SQL Server Driver][SQL Server]Database context has been changed to 'ADB_RD32'.
01000 5703 [Microsoft][ODBC SQL Server Driver][SQL Server]Language setting has been changed to italian
*** Access Error: invalid UTF-8 encoding: #{C0C1C2C3}
*** Where: pick-information
*** Near : info
*** Stack: do-file

(Error strings translated by me)

probe system/schemes/odbc/state/drivers

[
   "Microsoft Access-Treiber (*.mdb)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "2"
       "FileExtns" "*.mdb"
   )
   "Driver do Microsoft Paradox (*.db )" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.db"
   )
   "Driver do Microsoft Excel(*.xls)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.xls"
   )
   "Microsoft Text Driver (*.txt; *.csv)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.,*.asc,*.csv,*.tab,*.txt,*.csv"
   )
   {Driver da Microsoft para arquivos texto (*.txt; *.csv)} #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.,*.asc,*.csv,*.tab,*.txt,*.csv"
   )
   "Microsoft dBase-Treiber (*.dbf)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.dbf,*.ndx,*.mdx"
   )
   "SQL Server" #(
       "CPTimeout" "60"
       "SQLLevel" "1"
       "DriverODBCVer" "03.50"
       "APILevel" "2"
       "ConnectFunctions" "YYY"
       "UsageCount" "1"
       "FileUsage" "0"
   )
   "Microsoft Excel Driver (*.xls)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.xls"
   )
   "Driver do Microsoft dBase (*.dbf)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.dbf,*.ndx,*.mdx"
   )
   "Microsoft Paradox-Treiber (*.db )" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.db"
   )
   "Microsoft ODBC for Oracle" #(
       "CPTimeout" "120"
       "SQLLevel" "1"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYY"
       "UsageCount" "1"
       "FileUsage" "0"
   )
   "Microsoft Text-Treiber (*.txt; *.csv)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.,*.asc,*.csv,*.tab,*.txt,*.csv"
   )
   "Microsoft Excel-Treiber (*.xls)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.xls"
   )
   "Microsoft Access Driver (*.mdb)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "2"
       "FileExtns" "*.mdb"
   )
   "Driver do Microsoft Access (*.mdb)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "2"
       "FileExtns" "*.mdb"
   )
   "Microsoft Paradox Driver (*.db )" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.db"
   )
   "Microsoft dBase Driver (*.dbf)" #(
       "SQLLevel" "0"
       "DriverODBCVer" "02.50"
       "APILevel" "1"
       "ConnectFunctions" "YYN"
       "UsageCount" "1"
       "FileUsage" "1"
       "FileExtns" "*.dbf,*.ndx,*.mdx"
   )
   "SQL Server Native Client 11.0" #(
       "UsageCount" "1"
       "APILevel" "2"
       "ConnectFunctions" "YYY"
       "CPTimeout" "60"
       "DriverODBCVer" "03.80"
       "FileUsage" "0"
       "SQLLevel" "1"
   )
   "ODBC Driver 11 for SQL Server" #(
       "UsageCount" "1"
       "APILevel" "2"
       "ConnectFunctions" "YYY"
       "CPTimeout" "60"
       "DriverODBCVer" "03.80"
       "FileUsage" "0"
       "SQLLevel" "1"
   )
   "Microsoft Visual FoxPro Driver" #(
       "UsageCount" "1"
       "FileExtns" "*.dbf,*.cdx,*.idx,*.fpt"
       "DriverODBCVer" "02.50"
       "FileUsage" "1"
       "APILevel" "0"
       "SQLLevel" "0"
       "ConnectFunctions" "YYN"
   )
   "Microsoft Visual FoxPro-Treiber" #(
       "UsageCount" "1"
       "FileExtns" "*.dbf,*.cdx,*.idx,*.fpt"
       "DriverODBCVer" "02.50"
       "FileUsage" "1"
       "APILevel" "0"
       "SQLLevel" "0"
       "ConnectFunctions" "YYN"
   )
   "Driver para o Microsoft Visual FoxPro" #(
       "UsageCount" "1"
       "FileExtns" "*.dbf,*.cdx,*.idx,*.fpt"
       "DriverODBCVer" "02.50"
       "FileUsage" "1"
       "APILevel" "0"
       "SQLLevel" "0"
       "ConnectFunctions" "YYN"
   )
   "Microsoft FoxPro VFP Driver (*.dbf)" #(
       "UsageCount" "1"
       "FileExtns" "*.dbf,*.cdx,*.idx,*.fpt"
       "DriverODBCVer" "02.50"
       "FileUsage" "1"
       "APILevel" "0"
       "SQLLevel" "0"
       "ConnectFunctions" "YYN"
   )
]

>> probe system/schemes/odbc/state/sources
[
    "Excel Files" {Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)}
    "MS Access Database" "Microsoft Access Driver (*.mdb, *.accdb)"
    "ArcaBUO-OK" "SQL Server"
    "RD32" "SQL Server"
    "RD32B" "SQL Server"
    "Visual FoxPro Database" "Microsoft Visual FoxPro Driver"
    "Visual FoxPro Tables" "Microsoft Visual FoxPro Driver"
]

ODBC Panel
https://drive.google.com/file/d/1chhvrW7GTGDAJ6fmDK_FozfZ4D2t-6Bj/view?usp=sharing
Connection using IP Addess
connection: open make port! [
scheme: 'odbc
target: "driver={SQL Server};server=<IP_ADDRESS>;port=1433;database=ADB_RD32;uid=myuser;pwd=mypass"
]
Connection string using server name instead of IP
connection: open make port! [
scheme: 'odbc
target: "driver={SQL Server};server=SERVER\DB2014;port=1433;database=ADB_RD32;uid=myuser;pwd=mypass"
]
I Repeat the note written above:
One of the message I receive is "Invalid String Attribute" from the ODBC Driver. Removing the port=1433 has solved the problem

solved via commit fb1de49


Typo in the spack command

Just a simple typo in the spack install command:
$ spack install nvshmem +gpu_initiated_suppot +cuda +gdrcopy +ucx +mpi ^openmpi +cuda fabrics=ucx ^ucx +cuda +gdrcopy +dm +thread_multiple
just needs an "r" in support so:
$ spack install nvshmem +gpu_initiated_support +cuda +gdrcopy +ucx +mpi ^openmpi +cuda fabrics=ucx ^ucx +cuda +gdrcopy +dm +thread_multiple

Thanks a lot for letting us know. I have fixed the typo.


starkl?

With the release of the new ZK-STARK paper, it might be interesting to see how snarkl needs to change to accommodate it: https://eprint.iacr.org/2018/046.pdf

Many thanks for the pointer. Snarkl should port relatively easily to any crypto backend that uses Rank-1 Constraint Systems (R1CS) to encode arithmetized programs. However, ZK-STARK's arithmetization scheme, as described in Section 2 of the paper you cite above, uses a different representation (AIR, for algebraic intermediate representation). It's not clear to me yet exactly what would have to change to generate code in that representation.


Install script generation does not require edit permission

It probably should.

Fixed: 18238f51e591c147df83e990a4bb1078e22af493
Now it does.


MSVC: Allow static linking against the Visual C++ Runtime

This small contributution enables static linking against the Microsoft Visual C++ Runtime if it is explicitely requested by the user. Without the flag for MSVC, the linking may fail given the implicit dependency on vcruntime140.dll when using "bundled-" features.

I think this should get the data from the CARGO_CFG_TARGET_FEATURE env var. As it is, it checks the target feature of the build.rs, not at runtime.
See https://github.com/BurntSushi/memchr/blob/master/build.rs#L70-L74
Also, yeah, I think the is_compiler("msvc") check is not needed.

Thanks for your review! We do not need the is_compiler("msvc") as it is checked in cc internally. Just added it for the sake of clarity. I'm happy to remove it :)
Regarding the CARGO_CFG_TARGET_FEATURE, I'm a bit unsure. By default, cc already checks for this env var. However, it does not appear to be enough in my local test case which compiles and links well only when checking for the target feature of the build.rs additionally:
[...]
= note:    Creating library C:\Users\Admin\Documents\Projekte\matryoshka-sqlite\target\release\deps\matryoshka.dll.lib and object C:\Users\Admin\Documents\Projekte\matryoshka-sqlite\target\release\deps\matryoshka.dll.exp
          LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
          LINK : warning LNK4217: symbol 'free' defined in 'libucrt.lib(free.obj)' is imported by 'liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o)' in function 'sqlite3MemFree'
          LINK : warning LNK4217: symbol 'malloc' defined in 'libucrt.lib(malloc.obj)' is imported by 'liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o)' in function 'sqlite3MemMalloc'
          LINK : warning LNK4217: symbol '_msize' defined in 'libucrt.lib(msize.obj)' is imported by 'liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o)' in function 'sqlite3MemSize'
          LINK : warning LNK4217: symbol 'qsort' defined in 'libucrt.lib(qsort.obj)' is imported by 'liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o)' in function 'sqlite3Fts3SegReaderPending'
          LINK : warning LNK4217: symbol 'strncmp' defined in 'libucrt.lib(strncmp.obj)' is imported by 'liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o)' in function 'execSql'
          liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o) : error LNK2019: unresolved external symbol __imp_realloc referenced in function sqlite3MemRealloc
          liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o) : error LNK2019: unresolved external symbol __imp_strcspn referenced in function patternCompare
          liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o) : error LNK2019: unresolved external symbol __imp__localtime64_s referenced in function localtimeOffset
          liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o) : error LNK2019: unresolved external symbol __imp__beginthreadex referenced in function sqlite3ThreadCreate
          liblibsqlite3_sys-e960bef6b47b98ed.rlib(sqlite3.o) : error LNK2019: unresolved external symbol __imp__endthreadex referenced in function sqlite3ThreadProc
          C:\Users\Admin\Documents\Projekte\matryoshka-sqlite\target\release\deps\matryoshka.dll : fatal error LNK1120: 5 unresolved externals

Hm, that's very strange. I wonder if it's a cc bug.


support version changes

remove existing go to enforce installing the desired version

I prefer to leave the existing version in place and update a link to the new version to keep it more immutable.

Take a look at version 1.0.7 of the role that I just published to the galaxy. it will support version upgrades.

fair enough, thanks!


upgrade lib to support ocp 4.14

issue: https://issues.redhat.com/browse/ACM-6877

/hold

/lgtm

/unhold


JS editor autocomplete on small files

When file is small/new one autocomplete does not work as expected. We can parse system.js to extend  dictionary or even use reflection to get publick slots/Q_INVOKABLE methods from object.

Probably it is fixed, already, for 19.04

@iakov, can we close this issue?

Current state: extended autocompletion works, but shows methods only after the first letter is typed and the populated list is mixed with other tokens (type brick().i to see both i2c and if in the list).
Can be postponed until next version.


License options based on your plan for Microsoft 365 plans

Hi,
Could the license add-on options for the Microsoft 365 plans also be added ?
Which add-on license are for example possible with Microsoft 365 Business or Microsoft 365 E1 , etc ... ?
Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: abae7324-861d-7ca7-7239-a5df8fd927d5
Version Independent ID: 8a30bd2f-9bc3-a0a8-8b31-d53836ead4d0
Content: Microsoft Teams add-on licensing - Microsoft Teams
Content Source: Teams/teams-add-on-licensing/microsoft-teams-add-on-licensing.md
Service: msteams
GitHub Login: @tonysmit
Microsoft Alias: tonysmit

@YuriHerman Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

Hi @YuriHerman, thank you for your feedback.
You can find this information in the article, please go to:
https://docs.microsoft.com/microsoftteams/teams-add-on-licensing/microsoft-teams-add-on-licensing#license-options-based-on-your-plan
And select a link for more detail.

Hi @diaromws ,
The subscriptions in the link are only the Office365 subscriptions.
My suggestion was that this document would also explain what add-on licenses are possible for Microsoft365 subscriptions.

@YuriHerman Thank you.
Microsoft 365 Ex includes Office Ex. For example, O365 E3 is included on M365 E3:
https://www.microsoft.com/en-us/microsoft-365/compare-microsoft-365-e3-and-office-365-e3-enterprise
The same add-ons you can add on O365 you can add on M365.

@YuriHerman Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.


Support gauge typed panel

Support marshaling and unmarshaling Dashboard json which contains
gauge typed panel

To display this panel in vela core logging dashboard.


Eliminating the "/" char from the installdir parameter

In POSIX-compatible file systems, file names cannot contain the following characters: NUL (0x00), forward slash (0x2f). While others may also be disallowed, this requirement is present in all UNIX-inspired file systems.
This simple patch makes sure that even for games that contain the "/" character in their title, the install directory param is set in a way which does not violate this minimum requirement. I'm only trapping the "/" character since it's obviously impossible for games to have the NULL character in their title ;)

Thanks :)