CephFS CSI used with Kasten has issues with EAGAIN

Describe the bug
Using Kasten to do backups of workloads, some backups will get stuck in the copy-out part of the job.
During this period the application is mounting pvc's into a special copy-out pod. These mounts fail with Error 11 EAGAIN
FailedMount | MountVolume.MountDevice failed for volume "pvc-a51877a6-b630-4ee5-bc58-4bf9b11420e1" : rpc error: code = Internal desc = an error (exit status 11) occurred while running ceph args: [fs subvolume getpath kubes_ldr_cephfs csi-vol-37028e72-35ca-11eb-bb87-ca8c85f94050 --group_name csi -m stripped -c /etc/ceph/ceph.conf -n client.admin --keyfile=stripped]
Environment details

Image/version of Ceph CSI driver : quay.io/cephcsi/cephcsi:v3.1.1
Helm chart version : NA
Kernel version : Linux 5.4.0-48-generic #52 SMP Sat Sep 26 08:27:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its
krbd or rbd-nbd) :fuse
Kubernetes cluster version : Provider: K3os Kubernetes Version: v1.19.2+k3s1
Ceph cluster version : ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)

Steps to reproduce
Steps to reproduce the behavior:
Run backup.
Actual results
PVCs fail to mount with Error EAGAIN
Expected behavior
PVCs mount correctly
Logs
If the issue is in PVC mounting please attach complete logs of below containers.

csi-rbdplugin/csi-cephfsplugin and driver-registrar container logs from
plugin pod from the node where the mount is failing.

driver-registar.log
cephfsplugin.log
This was the most recent mount attempts.

@insanemal can you run ceph fs subvolume getpath kubes_ldr_cephfs csi-vol-37028e72-35ca-11eb-bb87-ca8c85f94050 --group_name csi in your ceph cluster and see the subvolume exists

Yeah I did try that it returns the same error EAGAIN saying the specified subvol is not ready.
From a different node if I mount up the cephFS filesystem I can browse to the volume subfolder and locate this particular one, however it's empty.
So I can only assume something has gone wrong at some point

But current theory is that the previous out of space event, that was triggered by the bug from my previous ticket has left the filesystem in a inconsistent state.

I don't see an issue from the cephcsi side, if the subvolume is not ready in the backend cephcsi cannot do anything.

So the issue is, the previous version of cephCSI had a bug with snapshots. That has been fixed I believe.
I hit that bug at the same time as running out of space.
 They can't be deleted because they have snapshots. The snapshots can't be deleted because they have pending clones. The clones aren't happening because of the previous bugs/ran out of space issue.
And I can't figure out how to cancel/delete the 'in progress' clones.
Basically you are both correct and not quite correct. I ended up in this situation because the CephCSI can't handle things failing and/or doesn't clean up after failures correctly.
So yeah, my filesystem appears to be kinda trashed and yeah CephCSI did it. And will probably do it again because it doesn't handle failures well

So the issue is, the previous version of cephCSI had a bug with snapshots. That has been fixed I believe.
I hit that bug at the same time as running out of space.

Yes the bug is fixed now

So now I have a bunch of stuck subvolumes. They can't be deleted because they have snapshots. The snapshots can't be deleted because they have pending clones. The clones aren't happening because of the previous bugs/ran out of space issue.
And I can't figure out how to cancel/delete the 'in progress' clones.

You have an option in ceph to cancel ongoing clone see https://docs.ceph.com/en/latest/cephfs/fs-volumes/#cloning-snapshots

Basically you are both correct and not quite correct. I ended up in this situation because the CephCSI can't handle things failing and/or doesn't clean up after failures correctly.
So yeah, my filesystem appears to be kinda trashed and yeah CephCSI did it. And will probably do it again because it doesn't handle failures well

Best option is now to clean up the stale snapshots and clones.

To cancel an ongoing clone you need the clone name. And how does one find the clone name of a clone that didn't start/finish?
It's not stored in the .meta file.
I can't find it listed in the subvolume listing "ceph fs subvolume ls blah blah"
So yeah that doesn't really work. It looks like the volume might have been cleaned up due to a delete retention policy.
An while that bug I hit might be fixed, I'm still not convinced that things will be handled correctly if you run out of space during a snapshot clone.
But please, if you can find actually helpful documentation on how to unravel this Gordian knot. I'm sorry if I sound frustrated, I've got considerable ceph experience. I know I should "just" clean up the stale state, but that is not working.
I haven't even mentioned the stuck k8s objects that won't delete and don't exist on the back end. No force deleting them also fails.
I'm a tad frustrated as there are some parts of this that feel distinctly half sucked. Like not being able to remount a RWX volume if it was mounted and a k8s node crashes without cleaning unmounting it. And that being deemed "compliant with K8s spec"
So I do apologise if I seem short, but I've just had this plug in potentially cause issues in my filesystem in a way that currently looks irreparable and I'm not convinced there is sufficient edge case testing, in part due to the overwhelming feeling from comments that seem to suggest that some k8s developers (and plugin developers) think that node failure and space exhaustion are things that only happen in imaginary lands that aren't production.

root@mgr:/mnt/ceph/volumes/csi$ceph fs subvolume snapshot rm kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi
Error EAGAIN: snapshot 'csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb' has pending clones
root@mgr:/mnt/ceph/volumes/csi$ ls -lah ../_index/clone/b94869cc-9147-4bc3-aa8b-00d21237d94c
lrwxrwxrwx 1 root root 57 Dec  4 10:00 ../_index/clone/b94869cc-9147-4bc3-aa8b-00d21237d94c -> /volumes/csi/csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050
root@mgr:/mnt/ceph/volumes/csi$ ls csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050
ls: cannot access csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050: No such file or directory
root@mgr:/mnt/ceph/volumes/csi$

Yeah exactly what I thought was the case. The 'target volume' doesn't exist anymore (or never did) so I can't actually rm the snapshots because "pending clones"
Yeah this is probably an upstream CephFS issue, but its something you can encounter with the CSI plugin.
Also it's causing me no end of issue because of the huge count of in progress clones it does know about taking up all the simultaneous clone slots.
And look I know I sound cranky but there seems to be a lot of "well its their fault" going around.

Okay we need to remove the stale subvolume created in the backend, we can get the list of subvolumes created by csi diver with
kubectl get pv -o=jsonpath='{.items[*].spec.csi.volumeAttributes.subvolumeName}'
make sure all PV has the subvolumeName in the spec. Once we get the list we need to get all created subvolumes list from  the ceph cluster
ceph fs subvolume ls myfs csi  -> replace myfs with the filesystem name.
Just do a comparison and if you see any subvolume present in the ceph cluster, not in the kubernetes cluster we can remove it with force command.
If its a clone check the clone is in completed status, if not and it's a stale clone we need to first cancel the clone and then remove it
Hope we have cleaned up the clones/stale subvolume we need to remove the snapshot on the subvolume. can you try to remove the snapshots with --force flag which might help to clean up snapshot if the clone is not present.

You would hope so, but alas that is not the case
 ceph fs subvolume snapshot rm kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi --force
Error EAGAIN: snapshot 'csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb' has pending clones

I wasn't kidding when I said irreparable. I wouldn't have opened a ticket if I could have solved it with a few sternly worded rm commands.

This apparently isn't super hard to get yourself into a state where even manual intervention isn't possible and there is nothing like it automatically making thing far worse during what is already a major outage event

@ShyamsundarR @kotreshhr can you please help here?

@insanemal As the Ceph cluster is lacking space (as per your troubleshooting) to complete operations, and due to the earlier CSI snapshot and clone leak, cleaning this up becomes more involved as clones would never complete.
We/I do acknowledge the fact that in progress clones have no way of being tracked from kubernetes to CephFS as things stand, but more on this later. Let's attempt to address cleaning up the pending operations and resources first.
The following (lengthy) procedure should help with cleaning up the orphaned and stuck resources,

Update to the latest ceph-csi images, if not already done, so that we can prevent further leaks

As clone is occurring asynchronously, just deleting the clone in progress subvolumes from CephFS will only trigger more clone creations from the kubernetes/ceph-csi front. To stop this from happening,

Scale down the ceph-csi provisioner Deployment to 0 (preventing any further dynamic provisioning operations)

As clones in progress do not have a usable CSI volumeHandle to find the subvolume that is the clone target, we would need to do the following,

Determine all bound PVCs from the kubernetes cluster
Determine all the subvolumes in CephFS
All subvolumes that are not traceable back to a bound PVC are possible clones or leaked clones (orphan subvolumes), this is the set we would work with going forward
Ensure that the orphan subvolumes are clones, inspecting their info/status outputs

If they are not, these may need more attention as we do not leak volumes

Cancel the clone operation for these subvolumes, if clone is still pending/inprogress
Remove these orphaned subvolumes

We would need to cleanup orphan snapshots in a similar manner

Determine all VolumeSnapshots from the kubernetes cluster
Determine all snapshots of subvolumes in CephFS
Again determine the orphan snapshot list
Unprotect the orphan snapshot

Explicit snapshots protection was removed as a per-requisite in Octopus release that you are using, but an unprotect does not hurt. As the pending clones are cancelled, snapshot unprotect or delete should work at this point.

Delete the orphan snapshot

Cleanup kubernetes state

Delete all PVCs that are not bound

This maybe optional, but if kasten has unbound PVCs due to backup requests generated earlier, it would be better to delete them to avoid recreating the same

Delete all VolumeSnapshots that are not ready

NOTE: Kubernetes VolumeSnapshot finalizers may prevent this deletion, in which case these may need deletion post step (6) once they report that they are ready or have a backing VolumeSnapshotContent

At this point the Ceph cluster should have recovered space and also not have any orphan resources, neither should the kubernetes cluster attempt creation of older PVCs/Snapshots as these are also deleted.

Scale up the provisioner Deployment (back to 3) for future provisioning requests

@ShyamsundarR
Sorry I wasn't super clear. It's no longer out of space. I've fixed that quite some time ago.
The issues I'm facing are from when it ran out of space while colliding at high speed with the bug that has been fixed.
I'm on the latest stable cephfs and ceph rbd CSI plugins.  That was first cab off the rank.
I don't have any more clones happening. I've already stopped that from happening.
The issue is, I have a volume with snapshots I cannot delete because they apparently have pending clones. And I cannot cancel the pending clones because they don't exist.
Unprotect does not work.
root@mgr:/mnt/ceph/volumes/csi$ ceph fs subvolume snapshot unprotect kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi
Deprecation warning: 'snapshot unprotect' call is deprecated and will be removed in a future release
root@mgr:/mnt/ceph/volumes/csi$ ceph fs subvolume snapshot rm kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi
Error EAGAIN: snapshot 'csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb' has pending clones

I know how to locate what is and isn't in K8s. I know how to do all of this stuff. I'm not trying to be rude here but nobody seems to understand that.
This has damaged my cephfs volume in a way that apparently cannot be undone. I would agree that the bug you have fixed might have resolved some of the issue.
And I'm starting to feel that this is an upstream issue with ceph/cephfs. But far as I can tell I'm hosed. I cannot untangle this mess. These operations should be atomic. This should not be possible.
Thanks for your assistance despite my frustration.

The issue is, I have a volume with snapshots I cannot delete because they apparently have pending clones. And I cannot cancel the pending clones because they don't exist.
Unprotect does not work.
root@mgr:/mnt/ceph/volumes/csi$ ceph fs subvolume snapshot unprotect kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi
Deprecation warning: 'snapshot unprotect' call is deprecated and will be removed in a future release
root@mgr:/mnt/ceph/volumes/csi$ ceph fs subvolume snapshot rm kubes_ldr_cephfs csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0 csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb csi
Error EAGAIN: snapshot 'csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb' has pending clones

There can be 2 cases here,

There really is a pending clone that is stuck
There is stale state for a clone that disallows the snapshot from being deleted

To determine what is the actual state if you could do the following, it may help taking this up further,

For the above snapshot that you presented, the UUID of the clone that is pending can be determined using the .meta file within the subvolume which should be (approximately) at, /volumes/csi/csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0. If you cat this file it should have sections one of which would be clone snaps which contains the required UUID to investigate in /volumes/clones/<UUID>
A list of pending clones can be seen as symlinks within the CephFS file system at /volumes/clones/<symlinks> where each of the symlinks are named using a UUID and points to a subvolume that is being cloned

Chasing down the above for the example you presented, may help in understanding why and where a clone is stuck (if at all) and further clone state to determine what else is incorrect and needs attention.

@ShyamsundarR
Yep I know and I already did that. Sorry you either missed it or I didn't explain it well enough. I checked the meta file then checked the clones folder... and
root@mgr:/mnt/ceph/volumes/csi/csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0$ cat .meta
[GLOBAL]
version = 1
type = subvolume
path = /volumes/csi/csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0/4c14b7d2-abbe-4a67-ae26-aaa1c1111521
state = complete

[clone snaps]
b94869cc-9147-4bc3-aa8b-00d21237d94c = csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb
ccb1ad88-d049-482d-a093-6d2bdf45a7d7 = csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb
bc15c6e8-3b88-4342-b014-8e5f88d2dfdd = csi-snap-b9d57cb3-35c3-11eb-bea0-7e65af608dfb
130026d8-1cbd-47e2-b999-342d452a1135 = csi-snap-e45d5f9a-368c-11eb-bea0-7e65af608dfb
a91a889c-5d4b-42da-b33a-826406eb565a = csi-snap-e45d5f9a-368c-11eb-bea0-7e65af608dfb
c6d0f4a1-936d-4be6-b22d-44c2f090464f = csi-snap-e45d5f9a-368c-11eb-bea0-7e65af608dfb
0865d464-4fbc-46a9-8d09-dae4b8611ac5 = csi-snap-d1afa2b6-3755-11eb-bea0-7e65af608dfb
24d7cd77-1885-4181-a797-4ac9c4fc238e = csi-snap-d1afa2b6-3755-11eb-bea0-7e65af608dfb
4a8c62c4-213e-4dd7-81b2-1da74916562e = csi-snap-d1afa2b6-3755-11eb-bea0-7e65af608dfb
a80ae340-6bea-4534-9799-dc9bc145d7eb = csi-snap-d6b5f01b-381e-11eb-bea0-7e65af608dfb
ab116728-3a4a-438a-b8f5-7fa1f986b170 = csi-snap-d6b5f01b-381e-11eb-bea0-7e65af608dfb

root@mgr:/mnt/ceph/volumes/csi/csi-vol-025051ea-19e3-11eb-9f05-ca15dfd3b3f0$ cd ..

root@mgr:/mnt/ceph/volumes/csi$ ls -lah ../_index/clone/b94869cc-9147-4bc3-aa8b-00d21237d94c
lrwxrwxrwx 1 root root 57 Dec  4 10:00 ../_index/clone/b94869cc-9147-4bc3-aa8b-00d21237d94c -> /volumes/csi/csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050

root@mgr:/mnt/ceph/volumes/csi$ ls csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050
ls: cannot access csi-vol-c8fbf801-35c3-11eb-bb87-ca8c85f94050: No such file or directory

They don't exist! I mean they have left marks all over the file system but ultimately something is very VERY wrong here

@ShyamsundarR
Yep I know and I already did that. Sorry you either missed it or I didn't explain it well enough. I checked the meta file then checked the clones folder... and

They don't exist! I mean they have left marks all over the file system but ultimately something is very VERY wrong here

Thank you, I did miss the fact that you checked the clones symlinks and the meta file as in this comment.
Can you let me know if other clone symlinks other than b94869cc-9147-4bc3-aa8b-00d21237d94c also do not exist? I would like to ensure the problem is stale symlinks and hence snapshot can never be deleted.
Did you delete the clone pending subvolumes using rmdir <mountpoint>/volumes/csi/csi-vol-<uuid> or use the subvolume series of commands to delete the subvolume? If the former, then the subvolume mgr module would not have had a chance to perform the index cleanup and could explain the issue.
Also, is the problem for other subvolumes as well, or only this subvolume?
I am not sure when all the events transpired, but if we could get the ceph-mgr logs it may help unravel the bug better.
If all boxes check off and we are looking at a bug in CephFS subvolume clone module, for recovery I would suggest removing the index entries and also cleaning up the clone snaps section of the .meta file. This would enable deleting the snapshots, as no further meta entries exist and also the index is empty/clean.

@ShyamsundarR I don't believe any of the subvols were "rm -rf"ed
This does however give me a way to move forward.
This isn't the only subvolume it's just the one I've had the most "Can't delete snapshot" errors in kubernetes about.
There are at least 3-4 others.
I can try and get you the mgr logs if it helps.
I want to be clear, I'm more worried about this being an unusual corner case when you run out of space while attempting to take snapshots. So not a straight out bug more an unhandled edge case.
But I try and get logs for your. And might have a crack at undoing the issue

@ShyamsundarR I appear to have untangled the issue. But have a new one. I have snapshotcontent objects that don't have snapshots on the filesystem.  I cannot delete these kubernetes objects because they fail to delete the ceph object with
E1218 04:00:52.375920       1 goroutinemap.go:150] Operation for "delete-snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11" failed. No retries permitted until 2020-12-18 04:02:54.375893038 +0000 UTC m=+772050.178120189 (durationBeforeRetry 2m2s). Error: "failed to delete snapshot \"snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11\", err: failed to delete snapshot content snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11: \"rpc error: code = Internal desc = volume not found\""
--force doesn't work either.

@ShyamsundarR I appear to have untangled the issue. But have a new one. I have snapshotcontent objects that don't have snapshots on the filesystem. I cannot delete these kubernetes objects because they fail to delete the ceph object with
E1218 04:00:52.375920 1 goroutinemap.go:150] Operation for "delete-snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11" failed. No retries permitted until 2020-12-18 04:02:54.375893038 +0000 UTC m=+772050.178120189 (durationBeforeRetry 2m2s). Error: "failed to delete snapshot \"snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11\", err: failed to delete snapshot content snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11: \"rpc error: code = Internal desc = volume not found\""
--force doesn't work either.

@insanemal So the CephFS snapshots do not exist, or maybe the subvolume itself does not exist on the CephFS volume, would that be a correct assumption?
@Madhu-1 Delete should succeed if the storage objects are no longer present, as per the CSI protocol, right? So if the above is true, can you check if we return success for a DeleteSnapshot in such cases, thanks!

@ShyamsundarR @Madhu-1
Snapshots do not exist. For at least one of them the subvolume also does not exist.
I haven't had a chance to see if the subvolume is gone for the few left stuck in k8s. I've been chasing other wupuses

@ShyamsundarR I appear to have untangled the issue. But have a new one. I have snapshotcontent objects that don't have snapshots on the filesystem. I cannot delete these kubernetes objects because they fail to delete the ceph object with
E1218 04:00:52.375920 1 goroutinemap.go:150] Operation for "delete-snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11" failed. No retries permitted until 2020-12-18 04:02:54.375893038 +0000 UTC m=+772050.178120189 (durationBeforeRetry 2m2s). Error: "failed to delete snapshot \"snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11\", err: failed to delete snapshot content snapcontent-5f6e1cb0-c123-4ce7-9fa2-2b1052787f11: \"rpc error: code = Internal desc = volume not found\""
--force doesn't work either.

@insanemal So the CephFS snapshots do not exist, or maybe the subvolume itself does not exist on the CephFS volume, would that be a correct assumption?
@Madhu-1 Delete should succeed if the storage objects are no longer present, as per the CSI protocol, right? So if the above is true, can you check if we return success for a DeleteSnapshot in such cases, thanks!

Looks like it's the same as https://github.com/ceph/ceph-csi/issues/1647


Seems to be F_Schlick misprint in shaders generated

Describe the bug
A clear and concise description of what the bug is.
In Filament.html document:
vec3 F_Schlick(float VoH, vec3 f0, float f90) {
return f0 + (vec3(f90) - f0) * pow(1.0 - VoH, 5.0);
}
In the shaders generated:
vec3 F_Schlick(const vec3 f0, float f90, float VoH) {
// Schlick 1994, "An Inexpensive BRDF Model for Physically-Based Rendering"
float f = pow5(1.0 - VoH);
return f + f0 * (f90 - f);
}
seems that f0 and f are swapped
No difference when f90 = 1.0 but in general this matters

Which shader are you talking about? This implementation is not in the engine: https://github.com/google/filament/blob/master/shaders/src/brdf.fs#L141

Filament.html does use this formulation but it specifically uses f90=1 to justify it. Let me know if the incorrect code still appears anywhere else.

Ok. Looks like I have rather old code. The problem I was talking about you already have fixed here:
96d3846 Fix Schlick Fresnel when f90 < 1

Oh yeah that was fixed a while ago. That code was never called with an f90 not set to 1.0 though (if I remember correctly).


no-filter filter has no effect in showing completed tasks

When a task/to-do is completed, it disappears. Choosing no-filter for the To-Dos view doesn't show completed items, as would be expected.
Windows 11, Unigraph 16d448f4

This should be fixed in #313 - no need to reinstall packages or anything, should work out of the box


feat: add ability to pass config path

closes #29
This PR adds a feature I originally intended for the first version but
forgot about until the opened issue. The tool now accepts a new flag
-conf and accepts a relative or absolute path to a config file. If the
flag is not present, it will fall back to the default config.

Thank you!!


Allow for generic behaviours.

This would allow network behaviours to be used together with generics.
It doesn't allow for the generic to be used in a syncvar or RPC methods. Also removed Test that checks if generic behaviours are blocked.

Test added back and working.

Test added back and working.


Premier League Table Bug

Searching for the current Premier League table shows Leicester City 7 points clear at the top.
This cannot possibly b....
oh wait :/

They are 7 points clear: http://premierleague.com/en-gb/league-table.html

Lol only 6 games left... Who knows
On Tuesday, 5 April 2016, Solomon Bier<EMAIL_ADDRESS>wrote:

Searching for the current Premier League table shows Leicester City 7
points clear on top.
This cannot possibly b....
oh wait :/
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/architv/soccer-cli/issues/76

COYS !!!!


Bundle accidentally updated extensions

Built-in, marketplace extensions, now auto-update after publish. @connor4312 and forgot about this and published later versions which are now installed for 1.66. To prevent the same update cycle for 1.66.1 we should simple bundle them

Hi


AMD Radeon driver Update Iissue for Radeon 660M

[Enter feedback here]
AMD driver update from microsoft make the screen stuck at maximum brightness

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 9c82359b-d6a0-4f42-a6e1-a5e83f1edfeb
Version Independent ID: 9c82359b-d6a0-4f42-a6e1-a5e83f1edfeb
Content: Get support for Partner Center dashboard issues - Windows drivers
Content Source: windows-driver-docs-pr/dashboard/hardware-submission-support.md
Product: windows-hardware
Technology: dashboard
GitHub Login: @anaharris-ms
Microsoft Alias: anaharris

Hi @admincyber-id,
This looks like it is a support issue, and not a documentation issue. Please follow the directions here: https://learn.microsoft.com/en-us/windows-hardware/drivers/dashboard/hardware-submission-support#technical-support-issues to submit a support ticket.
Thank you.


Default to Symphonia for MP3 decoding

Symphonia is already a supported backend for MP3 decoding in rodio, but it is not the default. I believe Symphonia MP3 decoder has matured, and it's time to switch to using it by default.
Currently rodio defaults to minimp3 for decoding, which is written in C and causes a number of issues:

Complicated cross-compilation
Potential security issues due to C code being entirely unsafe
iOS app store rejection (see #400)

Symphonia has an MP3 decoder written in 100% safe Rust. The decoding performance is exactly on par with ffmpeg in my tests.
I have tested the correctness of Symphonia's MP3 decoder on over 50,000 real-world files created by a variety of encoders to ensure correct decoding and compatibility with the files found in the wild, which resulted in multiple fixes (see https://github.com/pdeljanov/Symphonia/issues/72, https://github.com/pdeljanov/Symphonia/issues/78, https://github.com/pdeljanov/Symphonia/issues/99). These fixes are going to ship in the upcoming v0.5 along with support for gapless playback for MP3 files.
With safety, performance and compatibility all being in great shape for Symphonia's MP3 decoder, I believe it is time to switch to using it by default instead of minimp3.

Sounds like a plan! I don't really want to completely remove support for the C implementation as long as no extremely liberally licensed alternative exists though (MPL might be too strict for some people).

At a glance, most integration issues between Rodio and Symphonia have been fixed (#379, #401). #405 still outstanding, but the current mp3 adapter doesn't expose the total duration either, so it's not a blocker.

Symphonia v0.5 has now shipped with an extensively tested MP3 decoder (I've run over 500,000 MP3s through it alone, and compared the results to mpg123, libmad and ffmpeg).
This seems like a good time to switch!

I've checked the code and it seems that if you enable the symphonia-mp3 feature, it already is the default. What do you mean by making symphonia the default?

Note that minimp3 is also behind a feature. Both are default off.

Ah! I was under the impression that minimp3 was enabled by default, but it wasn't. This report is invalid then; but I'll open a new issue for switching to Symphonia v0.5

Nevermind, I see you have updated to Symphonia v0.5 in git already.

Actually I just checked the Cargo.toml and minimp3 is a default enabled feature, so there is something left to do, switching symphonia-mp3 with minimp3 in default.

Is there any way I can help move this forward? It should be a rather trivial change.

A PR would be welcome.


Drawing

I think the next best functionality to implement is drawing on NiiVue instances. The way that I've been trying to make this work is essentially experimenting with using drawing as another object class similar to Volumes or Meshes, albeit with different and appropriate attributes. This is how I currently have it written up as a prototype in the .widget file. However, as I look more into how drawings exist in the NiiVue source code, I'm not sure if this is the best approach. Here is how drawings are loaded in for NiiVue:
  async loadDrawingFromUrl(fnm: string, isBinarize = false): Promise<boolean> {
    if (this.drawBitmap) {
      log.debug('Overwriting open drawing!')
    }
    this.drawClearAllUndoBitmaps()
    let ok = false
    try {
      const volume = await NVImage.loadFromUrl(NVImageFromUrlOptions(fnm))
      if (isBinarize) {
        await this.binarize(volume)
      }
      ok = this.loadDrawing(volume)
    } catch (err) {
      log.error('loadDrawingFromUrl() failed to load ' + fnm)
      this.drawClearAllUndoBitmaps()
    }
    return ok
  }

@christian-oreilly I am going to keep experimenting with this as an approach but let me know if you can gather anything from the NiiVue code that would imply a better way to implement this.

as I look more into how drawings exist in the NiiVue source code, I'm not sure if this is the best approach

Your implementation looks clean to me. Did you notice any limitations/issues with the current design?

The problem with this has been that, while the implementation of the drawing and pen tool should've been fully functional, there was never actually an empty 'canvas' being generated to draw on, so nothing was happening.
Drawing is now functional. Before this can be merged, I am going to implement the ability to erase clusters and finish implementing loading in drawings.


uart9 missing from <st/h7/stm32h7.dtsi>

Hi, I've been using the dev kit stm32 nucleo_h723zg and noticed one of its uart (UART9) is not supported, Any particular reason?

@manoj153 No particular reason aside that it missed to no one until now. Feel free to add it via a pull request.

thanks, I am working on it. Will PR

here in my PR:
https://github.com/zephyrproject-rtos/zephyr/pull/33296/commits/6474d37bc6b61ae7df34345ae0153a566bb10883


modified:   second-highest-salary.sql

a second solution added

This is an automatically generated message.
<EMAIL_ADDRESS>is no longer with Yahoo! Inc.
Your message will not be forwarded.
If you have a sales inquiry, please email<EMAIL_ADDRESS>and someone will follow up with you shortly.
If you require assistance with a legal matter, please send a message to<EMAIL_ADDRESS>Thank you!


Integrate SPIRV-Tools validator

This integrates the validator from SPIRV-Tools. It replaces the existing shader_is_spirv function, and adds many more checks.
DON'T MERGE until #342 is merged.
Disabled one test (SPIR-V header version #, currently not covered by SPIRV-Tools). Will investigate adding support for this in that project.

:+1:

This breaks the android build. Please add the new include dir on android as well.

Build fix added.

cherry-picked and pushed to master


Removed required constraint on responses object

This change would allow the following to be a valid OpenAPI description
openapi: 3.1.0
info:
   title: An api with currently undefined responses
   version: 1.0.0
paths:
   /:
     get: {}

This is useful for doing initial design work on an API without having to define what request and response payloads should contain.  Currently to be a valid document it is required to include a responses object, which must contain at least one response code which then must contain a description.

No objections yet.  Leaving this open for folks to stew on it for a few days.


remove .DS_Store files

.DS_Store files shouldn't appear in a codebase. I've removed them all in that PR

Thanks @dbrrt. I missed this PR until now. Will merge asap.


Update vernemq.sh

Minor change in configuring nodename

Hi @vedavidhbudimuri could you explain a bit about the motivation for this change?

We aren't replacing the nodename with the one that is specified by the user.
If I define a subnet for the vernemq nodes and expect vernemq nodes to be communicating on that network then I guess I need an option to configure nodenames.
Will there be any issue if we provide this flexibility (just curious)


Migrate the project to NextJs/React

Is your feature request related to a problem? Please describe.
As the web app grows, we need to be able to maintain it efficiently, hence a framework for the same would be better.
Describe the solution you'd like
Using NextJs will make the app simpler while making sure it is future-secure.
Describe alternatives you've considered
We can stick to Vanilla Js and pure HTML as of now but that limits the learning and scope of this project.

@muditchoudhary and @Susmita-Dey what are your thoughts in this?

@muditchoudhary and @Susmita-Dey what are your thoughts in this?

Sounds good for the project but all credit will go to him after that. I mean to say that nothing would be left for what we did together to make the website.

No @Susmita-Dey , I'll only structure the nextjs codebase and copy the work you guys have done.

For example, we'll break the lengthy HTML file into chunks of small reusable react components.

For example, we'll break the lengthy HTML file into chunks of small reusable react components.

Then it's good.

Could you guys turn this into a hacktoberfest issue?
We can have multiple PRs supporting this.

Also, if we can have a branch on the repo which says react would be great. The react conversion will take some time hence we can't directly merge the upcoming PRs to main

fine! go for it

@akshitagupta15june just create a new branch called react. Only you can do so.

feel free to create any issues

+1 for this. Let me know if you need help creating the components. Also highly recommend using Supabase or another JAMstack compatible database for the backend

Hi, I would also like to contribute to this effort.

@akshaaatt please guide @Mominadar

s the web app gr

Yes, he is correct. I think we need a framework to maintain the website efficiently and speed up the process of development.

@muditchoudhary and @Susmita-Dey what are your thoughts in this?

Yes, he is correct. I think we need a framework to maintain the website efficiently and speed up the process of development.

@akshaaatt, correct me if I am wrong but, I don't think the nextjs server will be compatible with a node js server. Just create a react app for now, and integrate the backend later. @muditchoudhary what do you think?

@DugarRishab , NextJs is a fullstack framework which provides its own backend and server. It is similar to an express app, just the way of doing it is different.

I know that is why said that it might not work. The current backend (yet to be accepted and merged) is a node server.

@akshaaatt, correct me if I am wrong but, I don't think the nextjs server will be compatible with a node js server. Just create a react app for now, and integrate the backend later. @muditchoudhary @akshitagupta15june @Susmita-Dey what do you think?

I don't know much about them. But If next js has its own server and we are using node now how can we use both? I think it's not possible. It is sure we are going to use node for this one as we already have a PR.

@akshaaatt  Can I add a navbar and a footer on the react branch

@Anuran12 I did add a TopBar to the branch but it would be great if you could use Material UI components to make it and convert. Please add the footer. Thanks!

@akshaaatt  Okay..
By the way. I have a Proposal. we are making the react branch from scratch. So would not it be better if we change the UI. Like first make a good UI on figma and then start to make the website according to the UI.

art to make the website according to the UI.

yaa that's a good idea. Do you know Figma? and If yes, could you design the UI?

art to make the website according to the UI.

yaa that's a good idea. Do you know Figma? and If yes, could you design the UI?

I am learning. That's why i told you. And i would like to make a ui for this , it will give me a good practice. 😁✌

art to make the website according to the UI.

yaa that's a good idea. Do you know Figma? and If yes, could you design the UI?

I am learning. That's why i told you. And i would like to make a ui for this , it will give me a good practice. grinv

yaa go ahead. Awesome opportunity to learn it.


A mixed bag of fixes :)

Fixed Build badge link to CI
A few grammar & spelling mistakes
inline format args where appropriate - makes it a bit easier to read and follow
Removed un-needed borrow (per clippy suggestion)
Replaced .rev().next() with next_back() (per clippy)
Ran cargo fmt to keep things a bit tidier

P.S. in all seriousness though - I was looking at some unwrap() - seems they may crash during execution (I am still investigating) - and not certain if it would make sense to change them to Results instead?

Thanks, but I don't think I'm ready to switch to inline format. Let's keep it as is.

@RazrFalcon any objections to the other changes?

Hard to differentiate them, but seems ok.
I just don't like the fact that inline format doesn't support expressions. Feels half-backed.

Thanks. In general I do not accept code formatting and clippy fixes.


[feature request] show results from backtesting

is it possible to show the results from backtesting along side with the percentage chance of being correct?
i feel that this is an important way to show the accuracy of the predictions.
the backtest could be done on 95% of the historical data within the time frame selected.

I have included the requesed feature.  Any further details are in the description of commit 35b16cb.

backtesting is not the same as prediction price.
backtesting is testing the trained transformer on historical data to determine the profit it made using the algorithm.
the percentage accuracy is not good enough because for example, you can be right 95% of the time, winning $1 total, but the last 5% you can lose $100.

maybe i rephrase my question.
for example, if you start with $100 at the beginning of the year and follow the buy and sell signals, how much will you profit(or lose) 3 months later (in march)?

Your proposal sounds more plausible in conjunction with a specified trading strategy on trading a specified asset class.
That goes far and beyond the motivation and desired purpose for this project. Its structure and approach is a lot more general than what you are suggesting.
However, its backend structure can be used as a framework to build on trading strategies that can deliver your stated result. Though that’s a whole other project on its own, requiring a lot of trading expertise and I am not intending to further this project in that direction.


Add License

Hi,
Could you please add a license to your project (like MIT, BSD or Apache2) so others may utilize your code?
Thanks

Thanks for the reminder. License added.


Add sqlfluff as a linter for SQL

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/50
Originally created by @tayloramurphy on 2021-05-18 19:28:55

As we add more dbt models to the Hub implementation of Meltano, we can use sqlfluff to standardize the code.

View 4 previous comments from the original issue on GitLab


How to create ocr post correction from beginning?

I am an intern in one company and they gave me this ocr post correction project but i am new about machine learning and python. Your project exactly does what i want but i am getting lost while analyzing where to start and which steps should i follow. I know this takes your time but it is very important for me to understand and implement this project. I hope you will help. Thanks.

There is documentation on how to get started with nlppln: http://nlppln.readthedocs.io/en/latest/
But I think your question refers to ochre: https://github.com/KBNLresearch/ochre, so I'll close this issue.


Stage I: Pulumi abstraction

In this branch, I've got all our infrastructure replicated, and matched all the pulumi IDs. There are some I've changed the resource ID because it made things a lot easier, but I've catalogued them all here. Spreadsheet that links all the old to new IDs: https://docs.google.com/spreadsheets/d/1Jv1EVvW1qKhf9oUBeCyfcRBcGhM6XWqN933xDLQmfec/edit#gid=0

Hey @lgruen + @violetbrina,
In this branch, I've managed to get all our infrastructure replicated, and matched all the pulumi IDs. There are some I've changed the resource ID because it made things a lot easier, but I've catalogued them all here. Spreadsheet that links all the old to new IDs: https://docs.google.com/spreadsheets/d/1Jv1EVvW1qKhf9oUBeCyfcRBcGhM6XWqN933xDLQmfec/edit#gid=0
Soon I'll:

try deploying this to a new dataset
clean-up the PR + address the TODOs
hopefully get this reviewed.

It's unfortunately a chunky review, but I've totally reimplemented our infrastructure, it's set-up well for the next stages and should work on Azure without little intervention (except Azure specific chunks). A lot of the code in this PR is just boilerplate.
Eg: instead of:
access_group = create_group(...)

# it's now

@property
@lru_cache()
def access_group(self):
    return self.create_group("access")

This penalty is IMO very worth it given our we can organise infrastructure now, without having to pass state around to lots of variables.

Okay good news! Previously it was still trying to delete + recreate a few resources (like service-accounts, secrets) - with this PR (https://github.com/populationgenomics/analysis-runner/pull/460) I've managed to get all resources to match with no change (except an update to display names on a few service-accounts).
I'm getting a lot of "recreate" for memberships (BucketIAMMember, IAMMember, GroupMember, RepositoryIamMember, IamMember, SecretIamMember), but this is due to a member key change, as listed below. Although @lgruen, you mentioned this can sometimes cause a problem - the easiest may be to delete all the memberships and reapply them.
 Noting that there are no changes to any specific resources, just memberships.
+ 18 to create
~ 2 to update
- 21 to delete
+-3 to replace
44 changes. 193 unchanged

-   gcp:cloudidentity:GroupMembership   web-access-group-access-group-membership                  delete
+   gcp:cloudidentity:GroupMembership   full-group-cache-membership                               create
+   gcp:cloudidentity:GroupMembership   sample-metadata-main-read-group-cache-membership          create
+   gcp:cloudidentity:GroupMembership   sample-metadata-test-read-group-cache-membership          create
+   gcp:cloudidentity:GroupMembership   sample-metadata-test-write-group-cache-membership         create
+   gcp:cloudidentity:GroupMembership   standard-group-cache-membership                           create
+   gcp:cloudidentity:GroupMembership   test-group-cache-membership                               create
-   gcp:cloudidentity:GroupMembership   access-group-cache-full-access-level-group-membership     delete
-   gcp:cloudidentity:GroupMembership   access-group-cache-standard-access-level-group-membership delete
-   gcp:cloudidentity:GroupMembership   access-group-cache-test-access-level-group-membership     delete
-   gcp:cloudidentity:GroupMembership   sample-metadata-group-cache-main-read-group-membership    delete
-   gcp:cloudidentity:GroupMembership   sample-metadata-group-cache-main-write-group-membership   delete
-   gcp:cloudidentity:GroupMembership   sample-metadata-group-cache-test-read-group-membership    delete
-   gcp:cloudidentity:GroupMembership   sample-metadata-group-cache-test-write-group-membership   delete
+   gcp:cloudrun:IamMember              sample-metadata-access-group-cloudrun-invoker             create
+   gcp:cloudrun:IamMember              sample-metadata-main-read-cloudrun-invoker                create
+   gcp:cloudrun:IamMember              sample-metadata-main-write-cloudrun-invoker               create
+   gcp:cloudrun:IamMember              sample-metadata-test-read-cloudrun-invoker                create
+   gcp:cloudrun:IamMember              sample-metadata-test-write-cloudrun-invoker               create
-   gcp:cloudrun:IamMember              sample-metadata-service-account-full-invoker              delete
-   gcp:cloudrun:IamMember              sample-metadata-service-account-standard-invoker          delete
-   gcp:cloudrun:IamMember              sample-metadata-service-account-test-invoker              delete
+   gcp:projects:Service                iam-service                                               create
+   gcp:secretmanager:SecretIamMember   analysis-runner-access-group-cache-secret-accessor        create
+   gcp:secretmanager:SecretIamMember   analysis-runner-full-group-cache-secret-accessor          create
+   gcp:secretmanager:SecretIamMember   analysis-runner-standard-group-cache-secret-accessor      create
+   gcp:secretmanager:SecretIamMember   analysis-runner-test-group-cache-secret-accessor          create
-   gcp:secretmanager:SecretIamMember   access-group-cache-secret-accessor                        delete
-   gcp:secretmanager:SecretIamMember   analyis-runner-access-group-cache-secret-accessor         delete
-   gcp:secretmanager:SecretIamMember   full-group-cache-secret-accessor                          delete
-   gcp:secretmanager:SecretIamMember   sample-metadata-main-read-group-cache-secret-accessor     delete
-   gcp:secretmanager:SecretIamMember   sample-metadata-main-write-group-cache-secret-accessor    delete
-   gcp:secretmanager:SecretIamMember   sample-metadata-test-read-group-cache-secret-accessor     delete
-   gcp:secretmanager:SecretIamMember   sample-metadata-test-write-group-cache-secret-accessor    delete
-   gcp:secretmanager:SecretIamMember   standard-group-cache-secret-accessor                      delete
-   gcp:secretmanager:SecretIamMember   test-group-cache-secret-accessor                          delete
~   gcp:serviceAccount:Account          service-account-main-upload                               update      [diff: -displayName]
~   gcp:serviceAccount:Account          service-account-notebook-fewgenomes                       update      [diff: -displayName]
+   gcp:storage:BucketIAMMember         project-buckets-lister                                    create
+-  gcp:storage:BucketIAMMember         full-analysis-runner-config-viewer                        replace     [diff: ~member]
+-  gcp:storage:BucketIAMMember         standard-analysis-runner-config-viewer                    replace     [diff: ~member]
+-  gcp:storage:BucketIAMMember         test-analysis-runner-config-viewer                        replace     [diff: ~member]

Thanks @lgruen, I've updated all the stack from analysis-runner. I'll create a PR to remove all stack-related stuff from there.

🎉


Create  code coverage reports

Create code coverage reports for Core project

I do not know if this is helpful, but we could take a look at:
https://coveralls.io/


"Used by (count)" badge for GitHub Actions

:clipboard: Description

This badge should show how many other repositories use a given GitHub Action for their workflows.
:link: Data

Implementation by EndBug: https://github.com/EndBug/api/blob/master/api/github-actions/used-by.ts
Example usage with also useful click action: https://github.com/EndBug/add-and-commit/blob/0c4298a18d3a9d33e3b879c7b73b4a457547d949/README.md
:microphone: Motivation

It would give a better measurement about usage than stars do.

This mechanism of performing a free text search for the action name seems a bit crude, especially as GH's text search ignores the / character and views them as 2 strings. I suspect "EndBug add-and-commit" is sufficiently obscure not to raise false positives, but I can see other action names generating some false-positives.
While I probably wouldn't want to frame this as "here's how many people are using your GH action", maybe this concept has legs as a more general search results badge?

Ironically we were just discussing this (https://github.com/badges/shields/issues/4559#issuecomment-703169533)

maybe this concept has legs as a more general search results badge?

I could certainly see that as well


:art: Refactor string concatenation in model descriptions

Just some minor artsy cleanup --
I noticed that using the backslash for line continuation includes all the whitespace of the code indentation in the final string. So, the openapi docs would generate a description as:
The value is used once for a particular domain and window of time.                     This value is ...

So, quick fix: replace \ for line continuation with string concatenation

@jamshale Thanks! Forgot to check


Don't expire sessions

Sessions currently expire after two weeks of inactivity. There's no good reason for that. Sessions should last forever.

Sessions now last one year.


Installation on linux

I want to install the latest version on Linux. I have tried both ways.

pip install imagecodecs
It installs 2020.5.30 version, no matter what I do

python setup.py install
It tries to install the latest version(2021.3.31) but it throws an error.

Terminal output::

python setup.py install
running install
running bdist_egg
running egg_info
writing imagecodecs.egg-info/PKG-INFO
writing dependency_links to imagecodecs.egg-info/dependency_links.txt
writing entry points to imagecodecs.egg-info/entry_points.txt
writing requirements to imagecodecs.egg-info/requires.txt
writing top-level names to imagecodecs.egg-info/top_level.txt
reading manifest file 'imagecodecs.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '3rdparty/'
warning: no files found matching 'tests/.zfp'
warning: no files found matching 'tests/.tif'
warning: no previously-included files found matching 'tests/tiff/gray.movie.u1.tif'
warning: no previously-included files found matching 'tests/libaec/121B2TestData/ExtendedParameters/.'
warning: no previously-included files found matching 'tests/33792x79872.jpg'
warning: no previously-included files found matching 'tests/jpegxr.czi'
warning: no previously-included files matching 'pycache' found under directory ''
warning: no previously-included files matching '.py[co]' found under directory ''
warning: no previously-included files matching '-' found under directory ''
warning: no previously-included files matching '.html' found under directory ''
warning: no previously-included files matching 'Copy.' found under directory ''
writing manifest file 'imagecodecs.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
skipping 'imagecodecs/_aec.c' Cython extension (up-to-date)
skipping 'imagecodecs/_bitshuffle.c' Cython extension (up-to-date)
skipping 'imagecodecs/_blosc.c' Cython extension (up-to-date)
building 'imagecodecs._blosc' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iimagecodecs -I/usr/include/python3.6m -I/usr/local/lib/python3.6/dist-packages/numpy/core/include -c imagecodecs/_blosc.c -o build/temp.linux-x86_64-3.6/imagecodecs/_blosc.o
In file included from /usr/local/lib/python3.6/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1822:0,
from /usr/local/lib/python3.6/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /usr/local/lib/python3.6/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from imagecodecs/_blosc.c:593:
/usr/local/lib/python3.6/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with "
^
imagecodecs/_blosc.c:597:19: fatal error: blosc.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

v2020.5.30 is the last version that works with Python 3.6.
To build newer versions from source for Python >=3.7, try the instructions at https://github.com/cgohlke/imagecodecs/blob/2ef2a2ebb4e5b8ea7289c6e2fc6ea289e870af48/imagecodecs/imagecodecs.py#L140-L171


[FLINK-8804][Build] Integrate flink-shaded 3.0

What is the purpose of the change
This PR integrates flink-shaded 3.0 .
Brief change log

pin flink-shaded-netty to 2.0
bump flink.shaded.version to 3.0
replace jackson dependency in flink-sql-client
replace force-shading with flink-shaded-force-shading
remove jackson relocation in flink-dist (now unnecessary as service relocation is fixed)
replace jackson dependency in flink-docs
rebuild REST API docs that were affected by original jackson usage

Verifying this change

jackson dependency replacement are covered by compilation&tests

verify all maven properties are properly resolved

verify jackson services are properly relocated in flink-dist

verify REST API docs list the correct field names (conceptually! you don't need to check everything)

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

I've rebased the PR and minimized it to only contain the jackson changes.


[NeuralChat] Support vLLM serving

Type of Change
feature
Description
In this PR, we keep the native GPU-only implementation from vllm. There is also a CPU based PagedAttention implementation on IPEX but seems still experimental. We may support CPU based optimization through its ipex.optimize_transformers in the future.
A very hands-on example is given in this PR, targeting the example in https://docs.vllm.ai/en/latest/getting_started/quickstart.html. Users can replace the model name and add plugin just by editing the corresponding yaml.
Expected Behavior & Potential Risk
Support vllm serving
Batch inference compatibility is still under checking.
How has this PR been tested?
UT on GPU, intel_extension_for_transformers/neural_chat/tests/ci/server/test_textchat_server_vllm.py
Dependency Change?
vllm

To synchronize our detailed context and make some clarification,
This PR is enough for users to leverage the vLLM optimized models in NeuralChat pipelines (to replace the text generation part of the original chatbot). However vLLM is cuda-only so this PR requires a NV GPU.
Another GPU-free use-case is to deploy a chatbot on Xeon CPU only. For the Xeon CPU path, We cannot directly use or slightly adapt vLLM to get similar performance boost for serving on Xeon CPU for the following reasons:

PagedAttention Kernel is bounded to vLLM implementation, and is not really needed because CPU has enough continuous memory. That means the external fragments in paper Ch. 3.1 , which vLLM is mainly designed to solve, should not be a bottleneck on CPU
vLLM bound the model inference to CUDA ops, which cannot be reused by our CPU-only solutions
vLLM needs to swap-in/out blocks between CPU/GPU (when disable re-computation) treat CPU as an infinite memory warehouse when GPU does not have enough memory

Apart from the above inconsistencies, we can still learn from/re-implement vLLM continuous batching scheduling part for CPU. As a byproduct, the internal fragments in the paper can also be resolved although we do not initially aim for it. ITREX runtime kernels may be integrated here. Users just need to change the device to "CPU" to use this solution.


social media icons

Greetings, I want to add some social media icons in the footer section of the website

hello,
I would like to add the social media icon links to the footer section and make it responsive on all media widths for the best user experience.
Regards Priyam Maini

creating attractive social media icons for websites involves a combination of design principles and technical implementation.
Determine the design style: Consider the overall look and feel of your website and decide on a design style for the icons. They should match the aesthetics and branding of your website while being visually appealing.
Choose the right icon set: Select a set of social media icons that align with your design style. You can find many free and premium icon sets online. Ensure the icons are recognizable and represent the respective social media platforms clearly.
Customize the icons: To make the icons attractive and unique, you can customize them using graphic design software such as Adobe Photoshop or Illustrator. Here are some customization ideas:
Adjust the color scheme to match your website's color palette.
Add subtle gradients, shadows, or highlights to give them depth.
Experiment with different shapes, outlines, or borders.
Incorporate your logo or brand elements into the icons.
Optimize the icons for web: After customizing the icons, you need to optimize them for web use. Consider the following aspects:
Size: Resize the icons to an appropriate dimension that fits well within your website layout.
File format: Save the icons in a suitable web format, such as PNG or SVG, to maintain quality and ensure compatibility.
File size: Compress the icons to reduce file size without compromising too much on quality. This helps improve website loading speed.
Placement and visibility: Decide where to position the social media icons on your website. They are commonly placed in the header, footer, sidebar, or floating bar. Ensure the icons are easily noticeable and accessible to visitors.
Hover effects: Consider adding interactive hover effects to the icons. For example, the color could change, or a subtle animation or tooltip could appear when users hover over an icon. This enhances the user experience and makes the icons more engaging.
Linking the icons: Once you have your customized social media icons, you need to link them to your respective social media profiles. Use the appropriate URLs for each platform to ensure visitors are directed to the correct page.
Testing and responsiveness: Test the icons across different devices and screen sizes to ensure they display properly and are responsive. Icons should adapt well to different screen resolutions, maintaining their visibility and usability.

i am working it


bitcoin: Fix issue with parsing txs with 1 input and no outputs

As mentioned in #167, the Tx class in @node-dlc/bitcoin is currently unable to parse a base transaction with no inputs and 1 output.
This is because zero inputs and 1 output is serialized as "0001", which also happens to be the witness marker and version flag. However, these are invalid witness transactions because a Tx needs to have a witness to be considered a witness transaction.
Example tx:<PHONE_NUMBER>0100e1f5050000000017a9148fe46e05e329badba1c390a5ea2c0ad7de2059cd8700000000
There is support for parsing this type of transaction in Bitcoin Core and Bitcoin-s (perhaps others, I haven't checked).
This PR adds a throw/catch in Tx.parse to solve this issue.

[x] Add throw/catch to Tx.parse to allow base transactions with no inputs and 1 output to be parsed
[x] Add test for base transactions to Tx.spec.ts

Fixes #167

Codecov Report

Merging #168 (53a1cde) into main (546bd1d) will decrease coverage by 0.02%.
The diff coverage is 92.00%.

:exclamation: Current head 53a1cde differs from pull request most recent head 36c9bf9. Consider uploading reports for the commit 36c9bf9 to get more accurate results

@@            Coverage Diff             @@
##             main     #168      +/-   ##
==========================================
- Coverage   94.20%   94.17%   -0.03%
==========================================
  Files         168      168
  Lines        4520     4536      +16
  Branches      542      544       +2
==========================================
+ Hits         4258     4272      +14
  Misses        137      137
- Partials      125      127       +2

Impacted Files
Coverage Δ

packages/bitcoin/lib/Tx.ts
87.71% <92.00%> (-0.03%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 546bd1d...36c9bf9. Read the comment docs.


Add podspec, fix typo issues

There was typo in methods named
I added podspec file so that it can be added to a Podfile with the github link
I did move notifications.swift to core too.

I agree, this repo should be for OrangeIRCCore. Furthermore, I will rename OrangeIRCCore to just be OrangeIRC in order to simplify things.
Thank you for your contributions. I will create a new repo for the iOS app.


Add new ML cluster settings for 2.5

Signed-off-by: Naarcha-AWS<EMAIL_ADDRESS>Checklist

[x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Going to merge in for now and change the formatting to a table in a separate PR.


CBDT header version

Using emojicompat with https://github.com/googlefonts/emojicompat/pull/15 applied :
$ emojicompat --font ~/oss/noto-emoji/fonts/NotoColorEmoji-emojicompat.ttf
# dump of compat data omitted
WARNING: CBDT is at version 3.0. Version 2 is required, do NOT use it for emojicompat
         Running any emojicompat operation that saves the font will fix the problem
WARNING: CBLC is at version 3.0. Version 2 is required, do NOT use it for emojicompat
         Running any emojicompat operation that saves the font will fix the problem

$ emojicompat --font ~/oss/noto-emoji/fonts/Noto-COLRv1-emojicompat.ttf
# dump of compat data omitted
# does NOT print a warning as it has no bitmap tables so they can't be misversioned

$ emojicompat --font NotoColorEmoji-emojicompat.ttf --op setup_pua
0 PUA missing
0 PUA point at wrong glyph
3717 PUA correct
0 Emji entries did NOT match a glyph
WARNING: CBDT is at version 3.0. Version 2 is required, fixing that for you...
WARNING: CBLC is at version 3.0. Version 2 is required, fixing that for you...
Updating NotoColorEmoji-emojicompat.ttf

$ emojicompat --font ~/oss/noto-emoji/fonts/Noto-COLRv1-emojicompat.ttf
# now does *not* print a warning

Confirmed working on API23 emulator with prior repro


Proposal for fixing mirroring issues in Node-Red

Bi-directional languages are written from left-to-right. GUI mirroring is necessary for any application which has GUIs meant to be translated to a language based on bidirectional scripts such as Arabic and Hebrew. Several issues were noticed with the alignment and mirroring on viewing Node-red in right-to-left mode.
Attached is an assessment of the level of mirroring needed for Node-Red along with a proposed design to provide the mirroring support.
Node-Red Assessment.pdf

Thanks for raising this. We have a team who have also been looking at this topic, see issue #812.
Is this proposal a prelude to raising a pull-request with actual changes in? We'd need to co-ordinate anything you were proposing with the work being done by @tomerm and team.

Hi @knolleary , @AliaNaguib  is working for Egyptian Bidi team. Both teams join their efforts in addressing Bidi functionality in open source tools / projects. @AliaNaguib will most probably submit a patch after you finalize the design discussion. Please observe that code my team has contributed so far was around proper display of bidirectional data. Alia is tackling a different requirement - GUI mirroring (which is mandatory when GUI is translated to Arabic / Hebrew. Those two functional areas (GUI mirroring and proper display of bidirectional data) are orthogonal to each other. So, I would not expect any overlap or any contradiction / collision both in terms of location of suggested code changes and also functionally from end user perspective.

Hi @knolleary and @tomerm:
This is Ayah Shamandi - one of Egyptian Bidi Team too - and I am here to continue what @AliaNaguib started in, and very soon I will submit a patch for solving GUI mirroring defects based on Alia`s approach.
@Knolleary: I have a question, Enabling GUI mirroring is a pre-req for supporting Node-red with Arabic & Hebrew translation (or any Bi-directional  languages) so what is your procedure/process of translation?

Hi,
We need to discuss this work before you make any changes. We're currently working on a very tight schedule for our next release so need to make sure this work is as nondisruptive as possible.
Can we have a call later this week? Please drop me an email so we can discuss. I also need to sort out the ibm process side to allow us to accept any changes from you.
Thanks,
Nick

Closing as this work came to a halt.


Perl modules?

Does this still include perl modules to interface with cbitcoin?

You need libcbitcoin-perl, which is another one of my got repos.

On 2015/03/29, at 22:14, buzztiaan<EMAIL_ADDRESS>wrote:
Does this still include perl modules to interface with cbitcoin?

Reply to this email directly or view it on GitHub.

Derp, ty


Can   plot_cg(cfg.kb, 'out.png')    call use demangled_name for function node names?

The function callgraph generated by    plot_cg()     does not use demanged_names. (demanged_names appear to be available for non-stripped ELF binary files, and possibly for windows PE files w/ accompanying .pdb file though I have not tested this myself.)
Looks like in angr-dev/bingraphvis/bingraphvis/base.py in VisPipeLine.process() perhaps this can be handled? (I may have this totally wrong, I just looked around code a bit and thought this may be where support for use of demangled_names may go.)
Class VisPipeLine(object):
    ...
    def process(self, filter=None):
        if filter is None:
            graph = self.graph
        else:
            graph = self.graph.filtered_view(filter)

        for ea in self.edge_annotators:
            ea.set_graph(graph)

        for na in self.node_annotators:
            na.set_graph(graph)

        for n in graph.nodes:
            for c in self.content.values():
                c.render(n)
            for na in self.node_annotators:
                na.annotate_node(n)

        for e in graph.edges:
            for ea in self.edge_annotators:
                ea.annotate_edge(e)
        for c in self.clusterers:
            c.cluster(graph)

        return graph

Hi, sorry for my late answer!
I've not too much freetime lately, and forgot to react to the issue + i'm not working with angr since years, so I abandoned this project a bit :-(.
CG nodes are generated here, so probably this is the code that needs to be changed:
https://github.com/axt/bingraphvis/blob/0662ec6aaaa08c392b1f16d3fa8dffbdafe17466/bingraphvis/angr/content.py#L105
The CG pipeline is created here, so if you want to make demangling optional, you can add a new constructor parameter to the AngrCGHead class:
https://github.com/axt/bingraphvis/blob/0662ec6aaaa08c392b1f16d3fa8dffbdafe17466/bingraphvis/angr/factory.py#L66-L72
There is some preliminary demangle support in CFG visualization (in the comments for x86), probably only working on linux, and very ineffective, because it calls c++filt for every name resolution:
https://github.com/axt/bingraphvis/blob/0662ec6aaaa08c392b1f16d3fa8dffbdafe17466/bingraphvis/angr/x86/annotator.py#L85-L91
https://github.com/axt/bingraphvis/blob/0662ec6aaaa08c392b1f16d3fa8dffbdafe17466/bingraphvis/angr/x86/annotator.py#L111
AFAIK angr supports demangled names now out-of-the box, so using that name would be probably much better than calling c++filt.
Not sure if its still a relevant problem for you with 4 months delay (sorry again), but if you want to try to do the fix, I'm happy to accept PRs, otherwise I will add this eventually, but at the moment I don't have anything setup for angr development currently, and also not enough free time.


LD_PRELOAD should not be used for injecting security-critical code because of glibc fail-open behavior

If you use LD_PRELOAD for injecting security-critical code, that's extremely brittle because glibc will just plow ahead if loading LD_PRELOAD libraries fails:
$ LD_PRELOAD=/foo/bar/baz /bin/echo hello world
ERROR: ld.so: object '/foo/bar/baz' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
hello world

If the executable runs without CAP_SYS_ADMIN, an attacker could e.g. deliberately trigger such an error condition by temporarily opening so many files that the system-global limit is reached and open() returns -ENFILE.
Therefore, anything that relies on LD_PRELOAD for sandboxing is probably a bad idea, unless you have a safety net that somehow ensures that the LD_PRELOAD operation succeeded.

Yes, LD_PRELOAD is a bit fragile approach, that's why we propose an option of doing patchelf (or even linking on build time) to make the dependency permanent. In this case the application would refuse to start, if it can't load the library.

The README currently reads to me as if the LD_PRELOAD approach is a suggested usage pattern. If that is not the case, it may be a good idea to add a big warning at the top of the "Dynamically linked executables" section.

It is a suggested usage pattern, at least for now.

Just to clarify more: while LD_PRELOAD has its downsides, we think it should be offered as an option still, because it helps in some cases, where patchelf is not possible. For example, a system with enabled and enforced Linux IMA might not allow patching executables, so LD_PRELOAD is the only? option.
Also, LD_PRELOAD is easier to configure. Going back to the global file descriptor example: in most well monitored production systems a global file descriptor exhaustion will likely cause many more issues and would not go unnoticed in the first place.
Finally, some other users pointed out that there are similar solutions using similar approaches and LD_PRELOAD specifically, like Google minijail project, so it seems other engineers find value in the approach.

For example, a system with enabled and enforced Linux IMA might not allow patching executables, so LD_PRELOAD is the only? option.

Or you could use a different loader, or let your code signing infrastructure sign the patched executable, or...

in most well monitored production systems a global file descriptor exhaustion will likely cause many more issues and would not go unnoticed in the first place

Even if it is a deliberately targeted, extremely short spike?

Finally, some other users pointed out that there are similar solutions using similar approaches and LD_PRELOAD specifically, like Google minijail project

That's not a reason to repeat their mistakes.

Even if it is a deliberately targeted, extremely short spike?

This assumes the OS is already compromised. seccomp is as only good as a measure, if the OS is trustworthy. If it is not, any kernel security mechanism does not make sense. In other words, if I'm malicious and can do a deliberately targeted, extremely short spike or resource exhaust in the OS, most likely I will not bother with trying to bypass seccomp by timing exactly when a sandboxed process is starting, but will rather do other damage in an easier way.

In other words, even if an attacker only gains code execution in a process that has the ability to fork and to open /dev/null (and not much else), you consider the OS to be compromised?

I'm not sure that only with fork and open("/dev/null") it is possible to get a "deliberately targeted, extremely short spike"
especially if you just take defaults: https://github.com/torvalds/linux/blob/7c30b859a947535f2213277e827d7ac7dcff9c84/fs/file_table.c#L400

There are any number of other legitimate purposes for LD_PRELOAD (or other options like LD_AUDIT) that could be defeated if it fails to load the object for this kind of reason. If there's a bug here it can be argued that it's in glibc. If someone is that concerned, maybe a patch should instead be sent to glibc to add an environment variable like LD_EXIT_ON_ERROR=(preload,symbol-resolution,relocation,audit,...,any) so that one can be 100% confident that either a program starts with the LD_PRELOAD having been successful or it doesn't start at all.

I hope I'm not misunderstanding the intention of the comments in defense of LD_PRELOAD, but to me this reads as "yeah LD_PRELOAD has known security issues, you should use patchelf instead, but we still recommend it".
At this point the README should at very least document the risks of using LD_PRELOAD, in my opinion, or it can be misleading. Security should always come with sane defaults, but when it doesn't, it should warn the users loudly.

It seems the more generic problem outlined here is how to make sure the process we're sandboxing is actually sandboxed and the rules were applied. I've added a section to the README describing one of the ways: https://github.com/cloudflare/sandbox#check-the-status

Thank you for adding that section @ignatk ! It's not the big warning that I was hoping for, but it's definitely an improvement


Update tablediff-utility.md

The document does not clarify where the file is and it is not clear that it is not exists if you do not add the replication feature of the SQL Server. There are many questions in the forum regarding where the file is and why it is not in the location people claim it should be.

@markingmyname
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


How to access logged-in user data

I am currently running feathers v4, and using custom JWTStrategy to add user role into JWT token. With this implementation i am unable to access authenticated web-services.
Is there any way to get permission with default JWTStrategy ?
Please check my code
authentication.js
const { AuthenticationService, JWTStrategy } = require('@feathersjs/authentication');
const { LocalStrategy } = require('@feathersjs/authentication-local');
const { expressOauth } = require('@feathersjs/authentication-oauth');

class LegacyAuthenticationService extends AuthenticationService {
  async getPayload(authResult, params) {
    // Call original `getPayload` first
    const payload = await super.getPayload(authResult, params);
    const { user } = authResult;

    if (user && user.roles) {
      payload.roles = user.roles;
    }

    return payload;

  }
}

class LegacyJWTStrategy extends JWTStrategy {
  getEntityId(authResult) {
    const { authentication: { payload } } = authResult;

    return payload.roles || payload.sub;
  }
}

module.exports = app => {
  const authentication = new LegacyAuthenticationService(app);

  authentication.register('jwt', new LegacyJWTStrategy());
  authentication.register('local', new LocalStrategy());

  app.use('/authentication', authentication);
  // app.configure(expressOauth());
};

`
users.hooks.js
`const { authenticate } = require('@feathersjs/authentication').hooks;

const {
  hashPassword, protect
} = require('@feathersjs/authentication-local').hooks;

module.exports = {
  before: {
    all: [],
    find: [ authenticate('jwt') ],
    get: [ authenticate('jwt') ], <--- This function is unable to decode custom JWT
    create: [ hashPassword('password') ],
    update: [ hashPassword('password'),  authenticate('jwt') ],
    patch: [ hashPassword('password'),  authenticate('jwt') ],
    remove: [ authenticate('jwt') ]
  },

  after: {
    all: [
      // Make sure the password field is never sent to the client
      // Always must be the last hook
      protect('password')
    ],
    find: [],
    get: [],
    create: [],
    update: [],
    patch: [],
    remove: []
  },

  error: {
    all: [],
    find: [],
    get: [],
    create: [],
    update: [],
    patch: [],
    remove: []
  }
};
`

Error

GET Api Url: http://localhost:3030/users

{"name":"BadRequest","message":"Cast to ObjectId failed for value "admin" at path "_id" for model "users"","code":400,"className":"bad-request","errors":{}}

I am not using feather v4, here an example with working permissions
https://github.com/kfern/feathers-aor-test-integration

I have started my project by referring this example, but due to some major changes in authentication of feathersjs i am not able to figure out the issue i mentioned.

I am using feathers v4 with a similar configuration as you described without issues. Are you sure the error is due to authentication config? From your error message,
Cast to ObjectId failed for value "admin" at path "_id" for model "users"
It seems like it is trying to cast value "admin" to ObjectId type, which would fail as expected. Could the problem be that _id has the value "admin" and it might not be correct?


No data in Data folder

Hi Microsoft team,
Your project is great. However, I could not find data in CreditRiskPrediction project.
Could you update data?

Actually there is data but the description says "Link to the Full Data Set" and gives a size e.g. for transactionSimu.csv of 22,565 kb but when following the link the file acceleratoRs / CreditRiskPrediction / Data / transactionSimu_v3.csv has only 1.11 mb and seems to have the shape of the reduced dataset. @ZhouFang928 could you please check? Many thanks!

Hi all,
We used simulated dataset of larger size (described in the data description) to generate the html file for credit risk prediction and then reduced the size of the data in the repo so as to shorten the processing time. People can get the result more quickly by using the smaller data set for demonstration and learning purpose.

Thanks a lot for your updating.
I have had data and played with it.
Rgds
Anh
On 8 August 2017 at 09:56, Zhou Fang<EMAIL_ADDRESS>wrote:

Hi all,
We used simulated dataset of larger size (described in the data
description) to generate the html file for credit risk prediction and then
reduced the size of the data in the repo so as to shorten the processing
time. People can get the result more quickly by using the smaller data set
for demonstration and learning purpose.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/acceleratoRs/issues/3#issuecomment-320836192,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQnYxanzxhDpP6omJlRrn0cVUrXeH4Hiks5sV85tgaJpZM4OXwKn
.


Gem publishing function

This change adds a common function for use in a Jenkinsfile to publish a gem, which used to be performed with the Gem Publisher gem.
Getting this gem to work with Jenkins 2 was non-trivial, so this behaviour was migrated to Jenkins 2 on govuk_navigation_helpers. Since then, this behaviour has also been moved successfully to gds-api-adapters and slimmer.
This functionality is now being commonised into Puppet.

Thanks for doing this Alex, this solves the problems I was encountering, and gave up trying to fix with the gds-api-adapters build :)


Add Django 5.1+ support for CheckConstraint condition with backward compatibility

This PR adds support for CheckConstraint.condition in Django 5.1 while maintaining backward compatibility with Django 4.2 by dynamically selecting between condition and check based on the Django version.

https://github.com/RealOrangeOne/django-tasks/issues/100

@RealOrangeOne The current issue with mypy is that it indicates that condition is not defined. Should this be adjusted to match the version of django-stubs?
Here is the relevant line in django-stubs:
https://github.com/typeddjango/django-stubs/blob/df12deb6b30623c0661b920a4eece8135306b55d/django-stubs/db/models/constraints.pyi#L48C1-L67C19

If there's an overload, I'm surprised this is failing. If dependencies need updating, feel free to do so. Alternatively, I'm ok adding a type ignore for these lines.

If there's an overload, I'm surprised this is failing. If dependencies need updating, feel free to do so. Alternatively, I'm ok adding a type ignore for these lines.

I've added a type ignore for these lines as suggested. :)


cmd/compile: read and process import data lazily

As an experiment, I changed the compiler to report the number of packages, types, and functions built up as a result of processing imports for a given package:
$ git diff
diff --git a/src/cmd/compile/internal/gc/bimport.go b/src/cmd/compile/internal/gc/bimport.go
index ea9c02dea8..4fee3eee41 100644
--- a/src/cmd/compile/internal/gc/bimport.go
+++ b/src/cmd/compile/internal/gc/bimport.go
@@ -15,6 +15,7 @@ import (
 	"encoding/binary"
 	"fmt"
 	"math/big"
+	"os"
 	"strconv"
 	"strings"
 )
@@ -220,6 +221,10 @@ func Import(imp *types.Pkg, in *bufio.Reader) {

 	p.verifyTypes()

+	if os.Getenv("IMPORTDEBUG") != "" {
+		fmt.Fprintf(os.Stderr, "import %v: %d pkg + %d type + %d func\n", imp.Path, len(p.pkgList), len(p.typList), len(p.funcList))
+	}
+
 	// --- end of export data ---

 	typecheckok = tcok
$

This program has no imports, so no overhead:
$ cat x.go
package main

func main() {
	println("hello world")
}
$ IMPORTDEBUG=1 go build x.go
$

This program imports fmt, which turns out to mean processing 38 types and 20 functions, only one of which is needed:
$ cat x.go
package main

import "fmt"

func main() {
	fmt.Println("hello world\n")
}
$ IMPORTDEBUG=1 go build x.go
# command-line-arguments
import fmt: 2 pkg + 38 type + 20 func
$

This program imports net/http, which turns out to mean processing 231 types and 857 functions, only one of which is needed:
$ cat x.go
package main

import "net/http"

func main() {
	http.ListenAndServe(":8080", nil)
}
$ IMPORTDEBUG=1 go build x.go
# command-line-arguments
import net/http: 29 pkg + 231 type + 857 func
$

The overhead here is more than 100X. Many real-world packages are even bigger than the ones in the standard library.
In a typical source file or package, especially during an incremental build, I suspect the compiler spends significantly more type loading and type-checking all these unnecessary imported definitions than it does the necessary one and the actual source file.
Now that we have a clean encapsulation of import/export format handling, it seems like the next step should be to make the import/export format indexed in some way, so that it can be consulted on demand during the compilation rather than loaded entirely up front. That is, when you import "fmt" you'd open the export data for fmt, and then when you get to fmt.Println you'd pull in just what you need to understand fmt.Println (in this case, likely just the function type).
Similarly, for the HTTP server, invoking http.ListenAndServe would mean pulling in the function type and possibly then also learning about http.Handler, since that is the type of the second argument. That might in turn require learning about some more types, since Handler's ServeHTTP takes an http.ResponseWriter and a *http.Request. The latter is a struct with many other types, and it might require loading some large number of other types. Ideally this could be cut off through more lazy loading (none of this is required to type check that the argument
nil is in fact a *http.Request). Even if not, though, the total number of types and functions
should still be smaller than before.
The problem will only get worse as packages grow. For protobuf-generated code, where the export data summarizes essentially the entirety of all imported packages recursively, not loading this information into the compiler until it is needed would be a big win.
Even in the main repo, there are indications this could be a big win.  My guess is that, for any single package being compiled, nearly all of the function bodies included in export data are irrelevant (but which ones are irrelevant changes depending
on who is importing the package, of course). Under the assumption that on-demand processing of the function bodies would make -l=0 (inlining on) behave more like -l=1 (inlining off), that would reduce build times by about 10% (6.65s vs 7.27s). It would also keep the more aggressive inlining from increasing build times by 25% (9.28s vs 7.27s).
I'm assuming here that the increase in time is due to importing the new data and not writing it out. I haven't checked that, but the export data is typically read back many times but only written out once (or zero times) during a given build, so I think that's a reasonable assumption.
It's obviously too late for Go 1.9, but it seems like this would be an effective way to speed up the compiler in the Go 1.10 cycle.
/cc @aclements @griesemer @mdempsky

@mdempsky were just discussing this as well. I noticed it with the cmd/compile/internal/ARCH packages, whose compilation time appears to be dominated by reading the import data for the ssa package, which has lots of export data. @mdempsky was going to look into whether we can reduce the size of that export data by making better decisions about which functions might potentially be inlined by other packages, which might (maybe) happen for 1.9, and which would help both writing and reading.
One other bit of anecdotal evidence that this matters came from @davecheney's observation that the biggest compile time speedup last cycle came from switching to the binary export format.

Nice observation. I'd argue that in a more realistic scenario, a single package probably uses more functionality from an imported package; and that same package may be imported multiple times by a larger package (and then the import happens only the very first time), so the overhead is present but perhaps not as extreme. But there's no denying that this doesn't scale well and needs to be addressed eventually.
Here are some approaches that come to mind:

It should be relatively straight-forward to not read data for exported functions if they are not used. They are already in a separate section. The only thing we have to be careful about is to export types that are referred to from such functions (the original exported did that - the new format didn't have to because it could "inline" new type declarations). Such an approach should readily cut down the amount of data that needs to be read in and type-checked. Inlined function bodies produce a lot of nodes. (As an aside, ideally no type-checking should be needed for imported data - it was type-checked before. But that's a longer-term issue. And there's a tension between re-type-checking and exporting addition type information.)

Creating a more general index requires a bit more thought.

Longer-term: If we move to a model where installed packages are in some form of data base, export data could be in a separate data base which is suitable for quick lookup.

I suspect 1) could be done even for 1.9 if we permit say 1 or 2 weeks extra past the official freeze for this.

Related: #15734 and #15752.

The difference here is that more inlining means more function bodies that must be imported and type-checked.

True, but the importing and type-checking is not be the whole story. Inlining also means that functions get larger, which means more nodes, more time in the backend, more progs, etc. I'd wager that those effects are also non-trivial.
I still think this is a good idea, I'm just not sure that the inlining data is clear evidence for it, or that it will necessarily make mid-stack inlining much cheaper, compilation-wise.

One more thing: I suspect it's a common scenario that a package A, which imports a package C, partially exports pieces of C in the export data of A. Another package B may do the same, partially exporting a part of C, maybe the same as A did, maybe a different part.
A client of A and B now needs to import multiple parts of C, possibly different ones, possibly overlapping ones. Across a sufficiently large program it could be much more efficient to import C in full exactly once.

Could this finer-grained importing be turned around to create finer-grained dependences?  I worked on a Brand X The-Java-Programming-Language™ bytecode-to-native static compiler that did this down to the field-offset level, and it was nice.  Obviously we can end up spending more time on dependence paperwork than redoing the work, but perhaps there's a sweet spot at a finger granularity than current.

@dr2chase There's a separate issue for that #15752.

Ongoing work. For 1.11.

Yeah, I think this is substantially addressed by ca2f85fd3f32b2a4c863a2de602876bd31e9d956.
There's still more work that can be done to make type loading lazy. I think we can track that in #25056.


Hard Hat Icon

The helmet of a construction worker

Closing as duplicate of #191, but we're getting to it!


More tools for fast object investigation with TimeSeries

Few thoughts of what we could add, feel free to ad more or create sub-issues from these:

[ ] TimeSeries.plot()
matplotlib could be an optional dependency. Challenges: mags vs fluxes (mags should be inverted on plot); default colors for passbands
[ ] TimeSeries.fold(period)
Period-fold light curve. Challenge: what is the output? TimeSeries with .time to be a phase = time % period / time)?
[ ] TimeSeries.filter_band()
Convert to a single passband timeseries. Should it return a TimeSeries' sub-class / super-class instance?
[ ] TimeSeries.dropna()
Remove invalid data points, which could be interpreted wider than just inf/nan values, for example non-positive flux errors are invalid
[ ] TimeSeries.__repr__()/__str__()

Would be interested to talk about this more as well. We actually have an open ticket (#130) to deprecate TimeSeries in favor of serving users a minimal data structure that hands them light curves as arrays that they can work with as they see fit. The idea being that at the time we didn't think we really had much to offer users on the single-lightcurve visualization/processing front that they wouldn't already have their own tools (or just native pandas in the TimeSeries case) for. That being said, if you think the above would help in your workflows, we can definitely think about how much emphasis to put on the single lightcurve side!

From my point of view TAPE is missing tools for data exploration - this issue is basically about it, not about the particular interface. I agree that we cannot and shouldn't offer too much, but I believe it would be nice to offer some basic functionality.

Yeah, I agree for sure that we want to make sure users have tools for data exploration. I can definitely see opportunity on the plotting side to at least have a short-hand plotter function for users, and obviously they can just use matplotlib directly for any specific needs. Things like filter_band are more questionable to me, is something like TimeSeries.filter_band("g") much more convenient than the Pandas interface currently available (TimeSeries.data.loc["g"])?

TimeSeries.data.loc["g"] is cool, we should add it to docs / tutorials!

The need for a basic .plot function here blocks #404


Enable snappy by default

PR Description
Enable snappy by default to reduce confusion.
Fixed Issue(s)
fixes #2443
Documentation

[x] I thought about documentation and added the documentation label to this PR if updates are required.

Documentation-wise most users shouldn't notice this change - it only affects if you're using a custom network, not one of the builtin ones.


Segment and word durations too long

Hello there,
I am currently using the repo to modify a Whisper model and create word-level timestamps, which has yielded mostly great results.
The main challenge I am currently encountering arises when an audio has a lot of noise and subtle talking in the background. In such cases, the word-level segmentations sometimes "catch" the background noise as part of the word, resulting in excessively long segments. For instance:
[Segment(start=1.74, end=16.14, text=' This is not comment', [...], words=[WordTiming(word=' This', start=1.74, end=11.1, [...])]]
Is there an "in-house" method to address this issue without manually trying to shorten segments? I have also tried denoising the audio using the "noisereduce" package but it didn't work.

You can try to use demucs=True and/or vad=True.
There's also clamp_max() which was added in the recent commit.

Thank you so much for the quick response! Unfortunately, neither did demucs=True and vad=True help (separate and together), but will try using clamp_max().

Want to drop a line to say that I had the same problem with segment durations being too long and fitting too much words in.
clamp_max() worked like a charm!
Thank you for the lib! The update_seg_with_words that's auto ran on WhisperResult init took care of issues like showing segments early too. Clear and readable code. Thanks

Want to drop a line to say that I had the same problem with segment durations being too long and fitting too much words in.
clamp_max() worked like a charm!
Thank you for the lib! The update_seg_with_words that's auto ran on WhisperResult init took care of issues like showing segments early too. Clear and readable code. Thanks

Hi @hhursev can you explain more detailed how you get a solution to the problem using champ_max ?


Incorrect JAR version in Dockerfile

I tried to upgrade to using Release v2.1.0 and found the follow error:
clickhouse.local :) select * from jdbc('', 'show datasources')

SELECT *
FROM jdbc('', 'show datasources')

Query id: 996cec06-6230-48de-bf88-4aeabd683923

0 rows in set. Elapsed: 0.125 sec.

Received exception from server (version 22.6.1):
Code: 86. DB::Exception: Received from localhost:9000. DB::Exception: Received error from remote server /columns_info?connection_string=&table=show%20datasources&external_table_functions_use_nulls=true. HTTP status code: 500 Internal Server Error, body: No repository available for ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource. (RECEIVED_ERROR_FROM_REMOTE_IO_SERVER)

Upon investigation it appears the wrong version of the JAR file is being inserted into the Docker image.
± [qa] src/clickhouse-jdbc-bridge master ✓ $ docker run -it clickhouse/clickhouse-jdbc-bridge:2.1.0 bash
root@71c41750a53d:/app# ls -al *.jar
-rw-r--r-- 1 root root 10737611 Dec 16  2021 clickhouse-jdbc-bridge-2.0.7-shaded.jar
lrwxrwxrwx 1 root root       44 Jul 27 01:07 clickhouse-jdbc-bridge-shaded.jar -> /app/clickhouse-jdbc-bridge-2.0.7-shaded.jar

This seems to be due to the wrong download URL for JAR, its still referring to the ru/yandex repository:
https://github.com/ClickHouse/clickhouse-jdbc-bridge/blob/7faf5868040238251bd02f54eec4a8a50c0731fb/Dockerfile#L40

Looks like a bug but I cannot reproduce.
docker run --rm -it clickhouse/jdbc-bridge:2.1.0 bash
root@c6848870dc1b:/app# ls -alF
total 10528
drwxr-xr-x. 1 root root       24 Jul 27 02:11 ./
drwxr-xr-x. 1 root root       17 Jul 27 02:11 ../
-rw-r--r--. 1 root root 10740290 Apr 18 23:08 clickhouse-jdbc-bridge-2.1.0-shaded.jar
lrwxrwxrwx. 1 root root       44 Apr 18 23:09 clickhouse-jdbc-bridge-shaded.jar -> /app/clickhouse-jdbc-bridge-2.1.0-shaded.jar

Try building the Dockerfile from the current git head first docker build -t clickhouse/jdbc-bridge:2.1.0 .
This URL https://repo1.maven.org/maven2/ru/yandex/clickhouse/clickhouse-jdbc-bridge/maven-metadata.xml is incorrect on https://github.com/ClickHouse/clickhouse-jdbc-bridge/blob/master/Dockerfile#L40
It should be https://repo1.maven.org/maven2/com/yandex/clickhouse/clickhouse-jdbc-bridge/maven-metadata.xml
$ curl https://repo1.maven.org/maven2/ru/yandex/clickhouse/clickhouse-jdbc-bridge/maven-metadata.xml | grep latest
    <latest>2.0.7</latest>

$ curl https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc-bridge/maven-metadata.xml | grep latest
    <latest>2.1.0</latest>

Looks like it was due to this commit https://github.com/ClickHouse/clickhouse-jdbc-bridge/commit/f37446a4003c179d51b17e9d47cb0450b1ab603f

@zhicwu this is still an issue, please consider merging https://github.com/ClickHouse/clickhouse-jdbc-bridge/pull/156

@zhicwu this is still an issue, please consider merging #156

Sorry for the late response and thank you for the contribution, merged!
On a side note, it's no longer makes sense to continue development of JDBC bridge, because it's not actively maintained on server side. I'll added a new module in clickhouse-jdbc in the future to replace this one.


Download fails because of DBus.Error.NoReply

OS/Platform
Debian, Ubuntu, and derivatives
Installed
Compiled from source
Version
Version 103.0.5060.134 (Official Build, ungoogled-chromium) Debian GNU/Linux (64-bit)
Have you tested that this is not an upstream issue or an issue with your configuration?

[ ] I have tried reproducing this issue in Chrome and it could not be reproduced there
[ ] I have tried reproducing this issue in vanilla Chromium and it could not be reproduced there
[ ] I have tried reproducing this issue in ungoogled-chromium with a new and empty profile and it could not be reproduced there

Description
File download fails because of putative DBus NoReply error
How to Reproduce?
Download a file, choose location for it.
Actual behaviour
This is a brand new, upgraded install of Debian 11 with KDE Plasma. Nothing exciting going on. When I download a file, the normal system file dialog appears but when it is submitted nothing happens and the console prints the following:

[1608:1626:0104/143022.653046:ERROR:select_file_dialog_linux_portal.cc(719)] Portal returned error: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Expected behaviour
Naturally the file should be downloaded to the user's chosen destination.
Relevant log output
[1608:1626:0104/143022.653046:ERROR:select_file_dialog_linux_portal.cc(719)] Portal returned error: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Additional context
Workaround is to uncheck the "Ask where to save each file before downloading" option in chrome://settings/?search=downlo

IIRC there was a bug in Chromium with download manager, but it should be fixed with recent versions. Please try building any more recent version and check again.


Missing Files When Updating Sonarr

Errors using the script when updating Sonarr @coder8338

@coder8338 Anybody home?

Please try Ubuntu 20.04