digikam: fix media playback by adding ffmpeg to buildInputs

Motivation for this change
Since Digikam 6.0.0, media playback requires ffmpeg as a direct dependency.
This shouldn't really change the closure size, as another of Digikam's dependencies, libqtav, already depends on ffmpeg.
The difference is evident during the cmakeConfigurePhase:
--- before.txt	2019-02-21 10:10:24.312731211 +0000
+++ after.txt	2019-02-21 10:01:43.642602245 +0000
@@ -1,13 +1,13 @@
--- Could NOT find FFmpeg (missing: FFMPEG_LIBRARIES FFMPEG_INCLUDE_DIRS AVCODEC_LIBRARIES AVCODEC_INCLUDE_DIRS AVFILTER_LIBRARIES AVFILTER_INCLUDE_DIRS AVFORMAT_LIBRARIES AVFORMAT_INCLUDE_DIRS AVUTIL_LIBRARIES AVUTIL_INCLUDE_DIRS SWSCALE_LIBRARIES SWSCALE_INCLUDE_DIRS)
--- FFMPEG_FOUND        = FALSE
--- FFMPEG_INCLUDE_DIRS =
--- FFMPEG_LIBRARIES    =
--- FFMPEG_DEFINITIONS  =
+-- Found FFmpeg: /nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavcodec.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavfilter.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavformat.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavutil.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libswscale.so
+-- FFMPEG_FOUND        = TRUE
+-- FFMPEG_INCLUDE_DIRS = /nix/store/bsgkvbqniycnf94xv8dmlszazqrlas7n-ffmpeg-3.4.5-dev/include
+-- FFMPEG_LIBRARIES    = /nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavcodec.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavfilter.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavformat.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libavutil.so;/nix/store/naf33s4w5kpxv060vmjv6cz5jvcqnxrb-ffmpeg-3.4.5/lib/libswscale.so
+-- FFMPEG_DEFINITIONS  =
 -- QtAV search path: /nix/store/yjj7f7lypknwzbxvkvjkf8aa0av6ss0d-qtbase-5.12.0-dev/lib/cmake
 -- Found QtAV: /nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/lib/libQtAV.so;/nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/lib/libQtAVWidgets.so
 -- Found QtAV version 1.12.0
 -- QtAV_FOUND       = TRUE
 -- QtAV_INCLUDE_DIR = /nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/include/QtAV /nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/include/QtAVWidgets
 -- QtAV_LIBRARIES   = /nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/lib/libQtAV.so;/nix/store/ncxf8s0jlqmh1cvlp9day3byh45mxqb3-libqtav-1.12.0/lib/libQtAVWidgets.so
 -- QtAV_VERSION     = 1.12.0
--- ENABLE_MEDIAPLAYER option is enabled but FFMpeg cannot be found. Media player support is disabled.
+-- Media player support is enabled.

...and during program execution, in the Help → Components Information menu:

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build digikam
@callahad next time no need for an issue if you already have a PR.

Gotcha. For next time, what's the best way to get a pull request reviewed after it's been sitting for a few days?

Timed out... I'll try this again then:
@GrahamcOfBorg build digikam
@callahad to get your PRs merged you want to make them more appealing to people with commit access. You should look for people who might not have commit access, but have been doing a fair bit of work and review on nixpkgs. If you get people like this to review and approve your PR then people with commit access will be more like to accept your PR as the PR has already been reviewed and the amount of work required has been lowered.
https://github.com/NixOS/nixpkgs/issues/50105#issuecomment-452435126
Another effective way I've found has simply been to bug people on IRC or discourse until they merge your PRs :smiley:

@GrahamcOfBorg build enblend-enfuse vigra python27Packages.numpy openblas

Builds on my system. The aarch64-linux issue seems to be with dependencies and unrelated to this PR.

@markuskowa seeing as you're aware of the openblas issue on aarch64 issue, is there any chance you'd be willing to merge?

@markuskowa Thanks! :tada:

Thanks. LGTM. Let hydra do the build. This seems to be too big for OfBorg.


Fix #76 - improve avibase error handler, tests++

Fix #76 - improve avibase error handler, tests++

Coverage after merging 524 into main will be77.59%
Coverage Report for Changed FilesFileStmtsBranchesFuncsLinesUncovered Linessrc/servicesExternal   AviBaseService.js90.91%86.67%100%91.43%47–49, 59–61, 65–66

Coverage after merging 524 into main will be77.59%
Coverage Report for Changed FilesFileStmtsBranchesFuncsLinesUncovered Linessrc/servicesExternal   AviBaseService.js90.91%86.67%100%91.43%47–49, 59–61, 65–66


Add the second post

This commit should add the second post to the Blog web page.

Everything is OK.


Replace deprecated uses of set-output in Actions

Tin. Deadline is May 31.
Announcement: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Seems like PHPUnit needs some TLC, too. Version 10 appears to break YOURLS' tests, and doesn't work at all on PHP 8.0.

Indeed, php8.0 is deprecated, I won't be surprised PHPUnit 10 drops support for it.


Tab inside markdown preview moves focus around markdown preview element

markdown preview
try to tab out of the file, the focus is moved throught the element in the file
not sure if this is intended behavior because the file can be arbiterarly large and then it is hard fo focus out of the editor

As designed, otherwise how would you navigate to links in the markdown? You can use keys to focus the explorer to jump out.


[1.9.0-rc1] docker push performance regression

Pushing to an S3-backed private registry takes much longer on 1.9.0-rc1 than on 1.8.2, even though pushing to a local registry is substantially faster. When attached to a TTY, most of the time is spent after the "Pushing" progress bar hits completion, but before printing "Pushed". This time is negligible when pushing to a registry on the same machine backed by the local filesystem.
In 1.8.2, we spent way more time creating the tarball to send ("Buffering to Disk"), but there was no wait between the upload and "Image successfully pushed".
Some logs
In the comparisons below:

Same server is used, before and after the upgrade from 1.8.2 to 1.9.0-rc1
The top layers, 7191 and 8469 are both almost exactly 400MB
registry.shopify.io is an S3-backed private registry, hosted in EC2 along with the machine we're pushing from here.
This is actually indicative of a trend on sample sizes > 1 :)

Docker 1.8.2
Oct 14 18:56:19.277:	The push refers to a repository [registry.shopify.io/shopify-ci-branches] (len: 1)
Oct 14 18:56:19.294:	7191157cb30d: Buffering to Disk
Oct 14 18:56:32.647:	7191157cb30d: Image successfully pushed
Oct 14 18:56:32.928:	fa1c81cb6b28: Image already exists
Oct 14 18:56:33.267:	03c5add71c1d: Image already exists
Oct 14 18:56:33.686:	b25524db876d: Image already exists
Oct 14 18:56:34.024:	c290db7a5070: Image already exists
Oct 14 18:56:34.283:	454c2d886fcc: Image already exists
Oct 14 18:56:35.774:	ba5e0000ba5c: Image already exists
Oct 14 18:56:35.777:	ac192e88bf8434b72d6b237634ebf3bac5ab9ace: digest: sha256:30b3430df4b7d4bbd9f56b678b5a1c35b3632c7fd088eeca86ff93d8070065fe size: 12862

Docker 1.9.0-rc1
Oct 14 19:00:09.679:	The push refers to a repository [registry.shopify.io/shopify-ci-branches] (len: 1)
Oct 14 19:00:09.712:<PHONE_NUMBER>f7: Preparing
Oct 14 19:00:09.807:<PHONE_NUMBER>f7: Pushing
Oct 14 19:01:36.139:<PHONE_NUMBER>f7: Pushed
Oct 14 19:01:36.388:	a5323a34b08b: Image already exists
Oct 14 19:01:36.749:	03c5add71c1d: Image already exists
Oct 14 19:01:37.170:	b25524db876d: Image already exists
Oct 14 19:01:38.101:	c290db7a5070: Image already exists
Oct 14 19:01:38.369:	454c2d886fcc: Image already exists
Oct 14 19:01:38.684:	ba5e0000ba5c: Image already exists
Oct 14 19:01:38.687:	23168f9ff831f4097bed8cb6aeab38e3db9ea3c7: digest: sha256:17092b886b66afb5ec9f66c2df760533dc43c4fd13fd99a1238f51a88f307caf size: 11567

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

burke@locutus4.ec2:~ $ docker version
Client:
 Version:      1.9.0-rc1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9291a0e
 Built:        Wed Oct 14 03:04:22 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-rc1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9291a0e
 Built:        Wed Oct 14 03:04:22 UTC 2015
 OS/Arch:      linux/amd64
burke@locutus4.ec2:~ $ docker info
Containers: 1
Images: 134
Engine Version: 1.9.0-rc1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 16
Total Memory: 29.44 GiB
Name: locutus4.ec2.shopify.com
ID: 7R4J:UVLA:UOUE:OZNW:QAZK:QMED:MRF7:LI2L:UWFS:J4H2:WE6V:HSMU
WARNING: No swap limit support
burke@locutus4.ec2:~ $ uname -a
Linux locutus4.ec2.shopify.com 3.19.0-28-generic #30~14.04.1-Ubuntu SMP Tue Sep 1 09:32:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
burke@locutus4.ec2:~ $

I'm kind of poking at this myself, will see how far I get this afternoon. Looks at first glance like it's time spent in https://github.com/docker/docker/blob/master/graph/push_v2.go#L263-L266

Nope, that's not true, here's what I've got:
[t=0] about to io.Copy (push_v2.go#L257-L261)
[t=41] done io.Copy. about to digest (push_v2.go#L263-L266)
[t=41] enter *httpBlobUpload Commit
[t=82] enter *blobStatter.Stat
[t=84] exit *blobStatter.Stat
[t=84] exit *httpBlobUpload Commit
[t=84] done digest

Previously the whole process took 11 seconds.

Ping @aaronlehmann!

I think this is related to https://github.com/docker/docker/pull/15493
One thing I notice looking at the code is that 1.8.x used httpBlobUpload.ReadFrom, which would construct a PATCH request that read from a reader. But the code in 1.9 uses io.Copy, which invokes httpBlobUpload.Write repeatedly instead. Since Copy uses a 32kb buffer internally, this can result in many PATCH requests. I can see why that would be very slow with a S3 backend.
I will work on a fix for this. @burke, if you have time, could you please try reverting 349aef06d51a9425f70a877585fe45283366c2c4 and see if that makes a difference? It reverts cleanly except for a conflict in an integration test. You can just remove that integration test.
cc @stevvooe

I missed how io.Copy will use a ReadFrom method if available - so ignore the comment above. I'm continuing to investigate.

@burke: I think I understand what's going on here.
The upload to the registry is a two-step process. First the actual data is uploaded; then the engine asks the registry to commit that data to the blob store. The commit phase is where you see the pause.
When the registry uses S3 as the backend, committing to the blob store involves copying the data. This takes a significant amount of time. However, if the blob already exists on the registry, the copy is skipped.
I think the pushes you are doing with 1.8.2 upload layers that already exist, so they can skip the copy. But 1.9.0 is generating blobs with different hashes from those that existed previously, so they need to be copied to the right place.
I tested with 1.9.0 and 1.8.2, and with both versions I see pauses for some time at the end of the "pushing" step. I believe what you're seeing is not a bug, but an artifact of the way you're testing, where 1.8.2 happens to be uploading layers that already exist, and 1.9.0 is uploading layers which involve more work for the registry.

I don't think that's it. The image below shows push times by host. The last two were upgraded to 1.9.0 yesterday, and it would be exceedingly rare for an image in our hierachies to reference an existing upper layer that hadn't been referenced in the past 5-10 builds (i.e. any effect from having to write out new caches is long-since gone in this data):

@aaronlehmann I haven't had a chance to look at this in detail but you are probably on to something with ReadFrom getting hidden. io.Copy will use it, but if one wraps the reader and doesn't have a ReadFrom fallthrough, then it won't be used. We've had similar regressions in the registry that were hit when a Reader didn't have a ReadFrom implementation.

@burke: Thanks for the quick response.
Could you please elaborate a bit on what the workflow looks like? I'm having some trouble imagining why this situation would occur.
For each layer, the engine keeps track of the latest digest it has seen when pushing or pulling it. Before pushing a layer, it checks if the registry has a layer with that digest, and should avoid pushing it at all if the registry already has it.
So unless something is wrong there, the engine should only be pushing newly-built layers. And assuming these new layers are unique, those pushes should involve the overhead of the registry copying them to the final location in the blob store.

@stevvooe: I did some debugging and confirmed that ReadFrom was being used. There is only one PATCH request for each blob. Too bad, because that explanation would have made a lot of sense.

The logs do seem to indicate that both 1.8.2 and 1.9.0 are only pushing the new top layer of our images:
A recent 1.8.2 log:
Oct 15 18:21:45.996:	The push refers to a repository [registry.shopify.io/shopify-ci-branches] (len: 1)
Oct 15 18:21:46.015:	c96f70e4e306: Buffering to Disk
Oct 15 18:21:58.008:	c96f70e4e306: Image successfully pushed
Oct 15 18:21:58.478:	21615e90a830: Image already exists
Oct 15 18:21:58.771:	13fe0d421dcd: Image already exists
Oct 15 18:21:59.185:	b25524db876d: Image already exists
Oct 15 18:21:59.485:	c290db7a5070: Image already exists
Oct 15 18:22:00.021:	454c2d886fcc: Image already exists
Oct 15 18:22:00.427:	ba5e0000ba5c: Image already exists
Oct 15 18:22:00.430:	5f2a2bc0578ae071185203e39a47e1bd1fb39fe2: digest: sha256:e627abba81e9c9f577b1081ce9c44cd5a5e01836d3e33bd2704b5ab706167f14 size: 12862

A recent 1.9.0 log:
Oct 15 18:31:05.553:	The push refers to a repository [registry.shopify.io/shopify-ci-branches] (len: 1)
Oct 15 18:31:05.568:	9998d416aa37: Preparing
Oct 15 18:31:05.655:	9998d416aa37: Pushing
Oct 15 18:32:19.932:	9998d416aa37: Pushed
Oct 15 18:32:20.445:	50c6998536b6: Image already exists
Oct 15 18:32:20.802:	8db39f7a1862: Image already exists
Oct 15 18:32:21.047:	b25524db876d: Image already exists
Oct 15 18:32:21.456:	c290db7a5070: Image already exists
Oct 15 18:32:21.832:	454c2d886fcc: Image already exists
Oct 15 18:32:22.117:	ba5e0000ba5c: Image already exists
Oct 15 18:32:22.139:	06afc66ec7e9182d90aaeb45da93f45a5a4d8cb7: digest: sha256:07a8798a8491088dabe5063853a001b43f2f8c4fa5700a7bb7224406dd33a141 size: 11596

Our workflow is essentially:

The bottom four layers (ba5e0000 .. b255) constitute a baseimage that changes pretty rarely (weeks, sometimes months) and is always shared between build nodes.
the 8db3 and 13fe layers are the result of a fixed setup script in the source repo that changes very infrequently. It's usually the baseimage changing that causes this to rebuild. This layer is currently built differently on each build node, hence the different IDs.
the 50c6 and 2161 layers are the result of checking out some fixed point in the recent past (typically somewhere in the middle of the previous night) and running some things like bundle install in that context. This layer is generally shared by all images built later that day.
The top layer, which is the only one being pushed here in both versions (as expected) is the result of copying in the current source and finalizing compilation.

TL;DR: the bottom 6 layers only change once a day. It looks to me like docker is correctly identifying that only the top layer needs to be pushed, but for some reason it takes much longer on 1.9.0. this layer is 400MB: it includes a /var/lib/mysql :disappointed:

Is it possible that 1.8.2 would just shoot the blob up to the registry and return, but 1.9.0 is waiting for the data to become available via S3 or something like that?

@burke: Thanks for the info, this is very helpful.
Both 1.9.0 and 1.8.2 call the Commit method, which is where you are seeing lots of time spent per https://github.com/docker/docker/issues/17038#issuecomment-148176373. This method sends a very small PUT request to the registry, and waits for it to complete. The time is spent waiting for the registry, not doing anything inside the engine. The only explanation that makes sense for major differences in the time spent waiting here is the registry already having the blob in its blob store in one case but not the other. However, I'm at a loss as to why the Docker engine version would make a difference.
If you have access to the registry logs, we can dive into this from the registry's perspective. For each pushed layer that the registry didn't already have, you should see a line like this:
time="2015-10-15T17:52:07.731240035Z" level=debug msg="s3.Move(\"/docker/registry/v2/repositories/ubuntu/_uploads/db9528b7-bbcf-4bfb-a3b3-04ac0000a7aa/data\", \"/docker/registry/v2/blobs/sha256/7c/7ccf5baddc671df300711753fa329a2b6feb190ea523b019c634920ad66927cc/data\"" environment=development http.request.host="registry:5000" http.request.id=135ab698-9edd-4254-9a60-a6cbde3dc4ab http.request.method=PUT http.request.remoteaddr="<IP_ADDRESS>:27790" http.request.uri="/v2/ubuntu/blobs/uploads/db9528b7-bbcf-4bfb-a3b3-04ac0000a7aa?_state=24p2zYtFZBpOe6350AwOlaIdB2XgGwQZqorUniclJDt7Ik5hbWUiOiJ1YnVudHUiLCJVVUlEIjoiZGI5NTI4YjctYmJjZi00YmZiLWEzYjMtMDRhYzAwMDBhN2FhIiwiT2Zmc2V0IjoxOTcyNDkwMjQsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMTVUMTc6NTE6MjNaIn0%3D&digest=sha256%3A7ccf5baddc671df300711753fa329a2b6feb190ea523b019c634920ad66927cc" http.request.useragent="docker/1.9.0-dev go/go1.4.3 git-commit/9216b08-dirty kernel/4.0.7-boot2docker os/linux arch/amd64" instance.id=108f00d9-ef8f-4abf-b4eb-bc6ca9f75d22 service=registry trace.duration=11.251662362s trace.file="/home/ubuntu/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Move" trace.id=85165849-d44a-4830-a147-55fc0e180246 trace.line=135 vars.name=ubuntu vars.uuid=db9528b7-bbcf-4bfb-a3b3-04ac0000a7aa version=v2.1.1-54-g3fa64a9.m

Note the "trace.duration" key (11 seconds in this case).
My hypothesis was that on pushes from 1.8.2 (which apparently don't pause between the upload and "Image successfully pushed"), you won't see such a line in the logs. But if it does indeed appear, it would be interesting to know if it shows a consistently shorter trace.duration than an equivalent push with 1.9.0.

ping @burke are you able to provide these logs?

Yes, I'll get those tomorrow morning. Sorry, I got derailed toward the end of last week because we had Hack Days.

@burke no problem, I was just browsing through open issues <3

It turns out that there was a regression which means the layers are not getting compressed at all. That is probably why you're seeing slower pushes in 1.9.0-rc1. I've opened PR #17227 to fix this.

Thanks @aaronlehmann!

1.8.2: 16s
1.9.0-rc1: 89s
1.9.0-rc1 + 4dce280d: 47s

@burke is there a ready-to-use script you could provide us to save some time?

I'll try to put something together tomorrow.

@aaronlehmann @tonistiigi @LK4D4 if you guys could help with profiling this that'd be dope :)

@tiborvass Isn't this a dupe of #17227

@tonistiigi #17227 is supposed to fix this. It did help a lot according to https://github.com/docker/docker/issues/17038#issuecomment-150331384 (almost 50% faster) but nowhere near 1.8 according to burke.

@aaronlehmann @tonistiigi Probably you know better. in 1.8.2 99% of push is compression(probably on disk), however in 1.9 55% takes net/http/internal.(*chunkedWriter).Write function.

I haven't had a chance to look yet, but that sounds consistent with smaller chunks of data getting returned from the reader that's passed to httpBlobWriter's HTTP request. Each one would cause a separate HTTP-level chunk, which adds overhead. It makes sense that the gzip compressor would write data to its output in smaller increments than what we would get reading directly from a file.
I'll try buffering the compressor output and seeing if that helps.

@LK4D4 When I looked at the issues with 1.8.2, most of that "compression" was spent buffering to disk.

The HTTP chunk sizes were indeed tiny. PR #17291 should fix it.

Confirmed #17291 fixes the problem entirely :sparkles:.

Thanks @burke, and thanks for fixing @aaronlehmann, great job!


Unhandled panicking in create_reg_interval function in ezk_sip_ua

Hi,
I came across a bug in your ezk_sip_ua crate. It is an unhandled panicking in the create_reg_interval function.
If the period duration is smaller 10 seconds the program will panic at this line: https://github.com/kbalt/ezk/blob/main/crates/sip-ua/src/register/mod.rs#L104C1-L104C51
Change your Register example code like this to reproduce the bug:
...
    let mut registration = Registration::new(
        NameAddr::uri(id),
        NameAddr::uri(contact),
        registrar.into(),
        Duration::from_secs(5),  // <-- duration is smaller than 10 secs
    );
...

I haven't tested it yet, but this error can possibly occur if the SIP response contains an Expire header with less than 10 seconds and someone calls the receive_success_response function on the response because the faulty function is also called there.
A solution could be the checked_sub function, which can be used to check if the resulting duration of a subtraction would be negative/overflow and handle the problem by somehow. Or use the saturating_sub function.

Hi, thanks for finding!
I added a minimum value for period to work around this issue in https://github.com/kbalt/ezk/commit/6bc07a154e0b54f0a9d482f6b8fa526cbd97e956

Just out of curiosity, do you have a scenario where an interval as short as 5 seconds is required? Usually the expiry value with common registrars is at least 5mins.

I just stumbled across it when I was writing tests for an app that uses your crate.


new Response(readableStream).arrayBuffer() stopped working for streams longer than ~247 bytes at ~2023-08-03 20:44:17 +0000 UTC

From https://discord.com/channels/595317990191398933/1012440514135593083/1136901294196535387

(
This turned out to be a regression in workerd which will affect non-WfP workers too!
See this comment for the root cause and our workaround: https://github.com/cloudflare/workerd/issues/970#issuecomment-1665373143
)
We see a complete failure of certain routes that use WfP without any code changes on our end.
First health check failure: 2023-08-03 20:44:17 +0000 UTC
Last health check failure: 2023-08-04 05:59:04 +0000 UTC (Status 500)
The logs show:
  "exceptions": [
    {
      "name": "Error",
      "message": "The script will never generate a response.",
      "timestamp":<PHONE_NUMBER>644
    }
  ]

some failing routes:
https://rtedge.net/
https://elefunc.com/
(These routes now work after we applied the workaround in https://github.com/cloudflare/workerd/issues/970#issuecomment-1665373143)

const st = { // comp.rt.ht

  fo: ['gzip','deflate'][0],

  id: x => x,

  co: (s, f = st.fo) => { if (s === null || s === undefined) return s;
    const c = new CompressionStream(f);
    const w = c.writable.getWriter();
    /**/  w.write(new TextEncoder().encode(s));
    /**/  w.close();
    return new Response(c.readable).arrayBuffer();
  },

  de: (B, f = st.fo) => { if (B === null || B === undefined) return B;
    const d = new DecompressionStream(f);
    const w = d.writable.getWriter();
    /**/  w.write(B);
    /**/  w.close();
    return new Response(d.readable).arrayBuffer().then(B => new TextDecoder().decode(B)).catch(_ => new TextDecoder().decode(B));
  }

};

we have found the call triggering the regression after your update, it is in our exported fetch when we call:
await st.de(v) where v is an ArrayBuffer instance!
This used to work before 2023-08-03 20:44:17 +0000 UTC and now it does not.

If v.byteLength is 243 or less, decompression succeeds.
If v.byteLength is 247 or more, we now get The script will never generate a response..
Decompression never took more than 50ms in any of our deployed routes. I believe whatever causes the error triggers falsely now.

Here is what regressed!
This used to always work in st.de:
new Response(d.readable).arrayBuffer()

and has stopped working at 2023-08-03 20:44:17 +0000 UTC
with The script will never generate a response. errors,
for ReadableStreams longer than ~247 bytes.
Having GPT4 switch our code to use manual concatenation resolved the issue:
ReadableStream.prototype.arrayBuffer = async function() { return st.ab(this); };

const st = { // comp.rt.ht

  fo: ['gzip','deflate'][0],

  id: x => x,

  ab: async r => {
    const  R  = r.getReader();
    const  C  = []; while (true) { const { done, value } = await R.read(); if (done) break; C.push(value); }
    const  AB = new Uint8Array(C.reduce((a, b) => a + b.length, 0)); let p = 0; for (let c of C) { AB.set(c, p); p += c.length; }
    return AB;
  },

  co: (s, f = st.fo) => { if (s === null || s === undefined) return s;
    const c = new CompressionStream(f);
    const w = c.writable.getWriter();
    /**/  w.write(new TextEncoder().encode(s));
    /**/  w.close();
    return c.readable.arrayBuffer();
  },

  de: (B, f = st.fo) => { if (B === null || B === undefined) return B;
    const d = new DecompressionStream(f);
    const w = d.writable.getWriter();
    w.write(B);
    w.close();
    return d.readable.arrayBuffer().then(B => new TextDecoder().decode(B)).catch(_ => new TextDecoder().decode(B));
  }

};

@CetinSert here's a simple test using response of readable stream:
export const arrayBufferOfReadable = {
  async test() {
    const rs = new ReadableStream({
      type: 'bytes',
      start(controller) {
        const enc = new TextEncoder();
        controller.enqueue(enc.encode('0123456789'.repeat(1000)));
        controller.close();
      }
    });

    const read = await new Response(rs).arrayBuffer();
    assert.equal(10_000, read.byteLength);
  }
}

seems to pass if you add it to streams-test.js. Is there a critical step missing here?

I think I have a reproduce in #974 and revert is up in #973

@mikea – has this been reverted at all workerd deployments globally?

@CetinSert — we are in the process of reverting this change in production and will post here once that is complete.

@CetinSert the revert should be complete by now

@mikea – thank you for the prompt response!
Confirming with global Health Checks over the next 5 hours.
I will update and close the issue here by morning (New York).

Sorry for the summer-slow response!
This is just working flawlessly again.


fix G2 element subgroup membership check

Hi, I fix the bug mentioned by BitVM /issues/109, support G2 element subgroup membership check and the script len is about 990983 bytes.
Thanks for the backend of Bitlayer Labs.

What Bitlayer Backend?

What Bitlayer Backend?
I am an contributor from bitlayer lab.

Ah, that's what it meant. Thank you and the Bitlayer team!

Hi -- the implementation here seems to be for Point-On-Curve check and not subgroup membership check. From what I understand, both of these checks are essential and the issue raised was for the latter check.
https://github.com/Consensys/gnark-crypto/blob/11d2ed41a3d3470f9cf21a8e1cd4ca5d60a3983f/ecc/bn254/g2.go#L491
@dylanCai9, @wz14 -- please correct me if I'm wrong.
Thanks :)

Hi -- the implementation here seems to be for Point-On-Curve check and not subgroup membership check. From what I understand, both of these checks are essential and the issue raised was for the latter check. https://github.com/Consensys/gnark-crypto/blob/11d2ed41a3d3470f9cf21a8e1cd4ca5d60a3983f/ecc/bn254/g2.go#L491
@dylanCai9, @wz14 -- please correct me if I'm wrong. Thanks :)

Oh, I missed the additional check for the G2 subgroup, mistakenly assuming that verifying it the same way as for the G1 subgroup in BN254 would be sufficient. Thank you for pointing out this issue!

no worries, very happy to be of any help :)


Error handling is mostly redundant as it is only executed on success status codes.

As I understand and use Cypress in my work for 3 years the error handling is redundant.
https://docs.cypress.io/api/commands/request
The request has a build in error handler which is set to true. If the server answers with a non success status code like 4xx or 5xx the test failes automatically with a proper log.
The following code is only executed when the request answers with a success status code.
		.then((response) => {
				if (response.status !== 200) {
					throw new Error(`Failed to get messages. Status: ${response.status}, Body: ${response.body}`);
				}
				return response.body as MessagesSummary;
			});

property | default value| description
failOnStatusCode | true | Whether to fail on response codes other than 2xx and 3xx
To make the error handling work correctly set the property to false. Otherwise it can be removed from all commands.

Is it possible to contribute in this project? It will be my first one 🙈

Yes @TobiasHH. It will be great.

Can't pust a new feat branch. Need some help.
remote: Permission to pushpak1300/cypress-mailpit.git denied to TobiasHH. fatal: unable to access 'https://github.com/pushpak1300/cypress-mailpit.git/': The requested URL returned error: 403

@TobiasHH, please start by forking the repository to your account. After forking, make your changes in the forked repository, and then push those changes.
You can follow this guide for more detailed instructions: GitHub Forking Guide.

@TobiasHH Are you able to solve the issue or you need more help here ?

I just had time today to fork the project. I will try to implement the stuff the next days.

I have another question. The custom commands seem to lack a custom logging. Is it OK if I add it in my implemention?

Actual using the mailpit commands will display request in the command log. You can add custom names. So if you then use a mailpit command it will display the command name.
I will implement it in my branch and then you can have a look. If it is not good I can remove it.


CAMEL-20725 - Camel-Pinecone: Add more producer operations - Update o…

…peration
Description

Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[x] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test pinecone


issue-48 Add web fonts and font styles

Completes issue #48 .
Changes in this pull request:

Added Google fonts linking
Edited styles to call correct fonts
Replaced font-weight values with correct values

OK, this is updated


Publish YARD docs to vercel

The SDK is undergoing a rewrite/refresh and this issue applied to a previous version and is no longer applicable


Added the cooler aggregate cli command

I changed the former cooler/contribute/recursive_aggregate_onefile.py script to a cli option in the spirit of cooler cload and cooler balance.

merged in 9cb6e91


Create the prediction function

Your master branch is not in a mergeable state. I'm creating a branch from a reconstruction of your master relative from the merge-base to the current master.


provide special-case handling of Estimated{Row,Column}SizeHistogram

Borrows code from Cassandra to parse the array. Generates metric names that
conform to the same convention as the other histograms.
Disclaimer: This is Pretty Bad, both the copy-paste of code from Cassandra, and the special-casing of these two metrics.  Consider it a stop-gap until more time can be devoted to this (or another alternative is chosen).
Bug: T97024

/cc @filippog

cool @eevans ! I guess this would also fix some of the invalid values we've been seeing? https://phabricator.wikimedia.org/T97024

@filippog yeah, that's the idea.  It takes the metrics that were showing up like:
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.EstimatedRowSizeHistogram.value [J@3b173bdf<PHONE_NUMBER>

...and turns them into...
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.50percentile 215.0<PHONE_NUMBER>
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.75percentile 215.0<PHONE_NUMBER>
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.95percentile 215.0<PHONE_NUMBER>
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.98percentile 215.0<PHONE_NUMBER>
cassandra.restbase1004.org.apache.cassandra.metrics.ColumnFamily.local_group_wikipedia_T_parsoid_html.meta.99percentile 215.0<PHONE_NUMBER>
...

:+1: LGTM


Support optional compdef.In embed field for component constructors

What does this PR do?
Allow Provides structs that use the optional compdef.In embed field. Although this field is non-functional, it makes it a bit easier to convert from existing fx-aware components.
Motivation
Though the field compdef.In was added in a previous PR, the helper function ProvideComponentConstructor did not properly handle it when converting to and from fx types.
Additional Notes
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Functionality covered by unit tests

/merge

/merge


REPL code drop 1 bug fixes.

2 bug fixes:

Bug fix around inactive employee logic where we were never setting an employee to inactive
Change the way we query the open shift table to retrieve by org job path instead of scheduling group - this allows multiple org job paths to share the same scheduling group and sync correctly

Refactored OpenShiftController/ProcessOpenShiftsAsync


Update Content-Security-Policy response header

Set to resolve OWASP error detection

:tada: This PR is included in version 2.0.0-next.25 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Started Service Background Scanning

Currently my started service and run and discover device when the application is "foreground". Which means it's not minimized. The output is fine
[0:] BLE Bluetooth is On
[0:] BLE IS SCANNING
06-30 15:38:24.396 D/Mono    (27664): Assembly Ref addref Plugin.BLE[0x7f9a6ab500] -> System.Core[0x7f7aae1180]: 4
Adapter >= 21: Starting a scan for devices.
06-30 15:38:24.404 I/mono-stdout(27664): Adapter >= 21: Starting a scan for devices.
06-30 15:38:24.417 D/BluetoothAdapter(27664): STATE_ON
06-30 15:38:24.429 D/BluetoothLeScanner(27664): onClientRegistered() - status=0 clientIf=6
06-30 15:38:24.454 D/Mono    (27664): [0x7f758a0440] hill climbing, change max number of threads 6
06-30 15:38:24.729 D/Mono    (27664): DllImport searching in: '__Internal' ('(null)').
06-30 15:38:24.730 D/Mono    (27664): Searching for 'java_interop_jnienv_is_assignable_from'.
06-30 15:38:24.730 D/Mono    (27664): Probing 'java_interop_jnienv_is_assignable_from'.
06-30 15:38:24.730 D/Mono    (27664): Found as 'java_interop_jnienv_is_assignable_from'.
06-30 15:38:24.735 D/Mono    (27664): DllImport searching in: '__Internal' ('(null)').
06-30 15:38:24.735 D/Mono    (27664): Searching for 'java_interop_jnienv_get_byte_array_region'.
06-30 15:38:24.735 D/Mono    (27664): Probing 'java_interop_jnienv_get_byte_array_region'.
06-30 15:38:24.735 D/Mono    (27664): Found as 'java_interop_jnienv_get_byte_array_region'.
06-30 15:38:24.766 D/Mono    (27664): Assembly Ref addref Plugin.BLE.Abstractions[0x7f9a6ab400] -> System.Runtime.Extensions[0x7f7b642980]: 3
Adv rec [Type CompleteLocalName; Data 49-44-31-30-37-20-48-52]
Adv rec [Type Appearance; Data 41-03]
Adv rec [Type Flags; Data 06]
Adv rec [Type UuidsComplete16Bit; Data 0A-F0]
Adv rec [Type ManufacturerSpecificData; Data 11-00-02-00-EA-7A-9B-5B-50-4E-0B]
[0:] Trying to reconnect
[0:] Guid Found
[0:] Scanning
Adapter: Stopping the scan for devices.
Adapter >= 21: Stopping the scan for devices.

But once I minimize the application. It doesn't "receive" or discover anything.
[0:] BLE Bluetooth is On
Thread started: <Thread Pool> #8
[0:] BLE IS SCANNING
Adapter >= 21: Starting a scan for devices.
06-30 15:38:49.147 D/Mono    (27664): [0x7f7863f440] worker starting
06-30 15:38:49.150 I/mono-stdout(27664): Adapter >= 21: Starting a scan for devices.
06-30 15:38:49.154 D/BluetoothAdapter(27664): STATE_ON
06-30 15:38:49.158 D/BluetoothLeScanner(27664): onClientRegistered() - status=0 clientIf=7
Adapter: Scan timeout has elapsed.
06-30 15:39:09.205 I/mono-stdout(27664): Adapter: Scan timeout has elapsed.
Adapter: Stopping the scan for devices.
Adapter >= 21: Stopping the scan for devices.
06-30 15:39:09.209 I/mono-stdout(27664): Adapter: Stopping the scan for devices.
06-30 15:39:09.210 I/mono-stdout(27664): Adapter >= 21: Stopping the scan for devices.
06-30 15:39:09.219 D/BluetoothAdapter(27664): STATE_ON
[0:] BLE Bluetooth is On

So my concern is, how to get the scanning to receive the device in the background(as in minimized/when the application is not in focus).

There is an issue template ;) Please use it next time instead of deleting it. Please provide information about platform, plugin version, e.t.c.

Right. Sorry.
Visual Studio 2015
Xamarin.Android
Plugin.BLE version beta4
Compiled using latest Platform (6.0 marshmallow)

This is the sample of the Service demo we've tried. In focus/foreground of the application, it'll definitely reconnect. But not when it's minimized.
public class SimpleService : Service
    {
        private string _deviceID;
        private Plugin.BLE.Abstractions.Contracts.IAdapter _adapter;
        private IDevice currDevice;
        private IBluetoothLE _bleDevice;
        private System.Timers.Timer _timer;

        public override void OnCreate()
        {
            base.OnCreate();
            Console.WriteLine("OnCreate started");
            _bleDevice = CrossBluetoothLE.Current;
            _adapter = CrossBluetoothLE.Current.Adapter;
            _adapter.ScanTimeout = 20000;
	    _deviceID = "d8de624e-140f-4a22-8594-e2216b84a5f2"; //This is just a dummy GUID
	    _bleDevice.Adapter.DeviceDiscovered += reconnect;
            _bleDevice.Adapter.DeviceConnected += Adapter_DeviceConnected;
        }

        public override StartCommandResult OnStartCommand(Intent intent, StartCommandFlags flags, int startId)
        {
            Scanning(this, null);

            return StartCommandResult.NotSticky;
        }

        private async void Adapter_DeviceConnected(object sender, DeviceEventArgs e)
        {
            if (_adapter.IsScanning)
            {
                await _adapter.StopScanningForDevicesAsync();
            }

            currDevice = e.Device;
            System.Diagnostics.Debug.WriteLine("Connected");
        }

        public async void Scanning(object sender, EventArgs e)
        {
            await Task.Delay(5000); //We need the delay to have enough time to bring the app to background (out of focus/minimized)
            if (_bleDevice.IsAvailable)
            {
                if (_bleDevice.IsOn)
                {
                    System.Diagnostics.Debug.WriteLine("BLE Bluetooth is On");
                    if (!_bleDevice.Adapter.IsScanning)
                    {
                        System.Diagnostics.Debug.WriteLine("BLE IS SCANNING");
                        await _adapter.StartScanningForDevicesAsync();
                    }
                }
            }
        }

        public void reconnect(object sender, DeviceEventArgs e)
        {
            System.Diagnostics.Debug.WriteLine("Trying to reconnect");
            Guid id = new Guid();

            if (!Guid.TryParse(_deviceID, out id))
            {
                System.Diagnostics.Debug.WriteLine("Cannot find GUID");
                id = new Guid();
            }

            if (e.Device.Id == id)
            {
                System.Diagnostics.Debug.WriteLine("Guid Found");
                if (_adapter.IsScanning)
                {
                    System.Diagnostics.Debug.WriteLine("Scanning");
                    _adapter.StopScanningForDevicesAsync();
                }

                _adapter.ConnectToDeviceAync(e.Device);
            }

        }

        public override IBinder OnBind(Intent intent)
        {
            // This example isn't of a bound service, so we just return NULL.
            return null;
        }

        public override void OnDestroy()
        {
            base.OnDestroy();
            Log.Debug(TAG, "SimpleService destroyed at {0}.", DateTime.UtcNow);
        }
    }
}

Any updates?
So far I've tried the 1.1 version for the "ConnectToPreviousKnownDevice" in order for it to work since the background scanning is not receiving.

Not yet. We are focusing on 1.0. Do you have a minimal sample project on github? It would accelerate the process, when somebody is starting to investigate this issue.

Has there beeen a change in this issue ?

Solved it. I used the service object reference previously initalised from within a user thread.
I moved this initialization to function OnStartCommand() (within the service class), after which it worked.
Foreground/background - same performance.
My implementation uses the raw Android native API.

^ Hi do you have some sample code for this


Issue with passing password

We tried to script the export process for a DB and had issues with authentication as the password contained an exclamation mark. Trying to use the PowerShell methods of escaping the character failed. Any idea how to do it?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c1bdd674-b454-e863-c943-9c979b466a92
Version Independent ID: b15341f9-3f1b-00e0-a1c2-970f25a6ab03
Content: az sql db
Content Source: src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/_help.py
Service: sql-database
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

Do you have a solution for this ?
My password starts with a single quote (') and contains exclamation mark.
I tried to surround with double quotes ("), to escape with \ but no success.
E.g. : az login -u<EMAIL_ADDRESS>-p 'sd(!MH;du4

https://www.gnu.org/software/bash/manual/html_node/Double-Quotes.html
So, for you password: "'sd(!MH;du4"


Center the Shared Shelf button in the space below the image caption

...and let the caption run full width.

I was only able to test this on my iPhone7, but I still get the caption and button side-by-side in portrait mode.

Maybe I have the portrait breakpoint set too low. Do you know the pixel dimensions of your iphone screen?

I believe it's 1334x750px

I think the high dpi screens should be scaling to display correctly. Is your phone a 7+ or just 7?

I made the portrait breakpoint bigger. Let me know if this fixes it.


better output after formula-instances create

formula-instances create returns a 200 as the default successful output.
Something more user-friendly should be output.

Closed by #36


IPv4/single: Avoid a hanging reuse socket

Description
In a post on the FreeRTOS forum, Hubert Sack reported that a problem occurs when the SYN/ACK negotiation never finishes.
The orphaned socket will be marked as closed and deleted.
In case of a reuse socket, it should not be deleted because the application uses it as a server socket.
Solution: when a timeout is reached, the socket shall be put into eTCP_LISTEN mode again, so it can receive a new connection.
PS. A reuse socket is a TCP socket that turns itself into a client socket as soon as a connection is established. This method has two advantages: it uses less RAM, and it allows for a single connection only.
Test Steps
Have a client which only sends a SYN packet and then stop. After a timeout, the socket should be able to receive a new connection.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Finally I have setup a situation that simulates the problem, and I could see that the server socket recovered well.
Note that this only applies to sockets that have the FREERTOS_SO_REUSE_LISTEN_SOCKET socket option.
A telnet client sent a SYN, it was answered by a SYN/ACK, and from then, all incoming packets were dropped.
When the ipconfigTCP_HANG_PROTECTION noticed that the socket was inactive for 30 seconds, it was put in the eTCP_LISTEN state again.
A next connection would succeed normally again.

As I am currently also working on another issue, I think it is better to close this PR for now, and integrate the it with the other upcoming PR.
A summary: the normal path of a TCP connection is either:
"eTCP_LISTEN, eSYN_FIRST, eSYN_RECEIVED, eESTABLISHED, eCLOSE_WAIT"
or
"eCONNECT_SYN, eESTABLISHED, eCLOSE_WAIT"
This and the other PR handle the situation in which the status goes directly from SYN to CLOSED.
To be continued. Hein

Now closing, to be continued in another PR.

In PR #559 I addressed the above issue.


JSON Sort -> Sort by key

Is your feature request related to a problem? Please describe.
I have to sort big JSON-files with arrays. The array-entries should be sorted by a specific key
Describe the solution you'd like
When I hit JSON sort, it should ask me for keys to sort with, e.g. "date, last_name, ..."
Describe alternatives you've considered
I use VisualStudioCode which has support for this. But I'd rather sort directly in PHPStorm :)
Additional context
none

@simube, can you provide at least 1 data sample (pre and post sorting)?

OK, I'll try to give an example:
unsorted:
{
  "persons": [
    {
      "name": "Michael",
      "age": 30
    },
    {
      "name": "Andrew",
      "age": 42
    },
    {
      "name": "Simon",
      "age": 18
    },
    {
      "name": "Julia",
      "age": 33
    }
  ]
}

sorted with JSON sort:
{
  "persons": [
    {
      "age": 30,
      "name": "Michael"
    },
    {
      "age": 42,
      "name": "Andrew"
    },
    {
      "age": 18,
      "name": "Simon"
    },
    {
      "age": 33,
      "name": "Julia"
    }
  ]
}

(this just puts "age" before "name")
What I'd like to do:

Hit "JSON sort by key"
enter "name" (or multiple keys, e.g. first by "name", then by "age", or if there are multiple arrays to sort inside the file)
Then the file gets sorted

sorted by key "name" (Andrew first, Simon last):
{
  "persons": [
    {
      "age": 42,
      "name": "Andrew"
    },
    {
      "age": 33,
      "name": "Julia"
    },
    {
      "age": 30,
      "name": "Michael"
    },
    {
      "age": 18,
      "name": "Simon"
    }
  ]
}

There is this plugin for VS Code which does the task: https://github.com/fvclaus/vsc-sort-json-array
So it would be nice to have this functionality in PHPStorm too :)


[GEP-33] Machine Image Capabilities

How to categorize this PR?

/area documentation
/kind discussion
What this PR does / why we need it:
Introduces GEP-33 to add machine type and image capabilities. It aims to improve the operating system image selection and maintenance operations of worker groups.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
The missing GEP numbers were "reserved" during the hackathon and skipped to avoid conflicts. Corresponding PRs will be opened by other contributors.

/assign


Missing page/broken link for ColourSchemer

I can't find ColourSchemer in the list. Closing this issue.


Wrong import in run_swag script for new pytorch_transformers setup

https://github.com/huggingface/pytorch-transformers/blob/f2a3eb987e1fc2c85320fc3849c67811f5736b50/examples/single_model_scripts/run_swag.py#L35
It appears that WEIGHTS_NAME and CONFIG_NAME have moved out of {pytorch_transformers/pytorch_pretrained_bert}.file_utils (and instead can be imported directly from pytorch_transformers), as shown below :
>>> import pytorch_transformers as p
>>> p.__version__
'1.0.0'
>>> from pytorch_transformers.file_utils import WEIGHTS_NAME, CONFIG_NAME
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'WEIGHTS_NAME'
>>> from pytorch_transformers.file_utils import CONFIG_NAME
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'CONFIG_NAME'
>>> from pytorch_transformers import WEIGHTS_NAME
>>>

This seems incorrect in the new run_swag script (as shown above) (if true,  are there any other important imports that were overlooked here?)

I found a few other issues: 1) the script uses the old BertAdam (instead of AdamW)
https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L431

the train and test loops still use the old version of forward, i.e.,

https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L450
https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/examples/single_model_scripts/run_swag.py#L525
In the former case, I added the following:
                #loss = model(input_ids, segment_ids, input_mask, label_ids) # line 450 in train
                outputs = model(input_ids, segment_ids, input_mask, label_ids)
                loss = outputs[0]

And the latter:
                ## tmp_eval_loss = model(input_ids, segment_ids, input_mask, label_ids) ## line 525
                output = model(input_ids, segment_ids, input_mask, label_ids)
                tmp_eval_loss,logits = output[:2]

I'd be curious if the last two versions are correct.

Yes the run_swag script hasn't been updated to the new API yet.
Do you want to give it a look and submit a PR? I don't have plan to work on it in the short-term.


Release 0.9.20

Sekoilin huolella vscode mergetoolin kanssa, katotaan tätä kautta kuinka paljon sain rikki.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (main@b3e5000). Click here to learn what that means.

:exclamation: Current head 4cf067b differs from pull request most recent head 3af7591. Consider uploading reports for the commit 3af7591 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #20   +/-   ##
=======================================
  Coverage        ?   17.36%
  Complexity      ?       86
=======================================
  Files           ?        6
  Lines           ?      357
  Branches        ?        0
=======================================
  Hits            ?       62
  Misses          ?      295
  Partials        ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Dicom loading example broken

the example at https://rawgit.com/chafey/cornerstoneWADOImageLoader/master/examples/dicomfile/index.html seems to be broken
When loading an image I get cornerstoneWADOImageLoader is not defined
thx!

Whoops, this was broken by a recent update to the build process towards ES6 support. This has been fixed now by https://github.com/chafey/cornerstoneWADOImageLoader/pull/82.
Thanks for the report!


prep LunrCoreLmdb for NuGet

This is a PR for https://github.com/bleroy/lunr-core/issues/28.
It prepares LunrCoreLmdb to be a NuGet package.
If you can push it out the same way you're pushing out LunrCore, that would be great.
For now I can use the offline package for testing & development.

I will make these changes but I'm going to hold off until my mutable index feature is completed and tested, which should be done this week.

@danielcrenna Any update on the mutable index feature please?

I'll merge that for now, for the sake of having a package on NuGet, which is long overdue. Let's take the mutable index changes when they're done, separately.


GSoC 2022 Interfaces for Consistent API Design: Library-wide Discussion

For GSoC 2022, I'm working on designing more consistent interfaces to PySAL's exploratory and inferential statistics classes. The end goal is to design a sleek new API for PySAL's statistical classes that is highly self-consistent.
To couple with related issues in spreg, mgwr, esda, spopt, and spglm, I'm creating an issue here for discussion on library-wide concerns related to this project. From discussion in the other issues, it seems like a PySAL-specific set of base classes would be most preferable to developers. While scikit-learn and multiple dispatch might not be great fits for the library, it still needs consistent standards (whether or not they conform to external rules). Creating a PySAL-specific framework would have several key advantages, like in-house maintainability and customizability.
Importantly, I'm starting this API redesign in the modeling classes (spreg first, then likely moving to mgwr, spint, and beyond). Developing common constructs for these packages is a good subproblem before moving out of the model module. To this end, I'm interested in getting your feedback on the following questions:

What design elements would you like to see implemented in a new interface for PySAL's modeling classes?
What design elements does PySAL's current design get right, and what design elements could be deprecated?
Are there library/package/code structures elsewhere that might be beneficial to use or mimic in PySAL? What benefits would PySAL gain from drawing on these structures?

My drafts for this redesign can be found in pysal-base and in my copy of spreg. These are quite rough at present as development is ongoing. Descriptions of my development can be found on my blog.
Excited to hear your input!

Closing due to inactivity.


Server and CLI don' t exclude the same directories

I' m using ruff server in vscode.
Ruff version: 0.4.5
VSCode version : 1.89.1 (Universal)
My ruff.toml contains the following exclusion:
exclude = ["*/tests/*"]
I run ruff in a terminal at the same time as the ruff server in vscode.
The intersting part of the output of ruff -w is:
[23:26:28 PM] Starting linter in watch mode...

warning: Unexpected `# ruff: noqa` directive at conftest.py:10. File-level suppression comments must appear on their own line. For line-level suppression, omit the `ruff:` prefix.
warning: Unexpected `# ruff: noqa` directive at bikeanalysis/activities/models/activities.py:228. File-level suppression comments must appear on their own line. For line-level suppression, omit the `ruff:` prefix.
[23:26:28 PM] Found 19 errors. Watching for file changes.

Only 19 errors found
In vscode, the same errors are flagged, as expected, but I have also all the warnings for all my test files, that are in one of the tests/ directories.
For instance, this one:

def test_missing_all_streams():
    streams = []
    assert len(get_missing_streams(streams=streams)) == 11

triggers D103.
The ruff that runs in my terminal does not show any errors for files in my tests/ directory as expected from the configuration.
Am I missing something when configuring ruff server in VSCode?

Might be a real bug in the new server. I'll try to repro tonight.

exclude = ["*/tests/*"] doesn't actually work for me, even on the command-line. exclude = ["**/tests/*"] does work though.

I can reproduce with exclude = ["**/tests/*"] though.

I think ruff server just doesn't respect exclusions yet if you open an excluded file directly.

#11590 breaks my format workflow, I'd expect the vscode format document and format cell commands to format the python/notebook file even if it's excluded in configuration file, because I'm explicitly asking vscode to format it.
BTW, ruff format <path to file> works regardless of file exclusions config, so IMO manually call vscode format document should have the same behavior.

https://github.com/astral-sh/ruff/pull/11590 breaks my format workflow, I'd expect the vscode format document and format cell commands to format the python/notebook file even if it's excluded in configuration file, because I'm explicitly asking vscode to format it.

Hmm, I do understand the desire that explicitly issuing format document or format cells should format the code even if it is excluded. The challenge we face is that the LSP request, as far as I know, contains no information whether the request was triggered implicitly (by enabling format on save) or explicitly (by issuing the command). Implicit request should respect the ignores, I'm open to change that explicit requests don't.

An new option would be nice for people don't use format on save.

The challenge we face is that the LSP request, as far as I know, contains no information whether the request was triggered implicitly (by enabling format on save) or explicitly (by issuing the command)

Yeah, this is correct. The server has no way of knowing whether the request was triggered automatically or manually.
It seems reasonable to add a new option and is related to https://github.com/astral-sh/ruff/issues/12176. @messense Is it that the extension should format / lint every document that's opened in the editor, thus not considering the includes / excludes ?

Is this really about files that are opened explicitly? I'm finding that VScode flags warnings from files that are not open but are in my workspace. In my case these files:

Despite:
[tool.ruff]
exclude = [
  # Machine-generated files
  "**/.bootstrap/*",
  "**/.tox/*",
  "**/.vscode/*",
  "**/src/numbers_parser/generated/*",
]

Whether they're included by the server is a bit random. Is there a VScode log that would be helpful?

Is this really about files that are opened explicitly? I'm finding that VScode flags warnings from files that are not open but are in my workspace.

Can you say more about this? Is it that the diagnostics window includes violations from files that aren't opened in the editor? Are those diagnostics coming from Ruff because we don't support workspace wide diagnostics? Do you have python.analysis.diagnosticMode set to workspace? That setting is used by the Python extension to include workspace wide diagnostics.


BUG - User has the ability to spam someone with slack messages

BUG: When the confirmation button appears on the slack handle page, the user can spam the button to send multiple slack messages to the selected user.
DESCRIPTION
On the slack handle page (/slackname), when the user selects a user they must click the confirmation button to send the slack message. This doesn't prevent the user from clicking the button multiple times, meaning the user can span the selected user.
Steps to reproduce:

Navigate through either the snack chat or default path to the /slackname page
Select a slack account
Click the 'next' button multiple times
BUG - Multiple messages are sent to the selected user

Expected behaviour:

The user should only be able to send one single slack message. This could be by implementing a restriction on client side (UI restriction), server side (cloud function api restriction) or both.

Observed behaviour:

The user can send multiple slack messages to the selected slack account

For a more concrete UX, it would be good if we could lock the user out from doing anything on the page. At the moment, you can click 'Next' and then while it's in 'sending' mode, click on a different slack name.

Left comment in #337

Have added a change for #337 to this branch (slack-sending)
Safe to close this issue ?


Language translator

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Designed a language translator page which can translate any paragraph or word to any foreign languages according to the users

Hey @Swastik200, it seems like this is a duplicate issue for #51. Please provide better information for this. Make proper use of the template present for an issue to avoid unnecessary confusion.

Closing this issue due to inactivity.


Plan management panel with tag editing

Adds a new plan management panel for viewing plan metadata and editing plan tags. Thoughts on the name "Plan Management"?
Partially addresses #605

Think I've fixed the strict issues for all the relevant files in this PR


click child only creates append

click on the td of the second level does not obey the collapsed and expanded (red circle), in the second level, clicking on td only to create append of the children.

Resolvido colocando span, no evento click.
td.treegrid-control span
and
td.treegrid-control-open span


MSID: 95197 Version: 2 DOI: 10.1101/2<IP_ADDRESS>6898

MSID: 95197
Version: 2
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>6898
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=95197
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>6898

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/95197v2
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>6898
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/95197/v2/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!


"Preferences Sync" shows badge on the gear icon

VSCode Version:
Version: 1.45.0-insider (user setup)
Commit: 44c5185373d5f209cd6c0b5d29a216fb801da34e
Date: 2020-04-10T18:12:33.780Z
Electron: 7.2.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.18363

OS Version:

Steps to Reproduce:

Start VSCode insider version specified above
The gear icon always shows a blue badge because of "Preferences sync: Sign in to sync (1)".

Either do not show the badge at all (the best) or clear it as soon as someone clicks on "Preferences sync" menu.

Does this issue occur when all extensions are disabled?: Yes

Sorry, closing since duplicate of https://github.com/microsoft/vscode/issues/94822


dyld crash at launch

A colleague ran into a situation where the app would crash on launch. Important to note is that she was also seeing it with other apps, such as LinkedIn.app.
A restart of her device ‘fixed’ the issue.
These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.
Extracting the crash reports directly from the device revealed a dyld error.
Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.

TL;DR
This appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably this neat trick does so as well.

Info about the errors
Dyld Error Message:
Dyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack
  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy
  Reason: no suitable image found.  Did find:
    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C
  Dyld Version: 370.1

The error message and codes are always the same.
The error is not specific to a framework, it happens with various frameworks.
It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).

Known affected versions
These are the various versions for which we have crash reports.
iOS versions
~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq
OS Version:          iOS 9.1 (13B143)
OS Version:          iOS 9.2 (13C75)
OS Version:          iOS 9.2.1 (13D15)
OS Version:          iOS 9.2.1 (13D20)
OS Version:          iOS 9.3 (13E5214d)
OS Version:          iOS 9.3 (13E5225a)

dyld versions
~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq
  Dyld Version: 370.1
  Dyld Version: 370.6
  Dyld Version: 390.7

Device versions
~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq
Hardware Model:      iPad3,4
Hardware Model:      iPad4,4
Hardware Model:      iPad4,7
Hardware Model:      iPad5,3
Hardware Model:      iPad5,4
Hardware Model:      iPad6,7
Hardware Model:      iPad6,8
Hardware Model:      iPhone6,1
Hardware Model:      iPhone7,1
Hardware Model:      iPhone7,2
Hardware Model:      iPhone8,1
Hardware Model:      iPhone8,2

Retracing

The error is raised here in dyld.
Only place where mremap_encrypted could return -1 is here.
vn_getpath calls build_path, which in turn has a few possible locations where it could be returning -1:

https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L404
https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L446
https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L509

But trying to hunt that further down would take a lot more time, suffice to say that:

if mremap_encrypted returns -1, it most likely corresponds to ERESTART: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L270
if errno is 12, it most likely corresponds to ENOMEM: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L102

Caveats
Alas, the latest dyld sources to be released (at the time of writing) appear to be for version 360.18.
Taking a cursory look at the decompiled code for ImageLoaderMachOCompressed::registerEncryption from a firmware image for iOS 9.2 for iPad3,4, does not appear to have any real changes.
~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw
Archive:  iPad3,4_9.2_13C75_Restore.ipsw
  inflating: 058-25903-078.dmg
  inflating: 058-26242-078.dmg
  inflating: 058-26268-078.dmg
[…]

~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3
Writing out data..
[…]

~/t/ios-9.2-iPad3,4 » open extracted.dmg
~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld

int ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {
    r0 = arg0;
    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;
    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;
    r6 = arg1;
    r8 = r2;
    r4 = r0;
    if (r6 != 0x0) {
            r5 = 0x0;
            do {
                    r2 = *(*r4 + 0xe8);
                    if ((r2)(r4, r5, r2) != 0x0) goto (null);
                    r5 = r5 + 0x1;
            } while (true);
            r1 = r5;
            r2 = *(*r4 + 0x104);
            r0 = (r2)(r4, r1, r2);
            r10 = r6 + 0x8;
            asm{ ldm.w      sl, {r1, r5, sl} };
            r2 = *(r8 + 0xb2);
            r8 = *(r0 + 0x4);
            r6 = r0 + r1;
            r11 = *(r0 + 0x8);
            if (r2 != 0x0) {
                    dyld::log("                      0x%08lX->0x%08lX configured for FairPlay decryption\n");
            }
            r0 = _mremap_encrypted();
            r5 = r0;
            if (r5 != 0x0) {
                    r0 = ___error();
                    r3 = *(r4 + 0x4);
                    r2 = *r0;
                    r1 = r5;
                    r0 = dyld::throwf("mremap_encrypted() => %d, errno=%d for %s\n");
                    // …
            }
    }
    return r0;
}

Conclusion
The crash at launch definitely feels like a very bad experience and I feel like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.
I’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.
It would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.
Status

A couple of radar tickets have already been filed:

http://www.openradar.me/radar?id=4952667885404160 (mine)
http://www.openradar.me/radar?id=5054516256833536
http://www.openradar.me/radar?id=4992459683659776

A DTS ticket has been filed http://twitter.com/senior/status/707996353964728321
A guess has been made as to the issue being a jetsam change https://twitter.com/Catfish_Man/status/707955795493687296 (jetsam is the process in which the kernel basically kills apps to free up memory)

Framework binary sizes (output edited for readability):
$ find DerivedData/Artsy/Build/Products/Debug-iphoneos/Artsy.app/Frameworks -perm +u=x -type f -exec ls -lh {} \;
363K Adjust.framework/Adjust
104K Aerodramus.framework/Aerodramus
 80K AFNetworkActivityLogger.framework/AFNetworkActivityLogger
656K AFNetworking.framework/AFNetworking
151K AFOAuth1Client.framework/AFOAuth1Client
145K ALPValidator.framework/ALPValidator
294K Analytics.framework/Analytics
210K ARAnalytics.framework/ARAnalytics
 81K ARASCIISwizzle.framework/ARASCIISwizzle
131K ARCollectionViewMasonryLayout.framework/ARCollectionViewMasonryLayout
135K ARGenericTableViewController.framework/ARGenericTableViewController
138K ARTiledImageView.framework/ARTiledImageView
106K Artsy_UIButtons.framework/Artsy_UIButtons
 76K Artsy_UIColors.framework/Artsy_UIColors
 73K Artsy_UIFonts.framework/AGaramondPro-Bold.otf
101K Artsy_UIFonts.framework/AGaramondPro-Semibold.otf
 79K Artsy_UIFonts.framework/Artsy_UIFonts
 36K Artsy_UIFonts.framework/ITC_Avant_Garde_Gothic__Demi.ttf
106K Artsy_UILabels.framework/Artsy_UILabels
273K Bolts.framework/Bolts
368K CocoaLumberjack.framework/CocoaLumberjack
 74K DHCShakeNotifier.framework/DHCShakeNotifier
 80K DRKonamiCode.framework/DRKonamiCode
118K EDColor.framework/EDColor
956K FBSDKCoreKit.framework/FBSDKCoreKit
288K FBSDKLoginKit.framework/FBSDKLoginKit
128K FLKAutoLayout.framework/FLKAutoLayout
108K FXBlurView.framework/FXBlurView
1.4M HockeySDK_Source.framework/HockeySDK_Source
135K Interstellar.framework/Interstellar
146K iRate.framework/iRate
664B iRate.framework/iRate.bundle/ko.lproj/Localizable.strings
856B iRate.framework/iRate.bundle/no.lproj/Localizable.strings
888B iRate.framework/iRate.bundle/sl.lproj/Localizable.strings
100K ISO8601DateFormatter.framework/ISO8601DateFormatter
103K JLRoutes.framework/JLRoutes
101K JSBadgeView.framework/JSBadgeView
111K JSDecoupledAppDelegate.framework/JSDecoupledAppDelegate
109K Keys.framework/Keys
105K KSDeferred.framework/KSDeferred
240K libswiftAVFoundation.dylib
208K libswiftContacts.dylib
 13M libswiftCore.dylib
294K libswiftCoreAudio.dylib
208K libswiftCoreData.dylib
395K libswiftCoreGraphics.dylib
202K libswiftCoreImage.dylib
208K libswiftCoreLocation.dylib
248K libswiftCoreMedia.dylib
337K libswiftDarwin.dylib
240K libswiftDispatch.dylib
1.5M libswiftFoundation.dylib
255K libswiftObjectiveC.dylib
248K libswiftUIKit.dylib
208K libswiftWebKit.dylib
224K Mantle.framework/Mantle
101K MARKRangeSlider.framework/MARKRangeSlider
241K MMMarkdown.framework/MMMarkdown
 76K MultiDelegate.framework/MultiDelegate
163K NAMapKit.framework/NAMapKit
 78K NPKeyboardLayoutGuide.framework/NPKeyboardLayoutGuide
109K ObjectiveSugar.framework/ObjectiveSugar
 81K ORKeyboardReactingApplication.framework/ORKeyboardReactingApplication
104K ORStackView.framework/ORStackView
103K PBDCarouselCollectionViewLayout.framework/PBDCarouselCollectionViewLayout
1.0M ReactiveCocoa.framework/ReactiveCocoa
 96K RSSwizzle.framework/RSSwizzle
392K SDWebImage.framework/SDWebImage
 75K Then.framework/Then
 79K UIAlertView_Blocks.framework/UIAlertView_Blocks
 99K UICKeyChainStore.framework/UICKeyChainStore
 80K UIView_BooleanAnimations.framework/UIView_BooleanAnimations
136K VCRURLConnection.framework/VCRURLConnection

There is very few mentions about this crash that I could find, maybe because people don’t realise it is happening?

http://stackoverflow.com/questions/33718131/ios-crash-library-not-loaded-mremap-encrypted
http://stackoverflow.com/questions/34828744/swift-binary-rejected-and-i-cannot-reproduce-the-crash
https://forums.developer.apple.com/message/121962#121962

Sadly enough, two of those cases appear to get their app rejected, 1 followed up to say they recompiled and resubmitted and it was accepted. I bet that these were just cases of the reviewer’s device running out of memory and that the second case also got accepted after resubmitting.3

@JaviSoto had a discussion on twitter last week about the same issue but it looks like no solution surfaced. 🙁

@0xced Aye, he has let me know, it also sparked others chiming in. I have added a ‘Status’ section to the bottom of the OP.

The jetsam/memory pressure process is described here for reference: http://newosxbook.com/articles/MemoryPressure.html
If the issue can be reliably reproduced it may be worth testing explicit weak linking of some frameworks as described in SDK based development

My radar has been closed as a duplicate of 24278648.

@robinsenior Did you get any updates on your DTS?

They initially told me it was fixed in 9.3, but an Apple dev on Twitter told me otherwise. I see crash logs for it in beta 7 but not the full release (AFAIK, I haven't checked in a few days).

@robinsenior I have crash reports for 9.3 builds 13E233 and 13E234, so definitely not fixed.
If you don’t have any atm and want to follow-up with DTS I can send you some.

@alloy thanks; I just checked and we have some with those build numbers so I'll email them to DTS.
As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

Ace :+1:

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

That’s great news! Might update my personal device sooner after all :wink:

Apparently there have been some changes to dyld in 9.3 which resulted in significant performance improvements: https://github.com/stepanhruda/dyld-image-loading-performance
Didn't fix the crash from this issue though 😕

Yeah, that’s about #586.

Just got this update from DTS:

We’ve made some progress on this one. It seems to be related to the number of embedded frameworks an app has. I noticed that your app has 40 frameworks (12 of which are the Swift dylibs). You could reduce the chance of your app hitting this bug by combining some of those frameworks. Even cutting the number of frameworks in half would make a difference.

/facepalm

Oh jeez… So their progress is basically all of what this ticket already describes ¯_(ツ)_/¯
I do get that they want to give you something that’s actionable for you right now, but it would have been nice to know a bit more what their status is on the real issue. (rdar://24278648, which is the one mine got closed in favour of.)

Hi Alloy - Have you heard anything more about iOS dynamic library loading crasher? or are you guys making any changes to combine frameworks etc?
Thanks in advance for any insight.
-chris

@ctava Nope, if/when we would hear more, or decide a different course of action, it all goes public in this ticket. I don’t think combining frameworks would help, as it would probably still run out of memory and die in the same way. The only possibility to reduce (not fully get rid of) these issues would be to get rid of ‘fat’ (number of larger frameworks), which is not something we will be doing at this time.
To summarise, imo it’s clearly a bug in iOS, and the radar that mine was closed in favour of is still open, so they haven’t ruled it as “works as intended” yet. Until that time, I don’t think the number of crashes vs the number of users warrants a lot of work on our part. YMMV.

@robinsenior Thinking about that answer a bit more, could you ask them if they do in fact mean the number of libs vs the size of those libs? If they are positive that reducing the number of libs, regardless of size, is a way to workaround the issue, then that would definitely be helpful info.

@alloy done. I'll keep you posted.

Ace, thanks 👍

For another data point: we're seeing pretty commonly in our app as well since we switched to using dynamic frameworks with CocoaPods. We have 41 frameworks averaging about 1.5MB each. It's a pretty significant issue for us, so we're going back to a static library in our next release.

@maxmeyers Are you using Swift yet?

@alloy Fortunately not in production yet.

@maxmeyers Ok cool. Just wanted to check if the Swift dylibs were not causing this issue for you, but that explains it then. Thanks for the feedback 👍

@alloy DTS says it's the number of libs, not the size.

@robinsenior Well… that’s unexpected. Are you going to spend time on reducing the number of libs?

@alloy I'd rather not; our build system is already pretty convoluted due to shared frameworks across multiple projects, primarily built using Cocoapods. Do you know of a simple approach for mashing everything together?

@robinsenior I assume you’re using frameworks because you have some libs that use Swift? Do you have many of those vs objc ones?

@alloy of our 39 frameworks:

12 are libswift
17 are external dependencies
10 are internal frameworks, 7 of which contain Swift

@alloy Do you think it would be possible to trigger this issue even after a device reboot? Had this issue come to light today in our app with a colleague "fixing it" by restarting their device but another still appearing to run into problems.

@robinsenior Something like https://github.com/johnno1962/Accelerator is a ‘simple’ approach. Alternatively you could maybe have a separate Podfile for your objc pods and statically build those, you might need to do integration into your Xcode project yourself then, though. But make sure that the libs you would be doing this with will be able to lookup their resources, if they have any (wrt to NSBundle lookup).
If/when I decided to do this for our app, I’ll post a follow up with my steps.

@DarrylBayliss Seeing as DTS is reporting that it has to do with the number of libs, it might well be. Have you retrieved the crash reports from the device after that crash to verify it’s the same?

Want to also note a useful tweet about this issue

@alloy We have less than 20 frameworks including the swift ones and we’re still seeing the issue.

@alloy Managed to grab some logs and we actually had two problems. The dyld issue mentioned here and something unrelated we didn't catch. So nothing new to add here.

Is this still a thing for 9.3.1? I don't see it listed above but I haven't seen anything to confirm its fixed in 9.3.1

@ryang1428 This happens on 9.3.1 as well.

thanks @alloy .. did this mean to be 9.3.1 in your original post? You have 9.3 listed twice I now noticed OS Version:          iOS 9.3 (13E5225a)

Nope, 13E5214d & 13E5225a are both 9.3 betas iirc

@alloy I would like to thank you so much for doing such great writeup!! And also thanks Artsy for doing open source by default. It's great that you can share all details about an investigation of a weird crash! It helps others a lot!
We have noticed same issue with our app. I wonder if anybody is going to bring this topic to the labs at WWDC. Unfortunately I didn't get a ticket.

@yas375 You’re most welcome :bow:
That’s a good question. I’m personally not going, but maybe @orta is?

Not to beat a dead horse, but I've had this crash happen to me in several apps lately (including big ones like LinkedIn). Our app is getting pounded in app store reviews because of it. How is there not a bigger stink about it?

Just want to chime in on the conversation here. I ran into this issue this morning - lyft app crashed during startup. I reinstalled the app from app store and still to no avail. And then I found some other apps (not all) are having the same issue. Restarting my phone 'fixed' it.

Having looked at our crash reports in Xcode we do have a huge number of these crashes.

@robinsenior We're also getting destroyed, hundreds of 1 star reviews. 😢  I've been raising a fuss to everyone I know who could potentially help, and to no avail. It seems like many people don't realize they have these crashes. Since it's a crash on launch, their crash reporting tool (e.g. Crashlytics/Hockey) won't pick it up, and only iTunes Connect can relay it, they're probably living in ignorant bliss.

Hey folks, talked a bit to a DTS engineer that asked me to keep an eye out for this crash in iOS 9.3.2 beta 3. Might be nothing, but might as well add information here where I can.
Our app was also rejected from app review briefly for this specific problem which involved me coordinating between DTS engineering and the App Review team. Thankfully, we're approved and ready to go now, but things are getting hairy!

@brianmichel @JaviSoto just mentioned that he’s still seeing it on 9.3.2: http://twitter.com/Javi/status/726882761106038784.
Thanks for putting it on the radar of the App Review team 👍

As of now, both @JaviSoto and I have only seen crashes up to 9.3.2 beta 1 (13F51a). This combined with @brianmichel’s DTS feedback to lookout for this issue in beta 3 seems very hopeful.

any update on this issue. for example, how hopeful is 9.3.2 beta in terms of fixing the iOS crasher?

We were just rejected for this issue this morning. I didn't even bother with the appeal process, just submitted another build. Hopefully it gets approved.
Ironically, this is our first denial because of this issue, and I had just gotten done uninstalling 4 cocoapods we didn't absolutely need anymore.
This is really starting to worry me, and i'm wondering what percentage of our users are experiencing this and simply uninstalling the app and/or leaving bad reviews. We've already received a few 1 star reviews because of this.

We're seeing a number of 1 star reviews because of this too, so much so that in the next release notes we're going to have to specifically attribute this to being Apple/iOS's issue and not ours.

Here's what happened when we switched back from frameworks to static libraries (this is crashes as reported by iTunes Connect App Analytics):

@maxmeyers Did you not require frameworks for use with Swift or did you make use of https://github.com/johnno1962/Accelerator?

We didn't have any Swift dependencies. One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies.

@maxmeyers Do you use Swift in your app? If so, are you still getting this crash report (albeit in low numbers) due to libSwiftCore.dylib etc?

No Swift at all. This crash has completely disappeared.

One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies.

Or you could use a second Podfile for your Swift dependencies and use something like Rome, if you prefer CocoaPods ;)

@maxmeyers Ah, bugger, had hoped for an extra convincing case to Apple that shows that just using Swift leads to this issue. But great for you of course!

@alloy: That wouldn't solve the issue though would it? Only reduce the likelihood of the crash because of less frameworks. Either way, unfortunately our app is 50% Swift so most of our pods are used in both 😞

@SquaredTiki Correct.

Have any of you attempted to weak link frameworks and load them after launch using the NSBundle API?

With that technique, it sounds like you're starting to move into some of the territory covered in this issue: https://github.com/artsy/eigen/issues/586

There are two, related issues at work here, both centering on the dynamic loader. Memory pressure and performance. In the crash logs that I have seen memory pressure is the overriding concern. As the loader loads strongly linked frameworks at startup it has to verify their signatures. System processes involved in that increase memory pressure when it may have already been high on the device. The loader can't get enough memory to continue loading frameworks and their dependencies, the application can't be launched or is immediately terminated.
Weak-linking the frameworks and loading after launch using the NSBundle API gives the developer more control over this process.
Monitoring the performance of signature verification may also be helpful.

Apple has told us that weak linking does not help the problem.

We're having a similar problem and just wanted to reiterate the thanks for Artsy's open source ethos. This thread is very helpful.

In order to reproduce this crash consistently, we downloaded a number of apps from the App Store on an iOS 9.3.1 device (iPhone 6+). I opened 14 apps, and when trying to open the 15th app, I experienced a crash on launch. From there, any app I tried to open for the first time would crash on launch, while previously opened apps would continue to launch successfully.
I was able to reproduce this same sequence both after a reboot, and after manually killing all open apps via the app switcher.
After upgrading the phone iOS version to 9.3.2 Beta 4, I was unable to reproduce any crash. At the maximum, I opened 50+ apps successfully. Hopefully this means the issue is fixed or at least ameliorated with iOS 9.3.2 Beta 4.

@freshnitesh That’s great to hear, tanks for the update!

@alloy Happy to help :)

@freshnitesh thank you so much for the detailed instructions on how to reproduce the crash consistently. I'm also able to confirm that iOS 9.3.2 beta 4 fixes this issue. At some point, apps take longer to launch but no crash so far, which is way better.

9.3.2 is out. So far nothing is crashing, but will keep an eye on this in the coming days/weeks. 🙏 for success.

Yes, we believe the crashes on launch with mremap_encrypted -1 errno=12 are fixed in iOS 9.3.2 (as of beta 3, I think). Please file new bug reports if you see it again.

Alright, I'm calling this 👏
👍

Awesome! :clap:

This is great! 🚀

That's an amazing thread.
Despite the fact that is fixed in 9.3.2 (and I hope iOS 10), there's still the issue we're supporting quite a lot of old version devices. What you do on those cases?

Aside from the manual work shown in:

https://github.com/stepanhruda/dyld-image-loading-performance
https://github.com/johnno1962/Accelerator

A new CocoaPods plugin has been created by @Ruenzuo since then that will automate most of that work:

https://github.com/Ruenzuo/cocoapods-amimono

Other than that, there's no real choice to to write it off. At least 9.3 came with awesome emoji, and that got most of our users to update.

I am getting this error. App is working fine on simulator, but as I run it on device it crashes .I don't know why it's happening.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:1: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:15: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:20: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:29: missing_documentation: The recipe is missing a doc_url or doc_source_url
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:29: missing_dev_url: The recipe is missing a dev_url
WARNING: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:29: missing_description: The recipe is missing a description
ERROR: /tmp/abs_33j_jyzt18/clone/recipe/meta.yaml:29: missing_license_family: The recipe is missing the about/license_family key.
Errors were found


Small fix on logging on bootstrap.py

Since >= is used above before the break, example_idx then is already the length of the example, so the logging was sligthly misleading adding one more example count

actually it's more tricky, the +1 is correct when the bootstrapping goes over all the examples, but wrong when it finishes earlier due to reaching max_bootstraps. The way to make it work for both is moving the break at the end of the loop, I'll make this change


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v4.19.223",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=982b6ba1ce626ef87e5c29f26f2401897554f235"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "982b6ba1ce626ef87e5c29f26f2401897554f235",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "phonet/pep: refuse to enable an unbound pipe\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.19.223 by commit 982b6ba1ce626ef87e5c29f26f2401897554f235. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002791


Fix cub::DeviceHistogram::MultiHistogramEven() 1-Dimensional overload, when OffsetT is not an int.

When using gcc=7.5.0 and OffsetT=int64_t, the following template errors occur:
/usr/local/cuda/include/cub/device/device_histogram.cuh(144): error: no instance of overloaded function "dgl::cub::DeviceHistogram::MultiHistogramEven" matches the argument list
            argument types are: (void *, size_t, int32_t *, unsigned long long *[1], int [1], int32_t [1], int32_t [1], int64_t, int, unsigned long, cudaStream_t, __nv_bool)
          detected during:
            instantiation of "cudaError_t dgl::cub::DeviceHistogram::HistogramEven(void *, size_t &, SampleIteratorT, CounterT *, int, LevelT, LevelT, OffsetT, cudaStream_t, __nv_bool) [with SampleIteratorT=int32_t *, CounterT=unsigned long long, LevelT=int32_t, OffsetT=int64_t]"

This resolves this, by casting the 0 to an OffsetT, so as to ensure num_rows and num_row_samples are of the same type.

Thanks for the patch! This LGTM.
Would you mind updating this PR to fix the same issue on line 153 of this file, too?
run tests

@allisonvacanti Updated line 153 as well--which is actually the one where I encountered the issue. Thanks for catching that!

That was fast -- thanks! I'll get this in for the next release.

Tests look good -- I'll merge this soon.


Pre-install hook runs on upgrade

Hi, I've got problem with my chart.
I've created new chart and include there 2 dependencies, one of it has pre-install jobs, but it starts every time I'made upgrade.
upgrade.go:142: [debug] preparing upgrade for elk
upgrade.go:150: [debug] performing update for elk
upgrade.go:322: [debug] creating upgraded release for elk
client.go:133: [debug] creating 1 resource(s)
client.go:133: [debug] creating 1 resource(s)
client.go:133: [debug] creating 1 resource(s)
client.go:133: [debug] creating 1 resource(s)
client.go:133: [debug] creating 1 resource(s)
client.go:703: [debug] Watching for changes to Job pre-install-kibana with timeout of 5m0s
client.go:731: [debug] Add/Modify event for pre-install-kibana: ADDED
client.go:770: [debug] pre-install-kibana: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:731: [debug] Add/Modify event for pre-install-kibana: MODIFIED
client.go:770: [debug] pre-install-kibana: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:731: [debug] Add/Modify event for pre-install-kibana: MODIFIED
client.go:770: [debug] pre-install-kibana: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:731: [debug] Add/Modify event for pre-install-kibana: MODIFIED
client.go:770: [debug] pre-install-kibana: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:731: [debug] Add/Modify event for pre-install-kibana: MODIFIED
client.go:477: [debug] Starting delete for "kibana-helm-scripts" ConfigMap
client.go:477: [debug] Starting delete for "pre-install-kibana" ServiceAccount
client.go:477: [debug] Starting delete for "pre-install-kibana" Role
client.go:477: [debug] Starting delete for "pre-install-kibana" RoleBinding
client.go:477: [debug] Starting delete for "pre-install-kibana" Job
client.go:396: [debug] checking 17 resources for changes
client.go:684: [debug] Patch PodDisruptionBudget "elk-pdb" in namespace elk
client.go:675: [debug] Looks like there are no changes for ServiceAccount "fluent-bit"
client.go:675: [debug] Looks like there are no changes for Secret "elk-certs"
client.go:675: [debug] Looks like there are no changes for Secret "elk-credentials"
client.go:675: [debug] Looks like there are no changes for ConfigMap "elk-config"
client.go:675: [debug] Looks like there are no changes for ConfigMap "fluent-bit"
client.go:675: [debug] Looks like there are no changes for ConfigMap "kibana-config"
client.go:675: [debug] Looks like there are no changes for ClusterRole "fluent-bit"
client.go:675: [debug] Looks like there are no changes for ClusterRoleBinding "fluent-bit"
client.go:684: [debug] Patch Service "elk" in namespace elk
client.go:675: [debug] Looks like there are no changes for Service "elk-headless"
client.go:675: [debug] Looks like there are no changes for Service "fluent-bit"
client.go:675: [debug] Looks like there are no changes for Service "kibana"
client.go:684: [debug] Patch DaemonSet "fluent-bit" in namespace elk
client.go:684: [debug] Patch Deployment "kibana" in namespace elk
client.go:684: [debug] Patch StatefulSet "elk" in namespace elk
client.go:675: [debug] Looks like there are no changes for Ingress "kibana"
upgrade.go:157: [debug] updating status for upgraded release for elk

Any idea whats wrong? It should not start every time I've made upgrade...
Output of helm version: v3.11.0
Output of kubectl version: 1.21.7
Cloud Provider/Platform (AKS, GKE, Minikube etc.): Selfhosted

It might be helpful to see the object metadata for the resource you're having trouble with.

It might be helpful to see the object metadata for the resource you're having trouble with.

You mean this one:
  name: pre-install-kibana
  labels:
    app: kibana
    release: "elk"
    heritage: Helm
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": hook-succeeded```

Objects from this chart has metadata like that:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: elk
    meta.helm.sh/release-namespace: elk
  labels:
    app: kibana
    app.kubernetes.io/managed-by: Helm
    heritage: Helm
    release: elk
  name: kibana
  namespace: elk