unexpected high count of io_worker threads by using the IOSQE_ASYNC flag

I am not sure if this is expected or not but I do have pretty much as many io worker threads as I have IORING_OP_READ requests on different TCP sockets.
$ uname -a
Linux tau 5.12.3-arch1-1 #1 SMP PREEMPT Wed, 12 May 2021 17:54:18 +0000 x86_64 GNU/Linux

$ ps -L --pid 10334
    PID     LWP TTY          TIME CMD
  10334   10334 pts/0    00:00:00 test
  10334   10336 pts/0    00:00:00 Logging
  10334   10338 pts/0    00:00:00 Housekeeping
  10334   10339 pts/0    00:00:00 iou-mgr-10334
  10334   10341 pts/0    00:00:00 MariaDB
  10334   10342 pts/0    00:00:00 Account
  10334   10343 pts/0    00:00:00 Execution SVC
  10334   10344 pts/0    00:00:00 iou-mgr-10343
  10334   10345 pts/0    00:00:00 iou-wrk-10343
  10334   10346 pts/0    00:00:00 iou-wrk-10343
  10334   10347 pts/0    00:00:00 iou-wrk-10334
  10334   10348 pts/0    00:00:00 iou-wrk-10334
  10334   10349 pts/0    00:00:00 iou-wrk-10334
  10334   10350 pts/0    00:00:00 iou-wrk-10334
  10334   10351 pts/0    00:00:00 iou-wrk-10334
  10334   10352 pts/0    00:00:00 iou-wrk-10334
  10334   10353 pts/0    00:00:00 iou-wrk-10334
  10334   10354 pts/0    00:00:00 iou-wrk-10334
  10334   10355 pts/0    00:00:00 iou-wrk-10334
  10334   10356 pts/0    00:00:00 iou-wrk-10334
  10334   10357 pts/0    00:00:00 iou-wrk-10334
  10334   10358 pts/0    00:00:00 iou-wrk-10334
  10334   10359 pts/0    00:00:00 iou-wrk-10334
  10334   10360 pts/0    00:00:00 iou-wrk-10334
  10334   10361 pts/0    00:00:00 iou-wrk-10334
  10334   10362 pts/0    00:00:00 iou-wrk-10334
  10334   10363 pts/0    00:00:00 iou-wrk-10334
  10334   10364 pts/0    00:00:00 iou-wrk-10334
  10334   10365 pts/0    00:00:00 iou-wrk-10334
  10334   10366 pts/0    00:00:00 iou-wrk-10334
  10334   10367 pts/0    00:00:00 iou-wrk-10334
  10334   10368 pts/0    00:00:00 iou-wrk-10334
  10334   10369 pts/0    00:00:00 iou-wrk-10334
  10334   10370 pts/0    00:00:00 iou-wrk-10334
  10334   10371 pts/0    00:00:00 iou-wrk-10334
  10334   10372 pts/0    00:00:00 iou-wrk-10334
  10334   10373 pts/0    00:00:00 iou-wrk-10334
  10334   10374 pts/0    00:00:00 iou-wrk-10334
  10334   10375 pts/0    00:00:00 iou-wrk-10334
  10334   10376 pts/0    00:00:00 iou-wrk-10334
  10334   10377 pts/0    00:00:00 iou-wrk-10334
  10334   10378 pts/0    00:00:00 iou-wrk-10334
  10334   10379 pts/0    00:00:00 iou-wrk-10334
  10334   10380 pts/0    00:00:00 iou-wrk-10334
  10334   10381 pts/0    00:00:00 iou-wrk-10334
  10334   10382 pts/0    00:00:00 iou-wrk-10334
  10334   10383 pts/0    00:00:00 iou-wrk-10334
  10334   10384 pts/0    00:00:00 iou-wrk-10334
  10334   10385 pts/0    00:00:00 iou-wrk-10334
  10334   10386 pts/0    00:00:00 iou-wrk-10334
  10334   10387 pts/0    00:00:00 iou-wrk-10334
  10334   10388 pts/0    00:00:00 iou-wrk-10334
  10334   10389 pts/0    00:00:00 iou-wrk-10334
  10334   10390 pts/0    00:00:00 iou-wrk-10334
  10334   10391 pts/0    00:00:00 iou-wrk-10334
  10334   10392 pts/0    00:00:00 iou-wrk-10334
  10334   10393 pts/0    00:00:00 iou-wrk-10334
  10334   10394 pts/0    00:00:00 iou-wrk-10334
  10334   10395 pts/0    00:00:00 iou-wrk-10334
  10334   10396 pts/0    00:00:00 iou-wrk-10334
  10334   10397 pts/0    00:00:00 iou-wrk-10334
  10334   10398 pts/0    00:00:00 iou-wrk-10334
  10334   10399 pts/0    00:00:00 iou-wrk-10334
  10334   10400 pts/0    00:00:00 iou-wrk-10334
  10334   10401 pts/0    00:00:00 iou-wrk-10334
  10334   10402 pts/0    00:00:00 iou-wrk-10334
  10334   10403 pts/0    00:00:00 iou-wrk-10334
  10334   10404 pts/0    00:00:00 iou-wrk-10334
  10334   10405 pts/0    00:00:00 iou-wrk-10334
  10334   10406 pts/0    00:00:00 iou-wrk-10334
  10334   10407 pts/0    00:00:00 iou-wrk-10334
  10334   10408 pts/0    00:00:00 iou-wrk-10334
  10334   10409 pts/0    00:00:00 iou-wrk-10334
  10334   10410 pts/0    00:00:00 iou-wrk-10334
  10334   10411 pts/0    00:00:00 iou-wrk-10334
  10334   10412 pts/0    00:00:00 iou-wrk-10334
  10334   10413 pts/0    00:00:00 iou-wrk-10334
  10334   10414 pts/0    00:00:00 iou-wrk-10334
  10334   10415 pts/0    00:00:00 iou-wrk-10334
  10334   10416 pts/0    00:00:00 iou-wrk-10334
  10334   10417 pts/0    00:00:00 iou-wrk-10334
  10334   10418 pts/0    00:00:00 iou-wrk-10334
  10334   10419 pts/0    00:00:00 iou-wrk-10334
  10334   10420 pts/0    00:00:00 iou-wrk-10334
  10334   10421 pts/0    00:00:00 iou-wrk-10334
  10334   10422 pts/0    00:00:00 iou-wrk-10334
  10334   10423 pts/0    00:00:00 iou-wrk-10334
  10334   10424 pts/0    00:00:00 iou-wrk-10334
  10334   10425 pts/0    00:00:00 iou-wrk-10334
  10334   10426 pts/0    00:00:00 iou-wrk-10334
  10334   10427 pts/0    00:00:00 iou-wrk-10334
  10334   10428 pts/0    00:00:00 iou-wrk-10334
  10334   10429 pts/0    00:00:00 iou-wrk-10334
  10334   10430 pts/0    00:00:00 iou-wrk-10334
  10334   10431 pts/0    00:00:00 iou-wrk-10334
  10334   10432 pts/0    00:00:00 iou-wrk-10334

I keep reading the doc over and over... I have just noticed IORING_FEAT_FAST_POLL bit.
Does the fast poll feature totally removes the need of punting async requests to an io thread?
is the request/thread relation is a 1to1 relation?
What are the pros/cons of punting a request vs letting it be fast polled?
or am I mixing different concepts?

I keep reading the doc over and over... I have just noticed IORING_FEAT_FAST_POLL bit.
Does the fast poll feature totally removes the need of punting async requests to an io thread?

yes, partially, but that's far from being the whole story

is the request/thread relation is a 1to1 relation?

No, even without IORING_FEAT_FAST_POLL

What are the pros/cons of punting a request vs letting it be fast polled?

As I already told you, there are many means to do I/O asynchronous depending on many factors, worker threads is slow and expensive but sometime inevitable. io_uring tries to choose the best path, but IOSQE_ASYNC forces to go via worker threads. So, I'd highly discourage you to use the flag unless you know really know what it is.

thx for your feedback Pavel.
I keep reading fs/io_uring.c to figure out the answers myself but I have a hard time understanding this aspect of io_uring.
I hope perseverance will eventually pay off. I think that I got an ahah moment this morning. I think that maybe there is a single work queue per io_uring instance and it is the io_mgr thread that spawn io threads and do the work dispatching to the appropriate thread. I will read a bit fs/io-wq.c to validate this hypothesis.
Bottomline, I did open this issue because I wanted to make sure that the high thread count was normal. Since, you didn't comment on that specifically, I will assume that all is good and will close this issue.
thank you

I keep reading the doc over and over... I have just noticed IORING_FEAT_FAST_POLL bit.
Does the fast poll feature totally removes the need of punting async requests to an io thread?

yes, partially, but that's far from being the whole story

Do you have any pointers for me if I wanted to get the whole story?
design discussion threads, a small list of functions to study to get a better grasp of what io_uring do in that area... or anything else

is the request/thread relation is a 1to1 relation?

No, even without IORING_FEAT_FAST_POLL

but this is what I am experiencing with the test I did above... I did count my fds and I have 88 fds and this happens to be the exact number of io threads spawn.. Any ideay why I get this result?

oh, and I have another question... Since it is kinda related to this one, I am going slip it here instead of creating a new issue.
I am starting to consider playing with more than 2 io_uring instances and I have noticed the IORING_SETUP_ATTACH_WQ setup option.
That sounds like a very good idea!
What could be the reasons for NOT using this feature?
What is the best way to measure the load of the io workers to figure out that there is too much contention from the various io_uring instances to justify letting some instances using a private thread pool?

I did experiment with the IORING_SETUP_ATTACH_WQ option. TBH, by looking at the threads spawn by io_uring, I am not sure at all that I am understanding what it is doing. From the man page:

IORING_SETUP_ATTACH_WQ
This flag should be set in conjunction with struct io_uring_params.wq_fd being set to an existing io_uring ring file descriptor. When set, the io_uring instance being created will share the asynchronous worker thread backend of the specified io_uring ring, rather than create a new separate thread pool.

My confusion comes from what I think is an asynchronous worker thread backend and what it is actually. In my mind, a worker thread backend is the set of mgr/wrk threads sharing the same tid in their name.
This is obviously wrong because I have made several tests with 2 io_uring instances with different flags value:

Using IORING_SETUP_SQPOLL does add an additional mgr thread.
IORING_SETUP_ATTACH_WQ did never change the # of threads spawned

So maybe what a asynchronous worker thread backend really is, it is only the set of wrk threads but each actor, even with IORING_SETUP_ATTACH_WQ set, still needs its own mgr thread... It is hard to tell because in my current setup, I only have a single wrk thread (I will put back the IOSQE_ASYNC option to confirm ;-))
Case #1:
io_uring #1:
flags = 0
io_uring #2:
flags = 0
    PID     LWP TTY          TIME CMD
  13270   13270 pts/0    00:00:01 test
  13270   13275 pts/0    00:00:00 iou-mgr-13270
  13270   13280 pts/0    00:00:00 Execution SVC
  13270   13281 pts/0    00:00:00 iou-mgr-13280
  13270   13282 pts/0    00:00:00 iou-wrk-13270

Case #2:
io_uring #1:
flags = IORING_SETUP_SQPOLL
io_uring #2:
flags = 0
    PID     LWP TTY          TIME CMD
  13325   13325 pts/0    00:00:00 test
  13325   13330 pts/0    00:00:07 iou-sqp-13325
  13325   13331 pts/0    00:00:00 iou-mgr-13325
  13325   13332 pts/0    00:00:00 iou-mgr-13325
  13325   13337 pts/0    00:00:00 Execution SVC
  13325   13338 pts/0    00:00:00 iou-mgr-13337
  13325   13340 pts/0    00:00:00 iou-wrk-13325

Case #3:
io_uring #1:
flags = 0
io_uring #2:
flags = IORING_SETUP_ATTACH_WQ
    PID     LWP TTY          TIME CMD
  13390   13390 pts/0    00:00:01 test_kraken
  13390   13392 pts/0    00:00:00 Logging
  13390   13394 pts/0    00:00:00 Housekeeping
  13390   13395 pts/0    00:00:00 iou-mgr-13390
  13390   13397 pts/0    00:00:00 Kraken MariaDB
  13390   13398 pts/0    00:00:00 Kraken Account
  13390   13399 pts/0    00:00:00 Execution SVC
  13390   13400 pts/0    00:00:00 iou-mgr-13399
  13390   13401 pts/0    00:00:00 iou-wrk-13390

Case #3:
io_uring #1:
flags = IORING_SETUP_SQPOLL
io_uring #2:
flags = IORING_SETUP_ATTACH_WQ
$ ps -L --pid 9428
    PID     LWP TTY          TIME CMD
   9428    9428 pts/1    00:05:09 test
   9428    9435 pts/1    02:09:02 iou-sqp-9428
   9428    9436 pts/1    00:00:00 iou-mgr-9428
   9428    9437 pts/1    00:00:00 iou-mgr-9428
   9428    9442 pts/1    00:00:00 Execution SVC
   9428    9443 pts/1    00:00:00 iou-mgr-9442
   9428    9444 pts/1    00:00:00 iou-wrk-9428

I have redone Case #3 with IOSQE_ASYNC:
io_uring #1:
flags = 0
io_uring #2:
flags = IORING_SETUP_ATTACH_WQ
io_uring instance #2 still has its won iou-wrk thread (iou-wrk-13873) so, I have zero clue about what IORING_SETUP_ATTACH_WQ is supposed to do...
   PID     LWP TTY          TIME CMD
  13864   13864 pts/0    00:00:00 test
  13864   13869 pts/0    00:00:00 iou-mgr-13864
  13864   13873 pts/0    00:00:00 Execution SVC
  13864   13874 pts/0    00:00:00 iou-mgr-13873
  13864   13875 pts/0    00:00:00 iou-wrk-13873
  13864   13877 pts/0    00:00:00 iou-wrk-13864
  13864   13878 pts/0    00:00:00 iou-wrk-13864
  13864   13879 pts/0    00:00:00 iou-wrk-13864
  13864   13880 pts/0    00:00:00 iou-wrk-13864
  13864   13881 pts/0    00:00:00 iou-wrk-13864
  13864   13882 pts/0    00:00:00 iou-wrk-13864
  13864   13883 pts/0    00:00:00 iou-wrk-13864
  13864   13884 pts/0    00:00:00 iou-wrk-13864
  13864   13885 pts/0    00:00:00 iou-wrk-13864
  13864   13886 pts/0    00:00:00 iou-wrk-13864
  13864   13887 pts/0    00:00:00 iou-wrk-13864
  13864   13888 pts/0    00:00:00 iou-wrk-13864
  13864   13889 pts/0    00:00:00 iou-wrk-13864
  13864   13890 pts/0    00:00:00 iou-wrk-13864
  13864   13891 pts/0    00:00:00 iou-wrk-13864
  13864   13892 pts/0    00:00:00 iou-wrk-13864
  13864   13893 pts/0    00:00:00 iou-wrk-13864
  13864   13894 pts/0    00:00:00 iou-wrk-13864
  13864   13895 pts/0    00:00:00 iou-wrk-13864
  13864   13896 pts/0    00:00:00 iou-wrk-13864
  13864   13897 pts/0    00:00:00 iou-wrk-13864
  13864   13898 pts/0    00:00:00 iou-wrk-13864
  13864   13899 pts/0    00:00:00 iou-wrk-13864
  13864   13900 pts/0    00:00:00 iou-wrk-13864
  13864   13901 pts/0    00:00:00 iou-wrk-13864
  13864   13902 pts/0    00:00:00 iou-wrk-13864
  13864   13903 pts/0    00:00:00 iou-wrk-13864
  13864   13904 pts/0    00:00:00 iou-wrk-13864
  13864   13905 pts/0    00:00:00 iou-wrk-13864
  13864   13906 pts/0    00:00:00 iou-wrk-13864
  13864   13907 pts/0    00:00:00 iou-wrk-13864
  13864   13908 pts/0    00:00:00 iou-wrk-13864
  13864   13909 pts/0    00:00:00 iou-wrk-13864
  13864   13910 pts/0    00:00:00 iou-wrk-13864
  13864   13911 pts/0    00:00:00 iou-wrk-13864
  13864   13912 pts/0    00:00:00 iou-wrk-13864
  13864   13913 pts/0    00:00:00 iou-wrk-13864
  13864   13914 pts/0    00:00:00 iou-wrk-13864
  13864   13915 pts/0    00:00:00 iou-wrk-13864
  13864   13916 pts/0    00:00:00 iou-wrk-13864
  13864   13917 pts/0    00:00:00 iou-wrk-13864
  13864   13918 pts/0    00:00:00 iou-wrk-13864
  13864   13919 pts/0    00:00:00 iou-wrk-13864
  13864   13920 pts/0    00:00:00 iou-wrk-13864
  13864   13921 pts/0    00:00:00 iou-wrk-13864
  13864   13922 pts/0    00:00:00 iou-wrk-13864
  13864   13923 pts/0    00:00:00 iou-wrk-13864
  13864   13924 pts/0    00:00:00 iou-wrk-13864
  13864   13925 pts/0    00:00:00 iou-wrk-13864
  13864   13926 pts/0    00:00:00 iou-wrk-13864
  13864   13927 pts/0    00:00:00 iou-wrk-13864
  13864   13928 pts/0    00:00:00 iou-wrk-13864
  13864   13929 pts/0    00:00:00 iou-wrk-13864
  13864   13930 pts/0    00:00:00 iou-wrk-13864
  13864   13931 pts/0    00:00:00 iou-wrk-13864
  13864   13932 pts/0    00:00:00 iou-wrk-13864
  13864   13933 pts/0    00:00:00 iou-wrk-13864
  13864   13934 pts/0    00:00:00 iou-wrk-13864
  13864   13935 pts/0    00:00:00 iou-wrk-13864
  13864   13936 pts/0    00:00:00 iou-wrk-13864
  13864   13937 pts/0    00:00:00 iou-wrk-13864
  13864   13938 pts/0    00:00:00 iou-wrk-13864
  13864   13939 pts/0    00:00:00 iou-wrk-13864
  13864   13940 pts/0    00:00:00 iou-wrk-13864
  13864   13941 pts/0    00:00:00 iou-wrk-13864
  13864   13942 pts/0    00:00:00 iou-wrk-13864
  13864   13943 pts/0    00:00:00 iou-wrk-13864
  13864   13944 pts/0    00:00:00 iou-wrk-13864
  13864   13945 pts/0    00:00:00 iou-wrk-13864
  13864   13946 pts/0    00:00:00 iou-wrk-13864
  13864   13947 pts/0    00:00:00 iou-wrk-13864
  13864   13948 pts/0    00:00:00 iou-wrk-13864
  13864   13949 pts/0    00:00:00 iou-wrk-13864
  13864   13950 pts/0    00:00:00 iou-wrk-13864
  13864   13951 pts/0    00:00:00 iou-wrk-13864
  13864   13952 pts/0    00:00:00 iou-wrk-13864
  13864   13953 pts/0    00:00:00 iou-wrk-13864
  13864   13954 pts/0    00:00:00 iou-wrk-13864
  13864   13955 pts/0    00:00:00 iou-wrk-13864
  13864   13956 pts/0    00:00:00 iou-wrk-13864
  13864   13957 pts/0    00:00:00 iou-wrk-13864
  13864   13958 pts/0    00:00:00 iou-wrk-13864
  13864   13959 pts/0    00:00:00 iou-wrk-13864
  13864   13960 pts/0    00:00:00 iou-wrk-13864
  13864   13961 pts/0    00:00:00 iou-wrk-13864
  13864   13964 pts/0    00:00:00 iou-wrk-13864
  13864   13965 pts/0    00:00:00 iou-wrk-13864

ok, based on what I have understood from my fs/io_uring.c digging,
asynchronous worker thread backend = SQPOLL thread.
so maybe:

Clarify this in the man page
Maybe make sure that if IORING_SETUP_ATTACH_WQ is set, IORING_SETUP_SQPOLL is also set or else return EINVAL in io_uring_setup()

Since 5.12 IORING_SETUP_ATTACH_WQ does not affect io-wq, but is only for sqpoll sharing
io-wq is not related to SQPOLL, apart from that both sqpoll and non-sqpoll modes might use it.
io-wq is already shared, automatically and in a bit more complex manner.

ok, based on what I have understood from my fs/io_uring.c digging,
asynchronous worker thread backend = SQPOLL thread.

Wrong

Good point about creating too many threads, there are restrictions, but in some cases it will exhaust RLIMIT_NPROC. May make sense to allow to limit them.

ok, based on what I have understood from my fs/io_uring.c digging,
asynchronous worker thread backend = SQPOLL thread.

Wrong

Why?
The only places where wq_fd is accessed are:
io_attach_sq_data() and
io_sq_offload_create()
io_attach_sq_data() appears to be only called from io_get_sq_data() which is called from io_sq_offload_create()...
So in the end only when SQPOLL is created that wq_fd is accessed. Therefore my conclusion that it is only useful for it...

ok, based on what I have understood from my fs/io_uring.c digging,
asynchronous worker thread backend = SQPOLL thread.

Wrong

Why?
The only places where wq_fd is accessed are:
io_attach_sq_data() and
io_sq_offload_create()
io_attach_sq_data() appears to be only called from io_get_sq_data() which is called from io_sq_offload_create()...
So in the end only when SQPOLL is created that wq_fd is accessed. Therefore my conclusion that it is only useful for it...

Exactly, since 5.12 it's effectively used only for SQPOLL, but that doesn't mean that worker thread backend is SQPOLL. Legacy reasons... just think that it's IORING_SETUP_ATTACH and attach_fd and nothing with wq in naming.

Does the constraint of not being allowed to mix and match polled and non-polled I/O from using IORING_SETUP_IOPOLL is propagated to the second io_uring when it attaches itself with IORING_SETUP_ATTACH_WQ?

Case #5:
io_uring #1:
flags = IORING_SETUP_SQPOLL
io_uring #2:
flags = IORING_SETUP_SQPOLL|IORING_SETUP_ATTACH_WQ
    PID     LWP TTY          TIME CMD
  14189   14189 pts/0    00:00:00 test
  14189   14194 pts/0    00:00:11 iou-sqp-14189
  14189   14195 pts/0    00:00:00 iou-mgr-14189
  14189   14196 pts/0    00:00:00 iou-mgr-14189
  14189   14200 pts/0    00:00:00 Execution SVC
  14189   14202 pts/0    00:00:00 iou-wrk-14189

It has a single sqp thread which is pretty cool. SQPOLL is literally taking over a single CPU. If you have to have one, you better let all your io_ring instances benefit from it...
Now, I only have to find out what is the whole story with IORING_SETUP_IOPOLL. Would
io_uring #1:
flags = IORING_SETUP_SQPOLL|IORING_SETUP_IOPOLL
io_uring #2:
flags = IORING_SETUP_SQPOLL|IORING_SETUP_ATTACH_WQ
be valid?
or it would absolutely need to be:
io_uring #1:
flags = IORING_SETUP_SQPOLL|IORING_SETUP_IOPOLL
io_uring #2:
flags = IORING_SETUP_SQPOLL|IORING_SETUP_ATTACH_WQ|IORING_SETUP_IOPOLL

Does the constraint of not being allowed to mix and match polled and non-polled I/O from using IORING_SETUP_IOPOLL is propagated to the second io_uring when it attaches itself with IORING_SETUP_ATTACH_WQ?

No, can be intermixed

ie: Could the first io_uring be a IORING_SETUP_IOPOLL instance and the second attached io_uring using IORING_SETUP_ATTACH_WQ is not?

Yes

ok then. all is good. I'll be closing this issue for good. thx a lot for your help and feedback!


Refactor fine-tuning scripts

Light cleanup, fixed a few minor broken things.

Codecov Report

Merging #1797 (697f947) into master (dc41b37) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1797   +/-   ##
=========================================
  Coverage     57.71%   57.71%
  Complexity     1045     1045
=========================================
  Files           175      175
  Lines          9885     9885
  Branches       1362     1362
=========================================
  Hits           5705     5705
  Misses         3699     3699
  Partials        481      481

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc41b37...697f947. Read the comment docs.


internal/benchmarks: cleanup redundant bench tests

CL/183245 added a general purpose benchmark framework for all the various serialization formats.
Do we still need the others?

encoding/bench_test.go
proto/bench_test.go

\cc @neild @cybrcodr

Probably not, although proto/bench_test.go has been interesting for comparing the performance of very specific things (maps, oneofs, etc.) on a very micro level. It might be interesting to add some focused benchmarks for those to the benchmarks directory.

I just checked out benchmark/bench_test.go.  That's neat.  Yeah, that can replace encoding/bench_test.go.
Btw, just a note that passing in -v1 to benchmark text marshal/unmarshal may produce the same results depending on which v1 is being used.  With the go.mod file checked in, I believe it is using the latest v1 where text marshal/unmarshal are already wrappers to the v2 implementation.
In order to properly get v1 values, one will need to adjust go.mod to point to the real old v1 implementation.

I don't think there's much value in keeping an issue just for this. There's always a number of things that can be cleaned up.


[ZAP] The generated defaults in endpoint_config.h does not comply with endianness

Problem
When setting a default value for a particular attribute in the .zap file, the resulting generated file endpoint_config.h does not reflect the big-endian and little-endian versions correctly. They are both stored big-endian wise. That would be expected to be different in the two GENERATED_DEFAULTS tables for big- and little-endian.
Example:
Setting the LifetimeRunningHours in the PCC cluster to 0x123456 in the ZAP UI tool, and then saving the .zap file, results in the following content in the .zap file. Which is correct.
            {
              "name": "LifetimeRunningHours",
              "code": 21,
              "mfgCode": null,
              "side": "server",
              "included": 1,
              "storageOption": "RAM",
              "singleton": 0,
              "bounded": 0,
              "defaultValue": "0x123456",
              "reportable": 0,
              "minInterval": 1,
              "maxInterval": 65534,
              "reportableChange": 0
            },

Then running the ./scripts/tools/zap_regen_all.py script generates the following contents in the endpoint_config.h file.
#if BIGENDIAN_CPU
#define GENERATED_DEFAULTS                                                                                \
    {                                                                                \
.......
            /* Endpoint: 1, Cluster: Pump Configuration and Control (server), big-endian */                                        \
                                                                                \
            /* 1377 - LifetimeRunningHours, */                                                                                \
            0x12, 0x34, 0x56,                                                                                \
.......
#else // !BIGENDIAN_CPU
#define GENERATED_DEFAULTS                                                                                \
    {                                                                                \
.......
            /* Endpoint: 1, Cluster: Pump Configuration and Control (server), little-endian */                                     \
                                                                                \
            /* 1377 - LifetimeRunningHours, */                                                                                \
            0x12, 0x34, 0x56,                                                                                \
                                                                                \
.......
    }

#endif // BIGENDIAN_CPU

Proposed Solution
Make the ZAP tool generate the correct default values in the endpoint_config.h files.

It is a know issues which block FeatureMap for example

https://github.com/project-chip/connectedhomeip/pull/9725


Name collision

You might want to consider a rename.
https://github.com/dcramer/mangodb

ManosDB:
https://en.wikipedia.org/wiki/Manos:_The_Hands_of_Fate
¯_(ツ)_/¯

ThanosDB

How about PosMonDB,
Taking good of both worlds :3

MongrelDB -- a mongrel is "an animal resulting in the crossing of different breeds or types". Mixing Mongo and Postgres counts I think?

tbh change to MongrelDB seems good, the refactoring needed is minimal..

MangroveDB - like mangrove forest ...

MingaDB

ModuloDB?
In computing, the modulo operation returns the remainder or signed remainder of a division, after one number is divided by another (called the modulus of the operation).


Add ability to parse empty control with attributes

Okay, this is an admittedly weird thing to do but: this PR adds the ability to parse empty control blocks when they have attributes.
The Good
For example, this can be parsed now:
control {
  @sync; // <-- note the semicolon
}

The reason is that it gives us to experiment with "proto-keywords" like more easily. We currently have the philosophy of trying features out using attributes before we turn them into full blown keywords.
For example, the @sync keyword is an example. The current semantics is that for a program like this:
control {
  @sync A; B;
}

The synchronization happens before A executes. This means that sometimes we have to write programs like this:
control {
  if lt.out {
    @sync no_op;
  } else {
    @sync A; B;
  }
}

Where the no_op group doesn't do anything. With this change, we can write:
control {
  if lt.out {
    @sync;
  } else {
    @sync; A; B;
  }
}

This also has the pleasant outcome of making the meaning of @sync clearer---the synchronization occurs before the group executes. We also don't need to have special handling for invoke.
The Bad
The bad thing is that this is an empty which means that it shouldn't really have any semantics and should be erasable? This would make it important for passes to keep empty control programs around when they have an attribute attached.
On the other hand, imagine the possibilities–a @log keyword for print out values? @delay to insert fake delays between par threads to test their delay insensitivity?

I think this is a great thing as it certainly makes the semantics of sync clearer. Let me try to change the sync test cases to see if this gets compiled properly to barriers

hmm, I can kinda see this for sync but I have some misgivings. On the whole the idea of using empty groups with special attributes seems to me a means of hacking in a set of auxiliary control psuedo-commands (pragmas?) that influence compilation but don't correspond to group-type compilation flows. And if that's what we want, it might be better to let those types of commands stand alone.
lmk if this makes any sense. This is just my initial reaction so I am open to being persuaded.

@EclecticGriffin absolutely! Thinking of these as pseudo commands is pretty much what I was going for. I agree that we would eventually want to turn all of these into real control operators to do it “the right way”
Having this backdoor is more a question of making it easier to experiment with things quickly. Adding a new command is an infrastructure-wide change while adding a new attribute is trivial.
In fact, it also makes it easier for someone outside of the core team to play with new control like operators because all they have to do is add the attribute and write a pass to interpret it

makes sense to me!

Yeah, I too agree that this is a strange but creative solution to the problem! Seems like a good idea to run with it. In all fairness, attributes were already not "semantics-free"… especially in the case of @sync, your program's correctness really relies on the synchronization actually happening. So this is not much of a slide down the slippery slope, IMO, and it does open the door to interesting non-built-in control statements.


Fix http request format

Update the PUT requests, with help from @brian-harris
Fixes #22606

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 2 (0.0%)  changed or added relevant lines in 1 file are covered.
5 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.009%) to 41.686%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

extensions/azurecore/src/account-provider/auths/httpClient.ts
0
2
0.0%

Files with Coverage Reduction
New Missed Lines
%

extensions/notebook/src/dialog/configurePython/configurePathPage.ts
1
57.02%

extensions/sql-bindings/src/common/azureFunctionsUtils.ts
1
80.65%

extensions/sql-database-projects/src/common/httpClient.ts
1
66.18%

extensions/sql-database-projects/src/dialogs/publishDatabaseDialog.ts
2
50.74%

Totals

Change from base Build<PHONE_NUMBER>:
-0.009%

Covered Lines:
30348

Relevant Lines:
67401

💛  - Coveralls

FYI, I'm seeing an issue with calling getBlobs in azurecore - even though that's a get it seems it uses a post behind the scenes to get the storage account keys, but even with your change here I'm still seeing errors:

cc @cssuh @Charles-Gagnon @cheenamalhotra @brian-harris

@raymondtruong Could you copy the entire contents of the message? Seems to be a different error.

@raymondtruong Could you copy the entire contents of the message? Seems to be a different error.

sure thing, here's the error that gets populated after calling getBlobs:
RestError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:ae992142-101e-0070-2eea-679c5e000000
Time:2023-04-05T18:14:08.8937409Z
    at handleErrorResponse (c:\\Users\\ratruong\\source\\repos\\azuredatastudio2\\extensions\\azurecore\
ode_modules\\@azure\\core-http\\dist\\index.js:3174:19)
    at c:\\Users\\ratruong\\source\\repos\\azuredatastudio2\\extensions\\azurecore\
ode_modules\\@azure\\core-http\\dist\\index.js:3110:49
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async StorageRetryPolicy.attemptSendRequest (c:\\Users\\ratruong\\source\\repos\\azuredatastudio2\\extensions\\azurecore\
ode_modules\\@azure\\storage-blob\\dist\\index.js:14673:24)
    at async StorageClientContext.sendOperationRequest (c:\\Users\\ratruong\\source\\repos\\azuredatastudio2\\extensions\\azurecore\
ode_modules\\@azure\\core-http\\dist\\index.js:4893:31)
    at async ContainerClient.listBlobFlatSegment (c:\\Users\\ratruong\\source\\repos\\azuredatastudio2\\extensions\\azurecore\
ode_modules\\@azure\\storage-blob\\dist\\index.js:23381:30)"

@raymondtruong can you also paste the request here that you're able to see in debug session?

@raymondtruong can you also paste the request here that you're able to see in debug session?

@cheenamalhotra yep, I'll ping you guys offline if that's okay - full request and response is pretty large and I don't know what might be included so I'd rather not paste it here

the latest commit should fix the issues @raymondtruong was seeing.


Documentation: Specs link under Coding Rules in Developers.md not working

Discussion 🗣

Documentation: Specs link under Coding Rules in Developers.md not working
Context

When working on the source code yo ensure consistency throughout, all features or bug fixes must be tested by one or more specs.
here the specs link is pointing to the same file so from where we can see those specifications?, and if there's a correct link for this
Detailed Description

Hey!! @arpit-maurya, I'm not the maintainer for this project, Please ask @mttrbrts, @dselman, @irmerk for assigning this to you :)

Open a PR if you want to work on this.


fts5 syntax error when using punctuation

Hi Simon
I get a syntax error when using punctuation or special characters in a fulltext search (using fts5). I created the virtual table using sqlite-utils' "enable-fts"-command.
The same error appears on Niche Museums https://www.niche-museums.com/browse/search?q=park., but works fine in most of your other datasette-examples, e.g. register-of-members-interests https://register-of-members-interests.datasettes.com/regmem-98dc8b7/items?_search=mins.
What am I doing wrong? Many thanks!

I've not yet been able to figure out what the escaping rule are for FTS5 queries.
If we figure out how those work maybe we can bundle them as a custom function?
select ... where docs_fts match fts_escape(:search)

https://stackoverflow.com/a/43756146 Says that an escaping mechanism that works is this one:
select * from blah where term match '"bacon" "and" "eggs"'

So split ok whitespace and then encapsulate each search term in double quotes.


Devops: improve the development environment and  vs-code settings

[x] Update .gitignore file;
[x] Add vs-code launch configuration file in order to be able to run and debug the app from IDE directly;
[x] Add the code-spell-checker plugin entry to list of recommended plugins;
[x] Update spelling dictionary with unknown words and acronyms;
[x] BTW... lint readme.md.

I already got Github notifications,  you don't need to mention me. I'll handle it when I got free time.


fix(window): hopping window consumes too much memory

Unnecessary memory are holden when hopping window interval is bigger than length.

Close as it is not a normal usage of hopping window. Usually, the window length is bigger than hop size.


Won't load - was working fine

I had it all working and had agents running testing out some things.
I left it for a few hours doing nothing, and when I came back the UI was frozen or sorts, Refreshing the page did nothing.
I checked docker and it was still running.
I stopped docker and closed the UI,
I restarted docker using terminal inside VCS by entering docker-compose up --build it ran through the code and everything looked fine, but localhost:3000 won't load, I tried stopping and starting many times... It just won't load.
Any ideas? or anywhere I can see an error log or something?

@NeVeREire are you seeing any errors in docker logs? It's also not running for me, as I described in #269

@NeVeREire can you share the logs please?

@NeVeREire is the issue resolved with the newer version releases?

This seems to have solved with the newer version releases. Closing this.


Changing offload to NVMe instead of CPU causes error: TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Describe the bug
I'm able to run peft pre-trained models from:
https://github.com/huggingface/peft
I can successfully offload to CPU memory using deepspeed.
The model I used is "facebook/bart-large"
but when I try to run the exact same model to offload to NVMe, I get this error:
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
My accelerate config file "ds_zero3_nvme.yaml"
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  deepspeed_config_file: /home/ubuntu/peft/deepspeed_config.json
distributed_type: DEEPSPEED
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

My deepspeed config file "deepspeed_config.json":
  "zero_optimization": {
    "stage": 3,
    "stage3_max_live_parameters": 3e9,
    "stage3_max_reuse_distance": 3e9,
    "stage3_param_persistence_threshold": 1e5,
    "stage3_prefetch_bucket_size": 5e7,
    "contiguous_gradients": true,
    "overlap_comm": true,
    "reduce_bucket_size": 90000000,
    "sub_group_size": 1e9,
    "offload_optimizer": {
        "device": "nvme",
        "nvme_path": "/mnt/nvme1",
        "pin_memory": true,
        "buffer_count": 4,
        "fast_init": false
    }
  },
  "gradient_clipping": 1.0,
  "wall_clock_breakdown": true,
  "zero_allow_untested_optimizer": false,
  "aio": {
        "block_size": 262144,
        "queue_depth": 32,
        "thread_count": 1,
        "single_submit": false,
        "overlap_events": true
  }
}

I built deepspeed from source with  option DS_BUILD_AIO=1:
TORCH_CUDA_ARCH_LIST="7.5" DS_BUILD_AIO=1 DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 pip install . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1 | tee build.log

git clone https://github.com/huggingface/peft && cd peft
accelerate launch --config_file ds_zero3_nvme.yaml examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py
Error:
Traceback (most recent call last):
File "/home/ubuntu/peft/examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py", line 367, in
main()
File "/home/ubuntu/peft/examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py", line 231, in main
model, train_dataloader, eval_dataloader, test_dataloader, optimizer, lr_scheduler = accelerator.prepare(
File "/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 1220, in prepare
result = self._prepare_deepspeed(*args)
File "/home/ubuntu/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 1467, in _prepare_deepspeed
"train_batch_size": batch_size_per_device
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
Expected behavior
A clear and concise description of what you expected to happen.
ds_report output
[2024-02-13 07:19:25,899] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [YES] ...... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/ubuntu/.local/lib/python3.10/site-packages/torch']
torch version .................... 2.1.0+cu121
deepspeed install path ........... ['/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.12.7+43daf413, 43daf413, master
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.3
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 15.44 GB

System info (please complete the following information):

OS: [22.04.1-Ubuntu]
GPU count and types p4dn.2xlarge aws 1 GPU Tesla T4]

Launcher context
launched using accelerate

I wanted to update: I am able to manually setup NVMe SSD as a swap device and then force deepspeed to swap to NVMe SSD by limiting the CPU connected RAM available, but I still cant get the optimized, out of the box NVMe offload to function.

I'd appreciate if anyone else has this issue , and if you were able to resolve.
I wanted to find out if manually setting up NVMe SSD as a swap device, and limiting CPU RAM so that I force swap to NVMe SSD is similar in performance to the native DeepSpeed setup for offloading to NVMe

Can you post the rest of the ds_config? Based on the stack trace you gave the error is coming from accelerate and involves the batch size setting. Is that the only error?

The above is the entire ds_config file, there is nothing else. This is the only error. Batch size is set in the main python script:
    accelerator = Accelerator()
    model_name_or_path = "facebook/bart-large"
    dataset_name = "twitter_complaints"
    peft_config = LoraConfig(task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
    text_column = "Tweet text"
    label_column = "text_label"
    lr = 3e-3
    num_epochs = 2
    batch_size = 8
    seed = 42
    max_length = 64
    do_test = False
    set_seed(seed)

    dataset = load_dataset("ought/raft", dataset_name)
    classes = [k.replace("_", " ") for k in dataset["train"].features["Label"].names]
    dataset = dataset.map(
        lambda x: {"text_label": [classes[label] for label in x["Label"]]},
        batched=True,
        num_proc=1,
    )

I tried changing batch size to 1 but same error.

Since the error is coming from accelerate and the is no part of the DeepSpeed code in the stack trace, I believe it is a parsing error in their code. After review their code I suggest 1) You switch to the latest version. 2) Set train_micro_batch_size_per_gpu in your ds_config to ensure it is being parsed correctly.

If you are using accelerate, you have to define train_micro_batch_size_per_gpu in your deepspeed configuration file according to this, otherwise its None * int * int for train_batch_size


Automated attention checks

While attention checks can be performed manually, we can make it easier by having users specify a directory of attention checks and specify how many checks per participant should be performed. Then, we can automatically remove these examples from the statistical analysis.

Creation interface should optionally accept a directory of attention check files with known answers
Config file should allow specifying number of checks and specifying correct answers (varies for each test)

# MOS
attention_check:

    # Number of checks to perform per participant
    checks_per_participant: 2

    # Correct answers (a dictionary mapping conditions to a dictionary mapping filenames to scores)
    answers:

        <condition1>:
            <file1>: score
            <file2>: score
            ...
        <condition2>:
            ...

Integrate into frontend by adding checks as samples for participants to evaluate
End participants early if they fail attention check
Don't perform analysis on results of participants who fail the attention check


Error handling and retry policies for silent calls.

Library

[ ]<EMAIL_ADDRESS>or<EMAIL_ADDRESS>[x]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>
Description
Sorry if this has been asked earlier.
So i wanted to know what all errors fall into the case of a retry-able error (500s or any similar ones) or an error which can be retried after some time (eg. 429s) for silent calls such acquireTokenSilent or ssoSilent.
Is there a recommended way of dealing with these errors such as 429s which may require a back-off time.
I am assuming these are not already retried by default.
Is there a specific documentation for error handling in general? I could see some error handling here but not a lot of details were present on the scenarios other than InteractionRequiredAuthError.

We have some documentation in the repo on this topic.
The cases that cause a back-off are:

429 and 5xx statuses
Response contains Retry-After header without a 2xx status
InteractionRequiredError on the response

The default is that a call will be locked out for 60 seconds, but the server can specify various amounts of time. Interactive calls will bypass this and also alleviate the lockout if successful. They are also not retried by default. The idea is that you should always be able to alleviate the problem rather than just wait for the timeout to end.
The error and timing information is stored in a ThrottlingEntity object as a value in local storage.
The error currently gets exposed as a ServerError, though there was some discussion about making a ThrottlingError type that exposes this more explicitly with the time the call is locked out.
This is pretty deep information, so let me know if things aren't clear or you have more questions.

Thanks Jake for the info.
So if I understand correctly (feel free to correct me otherwise)

For 5xx or 429, we'll get ServerError thrown when making a silent call.
This can be fixed (or rather retried) using an interactive call for the same request?
Are there any specific error codes with respect to the ServerError object received in the error block for which an interactive call should be made?
I believe ServerError is currently not exported from @azure/msal-browser. I am assuming that might be because ServerError acts as a base class for other errors that we see.

For 5xx or 429, we'll get ServerError thrown when making a silent call.

5xx will be ServerError, 429 is just a generic backoff request that the library respects. There could be any errors associated with it as well.

This can be fixed (or rather retried) using an interactive call for the same request?

Yes, if you make an interactive call it bypasses any throttles set--and a successful interactive call will alleviate that throttle.

Are there any specific error codes with respect to the ServerError object received in the error block for which an interactive call should be made?

The main one is InteractionRequiredAuthError (different from ServerError, to be clear). It's up the the app developer to decide how they handle particular errors, but the most common cases are to always fall back to an interactive call if a silent calls fails, or to fall back to an interactive call if a silent call fails with InteractionRequiredAuthError.

I believe ServerError is currently not exported from @azure/msal-browser. I am assuming that might be because ServerError acts as a base class for other errors that we see.

Yes, exactly, it is exported from msal-common which msal-browser consumes.


Module not found

When I updated to version 1.83.5 I found that many imports were no longer available，
After checking the documentation, changing to the following method does not work either.
import getEditorServiceOverride from '@codingame/monaco-vscode-editor-service-override'
import getConfigurationServiceOverride, { updateUserConfiguration, configurationRegistry } from '@codingame/monaco-vscode-configuration-service-override'

They are separate packages now, you need to install them

Thanks for the answer, but it’s too bad that so many packages have to be installed manually. I suggest you reduce the number of packages.

There are 100+ software packages used in the demo. If you want to install these packages, you must run 100+ installation commands. If you need to update the version in the future, you need to perform this step again, which makes managing dependencies very disadvantageous.

The demo tries to demonstrate as much as possible so it requires almost all packages, that's not the case for most of the users
That's the only way to not make user download what they don't need: 1 package = 1 purpose
If you think some of them are likely to be used together, I'm open to create another meta-package that pulls them all
if you want to update them all, you can use npm-check-updates => ncu -i '@codingame/monaco-vscode-*'

Mainstream packaging tools do not package unused functions together, so I don’t mind downloading more unnecessary content.

Well, I do. Some are really heavy. With a meta package, everyone would be happy, right?


.github: fix workflow settings and allow branch pushes to main by com…

…mitters
This fixes the comment message for merge conflict checker and resets the
github branch protection for main in case of need by the RM.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

In the last run [1], it appears the workflow has worked but failed to ping the PR author (the variable wasn't expanded as documented in one of the wikis/blogs). This attempts to fix that. cc @nvazquez @weizhouapache @Pearl1594 - hope this helps with RM activities, and mundane tasks are automated.
[1] https://github.com/apache/cloudstack/pulls?q=is%3Apr+is%3Aopen+label%3Astatus%3Ahas-conflicts


enh: archive all button

Is your feature request related to a problem? Please describe.
Yes, currently users need to manually clean chat history and archive chats in Ollama WebUI, which can be time-consuming and inconvenient.
Describe the solution you'd like
I would like to add a "Cleanup All" or "Archive All" button to the chat section of the Ollama WebUI. This button should automatically clear chat history or archive all relevant conversations.
Describe alternatives you've considered
One alternative is to provide an option in settings to enable automatic cleanup or archiving after a certain period or number of messages. However, this would still require user intervention.

Delete all chats button already exists, archive all button would be a nice addition. PR welcome!

This feature is a commonly feature. It would be most beneficial to showcase it in the prime location of the homepage, ensuring easy access for users.

Closing in favour of #2483


Newer version of Javassist breaks shading

I know you've already rolled back from Javassist 3.19.0-GA to 3.18.2-GA, but I thought I would mention this since it took me a long time to track down. I shade HikariCP into my application. I cannot shade any version in [2.3.0, 2.3.7]. Specifically, any build between e3316e8618a1b00a180468041734a042738d161b and eadf8d1911da73bed9f96e5fa0d9feae11058328 causes the extremely helpful error
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.3:shade (default) on project insight-sql: Error creating shaded jar: null: RuntimeException -> [Help 1]

Using 3.18.2-GA makes it work normally again. Any chance you can cut a 2.3.8 release soon?

Yes, I'll try to publish one tonight.

@ams2990 HikariCP 2.3.8 has been published to the maven central repository.  It should be visible in approximately 3 hours.

Will Javassist 3.20.0-GA recently released fix this?

Fixed issues: https://issues.jboss.org/browse/JASSIST/fixforversion/12326488/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel

@guidomedina I guess it won't be relevant much anymore since hikari 2.4 won't have a Javassist dependency anymore.
Check the "dev" branch for the specifics.

Nice, one less jar for the project :D


[pytket-qsharp] import qsharp fails

On windows with python 3.8 the import of qsharp fails with the following error message:
from qsharp import compile as qscompile  # type: ignore
C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\qsharp_init_.py:198: in
client = start_client()
C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\qsharp\clients_init.py:34: in _start_client
client = qsharp.clients.iqsharp.IQSharpClient(kernel_name=kernel_name)
E AttributeError: partially initialized module 'qsharp' has no attribute 'clients' (most likely due to a circular import)
The fail occurs on the github ci:
We tried different qsharp version, for example 0.18.2107.153439 and 0.18.2108.160999. The error occurs in all of them.
https://github.com/CQCL/pytket-extensions/runs/3534918995?check_suite_focus=true
The problem was reported to qsharp on:
https://github.com/microsoft/iqsharp/issues/512

Still an issue with qsharp 0.20.


Show an unmatched destination path when uploading files to one subfolder for ADLS Gen2 accounts

Storage Explorer Version:1.7.0_20190313.2
Platform/OS: _Linux Ubuntu/macOS High Sierra/Windows 10
Architecture: _i86
Regression From: Previous release 1.7.0
Steps to Reproduce

Expand one ADLS Gen2 account -> 'Blob Containers' node.
Open one blob container -> Create a new folder then go to it.
Click 'Upload Files...' -> Check the path of the 'Destination directory' on 'Upload Files' dialog.

Expect Experience:
There shows the path of subfolder.
Actual Experience:
There only shows the root path.

More info:
This issue doesn't reproduce for normal blob containers and file shares.

Fix merged into master


Adapt to recent version of PHPUnit (8)

Thanks for this PR.
Merging this though would cause some major issues: As this code base hasn't been touched in quite a while, the minimum PHP version requirement is still PHP 5.3.
For PHP to be able to parse the Tests after your change, PHP 7.1 would be required. To actually use PHPUnit 8, we'd need PHP 7.2 and up.
While that is technically not a problem, this at the very least would require a new major release, raising the minimum version requirement to PHP 7.2. If we'd do that, we should also upgrade the main code base to make use of that version.
If we do that, we could also use PHPUnit 9 rather than just 8.
Not sure if that is worth the effort ;)

I implemented some last changes so the code works with PHP 8.1, that included moving the tests to PHP 7.3+ and thus the change of your PR. After this, this repository is going into archive mode.
There are too many thing I'd need to update, cleanup and basically rewrite to be still useful, breaking all BC.


using pnpm and metroResolverSymlink gets unable to resolve module interopRequireDefault during server

What happened?
Getting this error now:
> react-native rnx-start "clean"

                        #######
                   ################
                #########     #########
            #########             ##########
        #########        ######        #########
       ##########################################
      #####      #####################       #####
      #####          ##############          #####
      #####    ###       ######       ###    #####
      #####    #######            #######    #####
      #####    ###########    ###########    #####
      #####    ##########################    #####
      #####    ##########################    #####
      #####      ######################     ######
       ######        #############        #######
         #########        ####       #########
              #########          #########
                  ######### #########
                       #########

                    Welcome to Metro!
              Fast - Scalable - Integrated

To reload the app press 'r'
To open developer menu press 'd'
To quit, press 'control-c'
 BUNDLE  src/index.js

error: Error: Unable to resolve module @babel/runtime/helpers/interopRequireDefault from D:\OneDrive\Code\repos\RNSomething\MyApp\src\index.js: @babel/runtime/helpers/interopRequireDefault could not be found within the project or in these directories:
  D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules
  D:\OneDrive\Code\repos\RNSomething\node_modules

If you are sure the module exists, try these steps:
 1. Clear watchman watches: watchman watch-del-all
 2. Delete node_modules and run yarn install
 3. Reset Metro's cache: yarn start --reset-cache
 4. Remove the cache: rm -rf /tmp/metro-*
> 1 | /**
  2 |  * @format
  3 |  */
  4 |
    at ModuleResolver.resolveDependency (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\node-haste\DependencyGraph\ModuleResolution.js:129:15)
    at DependencyGraph.resolveDependency (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\node-haste\DependencyGraph.js:288:43)
    at Object.resolve (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\lib\transformHelpers.js:129:24)
    at resolve (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\DeltaBundler\traverseDependencies.js:396:33)
    at D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\DeltaBundler\traverseDependencies.js:412:26
    at Array.reduce (<anonymous>)
    at resolveDependencies (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\DeltaBundler\traverseDependencies.js:411:33)
    at processModule (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\DeltaBundler\traverseDependencies.js:140:31)
    at async traverseDependenciesForSingleFile (D:\OneDrive\Code\repos\RNSomething\MyApp\node_modules\.pnpm\metro@0.66.2\node_modules\metro\src\DeltaBundler\traverseDependencies.js:13

The file is where I'd expect it to be, so I'm not sure what is wrong now.
I tried installing watchman at one point and started getting an even weirder error, so I went to check the RN docs for setting up your environment, and the windows setup does not involve installing watchman. So uninstalled that and went back to this error.
Most of the sample repo has been copied right out of the test app in this repo.
Is there some config for the symlink resolver that would fix this? I poked around a bit at the files in the stack trace, and everything seems fine up until the point where it tries resolving the module path/name and fails, even though it's looking in the correct directory.
I'm getting a different error when I try bundling instead, so maybe the real error is just being obfuscated.
Affected Package
@rnx-kit/metro-resolver-symlinks
Version
0.1.3
Which platforms are you seeing this issue on?

[X] Android
[ ] iOS
[ ] macOS
[ ] Windows

System Information
➜  pnpx react-native info
info Fetching system and libraries information...
System:
    OS: Windows 10 10.0.19043
    CPU: (12) x64 Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
    Memory: 13.00 GB / 31.95 GB
  Binaries:
    Node: 16.8.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.11 - C:\Program Files\nodejs\yarn.CMD
    npm: 7.21.0 - C:\Program Files\nodejs\npm.CMD
    Watchman: Not Found
  SDKs:
    Android SDK:
      API Levels: 30
      Build Tools: 30.0.2, 31.0.0
      System Images: android-31 | Google APIs Intel x86 Atom_64
      Android NDK: Not Found
    Windows SDK: Not Found
  IDEs:
    Android Studio: Not Found
    Visual Studio: 16.9.31205.134 (Visual Studio Professional 2019)
  Languages:
    Java: javac 15
  npmPackages:
    @react-native-community/cli: ^6.0.0-0 => 6.0.0
    react: 17.0.2 => 17.0.2
    react-native: ^0.65.0-0 => 0.65.1
    react-native-windows: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

Steps to Reproduce
Repro repo
Steps:
npm install -g pnpm
git clone https://github.com/icesuicune/rnxSymlinkErrorRepro.git
cd rnxSymlinkErrorRepro/MyApp
pnpm install
pnpm run android

Don't own a Mac that doesn't belong to work so haven't tried running ios.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Can you try adding your app folder to watchFolders:
diff --git a/MyApp/metro.config.js b/MyApp/metro.config.js
index cc134d4..d8929d7 100644
--- a/MyApp/metro.config.js
+++ b/MyApp/metro.config.js
@@ -7,4 +7,7 @@ module.exports = makeMetroConfig({
   resolver: {
     resolveRequest: MetroSymlinksResolver(),
   },
+  watchFolders: [
+    __dirname
+  ]
 });

Metro only watches the path you specified at projectRoot. Since it's no longer the package root, you need to add it manually to watchFolders.

I had had tried a few different project roots but wasn't seeing any changes. That said, I seem to have caused this myself by overwriting my own config.  I think I had the project root set in three places.
In the package.json scripts I had
    "start": "react-native rnx-start --project-root src"
...
 "server": {
      "entryPath": "src/index.js",
      "detectDuplicateDependencies": {
        "ignoredModules": [
          "react-is"
        ]
      }

but I was only modifying the metro config project root value. I'm assuming the command line argument was overriding my root in metro.config. And entry path is only a valid property for the bundle config, and not the server? Or maybe not? I don't suppose there's a json schema definition or vscode extension that will help me out with the valid values for the rnx-kit section in package.json?
My app now bundles to 99% and then crashes violently with no error, so problem solved? Thanks for the help.

There's some documentation here: https://github.com/microsoft/rnx-kit/tree/main/packages/cli#readme
The gist is that you should provide most things in the config, and only use command line args when you need to override something. It's a good idea to have intellisense though.

My app now bundles to 99% and then crashes violently with no error

I was unable to repro this locally. Do you have a stack trace? Did you try using Node 14?

I ended up debugging it in android studio, and it seemed to be some odd initialization error. Supposedly some issue with the emulator configuration itself. I abandoned looking at it for the time being since I had a device on hand to test with.


freezing orbitals in FermionOperator

I have a FermionOperator, and I'd like to make the assumption that some of the orbitals are always empty and others are always full. (As I understand it, this is similar to the active space functionality in MolecularData.) My question is, what's the best way to do this?
What I'm thinking of doing is implementing two methods for FermionOperator, assume_occupied(index) and assume_unoccupied(index), that will remove the ladder operators that act on a given orbital and multiply the coefficients for the corresponding terms by 0, +1, or -1 as appropriate. I'd also need to implement a prune() method for SymbolicOperator that determines which indices don't appear in any terms and then renumbers the indices accordingly.
The idea is that
FermionOperator('1.0 [4^ 5 6]').assume_occupied(5).prune()
would yield -1.0 [0^ 1].
Does this seem like a reasonable way to go?

Ok, I just realized that it's a bit more complicated than I originally thought and the above example isn't right. I'll have to think about this some more...

I won't speak to its perfect accuracy, but I attach the following note I used a long time ago for deriving some parts of the active space routine in OpenFermion.  You may find the general methodology useful in deriving what you want to do:
ActiveSpaceHamiltonian.pdf

Ok thanks!


[QUERY] Which (ip)addresses to whitelist for ClientSecretCredential (and ConfidentialClientApplicationBuilder)

Library name and version
Azure.Identity 1.8.0
Query/Question
Within a big company we are using the Azure.Identity and Microsoft Identity classes to authenticate towards Azure using a client secret. Company policy restricts outgoing network traffic (using a firewall) on the machines that use these libraries. We would like to know which internet addresses the library (tries) to access to perform the authentication so that these (ip) addresses can be whitelisted.
Currently we are using these two calls the Azure Identity and Microsoft Identity library which result in connection issues due to firewall restrictions.
Azure Identity
 new ClientSecretCredential(tenantId, clientId, clientSecret, new TokenCredentialOptions());
Microsoft Identity
string authority = $"https://login.microsoftonline.com/{tenantId}/";
IConfidentialClientApplication app = ConfidentialClientApplicationBuilder
  .Create(clientId)
  .WithClientSecret(clientSecret)
  .WithAuthority(authority)
  .Build();

AuthenticationResult result = await app.AcquireTokenForClient(scopes).ExecuteAsync();

Environment
Windows Server 2019
.NET Framework 4.7.2

//cc: @christothes

Thank you for your feedback.  Tagging and routing to the team member best able to assist.  Please expect delayed responses due to the US holidays.

Hi @rogersberga - I'm not sure that there are any IP addresses documented for this, or that you could rely on any specific IP addresses remaining the same over time. My recommendation would be to trace the traffic in your environment(s) and build a list of IP ranges that way.


fix: Slack Notification aren't working

Close #132

The secrets context should take care of this. I've deployed some new fixes this morning which should fix notifications, so this is no longer needed. 😄


Fix Dart interop tests by upgrading to Dart 2

Fixes #15916
grpc-dart was failing to build because Dart version 1.x was installed in the Docker container.

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

Interop tests are passing

[microbenchmarks] No significant performance differences

Merging to fix the breakage.


I couldn't get "regulates" ancestors terms

Hello, I tried to get all ancestors terms following the "parents_and_ancestors.ipynb" notebook, but what I could get was only "is_a" ancestors terms.
Here is the screenshot from my jupyter notebook.

And here is the ancestor chart.

Could you help me with this problem?

@10myl200
Weird. I was able to get GO:008152.

Thank you for your quick reply! :)

I reinstalled goatools using conda, but I still got the same problem.

So, I downloaded up-to-date go-basic.obo files from https://geneontology.org/docs/download-ontology/

It seems like there was a problem with the obo file, somehow.

@10myl200
That's thorough investigation. Thanks.
If you still have the old obo, feel free to send it over and I'll inspect.
There could be a bug or a hidden filtering somewhere.

go-basic.obo.gz
That would be nice! Here is the old obo file.

@10myl200
Thanks. When I repeat the code on the file you sent it seemed ok. I'm not sure how to reproduce the issue. Or perhaps it is flaky? I'm not sure.
Let me know again if you continue to have issues with regulates which means it's flaky and I shall dig again.

ok. then we know it's flaky and certainly a bug.
I'll work on it soon. thanks


[Attention] Add out dtype for QK matmul

This PR adds an argument ~attn_weights_out_dtype~ qk_dtype for attention.py, essentially making matmul(q, k, out_dtype=qk_dtype).
The motivation is that some models (e.g. phi), when operating under float16, can observe INF in Q@K. Thus, we accumulate in float32 to avoid this.

In FlashInfer, there’s a parameter called “fp16_qk”, which we could tweak accordingly as well 😆

Ahh thanks for the pointer, I'll try and test a bit

also cc @jinhongyii @spectrometerHBH @MasterJH5574 , we likely will also need to pay attention to this in TIR KVCache kernels

Updated fp16_qk for FlashInfer; tested on llama


Scrolling Locked Misalignment - VSCode Insiders 1.88.0

Type: Bug
There is an aligning issue in the recently released Scrolling Locked functionality in VSCode Insiders.
When Scrolling Locked line-by-line alignment is enabled, when opening the search and/or search and replace box, the lines are misaligned.
The expected behavior is that the search and/or search and replace box, or no other box open in the editor, interferes with the text alignment.
 Opening the search widget causes some space to be reserved at the top of the editor pane for the widget. This causes the pane to scroll (can't be seen) and hence triggers scrolling on all other panes. I've got a fix for this

@benibenj Hi,
Testing VSCode Insiders - d994aede3529..., I noticed that the resolution (#208893) generated a new bug.
Page scrolling and sync broke when opening the search widget in both editors (2 search widgets). It's only scrolling up (even though you're scrolling down), and with a smooth scroll in mouse, the editor scrolls up thousands of lines.
Also when using the "Go To Row/Column" command (Ctrl + G), with the search widget open in both editors, the command does not go to the informed line.
It seems that the bug occurs when opening the second search widget, in any of the editors, as it misaligns the lines soon when opening the second widget. The problem lies in this.
When only have one search widget open, everything works fine.
Again I made a video to demonstrate it.
Video:
https://github.com/microsoft/vscode/assets/14878542/45a83ec3-0d28-4634-b08a-4c1c0fdbfbb9

Information about what I did in the video
I am using line number 40,000 as a reference.
When having the search widget open in just one of the editors, any of them, syncing and scrolling works fine. Go To Row/Column too.

When opening the second search widget in any of the editors, the bug occurs. Breaks sync and scrolling. Note that it only scrolls up, and skips thousands of lines with a smooth scroll (it was supposed to skip a few lines).
It also breaks the Go To Row/Column, it does not go to the informed line.

Note: This time I'm testing on macOS (Sonoma 14.4.1), and the video was made on it. My VSCode is in Brazilian Portuguese.
Do I need to open a new issue to report this bug, or can I report it right here?
Thank you for correcting these errors, it is an excellent feature.

@viniciusmi00 I also noticed this bug. I created a fix for it but it's not yet in insiders

@viniciusmi00 I also noticed this bug. I created a fix for it but it's not yet in insiders

Just to add, some (many) times when using horizontal page scrolling with Scrolling Locked activated, VSCode is freezing and have to restart it or wait. When I wait, I am thrown to line 1 of the editors.
If you can, take advantage and do some tests in this sense on large files (if necessary, I can provide one that I'm working on, it's public).
If it is not possible to fix this for now, perhaps it would be better to return it as it was, because although it misaligned when opening one of the search widgets, all you had to do was open both and it would be aligned again, and there was no performance problem with horizontal scrolling and these other problems I reported.


codec is deprecated and its use causes a error with GCC 6

Hi!
To get the compile working I will add -Wno-error=deprecated-declarations.
See: https://pmbs.links2linux.de/build/Multimedia/openSUSE_Tumbleweed/x86_64/groove/_log

Looks like you're building 4.3.0, which is reasonable, but the newest code is in master branch, not released yet, and depends on updated ffmpeg instead of libav. So I believe this error is fixed in master.

@andrewrk I still have this issue on master. Running macos Sierra, with ffmpeg version 3.3.2.
$ git clone https://github.com/andrewrk/libgroove
$ cd libgroove
$ mkdir build
$ cd build
$ cmake ..
Configuring libgroove version 5.0.0

Installation Summary
--------------------
* Install Directory            : /usr/local
* Build Type                   : Debug
* Build static libs            : ON
* Build example programs       : ON

System Dependencies
-------------------
* threads                      : OK
* libsoundio                   : OK
* ebur128                      : OK
* chromaprint                  : OK
* libavformat                  : OK
* libavcodec                   : OK
* libavfilter                  : OK
* libavutil                    : OK

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/ryan/libgroove/build
$ make
[  2%] Building C object CMakeFiles/libgroove_shared.dir/src/file.c.o
/Users/ryan/w/1egoman/termtrack/libgroove/src/file.c:176:42: error: 'codec' is deprecated [-Werror,-Wdeprecated-declarations]
    AVCodecContext *avctx = f->audio_st->codec;
                                         ^
/usr/local/include/libavformat/avformat.h:893:21: note: 'codec' has been explicitly marked deprecated here
    AVCodecContext *codec;
                    ^
/Users/ryan/w/1egoman/termtrack/libgroove/src/file.c:235:72: error: 'codec' is deprecated [-Werror,-Wdeprecated-declarations]
        AVCodecContext *avctx = f->ic->streams[f->audio_stream_index]->codec;
                                                                       ^
/usr/local/include/libavformat/avformat.h:893:21: note: 'codec' has been explicitly marked deprecated here
    AVCodecContext *codec;
                    ^
/Users/ryan/w/1egoman/termtrack/libgroove/src/file.c:288:46: error: 'codec' is deprecated [-Werror,-Wdeprecated-declarations]
    AVCodecContext *codec_ctx = f->audio_st->codec;
                                             ^
/usr/local/include/libavformat/avformat.h:893:21: note: 'codec' has been explicitly marked deprecated here
    AVCodecContext *codec;
                    ^
/Users/ryan/w/1egoman/termtrack/libgroove/src/file.c:384:45: error: 'codec' is deprecated [-Werror,-Wdeprecated-declarations]
        AVCodecContext *icodec = in_stream->codec;
                                            ^
/usr/local/include/libavformat/avformat.h:893:21: note: 'codec' has been explicitly marked deprecated here
    AVCodecContext *codec;
                    ^
/Users/ryan/w/1egoman/termtrack/libgroove/src/file.c:385:46: error: 'codec' is deprecated [-Werror,-Wdeprecated-declarations]
        AVCodecContext *ocodec = out_stream->codec;
                                             ^
/usr/local/include/libavformat/avformat.h:893:21: note: 'codec' has been explicitly marked deprecated here
    AVCodecContext *codec;
                    ^
5 errors generated.
make[2]: *** [CMakeFiles/libgroove_shared.dir/src/file.c.o] Error 1
make[1]: *** [CMakeFiles/libgroove_shared.dir/all] Error 2
make: *** [all] Error 2


Dbz 4655 incremental snapshots ga doc updates

DBZ-4655
This change updates connector files, shared files, and signaling files to:

Remove incubating/TP designations for signaling and ad hoc/incremental snapshots features.
Remove filter condition that prevents MongoDB incremental snapshots topic from being exposed downstream.

@jpechane:

TBD: Should incremental snapshots be designated as TP for MongoDB?
Should the MongoDB connector properties table include entries for:
-- incremental.snapshot.chunk.size -- Modify the description to refer to max # documents vs. rows?
-- signal.data.collection

Tested in local Antora build.
I based the branch for this change on main.
Please backport the change to 1.9.

Regarding the incremental.snapshot.chunk.size and signal.data.collection properties that are omitted from properties table, can you provide some input on whether these should be considered Advanced properties?
Would we consider these properties to be "required unless a default value is available"? Or to have "good defaults that will work in most situations and therefore rarely need to be specified in the connector's configuration"? The statements aren't exactly mutually exclusive. :)
FWIW, for the other connectors, the properties are listed inconsistently in the tables of Required and Advanced properties. Ideally, to make it easier for users who might work with multiple connectors, we provide the same experience across the different connectors.

@roldanbob Hi, thank you I think this is fine. I've just left one comment related to the MongoDB config options for incremental snapshotting. I am sorry for the confusion this topic took.

Please backport the change to 1.9.

There's no need to do that at this point in time. Only once the "main" branch represents 2.0 (i.e. after the upstream 1.9 Final release), this will be required.


Can't upload 2345.3.0 as a custom image on Digital Ocean

When trying to upload Flatcar 2345.3.0 to Digital Ocean as a custom image, the upload stays at "pending" for a while, then an error is shown saying the image couldn't be decompressed.
Decompressing the image locally seems to work. Uploading 2303.4.0 to Digital Ocean also works.
Repro Case
Scripts
cat ./upload_image

#!/bin/bash

set -euo pipefail

id=$(curl -s -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
    -d "{
        \"name\": \"flatcar_$FLATCAR_VERSION\",
        \"url\": \"https://stable.release.flatcar-linux.net/amd64-usr/$FLATCAR_VERSION/flatcar_production_digitalocean_image.bin.bz2\",
        \"distribution\": \"Unknown\",
        \"region\": \"fra1\"}" \
    "https://api.digitalocean.com/v2/images" | jq '.image.id')

echo "Upload started. Image ID: $id"

cat ./check_image

#!/bin/bash

set -euo pipefail

curl -s -X GET \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
    "https://api.digitalocean.com/v2/images/$IMAGE_ID" | jq '.'

Bad Case
export DIGITALOCEAN_TOKEN=xxxxxxxx
export FLATCAR_VERSION=2345.3.0
./upload_image

Upload started. Image ID: 60173630

export IMAGE_ID=60173630
./check_image

{
  "image": {
    "id": 60173630,
    "name": "flatcar_2345.3.0",
    "distribution": "Unknown",
    "slug": null,
    "public": false,
    "regions": [],
    "created_at": "2020-03-05T14:43:36Z",
    "min_disk_size": null,
    "type": "custom",
    "size_gigabytes": null,
    "description": "",
    "tags": [],
    "status": "pending",
    "error_message": ""
  }
}

After a while...
./check_image

{
  "image": {
    "id": 60173630,
    "name": "flatcar_2345.3.0",
    "distribution": "Unknown",
    "slug": null,
    "public": false,
    "regions": [],
    "created_at": "2020-03-05T14:43:36Z",
    "min_disk_size": null,
    "type": "custom",
    "size_gigabytes": null,
    "description": "",
    "tags": [],
    "status": "deleted",
    "error_message": "We had a problem decompressing your file. This typically happens if the archive is corrupt or contains multiple files. Please try decompressing the file locally and uploading the uncompressed image, or recompressing using gzip or bzip2."
  }
}

Good Case
Same as the bad case with the following change:
export FLATCAR_VERSION=2303.4.0

This image was uploaded and tested on DO during the release process, and I can also decompress and run in in QEMU, so not sure what suddenly changed.

Maybe we hit a timeout in DO because the bzip2 file takes ages to decompress.

I recompressed it with gzip -9 (it was just ~20 MB larger) and uploaded it manually without problems.

Another idea may be to approach them to use lbzip2/lbunzip2 for decompression.

Maybe DO fixed something on their backend, or our hosting has changed the duration of the transfer relative to the decompression - but I can't reproduce this today. I tried with the latest images (3033.2.1) and with (2345.3.0), in various regions (sfo3/fra1) and have had only success.
I'm adding a PR to docs (https://github.com/flatcar-linux/flatcar-docs/pull/204) to mention the possibility that it might fail and a possible solution with gzipping.


Unable to set extra http headers for basic authorization

I'm pretty sure I'm misusing the library: I'm trying to achieve a basic authorization with chromedp but it seems my extra headers are ignored.
My code:
err = c.Run(ctx, cdp.Tasks{
    cdp.ActionFunc(func(ctx context.Context, h cdpt.Handler) error {
         x := base64.StdEncoding.EncodeToString([]byte("login:password"))
	 headers := network.Headers{"Authorization": "Basic " + x}
	 return network.SetExtraHTTPHeaders(headers).Do(ctx, h)
    }),
    cdp.Navigate(url),
    // -> chrome displays me the authentication popup whereas I set the headers.
})

What's wrong with this code?
I took inspiration from the cookie example https://github.com/knq/chromedp/blob/master/examples/cookie/main.go

I'm not sure why you wouldn't just use the http://user:pass@host/ form for basic authentication?

There's a chance that you can't set that header via the Chrome Debugging Protocol, and that is probably entirely by design. I don't have time to look through the Chrome source tree, but I imagine that's the problem. Regardless, you should be able to specify this in the URL you're passing to Navigate, and besides that would be the more "correct" implementation.

Thank you for the answer, I'll try.
ChromeDP is awesome, great work!


Update chainlist

[x] https://github.com/Koniverse/SubWallet-ChainList/issues/233
[x] Update HydraDx information:
HydraDX -> Hydration
HydraDX Rococo -> Hydration Rococo
Logo chain + token HDX: https://www.figma.com/design/z9s6aLTLoUdjMwoXLyWcPz/01.SubWallet-Project-Logo?node-id=28-180&t=ybKApXhq2FC3p04r-4

[ ] https://www.notion.so/subwallet/2024-06-07-Chainlist-0-2-67-a0623048977948199942ff529293689a

Version test: http://<IP_ADDRESS>:8900/s/W6479KBaimtj9oR

[ ] Update some information for Hydration network

[ ] URL explorer
[ ] Get history

[ ] Re-check multi asset of WBTC (Bob)

Version test: http://<IP_ADDRESS>:8900/s/HwSrjC2xpNMR3zX
Validation passed in cases:

Add support BORK token

Show/hide token
Show balance
View explorer
Transfer (approve step)

Update HydraDX information:

Network Name and Network logo and Network details
Token logo
Show/hide network/token
Detect and show balance
View on explorer
Get and show history
Transfer token
Swap

Update providers and networks info


SWI-Prolog 8.5.16 Windows regression: nul no longer understood by the open/3-4 predicates

The SWI-Prolog 8.5.16 Windows version have a regression that breaks key libraries and a large number of tests: nulis  no longer understood by the open/3-4 predicates. This bug is not present in the previous version (8.5.15). If you're using running Logtalk with SWI-Prolog on Windows, avoid upgrading to version 8.5.16 and look for a possible fix in the next versions.

Thanks.  I see.   We can't use the "\?" prefix to get support for long file paths for the Windows "special" files :cry:
P.s. Is it an option to use open_null_stream/1 (inherited from Quintus, no clue how widely used it is).

Should be ok with SWI-Prolog/swipl-devel@1b9ec8d1cdcdafad37514e0c2e826fc60a26345d

P.s. Is it an option to use open_null_stream/1 (inherited from Quintus, no clue how widely used it is)?

Possible, yes. But would require use of conditional compilation directives in multiple places.
Thanks for the quick fix. I will try it when the next nightly build for the Windows binary installer become available.
Note that you can run Logtalk distribution tests automated on Windows using the provided scripts/logtalk_tester.ps1 PowerShell script. That would have readily flagged this regression.

Fix confirmed using the swipl-w64-2022-09-08.exe nightly build.


Madu/ensure sql idempotent

Description

References

#0000

Codecov Report

Merging #299 (5c12e79) into main (21978a1) will decrease coverage by 23.80%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##             main     #299       +/-   ##
===========================================
- Coverage   83.07%   59.28%   -23.80%
===========================================
  Files         456      381       -75
  Lines       10418     7893     -2525
===========================================
- Hits         8655     4679     -3976
- Misses       1438     3021     +1583
+ Partials      325      193      -132

Impacted Files
Coverage Δ

...ivedMessages/ArchivedMessageGetDocumentListener.cs
0.00% <ø> (ø)

.../ArchivedMessages/ArchivedMessageSearchListener.cs
0.00% <0.00%> (ø)

...rchivedMessages/GetArchivedMessageDocumentQuery.cs
0.00% <0.00%> (ø)

...cation/OutgoingMessages/Peek/PeekRequestHandler.cs
30.00% <0.00%> (-65.00%)
:arrow_down:

...rce/Application/SearchMessages/GetMessagesQuery.cs
0.00% <0.00%> (-12.50%)
:arrow_down:

source/Application/SearchMessages/MessageInfo.cs
0.00% <0.00%> (ø)

source/Application/SearchMessages/QueryBuilder.cs
0.00% <0.00%> (-97.37%)
:arrow_down:

source/Domain/ArchivedMessages/ArchivedMessage.cs
0.00% <0.00%> (-12.50%)
:arrow_down:

...ture/ArchivedMessages/ArchivedMessageRepository.cs
15.38% <0.00%> (-80.77%)
:arrow_down:

...regations/CalculationResultCompletedEventMapper.cs
3.15% <0.00%> (-54.74%)
:arrow_down:

... and 220 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Roboto-regular and bold contain redundant cmap_format_12 table?

When inspected using Google's TTX, the Roboto Regular and Bold fonts contain a cmap_format_12 table.
However, the content of this table is exactly the same as the cmap_format_4 table.
Furthermore, the presence of this table seems to cause problems with the Win32 API CreateFontPackage

They're almost exactly the same, but actually the format 12 table contains two glyphs outside the Unicode BMP which can't be included in a format 4 table: uni1F16A and uni1F16B.
I'd be surprised if that API doesn't support format 12 tables. But one solution might be to try subsetting the TTFs using nototools.

@Ziriax what are the problems you see with Win32 API CreateFontPackage?

That is strange, when comparing these tables with TTX, I did not see these glyphs in the table. Also, the italic faces do not have these tables. I will double check tomorrow.
When subsetting the font using that API call, the subset does not render at all in Google Chrome, but it does work in IE11 or DirectWrite. The subsetted italic faces work fine.
I will check out nonotools, thanks for the link!


[#239] Add schema and template for 'state_timeout_in_seconds'

and this was already in the readme?

Yep, you can find it in main: https://github.com/irods/irods_client_http_api/blob/cfb0c76e37bac83fa4a5f344e089e06ab251752b/README.md?plain=1#L269-L271
Blame says it was in 0.1.0.

Think this is the specific commit it was added in the readme: https://github.com/irods/irods_client_http_api/commit/f54fc3c56ff9f250519b30a3e650c65273b5bcfe

perfect.

Pounded.


Dictionary/List serialization issue/question

I would like to reach out for your guidance for couple of questions I have with Dictionary/List serialization.
Here's a sample XML serialized out of ExtendedXmlSerializer:
https://codebeautify.org/alleditor/y231598d5
And heres the object:
var xmlTestingObject = new List<Subject> {      new Subject     {         Count = 1,         Message = "Message1",         Destinations = new Dictionary<string, TestType>         {             {"Dest1", new TestType{ ABC = "abc1", DEF="def2"} },             {"Dest11", new TestType{ ABC = "abc11", DEF="def22"} }         }     },     new Subject     {         Count = 2,         Message = "Message2",         Destinations = new Dictionary<string, TestType>         {             {"Dest2", new TestType{ ABC = "abc21", DEF="def22"} },             {"Dest22", new TestType{ ABC = "abc21", DEF="def22"} }         }     } };

How do I get rid of :
<List  xmlns:exs="https://extendedxmlserializer.github.io/v2" exs:arguments="Subject">  <Capacity>4</Capacity>

How do I change the element name of "item" in a dictionary (also , is it possible to change the element name for "key" and "value" for a dictionary serialization.

Thanks and appreciate your inputs.

Hi @c0deExplorer thank you for writing in.  For my edification can you provide the container configuration code you used to generate the XML in question?  This will assist me in knowing how to proceed.

Thanks much @Mike-E-angelo . Here's the config I am trying out as of now:
`        IExtendedXmlSerializer serializerTest = new ConfigurationContainer()
.EnableImplicitTyping(typeof(Dictionary<string, TestType>),
typeof(List), typeof(Subject))
                                              .Create();`

Alright @c0deExplorer it took some time here to brush off the dust but I am happy to say that what you are after is possible.  Please check out this test/configuration:
https://github.com/ExtendedXmlSerializer/home/blob/a4f8a3509ed246bac0ab37c54ecc9e357055d14a/test/ExtendedXmlSerializer.Tests.ReportedIssues/Issue604Tests.cs
You will find that it emits the following:
<?xml version="1.0" encoding="utf-8"?>
<Issue604Tests-Subject>
	<Store>
		<Entry>
			<Key>Hello</Key>
			<Value>true</Value>
		</Entry>
		<Entry>
			<Key>World</Key>
			<Value>false</Value>
		</Entry>
	</Store>
</Issue604Tests-Subject>

So the extra attributes are removed and Item has been renamed to Entry.
Please let me know if you have any further questions and I will assist,

Thank you so much @Mike-E-angelo. Never knew about InspectedPropertyTypes.  It did do the trick!! Great work! I'll work on it and get back if I see anything different.  Again, Thanks a ton for your wonderful support and really appreciate your time spend to investigate this.

Thank you for the kind words.  If you ever need any further assistance please open a new issue and we will take a look into it.  Closing this for now.


The two leaderboards are the same as each other.

I cannot notice any difference between them...

This is due to a hack for computing results for the original HELM paper.

This is due to a hack for computing results for the original HELM paper.

Any plan to reformat it?

This is not planned.

This is not planned.

Thanks


Low classification road names are shown far too early

https://dev.streetspirit.xyz/#12.15/49.05457/8.33771

Not sure what classification these are, but they shouldn't be shown this early

Fixed in 2caf99d055fe1ae99ccc8ca69bcf4fb3eaa59053


Links keep getting selected even when I select a different link

Description:

When I click on a different link, the previously selected link should get unselected.
Steps to reproduce:
Affected Versions:
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

This is no longer reproducible, hence closing


cask github-desktop is outdated

now cask github-desktop is v216
new cask github-desktop is v217

November 10th 2015 v216
March 2nd 2016 v217

It has been updated to 217 12 days ago. Your setup is the one outdated. You need to brew update,


Fix requirements versions?

While trying to install your requirements-dev.txt into my virtual env, I got missing version errors for the following packages:
ERROR: Could not find a version that satisfies the requirement sip==4.19.13
ERROR: Could not find a version that satisfies the requirement mkl-random==1.0.2
ERROR: Could not find a version that satisfies the requirement mkl-fft==1.0.12
ERROR: Could not find a version that satisfies the requirement docutils==0.15.1

I've updated these packages to the lastest version listed on PyPi.
https://pypi.org/project/SIP/#history
https://pypi.org/project/mkl-random/#history
https://pypi.org/project/mkl-fft/#history
https://pypi.org/project/docutils/#history

@alxwrd good catch! I tested it myself and realized it is in fact broken on a fresh environment when using pip. This issue probably arose because I use conda.


DOC: Add link to slack space

As discussed in today's community meeting.
A few other details we discussed are already present at the Get Help page (stack overflow, for example) so I didn't add them here.

Thanks @melissawm! As a separate aside maybe we should have a community bullet point in the readme of the main repo linking to this page to improve visibility?

Thanks for the update. Let's get this in now.


wrap libtorch rank and shape routines

the tensor derived type now supports two methods:

get_rank
get_shape

These methods return the shape and rank of the tensor

all reads correctly to me.

Thanks @jatkinson1000 . Spot on.

I plan to add unit tests in the near future with pFUnit.
I did check locally by manually modifying the integration test but I will put a proper test in for the autograd work


Firing a click at co-ordinates...

Hello,
I'm trying to click on a button and I have this message :
Firing a click at co-ordinates [1043.5, 288] failed. Poltergeist detected another element with CSS selector '' at this position. It may be overlapping the element you are trying to interact with. If you don't care about overlapping elements, try using node.trigger('click').

If I look by css and I do trigger('click'), I have this message :
Unable to find css "provider_service_formula_id_3"

But when I'm trying with capybara-webkit or with chrome it works. What can I do?
Thanks!

This still seems to be a problem -- hidden elements don't register events, but Poltergeist still seems to be throwing a hidden-element-catcher error (see here). Hidden elements don't seem to catch events, so this seems like there should be a test of the elements visibility or something. Right?


Support semanticolor

Issue or RFC Endorsed by Atom's Maintainers
#19151
Description of the Change
Pass node text to grammar for leaf parser nodes.
Alternate Designs
This is the only way for grammars to use the text to determine styling.
Possible Drawbacks
Performance penalty of unmarshalling from native text buffer is mitigated by only passing text for leaf nodes.
Verification Process
I have been running with this locally for over a week with no issues.
Release Notes
Pass syntax node text to grammar for leaf parser nodes, to support grammars that determine styling based on token values.

please elaborate on performance issues, seems brief. mention no of ops, O(n) or O(log n), etc.

Re: performance penalty, I am referring to @maxbrunsfeld's comment here:

The .nodeText accessor on the TreeCursor has some performance cost. It needs to read a string from the native text buffer, and for internal syntax nodes (like a function declaration) it may be a very big string. And that line of code executes many times per syntax-highlighted line. So I'd like to avoid that performance cost if possible.

This solution avoids calling the .nodeText accessor on most long strings, by only calling it on syntax nodes with no children.

Hi to all Atom comunity, and all mantainers, if you can, you have a time and possibility, please promote this PR to positive decision to merge. Today more and more plugin developers are living for VSCode, but there are several gems that make Atom still the best, and great "SemanticColor" extension is one of them

Is there anything needed to help to get this through?
For a temporary fix, semanticolor still works in separately downloadable atom v1.42

@sharedprophet could you please make sure the CI is green, currently it's failing due to lint errors.
You can check the linting, by navigating to your repo and running script/lint from command line.

Looks like some unrelated tests timed out on Win64...?

@sharedprophet re-running the ci

Win64 tests seem flaky... something is up with the promises on there. It is failing on a PR to fix typos and that doesn't seem right.

Hello, as I see PR is ready and reviewed, but CI failed for non related stuff... What is the next step, maybe rerun CI again, and cross the fingers ? )

Hi, strange thing ... but oldier builds than that run is present

New CI Run https://github.visualstudio.com/Atom/_build/results?buildId=77881&view=results

Hello,
in the link, provided by @lkashef all state are green,

why CI state in this thread still shows some errors ?!

Thanks @ocoka for the heads up.
When the build is expired/deleted from Azure Devops, and a new manual run (instead of pushing a dummy commit) against the PR doesn't get picked up by the Github, that's why I provided the link above.

Thanks @sharedprophet for the contribution, I will go ahead and merge this PR

Is there a place I can find more info on this?  (Both accessing text nodes inside grammars and semanticolor)

@jeff-hykin the diff is pretty small, and there is a new test that presumably demonstrates the functionality. I hope that can shed some light on the changes.

@jeff-hykin inside a grammar, the change will sometimes provide the node text as a second parameter to the idForScope function. You can only take advantage of this if you are creating a grammar with code rather than a normal TreeSitter or TextMate grammar file, though.

@sharedprophet Ah thanks for the explanation that's what I was curious about 👍
And thanks @DeeDeeG


Return a sorted copy from array.sort()

Fixes #249
Note: sort() currently returns undefined when the observed property is undefined, while Ember's computed.sort() returns an empty array. Not sure if this difference is intended, I found it a bit surprising.
At the moment, it seems array/-utils.normalizeArray will accept a static default value, but I think we'd want to use a factory function that generates default values (new Array objects, in this case). Changing the normalizeArray API seems a bit outside the scope of this PR, so I haven't actually touched this, but have added a test for the current behavior (returning undefined) and a skipped test for the (in my mind) expected behavior (returning an empty Array).
Happy to open another issue/PR to address this if it's something you'd like changed.

ok, moved the tests from unit into the existing integration. If this looks good, happy to rebase out the extra commits

Looks great. Feel free to squash, or I can do it via github UI.

Cool, killed the redundant test commits, should be ready to merge (squashing further if you want) afaics

Great work. Released v0.31.3.

Sweet, thanks!


Training and Predicting using a single model type

Hi there,

i want to train an LightGBMClassifier model type only, so i included:
excluded_model_types=['NN', 'RF', 'XT', 'KNN', 'CAT', 'LR', 'custom']

The issue is that training includes LightGBMClassifierXT and weighted_ensemble, How to disable them also?

For prediction, is there a way to set only the individual model type that i want to use.
Because when i am trying to predict, it requires learner.pkl to be present, and also requires all the models trained to be present.

@Innixma @jwmueller @zhanghang1989

Answered here: https://github.com/jwmueller/KDD20-tutorial/issues/9

You can get rid of any other models by using predictor.delete_models(models_to_keep=['your_model_name'], dry_run=False) after training.
You can train just a LightGBM (+ weighted ensemble which can be deleted after) via task,fit(..., hyperparameters={'GBM': {}})


⚠ ClamAV has degraded performance

In 5d50024, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 197 ms

Resolved: ClamAV performance has improved in 4bcd633.