Memory is not released after hashing, but only when JVM exits

I noticed that a customer's application increased in memory usage every time some users logged in and the memory usage would not go down again even when everything was quiet. I profiled using async-profiler and found that the memory was being allocated by Scrypt. I managed to reduce it to the following reproducer:
public static void main(String[] args) throws InterruptedException {
    int NUM_THREADS = 10;
    int HASHES_PER_THREAD = 10;
    for (int i = 0; i < NUM_THREADS; i++) {
        new Thread(() -> {
            for (int j = 0; j < HASHES_PER_THREAD; j++) {
                //High memory usage during this loop is understandable
                //since we're actively using the native buffers
                SCryptUtil.scrypt("pwd", 16384, 8, 1);
            }
            //Once we're done, memory should be freed
            //However, it stays allocated until the JVM exists
            try {
                Thread.sleep(30000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }).start();
    }
    Thread.sleep(30000);
}

Just run this and watch the process' memory usage. It spikes up when the threads invoke scrypt and never goes down again. It gets worse the more threads you have. The hashes per thread also seem to play a role, but only to a certain extent. If I set that parameter to 1, I get ~50MB memory usage. If I set it to 10, I get 500MB, but setting it to 100 does not further increase it.
I played around with the parameters a bit and if I change N, the problem goes away. Only 2^14 exhibits the problem, other powers of 2 let the memory usage spike up and then go down again. I'm afraid I don't know enough C to be of assistance in figuring out why :)
I ran the reproducer on Linux 5.4.0-72 if that helps.

I'd also be interested why N is called the "CPU cost parameter" when it affects memory just as much.


Parallel dispatch calls freeze when Lua resolver yields

Problem summary: If you dispatch two parallel HTTP calls, and both call into Lua for DNS resolution, and both yield to wait for the DNS response, only the second one gets a response. The first one waits forever.
Background and some observations of the behavior
This has been observed first with Kong Gateway, but it is also reproducible in this module's test suite (see test cases in PR #523).
Using two different hostnames allows us to see in the logs that the second DNS request is the one that gets a response.
Yielding affects the behavior:

if you issue a request that triggers two parallel calls for the same hostname, it hangs because the first DNS request yields and never resumes. If you Ctrl-C the request and try again, the overall request succeeds, because the hostname was cached from the successful call of the previous request, and both succeed.
if you try to do the same with  a request that triggers two parallel calls for the different hostnames, the first request gets the same behavior (it hangs because the first DNS request yields and never resumes), but the second request hangs again, because it will try the first hostname which never resolved, yields, then tries the second hostname which is cached, and succeeds, but the overall request is still waiting for the first hostname.

In short, it seems that triggering the Lua resolver a second time discards the pending DNS resolution coroutine.
Logs
Enabling "debug_mail" logs on the Kong Gateway showed some interesting bits which may provide some clues. Here's what I've observed:
First we create a new Lua thread to resolve gobolinux.org, all is normal so far:
2024/03/22 16:25:45 [debug] 90985#0: *2 lua creating new thread
2024/03/22 16:25:45 [debug] 90985#0: lua ref lua thread 00005A6A986C62C0 (ref 649)
2024/03/22 16:25:45 [debug] 90985#0: *2 code cache lookup (key='wasm_lua_resolver_chunkdd7007ea3c1caf68390921fc5ddb939c', ref=0)
2024/03/22 16:25:45 [debug] 90985#0: *2 code cache miss (key='wasm_lua_resolver_chunkdd7007ea3c1caf68390921fc5ddb939c', ref=0)
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm running lua thread (lctx: 00005A6A96928E80, L: 00005A6A95FFCD30, co: 00005A6A986C62C0)
2024/03/22 16:25:45 [debug] 90985#0: *2 lua reset ctx
2024/03/22 16:25:45 [debug] 90985#0: *2 http lua finalize threads
2024/03/22 16:25:45 [debug] 90985#0: *2 lua run thread, top:0 c:1
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:13: wasm lua resolver thread
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:20: wasm lua resolver using existing dns_client
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:42: wasm lua resolving "gobolinux.org"

This causes the DNS client to trigger the UDP message for DNS resolution, epoll fd 34. It does not get an immediate response, so it yields:
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket network address given directly
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket resolve retval handler
2024/03/22 16:25:45 [debug] 90985#0: *2 UDP socket 34
2024/03/22 16:25:45 [debug] 90985#0: *2 connect to <IP_ADDRESS>:53, fd:34 #646
2024/03/22 16:25:45 [debug] 90985#0: *2 epoll add event: fd:34 op:1 ev:80002001
2024/03/22 16:25:45 [debug] 90985#0: *2 add cleanup: 00005A6A964C8870
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket connect: 0
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] client.lua:737: lookup(): WILL QUERY gobolinux.org
2024/03/22 16:25:45 [debug] 90985#0: *2 send: fd:34 31 of 31
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket calling receive() method
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket read timeout: 2000
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket receive buffer size: 4096
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket read data: waiting: 0
2024/03/22 16:25:45 [debug] 90985#0: *2 recv: fd:34 -1 of 4096
2024/03/22 16:25:45 [debug] 90985#0: *2 recv() not ready (11: Resource temporarily unavailable)
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp recv returned -2
2024/03/22 16:25:45 [debug] 90985#0: *2 event timer add: 34: 2000:10471706
2024/03/22 16:25:45 [debug] 90985#0: *2 lua resume returned 1
2024/03/22 16:25:45 [debug] 90985#0: *2 lua thread yielded

(we then get this which I don't if it's at all related)
2024/03/22 16:25:45 [debug] 90985#0: posted event 00005A6A9868DF30
2024/03/22 16:25:45 [debug] 90985#0: *2 delete posted event 00005A6A9868DF30

what follows immediately is the next dispatch, which needs resolving example.com:
2024/03/22 16:25:45 [debug] 90985#0: *2 proxy_wasm http dispatch connecting...
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm tcp socket resolving: example.com
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm tcp socket using default resolver
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm tcp socket resolving...
2024/03/22 16:25:45 [debug] 90985#0: *2 lua creating new thread
2024/03/22 16:25:45 [debug] 90985#0: lua ref lua thread 00005A6A9678DC20 (ref 650)
2024/03/22 16:25:45 [debug] 90985#0: *2 code cache lookup (key='wasm_lua_resolver_chunkdd7007ea3c1caf68390921fc5ddb939c', ref=0)
2024/03/22 16:25:45 [debug] 90985#0: *2 code cache miss (key='wasm_lua_resolver_chunkdd7007ea3c1caf68390921fc5ddb939c', ref=0)
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm running lua thread (lctx: 00005A6A97EE2D90, L: 00005A6A95FFCD30, co: 00005A6A9678DC20)

but unlike the dispatch above where the three lines lua reset ctx, http lua finalize threads and lua run thread were next to each other, we get instead this:
2024/03/22 16:25:45 [debug] 90985#0: *2 lua reset ctx
2024/03/22 16:25:45 [debug] 90985#0: *2 http lua finalize threads
2024/03/22 16:25:45 [debug] 90985#0: *2 lua finalize socket
2024/03/22 16:25:45 [debug] 90985#0: *2 lua close socket connection
2024/03/22 16:25:45 [debug] 90985#0: *2 event timer del: 34: 10471706
2024/03/22 16:25:45 [debug] 90985#0: *2 reusable connection: 0
2024/03/22 16:25:45 [debug] 90985#0: *2 lua deleting light thread 00005A6A986C62C0 (ref 649)
2024/03/22 16:25:45 [debug] 90985#0: *2 lua caching unused lua thread 00005A6A986C62C0 (ref 649)
2024/03/22 16:25:45 [debug] 90985#0: *2 lua run thread, top:0 c:1

Looks like it deleted coroutine ending-62C0, which was the one from the gobolinux.org DNS resolution above!?
The query for example.com then proceeds is a similar fashion to the previous one... (even with the same fd:34)
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:13: wasm lua resolver thread
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:20: wasm lua resolver using existing dns_client
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:42: wasm lua resolving "example.com"
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] [string "wasm_lua_resolver_chunk"]:44: has individual_toip: function: 0x5a6a962ef8f0
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket network address given directly
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket resolve retval handler
2024/03/22 16:25:45 [debug] 90985#0: *2 UDP socket 34
2024/03/22 16:25:45 [debug] 90985#0: *2 connect to <IP_ADDRESS>:53, fd:34 #647
2024/03/22 16:25:45 [debug] 90985#0: *2 epoll add event: fd:34 op:1 ev:80002001
2024/03/22 16:25:45 [debug] 90985#0: *2 add cleanup: 00005A6A96281D30
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket connect: 0
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] client.lua:737: lookup(): WILL QUERY example.com
2024/03/22 16:25:45 [debug] 90985#0: *2 send: fd:34 29 of 29

...but then after a while we do get data coming from fd:34 and we get a query result for example.com.
2024/03/22 16:25:45 [debug] 90985#0: epoll timer: 837
2024/03/22 16:25:45 [debug] 90985#0: epoll: fd:34 ev:0001 d:00005A6A9888E191
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket handler for "/anything?", wev 0
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket read handler
2024/03/22 16:25:45 [debug] 90985#0: *2 event timer del: 34: 10471706
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket read data: waiting: 1
2024/03/22 16:25:45 [debug] 90985#0: *2 recv: fd:34 85 of 4096
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp recv returned 85
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket waking up the current request
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm rctx reused: 00005A6A964161C0 (r: 00005A6A97B8D430, main: 1)
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm wev handler "/anything?" - timeout: 0, ready: 1 (main: 1, count: 1, resp_finalized: 0, state: 2)
2024/03/22 16:25:45 [debug] 90985#0: *2 wasm resuming lua thread (lctx: 00005A6A97EE2D90, L: 00005A6A95FFCD30, co: 00005A6A9678DC20)
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp operation done, resuming lua thread
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket calling prepare retvals handler 00005A6A943F1684, u:00005A6A98CEEED0
2024/03/22 16:25:45 [debug] 90985#0: *2 lua udp socket receive return value handler
2024/03/22 16:25:45 [debug] 90985#0: *2 lua run thread, top:0 c:1
2024/03/22 16:25:45 [debug] 90985#0: *2 [lua] client.lua:739: lookup(): QUERY GOT RESULT

Interesting: running #523 on CI with Valgrind produced the following after the test cases timed out:
https://github.com/Kong/ngx_wasm_module/actions/runs/8438836263/job/23111996715
 ==46021== 556 (48 direct, 508 indirect) bytes in 1 blocks are definitely lost in loss record 146 of 173
==46021==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==46021==    by 0x1E464A: ngx_alloc (ngx_alloc.c:22)
==46021==    by 0x1ACD49: ngx_create_pool (ngx_palloc.c:21)
==46021==    by 0x3ACBA9: ngx_wasm_lua_thread_new (ngx_wasm_lua.c:128)
==46021==    by 0x3AE14A: ngx_wasm_lua_resolver_resolve (ngx_wasm_lua_resolver.c:145)
==46021==    by 0x391B02: ngx_wasm_socket_tcp_connect (ngx_wasm_socket_tcp.c:406)
==46021==    by 0x3B7A0F: ngx_http_proxy_wasm_dispatch_resume_handler (ngx_http_proxy_wasm_dispatch.c:734)
==46021==    by 0x3B8628: ngx_http_proxy_wasm_dispatch_handler (ngx_http_proxy_wasm_dispatch.c:467)
==46021==    by 0x1DEB89: ngx_event_process_posted (ngx_event_posted.c:35)
==46021==    by 0x1DE086: ngx_process_events_and_timers (ngx_event.c:273)
==46021==    by 0x1ED637: ngx_single_process_cycle (ngx_process_cycle.c:323)
==46021==    by 0x1A8038: main (nginx.c:384)
==46021==
{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: definite
   fun:malloc
   fun:ngx_alloc
   fun:ngx_create_pool
   fun:ngx_wasm_lua_thread_new
   fun:ngx_wasm_lua_resolver_resolve
   fun:ngx_wasm_socket_tcp_connect
   fun:ngx_http_proxy_wasm_dispatch_resume_handler
   fun:ngx_http_proxy_wasm_dispatch_handler
   fun:ngx_event_process_posted
   fun:ngx_process_events_and_timers
   fun:ngx_single_process_cycle
   fun:main
}

...so it seems that an lctx is indeed being lost along the way :thinking:

Thanks for the test case! Long story short, yes it appears our Lua bridge cannot handle more than one "Lua thread" at a time (lctx).
At first I enabled some of the DDEBUG macros and ran the test as-is. I commented the error.log output here, which shows what is currently going on. "posted event" followed by "delete posted event" is not of concern, since its part of ngx_event_process_posted. The logs showed the issue was with a call to ngx_http_lua_reset_ctx in the Lua bridge.
I then commented-out the lua_reset_ctx call which prevented the waiting file descriptor for call 1 from being prematurely closed, but also showed that call 1 was never resumed, even after the Lua thread of call 2 had yielded and resumed all the way through. It then became apparent that rctx only supports a single lctx object, which is problematic since the http module resumes yielded Lua threads via its single lctx reference.
Maybe we should make lctx a single, long-lived Lua thread that schedules smaller co-routines? Which I hope we can do with the exposed OpenResty facilities. This could seem to be the underlying cause of the behavior we've seen with the Gateway's DNS semaphore not being released, although a single Lua thread is supposed to be resumed all the way through, so I am not sure.

Thanks for taking a look at it!

I commented the error.log output here, which shows what is currently going on.

Good, so I think I was looking at the right thing: that # fd:12 closed (!!) is the same block as Looks like it deleted coroutine ending-62C0 in the annotation above.

We could also make rctx store a queue of Lua threads instead of a single one, and resume all threads until its empty.

Something along those lines was the first thing that came to my mind when I saw the Valgrind output. I'll give that a try!

We could also make rctx store a queue of Lua threads instead of a single one, and resume all threads until its empty.

Something along those lines was the first thing that came to my mind when I saw the Valgrind output. I'll give that a try!

I did try this approach: https://github.com/Kong/ngx_wasm_module/tree/wip/lctxs-queue
I managed to collect all lctxs and resume them, but it doesn't quite work, because the socket data did not arrive correctly.
What is happening is:

when our Lua chunk issues a DNS request, resty/dns/resolver.lua yields at line 871 (buf, err = sock:receive(4096))
when the response from the DNS server arrives, this triggers the ngx_http_wasm_wev_handler, in which I go through my queue, waking up the yielded lctxs.
what happens then is that sock:receive() returns 0 bytes (which then confuses resty.dns.resolver and produces a "truncated" response error)

Apparently, the socket data connection is only plugged correctly to the most-recently-created lctx? I don't fully understand how data is supposed to flow there.

Maybe we should make lctx a single, long-lived Lua thread that schedules smaller co-routines?

If there is no way to map incoming data to the correct lctx at the "wev_handler", then that would be the way to go. But then, OpenResty does have its own socket scheduler, right? (i.e. it implements the equivalents of both luasocket and copas). Not sure how we would be able to put another scheduling layer on top of that. Lua coroutine schedulers are not composable out-of-the-box...

At first I noticed that in ngx_http_lua_socket_udp, the response length is stored in u->received which is set to 111 after ngx_udp_recv, but later in ngx_http_lua_socket_udp_receive_retval_handler it ends up being 0 (which bubbles up to sock:receive() returning ""). The value of u->received changed because u changed in-between the recv event and the udp_receive_retval_handler:
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp socket read handler
  2024/03/27 22:25:20 [debug] 1695661#0: *1 event timer del: 12: 155591122
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp socket read data: waiting: 1
  2024/03/27 22:25:20 [debug] 1695661#0: *1 recv: fd:12 111 of 4096
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp recv returned 111
1. ---> u is correct lua *** ngx_http_lua_socket_udp_read: u->received: 111 (u: 0x7f5a8569d9c0) at ../ngx_lua-0.10.26/src/ngx_http_lua_socket_udp.c line 1158.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp socket waking up the current request
  2024/03/27 22:25:20 [debug] 1695661#0: *1 rewrite phase: 3
  ngx_http_wasm_rewrite_handler: enter at /home/code/ngx_wasm_module/src/http/ngx_http_wasm_module.c line 672.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 wasm rctx reused: 000055CFADA90BB8 (r: 000055CFADA8FE40, main: 1)
  2024/03/27 22:25:20 [debug] 1695661#0: *1 any lua yielded? 1
  ngx_http_wasm_rewrite_handler: wasm lua forcing content wev handler at /home/code/ngx_wasm_module/src/http
  ngx_http_wasm_module.c line 685.
  ngx_http_wasm_wev_handler: enter at /home/code/ngx_wasm_module/src/http/ngx_http_wasm_module.c line 994.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 wasm rctx reused: 000055CFADA90BB8 (r: 000055CFADA8FE40, main: 1)
  2024/03/27 22:25:20 [debug] 1695661#0: *1 wasm wev handler "/t?" - timeout: 0, ready: 1 (main: 1, count: 1, resp_finalized: 0, state: 0)
  ngx_http_wasm_wev_handler: will resume lua thread (lctx: 0x55cfad7d9eb0) at /home/code/ngx_wasm_module/src/http/ngx_http_wasm_module.c line 1025.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 wasm resuming lua thread (lctx: 000055CFAD7D9EB0, L: 00007F5C8681A380, co: 00007F5C867E3DA8)
  ngx_http_wasm_set_resume_handler: resume_handler = ngx_http_core_run_phases (r: 0x55cfada8fe40, rctx: 0x55cfada90bb8) at /home/code/ngx_wasm_module/src/http/ngx_http_wasm_module.c line 1119.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp operation done, resuming lua thread
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp socket calling prepare retvals handler 000055CFAC3A0954, u:00007F5C867D5EF0
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua udp socket receive return value handler
  lua *** ngx_http_lua_socket_udp_receive_retval_handler: buffer: <CB>Q<81><80> at ../ngx_lua-0.10.26/src/ngx_http_lua_socket_udp.c line 1031.
2. ---> u should be 0x7f5a8569d9c0 lua *** ngx_http_lua_socket_udp_receive_retval_handler: u->received: 0 (u: 0x7f5c867d5ef0) at ../ngx_lua-0.10.26/src/ngx_http_lua_socket_udp.c line 1032.
  lua *** dumpstack: top: 1 at ../ngx_lua-0.10.26/src/ngx_http_lua_util.h line 738.
3. ---> we get "" on top of the stack with luaL_pushlstring(L, buf, 0) lua *** dumpstack: 1    string  "" at ../ngx_lua-0.10.26/src/ngx_http_lua_util.h line 747.
  2024/03/27 22:25:20 [debug] 1695661#0: *1 lua run thread, top:9 c:1
  lua *** ngx_http_lua_run_thread: ctx: 0x55cfad91ecd0, co: 0x7f5c867e9708, co status: 0, co is_wrap: 0 at ../ngx_lua-0.10.26/src/ngx_http_lua_util.c line 1150.
  lua *** ngx_http_lua_run_thread: 0x7f5c867e9708: saved co top: 0, nrets: 1, true top: 1 at ../ngx_lua-0.10.26/src/ngx_http_lua_util.c line 1162.
  lua *** ngx_http_lua_run_thread: co top elem: string at ../ngx_lua-0.10.26/src/ngx_http_lua_util.c line 1170.
  2024/03/27 22:25:20 [notice] 1695661#0: *1 [lua] resolver.lua:633: parse_response(): buf:  len: 0, client: <IP_ADDRESS>, server: localhost, request: "GET /t HTTP/1.1", host: "localhost"
  2024/03/27 22:25:20 [debug] 1695661#0: *1 [lua] resolver.lua:797: _tcp_query(): query the TCP server due to reply truncation

Since u is stored in coctx->data and since the coctx of all our Lua threads comes from &ctx->entry_co_ctx, I believe that we are currently scheduling multiple udp_socks in the same coroutine context, as you said it is not plugged in correctly. Now I believe that we are not initializing our coroutines correctly. We should look closer at facilities that create coroutines in OpenResty like ngx_http_lua_uthread.c, ngx_http_lua_coroutine.c, and ngx_http_lua_coroutine_create_helper + ngx_http_lua_post_thread. If we instantiate our own coroutine + coroutine contexts, and attach them to ctx->cur_co_ctx (or &ctx->entry_co_ctx), we should get closer to what we are trying to do...

Now I believe that we are not initializing our coroutines correctly.

Yep, that's what I feared.

We should look closer at facilities that create coroutines in OpenResty like ngx_http_lua_uthread.c, ngx_http_lua_coroutine.c, and ngx_http_lua_coroutine_create_helper + ngx_http_lua_post_thread.

This is going to be quite the journey. I've always been uncomfortable with that part of the codebase not knowing what exactly the various ctx-this-and-that variables exactly mean; about time I build a proper understanding of those bits. I'm going to deep dive into this, since it's a blocker for the other stuff I was doing. Thanks for the pointers!

I think we will figure it out; I made good progress this morning, so far both calls resume with the proper coctx locally (and proper socket + u), just need to tie some lose ends.


Format selection not working.

When I have only a portion of the file selected, and run the formatter, it formats the whole file.
Example:

Here are my settings:

@arxpoetica the code for that is on https://github.com/jdc0589/jsformat-atom/blob/master/lib/format.coffee#L75 and on https://github.com/jdc0589/jsformat-atom/blob/master/lib/format.coffee#L94. Maybe Atom's API has changed, see if you can play around and find out what's wrong now.

I'm also seeing this issue.


Migrate Visual Snapshot Tests to Percy

As discussed on Discord.
https://percy.io/
Work with Dave to set everything up.

A quick update on this. I heard back from BrowserStack support team this morning and they do not currently include a Percy plan as part of their open-source sponsored account. I've asked if they have plans to add it, but I think I'm speaking to the wrong person about it.
The idea is good, but we wouldn't be able to implement this on their free plan.

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

Yes, of course. I had a look before posting this and couldn't see any that offer anything obvious. Are there any you'd recommend?

I can't explicitly recommend any others as I've not tried them, but I have a few others which we've done some light investigations on so I'll do some digging and update you on my findings 👍

Are we still pursuing this or should we close the issue?

I chased quite hard and got back two responses that said they are not currently supporting open-source projects with Percy and the cost of the product on its own is way out of our reach.
That said, it would be good to fix this problem. It's not really that great that all snapshot tests need to come through my machine.

What we could/should do is set up a hosted runner using one of our Macs at MacStadium to run the snapshots and be the canonical source.
The only difficulty is to make sure whenever we change Xcode versions that machine can and does follow.

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.
Given BrowserStack's track record with open-source, I'm sure they will add Percy to their open-source programme (which is very generous) at some point. Their marketing page for the programme says that it covers "All products" and Percy is listed in the footer of that same page.
I think we keep this open for a little while and I'll add a reminder to chase again in a month or so.

Have we considered other third parties such as Applitools?
Should be same benefits, and has open-source pricing?

I'm not super excited in general about tying this to a 3rd party service, to be honest. We don't have that many screens and the PR diffs have worked fine so far.

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.

Unfortunately, this would only affect future growth because the images are in the history already. (Also, at 18M Tests isn't super big I'd say.)


Using SQL Alias

Details of the scenario you try and problem that is occurring:
I have been trying to refer to my Database Server using my SQL Alias (set in cliconfg.exe).  It seems that none of my commands work unless I use the actual Database Server name itself.
The DSC configuration that is using the resource:
For example: https://github.com/PowerShell/SharePointDsc/wiki/SPWebApplication
When I use DatabaseServer parameter I get "Cannot Find Server" if I use the SQL Alias.  It only works when I use the SQL Server machine name.
Version of the Operating System and PowerShell the DSC Target Node is running:
Windows Server 2012 R2
Powershell Version 5 (Windows Management Framework 5.0)
Version of the DSC module you're using:
Find-Module -Name SharePointDsc -Repository PSGallery | Install-Module

I also use SQL alias for my builds, and have not yet run into any issues with any of the DSC resources.
Other then ensuring the SQL alias is set on each SharePoint server, there really isn't anything else to configure. Can you try and perform the same actions without DSC to verify the SQL alias is configured properly?

Agree with Mike, I also use SQL aliases without any issues. I have seen issues in the past with aliases not working properly. Never figured out the cause, but the solution was to create a DNS record for the alias.

I had an issue because the "Dynamically determine port" was't checked.
Before installing SP, I always check the SQL connection with a UDL file.

@cathornton Can you try to create the web application through PowerShell directly to test if that works. All the DSC resource does is natively call out to New-SPWebApplication, so you should be able to test it by calling that directly
New-SPWebApplication -Name "Contoso Internet Site" -URL "https://www.contoso.com" -ApplicationPool "ContosoAppPool" -DatabaseServer "YOUR.ALIAS.HERE" -DatabaseName "WSS_Content"

As per the previous comments, my expectation is that this will show you the same error as the DSC resource is, which will likely relate to how the client alias has been configured.
On SQL aliases in general though, I do not recommend these to my customers as they actually break some of the health analyser rules in SharePoint that attempt to check things at SQL server, as they assume that the alias is the computer name and attempt to connect that way but will then fail as that isn't valid. If you are using aliases to give you the ability to easily move SQL server later on, I would suggest considering using DNS to help route you to an appropriate SQL server as a better alternative.
Let us know how you go testing the web app creation manually and we'll be able to help more from there.

thanks for the feedback everyone. Unfortunately I can no longer replicate this issue.  My test results have been a bit inconsistent so I suspect there may be other issues at play in my environment.
@BrianFarnhill your point on using DNS instead of SQL Alias is a good one and I will take it on board.

@cathornton No worries - hope you work it all out! Closing this issue off.


No Prior Mention of a Storage Account

Under the topic, Create a message endpoint, item 2a mentions a storage account that is supposed to have been created. However, there is  no prior point in the tutorial where this is done. One can easily create a storage account but correcting this would make the quickstart more coherent.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: aa9fcb9e-8d13-c4d1-a5d0-7eeb706c2b51
Version Independent ID: 642cbcbe-ed7d-1181-fc9a-ee3538446996
Content: Send custom events to web endpoint - Event Grid, Azure portal - Azure Event Grid
Content Source: articles/event-grid/custom-event-quickstart-portal.md
Service: event-grid
GitHub Login: @spelluru
Microsoft Alias: spelluru

@doyinosin
Thanks for your feedback! We will investigate and update as appropriate.

Fixed it. The updated article would go live sometime today. No storage account needed for this quickstart.
#please-close


Deleted features doesn't mark tile as expired

Context
If the changeset contains delete-only operation (eg. deleted a building) then it's import doesn't generate expired tile.
Expected Behavior
Expired tile should be generated.
Actual Behavior
Expired tile is not generated.
Steps to Reproduce

have ~/go/bin/imposm run -connection postgis://gis:secret@localhost/gis -mapping mapping.yaml -limitto limit.geojson -cachedir ./cache -diffdir ./diff -expiretiles-zoom 15 -expiretiles-dir ./expires running
delete single building from OSM
osm changeset is applied to the DB by imposm3 but it is not reflected in ./expires dir

Context
We are trying to have a map rendered by mapnik in sync with OSM data.
Your Environment

Version used: 5b0125cb85ad82ff6d2f5786a3e032944a10692e
Environment name and version: Debian Buster, PostgreSQL 11.3

Can you provide more information?
A delete of a way should be handled by Imposm and there is a test case:
https://github.com/omniscale/imposm3/blob/master/test/expire_tiles.osc#L40...L43
https://github.com/omniscale/imposm3/blob/master/test/expire_tiles_test.go#L103

Please extract files from https://drive.google.com/open?id=1eIfafN_gosS0rhBFn1JhdGm0FYTzgsFV:

diff file 230.osc.gz contains a change that deletes building imported from building.osm.pbf
mapping.yaml is the mapping file we use
diff file 277.osc.gz contains a changeset that adds a new building on different location

Then run:

~/go/bin/imposm import -connection postgis://gis:gis@localhost/gis -mapping mapping.yaml -read building.osm.pbf -diff -write -cachedir ./cache -diffdir ./diff -overwritecache
~/go/bin/imposm import -connection postgis://gis:gis@localhost/gis -mapping mapping.yaml -deployproduction
check that single building is present in osm_buildings table
~/go/bin/imposm diff -connection postgis://gis:gis@localhost/gis -mapping mapping.yaml -cachedir ./cache -diffdir ./diff -expiretiles-dir ./expires 230.osc.gz
check that there is no building in osm_buildings table but no expires dir is created even if it should
~/go/bin/imposm diff -connection postgis://gis:gis@localhost/gis -mapping mapping.yaml -cachedir ./cache -diffdir ./diff -expiretiles-dir ./expires 277.osc.gz
check that new building is created in osm_buildings table and expires is created with correct *.tiles file

@olt Is there anything else I can provide?
(Even maybe some microfunding, or credits at https://www.freemap.sk/ (credits modal is not ready yet))?
Thanks

@olt I did some more tests and narrowed down the conditions. I found that tile is not marked as dirty if you not only delete a way but also all its nodes. If you apply following patch then the test will fail:
diff --git a/test/expire_tiles.osc b/test/expire_tiles.osc
index b2d4772..19255c4 100644
--- a/test/expire_tiles.osc
+++ b/test/expire_tiles.osc
@@ -37,7 +37,13 @@
     </way>
   </modify>

-  <!-- delete way -->
+  <!-- delete way including its nodes -->
+  <delete>
+    <node id="20201" version="1" timestamp="2011-11-11T00:11:11Z" lat="2.0" lon="2.0001" />
+  </delete>
+  <delete>
+    <node id="20202" version="1" timestamp="2011-11-11T00:11:11Z" lat="2.0" lon="2.0002" />
+  </delete>
   <delete>
     <way id="20251" version="1" timestamp="2011-11-11T00:11:11Z" />
   </delete>

Failed test:
...
--- FAIL: TestExpireTiles (0.73s)
    --- FAIL: TestExpireTiles/CheckExpireFile (0.00s)
        expire_tiles_test.go:164: missing expire tile for delete way {8283 8100 14}
FAIL
FAIL	github.com/omniscale/imposm3/test	0.745s

Thanks for creating the failing test case. Imposm assumed that the changesets are always in a "consistent" order. This is now fixed.

Thakn you!


Does VSMonoDebugger Support Breakpoints on User Threads?

Does VSMonoDebugger Support Breakpoints on User Threads?

Yes, I can debug Tasks started via Task.Factory.StartNew and Threads  started via new Thread(ThreadMethod).


Rocket Error 404

This just randomly started to happen when starting up the application with the docker container.
The container logs don't show any errors. I got this error initially only in Firefox, but for some time also in Chrome. Later (after some refreshes) Chrome worked while Firefox still displays 404 every time. Restarting the daemon does not help. Clearing site data does not help, force refresh does not help.
Container logs:
docker run --rm -v$PWD/data/taker:/data -p8000:8000 ghcr.io/itchysats/itchysats/taker:preview --maker=<IP_ADDRESS>:9999 --maker-id=69a42aa90da8b065b9532b62bff940a3ba07dbbb11d4482c7db83a7e049a9f1e testnet
2021-11-11 07:52:53  INFO Initialized logger
2021-11-11 07:52:53  INFO Running version: 0.1.1
2021-11-11 07:52:56  INFO Connected to BitMex realtime API
2021-11-11 07:52:56 DEBUG Noise protocol initiator handshake is complete
2021-11-11 07:52:56  WARN 🔧 Configured for release.
2021-11-11 07:52:56  WARN address: <IP_ADDRESS>
2021-11-11 07:52:56  WARN port: 8000
2021-11-11 07:52:56  WARN workers: 4
2021-11-11 07:52:56  WARN ident: Rocket
2021-11-11 07:52:56  WARN limits: bytes = 8KiB, data-form = 2MiB, file = 1MiB, form = 32KiB, json = 1MiB, msgpack = 1MiB, string = 8KiB
2021-11-11 07:52:56  WARN temp dir: /tmp
2021-11-11 07:52:56  WARN keep-alive: 5s
2021-11-11 07:52:56  WARN tls: disabled
2021-11-11 07:52:56  WARN shutdown: ctrlc = true, force = true, signals = [SIGTERM], grace = 2s, mercy = 3s
2021-11-11 07:52:56  WARN log level: critical
2021-11-11 07:52:56  WARN cli colors: true
2021-11-11 07:52:56  WARN 📬 Routes:
2021-11-11 07:52:56  WARN (index) GET /<_paths..> text/html
2021-11-11 07:52:56  WARN (dist) GET /assets/<file..>
2021-11-11 07:52:56  WARN (feed) GET /api/feed
2021-11-11 07:52:56  WARN (get_health_check) GET /api/alive
2021-11-11 07:52:56  WARN (post_order_request) POST /api/cfd/order
2021-11-11 07:52:56  WARN (margin_calc) POST /api/calculate/margin
2021-11-11 07:52:56  WARN (post_cfd_action) POST /api/cfd/<id>/<action>
2021-11-11 07:52:56  WARN 📡 Fairings:
2021-11-11 07:52:56  WARN Shield (liftoff, response, singleton)
2021-11-11 07:52:56  INFO 🛡 Shield:
2021-11-11 07:52:56  WARN 🚀 Rocket has launched from http://<IP_ADDRESS>:8000
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T09:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T08:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T23:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T03:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T14:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T04:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T00:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T01:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T02:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T18:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T10:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T15:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T19:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T20:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T05:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T16:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T17:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T21:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T22:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T11:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T06:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-12T07:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T12:00:00.price?n=20
2021-11-11 07:53:01 DEBUG Fetching announcement for /x/BitMEX/BXBT/2021-11-11T13:00:00.price?n=20

It appears this is a docker port forwarding problem.
When using a different port (e.g. 8020) it works.
I have not idea why this happens. I re-launched docker and that fixed port 8000 behaviour as well.
Strange that this would happen just when I trying to present the UI 😅
Fixed by upgrading and restarting docker (which broke docker of course, but that could be solved by restarting the machine... because why not :/ ).
I don't think there is anything we can action here.


print the max number

Description
Description
Domain
CPP
Type of Contribution
Addition
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of LearnCPP.

/assign

/assign

/assign

/assign


Move ComponentListGenerator constants into common

Currently, ComponentListGenerator (components) defines constants used as keys in the descriptions of components that are consumed by Compiler (buildserver). However, rather than referencing the constants, we duplicate the definitions in Compiler.java. The appropriate thing to do here would be to refactor the constants into the common module and then reference the single definition from both the ComponentListGenerator and Compiler.

I can take a look at this issue. What do you mean by the common module?

@ZhengCornell I consider each directory under the top level appinventor directory to be a module (appengine, buildserver, etc.). There is one called common. However, after looking at how the sources are organized, I think that it might be better to put the constants into components/src/com/google/appinventor/components/common. This package gets compiled into a CommonConstants.jar file that is used by the appengine, buildserver, and components modules. You should create a new class, e.g. ComponentDescriptorConstants, and move the constants there. Once you've done that, you can update ComponentListGenerator and Compiler to reuse the shared constants rather than their own copies.

So only shared constants should be put into ComponentDescriptorConstants. Or all private static final constants must be put into this file? Besides, except these three files, do I have to modify others?

Fixed via 516cd43


GetBucketPolicy broken by PR #3276

Describe the bug
Trying to get a bucket policy consistently fails with fake error 500 Internal Server Error.
The issue comes from PR #3276 forcibly trying to parse the body as an XML document and failing as it's a JSON document and then considering it _looks_like_special_case_error due to this line: https://github.com/boto/botocore/blob/bab77d87db47c28dfa4329c9b849c5febf3998cc/botocore/handlers.py#L180
Regression Issue

[X] Select this option if this issue appears to be a regression.

Expected Behavior
The bucket policy document is returned successfully.
Current Behavior
An error is ClientError is raised.
Reproduction Steps
With adequate permissions, and a bucket with some policy configured, run:
aws --debug s3api get-bucket-policy --bucket test-bucket-with-policy

Possible Solution
Don't feed a response body whose header says Content-Type: application/json to an XML parser.
Additional Information/Context
No response
SDK version used
1.35.45
Environment details (OS name and version, etc.)
Ubuntu 22.04.5 LTS

Thanks for reporting, the team in looking into this issue and will post updates here.

Linking PR including fix: https://github.com/boto/botocore/pull/3288, which should be released soon.

@tim-finnigan just confirming, is this fix to be released in v1.35.46 or the following release? I'm seeing the same error with botocore=v1.35.46 right now.

@mukitmomin the fix for GetBucketPolicy should be available in our next release. You can track releases in our tags or via PyPI. We'll also post here once it's available.

@mukitmomin - release 1.35.47 is now out, please update your version of boto3/botocore in order to have access to this fix.


Plugin doesn't return anything when I think it should

When I add a query log in api project, file lib/query.js
 46       .apply(function(idsFrom, idsTo) {
 47         graph.shortestPath(idsFrom, idsTo, query.related, function(err, ids) {
 48           if (err) {
 49             callback(err);
 50           } else {
 51             graph.expand(ids, [], function(err, concepts) {
 52               callback(err, concepts);
 53                 console.dir(ids, concepts);
 54             });

ids seems to be [ 'urn:hg:geonames:2746203' ], but concepts is undefined.
this is what is in the graph for that id

Fixed in b8cec673e2516670313d6a41fe3eebf7b0ef5c51
Still can fail if multiple nodes with same id are found, see issue #4


Inconsistent clipping planes between terrain and tileset

The same clipping planes provide a different slice in terrain vs a 3D tileset (slicing in different directions).  Is there a straightforward way to get them to reconcile?

Sandcastle example

Ah it seems this is the solution


[UNR-1515] Replace toolbar images, in line with new branding guidelines.

Contributions: We are not currently taking public contributions - see our contributions policy. However, we are accepting issues and we do want your feedback.

Description
Replace toolbar images, in line with new branding guidelines.
Tests
Don't merge yet, I'm building the engine to check that this renders correctly.
Documentation
Don't merge yet, I'm updating the docs with new toolbar screenshots.


Support data downloading and loading

          > >

This change should solve the error you posted.
If you want the readers to use our artifact, you indeed need to tell them to copy this file into their project dir and to include the command that ensures that the artifact gets downloaded (Pkg.ensure_artifact_installed("uai2014", "Artifacts.toml")).

Instead of Pkg.ensure_artifact_installed("uai2014", "Artifacts.toml"), can we build the function into our package so that in the documentation we can write:
julia> TensorInference.download_data("uai2014")

julia> TensorInference.load_data("uai2014", name, task)

We still have Artifacts as a dependency, but will not download data automaticall for users.
Because it is very anoying when a user was told to find a specific file in his filesystem to follow a documentation. Especially for windows users.
Originally posted by @GiggleLiu in https://github.com/TensorBFS/TensorInference.jl/issues/41#issuecomment-1630819190

Since we already have read_instance, read_instance_from_string, we could name this new function read_instance_from_artifact to keep things consistent. I was also thinking that perhaps we could always do the check (Pkg.ensure_artifact_installed(artifact_name, artifact_toml)) inside this function so that it becomes transparent to the user.


build broken for Intel 17 compilers

see: http://rostam.cct.lsu.edu/builders/hpx_intel_17_2_174_boost_1_59_centos_x86_64_debug/builds/16/steps/build_core/logs/stdio

The same errors can be observed on clang when building the branch pack-short-circuit. They seem to come from a poisonous overload in the local/remote dataflow set, and are likely related to the recent work on pack traversal.

Pack traversal isn't used by dataflow since #2829 isn't merged yet.

In that case then there's a tiny chance that the work on pack traversal fixes it, as the error seems to come from a poisonous computation of a fused invocation. I'll apply pack-short-circuit to that PR and see what happens...

Unfortunately applying that PR does not solve the issue.
@Naios You can see from the logs that the poisonous overload is caused by a bad interaction with util::detail::functional_unwrap_impl, which I had assumed to be part of the pack traversal related work. There seems to be some sort of disagreement between what dataflow and unwrap think is callable, and to make it worse the issue seems to be instantiation order sensitive.
For completeness, here's the same failure as experienced with clang (on the pack-short-circuit branch): https://circleci.com/gh/STEllAR-GROUP/hpx/8033

I've reduced the clang/pack-short-circuit test case to this:
#include <hpx/lcos/future.hpp>
#include <hpx/util/unwrap.hpp>

void f(int) {}
int main() {
    hpx::util::unwrapping(f)
        (hpx::lcos::make_ready_future(42), hpx::lcos::make_ready_future());
}

Does this also fail on Intel 17 for the master branch?

@K-ballo is this test failing on clang/MSVC as well?

Did you test whether
#include <hpx/lcos/future.hpp>
#include <hpx/util/unwrap.hpp>

std::true_type tt =
  hpx::util::detail::is_functional_unwrap<
    decltype(hpx::lcos::make_ready_future(42)),
    decltype(hpx::lcos::make_ready_future())
  >{};

is still valid after applying the short circuit branch?
So there is no issue in the sort circuit implementation itself.
I took a short look over the branch and your improvements seem fine for me.

That snippet isn't valid neither after nor before. I fail to see how this is related, since the header that defines the trait isn't even included (including it doesn't make the code valid either).
In any event, I suggest focusing on fixing the issue with unwrapping on the master branch.

This particular issue has been determined to be caused by pack traversal failing to map ranges of future types for Intel (most likely a bug on their end); more specifically, by the mechanism for remapping a container's allocator.
See: https://github.com/STEllAR-GROUP/hpx/tree/fixing-2904 for the minimum changes that "fix" the core build.

This was fixed by merging #2981.


Put test data into repository

Currently the actual tests cannot be run because the data is only on authors' computer
Since there seems to be a special test batch, it can be checked into the repository.

The tests run through the following steps. They are linked via saved files with naming convention indicating which phase has created the data. First step takes in the huge raw data we have no permission to share. First step also samples it down and saves for steps 2 and 3. Step 0 is needed for downloading so that anyone can execute the full pipeline. Currently, step 0 has been done manually.

https://github.com/EvoTestOps/LogLead/blob/c1c749829d04914d1ed5fd5d0415f927ea9cae2b/tests/loaders.py
https://github.com/EvoTestOps/LogLead/blob/c1c749829d04914d1ed5fd5d0415f927ea9cae2b/tests/enhancers.py
https://github.com/EvoTestOps/LogLead/blob/c1c749829d04914d1ed5fd5d0415f927ea9cae2b/tests/anomaly_detectors.py

The latest commit fixes this. Now data is downloaded when one trie to run main.py in tests folder.
https://github.com/EvoTestOps/LogLead/commit/ddc85341cff53b9d53310c6467fe1d0ee44d59c3
Download data can also be run separatly. It does not overwrite. Rather it check if folder exists it does not download
https://github.com/EvoTestOps/LogLead/blob/main/tests/download_data.py
This config file controls what gets downloaded and tested. Commenting out rows disables downloading and testing
https://github.com/EvoTestOps/LogLead/blob/main/tests/datasets.yml

@jnyyssol you had found couple of new datasets. Can you add them to config so they also get downloaded? Please also add them to tests.

@jnyyssol you had found couple of new datasets. Can you add them to config so they also get downloaded? Please also add them to tests.

This is a bit tricky, because the ADFA and AWSCTD datasets already consist of event IDs. Therefore most of the enhancements don't make sense, and some even cause it to crash. I got the tests to run with ADFA and AWSCTD by doing the following:

Download the data (needs new py7zr package to unpack .7z)
Load the data and save these two directly with _eh which indicates they have been enhanced
This will cause the enhancer to skip them
Add a check in anomaly_detectors.py to ensure numeric columns exist (which they don't in these datasets, so many tests are skipped)

@mmantyla do you think it makes sense to include these two in the tests given that they are so different? Before pushing I still need to check that my changes didn't break anything regarding the other datasets.

I am closing this. Full test data will never be in LogLead repo. However, tests folders already has mechanism of downloading all supported datasets


Try to fix plotting on Windows for the Python Console

Plotting was broken for the Python console and this fixes it in my local tests. It could also be a fix for issue #1831 (at least I don't get more hangups after pressing twice F5 with plots in a file).
@divenex, could you test this pull request and verify if that's true or not for you? Thanks ;-)

I need to release beta7 today. @divenex, please let me know if this fixes #1831 after I release it.


Bump v0.4.0rc1

What's Changed

Don't use args.exp_name by @KKIEEK in https://github.com/SIAnalytics/siatune/pull/139
[Enhance] Add support for dict input in Choice by @KKIEEK in https://github.com/SIAnalytics/siatune/pull/138
Add openmim in requirements/runtime by @KKIEEK in https://github.com/SIAnalytics/siatune/pull/145
[Feature] Support trial name template by @KKIEEK in https://github.com/SIAnalytics/siatune/pull/146

Full Changelog: https://github.com/SIAnalytics/siatune/compare/v0.4.0rc0...v0.4.0rc1

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@f0d17aa). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #147   +/-   ##
=======================================
  Coverage        ?   62.82%
=======================================
  Files           ?       65
  Lines           ?     2028
  Branches        ?      313
=======================================
  Hits            ?     1274
  Misses          ?      650
  Partials        ?      104

Flag
Coverage Δ

unittests
62.82% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


build(bazel): add buildifier and format build files

resolves DSP-1668
buildifier is a tool for formatting bazel BUILD and .bzl files with a standard convention.

add buildifier target (make buildifier)
run buildifier

Thanks for the review :-)


[CodeGen][LLVM] Lower if_then_else() with select

Hi, there. It seems better to use select directly for if_then_else() expr. Since if_then_else() is common in padding representation, it could hopefully save lots of cross-bb overheads.
My two remaining questions:
(1) Are there any places to test certain codegen features?
(2) Are there known llvm-based target which do not handle select instruction well?

Quick question: in TIR, T.Select means both branches are executed no matter which value the conditional is evaluated to - which brings benefits that make it vectorizable in certain cases. However, if the conditional is a safeguard to avoid access out of buffer boundary, replacing it with T.Select will lead the program to crash. Therefore, I would love to learn if our implementation here in LLVM codegen will lead to the same effect? Thanks a lot!

the conditional is a safeguard

Thanks for the reply! So is the if_then_else  also take short-circuit evaluation semantic like if stmt? If so I think my replacement is unsafe.  What about add a check like SideEffect(e) <= kReadState?

After some thinking, it seems to be better to make TIR transformation if_then_else() -> T.select if neccesary, thus do not complex the llvm codegen.
Thank you all the same! @junrushao1994

@wrongtest Thank you so much for the discussion!


Extension issue: Unresponsive, Performance Issue

Issue Type: Bug
Extension Name: python
Extension Version: 2021.1.502429796
OS Version: Windows_NT x64 10.0.18363
VSCode version: 1.53.2

:warning: We have written the needed data into your clipboard. Please paste! :warning:

Thanks for the bug report, but unfortunately we need some more information from you in order to triage this issue appropriately. Please create a new bug by providing a filled-in copy of our issue template.


Serial connection support for macOS

It recently came up on Slack that it would be good to have support for the serial connection on macOS.

@julianoes This concerns me - I was not aware that any of our APIs were "platform specific". If we don't support an API on a particular platform then that should be part of the documentation.
So does this work on Windows? Are there any other APIs that we know only work on Linux, Windows or Mac?

@hamishwillee: For what I understand from the code, the serial connection is now supported on Linux and macOS, and not on Windows.

@hamishwillee: For what I understand from the code, the serial connection is now supported on Linux and macOS, and not on Windows.

Thanks. Then we should probably add a warning to DroneCore::add_any_connection() and add_serial_connection() docs. Something like

@warning This method is not supported on Windows (it is supported on Linux and macOS).

I am making the assumption that everything else works everywhere.

I am making the assumption that everything else works everywhere.

I wouldn't know :grinning:.

@hamishwillee you're right.
Let's add that comment about Windows.
UDP and TCP connections are supported on all 3 operating systems.


Append date when marking <

Inspired by this issue, could we get a prompt whenever we mark a task with > to write a date (preferably in natural language) then append it to the task in this dataview supported format [due: YYYY-MM-DD] ?
This will be especially useful when it'll be possible to mark task right from the dataview query.

How do you see this working?

always prompt?
Only prompt sometimes?
Or .. only prompt when delegating? This could require/recommend a specific text append/replacement string per type, which gets interesting...

In my case, always prompt but only when marking with >. I don't need a prompt for the other types.

Sorry, I meant a prompt the < : schedulling type, not > : forwarded

Could this be generalized to expand the "Additional task types" setting to a list of each type where each item can be given a character, text to append when marking (rolling "Append to completed task" into this task type list), or leaving that field blank if a type shouldn't get marked.

This is in flight right now.

This will be done in 1.0.x (already present in BRAT release).


Fix: GitHub Workflow trigger

GitHub Workflow was triggered only when push event was fired.
Now it is also triggered when pull_request event is fired.
This enables external contributors to run tests when they send pull requests.
(push event is fired when push is done to forked repository, but push event is not fired when push is done to kitsuyui's repository)

Codecov Report
Patch and project coverage have no change.

Comparison is base (40086f0) 100.00% compared to head (9639aa8) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #50   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            2         2
  Lines          228       228
=========================================
  Hits           228       228

Flag
Coverage Δ

unittests
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Rewired all logger names so slf4j / odl works properly on WCS 12 running on Weblogic 12c

Hi Freddy, javadoc plugin complaining on the package step, sh package.sh fails:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (attach-javadocs) on project gsf-facades: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /gst-foundation/gsf-facades/src/main/java/com/fatwire/gst/foundation/DebugHelper.java:213: error: reference not found
[ERROR] * logger {@link DebugHelper#TIME_LOGGER}.

In gsf-facades/src/main/java/com/fatwire/gst/foundation/DebugHelper.java:213 the reference to TIME_LOGGER should be replaced with LOG_TIME.

Give it a try now


Trtexec multi-source (streams) and multi-batch performance test failed

Description
I want to test the performance of the model in multi-streams and multi-batches (https://github.com/NVIDIA-AI-IOT/yolo_deepstream#performance) with the trtexec command, and I test it with the following command
/usr/src/tensorrt/bin/trtexec --loadEngine=yolov7_b16_int8_qat_640.engine --shapes=images:4x3x640x640 --streams=4

ps:
The .engine model source is converted by the following command(dynamic batch)
/usr/src/tensorrt/bin/trtexec --verbose --onnx=yolov7_qat_640.onnx --workspace=4096 --minShapes=images:1x3x640x640 --optShapes=images:12x3x640x640 --maxShapes=images:16x3x640x640 --saveEngine=yolov7_b16_int8_qat_640.engine --fp16 --int8

but the following error occurs.
[06/02/2023-09:24:37] [I] === Model Options ===
[06/02/2023-09:24:37] [I] Format: *
[06/02/2023-09:24:37] [I] Model:
[06/02/2023-09:24:37] [I] Output:
[06/02/2023-09:24:37] [I] === Build Options ===
[06/02/2023-09:24:37] [I] Max batch: explicit batch
[06/02/2023-09:24:37] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[06/02/2023-09:24:37] [I] minTiming: 1
[06/02/2023-09:24:37] [I] avgTiming: 8
[06/02/2023-09:24:37] [I] Precision: FP32
[06/02/2023-09:24:37] [I] LayerPrecisions:
[06/02/2023-09:24:37] [I] Calibration:
[06/02/2023-09:24:37] [I] Refit: Disabled
[06/02/2023-09:24:37] [I] Sparsity: Disabled
[06/02/2023-09:24:37] [I] Safe mode: Disabled
[06/02/2023-09:24:37] [I] DirectIO mode: Disabled
[06/02/2023-09:24:37] [I] Restricted mode: Disabled
[06/02/2023-09:24:37] [I] Build only: Disabled
[06/02/2023-09:24:37] [I] Save engine:
[06/02/2023-09:24:37] [I] Load engine: yolov7_b16_int8_qat_640.engine
[06/02/2023-09:24:37] [I] Profiling verbosity: 0
[06/02/2023-09:24:37] [I] Tactic sources: Using default tactic sources
[06/02/2023-09:24:37] [I] timingCacheMode: local
[06/02/2023-09:24:37] [I] timingCacheFile:
[06/02/2023-09:24:37] [I] Heuristic: Disabled
[06/02/2023-09:24:37] [I] Preview Features: Use default preview flags.
[06/02/2023-09:24:37] [I] Input(s)s format: fp32:CHW
[06/02/2023-09:24:37] [I] Output(s)s format: fp32:CHW
[06/02/2023-09:24:37] [I] Input build shape: images=4x3x640x640+4x3x640x640+4x3x640x640
[06/02/2023-09:24:37] [I] Input calibration shapes: model
[06/02/2023-09:24:37] [I] === System Options ===
[06/02/2023-09:24:37] [I] Device: 0
[06/02/2023-09:24:37] [I] DLACore:
[06/02/2023-09:24:37] [I] Plugins:
[06/02/2023-09:24:37] [I] === Inference Options ===
[06/02/2023-09:24:37] [I] Batch: Explicit
[06/02/2023-09:24:37] [I] Input inference shape: images=4x3x640x640
[06/02/2023-09:24:37] [I] Iterations: 10
[06/02/2023-09:24:37] [I] Duration: 3s (+ 200ms warm up)
[06/02/2023-09:24:37] [I] Sleep time: 0ms
[06/02/2023-09:24:37] [I] Idle time: 0ms
[06/02/2023-09:24:37] [I] Streams: 4
[06/02/2023-09:24:37] [I] ExposeDMA: Disabled
[06/02/2023-09:24:37] [I] Data transfers: Enabled
[06/02/2023-09:24:37] [I] Spin-wait: Disabled
[06/02/2023-09:24:37] [I] Multithreading: Disabled
[06/02/2023-09:24:37] [I] CUDA Graph: Disabled
[06/02/2023-09:24:37] [I] Separate profiling: Disabled
[06/02/2023-09:24:37] [I] Time Deserialize: Disabled
[06/02/2023-09:24:37] [I] Time Refit: Disabled
[06/02/2023-09:24:37] [I] NVTX verbosity: 0
[06/02/2023-09:24:37] [I] Persistent Cache Ratio: 0
[06/02/2023-09:24:37] [I] Inputs:
[06/02/2023-09:24:37] [I] === Reporting Options ===
[06/02/2023-09:24:37] [I] Verbose: Disabled
[06/02/2023-09:24:37] [I] Averages: 10 inferences
[06/02/2023-09:24:37] [I] Percentiles: 90,95,99
[06/02/2023-09:24:37] [I] Dump refittable layers:Disabled
[06/02/2023-09:24:37] [I] Dump output: Disabled
[06/02/2023-09:24:37] [I] Profile: Disabled
[06/02/2023-09:24:37] [I] Export timing to JSON file:
[06/02/2023-09:24:37] [I] Export output to JSON file:
[06/02/2023-09:24:37] [I] Export profile to JSON file:
[06/02/2023-09:24:37] [I]
[06/02/2023-09:24:37] [I] === Device Information ===
[06/02/2023-09:24:37] [I] Selected Device: Xavier
[06/02/2023-09:24:37] [I] Compute Capability: 7.2
[06/02/2023-09:24:37] [I] SMs: 8
[06/02/2023-09:24:37] [I] Compute Clock Rate: 1.377 GHz
[06/02/2023-09:24:37] [I] Device Global Memory: 31002 MiB
[06/02/2023-09:24:37] [I] Shared Memory per SM: 96 KiB
[06/02/2023-09:24:37] [I] Memory Bus Width: 256 bits (ECC disabled)
[06/02/2023-09:24:37] [I] Memory Clock Rate: 1.377 GHz
[06/02/2023-09:24:37] [I]
[06/02/2023-09:24:37] [I] TensorRT version: 8.5.2
[06/02/2023-09:24:38] [I] Engine loaded in 0.0275892 sec.
[06/02/2023-09:24:38] [I] [TRT] Loaded engine size: 39 MiB
[06/02/2023-09:24:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +41, now: CPU 0, GPU 41 (MiB)
[06/02/2023-09:24:39] [I] Engine deserialized in 1.04122 sec.
[06/02/2023-09:24:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +364, now: CPU 0, GPU 405 (MiB)
[06/02/2023-09:24:39] [I] Setting persistentCacheLimit to 0 bytes.
[06/02/2023-09:24:39] [I] [TRT] Could not set default profile 0 for execution context. Profile index must be set explicitly.
[06/02/2023-09:24:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +363, now: CPU 0, GPU 768 (MiB)
[06/02/2023-09:24:39] [I] Setting persistentCacheLimit to 0 bytes.
[06/02/2023-09:24:39] [E] Error[1]: Unexpected exception cannot create std::vector larger than max_size()
[06/02/2023-09:24:39] [I] [TRT] Could not set default profile 0 for execution context. Profile index must be set explicitly.
[06/02/2023-09:24:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +363, now: CPU 0, GPU 1131 (MiB)
[06/02/2023-09:24:39] [I] Setting persistentCacheLimit to 0 bytes.
[06/02/2023-09:24:39] [E] Error[1]: Unexpected exception cannot create std::vector larger than max_size()
[06/02/2023-09:24:39] [I] [TRT] Could not set default profile 0 for execution context. Profile index must be set explicitly.
[06/02/2023-09:24:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +1, GPU +364, now: CPU 1, GPU 1495 (MiB)
[06/02/2023-09:24:39] [I] Setting persistentCacheLimit to 0 bytes.
[06/02/2023-09:24:39] [E] Error[1]: Unexpected exception cannot create std::vector larger than max_size()
[06/02/2023-09:24:39] [I] Using random values for input images
[06/02/2023-09:24:39] [I] Created input binding for images with dimensions 4x3x640x640
[06/02/2023-09:24:39] [I] Using random values for input images
[06/02/2023-09:24:39] [I] Created input binding for images with dimensions 4x3x640x640
[06/02/2023-09:24:39] [I] Using random values for input images
[06/02/2023-09:24:39] [I] Created input binding for images with dimensions 4x3x640x640
[06/02/2023-09:24:39] [I] Using random values for input images
[06/02/2023-09:24:39] [I] Created input binding for images with dimensions 4x3x640x640
[06/02/2023-09:24:39] [I] Using random values for output outputs
[06/02/2023-09:24:39] [I] Created output binding for outputs with dimensions 4x25200x85
[06/02/2023-09:24:39] [I] Using random values for output outputs
[06/02/2023-09:24:39] [I] Created output binding for outputs with dimensions 4x25200x85
[06/02/2023-09:24:39] [I] Using random values for output outputs
[06/02/2023-09:24:39] [I] Created output binding for outputs with dimensions 4x25200x85
[06/02/2023-09:24:39] [I] Using random values for output outputs
[06/02/2023-09:24:39] [I] Created output binding for outputs with dimensions 4x25200x85
[06/02/2023-09:24:39] [I] Starting inference
[06/02/2023-09:24:39] [E] Error[2]: [executionContext.cpp::enqueueV3::2386] Error Code 2: Internal Error (Assertion mOptimizationProfile >= 0 failed. )
[06/02/2023-09:24:39] [E] Error occurred during inference

Environment
TensorRT Version : 8.5.2
GPU Type : J etson AGX Xavier
Nvidia Driver Version :
CUDA Version : 11.4.315
CUDNN Version : <IP_ADDRESS>
Operating System + Version : 35.2.1 ( Jetpack: 5.1)
Python Version (if applicable) : Python 3.8.10
TensorFlow Version (if applicable) :
PyTorch Version (if applicable) : 1.12.0a0+2c916ef.nv22.3

Hi, I am not sure if you still can meet the issue on lastest tensorrt. pls have a try


Add 'auto' tactic and STLC example

Description
This change introduces a Coq-like auto tactic using user-defined hints and the existing theorem context, as well as a test case for formalized STLC.
At the moment the auto tactic does a simple BFS brute force scan with minor pruning, but it may be possible to introduce some more intelligent pattern matching or tactic/id re-ordering in the future.
Points to note/for discussion:

I benchmarked the auto.rkt test file to take around 30 seconds for proving five (trivial) theorems from scratch, with some taking a bit longer than others (inductive example seems to work almost instantaneously, for instance)
As it is, the hints need to be cleared each time after use otherwise they bleed into the subsequent auto runs
I'm not entirely sure if the changes related to apply and by-intros was necessary, though I'm not quite sure how to work with syntax methods that aren't tactics operating on a tree zipper.

Tests performed:

Just ran the two test files and saw no errors

We're going to make the hint database automatically lexically scoped by redefining ntac and define-theorem to parameterize the hint database in the scope of a new ntac call, and fix a few code smells before merging.

Pushed some changes to this in my branch auto; take a look.

Pushed some changes to this in my branch auto; take a look.

Merged in the changes. Just to be clear, it's fine that we have begin-for-syntax blocks for the parameters right? e.g.
(begin-for-syntax
  (current-max-auto-depth 4))

And the remaining TODO comments to remove the methods are so that we only let the user set hints at the top level similar to the max auto depth?

Something like this interface, where scope manages removes hints automagically.
(define-theorem ...
  #:hints (e ...)

  (hints e1 ...
    auto
    simpl))

I've rebased, squashed commits, and fixed some stylistic things more in origin/auto.
We should add documentation for this. Most of the documentation is out of date, but the ntac stuff is still correct, so it' be good to stay on top of it. Think you can figure out how to do that?

I've rebased, squashed commits, and fixed some stylistic things more in origin/auto.
We should add documentation for this. Most of the documentation is out of date, but the ntac stuff is still correct, so it' be good to stay on top of it. Think you can figure out how to do that?

Will do - I'll merge in your changes and add documentation here. Thanks!

You may not want to merge my changes since I’ve done a lot of rebasing and squashing; I can always merge documentation and such.
--
Sent from my phoneamajig

On Oct 18, 2019, at 10:48, Paul<EMAIL_ADDRESS>wrote:
I've rebased, squashed commits, and fixed some stylistic things more in origin/auto.
We should add documentation for this. Most of the documentation is out of date, but the ntac stuff is still correct, so it' be good to stay on top of it. Think you can figure out how to do that?
Will do - I'll merge in your changes and add documentation here. Thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

Current state of affairs for the documentation:

Some red squiggles for forall, ->, Nat, define, I'm guessing it'll stay this way until we add documentation for these again?
Some weird documentation behaviour with duplicate names across base and hints. For now, I've made define-theorem point to the one defined in hints while everything else (e.g. ntac) points to base as before.
Removed some of the unused tactics from the documentation (I made sure to check commit history for the files, and it looks like these variants were indeed removed)
by-obvious wasn't working on types since they aren't considered IDs

Please let me know if I should add anything else!

Looks like the blocking issue in Racket was fixed via https://github.com/racket/racket/issues/2981

Merged in e86c676925641bd4ece88ba3972ae0c80561cb68


Can get softlocked in FJ Temple

Probably just update the checkpoint to jungle-start if they have both temple cells

Fixed


Register as package

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/17217
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.0 -m "<description of version>" bb2c4994d93dcb819585fde4c4a484234ec1fd38
git push origin v0.1.0

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/17217
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.0 -m "<description of version>" 3a04173de4e72aba2a0ea23359b2019b0d6ffa24
git push origin v0.1.0

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/17217
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.0 -m "<description of version>" 8476c32cee28006d3e006382b8be9ca5276f0ff1
git push origin v0.1.0

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/18573
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.1 -m "<description of version>" cb6950f2b4d9db9e5f560516f50d299b7a66696b
git push origin v0.1.1

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/21475
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.2 -m "<description of version>" 60fe6145dd896155645ff292d9049e35bb0488bf
git push origin v0.1.2

@JuliaRegistrator register()

Error while trying to register: "Tag with name v0.1.2 already exists and points to a different commit"

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/21517
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 24dfb6cffa5ef45346c247ec5f7138f7d9603b82
git push origin v0.1.3

Also, note the warning: Version 0.1.3 skips over 0.1.2
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/21475
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.2 -m "<description of version>" d40ea3cf13c89445fd8984fe3e317a856345e300
git push origin v0.1.2

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/23317
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" d4bd2688846d4db79af43410eccba00e17c36208
git push origin v0.1.3

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/23317
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 76d4c38fa319b6880f3e8b07a4a70767df882471
git push origin v0.1.3

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/25089
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" 5dff7633656d06baf38bc496329b62abf7d78287
git push origin v0.1.5

Also, note the warning: Version 0.1.5 skips over 0.1.4
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/25389
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.4 -m "<description of version>" bbfecc2a6d9b88cd610bd9b877bd23c334e9e536
git push origin v0.1.4

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/26175
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" 416256f3bfd3af9cb07995beb9c52a6fca532555
git push origin v0.1.5

@JuliaRegistrator register()

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/26175
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" fc8ebed6f179d1b5b36b46fefa038c5c67d34a0c
git push origin v0.1.5

Registration pull request updated: JuliaRegistries/General/26175
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" fc8ebed6f179d1b5b36b46fefa038c5c67d34a0c
git push origin v0.1.5

@JuliaRegistrator register()

@JuliaRegistrator register()

Registration pull request updated: JuliaRegistries/General/26175
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" b30dbf0f58b014f36dfa5ff738182d125af2ac9a
git push origin v0.1.5

Registration pull request updated: JuliaRegistries/General/26175
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" b30dbf0f58b014f36dfa5ff738182d125af2ac9a
git push origin v0.1.5


Add quick start in readme

Add a clear and concise quick start section in Project Readme itself for beginners to setup project on their local machines.

@tdruez Can I work on adding steps in Readme itself on how to run project on their machines. It will be beneficial for beginners to setup project and contribute to it.

Our standard is to use RTD for documentation - otherwise you have duplicate information which is likely to be incorrect in at least one location. It is also cleaner to synch code with doc using RTD.

There is adequate starting documentation here: https://scancodeio.readthedocs.org/, closing this.

@majurg @mjherzog Sure, will look into it. Looking forward for more contributions.


Handle login and logout correctly

If I log in as an operator and then click the "back" button in the browser, I'm still logged in even though I'm on the Login page.
What to do?

When an operator is logged in and refreshes the page (clicks F5 on the keyboard or clicks the browser's refresh button) it logs him out.
Sometimes hitting the browser's back button also logs operator out.

When an operator is logged in and refreshes the page (clicks F5 on the keyboard or clicks the browser's refresh button) it logs him out.
Sometimes hitting the browser's back button also logs operator out.

Solved in 24c688b, 3f193b1 and 08ce230 by correcting client-side routing in the MUI components. We use react-router-dom Link component or navigate function from the useNavigate() hook.


Add migration for accounts

Needed to run the migration introduced with api-plugin-accounts#17.
Merge when the next version of api-plugin-accounts gets released.

There you go guys! Should be ready to merge now.


optimize.minimize BFGS oscillates, does not converge

In my application, I find that BFGS often oscillates near the optimum but does not converge (repeating values in the input array x of the objective function). I am doing some image analysis, so it's hard to recreate the issue without the raw data. I am not able to exit early: maxiter doesn't work, I guess because it's stuck within a single iteration? Why is this? It will exit early if maxiter is small enough, but will oscillate indefinitely otherwise. How else can I exit the optimizer early without killing the process if it starts to oscillate like this?
This is an example of the oscillating behavior, note that it repeats after 5 lines:
f(x)              x[0]        x[1]       x[2]
5426.478690485285 [2.57262195 0.25218706 0.28089962]
5386.242021794702 [2.57262195 0.25218706 0.18089962]
5386.242021794702 [2.57262195 0.25218706 0.18089962]
5317.307733034538 [2.67262195 0.25218706 0.18089962]
5258.679479779108 [2.57262195 0.35218706 0.18089962]
5426.478690485285 [2.57262195 0.25218706 0.28089962]
5386.242021794704 [2.57262195 0.25218706 0.18089962]
5386.242021794704 [2.57262195 0.25218706 0.18089962]
5317.307733034539 [2.67262195 0.25218706 0.18089962]
5258.679479779108 [2.57262195 0.35218706 0.18089962]
5426.478690485287 [2.57262195 0.25218706 0.28089962]
5386.242021794702 [2.57262195 0.25218706 0.18089962]
5386.242021794702 [2.57262195 0.25218706 0.18089962]
5317.307733034538 [2.67262195 0.25218706 0.18089962]
5258.679479779108 [2.57262195 0.35218706 0.18089962]
5426.478690485285 [2.57262195 0.25218706 0.28089962]

I've seen several issues regarding maxiter "not working" with BFGS, but none specifically quote this oscillating behavior.

What options are you calling it with? Which Scipy version?
If you are not providing an explicit gradient, then that's probably the numerical differentiation with a fairly coarse step, not oscillation. Hit Ctrl-C during the execution to get a traceback about what it is doing. maxiter counts the outer iterations, not line search and gradient estimation, so you can try setting to a smaller number.

I am calling minimize with these options: options={'gtol':1, 'eps':1e-6}, though in the example above I had 'eps':1e-1, changing these numbers helps sometimes but not always. I am not supplying a gradient function, correct. Scipy version is 1.4.1.
This is the relevant part of the traceback:
  File "C:\Users\nwestern\Documents\python_scripts\pl-image-utils\wafer_edge_reg.py", line 121, in edge_detect
    options={'gtol':1, 'eps':1e-6}) # BKM
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\_minimize.py", line 604, in minimize
    return _minimize_bfgs(fun, x0, args, jac, callback, **options)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 1028, in _minimize_bfgs
    old_fval, old_old_fval, amin=1e-100, amax=1e100)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 845, in _line_search_wolfe12
    **kwargs)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\linesearch.py", line 101, in line_search_wolfe1
    c1=c1, c2=c2, amax=amax, amin=amin, xtol=xtol)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\linesearch.py", line 176, in scalar_search_wolfe1
    derphi1 = derphi(stp)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\linesearch.py", line 90, in derphi
    gval[0] = fprime(xk + s*pk, *newargs)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 327, in function_wrapper
    return function(*(wrapper_args + args))
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 765, in approx_fprime
    return _approx_fprime_helper(xk, f, epsilon, args=args)
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 697, in _approx_fprime_helper
    df = (f(*((xk + d,) + args)) - f0) / d[k]
  File "C:\Users\nwestern\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\optimize\optimize.py", line 327, in function_wrapper
    return function(*(wrapper_args + args))

This is an intermittant issue for me. Irrespective of the cause of this effect, is there a way that I can gracefully exit the optimizer early?

I would exit the optimizer early by raising an Exception in the callback or objective functions. The former is preferred because you can keep track of the best solution vector so that you can retrieve it once you've handled the exception.
e.g.
class Callback:
    def __init__(self):
         self.x = None
    def __call__(self, x):
        self.x = x
        if stopping_condition:
             raise StopIteration

my_callback = Callback()


http: delaying filter stack creation until full headers have been received

This is a precursor to creating a custom filter stack for websocket vs HTTP.
Risk Level: High
Testing: new unit tests to verify correct watermark behavior
Docs Changes: n/a
Release Notes: yes.

Oops looks like master merge needed.

(intentionally waiting until after we cut the release)


Allow variable number of references for BLEU via API

This allows BLEU to use a variable number of references (different number of references for each sentence) if called via the API (see #130). If a sentence has fewer than the maximum number of references, None should used to fill remaining reference streams.

Thanks for this! I suggest we merge into #133 and including it with the changes to version 1.5.

Thanks for this! I suggest we merge into #133 and including it with the changes to version 1.5.

I'm happy to have this merged, but it needs a note added to the CHANGELOG.

I'm happy to have this merged, but it needs a note added to the CHANGELOG.

Just after merging this, I figured out that this also removes empty hypotheses, which could have been produced by a decoder that only generated "EOS" token in the early training phase. In the case of multi-references, first ref would then be taken as the hyp, leading to erroneous computation:
In [22]: sys
Out[22]:
['The dog bit the man.',
 "It wasn't surprising.",
 'The man had just bitten him.']

In [23]: refs
Out[23]:
[['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
 ['The dog had bit the man.',
  'No one was surprised.',
  'The man had bitten the dog.']]

In [24]: empty_sys
Out[24]: ['', '', '']

In [25]: sacrebleu.corpus_bleu(sys, refs)
Out[25]: BLEU = 48.53 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

In [26]: sacrebleu.corpus_bleu(empty_sys, refs)
Out[26]: BLEU = 23.93 64.7/35.7/18.2/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)

Just after merging this, I figured out that this also removes empty hypotheses, which could have been produced by a decoder that only generated "EOS" token in the early training phase. In the case of multi-references, first ref would then be taken as the hyp, leading to erroneous computation:
In [22]: sys
Out[22]:
['The dog bit the man.',
 "It wasn't surprising.",
 'The man had just bitten him.']

In [23]: refs
Out[23]:
[['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
 ['The dog had bit the man.',
  'No one was surprised.',
  'The man had bitten the dog.']]

In [24]: empty_sys
Out[24]: ['', '', '']

In [25]: sacrebleu.corpus_bleu(sys, refs)
Out[25]: BLEU = 48.53 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

In [26]: sacrebleu.corpus_bleu(empty_sys, refs)
Out[26]: BLEU = 23.93 64.7/35.7/18.2/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)


Roll Flutter Engine from 3d8c5ef1060c to 3470fa84826e (1 revision)

https://github.com/flutter/engine/compare/3d8c5ef1060c...3470fa84826e
2023-01-01<EMAIL_ADDRESS>Roll Skia from 486deb23bc2a to a31d9c3b4583 (2 revisions) (flutter/engine#38586)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Add GenPhrase

Add GenPhrase, a secure passphrase generator for PHP 5.3+ applications:
https://github.com/timoh6/GenPhrase

Thanks, this looks great.


[Enhancement] Class Attribute Per Level

Describe the solution you'd like
Starting attributes can not be decimals. Attribute per level can be decimal
in which 0.2 means 1 point every 5 levels however there is no way for it to
account for the first level making it harder to come up with the numbers
to account for this issue.

Or I can just add 0.01 now that I think about it


Constructor of all Node always NodeClass.Object

Hello,
Thank you for your OPC implemantation !
In the file AddressSpace, you create all types of nodes but you always create them in NodeObject type rather than the right type, is that intentional?
For example, for the NodeObjectType the constructor is :
			public NodeObjectType(NodeId Id, QualifiedName BrowseName, LocalizedText DisplayName, LocalizedText Description, UInt32 WriteMask, UInt32 UserWriteMask, bool IsAbstract)
				: base(Id, NodeClass.Object, BrowseName, DisplayName, Description, WriteMask, UserWriteMask)

But it will be better is if was that ?

			public NodeObjectType(NodeId Id, QualifiedName BrowseName, LocalizedText DisplayName, LocalizedText Description, UInt32 WriteMask, UInt32 UserWriteMask, bool IsAbstract)
				: base(Id, NodeClass.ObjectType, BrowseName, DisplayName, Description, WriteMask, UserWriteMask)

Oops, fixed. Thanks!


fix: Do not suggest to disable schema validation if it is already disabled

Fixes #914

Codecov Report

Merging #956 (fafc9f9) into master (ffdcee6) will decrease coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #956      +/-   ##
==========================================
- Coverage   99.77%   99.68%   -0.10%
==========================================
  Files          66       66
  Lines        4494     4451      -43
  Branches      740      740
==========================================
- Hits         4484     4437      -47
- Misses          3        5       +2
- Partials        7        9       +2

Flag
Coverage Δ

unittests
99.68% <100.00%> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/schemathesis/cli/__init__.py
100.00% <100.00%> (ø)

src/schemathesis/cli/context.py
100.00% <100.00%> (ø)

src/schemathesis/cli/output/default.py
100.00% <100.00%> (ø)

src/schemathesis/extra/pytest_plugin.py
97.72% <0.00%> (-2.28%)
:arrow_down:

src/schemathesis/cli/callbacks.py
98.19% <0.00%> (-1.81%)
:arrow_down:

src/schemathesis/lazy.py
98.38% <0.00%> (-0.03%)
:arrow_down:

src/schemathesis/models.py
99.52% <0.00%> (-0.02%)
:arrow_down:

src/schemathesis/runner/impl/core.py
98.68% <0.00%> (-0.01%)
:arrow_down:

src/schemathesis/hooks.py
100.00% <0.00%> (ø)

src/schemathesis/targets.py
100.00% <0.00%> (ø)

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffdcee6...fafc9f9. Read the comment docs.


CI upgrade

It's most useful if the commit messages are in the present tense (e.g. "Add pyOpenSSL"), and include the motivation for the change, as its not immediately clear from your first commit why pyOpenSSL is being added (there is a Git styleguide here [1]).
1: https://github.com/alphagov/styleguides/blob/master/git.md

@cbaines fixed. thanks for a review.

@cbaines fixed. thanks for a review.

:+1: I see that Jenkins has now picked up this branch which is good, but the build has failed.

@cbaines yeah, it failed. will debug it as soon as we get access to https://ci.integration.publishing.service.gov.uk/

@cbaines yeah, it failed. will debug it as soon as we get access to https://ci.integration.publishing.service.gov.uk/

Have you got access issues with Jenkins itself, or with the CI agent over SSH?

access issues with jenkins/ci website because of vpn. should be resolved today evening.


React router dynamic routes not rendering component

Hey, I tried to do the dynamic routes thing, but can't make it work properly. I found a similar SO question, I have almost the same code, but the App component still doesn't render.
If I import the App component and change the code to the below one, it works:
import App from "./components/App";

// ...

const routes = {
  component: 'div',
  childRoutes: [{
    path: '/',
    component: App,
    childRoutes: [
      // ...
    ]
  }]
};

Why is it so, the docs are wrong or I do something wrong? Can't find any info :(

This will also work:
component: require('./components/App').App,

How to make it work with component: require('./components/App') ?

Thanks for your question!
We want to make sure that the GitHub issue tracker remains the best place to track bug reports and feature requests that affect the development of React Router.
Questions like yours deserve a purpose-built Q&A forum. Would you like to post this question to Stack Overflow with the tag #react-router? https://stackoverflow.com/questions/ask?tags=react-router.
We also have an active and helpful React Router community on Reactiflux, which is a great place to get fast help with React Router and with the rest of the React ecosystem. You can join at https://discord.gg/0ZcbPKXt5bYaNQ46.

@taion I've already searched Google, Stack Overflow and found the same question with no answer, Discord also no answer.
Here is the small repo with code. The file is required and if I console log something - it works, but this class doesn't render.
Can you please take a minute and look at it? I really don't know where to ask.

I'm sorry - we need to keep the issue tracker restricted just to trackable project issues. It's not fair to anybody else if we make exceptions on support/Q&A on the issue tracker. Try e.g. asking again on Discord, or just wait for someone to answer the SO question.

Okay, the reason was babel 6 and changed behaviour of module.exports in commonjs.
Now, we have to use component: require('./components/App').default.

:+1:

I was stuck 'coz of this as well. Thanks @VladShcherbin for the .default
It would be great if this gets updated in the official doc.


fix: when deleting dont validate expiration

closes #184

First PR 100% made in the cloud ever 🔥

oh nice 💥


Deployment commit for stage-stable

Merged main branch to stage-stable.
Use latest commit to update namespace ref in app-interface repo. Don't use merge commit, SHAs must be unique when images are created for each branch.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 65.72%. Comparing base (eb05f8d) to head (a9cc479).

Additional details and impacted files
@@              Coverage Diff              @@
##           stage-stable    #3893   +/-   ##
=============================================
  Coverage         65.72%   65.72%
=============================================
  Files               515      515
  Lines             10001    10001
  Branches           2198     2198
=============================================
  Hits               6573     6573
  Misses             3424     3424
  Partials              4        4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Check that bytes received equals the Content-Length header; retry if not

Can we explore using got instead? It seems more reliable, see @hyperknot's comments on #490

I'm gonna merge this, but we should look into got ASAP!