Initial sync is worryingly slow & big to get to clients.

While chasing down https://github.com/matrix-org/matrix-rust-sdk/issues/1308 i was seeing initial sync reqs taking 36s to arrive. This was in part exasperated by the req having a 10s timeout on it, but even after fixing that, things are taking a worryingly long time - especially after ~12h of being offline.
Here's one i just profiled taking ~5s to execute (after ~2h of being offline). Client saw:
2022-12-28T00:21:59.635174Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some(10), filters: None }], room_subscriptions: {}, unsubscribe_rooms: [], extensions: ExtensionsConfig { to_device: Some(ToDeviceConfig { enabled: Some(true), limit: None, since: None }), e2ee: Some(E2EEConfig { enabled: Some(true) }), account_data: Some(AccountDataConfig { enabled: Some(true) }), other: {} } } config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver=Some("https://slidingsync.lab.element.dev/")}:send{config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver="https://slidingsync.lab.element.dev/" user_id=Some("@matthew:matrix.org") server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] request_type="ruma_client_api::sync::sync_events::v4::Request"}: matrix_sdk::http_client: Sending request
2022-12-28T00:22:04.412702Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some(10), filters: None }], room_subscriptions: {}, unsubscribe_rooms: [], extensions: ExtensionsConfig { to_device: Some(ToDeviceConfig { enabled: Some(true), limit: None, since: None }), e2ee: Some(E2EEConfig { enabled: Some(true) }), account_data: Some(AccountDataConfig { enabled: Some(true) }), other: {} } } config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver=Some("https://slidingsync.lab.element.dev/")}:send{config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver="https://slidingsync.lab.element.dev/" user_id=Some("@matthew:matrix.org") server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] request_type="ruma_client_api::sync::sync_events::v4::Request"}: matrix_sdk::http_client: Got response: Response { status: 200, version: HTTP/1.1, headers: {"access-control-allow-headers": "Origin, X-Requested-With, Content-Type, Accept, Authorization", "access-control-allow-methods": "GET, POST, PUT, DELETE, OPTIONS", "access-control-allow-origin": "*", "content-type": "application/json", "date": "Wed, 28 Dec 2022 00:22:01 GMT", "transfer-encoding": "chunked", "set-cookie": "......; path=/; HttpOnly; Secure; SameSite=None"}, body: b"{\"lists\":[{\"ops\":[{\"op\":\"SYNC\",\"range\":[0,20],

Server saw:
00:22:00 INF created new connection conn_id=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 user=@matthew:matrix.org
00:22:01 WRN Client did not increment since token: possibly sending back duplicate to-device events! device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 initial=true last_sent=6950241 recv=0 user=@matthew:matrix.org
00:22:01 INF  ag=25 d=1 duration=1111.414264 p=0 q=1 r=21 size=296993 status=200 u=@matthew:matrix.org
00:22:04 INF  dl-c=71 dl-l=8 duration=32.985866 p=1 q=2 r=0 size=2200 status=200 u=@matthew:matrix.org
00:22:05 INF  duration=44.822427 p=2 q=3 r=0 size=327 status=200 u=@matthew:matrix.org

Server pprof saw (for the OnIncomingRequestInitial):
When	Elapsed	Goroutine ID	Events
4.451185334s	1.105535526s		Task 1964 (goroutine view) (complete)
4.451185334	 .         	39366	task OnIncomingRequestInitial (id 1964, parent 0) created
4.451192886	 .     7552	39366	region load started (duration: 42.257935ms)
4.493469125	 . 42276239	39366	region onIncomingListRequest started (duration: 32.959255ms)
4.526486083	 . 33016958	39366	region buildRooms started (duration: 1.000488985s)
4.558728129	 . 32242046	39366	new goroutine 39367: database/sql.(*DB).beginDC·dwrap·17
4.560168230	 .  1440101	39366	new goroutine 39395: database/sql.(*Rows).initContextClose·dwrap·27
4.562780862	 .  2612632	39366	region RoomStateAfterEventPosition started (duration: 941.937482ms)
4.563280573	 .   499711	39366	new goroutine 39294: database/sql.(*DB).beginDC·dwrap·17
4.568830597	 .  5550024	39366	new goroutine 39368: database/sql.(*Rows).initContextClose·dwrap·27
4.570190699	 .  1360102	39366	new goroutine 39369: database/sql.(*Rows).initContextClose·dwrap·27
5.527456348	 .957265649	39366	region extensions started (duration: 29.246106ms)
5.549077859	 . 21621511	39366	new goroutine 39295: database/sql.(*DB).beginDC·dwrap·17
5.551339387	 .  2261528	39366	new goroutine 39296: database/sql.(*DB).beginDC·dwrap·17
5.552959072	 .  1619685	39366	new goroutine 39396: database/sql.(*Rows).initContextClose·dwrap·27
5.553884664	 .   925592	39366	new goroutine 39297: database/sql.(*DB).beginDC·dwrap·17
5.555373534	 .  1488870	39366	new goroutine 39373: database/sql.(*Rows).initContextClose·dwrap·27
5.556710390	 .  1336856	39366	region liveUpdate started (duration: 6.246µs)
5.556720860	 .    10470	39366	task end

So: if it took 1111ms to generate on the server (which is already quite a lot; the 300KB request size is way too large thanks to including 130KB of redundant account_data on it, and 20*20 = 400 events of scrollback), how come it took the client another 3s to receive it? I guess this could be slow internet connectivity whilst travelling for xmas, but empirically i seem to be able to download stuff at ~4MB/s.
To address the size, i think account_data has to be incremental, as per https://github.com/matrix-org/matrix-spec-proposals/pull/3575/files#r1057933023 - and we should always avoid syncing in scrollback in rooms during initial sync. However, it doesn't explain whey the request took 1s to execute but blocked the client for 5s...

Here's another one, this time after 10h of downtime:
New run after being offline for ~10h:
Clientside:
2022-12-28T11:00:44.594228Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some(10), filters: None }], room_subscriptions: {}, unsubscribe_rooms: [], extensions: ExtensionsConfig { to_device: Some(ToDeviceConfig { enabled: Some(true), limit: None, since: None }), e2ee: Some(E2EEConfig { enabled: Some(true) }), account_data: Some(AccountDataConfig { enabled: Some(true) }), other: {} } } config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver=Some("https://slidingsync.lab.element.dev/")}:send{config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver="https://slidingsync.lab.element.dev/" user_id=Some("@matthew:matrix.org") server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] request_type="ruma_client_api::sync::sync_events::v4::Request"}: matrix_sdk::http_client: Serializing request
2022-12-28T11:00:44.594372Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some(10), filters: None }], room_subscriptions: {}, unsubscribe_rooms: [], extensions: ExtensionsConfig { to_device: Some(ToDeviceConfig { enabled: Some(true), limit: None, since: None }), e2ee: Some(E2EEConfig { enabled: Some(true) }), account_data: Some(AccountDataConfig { enabled: Some(true) }), other: {} } } config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver=Some("https://slidingsync.lab.element.dev/")}:send{config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver="https://slidingsync.lab.element.dev/" user_id=Some("@matthew:matrix.org") server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] request_type="ruma_client_api::sync::sync_events::v4::Request"}: matrix_sdk::http_client: Sending request
2022-12-28T11:00:51.699526Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some(10), filters: None }], room_subscriptions: {}, unsubscribe_rooms: [], extensions: ExtensionsConfig { to_device: Some(ToDeviceConfig { enabled: Some(true), limit: None, since: None }), e2ee: Some(E2EEConfig { enabled: Some(true) }), account_data: Some(AccountDataConfig { enabled: Some(true) }), other: {} } } config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver=Some("https://slidingsync.lab.element.dev/")}:send{config=Some(RequestConfig { timeout: 60s, retry_limit: None, retry_timeout: None }) homeserver="https://slidingsync.lab.element.dev/" user_id=Some("@matthew:matrix.org") server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] request_type="ruma_client_api::sync::sync_events::v4::Request"}: matrix_sdk::http_client: Got response: Response { status: 200, version: HTTP/1.1, headers: {"access-control-allow-headers": "Origin, X-Requested-With, Content-Type, Accept, Authorization", "access-control-allow-methods": "GET, POST, PUT, DELETE, OPTIONS", "access-control-allow-origin": "*", "content-type": "application/json", "date": "Wed, 28 Dec 2022 11:00:50 GMT", "transfer-encoding": "chunked", "set-cookie": ".........; path=/; HttpOnly; Secure; SameSite=None"}, body: b"{\"lists\":[{\"ops\":[{\"op\":\"SYNC\",\"range\":[0,20],

Serverside:
11:00:45 INF created new connection conn_id=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 user=@matthew:matrix.org
11:00:46 WRN Client did not increment since token: possibly sending back duplicate to-device events! device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 initial=true last_sent=6956510 recv=0 user=@matthew:matrix.org
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=FAF1AFEB-E0DE-4A1C-8EFE-9A725BFCA605
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=76fa1180-a2ce-4e9b-a384-adffc4bed0d5
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=a6cf24b4-b675-457c-861e-4700fa7ae7fa
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=a8206e85-a317-464b-bba3-79954af724c0
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=fc927ff3-ca16-4a47-a8df-e561249fd674
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=d994ca53-f687-4f8b-8da7-d2db345cfda9
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=164d818f-a404-4128-bf8a-239ac1a24479
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=fbfc3a92-3fe0-4e93-bcdb-eed23251306e
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=e4583817-f443-43cb-bc4d-540cf2e4f126
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=F6F1C103-752C-46A1-929F-91D29F0B22B9
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=be463522-0df9-4dac-a1a9-0f0d05a40d66
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=df7f0af8-01af-4ee4-ba32-dfc38ae2cd45
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=2090c94c-db68-40b5-8de9-bdd983af8d55
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=f80e2966-6b9d-49e9-99fc-dc677c9e3d27
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=2df9b15e-96bb-4c03-a220-caebc8e2f5ca
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=7c91959e-6f36-4848-8b6f-6dfaaef6f70b
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=ed9f0404-5f8c-421a-8fba-f48322514302
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=8b52098a-0879-4cc0-acf2-146310b2e5a3
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=91d299b5-74a3-4523-9ef8-97815a7be201
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=685c4b34-266d-4b60-9a33-7e47eba16a27
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=94AC7AC8-DE15-4600-8D98-BE62F1DC77EE
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=34d62e4a-9847-4dcb-8873-d39805dcddaa
11:00:50 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=878c0399-6d6d-4899-9dc8-2c08e7d4a654
11:00:50 INF  ag=25 d=23 duration=5070.082967 p=0 q=1 r=21 size=399763 status=200 u=@matthew:matrix.org
11:00:52 INF  dl-c=352 dl-l=35 duration=66.935369 p=1 q=2 r=0 size=10312 status=200 u=@matthew:matrix.org
11:00:52 INF  duration=71.452315 p=2 q=3 r=1 size=846 status=200 u=@matthew:matrix.org
11:00:56 INF  duration=3389.829977 p=3 q=4 r=0 size=313 status=200 u=@matthew:matrix.org
11:00:57 INF  duration=36.677605 p=4 q=5 r=1 size=717 status=200 u=@matthew:matrix.org
11:01:05 INF  duration=7519.056685 p=5 q=6 r=1 size=908 status=200 u=@matthew:matrix.org
11:01:05 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=F0BAA31D-3315-4ED6-B005-04C122DD8558
11:01:05 INF  d=1 duration=33.431959 p=6 q=7 r=0 size=366 status=200 u=@matthew:matrix.org
11:01:06 INF  duration=39.089825 p=7 q=8 r=1 size=808 status=200 u=@matthew:matrix.org

Pprof of 5s OnIncomingRequestInitial:
When	Elapsed	Goroutine ID	Events
6.081120159s	5.063043811s		Task 2559 (goroutine view) (complete)
6.081120159	 .         	46984	task OnIncomingRequestInitial (id 2559, parent 0) created
6.081129222	 .     9063	46984	region load started (duration: 693.533809ms)
6.774674218	 .693544996	46984	region onIncomingListRequest started (duration: 28.299495ms)
6.803026782	 . 28352564	46984	region buildRooms started (duration: 631.54459ms)
6.852989439	 . 49962657	46984	new goroutine 46986: database/sql.(*DB).beginDC·dwrap·17
6.855811658	 .  2822219	46984	new goroutine 46971: database/sql.(*Rows).initContextClose·dwrap·27
6.858783687	 .  2972029	46984	region RoomStateAfterEventPosition started (duration: 551.195066ms)
6.859238189	 .   454502	46984	new goroutine 46987: database/sql.(*DB).beginDC·dwrap·17
6.862468574	 .  3230385	46984	new goroutine 46972: database/sql.(*Rows).initContextClose·dwrap·27
6.863682627	 .  1214053	46984	new goroutine 46973: database/sql.(*Rows).initContextClose·dwrap·27
7.435256453	 .571573826	46984	region extensions started (duration: 3.708890621s)
11.120415767	3.685159314	46984	new goroutine 46988: database/sql.(*DB).beginDC·dwrap·17
11.123099899	 .  2684132	46984	new goroutine 46996: database/sql.(*DB).beginDC·dwrap·17
11.134975984	 . 11876085	46984	new goroutine 46997: database/sql.(*Rows).initContextClose·dwrap·27
11.136070844	 .  1094860	46984	new goroutine 46977: database/sql.(*DB).beginDC·dwrap·17
11.141054725	 .  4983881	46984	new goroutine 46998: database/sql.(*Rows).initContextClose·dwrap·27
11.144153909	 .  3099184	46984	region liveUpdate started (duration: 5.683µs)
11.144163970	 .    10061	46984	task end
GC:0s

So, 5s to generate a response on the server, resulting in a 7s delay on the client.

region extensions started (duration: 3.708890621s) probably device events

We may be reaching database limits at this point, given how sporadic the load times are (42 or 693ms to region load load the room list is suspicious)

region extensions started (duration: 3.708890621s) probably device events

but there's only ~22 to-device events?

Yes but it needs to query a to device events table that was at last check 7 million rows.

Here's another excruciating 37s one:
2022-12-31T10:00:28.495451Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some>
2022-12-31T10:00:28.496058Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some>
2022-12-31T10:01:06.848306Z TRACE send_inner{request=Request { pos: None, txn_id: None, timeout: Some(30s), lists: [SyncRequestList { slow_get_all_rooms: false, ranges: [(0, 20)], sort: ["by_recency", "by_name"], required_state: [(RoomAvatar, ""), (RoomEncryption, "")], timeline_limit: Some>

and
10:00:28 INF created new connection conn_id=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 user=@matthew:matrix.org
10:00:29 WRN Client did not increment since token: possibly sending back duplicate to-device events! device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 initial=true last_sent=7250544 recv=0 user=@matthew:matrix.org
10:01:06 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=6a890877-9f58-4f5f-8414-62c5fbbc0974
10:01:06 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=b52d1413-7266-40ec-b020-ed1df9a42224
10:01:06 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=95d500b8-d8a9-4b48-8b28-ee4aa48062e4
10:01:06 INF ToDeviceTable.Messages device=ec725749c89702fe5e1fc5e019e77ce22dc0f3ae0717483f307a02e2222c6631 msgid=7bd978c0-f431-47f0-aeb3-625a0f4f367b
10:01:06 INF  ag=25 d=4 dl-c=1 duration=37591.21913 p=0 q=1 r=21 size=347050 status=200 u=@matthew:matrix.org

Presumably to fix this:

We need to actually delete the to-device messages to the device which the device has acknowledged receiving(!)
Or we could at least slap an index on the db. 7m rows is nothing.

We need to actually delete the to-device messages to the device which the device has acknowledged receiving(!)

We've always done this.

Or we could at least slap an index on the db. 7m rows is nothing.

We have indexes.
After much analysis, the conclusions:

add another index for position, device_id - this was done in https://github.com/matrix-org/sliding-sync/commit/f47e71c726839ae94d068bc6aadbf4e1e19ae3f8
Make sure clients (EX) persist the next_batch token between sessions. If you keep sending up since=0 then postgres has to scan a lot more data than is required, adding ~500ms to the average timings.

The very slow 37s one was also caused by the to-device table, but it's hard to know if these conclusions will be enough to resolve them. Clearing out the to-device table of stale msgs (deleted devices, etc) would also help constrain growth.

Closing this as we have done many performance improvements for the initial sync:

region RoomStateAfterEventPosition now takes <100ms, as opposed to 500-1000ms. https://github.com/matrix-org/sliding-sync/commit/00e4b8238ce144054d93ab75a43785f63674990b
region extensions now has a better to_device index. https://github.com/matrix-org/sliding-sync/commit/f47e71c726839ae94d068bc6aadbf4e1e19ae3f8

Further gains can be made (thanks Stefan!) by:

Batching up TransactionIDForEvents to be a single query for all returned rooms, currently it's one query per room.
Batching up SelectClosestPrevBatchByID to be a single query for all returned rooms, currently it's one query per room.

This will save 50-100ms more.


FileSystem shows custom Resource icon of first external dependancy instead of correct icon

Tested versions
v4.3.beta2.official [b75f0485b]
System information
Godot v4.3.beta2 - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2060 (NVIDIA; <IP_ADDRESS>23) - Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz (4 Threads)
Issue description
If a saved custom resource file contains an exported subresource, and that subresource script appears first in the list of dependencies, then the subresource icon will be shown in the FileSystem dock instead of the customized resource icon.
It looks like there's an assumption that the resource's own script would always be the first listed external dependency, but that's not the case, the order seems to shuffle unpredictably whenever the file is overwritten. Manually correcting the order in the .tscn file fixes the problem temporarily, but doesn't prevent the editor from suddenly changing it again later on.
Only the FileSystem dock is affected, the correct icon is still shown elsewhere in the editor.
This file shows the correct custom resource icon:
[gd_resource type="Resource" script_class="CustomResource" load_steps=4 format=3 uid="uid://becgbjt2xt25t"]

[ext_resource type="Script" path="res://scripts/custom_resource.gd" id="1_1ahaa"]
[ext_resource type="Script" path="res://scripts/sub_resource.gd" id="2_ra8cr"]

[sub_resource type="Resource" id="Resource_5gvxf"]
script = ExtResource("2_ra8cr")
some_value = 0

[resource]
script = ExtResource("1_1ahaa")
sub_resource = SubResource("Resource_5gvxf")

This file mistakenly shows the icon of the subresource, note the ext_resource order:
[gd_resource type="Resource" script_class="CustomResource" load_steps=4 format=3 uid="uid://dlvvxjopcqthw"]

[ext_resource type="Script" path="res://scripts/sub_resource.gd" id="2_lm6jx"]
[ext_resource type="Script" path="res://scripts/custom_resource.gd" id="1_efmpu"]

[sub_resource type="Resource" id="Resource_peacu"]
script = ExtResource("2_lm6jx")
some_value = 1

[resource]
script = ExtResource("1_efmpu")
sub_resource = SubResource("Resource_peacu")

Steps to reproduce
I couldn't find steps that would predictably make the editor reverse the dependency order of a file, but it's happening to me frequently with normal editing, so I assume the order just isn't guaranteed.
Minimal reproduction project (MRP)
file_icon_subresource_issue.zip

I look a bit into this problem and i don't see an easy way to use the dependencies correctly to find the right custom icon. If we loop on all deps, how can we find the right one?
I suggest using the script class which is already available in EditorFileSystemDirectory::FileInfo.resource_script_class. Based on this class name, the icon path should be available with EditorData::script_class_get_icon_path. I think that before 4.3 it was not mandatory to have a class name for custom resource with icon, but how, I think it is? Maybe a fallback could be to use the first dependency when the resource does not have a class name?

Seems reasonable to me to for icons to require a class_name. In that case I think it would be better not to use the first dependency as a fallback, since it can give misleading results, and show a warning instead.
Tested with v4.3.beta2.official [b75f0485b], no changes to this behavior from #93147.


if we only use mask LM in training and disable the'next sentence', how should I modify the create_pretraining_data.py

in pre-training code, I figure out just disable the 'next sentence loss' code

change line 274 from "if len(current_chunk) == 1 or rng.random() < 0.5" to "if len(current_chunk) == 1" may help your problem.  all next_sentence labels are 0 now.


adding security scopes to dashboard

Detailed Description
This issue is the beginning of a discussion of how to add authorization to the dashboard.

Discuss how to add auth0 scopes to the app
Add warning to if we are adding a non ocf user to the ocf group.
Spike - find popup component to use in app to double check user agrees to an action (ie. update, delete, etc.)
Currently users who might add users or add sites are Peter, Brad, Sol, Zak, and Rachel

Context
The main users of the ocf dashboard will be internal, but it's good to think about who will have permission to do what. If we're deleting a site, it would be good that only certain people have that authorization.

https://auth0.com/blog/streamlit-user-and-api-authentication/

this simple version looks good - https://github.com/conradbez/streamlit-auth0-example/blob/main/test.py

`from auth0_component import login_button
import streamlit as st
clientId = "...."
domain = "...."
user_info = login_button(clientId, domain = domain)
st.write(user_info)`

thanks, @peterdudfield, I'll have a go at this. did you try anything already?

See comment here for trying to logout - https://github.com/conradbez/streamlit-auth0/issues/18

Might be better to use https://github.com/chris-garrett/garrett-streamlit-auth0

Might be better to use https://github.com/chris-garrett/garrett-streamlit-auth0

This one works


Added new module: Unit conversion

Closes #66
Unit conversion API added, used API: https://rapidapi.com/neutrinoapi/api/convert-1

@adityabisoi Solve the conflicts.

@TeAmP0is0N done.

Needs a corresponding PR in jarvis-core.


feat(update-notifier): Add update notifer

Every time the cli si run update notifier will check for updates with
the default options.

Addresses #5

Hey @dnlup,
Thanks for the PR 😊
It looks good to me, except that it doesn't follow the formatting styles used by the project.
To fix that, pls run
$ ./node_modules/.bin/prettier --write src/index.js

And push those changes, we can merge this then.

@jaydp17 Sorry :smile: , updated it.


Removed seed-isort-config repo as its deprecated

Possible fix for : #60

Thanks a lot, @pkesavap!

Just a quick tip: if you write fixes #60 or closes #60 the issue gets automatically closed one the pull requests gets accepted.
see https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
possible fix is not yet on the list! It's about time!! :-)

Just a quick tip: if you write fixes #60 or closes #60 the issue gets automatically closed once the pull requests gets accepted.
see https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
possible fix is not yet on the list! It's about time!! :-)

wow, i did not know that , thanks for the link,

@jugmac00  sorry to bother you, can you label the PR with hacktoberfest tag, its not reflecting in my profile , wonder why,
\thanks

For #hacktoberfest only pull requests between Oct 1st and Oct 31st are counted :-/
But there are many more...
https://github.com/search?q=org%3Amorepath+hacktoberfest&type=issues

For #hacktoberfest only pull requests between Oct 1st and Oct 31st are counted :-/
But there are many more...
https://github.com/search?q=org%3Amorepath+hacktoberfest&type=issues

oh no worries , :D


Spyder says it's on version 5.4.0 even though updated to 5.4.3

Issue Report Checklist

[Y] Searched the issues page for similar reports
[Y] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[Y] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[ ] Could not reproduce inside jupyter qtconsole (if console-related)
[ ] Tried basic troubleshooting (if a bug/error)

[Y] Restarted Spyder
[Y] Reset preferences with spyder --reset
[Y] Reinstalled the latest version of Anaconda
[Y] Tried the other applicable steps from the Troubleshooting Guide

[Y] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
I got this very strange problem: I installed spyder via pacman before, and I decided to remove it and installed a conda version instead. I remove the old spyder and install conda, which comes with a 5.4.3 spyder. conda list shows
spyder                    5.4.3           py310h06a4308_1

But when I start spyder, it still notify me to update to 5.4.3:

And the version shows it's still on 5.3.0:

I removed anaconda and reinstall multiple times but the problem still exist. I also make sure that the spyder I started is from conda, by go in the ~/miniconda3/bin/ and start it directly, none of it helped. I'm really confused and don't know what to do now.
 Could you check what sys.executable returns inside Spyder and what it returns from the terminal where you used the conda list  command? So from Spyder, run in the IPython console something like:
import sys
sys.exectuble

and from the cmd something like:
python -c "import sys; print(sys.executable)"

Are the results shown different in some way?
Not totally sure what could be happening :thinking but any new info in order to better understand this issue is greatly appreciated. Let us know!

Hi @dalthviz, thanks for the reply! 'python -c "import sys; print(sys.executable)"' returns:
/home/user/miniconda3/bin/python

In the IPython console sys.executable return
sys.executable
Out[5]: '/home/user/miniconda3/bin/python'

If you want to know any more info just let me know!

I don't know if this is related but I just found out my spyder can't open DataFrame in variable explorer and gives an error:
Traceback (most recent call last):
  File "/home/sphaela/.local/lib/python3.10/site-packages/spyder/plugins/variableexplorer/widgets/collectionsdelegate.py", line 203, in createEditor
    if not editor.setup_and_check(value, title=key):
  File "/home/sphaela/.local/lib/python3.10/site-packages/spyder/plugins/variableexplorer/widgets/dataframeeditor.py", line 946, in setup_and_check
    self.dataModel = DataFrameModel(data, parent=self)
  File "/home/sphaela/.local/lib/python3.10/site-packages/spyder/plugins/variableexplorer/widgets/dataframeeditor.py", line 135, in __init__
    self.max_min_col_update()
  File "/home/sphaela/.local/lib/python3.10/site-packages/spyder/plugins/variableexplorer/widgets/dataframeeditor.py", line 244, in max_min_col_update
    for __, col in self.df.iteritems():
  File "/home/sphaela/miniconda3/lib/python3.10/site-packages/pandas/core/generic.py", line 5989, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'iteritems'. Did you mean: 'isetitem'?

Thank you for the new info @JGSphaela ! The traceback you are seeing is related with https://github.com/spyder-ide/spyder/issues/20496 (which is an issue that was solved for Spyder 5.4.3).
However, something interesting is that the path shown related to Spyder doesn't point to your miniconda installation but instead to you .local directory in your user home  directory. Did you install Spyder in your machine before using pip with the --user flag? I think the problem comes from miniconda launching Spyder from your ~/.local directory? What happens if you import spyder and check print spyder? Is the path shown pointing to ~/.local?
If that is not the case I think you will need to remove the Spyder installation at ~/.local
Let us know if the info above helps!

Hi @dalthviz, that's exactly the cause of the problem! I didn't know I have the spyder installed with pip before, I removed it using pip uninstall spyder outside conda and now everything works! Thank you so much!


Update diffx-scalatest to 0.5.4

Updates com.softwaremill.diffx:diffx-scalatest from 0.4.5 to 0.5.4.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.softwaremill.diffx", artifactId = "diffx-scalatest" } ]

labels: test-library-update, semver-minor

Superseded by #286.


Multiple instances of lua-language-server (sumneko_lua) are running when I open a single file

nvim --version:

NVIM v0.5.0-1869f86e3
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/clang -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNDEBUG -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/tmp/neovim-20200809-42220-drihg/build/config -I/tmp/neovim-20200809-42220-drihg/src -I/usr/local/include -I/tmp/neovim-20200809-42220-drihg/deps-build/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/usr/local/opt/gettext/include -I/tmp/neovim-20200809-42220-drihg/build/src/nvim/auto -I/tmp/neovim-20200809-42220-drihg/build/include
Compiled by ahmed@pandoras-box

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/local/Cellar/neovim/HEAD-1869f86/share/nvim"

Run :checkhealth for more info

nvim-lsp version(commit hash): 3f72236

:checkhealth

health#targets#check
========================================================================
  - OK: No conflicting mappings found

health#nvim_lsp#check
========================================================================
## Checking language server protocol configuration
  - INFO: sumneko_lua: configuration checked.
  - INFO: ocamlls: configuration checked.
  - INFO: yamlls: configuration checked.
  - INFO: pyls: configuration checked.
  - INFO: cssls: configuration checked.
  - INFO: rls: configuration checked.
  - INFO: vimls: configuration checked.
  - INFO: jsonls: configuration checked.
  - INFO: tsserver: configuration checked.
  - INFO: bashls: configuration checked.

health#gitmessenger#check
========================================================================
  - OK: +job is available to execute Git command
  - OK: Git command `git` is available: git version 2.28.0
  - OK: Floating window is available for popup window

health#completion_nvim#check
========================================================================
## general
  - OK: neovim version is supported

## completion source
  - OK: all completion source are valid

## snippet source
  - OK: You are using vim-vsnip as your snippet source

health#nvim#check
========================================================================
## Configuration
  - OK: no issues found

## Performance
  - OK: Build type: Release

## Remote Plugins
  - OK: Up to date

## terminal
  - INFO: key_backspace (kbs) terminfo entry: key_backspace=\177
  - INFO: key_dc (kdch1) terminfo entry: key_dc=\E[3~
  - INFO: $COLORTERM='truecolor'

## tmux
  - OK: escape-time: 0
  - INFO: Checking stuff
  - OK: focus-events: on
  - INFO: $TERM: tmux-256color

health#provider#check
========================================================================
## Clipboard (optional)
  - OK: Clipboard tool found: pbcopy

## Python 2 provider (optional)
  - INFO: Using: g:python_host_prog = "/usr/local/bin/python2"
  - INFO: Executable: /usr/local/bin/python2
  - INFO: Python version: 2.7.17
  - INFO: pynvim version: 0.4.1
  - OK: Latest pynvim is installed.

## Python 3 provider (optional)
  - INFO: Using: g:python3_host_prog = "/usr/local/bin/python3"
  - INFO: Executable: /usr/local/bin/python3
  - INFO: Python version: 3.8.5
  - INFO: pynvim version: 0.4.1
  - OK: Latest pynvim is installed.

## Python virtualenv
  - OK: no $VIRTUAL_ENV

## Ruby provider (optional)
  - INFO: Disabled (g:loaded_ruby_provider=0).

## Node.js provider (optional)
  - INFO: Node.js: v14.7.0
  - INFO: Nvim node.js host: /usr/local/bin/neovim-node-host
  - OK: Latest "neovim" npm/yarn package is installed: 4.9.0

## Perl provider (optional)
  - INFO: Disabled (g:loaded_perl_provider=0).

What language server(If the problem is related to a specific language server): sumneko_lua
Operating system/version: macOS 10.15.6
Terminal name/version: Kitty 0.18.2
$TERM:

Inside TMUX: tmux-256color
Outside TMUX: xterm-kitty

How to reproduce the problem from neovim startup
Open any .lua file, in my case it's this file https://github.com/ahmedelgabri/dotfiles/blob/e71e9ac3d319cc0af909e7156212ae445de27a10/roles/vim/files/.vim/lua/lsp.lua
Actual behaviour

It locks the editor for a couple of seconds
Runs multiple instances of the LSP
Sometimes even can consume 90+% CPU (can't reproduce this now, but saw it happening multiple times)

Expected behaviour

Run async if possible & don't lock my editor
Run a single instance of the LSP

I seem to recall @norcalli mentioning that sumneko server had various problems. Maybe he can comment on his current setup for Lua LSP

I'm having the same issue

They don't shut down after closing neovim too

I think this has been improved since then? I never experienced it anymore and they always shut down after closing Neovim. I also tried to open 4 files and it only launch 1 instance of lua-language-server. Though, the LSP sometimes uses a ton of my CPU but it never locks my Neovim, it's still usable. After several seconds, it went back to normal.

I think this has been improved since then? I never experienced it anymore and they always shut down after closing Neovim. I also tried to open 4 files and it only launch 1 instance of lua-language-server. Though, the LSP sometimes uses a ton of my CPU but it never locks my Neovim, it's still usable. After several seconds, it went back to normal.

I personally haven't noticed this in awhile, I also feel like this is an upstream issue and we can't/shouldn't try to implement more advanced process management for language servers that refuse a shutdown command. The only thing we could try to do is store the PID and pkill it when it fails, but if the language server crashes (as they do), I don't know that we can reliably identify that and not fill whatever process ends up using the same PID.

I personally haven't noticed this in awhile, I also feel like this is an upstream issue and we can't/shouldn't try to implement more advanced process management for language servers that refuse a shutdown command. The only thing we could try to do is store the PID and pkill it when it fails, but if the language server crashes (as they do), I don't know that we can reliably identify that and not fill whatever process ends up using the same PID.

I didn't notice this happening either for a while & after merging this #498 I don't think it makes sense to keep this open anymore.

I didn't notice this happening either for a while & after merging this #498 I don't think it makes sense to keep this open anymore.

Unfortunately a lot of bugs seem to be caused by having older language servers, and before when we were managing installs via neovim they would become rather out of date 😞  I'm glad sumneko seems to be working now!

Unfortunately a lot of bugs seem to be caused by having older language servers, and before when we were managing installs via neovim they would become rather out of date 😞  I'm glad sumneko seems to be working now!

@mjlbach Thank you!

Apologies to pull this back up, but this issue is still lurking. It also occurs with Null-ls and Pyright.
Im running sumneko_lua (via brew) 2.6.7 and still have the multiple-spawn issue occurring. Anytime I open new files within the same directory, or a same project, a new LSP is spawned. The picture is much like the two posted beforehand.

I've been experiencing the same issue, except a new LSP is spawned whenever the file is written to (e.g. :w).

yep, it still happens to me too. If I work on my lua config, my RAM gets full in 10 minutes and OOM killer triggers.

:lua vim.lsp.stop_client(vim.lsp.get_active_clients())
This command got rid of the multiple instances for me. It's just a temporary workaround to get rid of the instances while working on the config but it gets rid of the annoying instances.
Multiple instances occurred when I sourced the file :so %

:lua vim.lsp.stop_client(vim.lsp.get_active_clients())
This command got rid of the multiple instances for me. It's just a temporary workaround to get rid of the instances while working on the config.
Multiple instances occurred when I sourced the file :so %

:lua vim.lsp.stop_client(vim.lsp.get_active_clients())
This command got rid of the multiple instances for me. It's just a temporary workaround to get rid of the instances while working on the Lua file.
Multiple instances occurred when I sourced my lsp config file.

:lua vim.lsp.stop_client(vim.lsp.get_active_clients())
This command got rid of the multiple instances for me. It's just a temporary workaround to get rid of the instances while working on the Lua file.
Multiple instances occurred when I sourced my lsp config file.

Thank you for your tip @histeward !
I ended up using this key map in my init.lua, so I can reload my settings quickly and avoid multiple LSP servers for the same file type from running:
vim.keymap.set("n", "<leader><CR>", ":execute 'lua vim.lsp.stop_client(vim.lsp.get_active_clients())' | luafile ~/.config/nvim/init.lua<CR>", {noremap = true})

@ahmedelgabri
I myself discovered this issue today while I was trying to set up nvim-lspconfig. But because all my plugin configurations are inside packer.nvim's plugin specification file (plugins.lua), I managed to solve this by adding a small chunk of logic that would skip the setup() call for sumneko_lua if there is already an instance of it attached to the buffer.
Here's what it would look like in your plugins.lua file:
vim.cmd([[
      augroup packer_user_config
          autocmd!
          autocmd BufWritePost plugins.lua source <afile> | PackerCompile
      augroup end
  ]])

...Your other config...

--Setting up language servers
local lspconfig = require("lspconfig")
local start_sumneko_lua = true --boolean defaults to true, otherwise, the server will never start
local current_buf_id = vim.api.nvim_get_current_buf()
local servers_attached_to_current_buf = vim.lsp.get_active_clients({bufnr = current_buf_id})

for _, server in ipairs(servers_attached_to_current_buf) do
    if server.name == "sumneko_lua" then --an instance of sumneko_lua is already attached to the buffer
        start_sumneko_lua = false
    end
end

--After that, all you have to do is to wrap sumneko lua's setup() call inside a conditional using the boolean

if start_sumneko_lua then
    lspconfig.sumneko_lua.setup{
         -- yada yada yada
    }
end

--Other servers don't need this, so their setup can be called as usual
lspconfig.pyright.setup{

}

--yada yada yada...

The idea is to wrap an if-conditional around the lspconfig.sumneko_lua.setup() call and then decide whether or not to start the language server based on a "flag." Turning off single_file_support is certainly a quick and dirty way to make this issue go away, but sometimes we need single_file_support as well. If that's what you need, then this is one way to do it. I've tested it on my setup and haven't run into issues so far. The code seems to be working fine for now.

In my case, a new sumneko_lua LSP client is spawned on different root directory, even with the single_file_support set to true(this is set upon nvim-lspconfig):


Mask subscription ID in all JSON

Sometimes a sub id makes it way into the JSON responses, must be masked out.

Fixed in function, can be closed.


Volvo integration is broken

Volvo has discontinued v1 of the API ('error': {'message': 'GONE', 'description': 'Version 1 is removed.) The module no longer returns any values :-(
I'm already updating the code to support API v2...

Thanks!


fix: make TGet covariant on Attribute stub

[x] Added or updated tests

Changes
Hello!
I'm using the PHPStan bleeding edge config, and I started getting the following errors with the Model Attributes:
    /** @return Attribute<string|null, never> */
    protected function surveyedName(): Attribute
    {
        return Attribute::get(function (): ?string {
            if (! $this->surveyable_type) {
                return null;
            }

            return Str::of($this->surveyable_type)
                ->replace('.', ' ')
                ->title()
                ->toString();
        });
    }

  148    Method <class>::surveyedName() should return Illuminate\Database\Eloquent\Casts\Attribute<string|null, never> but returns
         Illuminate\Database\Eloquent\Casts\Attribute<string|null, never>.
         🪪  return.type
         💡 Template type TGet on class Illuminate\Database\Eloquent\Casts\Attribute is not covariant. Learn more: https://phpstan.org/blog/whats-up-with-template-covariant

This PR updates the TGet template to be covariant and fixes some other issues with the stub.
Thanks!
Breaking changes
None

@szepeviktor, yes I believe that should be fine, the bleeding edge config is a sneak peak at the next version of phpstan

AFAIR this was a bug in PHPStan. Making the type covariant is not the correct solution. Closing it for now.


No Internet Connection Results page

What problem does this feature solve?
if a user doesnt have network connection have a result page to show lack of connectivity
What does the proposed API look like?
<nz-result nzStatus="Network" nzTitle="No Internet" nzSubTitle="Please check your network connectivity">       <div nz-result-extra>         <button nz-button nzType="primary">Retry</button>       </div>     </nz-result>

Hi. The requested feature needs to be designed by our friends from antd. Would you please open an issue there?

Closed due to inactivity. Please open an issue on antd first since this feature requires a new illustration.


[Select] Popup menu width does not align with the control width

With the newly added Select control, the default pop up width of the contained MenuItems does not align with the control:

And there does not seem to be simple way to make them align aside from using a native input.
I think in this respect the Select control should behave like SelectField in version 0.19 by default.

Tech
Version

Material-UI
1.0.0-beta.11

MenuItems does not align with the control:

What do you mean by aligning? Please follow the issue template. The expected part is too vague.

Sorry for not making myself clear enough.
By 'aligning' I mean my expected behavior of a popup menu brought up by clicking the trigger in the Select control should be that:

The left side of the menu is in line with the left side of the Select control,
The width of the menu shall be the same as the width of the Select control so that
The right edge of the menu is also in line with the right side of the right side of the Select control
Except when the width of the menu item exceeds the width of the Select control.

So in the previous example from the Select demo, the menu shall have the size of the red box below:

In the 0.1x release as well as in current demo of Select control with native control in 1.0 release, the width and position of the popup menu is just as I expected.
Screenshot of SelectField of 0.19:

Actually the current Select control of 1.0.0 behaves like SelectField of 0.19 with autoWidth set to true:

I'm currently looking for a fix. While we're at it, the menu should also align so that the text is at the same location as the text in the field, like shown here at "Mobile Dropdowns"

I'm currently working on a fix.

👍

...the menu should also align so that the text is at the same location as the text in the field, like shown here at "Mobile Dropdowns".

In your link, the text of the selected item is aligned vertically with the value presented in the component when the dropdown is not visible.  However, horizontally, it is not in the same position.  You can see this by using the field label as a reference.  It is shifted right by the size of the dropdown's left gutter.  This is more or less what the Select's dropdown does today when it is not in native mode.


Add book recommendation for mentor : Olga Volkova

Task
Add recommendations for Olga Volkova in Resources section.
(Use the same format that has been used for other mentors e.g. Eleonora)
Books
Designing Machine Learning Systems
Machine Learning System Design Interview
Resources tab

Acceptance criteria
Screenshot of the resources tab

Could I please also add one more book to the list:
Designing Data-Intensive Applications


Fixed config merge (array of objects) and added options.configOutput.

@src-code

Looks good, assuming that arrays are the only data type that need special handling in merge()...

For now that's the only problem I found. I'll do another PR if I can find any other issues with the config merges.


Inputs to be specified and designated to the chose date, for the read mode.

From local storage, each task is going to be shown only if the related date is chosen.

Done


Make repository pip installable

Turning the repository into a pip installable package to facilitate code development.

@aflaxman @rmudambi We should create a template to enable doing this automatically for all our "research" repositories, similar to the template vivarium_research_template used by the SWE's for all the Vivarium models.


Stream-k initial support

This PR introduces initial stream-k support into hipBLASLt. SGEMM, HGEMM, and BGEMM have been validated with a large number of parameter combinations and problem sizes, and this PR includes some new CI tests for these problem types. The stream-k algorithm helps deliver improved and stable performance with smaller library size. For full details, see the presentations or paper.
Most of the code changes for stream-k are isolated in a component class to help with code readability. Since stream-k is not compatible with GSU, all the GSU related code has also been moved from the KernelWriter classes into it's own component class, and an additional setting has been added to allow generating kernels with GSU disabled (GSU=0 disables it, GSU=1 is still the default, a new CI test for both settings is included). A component for persistent looping has also been added since stream-k operates on a loop. Descriptions of how to use the stream-k options and run-time environment variables are included in the comments in Common.py.
Full stream-k support in hipBLASLt is still a work in progress, and this PR is meant to introduce the initial core functionality. There are a number of known issues that I will work in in future PRs and there are tickets to track, including:

Support and validation for all data types
Consolidate some of the fixup and partials write procedures with the general global write module
Compare and consolidate the stream-k XCC mapping feature with the WorkGroupMappingXCC option
Group gemm support
Additional plans for code improvement and porting of more stream-k features from Tensile repo are described in tickets

I tried HHS TN + StreamK: [0,1,2,3] but I see some issues.
(1) warnings
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool0 still in use
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool1 still in use
TensileInstructions::WARNING: RegisterPool::add(0,149) pool0 already available
TensileInstructions::WARNING: RegisterPool::add(0,149) pool1 already available
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool0 still in use
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool1 still in use
TensileInstructions::WARNING: RegisterPool::add(0,149) pool0 already available
TensileInstructions::WARNING: RegisterPool::add(0,149) pool1 already available
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool0 still in use
TensileInstructions::WARNING: RegisterPool::remove(0,95) pool1 still in use
TensileInstructions::WARNING: RegisterPool::add(0,149) pool0 already available
TensileInstructions::WARNING: RegisterPool::add(0,149) pool1 already available
(2) error
File "tensilelite/Tensile/Components/StreamK.py", line 1792, in storeBranches
module.add(self.storeBranchesCommon(writer, kernel, skPartialsLabel, vectorWidths, element, tmpVgpr, cvtVgprStruct))
NameError: name 'element' is not defined
This is the folkparameters I tried.
- 1LDSBuffer: [1]#[0,1]
- DepthU: [64]#[32, 64, 128, 256]
- MatrixInstruction:
- [16, 16, 16,  1,  1,  4,  8,  4,  1] # MT 256x128
- PrefetchGlobalRead: [2]#[1,2]#[2]
- PrefetchLocalRead: [1]
- SourceSwap: [1]#[0,1]
- StaggerU: [0]#[0,8]#[0,2,4]
- StreamK: [0,1,2,3]

Please make sure all CI tests pass on your local node

tensilelite tox
hipblaslt-test

Thanks for the notes. I fixed issue #2 as well as a bug in the xccmap. Still working on debugging issue #1.

Thanks for your update.
Since "noCI" label is added, please run all CI tests (tensilelite tox + hipblaslt-test) on your local node and paste the result here.

Finished running tensilelite tox tests, working on hipblaslt-test
=================================================================================== 75 passed, 31 skipped, 44 warnings in 7744.02s (2:09:04) ====================================================================================
py310: exit 0 (7744.30 seconds) /home/stebrown/hipBLASLt2/tensilelite> py.test -v --basetemp=/home/stebrown/hipBLASLt2/tensilelite/.tox/py310/tmp --junit-xml=/home/stebrown/hipBLASLt2/tensilelite/python_tests.xml --junit-prefix=py310 --color=yes -n 4 --prebuilt-client=/home/stebrown/hipBLASLt2/tensilelite/.tox/py310/client/0_Build/client/tensile_client ./Tensile/Tests/ -m common pid=1141334
py310: OK (7888.08=setup[14.73]+cmd[1.80,127.24,7744.30] seconds)
congratulations :) (7888.16 seconds)

All hipblaslt-test test cases passing as well:
[----------] Global test environment tear-down
[==========] 48206 tests from 13 test suites ran. (2278421 ms total)
[  PASSED  ] 48206 tests.


fixed incorrect logging order

What type of PR is this?
/kind code-refactoring
What does does this PR do / why we need it:
I found this bit of logging order a bit disinforming.
"\nPushing devfile component %s" should be logged before the actually Pushing logic starts. not after that.

looks good to me.
/lgtm

/hold
@dev-gaur the part of code being moved here is specific to else block. What's being proposed as a part of this PR is to log it for both if and else blocks. I get it that it's just some logging so not a big deal. But, IMO, it's best to do git blame and discuss this quickly with the original contributor.

/hold
@dev-gaur the part of code being moved here is specific to else block.

^ Which was wrong in the first place, right?

/hold
@dev-gaur the part of code being moved here is specific to else block.

^ Which was wrong in the first place, right?

TBH, I'm not sure. :disappointed:
@jgwest can you PTAL since it looks like you added the code that @dev-gaur thinks needs a change.

@dev-gaur I agree with your concern but with your PR  the line ✓  Changes successfully pushed to component  is looking misplaced. i believe the log Pushing devfile component nodejs is to denote success not process.
$ odo push

Pushing devfile component nodejs

Validation
 ✓  Validating the devfile [46035ns]

Creating Kubernetes resources for component nodejs
 ✓  Waiting for component to start [4ms]

Applying URL changes
 ✓  URLs are synced with the cluster, no changes are required.

Syncing to component nodejs
 ✓  Checking file changes for pushing [523446ns]
 ✓  No file changes detected, skipping build. Use the '-f' flag to force the build.
 ✓  Changes successfully pushed to component

$ odo push
Validation
 ✓  Validating the devfile [50844ns]

Creating Kubernetes resources for component nodejs
 ✓  Waiting for component to start [1ms]

Applying URL changes
 ✓  URLs are synced with the cluster, no changes are required.

Syncing to component nodejs
 ✓  Checking file changes for pushing [526513ns]
 ✓  Syncing files to the component [109ms]

Executing devfile commands for component nodejs
 ✓  Executing install command "npm install", if not running [3s]
 ✓  Executing run command "npm start", if not running [1s]

Pushing devfile component nodejs
 ✓  Changes successfully pushed to component

@jgwest can you PTAL since it looks like you added the code that @dev-gaur thinks needs a change.

Re: the specific ordering of these blocks, it wasn't I that added/changed them, I preserved the behaviour of that code as I found it (https://github.com/openshift/odo/commit/e28812a34d4a2443d1e816141c401e3562a5c6ae)

@dev-gaur https://github.com/openshift/odo/pull/3814#issuecomment-679975107 makes me think that the change you've proposed is likely not required. Unless there's some reasoning to take this further, we can close this PR. WDYT?

@adisky thanks for pointing that out.. I didn't realise this small a change would disrupt the logging order on that scale.
@dharmit sure.. if this is trivial then we can close this.


Add document column blocks

1. Why is this change necessary?
There is no way of appending or prepending data to the default columns in document positions - e.g. append to label column.
2. What does this change do, exactly?
Add new blocks inside of table columns
3. Describe each step to reproduce the issue or behaviour.
Try extending document position label column, you either have to add a new column or overwrite the existing one
4. Please link to the relevant issues (if any).
N/A
5. Checklist

[x] I have rebased my changes to remove merge conflicts
[ ] I have written tests and verified that they fail without my change
[x] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/NEXT-24107
Please use this issue to track the state of your pull request.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/NEXT-24211
Please use this issue to track the state of your pull request.


Reimplementation of  Blocks

As mentioned in #550, this is the final PR needed for fixing the biggest issues with blocks.

This simplifies handling of blocks by more explicitly storing them, depending on how they are encountered while parsing. There are three "types" of blocks when it comes to parsing:

Defined Blocks - These are blocks which are defined using the {% block... %} tag. They are stored on the Twig.Template instance where the block tag was parsed. They are stored with a reference to this template.
Imported Blocks - These are blocks which are imported using the {% use... %} tag. This includes all the blocks resolved from the template referenced by the use tag. They are stored on the Twig.Template instance where the use tag was parsed.
Override Blocks - These are blocks which are passed when creating a Twig.ParseState instance. This is used internally to render a parent template referenced by an {% extends... %} tag. Any blocks defined in the template where the extends tag was parsed, will be passed to the Twig.ParseState instance used when rendering the parent template in order to override properly.

The logic for determining what to output when using a {% block %} tag, block() function, or parent() function is now better defined and more consistent. Much of the logic and complexity was removed from the associated parse functions in twig.logic.js and twig.function.js. There are now four methods which handle this logic internally:

Twig.Template.getBlock - This method attempts to find the first block matching a specified name on the Twig.Template instance, in the following order:

Defined Block
Imported Block

Twig.Template.getBlocks - This method returns an object with all the available blocks on the Twig.Template instance, using the following precedence:

Defined Blocks
Imported Blocks

Twig.ParseState.getBlock - This method attempts to find the first block matching a specified name in the following order:

Override Block
Defined Block or Imported Block resolved from the associated template using Twig.Template.getBlock
When extending, Defined Block or Imported Block resolved from the parent template using Twig.Template.getBlock

Twig.ParseState.getBlocks - This method returns an object with all the currently available blocks using the following precedence:

Override Blocks
Defined Blocks or Imported Blocks resolved from the associated template using Twig.Template.getBlocks
When extending, Defined Blocks or Imported Blocks resolved from the parent template using Twig.Template.getBlocks

The output option has been removed from the params argument on the Twig.Template.render method (which accepted blocks or macros as a value) due to unreliability and it no longer being used internally. This is a BREAKING CHANGE.
This includes tests for, and fixes issues #257, #382, and #436.

Any updates here?

I'm planning on getting this rolling again when I have a little more time, hopefully in January.

Anything needing being done here?

There's nothing else that I have on my list to do. This can be merged. I can then do a PR to bring this branch inline with the master branch, allowing a new version to be released.

@willrowe I do not see renderBlock mentioned in the description. Is renderBlock already existed or any plan about rendering a block individually?

@willrowe I actually found the answer after I read your PR carefully. I can use getBlock and BlockRender to render a block. Please correct me if wrong. Thanks

@wz185 the getBlock methods resolve a single Twig.Block instance. The render method can then be called on that instance.

@willrowe thanks for confirming. @RobLoach is there any ETA of the next release?

We are in need of the the blocks refactor. I merged into master locally, resolving conflicts. Works great. Anything not finished yet or that's keeping this from being merged?

@owzim there is nothing more that I need to do, this can be merged.

I'll tag a release for what's in there now, merge this, and merge this so we can do more testing. Looking great, @willrowe et al!

Thanks @RobLoach

@RobLoach @willrowe I had the issue that code executed in a parent template block was also executed in the child's template block, although it was empty, or filled with different code. With my manual this-branch-into-current master-merge, this was fixed, with the latest release 1.13.3 the problem persists.

@owzim as far as I know there has not been a release with the new block implementation yet. When it does come it should be a v2.0.0 release.

@willrowe thanks for the reply, but looking at the 1.13.3 release, I see the same commits like in here. What I am missing?

Switch the base and compare. master has all the commits of 1.13.3 and more.

I can make a 2.0.0-alpha.1 release for this to allow some additional testing.... Within the next hour.

Great, thanks @RobLoach!

When will there be a release containing this?


save output by default in curl test script

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 25.39%. Comparing base (0909387) to head (de36889).

:exclamation:  There is a different number of reports uploaded between BASE (0909387) and HEAD (de36889). Click for more details.
HEAD has 1 upload less than BASE

Flag
BASE (0909387)
HEAD (de36889)

g3proxy
1
0

Additional details and impacted files
@@             Coverage Diff             @@
##           master     #371       +/-   ##
===========================================
- Coverage   42.01%   25.39%   -16.62%
===========================================
  Files        1089      656      -433
  Lines       98410    48427    -49983
===========================================
- Hits        41344    12297    -29047
+ Misses      57066    36130    -20936

Flag
Coverage Δ

g3proxy
?

lib
25.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


New option: .dismissOnPopover

Hey @corin8823 ! Love the simplicity of the popover, and I'd like to use it in my app! 🐻 I was missing one option: to dismiss on popover tap, so I added it!
Also, I spotted one bug. When you define a method in example project, like this:
  @IBAction func tappedLeftBottomButton(_ sender: UIButton) {
    let width = self.view.frame.width / 4
    let aView = UIView(frame: CGRect(x: 0, y: 0, width: width, height: width))
    let options = [
      .type(.up),
      .showBlackOverlay(false)
      ] as [PopoverOption]
    let popover = Popover(options: options, showHandler: nil, dismissHandler: nil)
    popover.show(aView, fromView: self.leftBottomButton)
  }

Then you can't dismiss this popup by tapping on button. The solution is:
  open func show(_ contentView: UIView, point: CGPoint, inView: UIView) {
    if let visiblePopover = inView.subviews.flatMap({ $0 as? Popover }).first, showBlackOverlay == false {
        visiblePopover.dismiss()
        return
    }

But I didn't add it, because it would dismiss other opened popover. But if you have only on popover on-screen then this would do the trick.
Cheers!

It is not supposed to put out two popover on the screen.
So you do not need to increase options for this fix.
Thanks for the bug report!


Make extent in NavigateTo not nullable

Follow up of this comment
https://github.com/Mapsui/Mapsui/pull/1383#discussion_r748798092

My knowledge in this area isn't very good, so I could only look astonished on all this work going on.
But isn't it better to check in the Navigator, if the Extent is null and then do nothing? So you have to change it only at one place.

Yes that's is how it is currently implemented. In the NavigatedTo Implementation it returns when the extent is null.  So I can live with both ways extent allow to be nullable or always to have to specify the extent.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: inforithmics:x: Thomas StockerThomas Stocker seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Here in the Nullable Design Document it is stated that NavigateTo should not be nullable.
https://github.com/Mapsui/Mapsui/issues/1360#issuecomment-962825664

@charlenni The idea is that it is better to avoid problems than to fix them. This is very general principle. For instance.

With regard to errors it is better to detect a problem early and immediately throw an exception than it is to try to keep running in a corrupt state (fail fast). This is not always possible, especially with regard to things that can fail like network errors, but where you can you should.
With regard to IDiposable you would rather create a class that does not need to be disposed instead of calling dispose on all disposable classes.

Nullability is by some language designers seen as a big mistake. Often we can not avoid it but if we can we should. Checking for null can avoid a crash but can also lead to surprising behavior. If nothing happens while it should this can be a problem that is hard to diagnose. Better to make it impossible and force the caller make sure it is not null. There is now still a null check in the caller of Navigator, but we will push nullability back step by step.

@inforithmics There is a conflict. I could have look at this tomorrow.

@pauldendulk But isn't it possible, that the developer calls NavigateTo() with a null? We are not the only one, that use this function. Perhaps throwing an ArgumentNullException would better here?

@charlenni Where should the ArgumentException be thrown?

@pauldendulk In all public NavigateTo(), that expect an Extent.

@charlenni Looking at that code now, and although the interface in not nullable its implementation is still nullable. It should be not nullable as well, just following the interface, and I think it would make sense to throw an ArgumentNullException if the argument is not null. It that what you had in mind?
@inforithmics What do you think?


Editor GPU: Handle the case where glyphs do not fit into standard atlas page slabs

Repro:

macOS (or a high DPI monitor preferably)
Enable editor gpu acceleration
Zoom in a lot, eventually things will all break as glyphs won't fit into the atlas pages

Related: https://github.com/microsoft/vscode/issues/232979

https://github.com/microsoft/vscode/blob/065d72c268bb9a8912fc1a09b5032718ece71149/src/vs/editor/browser/gpu/atlas/textureAtlasPage.ts#L84-L87


Update hash and version of testing dataset

Following: https://github.com/mne-tools/mne-testing-data/pull/115

@larsoner @drammock I think we have the tests ubuntu-20.04 / old / 3.9 and ubuntu-latest / minimal / 3.9 set in the branch protection rule, but those tests don't exist anymore since 3.9 was dropped.


Errors on fresh installation with demo data populated

Response on progress API call with user id 1:
{"achievements": {}, "achievement_errors": {"1": {"id": 1, "error": "Traceback (most recent call last):\n  File "/home/fantaus/Documents/cg/gengine/views.py", line 92, in ea\n    return Achievement.evaluate(user, achievement["id"])\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 763, in evaluate\n    return cache_achievement_eval.get_or_create("%s_%s" % (user["id"],achievement_id),generate)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 657, in get_or_create\n    async_creator) as value:\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 158, in enter\n    return self._enter()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 98, in _enter\n    generated = self._enter_create(createdtime)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 149, in _enter_create\n    created = self.creator()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 625, in gen_value\n    created_value = creator()\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 682, in generate\n    Goal.evaluate(goal, user_id, user_wants_level,None)\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 1002, in evaluate\n    users_progress = Goal.compute_progress(goal,user_id)\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 993, in compute_progress\n    q = cache_goal_statements.get_or_create(str(goal["id"]),generate_statement_cache)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 657, in get_or_create\n    async_creator) as value:\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 158, in enter\n    return self._enter()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 98, in _enter\n    generated = self._enter_create(createdtime)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 149, in enter_create\n    created = self.creator()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 630, in gen_value\n    self.backend.set(key, value)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/backends/file.py", line 239, in set\n    compat.pickle.HIGHEST_PROTOCOL)\nPicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed\n"}, "2": {"id": 2, "error": "Traceback (most recent call last):\n  File "/home/fantaus/Documents/cg/gengine/views.py", line 92, in ea\n    return Achievement.evaluate(user, achievement["id"])\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 763, in evaluate\n    return cache_achievement_eval.get_or_create("%s%s" % (user["id"],achievement_id),generate)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 657, in get_or_create\n    async_creator) as value:\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 158, in enter\n    return self._enter()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 98, in _enter\n    generated = self._enter_create(createdtime)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 149, in _enter_create\n    created = self.creator()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 625, in gen_value\n    created_value = creator()\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 682, in generate\n    Goal.evaluate(goal, user_id, user_wants_level,None)\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 1002, in evaluate\n    users_progress = Goal.compute_progress(goal,user_id)\n  File "/home/fantaus/Documents/cg/gengine/models.py", line 993, in compute_progress\n    q = cache_goal_statements.get_or_create(str(goal["id"]),generate_statement_cache)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 657, in get_or_create\n    async_creator) as value:\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 158, in enter\n    return self._enter()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 98, in _enter\n    generated = self._enter_create(createdtime)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.core-0.4.1-py2.7.egg/dogpile/core/dogpile.py", line 149, in _enter_create\n    created = self.creator()\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/region.py", line 630, in gen_value\n    self.backend.set(key, value)\n  File "/usr/local/lib/python2.7/dist-packages/dogpile.cache-0.5.7-py2.7.egg/dogpile/cache/backends/file.py", line 239, in set\n    compat.pickle.HIGHEST_PROTOCOL)\nPicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed\n"}}}

There was an error in the correct configuration of the caching backends. Some queries are not supported by the file caching backend due to serialization issues. I have changed it to an in memory backend in commit 61358338c5bcab4bd0f6c48158a88adc34cc8537. The new version is already released https://github.com/ActiDoo/gamification-engine/releases/tag/v0.1.36


Fix word generator to use word ammount, not len value

This trivial PR fixes word generator to not generate empty words.

Nice!


[Experiment] Update icon component to use raw SVG data

It’s bugged me for a while that this is using React and not just SVG text. Turns out the gain in the bundle is uh, not that much (28.94 KB vs 20.41 KB) but on the plus side this means we’re relying on Webpack rather than arbitrary lists in the component file to figure out what’s available!

I'm cool with this change! How does the raw-loader work in prod?

Things that are raw-loaded are included in the bundle as usual. The difference being they’re SVG text rather than JS. We could take this further and use something like svgo to save a few more bytes but I suspect that wouldn’t be too important (it’d just be trimming whitespace at this point).

Sweet, let's do it!


feat: toggle unescape strings

Hey, thanks for putting this great tool together. I often run into logs that have some escaped string, like "This is a newline\n". When these strings get long, it's very hard to read, and sometimes they can be multiple levels of escape being nested. I've ran into bugs that I could've solved a lot earlier if I could easily inspect escaped logs.
It would be great if there is a toggle to escape one level (with support for escaping more than one level).
I unfortunately don't have time to contribute atm but leaving a request here in case any one else also has similar needs. Happy to come back and contribute this feature when I have some time.

Hi! Thank you for the issue.
Let's take the following logs:
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}
{"message": "123123123123\n123123123123123\n123123123123123"}

They look as on the picture:

How do you want to see this?

Naively, I think just escaping the sequences as is would work. The important thing is changing a long piece of text that's practically unreadable into something that at least has structure. The lack of indentation would be a bit weird, but this is also how Python triple quote strings work so it least has precedance.

Alternatively, something like this might make sense, but I realize this is probably straying a bit far from any JSON conventions

what about expanding such strings on click?
and probably this issue should go to https://github.com/antonmedv/fx (this widget is reused in json-log-viewer).

I think that would be a good UX. Let me post to the fx repo. Thanks :)


Lab instructions are incorrect

The step-by-step instructions are not correct for the code used in this module:
https://github.com/dusty-nv/jetson-inference/blob/master/docs/imagenet-example-python-2.md
E.g. the complete code uses "args" but the step instructions use "opt" and "args". So if you build the code section-by-section instead of copying the entire code it will not run.

Thanks for pointing this out @bryonbaker - just fixed this in https://github.com/dusty-nv/jetson-inference/commit/6bf94f753c727ea50f256fdec5fbe74bee540773


Transmorph paper

Hello,
I'm wondering what is the library used in the paper to display the Dice scores as box plots in the result part. And is it possible to get a code snippet ?
Thanks in advance

Hi @kiwibruyant ,
Please check out this script for the plots:
https://github.com/junyuchen245/TransMorph_Transformer_for_Medical_Image_Registration/blob/main/IXI/analysis.py
Junyu


v5 - "Cannot form Range" crash when AudioKit receives a particular MIDI sysex message

I'm building a SysEx editor for Emu Proteus 2K family devices, using AudioKit for the MIDI side. One particular sysex response from my test hardware (an Emu MoPhatt sound module) is causing a crash in MIDI+Receiving.swift:
private func processUMPSysExMessage(with bytes: [UInt8]) -> [UInt8]? {
    // Chapter 4.4 of Universal MIDI Packet (UMP) Format and MIDI 2.0 Protocol, Version 1.0
    // http://download.xskernel.org/docs/protocols/M2-104-UM_v1-0_UMP_and_MIDI_2-0_Protocol_Specification.pdf

    let umpSysExStatus = UMPSysEx7bitStatus(rawValue: getMSB(from: bytes[0])) // status byte
    let validBytesCount = getLSB(from: bytes[0]) // valid bytes count field
    let validBytes = Array(bytes[1...Int(validBytesCount)])    // <--- CRASH OCCURRING HERE

    guard (umpSysExStatus != nil) else {
        Log("UMP SYSEX - Got unsupported UMPSysEx7bitStatus", log: OSLog.midi)
        return validBytes
    }

    ...

The crash (marked on the code above) is:

Fatal error: Can't form Range with upperBound < lowerBound

This is because validBytesCounts is 0, which is clearly less than the lower bound of 1.
The sysex response I'm receiving (which I've confirmed is the correct response for the message I'm sending) is:
F0 18 0F 04 55 0B 01 01  00 0D 00 62 74 73 3A 54 10  68 65 20 55 6C 74 69 6D  61 74 65 F7

I have other sysex messages being received correctly, and as far as I can tell there isn't something in my code that is causing this (as I said, the MoPhatt is sending the expected SysEx message to my app, and then this crash occurs before my code is hit).
To replicate:
I realise few will have this MIDI hardware available, but I would assume that "playing" the sysex above out from SysEx Librarian into a MIDI receiving AudioKit app should trigger the same crash.
Setup:

Type: iPhone X
OS: iOS 14.3
AudioKit Version: v5 main (last updated 2021-01-15)
Emu MoPhatt MIDI sound module running OS v2.26
Cakewalk UM-2G MIDI interface, connected to iPhone via Apple Camera Connector

Can you make it fail in a sample project that both sends and receives sysex? This will be a while before one of us goes to the trouble of setting up your situation. If you want to do a screenshare, that could be the most efficient.

Can you make it fail in a sample project that both sends and receives sysex? This will be a while before one of us goes to the trouble of setting up your situation. If you want to do a screenshare, that could be the most efficient.

I've created a minimal project that demonstrates the crash – https://github.com/adamjansch/AudioKit-SysEx-crash. This project only implements sysex reception.
To reproduce
You will need some way of sending the sysex file at https://github.com/adamjansch/AudioKit-SysEx-crash/blob/main/Sysex/MoPhatt Preset Name response.syx – I used SysEx Librarian for Mac (https://www.snoize.com/SysExLibrarian/).

Run the project on an iOS device. When ready it should display a "Hello World!" message.
Make a MIDI connection between a Mac running SysEx Librarian and the iOS device (note: I tested using hardware MIDI, I don't know if network MIDI will work out of the box). The Mac should run MIDI out to iOS device MIDI in.
Start SysEx Librarian, set the MIDI out endpoint to point to the iOS device, and load in https://github.com/adamjansch/AudioKit-SysEx-crash/blob/main/Sysex/MoPhatt Preset Name response.syx.
Press Play in SysEx Librarian.

If the MIDI connections are all correct then the iOS app should crash as described.

I've created a minimal project that demonstrates the crash – https://github.com/adamjansch/AudioKit-SysEx-crash. This project only implements sysex reception.
To reproduce
You will need some way of sending the sysex file at https://github.com/adamjansch/AudioKit-SysEx-crash/blob/main/Sysex/MoPhatt Preset Name response.syx – I used SysEx Librarian for Mac (https://www.snoize.com/SysExLibrarian/).

Run the project on an iOS device. When ready it should display a "Hello World!" message.
Make a MIDI connection between a Mac running SysEx Librarian and the iOS device (note: I tested using hardware MIDI, I don't know if network MIDI will work out of the box). The Mac should run MIDI out to iOS device MIDI in.
Start SysEx Librarian, set the MIDI out endpoint to point to the iOS device, and load in https://github.com/adamjansch/AudioKit-SysEx-crash/blob/main/Sysex/MoPhatt Preset Name response.syx.
Press Play in SysEx Librarian.

If the MIDI connections are all correct then the iOS app should crash as described.

This crash has nothing to do with your hardware.
It is caused by a malformed range expression (MIDI+Receiving.swift)
// this range will cause a crash when validBytesCount is 0
 1...Int(validBytesCount)

It should be written like this:
// correct
 1..<1+Int(validBytesCount)

You can confirm this be running the following tests:
import XCTest

extension UInt8 {
    var lowNibble: UInt8 { self & 0x0F }
}

// this is how bytes are extracted from the UMP64 (MIDI+Receiving.swift)
func getBytesFromUMPWords(word0: UInt32, word1: UInt32) -> [UInt8] {
    let bytes0 = withUnsafeBytes(of: word0.bigEndian, Array.init)
    let bytes1 = withUnsafeBytes(of: word1.bigEndian, Array.init)
    return Array(bytes0[1..<4]) + bytes1
}

// this is how data is read (MIDI+Receiving.swift)
func getDataFromUMPBytes(bytes: [UInt8])->[UInt8] {
    let validBytes = Int(bytes[0].lowNibble)

    // this is the correct range
    let validBytesRange = 1..<1+validBytes

    return Array(bytes[validBytesRange])
}

class UMPSysexGetDataTests: XCTestCase {

    func testWith4DataBytes() throws {
        // group: 3, umpType: 5, sysexType: 0, validbytes: 4, data0: 1, data1: 2
        let word0: UInt32 = 0x35_04_01_02

        // data2: 3, data3: 4, data4: 0, data5: 0
        let word1: UInt32 = 0x03_04_00_00

        let bytes = getBytesFromUMPWords(word0: word0, word1: word1)
        let data = getDataFromUMPBytes(bytes: bytes)

        XCTAssertEqual(data, [1, 2, 3, 4])
    }

    func testWithNoDataBytes() throws {
        // group: 3, umpType: 5, sysexType: 0, validbytes: 0, data0: 0, data1: 0
        let word0: UInt32 = 0x35_00_00_00

        // data2: 0, data3: 0, data4: 0, data5: 0
        let word1: UInt32 = 0x00_00_00_00

        let bytes = getBytesFromUMPWords(word0: word0, word1: word1)
        let data = getDataFromUMPBytes(bytes: bytes)

        XCTAssertEqual(data, [])
    }
}

This crash has nothing to do with your hardware.
It is caused by a malformed range expression (MIDI+Receiving.swift)
// this range will cause a crash when validBytesCount is 0
 1...Int(validBytesCount)

It should be written like this:
// correct
 1..<1+Int(validBytesCount)

You can confirm this be running the following tests:
import XCTest

extension UInt8 {
    var lowNibble: UInt8 { self & 0x0F }
}

// this is how bytes are extracted from the UMP64 (MIDI+Receiving.swift)
func getBytesFromUMPWords(word0: UInt32, word1: UInt32) -> [UInt8] {
    let bytes0 = withUnsafeBytes(of: word0.bigEndian, Array.init)
    let bytes1 = withUnsafeBytes(of: word1.bigEndian, Array.init)
    return Array(bytes0[1..<4]) + bytes1
}

// this is how data is read (MIDI+Receiving.swift)
func getDataFromUMPBytes(bytes: [UInt8])->[UInt8] {
    let validBytes = Int(bytes[0].lowNibble)

    // this is the correct range
    let validBytesRange = 1..<1+validBytes

    return Array(bytes[validBytesRange])
}

class UMPSysexGetDataTests: XCTestCase {

    func testWith4DataBytes() throws {
        // group: 3, umpType: 5, sysexType: 0, validbytes: 4, data0: 1, data1: 2
        let word0: UInt32 = 0x35_04_01_02

        // data2: 3, data3: 4, data4: 0, data5: 0
        let word1: UInt32 = 0x03_04_00_00

        let bytes = getBytesFromUMPWords(word0: word0, word1: word1)
        let data = getDataFromUMPBytes(bytes: bytes)

        XCTAssertEqual(data, [1, 2, 3, 4])
    }

    func testWithNoDataBytes() throws {
        // group: 3, umpType: 5, sysexType: 0, validbytes: 0, data0: 0, data1: 0
        let word0: UInt32 = 0x35_00_00_00

        // data2: 0, data3: 0, data4: 0, data5: 0
        let word1: UInt32 = 0x00_00_00_00

        let bytes = getBytesFromUMPWords(word0: word0, word1: word1)
        let data = getDataFromUMPBytes(bytes: bytes)

        XCTAssertEqual(data, [])
    }
}

Want to make a pull request with the fix?

Want to make a pull request with the fix?

Ok. I can do it this afternoon.

Ok. I can do it this afternoon.

Just gave the fix a test, looks good at this end! Thanks for your help @jrmaxdev and @aure :-)

Just gave the fix a test, looks good at this end! Thanks for your help @jrmaxdev and @aure :-)


Clean up validation code

Validation code should only do validation, shouldn't also set default value. The default value setting should be done in conversion since it matters in different api version.

Done. We now have a defaulting pass.

See #1502 for more details.


Feature/auto shard rebalance

Scope & Purpose
The purpose of this PR is to integrate the autorebalance tool into arangodb.

[ ] :hankey: Bugfix
[x] :pizza: New feature
[ ] :fire: Performance improvement
[ ] :hammer: Refactoring/simplification

Checklist

[ ] Tests

[ ] Regression tests
[ ] C++ Unit tests
[ ] integration tests
[ ] resilience tests

[x] :book: CHANGELOG entry made
[ ] :books: documentation written (release notes, API changes, ...)

Related Information

[ ] Docs PR:
[ ] Enterprise PR:
[ ] GitHub issue / Jira ticket:
[ ] Design document: https://github.com/arangodb/documents/blob/master/DesignDocuments/01_IDEAS/AutoRebalance.md

* Test "restart"
    [FAILED]  tests\js\client\restart\test-soft-shutdown-cluster.js

      "test" failed: server unavailable for testing. Instance not healthy!

and
17:24:35 FATAL: command execution failed
17:24:35 java.io.EOFException
17:24:35 	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905)
17:24:35 	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400)
17:24:35 	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936)
17:24:35 	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
17:24:35 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:4


Avoid async infinite loop on tab close

When a tab gets closed, _RemoveTab will call TabBase::Shutdown(),
which then re-raises the Closed event, which will end up calling
_RemoveTab again, etc. The only reason this didn't crash WT so far
is because _RemoveOnCloseRoutine contains a resume_foreground,
which would resolve the recursion and turn it into CPU usage.
It would spin as long as WinUI hasn't discard the tab object,
which takes an unpredictable amount of time.
Raising the Closed event from Shutdown() is unnecessary, because
the handlers of the event end up calling _RemoveTab anyways.
Technically the entire Closed event can be removed now, but I left it
in anyways because resolving the architectural "knot" around the way
tab closing after the last pane closes is implemented requires much
more significant changes.
This commit additionally removes the _createCloseLock mutex in Pane
as it was very likely not working as intended anyways. Only some methods
were protected by it and it doesn't avoid any STA/MTA/NA issues either.
Validation Steps Performed

Closing tabs and panes always ends up calling Shutdown() ✅

Are we certain that Close always comes in on the right thread?

You're right, it didn't. In my original PR it worked, because I also made it so that the close event gets emitted on the UI thread. I now brought those changes into this PR as well, but unfortunately it increases the diff size. It's easier to view with whitespace changes suppressed.

Oh, you removed it from 1.18. OK

Yeah I think I'd like to wait with this until 1.19... There's still these comments in the code and I'd like to 110% grasp what they mean in practice and if it was a practical concern to begin with:
// It's possible that this event handler started being executed, then before
// we got the lock, another thread created another child. So our control is
// actually no longer _our_ control, and instead could be a descendant.

...because to be honest I don't think any of this worked the way we imagined it would.

@carlos-zamora Sorry for the late response. As far as I can tell, _ControlConnectionStateChangedHandler is the only function that is being called from a background thread. It would end up calling Close() which I assume is why the mutex was called _createCloseLock.
The reason I removed it is simply because that mutex provides a false sense of security. We can't use it to make shutdown secure, when Pane has ~50 member functions and only 6 of them are thread safe. 😅 It just doesn't work at all and never did. 😅😅

This has ✅✅, but never merged while I was out. I haven't read it yet. Should we include this in the bug bash?

Yes, please. I've merged in main just now to resolve the conflict.


109-validate-licenses-of-used-libraries-and-tools

Blocked by this PR on the required package: https://github.com/FHPythonUtils/LicenseCheck/pull/29


Next major release queue

:tada: This issue has been resolved in v4.0.0
If you appreciate this project, please consider supporting this project by sponsoring :heart: :pray:


Using icon component with custom icon sets

I think it would be nice to allow people to use our icon component with custom, external icon sets. I think we can inspire from Shoelace for this and give the ability to register custom icon sets by still keeping current icon set by default.
<script type="module">
  import { registerIconLibrary } from '@trendyol/baklava';

  registerIconLibrary('bootstrap', {
    resolver: name =><EMAIL_ADDRESS>  });
</script>

<bl-icon library="bootstrap" name="arrow-up"></bl-icon>

We have decided against implementing this feature for the time being.
Users can change the icon pack by using the setIconPath utility, provided they supply all icons with the same names. However, we do not endorse or officially support this method.
We may revisit this feature in the future. If anyone feels the need for this functionality, please feel free to reopen the issue.


CORS issue but the rest of Medusa backend works

Hi,
I am facing an issue with plugin when moving from local development to a Railway server. I am using the plugin for authenticating admin users with Firebase. The plugin works as expected in my local environment but now when I have deployed my site and Medusa backend to Railway I receive this error:
❌  "Access to XMLHttpRequest at 'https://.up.railway.app/admin/auth/firebase' from origin 'https://.up.railway.app' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource."
And got a "Referrer Policy: strict-origin-when-cross-origin" when inspected the network request.
I have added these environment variables in Railway:
BACKEND_URL=https://.up.railway.app
ADMIN_URL=https://.up.railway.app
STORE_URL=https://.up.railway.app
I've also tried to add https://.up.railway.app and https://.up.railway.app to the Authorised Domains in Firebase Authentication with no success.
The rest of Medusa works fine with no CORS issues.

@dPreininger could you have a look at this please 😍

@rasmuslian I believe the cause for this is that the url is incorrect. You have a dot right after https://. Try urls without this dot like so: https://up.railway.app.

any news @rasmuslian

Sorry for late reply, I’ve gone on vacation today and will be afk for the next 1.5 weeks. Will update after that.

@adrien2p However when looking at this now I see that my urls in the question are wrong, they are not https://.up.railway.app but it should be for example BACKEND_URL=https://medusa-server.up.railway.app and STORE_URL=https://my-frontend.up.railway.app

@adrien2p @dPreininger Hi again, I’m now back from vacation, any updates? See my previous answer also :)

Update: I tested and redeployed the whole thing, turns out that probably I mis-set the environment variables after all somehow. Sorry for all the confusion, closing now.


Simple voice chat, double container, port issue

Hello, more of a question than an issue but i encounter a strange behaviour here :
I Actually run 2 differents container, each one with a minecraft server from ITZG. In both container (one forge, and one fabric if that matter) i have simple voice chat installed. Here is my configuration for it regarding port
Server 1
minecraft-server:
    image: itzg/minecraft-server:latest
    container_name: server1
    ports:
      - 25565:25565
      - 24454:24454/udp

Server 2
minecraft-server:
    image: itzg/minecraft-server:latest
    container_name: server2
    ports:
      - 25566:25565
      - 24455:24454/udp

The default port for simple voice chat is 24454 UDP that why. But with this configuration, the mods didn't work on the server2 and to fix it, i had to change the port config to - 24455:24455/udp (and make the change to simple voice chat config file inside /data in order to change the default port to 24455)
In my understanding, docker container should run as an Isolated Resource. So i'm wondering why i encounter a port conflict ?
Thanks for your answer

Yes your understanding of Docker and network isolation is correct. I'm guessing in this case the protocol the voice chat is using over UDP advertises the configured port to the external endpoints. As a result it's still needing the ports to line up.

Ok thanks for your answer.
So either each user have to configure the right port client's side, or i need to do manually the configuration server's side.
I prefer the solution that make users do less manipulations.
You can close the issue thanks you.