[TW#20805] Enterprise PEAP connection issue

I'm testing wpa2 enterprise example but the ESP32 is unable to connect I'm setting the EAP method to EAP_PEAP. Below is a log for the AP when the device is trying to connect. I tried different versions of the IDF with no joy.
My laptop and phone can connect to the same SSID using the username and password with no issues.
Any ideas what might be going wrong? why the authentication is failing?

Supplicant used different EAP type: 3
authentication failed - EAP type: 4

Hi, did the above suggestion help you resolve the issue?

No it did not I'm using the correct authentication method in the ESP32 firmware. I'm still unable to use wpa2 enterprise, even with the latest update to IDF v3.0.
Can you please share a test setup where the wpa2 enterprise example was tested?

Hi, @ali-slim-sst, if you want to use the wpa_enterprise example, I think most important thing is make sure your have correct certificate, because the server and client certificates should be matched so that it can build TLS tunnel after check certificates, if you can use the server.key, server.crt and server.pem in example instead of them in the server, it would be OK, or you can try whether you have client cerificate to instead of them in the example. If you don have client certificate or can't exchange the certificate in server, may be you can mark down esp_wifi_wpa_ent_set_ca_cert() and esp_wifi_sta_wpa2_ent_set_cert_key() so that it will jump certificate check. If it still fail, maybe you can send us the air packet and let me check when it fail

Hi @XinDeng11 , please explain why do i need the certificates if I'm using PEAP? If i remove esp_wifi_wpa_ent_set_ca_cert() this should skip the certifcate check?

PEAP methds do security check based on certificate and username/password, it need check certificate to build TLS tunnel, and then do MSCHAPv2 check in the TLS tunnel by username and password, because server need decode the pre master secret created by client according to the server key which is in the certificate, and the pre master secret created by client was encoded by public key in the ca certificate, so that why certificate is necessary in PEAP/TTLS/TLS method. Why the certificate should be matched is due to RSA algrithm, there are two private keys and one public key, each private key is in the client certificate and server certificate, and public key is in the certificate, so it need check the ca certificate to make sure client and server are safe. Remove ca cert check and set key will jump this step and build a tunnel directly, but it can't in the TTLS method.

Thanks @XinDeng11 for the explanation. I removed esp_wifi_sta_wpa2_ent_set_ca_cert() and esp_wifi_sta_wpa2_ent_set_cert_key() but I still cannot connect in PEAP mode. Let me know if there's anything else I can try.
Please note that I'm able to connect other devices like my phone and laptop to the same router using PEAP just using username and password.

If you can offer sniffer log(air packets), we can do a continue check when it failed, because we 100% sure the example is OK when connect to a radius server.

@XinDeng11 I've tried to capture some useful wifi air packets, but I'm not getting anything useful so far, can you please share how you're sniffing the packets? Thanks.

@ali-slim-sst what's your software and tool use

If  it that, I just wonder whether you are in the correct channel, because use wireshark in Linux need set channel by command in terminal. If you still can't found useful message, you can try to open the log by menuconfig, the log output option in the components option. If you still fail to get message, you have to wait for the 3.1 version release. We will open source the wpa enterprise code at that time

@XinDeng11 below is debug output. What are reasons 204 and 201?
I (0) cpu_start: App cpu up.
I (332) heap_init: Initializing. RAM available for dynamic allocation:
D (339) heap_init: New heap initialised at 0x3ffae6e0
I (344) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
D (350) heap_init: New heap initialised at 0x3ffb8f28
I (355) heap_init: At 3FFB8F28 len 000270D8 (156 KiB): DRAM
I (361) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (367) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
D (374) heap_init: New heap initialised at 0x4009154c
I (379) heap_init: At 4009154C len 0000EAB4 (58 KiB): IRAM
I (385) cpu_start: Pro cpu start user code
D (397) clk: RTC_SLOW_CLK calibration value: 3332237
V (71) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): checking args
V (71) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): Args okay. Resulting flags 0xE
D (76) intr_alloc: Connected src 46 to int 2 (cpu 0)
V (81) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): checking args
V (87) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): Args okay. Resulting flags 0xC0E
D (95) intr_alloc: Connected src 57 to int 3 (cpu 0)
V (100) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): checking args
V (107) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): Args okay. Resulting flags 0x40E
D (115) intr_alloc: Connected src 24 to int 9 (cpu 0)
I (120) cpu_start: Starting scheduler on PRO CPU.
V (0) intr_alloc: esp_intr_alloc_intrstatus (cpu 1): checking args
V (0) intr_alloc: esp_intr_alloc_intrstatus (cpu 1): Args okay. Resulting flags 0x40E
D (10) intr_alloc: Connected src 25 to int 2 (cpu 1)
I (10) cpu_start: Starting scheduler on APP CPU.
D (170) heap_init: New heap initialised at 0x3ffe0440
D (170) heap_init: New heap initialised at 0x3ffe4350
V (180) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): checking args
V (180) intr_alloc: esp_intr_alloc_intrstatus (cpu 0): Args okay. Resulting flags 0xE
D (180) intr_alloc: Connected src 16 to int 12 (cpu 0)
D (200) nvs: nvs_flash_init_custom partition=nvs start=9 count=6
D (270) nvs: nvs_open_from_partition misc 1
D (270) nvs: nvs_get_str_or_blob log
I (270) wifi: wifi firmware version: 6450435
I (270) wifi: config NVS flash: enabled
I (270) wifi: config nano formating: disabled
I (280) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (290) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
D (290) nvs: nvs_open_from_partition nvs.net80211 1
D (300) nvs: nvs_get opmode 1
D (300) nvs: nvs_get_str_or_blob sta.ssid
D (310) nvs: nvs_get_str_or_blob sta.mac
D (310) nvs: nvs_get sta.authmode 1
D (310) nvs: nvs_get_str_or_blob sta.pswd
D (320) nvs: nvs_get_str_or_blob sta.pmk
D (320) nvs: nvs_get sta.chan 1
D (330) nvs: nvs_get auto.conn 1
D (330) nvs: nvs_get bssid.set 1
D (330) nvs: nvs_get_str_or_blob sta.bssid
D (340) nvs: nvs_get sta.phym 1
D (340) nvs: nvs_get sta.phybw 1
D (340) nvs: nvs_get_str_or_blob sta.apsw
D (350) nvs: nvs_get_str_or_blob sta.apinfo
D (350) nvs: nvs_get sta.scan_method 1
D (360) nvs: nvs_get sta.sort_method 1
D (360) nvs: nvs_get sta.minrssi 1
D (360) nvs: nvs_get sta.minauth 1
D (370) nvs: nvs_get_str_or_blob ap.ssid
D (370) nvs: nvs_get_str_or_blob ap.mac
D (370) nvs: nvs_get_str_or_blob ap.passwd
D (380) nvs: nvs_get_str_or_blob ap.pmk
D (380) nvs: nvs_get ap.chan 1
D (390) nvs: nvs_get ap.authmode 1
D (390) nvs: nvs_get ap.hidden 1
D (390) nvs: nvs_get ap.max.conn 1
D (400) nvs: nvs_get bcn.interval 2
D (400) nvs: nvs_get ap.phym 1
D (400) nvs: nvs_get ap.phybw 1
D (410) nvs: nvs_get ap.sndchan 1
D (410) nvs: nvs_set_blob sta.mac 6
D (420) nvs: nvs_set_blob ap.mac 6
I (420) wifi: Init dynamic tx buffer num: 32
I (420) wifi: Init data frame dynamic rx buffer num: 32
I (420) wifi: Init management frame dynamic rx buffer num: 32
I (430) wifi: wifi driver task: 3ffc0680, prio:23, stack:4096
I (440) wifi: Init static rx buffer num: 10
I (440) wifi: Init dynamic rx buffer num: 32
I (440) wifi: wifi power manager task: 0x3ffc52c0 prio: 21 stack: 2560
I (450) example: Setting WiFi configuration SSID SSTest...
I (460) wpa: WPA2 ENTERPRISE VERSION: [v2.0] enable
D (460) RTC_MODULE: Wi-Fi takes adc2 lock.
D (460) phy_init: loading PHY init data from application binary
D (470) nvs: nvs_open_from_partition phy 0
D (470) nvs: nvs_get cal_version 4
V (480) phy_init: phy_get_rf_cal_version: 366
D (480) nvs: nvs_get_str_or_blob cal_mac
D (490) nvs: nvs_get_str_or_blob cal_data
D (500) nvs: nvs_close 3
V (500) phy_init: register_chipv7_phy, init_data=0x3f412e08, cal_data=0x3ffc5648, mode=0
I (520) phy: phy_version: 366.0, ba9923d, Dec 29 2017, 14:25:06, 0, 0
I (520) wifi: mode : sta (24:0a:c4:08:62:6c)
D (520) event: SYSTEM_EVENT_STA_START
V (520) event: enter default callback
D (530) tcpip_adapter: check: local, if=0 fn=0x401224f0
0x401224f0: tcpip_adapter_start_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
D (530) tcpip_adapter: call api in lwip: ret=0x0, give sem
D (540) tcpip_adapter: check: remote, if=0 fn=0x401224f0
0x401224f0: tcpip_adapter_start_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
V (550) event: exit default callback
I (2960) wifi: n:6 0, o:1 0, ap:255 255, sta:6 0, prof:1
I (3940) wifi: state: init -> auth (b0)
I (3940) wifi: state: auth -> assoc (0)
I (3950) wifi: state: assoc -> run (10)
I (3950) wpa: wpa2_task prio:2, stack:6656
I (4530) example: ~~~~~~~~~~~
I (4530) example: IP:<IP_ADDRESS>
I (4530) example: MASK:<IP_ADDRESS>
I (4530) example: GW:<IP_ADDRESS>
I (4530) example: ~~~~~~~~~~~
I (6530) example: ~~~~~~~~~~~
I (6530) example: IP:<IP_ADDRESS>
I (6530) example: MASK:<IP_ADDRESS>
I (6530) example: GW:<IP_ADDRESS>
I (6530) example: ~~~~~~~~~~~
I (6950) wifi: pm start, type:0
I (8530) example: ~~~~~~~~~~~
I (8530) example: IP:<IP_ADDRESS>
I (8530) example: MASK:<IP_ADDRESS>
I (8530) example: GW:<IP_ADDRESS>
I (8530) example: ~~~~~~~~~~~
I (10530) example: ~~~~~~~~~~~
I (10530) example: IP:<IP_ADDRESS>
I (10530) example: MASK:<IP_ADDRESS>
I (10530) example: GW:<IP_ADDRESS>
I (10530) example: ~~~~~~~~~~~
I (12530) example: ~~~~~~~~~~~
I (12530) example: IP:<IP_ADDRESS>
I (12530) example: MASK:<IP_ADDRESS>
I (12530) example: GW:<IP_ADDRESS>
I (12530) example: ~~~~~~~~~~~
I (14530) example: ~~~~~~~~~~~
I (14530) example: IP:<IP_ADDRESS>
I (14530) example: MASK:<IP_ADDRESS>
I (14530) example: GW:<IP_ADDRESS>
I (14530) example: ~~~~~~~~~~~
I (16530) example: ~~~~~~~~~~~
I (16530) example: IP:<IP_ADDRESS>
I (16530) example: MASK:<IP_ADDRESS>
I (16530) example: GW:<IP_ADDRESS>
I (16530) example: ~~~~~~~~~~~
I (18530) example: ~~~~~~~~~~~
I (18530) example: IP:<IP_ADDRESS>
I (18530) example: MASK:<IP_ADDRESS>
I (18530) example: GW:<IP_ADDRESS>
I (18530) example: ~~~~~~~~~~~
I (20530) example: ~~~~~~~~~~~
I (20530) example: IP:<IP_ADDRESS>
I (20530) example: MASK:<IP_ADDRESS>
I (20530) example: GW:<IP_ADDRESS>
I (20530) example: ~~~~~~~~~~~
I (22530) example: ~~~~~~~~~~~
I (22530) example: IP:<IP_ADDRESS>
I (22530) example: MASK:<IP_ADDRESS>
I (22530) example: GW:<IP_ADDRESS>
I (22530) example: ~~~~~~~~~~~
I (24530) example: ~~~~~~~~~~~
I (24530) example: IP:<IP_ADDRESS>
I (24530) example: MASK:<IP_ADDRESS>
I (24530) example: GW:<IP_ADDRESS>
I (24530) example: ~~~~~~~~~~~
I (26530) example: ~~~~~~~~~~~
I (26530) example: IP:<IP_ADDRESS>
I (26530) example: MASK:<IP_ADDRESS>
I (26530) example: GW:<IP_ADDRESS>
I (26530) example: ~~~~~~~~~~~
I (28530) example: ~~~~~~~~~~~
I (28530) example: IP:<IP_ADDRESS>
I (28530) example: MASK:<IP_ADDRESS>
I (28530) example: GW:<IP_ADDRESS>
I (28530) example: ~~~~~~~~~~~
I (30530) example: ~~~~~~~~~~~
I (30530) example: IP:<IP_ADDRESS>
I (30530) example: MASK:<IP_ADDRESS>
I (30530) example: GW:<IP_ADDRESS>
I (30530) example: ~~~~~~~~~~~
I (32530) example: ~~~~~~~~~~~
I (32530) example: IP:<IP_ADDRESS>
I (32530) example: MASK:<IP_ADDRESS>
I (32530) example: GW:<IP_ADDRESS>
I (32530) example: ~~~~~~~~~~~
I (33950) wifi: state: run -> init (2)
I (33950) wifi: pm stop, total sleep time: 0/27001411
I (33950) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
D (33950) event: SYSTEM_EVENT_STA_DISCONNECTED, ssid:SSTest, ssid_len:6, bssid:cc:5d:4e:ec:fe:4f, reason:204
I (33960) wpa: wpa2 task delete
V (33960) event: enter default callback
D (33970) tcpip_adapter: check: local, if=0 fn=0x40122848
0x40122848: tcpip_adapter_down_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
D (33970) tcpip_adapter: if0 start ip lost tmr: enter
D (33980) tcpip_adapter: if0 start ip lost tmr: no need start because netif=0x3ffc5c4c interval=120 ip=0
D (33990) tcpip_adapter: call api in lwip: ret=0x0, give sem
D (33990) tcpip_adapter: check: remote, if=0 fn=0x40122848
0x40122848: tcpip_adapter_down_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
V (34000) event: exit default callback
I (34530) example: ~~~~~~~~~~~
I (34530) example: IP:<IP_ADDRESS>
I (34530) example: MASK:<IP_ADDRESS>
I (34530) example: GW:<IP_ADDRESS>
I (34530) example: ~~~~~~~~~~~
D (36410) event: SYSTEM_EVENT_STA_DISCONNECTED, ssid:SSTest, ssid_len:6, bssid:00:00:00:00:00:00, reason:201
V (36410) event: enter default callback
D (36420) tcpip_adapter: check: local, if=0 fn=0x40122848
0x40122848: tcpip_adapter_down_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
D (36420) tcpip_adapter: if0 start ip lost tmr: enter
D (36430) tcpip_adapter: if0 start ip lost tmr: no need start because netif=0x3ffc5c4c interval=120 ip=0
D (36440) tcpip_adapter: call api in lwip: ret=0x0, give sem
D (36440) tcpip_adapter: check: remote, if=0 fn=0x40122848
0x40122848: tcpip_adapter_down_api at /home/esp32/esp/esp-idf/components/tcpip_adapter/./tcpip_adapter_lwip.c:1071
V (36450) event: exit default callback
I (36530) example: ~~~~~~~~~~~
I (36530) example: IP:<IP_ADDRESS>
I (36530) example: MASK:<IP_ADDRESS>
I (36530) example: GW:<IP_ADDRESS>
I (36530) example: ~~~~~~~~~~~
I (38530) example: ~~~~~~~~~~~
I (38530) example: IP:<IP_ADDRESS>
I (38530) example: MASK:<IP_ADDRESS>
I (38530) example: GW:<IP_ADDRESS>
I (38530) example: ~~~~~~~~~~~
I (38860) wifi: n:6 0, o:6 0, ap:255 255, sta:6 0, prof:1
I (38860) wifi: state: init -> auth (b0)
I (38870) wifi: state: auth -> assoc (0)
I (38870) wifi: state: assoc -> run (10)
I (38870) wpa: wpa2_task prio:2, stack:6656
I (39960) wpa: >>>>>wpa2 FIALED
I (39970) wpa: wpa2 task delete
I (40530) example: ~~~~~~~~~~~
I (40530) example: IP:<IP_ADDRESS>
I (40530) example: MASK:<IP_ADDRESS>
I (40530) example: GW:<IP_ADDRESS>
I (40530) example: ~~~~~~~~~~~
I (41870) wifi: pm start, type:0

Hi @XinDeng11 , I found the issue with eap connection. ESP32 EAP messages does not specify which eap type it will be using, so the radius server is using the set default which was md5.
I changed the freeradius default_eap_type from md5 to peap in the configuration file "..mods-enabled/eap", and the device was able to connect.
Is there a way to set the eap type in the ESP32 eap connection messages?

Any updates? @XinDeng11 @FayeY

@ali-slim-sst actually, we only support PEAP/TTLS/TLS in 32, and they have been start when enable, STA can not set eap type by its own, just start one of the methods base on radius server, server decides use which method. When start handshake, sever will send the type to STA by EAP frame, if STA support this method, it will go on, otherwise the connect would stop

Hi @FayeY  no not really.
@XinDeng11  there is a bit of misunderstanding, the device i.e the STA should be able to specify EAP type  in the EAP frame, like when I connect using another device such as my phone I have to choose a EAP method, in case of the ESP32 it's not specifying the EAP method and failing to connect.
I will post more references later.
Thanks.

@ali-slim-sst this is just products behavior, real flow is decided by server

look at point5 and 6

@ali-slim-sst any other question?

if STA support this method, it will go on, otherwise the connect would stop

Where a peer receives a Request for an unacceptable authentication
Type (4-253,255), or a peer lacking support for Expanded Types
receives a Request for Type 254, a Nak Response (Type 3) MUST be
sent.  The Type-Data field of the Nak Response (Type 3) MUST
contain one or more octets indicating the desired authentication
Type(s), one octet per Type, or the value zero (0) to indicate no
proposed alternative.

@XinDeng11
Below is form the Radius server configuration file. When the EAP-Identity message does not specify the EAP type, it will use the default_eap_type.
In your test setup, the ESP32 is able to connect if default_eap_type = md5 ?
https://github.com/FreeRADIUS/www.freeradius.org/blob/master/radiusd/raddb/eap.conf

eap {
#  Invoke the default supported EAP type when
#  EAP-Identity response is received.
#
#  The incoming EAP messages DO NOT specify which EAP
#  type they will be using, so it MUST be set here.
#
#  For now, only one default EAP type may be used at a time.
#
#  If the EAP-Type attribute is set by another module,
#  then that EAP type takes precedence over the
#  default type configured here.
#
default_eap_type = md5

@ali-slim-sst No, we didn't support md5 yet

@negativekelvin Yes your right, it should keep negotiating
https://tools.ietf.org/html/rfc3748#section-5.3.1
@XinDeng11 Freeradius debug output
Thu May 24 11:56:10 2018 : Debug: (9) eap: Peer sent packet with method EAP NAK (3)
Thu May 24 11:56:10 2018 : Debug: (9) eap: Peer NAK'd indicating it is not willing to continue
Thu May 24 11:56:10 2018 : Debug: (9) eap: Sending EAP Failure (code 4) ID 139 length 4

@XinDeng11 The picture below is form the wireshark capture, the device on the left is my phone and the device on right is the ESP32. Notice that the phone is specifying the "Desired Auth Type" (EAP-PEAP) (25) while the ESP32 just sends "0".

I can send the logs by email if needed.

@ali-slim-sst OK, please attach the sniffer log in the Github, we will discussion whether we can add this in the next version, thanks

@ali-slim-sst we have locked the feature development in our 3.1 version, now we can't add feature casual. However, we are doing plan for 4.0 version now, we can add this feature to a high priority. If it is urge, you can contract with our business affairs, we can do this ahead of schedule after it, thanks

This is not really a feature, it is a bugfix.  Add 3 bytes to the nak is minor.

I agree with @negativekelvin, this is definitely a bug (or at least a deviation from the standard) that stops ESP32 wifi enterprise from functioning in normal enterprise IT environments where it is simply not possible to pre-configure a radius server to work around a non-standard behaviour in the ESP32.
All that is needed is a simple additional field for desired auth type to be added to the IDF. Really not a big deal to add.
@XinDeng11: This is very high priority for us (it stops our product from working). We are prepared to test this bugfix immediately.

I investigated this error and now that parts of wpa_supplicant code was moved to esp-idf sources I was able to find a solution.
I'm not sure whether this is an error done when porting supplicant code to esp-idf or done on purpose to solve some specific interop problem but anyway. In function eap_sm_build_nak:
esp-idf loops through all registered eap_methods but set the desired method to EAP_TYPE_NONE:
	for (m = methods; m; m = m->next) {
		if (type == EAP_TYPE_EXPANDED) {
			wpabuf_put_u8(resp, EAP_TYPE_EXPANDED);
			wpabuf_put_be24(resp, m->vendor);
			wpabuf_put_be32(resp, m->method);
		} else
			wpabuf_put_u8(resp, EAP_TYPE_NONE);
		found++;
	}

This is changed compared to corresponding code in supplicant (function eap_sm_buildNak). In mainline supplicant type is retrieved from registered method(s). Like this:
wpabuf_put_u8(resp, m->method);
That simple change makes wpa eap functional on esp-idf also with servers that not necessarily are configured to make the initial attempt with a supported eap type. With this change esp device will respond with suggested type and server continue trying alternative methods until one that both sides allow is found.
I suggest to first apply the change above and then expose an api for configuring accepted methods. Mainline wpa_supplicant support this through a space separated config in wpa_supplicant.conf (network "eap" option). See functions eap_allowed_method and wpa_config_parse_eap. Something similar should be easy to implement for esp-idf too.

@mikaelkanstrup, you are right! It's the root cause, we will fix it soon.

@ali-slim-sst this bug has been fixed, the MR is what liuzfesp  pushed above

How can i set to mschapv2 method? i have a customer with wpa2 security, when i connected on my mac the authorize was GTC, on linux i can set to mschapv2 and it works, i need to configure EPS32 use mschapv2, please help


support for Red Hat 8

support Red Hat 8 systems
just copy pasted config from redhat-7

fixed in https://github.com/idiv-biodiversity/ansible-role-postfix/pull/6


How to integrate algo Multi-Agent Transformer in BenchMARL

Hi there,
First off, I want to say that BenchMARL looks like a fantastic resource for the MARL community! I've been following the project and I'm excited about the potential for benchmarking various algorithms.
I recently came across the Multi-Agent Transformer (MAT) by PKU-MARL, which is showcased in their GitHub here and the accompanying paper on arXiv. MAT introduces an innovative approach to MARL with its centralized joint-observation-action Transformer structure, and I believe it could be a valuable addition to BenchMARL.

I'm curious about how I might go about integrating MAT into BenchMARL. Could you provide some guidance or suggestions on this process? I'm eager to contribute to the project and explore the synergy between MAT and BenchMARL.
Thanks for your great work on this, and I'm looking forward to any advice you can offer!

Hey!
Thanks for the comment!
So, to understand better how we can frame this I have a few points:

Is this a new proposed model for a policy and as such this policy can be trained in any actor critic algo? or is it a new algorithm that prescribes the model to be used?
it seems that it has the requirement of accessing past observations and past actions which is currently not available in benchmarl but should not be too hard with a transform on the env
one main challenge may be that the model behaves differently during execution (1-step prediction) and training (n-step predictions), but also this should be doable
if i understand correctly the transformer only uses attention over time and not over agents ? i.e. it is the same architecture replicated for each agent?

Dear Developer,
The algorithm I cited is actually a centralized multi-agent reinforcement learning algorithm independent of the time dimension. At any given timestep, it takes in the joint observations of all agents and then outputs their corresponding actions via a Transformer-based model. During the training phase, its decoder conducts parallel predicting; however, during the execution phase, the decoder part outputs agents autoregressively. My current question is whether the framework supports the implementation of such a cross-agent centralized Transformer-based network structure.
Thank you!

Ah I see! Even better then. It is possible to implement it as a model (to be used as a policy) and train it with any algorithm in the library. The only thing that is strange is that it would be a centralized policy (as it needs access to all other agents' info).
You can also create a a new algorithm representing the one described in the paper and train it with that.
When you say that the actions are computed parallelly during training, what do you mean? You still need to access the actions of the prior agents to compute the next agent action right? is it just that they are outputed at the same time? cause this would also need to be the case for execution (ie. we run the model autoregressively for all agents within one forward pass for each step)
If you want to look at a model that has similar behavior to understand how this would work in benchmarl, you can look at the GNN https://github.com/facebookresearch/BenchMARL/blob/main/benchmarl/models/gnn.py . In particular, if you use a gnn with GATv2 and full topology in benchmarl you would obtain a fully connected attention layer over the agents (similar to a transformer). The difference between that and this one seems to be the autoregressive nature of action prediction, where agents are given an order and later agents have access to earlier agents info.
Regarding this ordering, I guess my main question is related to how we can choose the ordering? what makes us establish that an agent is first and has access to less info than the others?
Also, one other concern I have is regarding this centralization. If the transformer accesses all agents info, what makes this a multiagent problem and not simply a single agent problem with a factored action and observation space? When I designed benchmarl, I did not envision the possibility of centralized policies as in my idea they would make the problem not MARL.
In the arxiv the authors propose a Decentralized version of the model (which still relies on fully observability but removes the autoregressive actions). This would be equivalent to the fully connected GNN with  GATv2 i mentioned earlier (which we already have in the library)

Hi！ Firstly, thanks for your reply!
Regarding the action prediction of MAT, in the training stage, since we can directly sample from the off-policy buffer or use on-policy samples, the joint action of agents at a timestep is determined. That is to say, we can directly use the sampled actions as the label of the Transformer and then calculate the RL loss (such as PPO loss). At this time, the agent order can be arbitrary.
However, in the execution stage, as you think, we first need to determine an agent order and then output actions agent by agent autoregressively. There are also some studies on how to choose this order. But for MAT, the impact is not significant, and we can ignore the impact of the output order.
Finally, regarding your concern about centralized policies. Indeed, for real-world applications, this CTCE mode cannot be called a multi-agent RL algorithm to some extent. But some recent studies have proved that the performance of the decentralized policy formed by further policy distillation based on this centralized policy has also achieved stronger performance compared to training directly using the CTDE paradigm. On the other hand, I think this structure can also be regarded as a MARL paradigm that allows implicit communication between agents, and there are many research studies on MARL with communicaiton. Therefore, I personally think that MAT still has certain implementation value.
I have also looked at the implementation of GNN in the repo. Analogously to MAT, it can indeed be regarded as its Encoder part. What I currently want to figure out is how to implement the neural network of the Decoder part. From another view, regarding MAT as a single-agent version of a centralized policy and then going back to the dependent TorchRL, MAT is completely different from the offline Decision Transformer implemented by TorchRL. If you can give me some advice on how to implement MAT in TorchRL or BenchMARL, I will be very grateful to you!

Ok got your points, thanks!
so it seems that the core is implementing the decoder since we already can do the encoder
i think the main problem there is that benchmarl  policies output the parameters of distributions and the sampling of the actions is done later. Thus, it seems that to run this in the proposed implementation we would need n_agents forward passes for each environment step. But the way benchmarl works is that we take one forward pass for all agents per step.
Therefore in this forward pass we cannot compute actions  autoregressively as actions are sampled outside the model. What we could do is output the logits of agents autoregressively in one pass (one agent at a time)
this would change the paradigm to be autoregressive on logits and not on actions .

Your understanding is correct, and the way of tokenizing logits is different in training and inferencing phases. You can read the following code given encoder's embeddings which is copied in MAT's repo for more details:
import torch
from torch.distributions import Categorical, Normal
from torch.nn import functional as F

def discrete_autoregreesive_act(decoder, obs_rep, obs, batch_size, n_agent, action_dim, tpdv,
                                available_actions=None, deterministic=False):
    shifted_action = torch.zeros((batch_size, n_agent, action_dim + 1)).to(**tpdv)
    shifted_action[:, 0, 0] = 1
    output_action = torch.zeros((batch_size, n_agent, 1), dtype=torch.long)
    output_action_log = torch.zeros_like(output_action, dtype=torch.float32)

    for i in range(n_agent):
        logit = decoder(shifted_action, obs_rep, obs)[:, i, :]
        if available_actions is not None:
            logit[available_actions[:, i, :] == 0] = -1e10

        distri = Categorical(logits=logit)
        action = distri.probs.argmax(dim=-1) if deterministic else distri.sample()
        action_log = distri.log_prob(action)

        output_action[:, i, :] = action.unsqueeze(-1)
        output_action_log[:, i, :] = action_log.unsqueeze(-1)
        if i + 1 < n_agent:
            shifted_action[:, i + 1, 1:] = F.one_hot(action, num_classes=action_dim)
    return output_action, output_action_log

def discrete_parallel_act(decoder, obs_rep, obs, action, batch_size, n_agent, action_dim, tpdv,
                          available_actions=None):
    one_hot_action = F.one_hot(action.squeeze(-1), num_classes=action_dim)  # (batch, n_agent, action_dim)
    shifted_action = torch.zeros((batch_size, n_agent, action_dim + 1)).to(**tpdv)
    shifted_action[:, 0, 0] = 1
    shifted_action[:, 1:, 1:] = one_hot_action[:, :-1, :]
    logit = decoder(shifted_action, obs_rep, obs)
    if available_actions is not None:
        logit[available_actions == 0] = -1e10

    distri = Categorical(logits=logit)
    action_log = distri.log_prob(action.squeeze(-1)).unsqueeze(-1)
    entropy = distri.entropy().unsqueeze(-1)
    return action_log, entropy

def continuous_autoregreesive_act(decoder, obs_rep, obs, batch_size, n_agent, action_dim, tpdv,
                                  deterministic=False):
    shifted_action = torch.zeros((batch_size, n_agent, action_dim)).to(**tpdv)
    output_action = torch.zeros((batch_size, n_agent, action_dim), dtype=torch.float32)
    output_action_log = torch.zeros_like(output_action, dtype=torch.float32)

    for i in range(n_agent):
        act_mean = decoder(shifted_action, obs_rep, obs)[:, i, :]
        action_std = torch.sigmoid(decoder.log_std) * 0.5

        # log_std = torch.zeros_like(act_mean).to(**tpdv) + decoder.log_std
        # distri = Normal(act_mean, log_std.exp())
        distri = Normal(act_mean, action_std)
        action = act_mean if deterministic else distri.sample()
        action_log = distri.log_prob(action)

        output_action[:, i, :] = action
        output_action_log[:, i, :] = action_log
        if i + 1 < n_agent:
            shifted_action[:, i + 1, :] = action

        # print("act_mean: ", act_mean)
        # print("action: ", action)

    return output_action, output_action_log

def continuous_parallel_act(decoder, obs_rep, obs, action, batch_size, n_agent, action_dim, tpdv):
    shifted_action = torch.zeros((batch_size, n_agent, action_dim)).to(**tpdv)
    shifted_action[:, 1:, :] = action[:, :-1, :]

    act_mean = decoder(shifted_action, obs_rep, obs)
    action_std = torch.sigmoid(decoder.log_std) * 0.5
    distri = Normal(act_mean, action_std)

    # log_std = torch.zeros_like(act_mean).to(**tpdv) + decoder.log_std
    # distri = Normal(act_mean, log_std.exp())

    action_log = distri.log_prob(action)
    entropy = distri.entropy()
    return action_log, entropy

Is there some ways to implement this in BenchMARL and make it work?

As I said above, BenchMARL models do not deal with distributions and just output the logits (agnostic of what distribution these withh got to)
The way you could implement this in benchmarl is by feeding sifted_logits instead of shifted_action to the decoder so that your remove any distribution dependence from the model

thank you for your reply, i will consider how to implement it^_^

Reopening this as I am very interested in it and am thinking more on how to practically do it.
I'll add it to the TODOs


Fixed Deprecations

Fixed Deprecations in the keymap file as well as the method call to register the commands. Fixes Issue #2.

Thx


Feature show value new window

Summary of the Pull Request
What is this about:
About #4 and #5
Description:
Directly display some method's value;
Add right click to open a new lookup window;
Fix navigation bar's bug;

Still have small problem when we click more that about larger 5 item ,

yes, it's longer than the window's width.Let me try to find a way to fix.

Changed the scrollviewer's style;

Fix bug about muti-window‘s messager has muti-recipient;
Aviod Dispose() Method call;
Remove back items in navigationbar when user selected;


WIP: External code snippets and example code

A first stab at #29
We want to move all example code out of the docs pages and into separate code snippets so we could use a central repository for these (for testing, benchmarking and easier maintenance). The first step is making it possible to pull in external code snippets which is already working here with files included in the /website/code directory (these can be imported using the @code alias).
The above is achieved by using Webpack's "Source Asset modules" feature, and the next steps would be further reducing boilerplate by allowing MultiLanguageCodeBlock-s to load their contents from external files as opposed to being provided inline. This is going to be tricky because of the sync/async import boundary so still figuring out our options.
Resources related to the initial commit in this PR:

https://webpack.js.org/guides/asset-modules/#source-assets
https://docusaurus.io/docs/markdown-features/react#importing-components
https://docusaurus.io/docs/api/docusaurus-config#plugins
https://docusaurus.io/docs/api/plugin-methods/

After initial build this PR could be previewed at: https://suborbital.github.io/docs/flaki/29-code-examples/

An example of an externally loaded snippet can be seen at: https://suborbital.github.io/docs/flaki/29-code-examples/atmo/runnable-api/http-client#example-runnable

So I have been digging around this house of cards awhile, I'm going to write up the results of that so that people with a better understanding/more experience in the infrastructure can chime in with regards to what would be the best way forward, cc @ospencer?
The goal
Is to be able to create tabbed <MultiLanguageCodeBlock>s from external files by pulling in the contents of an e.g. Rust or JS code snippet.
The markdown/mdx should look like something like this:
<MultiLanguageCodeBlock>

 ` ` `rust
// this is an inline Rust code block
 ` ` `

<CodeBlock language="go">
// this is an inline Go codeblock
</CodeBlock>

<CodeBlock<EMAIL_ADDRESS>// this is an external typescript runnable example, loaded from the local "@code" alias folder
// the "language" prop can be elided if its value may be inferred automatically from the file extension
</CodeBlock>

</MultiLanguageCodeBlock>

Users can mix & match inline definitions of code blocks and external ones within the same block, and the language tab will be inferred from the snippet type or file extension automatically.
It would also be really useful if we could simply reference external files, e.g. straight out of a separate repo in the URL. These resources would be fetched during the build and included in the document auto-magically. We could also provide a suitable alias for the resources to further reduce boilerplate.
Loading external code snippets straight from GitHub would also go well with the current effort of moving example runnables into a centralized repo, and referencing the files from Git would allow us to later reference explicit file versions of the files, that correspond to the given version of the docs, which we will be needing anyway once we start versioning the docs.
Preceding work
The initial <MultiLanguageCodeBlock> component made it possible to reduce the boilerplate of having to manually define the tabbed structure for every code block, and centralize language information for languages supported by Reactr.
All this initial work still relied solely on inline code blocks, but supported both markdown fenced code blocks and explicitly declared <CodeBlock> components.
Current approach
This pull request adds support for pulling in external files using static import statements in the individual markdown files and using their contents in <CodeBlock>-s. As described above this approach uses Webpack 5's source assets as opposed to the raw-loader method proposed by the Docusaurus docs.
The current approach also lives in its own plugin which will make it possible to move all the earlier bits into the plugin itself (and make them more reusable by other users, eventually moving the plugin into their own, separately installable module, and potentially upstreaming it in full or certain modifications to e.g. the <CodeBlock> component.
This approach described above is already useful and a big step forward, but according to my research, it has several limitations that make it unlikely to be able to support further improvements. I willl describe these findings in detail below.
Next steps
The next step to further enhance the current approach and reduce boilerplate has to balance two requirements:

Further reduce boilerplate by making it possible to declare the file path on the <CodeBlock> component itself (or a wrapper component), and drop the requirement for having to explicitly import the file in an import statement, e.g.:
Instead of:
import myCode from<EMAIL_ADDRESS>
<CodeBlock>{myCode}</CodeBlock>

Allow usage to be just:
<[External]CodeBlock<EMAIL_ADDRESS>/>

Make it possible to load files from an arbitrary source. This would make it possible to store the code snippets in an entirely separate repository without having e.g. explicitly pull in that repo (such as as a submodule), and further expand the flexibility of the system.
Such as allowing for usage like:
<[External]CodeBlock src="https://raw.githubusercontent.com/suborbital/examples/my/code.rs" />

The solution to 1. may have an effect of the solvability of 2. so it's best to consider these requirements together. There are multiple avenues of solving for these requirements but some of them seem to be leading to dead ends. I'm going to talk about my research into these avenues below.
Solving dynamic imports
To be able to load arbitrary files we can imagine multiple approaches, but I'm going to argue that we wouldn't want to do client-side loading of these files, and especially not from external URLs. Because of how Docusaurus is built, escape hatches are provided to execute client-side code that transform the page beyond what is possible during server-side rendering. My view is that resorting to something like this would have to be a last resort, especially given that it limits really interesting future possibilities around static generation.
With that out of the way there are various approaches to making this happen:

Dynamic import():
We load the URL referenced by the <[External]CodeBlock> component src prop as a Webpack asset within using dynamic import()s.
Docusaurus plugin APIs:
We load the content as external content in Docusaurus' dedicated lifecycle APIs, components reference this content exposed by the lifecycle APIs during build
Preprocessing step:
We preprocess the docs and preload/cache the URLs referenced by the <[External]CodeBlock> src props, we manually insert (prepend) the relevant import statements into the files and leave it to Webpack etc. to take care of loading/bundling them (the rest is the same as the current approach)

Any approach that involves dynamic imports not only suffers from the added complexity of Webpack's dynamic imports, but will also require the component itself to be async.

While googling around I uncovered this thread that mentions some of the issues inherent to webpack and dynamic imports. The linked comment in particular suggests building out a map of imports, to create a pseudo-dynamic import where at least each path is known ahead of time. With this approach I was able to sort of acheive success, but only by defining the map statically. The closest I've got to dynamic imports is a large hand rolled map of dynamic imports. --
jemjam

It seems like Docusaurus does not even support rendering async components, and expects all data to be available already during rendering.
These hurdles pretty much exclude Option 1.
It is possible to do some processing in a plugin and expose that data as lifecycle hooks. It does not seem like that this API can be efficiently used to expose data to the docs and while in theory I think setGlobalData() can be twisted into something like this it would be very bad for page load times so I don't think Option 2 would be feasible here, either.
Option 3 seems like the most feasible solution right now, and would probably be most easily achieved using a custom MDX plugin. Keep in mind that Docusaurus currently only supports the earlier MDX version, not MDX v2, the support for which is slated for Docusaurus v3. Also notable that Docusaurus is currently migrating to ES modules, so all plugins should be written in require() module syntax still (see note) and ported to ESM once that upgrade takes place.
Implementation
The current plan to implement this feature is a completely new custom remark transformer, not unlike to the one that implements Docusaurus Admonitions today.
The plugin would (pre)process every MDX document  to:

Find where external CodeBlock src loads are present
If these are not local file references (e.g. HTTP urls) it would cache the contents locally and rewrite the URL
Prepends the required import statement to the top of the file that would later load the contents via Webpack
Replaces the code block with the <CodeBlock> element parametrized correctly (e.g. does detection of the highlighting type when necessary) and with the import-ed variable passing in the file source content
It may even further expand the markdown syntax with additional medatada and processing

It may be possible to completely replace the current <MultiLanguageCodeBlock> implementation with a sufficiently advanced remark transformer (whether it would be worth it is another question). We should also ensure that we do some caching on the external files, particularly for making sure that local builds (e.g. while developing) are not impacted by the extra fetches. Remark transformers are async functions so the fetches and caching can be performed as part of the initial discovery/rewriting of the markdown content. There are existing plugins that can handle caching external content, but they seem to require a preconfigured list of content to be preloaded, so they are not really useful for our usecase but may inform implementation.

Earlier @javorszky had a couple questions regarding mostly the Preceding work section. While a lot of this work still evolving, I realize the current state wasn't really documented anywhere outside of PR descriptions so I'm going to be adding some docs in #92

The latest commit brings a prototype of the external code snippet caching.
It can be previewed using the test.md document as deployed here:
https://suborbital.github.io/docs/flaki/29-code-examples/test/
The missing ES6 support mentioned above is causing a lot of headaches, and mandates that we use outdated modules in the transformer, as all newer versions are ESM-only. We might want to consider updating to ESM/MDX v2 manually (but right now I would not recommend it as there is frequent version bumps in Docusaurus v2 due to the beta so we that would involve a lot of patching on updating).
Another thing is that a the boundaries of a custom Suborbital CodeBlockExtensions plugin is already taking shape and initially (and probably before the landing of this PR) I will move all our components into a separate plugin, but still within this repo and later we can evaluate to move it out into its own repo.
Another think I'm currently also mulling over is how to handle the imported snippets. There are two possibilities, and each come with its own pros and cons:

use asset imports:

this means pre-caching the files we pull from URLs (this already happens)
add the cached file paths as asset imports in the top of the md file
use a react {variableInterpolation} within a <CodeBlock> of fenced code block to pull in the file content

use direct embeds:

this one only needs ephemerally loading the file contents
put the contents directly into the currently md file's AST
that's it

Currently we use 2. and it seems fine but I am not completely sure whether asset imports wouldn't have an advantage still. Asset imports definitely need caching, which is good from a local-development-rebuild perspective, but bad from a "now we may need a way to cache-bust". Apropos local development, we need a story there without making users to manually change paths in the files. Thinking of allowing changes to the .cache directory -- again, asset imports may make this easier. Fwiw, we can probably make asset imports the default for local development, and direct embeds for docusaurus build.
Another note on versioning: external URLs should point to a very specific repo, and the urls should make sure they link to a tagged version of the file. This ensures e.g. reproducible builds and paves the way to #23
Note that the above will probably require us to allow people to embed this version in URLs as a placeholder and do a replacement on this during pre-processing. Since interpolations are not expanded (yet) at pre-processing time this will probably need a custom mechanism.


chore: add YARN_CLASSIC as a supported value for package manager

A new value, YARN_CLASSIC, is now the default.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mergifyio backport maintenance-v5.0 maintenance-v5.1


Dashboard Email Reports  Selector displays ID instead of Name

Screenshot

Description
When creating a Dashboard Email report the selector displays the ID instead of the name.
For Example Dashboard <1> instead of World Bank's Data
I have observed this in version 1.0.0 and 1.0.1
Chart Email reports work just fine.
Design input
Display Dashboard Name.
@junlincc

There is an open PR for this: https://github.com/apache/superset/pull/11776
Although you can now also try out the new Alert & Report manager by turning on the ALERT_REPORTS feature flag.

Hey @ktmud
Thank you
I have the following set up
FEATURE_FLAGS = {
    "KV_STORE": True,
    "SHARE_QUERIES_VIA_KV_STORE": True,
    "ENABLE_TEMPLATE_PROCESSING": True,
    "ENABLE_REACT_CRUD_VIEWS": False,
    "ROW_LEVEL_SECURITY": True,
    "ALERT_REPORTS": True,
    "DASHBOARD_NATIVE_FILTERS": True,
    "THUMBNAILS": True,
  }

However, when I visit Settings>Alerts & Reports I get a 404

Have you turned off ENABLE_REACT_CRUD_VIEWS by any chance?
The new alerts and reports kind of depend on it: https://github.com/apache/superset/blob/52ba82fb8b9f1ce5519f1a13f6417a0ce486e7df/superset/views/alerts.py#L80
@lilykuang @nytai do you think whether it makes sense to force turning on ENABLE_REACT_CRUD_VIEWS when  ALERT_REPORTS is enabled?

@ktmud I had turned it off. Let me turn it on and retry.

@ktmud
I am now able to view the reports and have set up a test report to be sent every minute.
I have celery workers and beat running as well as cache configured.

I have not received any so far, is there any further configuration I need to do?

Managed to find these from github search
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
          'email_reports.schedule_hourly': {
              'task': 'email_reports.schedule_hourly',
              'schedule': crontab(minute='*/59'),
          },
          'alerts.schedule_check': {
            'task': 'alerts.schedule_check',
            'schedule': crontab(minute='*/4'')
          },
          'reports.scheduler': {
            'task': 'reports.scheduler',
            'schedule': crontab(minute='*/5'),
          },
          'reports.prune_log': {
              'task': 'reports.prune_log',
              'schedule': crontab(minute=0, hour=0),
          },
      }

@ktmud We could throw a warning if ALERT_REPORTS is enabled, but ENABLE_REACT_CRUD_VIEWS is not. I think that fully deprecating the ENABLE_REACT_CRUD_VIEWS feature flag makes more sense, I don't think there's any strong case for continuing to support the old FAB views at this point.

@nytai agree, planning on removing the MVC CRUD ModelView code also. But we can deprecate the flag now

@jawabuu  can you please confirm if this issue is still existing / close this GH# if it's resolved? Thanks

@zuzana-vej The issue still exists. The PR to close this has not been merged.


figure out category hierarchy url patterns

Since categories names and slugs are unique, we can throw anything under /category to the category view and have a regex find the last component of the URL and query that.
We should also probably check the validity of the URL path all the way back up the tree and raise 404 if it doesn't check out. Otherwise, we could potentially clutter the site-name.com/*/ namespace with subcategory URLs.

Story.sections should be changed from a ManyToMany to a ForeignKey. If we do this, a simple unique_together of parent and slug should work fine, where parent is a primary key of a Section which is by extension related to a single Site (proof by induction).

Frontend URL parser will start at the leftmost component of the URL and make a query for each component, saving the parent id. If at any point a component does not exist, a 404/not found should be raised. Will possibly want to consider a better – O(1) instead of O(n) where n is number of URL components – caching scheme.


Rightshift >> does not work on values where the rhs is larger than bit size

Aim
nargo execute
fn main() {
    let a:u8 = 0;
    let b:u8  = 10;

    let c = a >> b;
    std::println(c);
}

Expected Behavior
return 0
Bug
The application panicked (crashed).
Message:  attempt to divide by zero
Location: /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/num/mod.rs:933

To Reproduce

Installation Method
Compiled from source
Nargo Version
nargo 0.10.2 (git version hash: c11c810d343adcf5286445b8863ceb1f19efcb52, is dirty: false)
Additional Context
this causes https://github.com/shuklaayush/noir-bigint/ test shl1 to panic i think cc @kevaundray
Would you like to submit a PR for this Issue?
Maybe
Support Needs
No response

Looks to be a duplicate of https://github.com/noir-lang/noir/issues/2250


Sorry... It seems like a little thing but is just bugging me.

before_install really should come before install.

I put this into #2015


Not show remaining days if it passed end date

Currently, if the remaining days are only shown when the challenge is active, but if the challenge passes the endDate and the status is not updated:

Add extra validation, in case the challenge status is not up to date. Optionally, we can auto correct the status if they are out of date.

Another bug related to remaining days. It shows active and both dates presented, but the progress gives negative values. Looking into it


Run Cilium CLI L7 tests

Part of https://github.com/cilium/cilium/issues/23784 to remove the L7 TLS test to cilium-cli. The test case got merged in https://github.com/cilium/cilium-cli/pull/1406 but does not yet run inside the CI. This PR removed the old test and sets up the CI to be able to run the test. The test will run once it reaches a release of the CLI tool
Migrate L7 TLS Ginkgo tests to cilium-cli

/test

Heads up that you'll need to read the ### FOR TESTING PURPOSES note at the top of the workflows and follow those steps in order to test the new functionality. It's likely that the current CI runs for the GitHub workflows are not testing the changes in this PR.

[=] Test [client-egress-l7-tls-deny-without-headers]
..
[=] Test [client-egress-l7-tls-headers]
..

Tests run now, even on cloud providers in https://github.com/cilium/cilium/pull/24489


chore: disable floating promises lint rule in tests

Summary
Recently I've been having some local lint errors in tests for floating promises, never seen it before but I've disabled the lint rule for tests.
Checklist

[x] All changes are covered by tests (or not applicable)
[x] All changes are documented (or not applicable)
[x] I reviewed the entire PR myself (preferably, on GH UI)
[x] I described all Breaking Changes (or there's none)

Reason


Ru migrate

Description

Fixed typo
-Corrected file name for script

#sign-off. CC @mikefrobbins
Thanks.


Update the go version in docs

In CONTRIBUTING.md in prerequisites section Go version is set as 1.13.9 since thanos is using 1.18 version

// @matej-g @yeya24


add Google Cloud Storage / nearline to storages

in march 2015 google introduced the Cloud Storage / Nearline plattform.
using this service as a backup-storage would make a great improvement to this project.
regards, Bernhard
http://googlecloudplatform.blogspot.co.at/2015/03/introducing-Google-Cloud-Storage-Nearline-near-online-data-at-an-offline-price.html

+1

+1
But for what I've just read, I think the feature requests tracker is this one: https://github.com/meskyanichi/backup-features
We should open an issue there to ask for this feature. @bmaeser do you want to do it (as you are the OP)?

ah, didn't see that, thanks for pointing it out @cristiangauma .
issue moved to meskyanichi/backup-features#22 , i am closing this one.


Write up instructions for installation

As a product owner I will need instructions written on how to download, install, and run from Azure.

Cool Elise you are already assigned this


Missing image resin-cli preload uninformative

resin-cli version: 7.10
Operating system and architecture:
Ubuntu x86-64

Hi I'm trying to follow the resin preload program for the Jetson tx2, but I'm always encountering 'Missing Image' the img file I already verified is there and I have access to my devices and the app id.
Any ideas of what is going on? thanks

I'm not sure I'm afraid. @zvin do you know what might cause this?

@Davidnet what command are you running ?

@Davidnet the correct command would be
resin preload resin-XXXXXXXXX-jetson-tx2-2.17.0+rev1-v7.16.6.img --app $APP_NUMBER

You can run resin preload --help for all available options.

Awesome! Is there any guidelines that I can do for adding that example to the documentation, I think it would save a time. Thanks you're awesome!


Fix build

This PR fixes build with new GCC and with Python 3.
This will fix https://github.com/ARM-software/ComputeLibrary/issues/405

Looks good, thanks for fixing these issues


Dup of #211


No releases found that matches specified selector and environment - v0.156.0

Operating system
Mac
Helmfile Version
0.156.0
Helm Version
v3.12.3
Bug description
When I execute:
helmfile -e dev -l layer=infra,release-name=argocd --no-color --log-level debug diff
I get:
err: no releases found that matches specified selector(layer=infra,release-name=argocd) and environment(dev), in any helmfile
Example helmfile.yaml
I have 2 helmfile.
The main helmfile:
environments:
    dev:
        missingFileHandler: Error
        kubeContext: test
helmDefaults:
  atomic: true
  cleanupOnFail: false

helmfiles:
  - path: helmfiles/argo/cd/helmfile.yaml

templates:
  argocd-apps: &argocd-apps
  chart: helm-charts/argocd-apps

releases:
  - name: argo-rollouts
    <<: *argocd-apps
    namespace: argo-rollouts
    disableValidation: true

The subhelmfile(helmfiles/argo/cd/helmfile.yaml):
releases:
  - name: argocd
    namespace: argocd
    createNamespace: true
    chart: argo/argo-cd
    version: 5.21.0
    missingFileHandler: Error
    atomic: true
    verify: false
    wait: true
    waitForJobs: true
    labels:
      release-name: "{{ `{{ .Release.Name }}` }}"
      layer: "infra"
    values:
      - "{{ .Environment.Name }}/values.yaml"

Error message you've seen (if any)
err: no releases found that matches specified selector(layer=infra,release-name=argocd) and environment(dev), in any helmfile
Steps to reproduce
n/a
Working Helmfile Version
v3.12.3
Relevant discussion
No response

Looks like i have to add the environment defined in the main helmfile.yaml to the sub-helmfile.yaml file ?  Is a way to skip this declaration again and the sub-helmfile.yaml inherits the variables and the environment from the main helmfile.yaml

helmfiles:
- # Path to the helmfile state file being processed BEFORE releases in this state file
  path: path/to/subhelmfile.yaml
  # Label selector used for filtering releases in the nested state.
  # For example, `name=prometheus` in this context is equivalent to processing the nested state like
  #   helmfile -f path/to/subhelmfile.yaml -l name=prometheus sync
  selectors:
  - name=prometheus
  # Override state values
  values:
  # Values files merged into the nested state's values
  - additional.values.yaml
  # One important aspect of using values here is that they first need to be defined in the values section
  # of the origin helmfile, so in this example key1 needs to be in the values or environments.NAME.values of path/to/subhelmfile.yaml
  # Inline state values merged into the nested state's values
  - key1: val1

you can try to use:
  # Override state values
  values:

@yxxhero In my main helmfile.yaml are defined the environments, my idea is that any sub-helmfile.yaml inherit these environments. I do not want to pass any additional values to sub-helmfile.yaml or is mandatory to create additional.values.yaml file with the environment definition that are also included in my helmfile.yaml
Regarding the document in the last release it is possible, in the way that I did, but it does not work.

@adecchi-2inno sub-helmfile is designed to run on its own. you can try to see: https://helmfile.readthedocs.io/en/latest/writing-helmfile/#re-using-environment-state-in-sub-helmfiles

@mumoshu WDYT?

@yxxhero
Thank you for your response. I tried that option but the error persists.
The sub-helmfile(helmfiles/argo/cd/helmfile.yaml):
releases:
  - name: argocd
    namespace: argocd
    createNamespace: true
    chart: argo/argo-cd
    version: 5.21.0
    missingFileHandler: Error
    atomic: true
    verify: false
    wait: true
    waitForJobs: true
    labels:
      release-name: argocd
      layer: "infra"
    values:
      - "{{ .Environment.Name }}/values.yaml"

The main helmfile.yaml
bases:
  - environments.yaml
  - templates.yaml

helmDefaults:
  atomic: true
  cleanupOnFail: false

releases:
  - name: argo-rollouts
    inherit:
      - template: argocd-apps
    namespace: argo-rollouts
    disableValidation: true
    #missingFileHandler: Warn
    labels:
      release-name: "{{ `{{ .Release.Name }}` }}"
      layer: "infra"
    set:
      - name: source.helm.file
        file: "helmfiles/argo/rollouts/{{ .Environment.Name }}/values.yaml"
      - name: source.repoURL
        value: "https://argoproj.github.io/argo-helm"
      - name: source.targetRevision
        value: 2.9.3
      - name: source.chart
        value: "argo-rollouts"
      - name: destination.namespace
        value: "argo-rollouts"
    needs:
      - argocd/argocd

helmfiles:
  - path: helmfiles/argo/cd/helmfile.yaml
    selectorsInherited: true
    values:
      - {{ toYaml .Values }}

But continue not finding the release.
0 release(s) matching layer=infra,release-name=argocd found in helmfile.yaml

We have also the same issue, when using a root helmfile and including files to helmfiles[0].path


Renderdoc Support for embedded systems

Description
Hi,
I'm trying to use renderdoccmd tool to capture frames of our armv7e applications but it fails to connect to the program during initialisation stage. The host system is a TV board. The TV has a Wayland Compositor system, So I'm trying to capture frames of Wayland server and client applications.
All of these applications are using OpenGLES for rendering.
Renderdoc binaries built with the following configuration.

ENABLE_EGL=ON
ENABLE_GL=OFF
ENABLE_GLES=ON
ENABLE_PYRENDERDOC=OFF
ENABLE_QRENDERDOC=OFF
ENABLE_RENDERDOCCMD=ON
ENABLE_VULKAN=OFF
ENABLE_XCB=OFF
ENABLE_XLIB=OFF
QRENDERDOC_ENABLE_PYSIDE2=OFF
ENABLE_UNSUPPORTED_EXPERIMENTAL_POSSIBLY_BROKEN_WAYLAND=ON
CMAKE_BUILD_TYPE=Debug

Below is toolkit used to build the binaries.
export C="arm-starfishmllib32-linux-gnueabi-gcc  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=softfp -mcpu=cortex-a7 -mtune=cortex-a7 -funwind-tables -rdynamic --sysroot=$SYSROOT"

export CXX="arm-starfishmllib32-linux-gnueabi-g++  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=softfp -mcpu=cortex-a7 -mtune=cortex-a7 -funwind-tables -rdynamic --sysroot=$SYSROOT"

export CPP="arm-starfishmllib32-linux-gnueabi-gcc -E  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=softfp -mcpu=cortex-a7 -mtune=cortex-a7 -funwind-tables -rdynamic --sysroot=$SYSROOT"

I'm wondering whether renderdoc can be made work in this particular system, please let me know if it's possible. it would be really helpful for our developing environment.
Steps to reproduce
When i tried launch for application for capture with following command
./renderdoccmd capture --opt-hook-children systemctl start surface-manager
It throws this error

Failed to create & inject: RenderDoc injection failed: Couldn't connect to target program. Check that it didn't crash or exit during early initialisation, e.g. due to an incorrectly configured working directory.

Environment

RenderDoc version: renderdoccmd x86 v1.32 built from 10d402358cb5eac9d35bc66216eb77eddc6e9098
Operating System: linux
Graphics API: OpenGLES 2

RenderDoc does not support ARM linux nor does it support wayland as indicated by the name of the option you enabled.


(chore) Reiterate over current elasticsearch configuration

After this is deployed we need to run the elasticsearch:vacancies:index to recreate the indices.
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [vacancy]: The [string] type is removed in 5.0 and automatic upgrade failed because parameters [ignore_malformed] are not supported for automatic upgrades. You should now use either a [text] or [keyword] field instead for field [maximum_salary]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [vacancy]: The [string] type is removed in 5.0 and automatic upgrade failed because parameters [ignore_malformed] are not supported for automatic upgrades. You should now use either a [text] or [keyword] field instead for field [maximum_salary]","caused_by":{"type":"illegal_argument_exception","reason":"The [string] type is removed in 5.0 and automatic upgrade failed because parameters [ignore_malformed] are not supported for automatic upgrades. You should now use either a [text] or [keyword] field instead for field [maximum_salary]"}},"status":400}

 Reiterate over current ElasticSearch configuration.


Install dependencies for MyPy analysis

Changes
Install a full environment to make sure, MyPy checks all the input types.
Related Issues
Closes #752
Note that this will introduce like 80 type errors we collected since last year August.

Okay, if I didn't destroy anything else with my last commit, this should be ready now.

What do we do about networkx? Pin it in another branch, merge it into master, then update this one and merge it too?

I think we should add a Github actions cache for the mypy sqlite cache file, this would speed up the mypy action greatly (like total runtime 10s)

I think we should add a Github actions cache for the mypy sqlite cache file, this would speed up the mypy action greatly (like total runtime 10s)

works like a charm, nice 🚀


Use cypress (frontend UI testing) and integration image for e2e/integration testing off the dApp

Description
Use cypress (frontend UI testing) and integration image for e2e/integration testing off the dApp.
Acceptance criteria

We need to check if the wizard solution (raw private key) still works (it's merged in master)

Tasks

[ ]

@nephix @kelsos Based on your feedback. Should we go ahead with cypress or should we implement #1511 first and check how this works?
Personally I like manual testing as it also gives useful feedback and preserves a feeling for the actual dApp.

Personally I like manual testing as it also gives useful feedback and preserves a feeling for the actual dApp

I agree

I agree we should fix the manual testing first. I don't see the automated tests as replacing manual testing but as complementing.

@nephix @kelsos Thanks for your feedback. I have been moving this a little bit down to tackle it in the next iteration (not in this one starting today)


refactor: softly connect to cwgo-pkg

What type of PR is this?

What this PR does / why we need it (English/Chinese):

Which issue(s) this PR fixes:
软连接到cwgo-pkg

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: smx-Morgan:x: JiChenSSGYou have signed the CLA already but the status is still pending? Let us recheck it.


[0.10.0] UI won't load if custom authorizer rejects anonymous access to ListNamespaces

Describe the bug
We are seeing a blank screen when loading the UI.  The network panel in the browser shows 401 responses to the backend for requests to api/v1/namespaces?.  We have the UI configured for OAuth SSO, but we never get to the point where the user can log in.  Our temporal server has a custom authorizer in place that does not permit anonymous requests to the ListNamespaces rpc operation.
To Reproduce
Put an authorizer in place that requires a bearer token from GRPC clients.
Configure the UI to use and SSO provider.
Expected behavior
UI should not attempt anonymous GRPC requests prior to the user logging in.
Desktop (please complete the following information):

OS: MacOS
Browser Firefox

thx for testing & reporting. Should be fixed in 0.10.1


JSON output? (Or other structured non-ANSI-colored format)

I'm trying to simply search for all expressions that match a pattern and summarize them in a downstream piped command. It'd be nice to not have to parse ANSI codes to do that. 😕 Great tool otherwise!

Sure! I'm always planning to support json out put! Probably I will add an Vim compatible output later.

Fixed in 0c031c19e2654c88ed2506ce151423fb013768e7!

Example showcase!


Build fails when building in parallel

Building Piranha with mvn clean install -T8 results in errors. mvn clean install without -T8 works, but takes like 10 minutes in my system.
Can you fix the errors during the parallel build? Thanks!

Hi @folker-kuhn!
We have some integration tests that currently can't run in parallel. What you could do is skip the tests and the Javadoc, as most of the build time is because of them.
mvn clean install -T8 -DskipTests -Dmaven.javadoc.skip=true

Oh, that is not a bad idea, thanks!
I think this fixes my problem, but out of the box parallel would be good to help others, not? Why dont integration test can run in parallel? Can this be fixed?

@Thihup Do you need any help here? If so let us know!

@mnriem What I'm seeing is that when testing Piranha Micro, the Shrinkwrap Maven resolver changes something in the filesystem (folder ~/.m2) in the same instant as Maven is doing something else. I actually don't know how to solve it.
The changes in the PR already improves time using mvn package, but mvn verify / mvn install still fail.

@arjantijms Any insight here? @Thihup do these changes require any changes to the current command line and or can they be committed regardless and for now only improve mvn package ?

@mnriem The changes can be already committed. I didn't merged it yet because it didn't improve the mvn install, but we can merge it now and latter fix the mvn install 😉

I suggest we merge it and open a new issue to address the remaining work later.


spark

Hello,
I want to use spark over cassandra & elasticsearch the same way you use it for heroic.
My first step is to understand how you extract data from elasticsearch and cassandra.
Could you tell me were in the code you translate query from heroic to elasticsearch and to cassandra ?
thanks
Lucile

Any translation happens in the a backend, MetricBackend for data, MetadataBackend for metadata. Look for child classes of these.
Queries are basically:

MetadataBackend, translate api query to elasticsearch query, run it to get a list of time series.
MetricBackend, for each time series, fetch the specified range and feed into aggregation framework.

The specific implementation depends on the backend. That's where you should look.

ok
i use cassandra backend so
how to query cassandra with series_id from elasticsearch ?
how to use timestamp to query cassandra ?
what is metric_key ? how to calculate it ?
what is data_time_offset ? how from timestamp you calculate it ?
thank you

hello . i wandered how you make the key serie ? hash ?
thank you

I'm not familiar with the Cassandra side of things but I believe its similar enough to Bigtable.

how to query cassandra with series_id from elasticsearch ?
how to use timestamp to query cassandra ?
what is metric_key ? how to calculate it ?
what is data_time_offset ? how from timestamp you calculate it ?

With bigtable there is a row key serializer that takes the key + tags of a series and then calculates an offset on the timestamp of the actual metric.
Most of the heavy work is done here - https://github.com/spotify/heroic/blob/master/metric/bigtable/src/main/java/com/spotify/heroic/metric/bigtable/MetricsRowKeySerializer.java#L123

thanks for your explanation.
in fact in my cassandra table i have this:
metric_key                                                                                | data_timestamp_offset | data_value --------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------  0x036e72680604636f6f7205302e3132350166043135303904686f7374047273646203696e740131036f72690265770474797065047669737588fafefebe26 |                  2616 |    2.41035
so the metric key is:
0x036e72680604636f6f7205302e3132350166043135303904686f7374047273646203696e740131036f72690265770474797065047669737588fafefebe26
and from blob to varchar i have:
nrhcoor0.125f1509hostrsdbint1oriewtypevisu�����&,
it is praticaly understandable
my tags and key with values are
nrhcoor0.125f1509hostrsdbint1oriewtypevisu
what i don't understand is the end:
88fafefebe26
???
i don't know how to make it ? from the metadata ? hash ?
thanks for replying

hmm - I'm not sure I understand what is at the end either. What i have found helpful in the past is to use the debugger as part of your IDE (I use Intelij) and set a bunch of break points in classes and methods that involve serializing and deserializing the row key.


Finish bug reducer additions for today with the ability to test runtime miscompiles/runtime crashers.

This PR contains support in bug-reducer-tester for creating test cases with miscompiles and runtime crashes.

@swift-ci Please smoke test and merge


Check the font path is the file

If the font path is the directory then it's undefined behavior.

It's internal changes. Should I update RELEASE_NOTES.md?


Missing: Linked List operations based on Arrays

DESCRIPTION
There is a file containing all the operations that can be performed on a linked list in Algorithms/Competitive Coding/Linked List  but it's only pointer based.
I would like to add an array based 'Linked_list_operation' file using C++ and if allowed I would like to rename the current one to 'Linked_list_operation_based_on_pointers'
STEPS TO REPRODUCE

Adding the new file containing Linked List operations based on array
Renaming the current file to 'Linked_list_operation_based_on_pointers'

EXPECTED OUTCOME
The repository will now have both array based and pointer based Linked List operations files.
ACTUAL OUTCOME
--
Proposed Solution
I would like to add an array based 'Linked_list_operation' file using C++ and if allowed I would like to rename the current one to 'Linked_list_operation_based_on_pointers'

create two folders then.
The already present one under Pointer based LLs and the other one under Array Implementation

Thank You Sir, I'll submit a PR soon


Add option to open torrent's folder with shortcut CTRL+L

I personally needed this option,the reason for this is while content is downloading I have to find it in /tmp/torrent-stream/... to be able to open it with a custom player (in my case beamer). Now all you need to do is hit CTRL+L to open that folder.
MaxOSX, Ubuntu, Windows are supported and tested. Other OS will be opened with command xdg-open which should be supported on other UNIX distributions.

could you fix the standard errors? otherwise LGTM

Issue fixed, thanks @mafintosh

Awesome! Released in 0.34.0

I found a bug with this. If you're selecting from an index (peerflix "magnet-link" --list) CTRL + L doesn't work.


Convert the data from bin100 to cellbin resolution

Hi,
I am currently working on the Stereo-seq data of mouse brain samples. l need first to use 100 bin size for structure and are identification and then perform further analysis at cellbin resolution in a specific region identified by 100 bin size. I am wondering if there is any way to achieve this.
Thanks so much for your help!

Do you mean you want to first do annotation on a bin-size 100 data and then convert it to a cell-bin to do further analysis?

Yes! Since I first select the region of interest based on the bin-size 100 and want to perform further analysis at single-nucleus resolution in that area, I need to convert the selected area from bin100 to cellbin data.
Thanks!
From: tanliwei @.>
Date: Wednesday, July 17, 2024 at 10:53 PM
To: STOmics/Stereopy @.>
Cc: Gong, Yun @.>, Author @.>
Subject: Re: [STOmics/Stereopy] Convert the data from bin100 to cellbin resolution (Issue #307)
External Sender. Be aware of links, attachments and requests.
Do you mean you want to first do annotation on a bin-size 100 data and then convert it to a cell-bin to do further analysis?
—
Reply to this email directly, view it on GitHubhttps://github.com/STOmics/Stereopy/issues/307#issuecomment-2235268005, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHJD5DX27JQYDR6HUVPTH2LZM43Z5AVCNFSM6AAAAABLA2TTSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZVGI3DQMBQGU.
You are receiving this because you authored the thread.Message ID: @.***>

If you already have the cell-bin file (.gef or .gem) corresponding to the square-bin file you are using, you can use stereo.utils.cluster_bins_to_cellbins to map the annotations of square-bin to the corresponding cell-bin.

If you already have the cell-bin file (.gef or .gem) corresponding to the square-bin file you are using, you can use stereo.utils.cluster_bins_to_cellbins to map the annotations of square-bin to the corresponding cell-bin.
Example codes:
import stereo as st
from stereo.utils import cluster_bins_to_cellbins

data_bin = st.io.read_gef(bin_file, bin_type='bins', bin_size=100)

# do annotation on data_bin and name the annotation as 'annotation'

data_cellbin = st.io.read_gef(cellbin_file, bin_type='cell_bins')

cluster_bins_to_cellbins(data_bin, data_cellbin, 'annotation')

# do analysis on data_cellbin, the annotation mapped to data_cellbin is named as 'annotation_from_bin'

Got it!.
Thanks so much!
From: tanliwei @.>
Date: Thursday, July 18, 2024 at 1:00 AM
To: STOmics/Stereopy @.>
Cc: Gong, Yun @.>, Author @.>
Subject: Re: [STOmics/Stereopy] Convert the data from bin100 to cellbin resolution (Issue #307)
External Sender. Be aware of links, attachments and requests.
If you already have the cell-bin file (.gef or .gem) corresponding to the square-bin file you are using, you can use stereo.utils.cluster_bins_to_cellbinshttps://stereopy.readthedocs.io/en/latest/content/stereo.utils.cluster_bins_to_cellbins.html to map the annotations of square-bin to the corresponding cell-bin.
—
Reply to this email directly, view it on GitHubhttps://github.com/STOmics/Stereopy/issues/307#issuecomment-2235678402, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHJD5DTJR5YJNSSVACUTUUTZM5KYTAVCNFSM6AAAAABLA2TTSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZVGY3TQNBQGI.
You are receiving this because you authored the thread.Message ID: @.***>


Have documentation + samples for PackageReference workarounds

We have been talking to customers about the workarounds for the following limitations with PackageReference. We should have it documented along with samples.

Web.config transforms
Copying contents files to project
...

Not eligible for docs day.

@nkolev92 , arent there already some samples? Can you link them here?

There's docs for https://github.com/NuGet/docs.microsoft.com-nuget/blob/master/docs/create-packages/Source-and-Config-File-Transformations.md#xdt-transforms
The content files part is a harder to get right and we have no sample that I could find at this point.

Much of this issue seems to be addressed with previous migration docs. Let's close this for now and open new issues if there's not sufficient docs here.


Digest::Digest is deprecated; use Digest

I get the warning multiple times in each execution. Could upgrade to new versions of digest please?

I concur - I'd love to get rid of these warnings.

I think this may actually be an issue with right_aws which has the bug fixed in master, but unfortunately the most recent gem is right_aws-3.1.0 was released in June 13, 2013, whereas the fix wasn't applied until December 30, 2013. So I guess pull the repo, build a gem and install your own gem to fix the issue.


deepin.qcef: 1.1.7 -> 1.1.8

Motivation for this change
Update to version 1.1.8
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 88960 1

2 packages built:
- deepin.qcef
- netease-cloud-music


Shader on MeshInstance2D has no effect

Godot version
3.3.2.stable.arch_linux
System information
Arch Linux, GLES3, Ryzen 7 PRO 4750U integrated graphics, AMDGPU driver
Issue description
When creating a MeshInstance2D and adding a QuadMesh with a ShaderMaterial (directly assigned to the mesh), the shader has no effect. This happens for both vertex and fragment shaders.
I'm aware that adding shaders to the material property of the MeshInstance2D itself will work, but I'm not sure if this is the correct approach when using MultiMeshInstances (which exhibit the same behavior).
If this is intended, a note in the docs would be awesome.
Steps to reproduce

Create a MeshInstance2D.
Create and assign a QuadMesh.
Create and assign a ShaderMaterial to the QuadMesh.
Create a new shader and assign it to the ShaderMaterial:

shader_type canvas_item;

void vertex() {
	VERTEX.y *= sin(TIME);
}

void fragment() {
	COLOR = vec4(0.0, 0.0, 0.0, 1.0);
}

Minimal reproduction project
Very small example repo: https://github.com/raffomania/godot-2dmesh-shader-bug

@raffomania Can you reproduce this after disabling batching in the Project Settings? Also, can you reproduce this after switching to the GLES2 renderer (Rendering > Quality > Driver > Driver Name)?

The current logic for materials in 2D is based on inheriting materials from parents. When rendering an object the 2D renderer looks to see if the object inherits from a parent, and if not, it uses the item's material.
https://github.com/godotengine/godot/blob/1cd10461ca966b5e05f47f0648b89fc268af51d8/drivers/gles3/rasterizer_canvas_gles3.cpp#L146-L148
It has been like this since long before MeshInstance2D was created. So it may be worth rewriting parts of the 2D renderer to include per-mesh materials. But see this comment
https://github.com/godotengine/godot/blob/1cd10461ca966b5e05f47f0648b89fc268af51d8/drivers/gles3/rasterizer_canvas_gles3.cpp#L817
Which says that it may be difficult to do.
The reason for the difficulty is that canvas items support drawing multiple items (for example, through the drawing API) so a single CanvasItem can be responsible for drawing all kinds of primitives (including meshes, polygons, rects, etc.) The CanvasItem can be seen as a primitive which joins other primitives together under a common material.
To support Meshes with unique materials (like in the 3D renderer) we would need to add a system into the 2D renderer that breaks meshs off into their own CanvasItem internally so that they do not get rendered along with other primitives in the CanvasItem. As the comment linked above says, this would be rather difficult to do (but probably not impossible!).

Thanks for the detailed explanation! This clarifies a lot. Do you think it makes sense to add a note to the docs regarding this? I'd be happy to write a PR.

Do you think it makes sense to add a note to the docs regarding this? I'd be happy to write a PR.

Yes! That would be very helpful.

Has anything changed since than? All my 2D Mesh instances show up as completely white with an assigned surface, I am unsure if this is related?


fix(dashboard): fix card range actions

fix(dashboard): rewire the onCardAction support correctly
fix(cardUtilityFunctions): add support the the lastQuarter, nextQuarter and export the determineCardRange utility for downstream consumers

:tada: This PR is included in version 1.97.1 :tada:
The release is available on:

npm package (@release-1.x dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


allow formatErrorMessage callback

something like this can be useful for logging the cost. wdyt?

I'd rather have some callback for formatting the error message to send up to the client.

ok. can you elaborate on the api? getErrorMessage(cost, documentNode) maybe?

just formatErrorMessage(cost) perhaps

Wait, is the idea here that you want to log all costs on queries? Or what's the goal here? Maybe it'd be better to re-use the visitor to create a different rule that only does logging?

initially i thought about logging all costs but figured it makes sense to do it only when validation fails. no? so logging when validation fails is the goal.

I think there are two separate concerns – what to do right now in dev to figure out how to set cost limits, then what we'll want to do in a production deploy.

Thinking about this some more, probably https://github.com/4Catalyzer/graphql-validation-complexity/pull/3/commits/4226b5840824d63f46e724494765437256e5c00a is the right way to go.

it'd be better to re-use the visitor to create a different rule that only does logging. idk how that would work.
@taion maybe do both. onCost and formatErrorMessage. onCost solves all our usecases but i can imagine somebody needing formatErrorMessage. wdyt?

Sure – still could be useful to tell without e.g. looking at server logs more details on queries that exceed the cost limit. You'd only want to turn it on in dev, though – would probably be a security issue to enable it in prod.

@taion anything left here?

@taion updated.


DS-2491 set deletedRecord=transient in OAI Identify

https://jira.duraspace.org/browse/DS-2491

As said on Jira, +1


feat: allow client side translation loading

Tried to apply the changes done in #1170 to the latest version.

@isaachinman any feedback?

Hey @adrai – sorry for the delay. I should be able to look at this on Friday.

Just a few thoughts to get the discussion going.
Also, in general I don't think I'd like to maintain a separate client side example, as it may mislead users. The main use case I've seen for client side loading is error pages, so maybe we can drop a custom _error.tsx into the existing simple example instead? Link for more context: #1668 (comment)

but then we would also need to inject the config here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/pages/_app.js#L5 like https://github.com/isaachinman/next-i18next/pull/1726/files#diff-358d2e15df81c4ef4f244307017badb7fb1aef3347af41f43f760ed4fb2f0d66R6

@adrai Couldn't we pass the config via getInitialProps on _error.tsx? I think most users rely on getStaticProps and getServerSideProps anyways, so we may as well try to come up with a pattern that's useful for the majority. I'm happy to change any internals that make sense to change.

I don't know if I understood your idea, but at the end this will lead to changing the next-i18next.config.js file in your simple example to look more like the one of the other example. Do you really want to mix this stuff?

@maurobalbi @hoangtrung99 @Javimtib92 @cmaerz @natterstefan @DanPurdy @nicolasconnault @peter-jozsa @kfazinic

@isaachinman if I move the client-loading example to i18next-http-backend, do you think this PR could land next week?
Seems I'm more interested in this to land now, than the devs that commented on the first PR 😁

Thank you for pinging me @adrai but unfortunately we ended up not using this package because of the app re-mount issues (if I can remember well) so I think I'm not best person for this job. But I hope these fixes will land in this package and maybe we can switch back to it in the future. Thanks for all your efforts!

Thanks for the ping as well, @adrai. I am afraid I am not the right person to ask for help in this case. We constantly vacillate between whether or not to stay with this library, it has its quirks (also in relation to Storybook and other tools). Thank you for taking care of this and investing time to improve it.

Thanks for the ping, but this was a dealbreaker, and we have moved on in the meantime. Which is a shame, because it feels like exploitation of the open source spirit.  Deadlines can be harsh…

@peter-jozsa That issue has been fixed.
@natterstefan Please feel free to open an issue.
@maurobalbi Loading translations on the client side has always been possible, although perhaps support couldn't be considered "first class".
@adrai I will re-review this weekend.

@peter-jozsa That issue has been fixed.

Thanks for informing me! In this case I think the library is back in the game for us 😉

if I https://github.com/isaachinman/next-i18next/pull/1726#discussion_r825792561 to i18next-http-backend

@adrai Yes, I would prefer that.

@isaachinman should I keep the readme or should I also remove that? https://github.com/isaachinman/next-i18next/pull/1726/files#diff-4e752105f99f00968883e3580ff21527c960ecdd8954c3cd11b50a64c6dfe957

@adrai I think we can just include a short note/link in the main README.

ok, done so.

@isaachinman ready to be merged?

@isaachinman fyi: rebased with master to resolve the conflicts...

... ok probably next week 😉

@isaachinman don't worry... I know... I'm in the same situation.
I can wait, no problem... just wanted to trigger a reminder notification 😉

I elaborated all your comments, the only one that I've not done is: https://github.com/isaachinman/next-i18next/pull/1726/files#r841191426

A huge thank you to @adrai for all your help and patience 🙏

I have to thank you ;-)

Published in v11.0.0.

The example is published: https://github.com/i18next/i18next-http-backend/tree/master/example/next
And there's also a specific applied blog post describing it


Adding a MudBlazor component to a component trigger a full rebuild instead of Hot Reload

Bug type
Other
Component name
No response
What happened?
Hello,
When I add a blazor component a full rebuild is needed and we need to manually refresh the page in the browser
Expected behavior
Smooth hot reload
Reproduction link
https://github.com/Socolin/ReproBugBlazor
Reproduction steps

Create a new project dotnet new blazorwasm
add blazor https://mudblazor.com/getting-started/installation#ready-for-more?
run dotnet watch
Add <MudButton>Hello !</MudButton>  in Index.razor

Relevant log output
watch : File changed: /tmp/tmp1/Pages/Index.razor.
watch : Unable to apply hot reload because of a rude edit.
watch : Do you want to restart your app - Yes (y) / No (n) / Always (a) / Never (v)?

Version (bug)
6.0.2
Version (working)
No response
What browsers are you seeing the problem on?
Chrome
On what operating system are you experiencing the issue?
Linux
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@mikes-gh ?

Yes I'm also having this behavior... the lost of hot reload when using MudBlazor :'(

Is there any progress on fixing this issue? I see the last comment is over a year old by now.

Hi

Is there any progress on fixing this issue? I see the last comment is over a year old by now.

It's not really clear what to fix and how, There are no rules by Microsoft on what shouldn't be done in the component to not break the hot reload, the MudButton has pretty simple and common code for a Blazor component.
It looks like a more suitable issue to create in the dotnet repository where hot reload lives.
For this reason I will close this issue, otherwise it will be forever stale as I don't anything what could be done from our side.


[Storage] Migrate test_retry.py to test proxy

This PR migrates test_retry.py and test_retry_async.py to test proxy.

API change check
API changes are not detected in this pull request.


Incorrect instructions for asking price aspect of add property command

When in property mode, the add command instruction has incorrect instructions with regards to the asking price. As seen in the picture, the inputted a/100000,000 technically "has 3 digits after the comma", but it's still marked as incorrect. While the inputted number shouldn't actually be valid, the instructions can perhaps be clearer.
To reproduce:

Run the ABCLI app
Switch to Property mode using the switch p command
Input an invalid asking price in an add command, e.g. add n/John Doe p/98765432 l/Clementi a/100000,000 t/HDB

Discussed: Resolved.


fixed for python3 and for new site

Hi @rushic24,
Thanks for your PR.
I checked your code, tried it too,
Please remove all the useless parenthesis before I merge your PR

Hi @rushic24,
Thanks for your PR.
I checked your code, tried it too,
Please remove all the useless parenthesis before I merge your PR

Hi,
useless parenthesis as in ? Its for python3, print() needs them


Minor : Update database.displayName in search indexes from text to keyword

Describe your changes:
Fixes  exception on  search query for all

I worked on ... because ...

Type of change:

[x] Bug fix
[ ] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@sonika-shah if I am not mistaking you can always do terms/aggregation against this field with displayName.keyword since we have this mapping
          "displayName": {
             "type": "text",
             "fields": {
               "keyword": {
                 "type": "keyword",
             }
           },

if we change the mapping for display name to keyword, then search query such as foo for a My Foo Table display name will not return a match.


Constant pooling for constant buffers

To avoid per-invocation buffer allocation and copying we can outline constant buffers to globals and then via #1144 deduplication fold many of them together. From there we can rewrite the constant accesses to be subviews into a larger constant pool.
At the flow level we could have flow.constant_pool containing flow.constant ops, flow.constant_pool.lookup to get the tensor by symbol, etc. This allows us to group constants by usage pattern (if we care) and placements we know at the flow level. It would also let us insert slicing ops that have a chance of getting fused into dispatch regions such that they became free indexing later on. If each constant pool was mapped to a hal.buffer this would save us lots of descriptor set/binding overhead.
At the hal level we can lower flow.constant_pool to a hal.buffer variable and initialize it with the constant contents. This would be a good chance to ensure we are properly allocating the constants as mapped memory vs. alloc+copy; a hal device property (unified memory/discrete memory) could be used in a hal.device.switch to either create a command buffer to issue on a dma queue or just use the allocator to get a hal.buffer from the constant rodata.

Currently iterating on this (and trying to keep both buffer allocation and device placement in mind). Have a flow pass that outlines large constants, which then if there's flow-level stream placement can use that to determine which constants may need to be sharded. The constants are represented as immutable flow.variables so flow.variable.load/flow.variable.load.indirect both work just as with normal variables. HAL will then take those immutable variables and pool them based on target device type and usage semantics (initialization-only vs. continued usage). Deduping in #1144 will handle taking care of the immutable flow.variables that come from multiple functions nicely.

(the confounding factor here in HAL is that we need to observe potential hardware requirements; either with respect to unified vs. discrete memory and things like requirements on allowable offsets that can be bound to the device: https://vulkan.gpuinfo.org/displaydevicelimit.php?name=minStorageBufferOffsetAlignment&platform=android)

(another confounding factor is maximum buffer size; https://vulkan.gpuinfo.org/displaydevicelimit.php?name=maxStorageBufferRange&platform=android - systems with a max of 128MB are common - so pools will need to be able to be split across storage buffers)

oh cool, maxStorageBufferRange is actually just the maximum window size. Still will likely be an issue if any single layer ends up with parameters >128MB, but that's unlikely on Android and on Windows 4GB is the most common value.

(maxMemoryAllocationSize is the query for maximum allocation size, however gpuinfo doesn't track that and it's really hard to get access to)

2/3 done;
so far, constant pools are identified by finding flow.variables with constant values:
flow.variable @_large_const_0 dense<1.0> : tensor<1xf32>
flow.variable @_large_const_1 dense<2.0> : tensor<64xf32>
func @fn1() -> (tensor<1xf32>, tensor<4xf32>, tensor<3xi8>) {
  %cst0 = flow.variable.load @cst0 : tensor<1xf32>
  %cst1 = flow.variable.load @cst1 : tensor<4xf32>
  %cst2 = flow.variable.load @cst2 : tensor<3xi8>
  return %cst_0, %cst_1 : tensor<1xf32>, tensor<4xf32>, tensor<3xi8>
}

->
hal.constant_pool @pool attributes {
    buffer_constraints = #hal.buffer_constraints<max_allocation_size =<PHONE_NUMBER>,
                                                 min_buffer_offset_alignment = 32,
                                                 max_buffer_range = 134217728,
                                                 min_buffer_range_alignment = 4>
  } {
  hal.constant_pool.value @cst0 = dense<1.000000e+00> : tensor<1xf32>
  hal.constant_pool.value @cst1 = dense<[2.1, 3.2, 4.3, 5.4]> : tensor<4xf32>
  hal.constant_pool.value @cst2 = dense<[6, 7, 8]> : tensor<3xi8>
}
func @fn1() -> (tensor<1xf32>, tensor<4xf32>, tensor<3xi8>) {
  %cst0 = hal.constant.load @pool::@cst0 : tensor<1xf32>
  %cst1 = hal.constant.load @pool::@cst1 : tensor<4xf32>
  %cst2 = hal.constant.load @pool::@cst2 : tensor<3xi8>
  return %cst0, %cst1, %cst2 : tensor<1xf32>, tensor<4xf32>, tensor<3xi8>
}

then the pools are packed into one or more storage buffers based on the buffer constraints:
  hal.constant_pool @pool attributes {buffer_constraints = #hal.buffer_constraints<max_allocation_size =<PHONE_NUMBER>, min_buffer_offset_alignment = 32, max_buffer_range = 134217728, min_buffer_range_alignment = 4>} {
    hal.constant_pool.splat @cst0 = dense<1.000000e+00> : tensor<1xf32>
    hal.constant_pool.span @cst1 : tensor<4xf32> = @pool_storage_0[#hal.byte_range<0, 16>]
    hal.constant_pool.span @cst2 : tensor<3xi8> = @pool_storage_0[#hal.byte_range<32, 3>]
  }
  hal.variable @pool_storage_0 dense<[102, 102, 6, 64, -51, -52, 76, 64, -102, -103, -119, 64, -51, -52, -84, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 7, 8, 0]> : vector<36xi8>
  func @fn1() -> (tensor<1xf32>, tensor<4xf32>, tensor<3xi8>) {
    %const = hal.constant.load @pool::@cst0 : tensor<1xf32>
    %const_0 = hal.constant.load @pool::@cst1 : tensor<4xf32>
    %const_1 = hal.constant.load @pool::@cst2 : tensor<3xi8>
    return %const, %const_0, %const_1 : tensor<1xf32>, tensor<4xf32>, tensor<3xi8>
  }

(splat is an example here, but this also allows us to express different compression/packing techniques)
the final part will be materializing the runtime buffers that the storage is unpacked into (or mapped into, if possible), which will cause the hal.constant_pool to be disposed and the hal.constant.load ops to be replaced with constant references to the runtime buffers and ranges.
In the ideal case, we'll end up with a handful of well-aligned storage buffers as rodata in the final binary and some generated IR that either maps them directly into memory for use or unpacks and uploads the storage->runtime buffers using DMA.
Future work is really easy to slot in to the pipeline for things like cross-constant referencing ("small constant A is found in large constant B, so just reuse the storage by slicing into B"), RLE encoding, special value construction (ala #1152) like "make this a scaling matrix of xyz" etc. We can also do storage layout optimizations to make paging in the mapped memory/DMA more efficient, profile-guided runtime placement (storing constants accessed near each other in time near each other in space), etc. What's there is the basics only :)

(the other thing this enables is externalized constant storage, where we could have an #iree.external_constant attribute that references a span of bytes in an external file, keeping compiler overhead lower)

no real progress today due to meetings 😢
targeting end-of-week

Landed in #3424!