Fire Remote with 4 Bottom Buttons? How do I decode the buttons?

I have the newest fire tv voice remote 3rd gen and I have no idea how to decode the buttons to put them into the keymap. Is there any way I can do this?

Unfortunately for anything I connect as soon as I hit a button it panics and reboots

If you could upload the logs, then maybe I can have a better understanding what is going on. Unfortunately I have no possibility to test things at the moment because I am away from home for a longer period of time.
If you use Home Assistant maybe look at the new Bluetooth Proxy feature that allows to connect Bluetooth devices directly to HomeAssistant via an ESP32.

sorry for the late responce haha this is what i get when i press the down button it registers the button and identifys it but panics
[07:57:58][I][ble_hid_client.cpp:644]: Start scanning for HID-BLE-Devices for 3600 seconds [07:57:58][W][ota:095]: Last Boot was an unhandled reset, will proceed to safe mode in 8 restarts [07:58:33][I][ble_hid_client.cpp:463]: Found HID device: [07:58:33][I][ble_hid_client.cpp:466]:  address: f8:fc:e1:b4:f9:b9 [07:58:33][I][ble_hid_client.cpp:468]:  name: AR [07:58:33][I][ble_hid_client.cpp:471]: connect to the remote device. [07:58:34]lld_pdu_get_tx_flush_nb HCI packet count mismatch (0, 1) [07:58:34][I][esp-idf:000]: W (98104) BT_HCI: DiscCmpl evt: hdl=0, rsn=0x3e [07:58:34] [07:58:36][I][ble_hid_client.cpp:443]: Pairing success [07:58:36][I][ble_hid_client.cpp:447]: auth mode = ESP_LE_AUTH_BOND [07:58:44][I][ble_hid_client.cpp:307]: Pressed key DOWN [07:58:44]Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled. [07:58:44] [07:58:44]Core  0 register dump: [07:58:44]PC      : 0x400d33eb  PS      : 0x00060130  A0      : 0x800d2c10  A1      : 0x3ffdbbf0 WARNING Decoded 0x400d33eb: esphome::BleHidClientComponent::getKeycodeSensor() at C:\Users\Brett\Downloads\esphome-ble-remote-master\esphome-ble-remote-master\.esphome\build\ble-remote-connector/src/ble_hid_client.cpp:650  (inlined by) gattc_profile_event_handler at C:\Users\Brett\Downloads\esphome-ble-remote-master\esphome-ble-remote-master\.esphome\build\ble-remote-connector/src/ble_hid_client.cpp:308 [07:58:44]A2      : 0x00000000  A3      : 0x00000051  A4      : 0x00000051  A5      : 0x3ffb7dfc [07:58:44]A6      : 0x3ffc60e0  A7      : 0x3ffdbbf0  A8      : 0x800d33e8  A9      : 0x3ffdbb90 [07:58:44]A10     : 0x00000003  A11     : 0x3f40015d  A12     : 0x00000133  A13     : 0x3f40061b [07:58:44]A14     : 0x3ffdbc18  A15     : 0x00000006  SAR     : 0x00000004  EXCCAUSE: 0x0000001c [07:58:44]EXCVADDR: 0x00000014  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff [07:58:44] [07:58:44]Backtrace:0x400d33e8:0x3ffdbbf0 0x400d2c0d:0x3ffdbc60 0x400e1cd2:0x3ffdbc80 0x400e21fd:0x3ffdbca0 0x400fd601:0x3ffdbce0 0x400ff54f:0x3ffdbd00 WARNING Found stack trace! Trying to decode it WARNING Decoded 0x400d33e8: esphome::BleHidClientComponent::getKeycodeSensor() at C:\Users\Brett\Downloads\esphome-ble-remote-master\esphome-ble-remote-master\.esphome\build\ble-remote-connector/src/ble_hid_client.cpp:650  (inlined by) gattc_profile_event_handler at C:\Users\Brett\Downloads\esphome-ble-remote-master\esphome-ble-remote-master\.esphome\build\ble-remote-connector/src/ble_hid_client.cpp:308 WARNING Decoded 0x400d2c0d: esphome::esp_gattc_cb(esp_gattc_cb_event_t, unsigned char, esp_ble_gattc_cb_param_t*) at C:\Users\Brett\Downloads\esphome-ble-remote-master\esphome-ble-remote-master\.esphome\build\ble-remote-connector/src/ble_hid_client.cpp:534 WARNING Decoded 0x400e1cd2: btc_gattc_cb_to_app at C:/Users/Brett/.platformio/packages/framework-espidf/components/bt/host/bluedroid/btc/profile/std/gatt/btc_gattc.c:24 WARNING Decoded 0x400e21fd: btc_gattc_cb_handler at C:/Users/Brett/.platformio/packages/framework-espidf/components/bt/host/bluedroid/btc/profile/std/gatt/btc_gattc.c:1002 WARNING Decoded 0x400fd601: btc_thread_handler at C:/Users/Brett/.platformio/packages/framework-espidf/components/bt/common/btc/core/btc_task.c:184 WARNING Decoded 0x400ff54f: osi_thread_run at C:/Users/Brett/.platformio/packages/framework-espidf/components/bt/common/osi/thread.c:68 [07:58:44] [07:58:44] [07:58:44]ELF file SHA256:<PHONE_NUMBER>409416 [07:58:44] [07:58:44]Rebooting... [07:58:44]ets Jun  8 2016 00:22:57 [07:58:44] [07:58:44]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [07:58:44]configsip: 0, SPIWP:0xee [07:58:44]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [07:58:44]mode:DIO, clock div:2 [07:58:44]load:0x3fff0030,len:7580 [07:58:44]ho 0 tail 12 room 4 [07:58:44]load:0x40078000,len:14912 [07:58:44]load:0x40080400,len:3688 [07:58:44]entry 0x4008067c
i will look into the proxy as well thank you!

Also with 9.0 I'll see if Bluetooth will work natively since I am close to the server

Still can't get it to work what esp do you use? Maybe I need one with more ram?

I have an ESP32-WROOM-32. Have you changed anything in the .yaml file other than what is needed to be changed according to the comments?

Nothing changed other than what I needed to change :-)

Any thoughts on what I'm doing wrong?

If there's anything I can do to help, I'm happy to provide what I can haha I see a few new commits and reverts I appreciate the work your putting in so much!!!

Just ran into the same issue, let me know if you need more input, I tried going back 2 commits, but ran into compile issues... did not try to dig further yet though...

So I finally got around to look at the issue and was able to fix it with pull request #11. Please let me know if it is working for you now, so I cn close this issue. Thank You.

Now once it compiles and flashes it scans and hangs showing
[10:16:29][I][ble_hid_client.cpp:646]: Start scanning for HID-BLE-Devices for 3600 seconds
I've tried different remotes all in pairing mode and never goes from that line

I have just added a bit of code that hopefully fixes it for you, if not it would be great if you could set the logger to DEBUG mode and upload the log. The new version is in this branch.

One thing though devices that used to connect in the reboot version no longer try to connect. I'll get a log but it just scans and continues without connecting.

Thought I would reopen it just in case other remotes are possible. If not please close this thanks again!

[18:33:13]E (51691) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time: [18:33:13]E (51691) task_wdt:  - loopTask (CPU 1) [18:33:13]E (51691) task_wdt: Tasks currently running: [18:33:13]E (51691) task_wdt: CPU 0: IDLE [18:33:13]E (51691) task_wdt: CPU 1: IDLE [18:33:13]E (51691) task_wdt: Aborting. [18:33:13] [18:33:13]abort() was called at PC 0x4012126c on core 0 [18:33:13] [18:33:13] [18:33:13]Backtrace:0x40082b7a:0x3ffbe3d00x40090eb5:0x3ffbe3f0 0x4009708e:0x3ffbe410 0x4012126c:0x3ffbe480 0x40083541:0x3ffbe4a0 0x4019fe57:0x3ffbd110 0x401219c5:0x3ffbd130 0x40092288:0x3ffbd150  [18:33:13] [18:33:13] [18:33:13] [18:33:13] [18:33:13]ELF file SHA256: 9e09c223ef36115e [18:33:13] [18:33:13]Rebooting... [18:33:13]ets Jul 29 2019 12:21:46 [18:33:13] [18:33:13]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [18:33:13]configsip: 0, SPIWP:0xee [18:33:13]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [18:33:13]mode:DIO, clock div:2 [18:33:13]load:0x3fff0030,len:6656 [18:33:13]load:0x40078000,len:14848 [18:33:13]load:0x40080400,len:3792 [18:33:13]entry 0x40080694 [18:33:13]I (27) boot: ESP-IDF 4.4.2 2nd stage bootloader [18:33:13]I (27) boot: compile time 18:22:37 [18:33:13]I (27) boot: chip revision: 3 [18:33:13]I (30) boot_comm: chip revision: 3, min. bootloader chip revision: 0 [18:33:13]I (37) boot.esp32: SPI Speed      : 40MHz [18:33:13]I (42) boot.esp32: SPI Mode       : DIO [18:33:13]I (46) boot.esp32: SPI Flash Size : 4MB [18:33:13]I (51) boot: Enabling RNG early entropy source... [18:33:13]I (56) boot: Partition Table: [18:33:13]I (60) boot: ## Label            Usage          Type ST Offset   Length [18:33:13]I (67) boot:  0 otadata          OTA data         01 00 00009000 00002000 [18:33:13]I (74) boot:  1 phy_init         RF data          01 01 0000b000 00001000 [18:33:13]I (82) boot:  2 app0             OTA app          00 10 00010000 001c0000 [18:33:13]I (89) boot:  3 app1             OTA app          00 11 001d0000 001c0000 [18:33:13]I (97) boot:  4 nvs              WiFi data        01 02 00390000 0006d000 [18:33:13]I (104) boot: End of partition table [18:33:13]I (109) boot_comm: chip revision: 3, min. application chip revision: 0 [18:33:13]I (116) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=2af68h (175976) map [18:33:13]I (188) esp_image: segment 1: paddr=0003af90 vaddr=3ffbdb60 size=05088h ( 20616) load [18:33:13]I (196) esp_image: segment 2: paddr=00040020 vaddr=400d0020 size=d2c58h (863320) map [18:33:13]I (509) esp_image: segment 3: paddr=00112c80 vaddr=3ffc2be8 size=00280h (   640) load [18:33:13]I (509) esp_image: segment 4: paddr=00112f08 vaddr=40080000 size=1cae0h (117472) load [18:33:13]I (563) esp_image: segment 5: paddr=0012f9f0 vaddr=50000000 size=00010h (    16) load [18:33:13]I (577) boot: Loaded app from partition at offset 0x10000 [18:33:13]I (577) boot: Disabling RNG early entropy source... [18:33:13]I (589) cpu_start: Pro cpu up. [18:33:13]I (589) cpu_start: Starting app cpu, entry point is 0x4008223c [18:33:13]I (576) cpu_start: App cpu up. [18:33:13]I (606) cpu_start: Pro cpu start user code [18:33:13]I (606) cpu_start: cpu freq: 160000000 [18:33:13]I (606) cpu_start: Application information: [18:33:13]I (610) cpu_start: Project name:     so-ring [18:33:13]I (615) cpu_start: App version:      2022.12.3 [18:33:13]I (620) cpu_start: Compile time:     Jan 15 2023 18:20:11 [18:33:13]I (626) cpu_start: ELF file SHA256:  9e09c223ef36115e... [18:33:13]I (632) cpu_start: ESP-IDF:          4.4.2 [18:33:13]I (637) heap_init: Initializing. RAM available for dynamic allocation: [18:33:13]I (644) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM [18:33:13]I (650) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM [18:33:13]I (656) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM [18:33:13]I (663) heap_init: At 3FFCB098 len 00014F68 (83 KiB): DRAM [18:33:13]I (669) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM [18:33:13]I (675) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM [18:33:13]I (682) heap_init: At 4009CAE0 len 00003520 (13 KiB): IRAM [18:33:13]I (689) spi_flash: detected chip: generic [18:33:13]I (693) spi_flash: flash io: dio [18:33:13]I (699) cpu_start: Starting scheduler on PRO CPU. [18:33:13]I (0) cpu_start: Starting scheduler on APP CPU. [18:33:14][I][logger:258]: Log initialized [18:33:14][C][ota:469]: There have been 0 suspected unsuccessful boot attempts. [18:33:14][D][esp32.preferences:113]: Saving 1 preferences to flash... [18:33:14][D][esp32.preferences:142]: Saving 1 preferences to flash: 0 cached, 1 written, 0 failed [18:33:14][I][ble_hid_client.cpp:568]: Trying to connect to ble server address: 62:50:7b:9a:0b:dc [18:33:14][W][app:022]: Component custom_component already registered! (0x3ffbca5c) [18:33:14][I][app:029]: Running through setup()... [18:33:14][C][wifi:037]: Setting up WiFi... [18:33:14][C][wifi:038]:   Local MAC: 78:21:84:92:FC:0C [18:33:14][D][esp-idf:000]: I (258) wifi: [18:33:14][D][esp-idf:000]: wifi driver task: 3ffd4ec8, prio:23, stack:6656, core=0 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1251) system_api: Base MAC address is not set [18:33:14] [18:33:14][D][esp-idf:000]: I (1252) system_api: read default base MAC address from EFUSE [18:33:14] [18:33:14][D][esp-idf:000]: I (1264) wifi: [18:33:14][D][esp-idf:000]: wifi firmware version: eeaa27d [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1272) wifi: [18:33:14][D][esp-idf:000]: wifi certification version: v7.0 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1292) wifi: [18:33:14][D][esp-idf:000]: config NVS flash: enabled [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1302) wifi: [18:33:14][D][esp-idf:000]: config nano formating: disabled [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1313) wifi: [18:33:14][D][esp-idf:000]: Init data frame dynamic rx buffer num: 32 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1323) wifi: [18:33:14][D][esp-idf:000]: Init management frame dynamic rx buffer num: 32 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1343) wifi: [18:33:14][D][esp-idf:000]: Init management short buffer num: 32 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1353) wifi: [18:33:14][D][esp-idf:000]: Init dynamic tx buffer num: 32 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1364) wifi: [18:33:14][D][esp-idf:000]: Init static rx buffer size: 1600 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1374) wifi: [18:33:14][D][esp-idf:000]: Init static rx buffer num: 10 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1394) wifi: [18:33:14][D][esp-idf:000]: Init dynamic rx buffer num: 32 [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1405) wifi_init: rx ba win: 6 [18:33:14] [18:33:14][D][esp-idf:000]: I (1406) wifi_init: tcpip mbox: 32 [18:33:14] [18:33:14][D][esp-idf:000]: I (1406) wifi_init: udp mbox: 6 [18:33:14] [18:33:14][D][esp-idf:000]: I (1415) wifi_init: tcp mbox: 6 [18:33:14] [18:33:14][D][esp-idf:000]: I (1425) wifi_init: tcp tx win: 5744 [18:33:14] [18:33:14][D][esp-idf:000]: I (1425) wifi_init: tcp rx win: 5744 [18:33:14] [18:33:14][D][esp-idf:000]: I (1435) wifi_init: tcp mss: 1440 [18:33:14] [18:33:14][D][esp-idf:000]: I (1446) wifi_init: WiFi IRAM OP enabled [18:33:14] [18:33:14][D][esp-idf:000]: I (1446) wifi_init: WiFi RX IRAM OP enabled [18:33:14] [18:33:14][D][esp-idf:000]: I (1457) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07 [18:33:14] [18:33:14][D][esp-idf:000]: I (1571) wifi: [18:33:14][D][esp-idf:000]: mode : sta (78:21:84:92:fc:0c) [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1572) wifi: [18:33:14][D][esp-idf:000]: enable tsf [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][esp-idf:000]: I (1575) wifi: [18:33:14][D][esp-idf:000]: Set ps type: 1 [18:33:14] [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:14][D][wifi:386]: Starting scan... [18:33:14][D][esp-idf:000]: I (1588) wifi: [18:33:14][D][esp-idf:000]: Set ps type: 1 [18:33:14] [18:33:14][D][esp-idf:000]:  [18:33:14] [18:33:16][D][wifi:401]: Found networks: [18:33:16][I][wifi:445]: - 'DragRacer-5G' (88:DC:96:3E:8F:62) ▂▄▆█ [18:33:16][D][wifi:446]:     Channel: 5 [18:33:16][D][wifi:447]:     RSSI: -24 dB [18:33:16][D][wifi:450]: - 'wzap26758302' (7C:78:B2:AB:21:D3) ▂▄▆█ [18:33:16][D][wifi:450]: - 'HP-Print-45-Officejet Pro 8600' (88:51:FB:21:B5:45) ▂▄▆█ [18:33:16][I][wifi:257]: WiFi Connecting to 'DragRacer-5G'... [18:33:16][D][esp-idf:000]: I (4001) wifi: [18:33:16][D][esp-idf:000]: new:<5,0>, old:<1,0>, ap:<255,255>, sta:<5,0>, prof:1 [18:33:16][D][esp-idf:000]:  [18:33:16] [18:33:17][D][esp-idf:000]: I (4810) wifi: [18:33:17][D][esp-idf:000]: state: init -> auth (b0) [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4816) wifi: [18:33:17][D][esp-idf:000]: state: auth -> assoc (0) [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4822) wifi: [18:33:17][D][esp-idf:000]: state: assoc -> run (10) [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4839) wifi: [18:33:17][D][esp-idf:000]: connected with DragRacer-5G, aid = 9, channel 5, BW20, bssid = 88:dc:96:3e:8f:62 [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4841) wifi: [18:33:17][D][esp-idf:000]: security: WPA2-PSK, phy: bgn, rssi: -23 [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4852) wifi: [18:33:17][D][esp-idf:000]: pm start, type: 1 [18:33:17] [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: W (4871) wifi: [18:33:17][D][esp-idf:000]: <ba-add>idx:0 (ifx:0, 88:dc:96:3e:8f:62), tid:0, ssn:0, winSize:64 [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:17][D][esp-idf:000]: I (4941) wifi: [18:33:17][D][esp-idf:000]: AP's beacon interval = 102400 us, DTIM period = 2 [18:33:17][D][esp-idf:000]:  [18:33:17] [18:33:19][D][esp-idf:000]: I (6734) esp_netif_handlers: sta ip: <IP_ADDRESS>, mask: <IP_ADDRESS>, gw: <IP_ADDRESS> [18:33:19] [18:33:19][C][wifi:374]:   Subnet: <IP_ADDRESS> [18:33:19][C][wifi:375]:   Gateway: <IP_ADDRESS> [18:33:19][C][wifi:376]:   DNS1: <IP_ADDRESS> [18:33:19][C][wifi:377]:   DNS2: <IP_ADDRESS> [18:33:19][C][ota:093]: Over-The-Air Updates: [18:33:19][C][ota:094]:   Address: so-ring.local:3232 [18:33:19][C][ota:097]:   Using Password. [18:33:19][C][api:025]: Setting up Home Assistant API server... [18:33:19][D][ble_hid_client.cpp:586]: Start of setup [18:33:19][D][esp-idf:000]: I (6768) BTDM_INIT: BT controller compile version [bf9c077] [18:33:19] [18:33:19][D][ble_hid_client.cpp:665]: Finished setup [18:33:19][D][ble_hid_client.cpp:537]: EVT 0, gattc if 3 [18:33:19][D][ble_hid_client.cpp:141]: REG_EVT [18:33:19][D][sensor:127]: 'WiFi Signal': Sending state -24.00000 dBm with 0 decimals of accuracy [18:33:19][D][ble_hid_client.cpp:383]: Setting scan parameters complete [18:33:19][I][app:102]: ESPHome version 2022.12.3 compiled on Jan 15 2023, 18:20:10 [18:33:19][C][wifi:504]: WiFi: [18:33:19][C][wifi:362]:   Local MAC: 78:21:84:92:FC:0C [18:33:20][C][wifi:363]:   SSID: 'DragRacer-5G' [18:33:20][C][wifi:364]:   IP Address: <IP_ADDRESS> [18:33:20][C][wifi:366]:   BSSID: 88:DC:96:3E:8F:62 [18:33:20][C][wifi:367]:   Hostname: 'so-ring' [18:33:20][C][wifi:369]:   Signal strength: -24 dB ▂▄▆█ [18:33:20][C][wifi:373]:   Channel: 5 [18:33:20][C][wifi:374]:   Subnet: <IP_ADDRESS> [18:33:20][C][wifi:375]:   Gateway: <IP_ADDRESS> [18:33:20][C][wifi:376]:   DNS1: <IP_ADDRESS> [18:33:20][C][wifi:377]:   DNS2: <IP_ADDRESS> [18:33:20][C][logger:293]: Logger: [18:33:20][C][logger:294]:   Level: DEBUG [18:33:20][C][logger:295]:   Log Baud Rate: 115200 [18:33:20][C][logger:296]:   Hardware UART: UART0 [18:33:20][I][ble_hid_client.cpp:671]: Start scanning for HID-BLE-Devices for 3600 seconds [18:33:20][D][ble_hid_client.cpp:393]: Scan start success [18:33:20][C][restart:076]: Restart Switch 'Restart' [18:33:20][C][restart:078]:   Icon: 'mdi:restart' [18:33:20][C][restart:099]:   Restore Mode: restore defaults to OFF [18:33:20][C][mdns:103]: mDNS: [18:33:20][C][mdns:104]:   Hostname: so-ring [18:33:20][C][ota:093]: Over-The-Air Updates: [18:33:20][C][ota:094]:   Address: so-ring.local:3232 [18:33:20][C][ota:097]:   Using Password. [18:33:20][C][api:138]: API Server: [18:33:20][C][api:139]:   Address: so-ring.local:6053 [18:33:20][C][api:143]:   Using noise encryption: NO [18:33:20][C][wifi_signal.sensor:009]: WiFi Signal 'WiFi Signal' [18:33:20][C][wifi_signal.sensor:009]:   Device Class: 'signal_strength' [18:33:20][C][wifi_signal.sensor:009]:   State Class: 'measurement' [18:33:20][C][wifi_signal.sensor:009]:   Unit of Measurement: 'dBm' [18:33:20][C][wifi_signal.sensor:009]:   Accuracy Decimals: 0 [18:33:20][D][ble_hid_client.cpp:458]: Found device: [18:33:20][D][ble_hid_client.cpp:464]: 	address: 62:50:7b:9a:0b:dc [18:33:20][D][ble_hid_client.cpp:466]: 	name: So+ [18:33:20][D][api:102]: Accepted ::FFFF:<IP_ADDRESS> [18:33:20][D][api.connection:918]: Home Assistant 2022.12.8 (::FFFF:<IP_ADDRESS>): Connected successfully [18:33:55][D][sensor:127]: 'WiFi Signal': Sending state -34.00000 dBm with 0 decimals of accuracy
Here is the log from a ring i have it finds it no problem but seems to never attempt to connect even though its in pairing mode


Bug (Java 21) - sha256sum.txt bug for automatic resolver

Describe the bug
Thank you for adding support for Java 21 ✨
 wget --progress=dot:giga -O sha256sum.txt  https://aka.ms/download-jdk/microsoft-jdk-21-linux-aarch64.tar.gz.sha256sum.txt

The contents of the  sha256sum.txt file form above ^ command does not match with the contents returned with following command 👇
 wget --progress=dot:giga -O sha256sum.txt  https://aka.ms/download-jdk/microsoft-jdk-21.0.0-linux-aarch64.tar.gz.sha256sum.txt

To me, it seems like an issue with the automatic resolver.
MSFT Build of OpenJDK Version:
Java 21
Screenshots, Logs etc

Yes, this is a shortcoming of our release thus far! You are a fast adopter for  sure, thanks for being a keen user of the Microsoft Build of OpenJDK, and for your feedback! 😄
While we have indeed published the binaries in our download page, we haven't published our 'vanity' links for JDK21.
We will have this corrected by EOD (in PST timezone) tomorrow, including links for the .sha256sum.txt files that correspond to each binary.
Future releases will have this step incorporated into the initial publish step.

The Aka links have been updated. Please go ahead and try again, @samruddhikhandale.

That worked, thank you! 🎉


Capstone Full Stack - Set Up Kanban

In this capstone 🚀:

This capstone will be developed by 5 members: @muneeb3123, @LarryIVC, @abiola-arobieke, @Ashnaali3255,  and @sameedmohsin28

We met in a Zoom call  💻 to decide the workflow and approach for this project 📦 , and we all agreed to add each task that will be assigned after discussing it between us 🧑🤝🧑

👉 On the go, we will actively participate and collaborate to progressively add each new task to the Kanban project 👈 .

The link of the kanban project is in the next link: 👇
https://github.com/users/muneeb3123/projects/4
We are open to your feedback.

Hi @LarryIVC,
Wow, you did it 🎉

Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
To highlight

All cards have descriptions, assignees, and pointings.  The pointings are fair according to the requirement. ✔

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Feature/applics 813 search split term

Changed the representations search to an AND when splitting search terms instead of OR and added a stop-word filter list to ignore common terms .
https://pins-ds.atlassian.net/browse/APPLICS-813
Updates to BO and NI repositories.
Useful information to review or test
Example link - https://applications-service-dev.planninginspectorate.gov.uk/projects/TR010027/representations?searchTerm=Winthorpe+Primary+School&itemsPerPage=25.  Previously this would return a lot of unrelated results as it would be searching on "Winthorpe" OR "primary" OR "School".  Instead we target the inclusion of all of these terms.
Unit tests have been updated in line with the code updates.
NB.  The BO search still uses OR functionality for it's first name and last name fields due to the restrictive nature of having the data separated into two fields.
Type of change 🧩

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update
[ ] Other (please explain in the description section above)

Checklist before requesting a review

[X] I have performed a self-review of my own code
[X] I have double checked this work does not include any hardcoded secrets or passwords
[X] I have made corresponding changes to the documentation
[X] I have provided details on how I have tested my code
[X] I have referenced the ticket number above
[X] I have provided a description of the ticket
[X] I have included unit tests to cover any testable code changes

:tada: This PR is included in version 1.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.38.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Automate Quants creation with a Hugging Face space.

Hey hey @CharlesMod,
I hope you are doing well. I'm VB I lead advocacy efforts for open source at Hugging Face. I love your project. I think it is a brilliant way to decentralise quant creation (especially GGUF).
A great way to increase awareness for your scripts could be by creating a Hugging Face space similar to the safetensors convert bot.
I'd be happy to help you put one together!
Thanks again for your contribution and open-source efforts.
Cheers!
VB

Hi @Vaibhavs10 ! I am doing well thanks, I hope the same for you.
Likewise, many thanks! Please feel free to help in any way you'd like, I'm game. Just let me know how I can help in turn.
Regards,
Charles


host arg for app object (instead of hardcode <IP_ADDRESS> interface)

Fixes broken ipv6 support (hardcoded <IP_ADDRESS> listen interface in app.py)
Changes proposed in this pull request:

Added host arg for App object, now we can redefine host to listen on
By default listen host='::' interface instead host='<IP_ADDRESS>'. App would think its in ipv6-only world, but you could access it thru ipv4 cause OS would map your ipv4 addr to ipv6 (see http://stackoverflow.com/a/21689979)

didnt see #298 pull, sorry
But: what about listening '::' by default?

https://github.com/zalando/connexion/pull/298#issuecomment-250358709
I believe this would break support for windows.


Manu bar still blue under Windows 10

Here's a screenshot:

So, from what I can tell, RStudio's menu bar is outside the scope of what's customizable through injecting css into index.htm. The menu bar and editors (the four panes) are built using different frameworks; Qt is used for the menu bar, while Ace is used for the editors.
However, someone else has made a package that solves this problem! Check out rscodeio and let me know if this is what you expected the menu bar to look like.

This works as I was hoping, thanks a lot!

No problem! Since the menu bar isn't within the scope of this project, I'm going to close this issue. I'm open to reopening the issue if anyone feels that styling the menu should be included here, or if anyone wants to open a PR.


Very large memory being used for a 50mb excel file

When I make the call to ExcelReaderFactory.CreateReader(Stream stream),  which stream contains the contents to a 50mb excel file - a total of 800 mb gets eaten up by that call (halting execution while doing so).
Is this using DOM? I thought only a single element/line was ever loaded at a time in the memory.

Hi,
ExcelDataReader is not using DOM. And indeed, only one row is loaded in the memory at once (with some exceptions).
You're not measuring memory usage after calling AsDataSet()? 16x would not be surprising in that case. Please show your code, how you measure memory usage, and preferably the file that's causing problems. If you can't share that specific file, can you create a new one in which the problem is reproducible for us too? At the very least let us know if its XLS, XLSX or XLSB

Hi,
ExcelDataReader is not using DOM. And indeed, only one row is loaded in the memory at once (with some exceptions).
You're not measuring memory usage after calling AsDataSet()? 16x would not be surprising in that case. Please show your code, how you measure memory usage, and preferably the file that's causing problems. If you can't share that specific file, can you create a new one in which the problem is reproducible for us too? At the very least let us know if its XLS, XLSX or XLSB

The file is .xlsx.
I tried something out, and it appears there's more to this.
When I tested the code in a .NET 5 Console application, by simply invoking the following for the FileStream
var stream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using IExcelDataReader reader = ExcelReaderFactory.CreateReader(stream);

I didn't see any of that highly insane memory usage.
But when I did it in a ASP.NET 5 project, which file was passed through ModelBinding via a IFormFile, and whose stream I received via a call to IFormFile.OpenReadStream() and calling the same constructor as above:
using IExcelDataReader reader = ExcelReaderFactory.CreateReader(stream);

The memory gobbles up to a 800 megabytes, if not even a gigabyte. I don't think it has something to do with my specific file, but something to do with how you handle a Microsoft.AspNetCore.Http.ReferenceReadStream, returend from IFormFile.ReadOpenStream().
If you could please test this out on a ASP.NET 5 project with a large enough file (20-50 mb), and confirm that this is indeed causing the issues, if not - I'll try to create a github repository with a file inside you can use to reproduce the issue, when I get the time to do so (Preferably at the end of the week).

It seems that you don't even need the model binding of an IFormFile. Simply executing
var stream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using IExcelDataReader reader = ExcelReaderFactory.CreateReader(stream);

from any action in the ASP.NET Core project was enough to see those memory spikes.

You could profile your app to pinpoint exactly which method uses up this much memory and share the data here.

I'm not sure how exactly to pinpoint the method, but taking a snapshot of my memory, it shows a whooping 64, 435, 728 bytes for object type ExcelDataReader.Core.OpenXmlFormat.XlsxSST

Can you also (rename the xlsx to zip and) unzip the XLSX and let us know the size of xl/sharedStrings.xml from inside the zip?
XlsxSST is the "Shared String Table", and is essentially a List which contains all the strings in the workbook. When this object takes ~64mb, it means there are either 64mb/16 = 4 million strings in it, or the runtime has allocated 64mb in total for all strings in the spreadsheet (I'm not sure if/how the .NET runtime optimizes this).
In the former case, if the spreadsheet contains 4 million strings taking up most of the 50mb of compressed data, uncompressed at ~5:1 ratio, and double that for the internal UTF-16 string representation, gives approx 500mb of allocations. Plus overhead per string object and for the SST we're not that far from seeing expected behavior.
Can you tell a bit more about the characteristics of the workbook you are testing? How many worksheets are there, how many rows/cols, what is the distribution of number cells vs text cells? What is the average length of the strings?

Can you also (rename the xlsx to zip and) unzip the XLSX and let us know the size of xl/sharedStrings.xml from inside the zip?
XlsxSST is the "Shared String Table", and is essentially a List which contains all the strings in the workbook. When this object takes ~64mb, it means there are either 64mb/16 = 4 million strings in it, or the runtime has allocated 64mb in total for all strings in the spreadsheet (I'm not sure if/how the .NET runtime optimizes this).
In the former case, if the spreadsheet contains 4 million strings taking up most of the 50mb of compressed data, uncompressed at ~5:1 ratio, and double that for the internal UTF-16 string representation, gives approx 500mb of allocations. Plus overhead per string object and for the SST we're not that far from seeing expected behavior.
Can you tell a bit more about the characteristics of the workbook you are testing? How many worksheets are there, how many rows/cols, what is the distribution of number cells vs text cells? What is the average length of the strings?

@andersnm , please do not forget that this "expected" behavior of seeing 500+ mb in memory is only happening on the ASP.NET Core framework, not in a Console application, which is my main concern. I'm using the same file, with the same NuGet package. Everything is the same, with the exception that one project is a simple Console App, while the other - an ASP.NET Core project.
In the very first lines of code - actually THE very first lines of code in both projects are the same - opening the file stream, and then opening an excel reader via the current library.
Does this not happen for you, when you test the very same file in these different environments? (Both uses .NET 5)
sharedStrings.xml is 29.3 mb large. The file has 244 870 rows, a single worksheet/spreadsheet. 39 Columns, sorry but the rest I don't have the time right now to diverse - there are string columns, numbers, and dates.

@SpiritBob Thanks for the additional information! This should be enough to attempt a repro at least. Alas no .NET 5 here, and no free diskspace to install, so it's likely going to be a while to repro myself.
This all sounds very strange. Not sure exactly how the memory usage was measured, so still keeping the door open for a "normal" explanation. Hopefully, this is not some peculiar streams implementation detail bug deep inside the .NET runtime that only affects ASP.NET Core and takes a week to investigate.

@SpiritBob Thanks for the additional information! This should be enough to attempt a repro at least. Alas no .NET 5 here, and no free diskspace to install, so it's likely going to be a while to repro myself.
This all sounds very strange. Not sure exactly how the memory usage was measured, so still keeping the door open for a "normal" explanation. Hopefully, this is not some peculiar streams implementation detail bug deep inside the .NET runtime that only affects ASP.NET Core and takes a week to investigate.

Come weekend I'll try to help out by creating actual github repos reproducing the issue for you (hopefully).

For the love of me - I could not create a sample file that mimics the behavior of the original one I've got. Sadly, I'm unable to post it as the data there is quite sensitive.
https://github.com/SpiritBob/ExcelReading/blob/main/WebProject/Program.cs
https://github.com/SpiritBob/ExcelReading/blob/main/ConsoleApp/Program.cs
Those are the only files worthy to note in the repo. When tested with the file I've got - the memory my debugger reports as being consumed on the Web project is 500-700 mb, while on the console project - it reaches only 100 mb (as reported by my debugger). Upon taking a memory snapshot after both executions, the Heap size is the same, so that's kind of confusing (in both cases around 64 mb). I'm not sure what exactly I need to look for, in order to understand from where those extra 400-500 mb are coming from.

Throwing out some random thoughts here. Just to be sure, can you confirm if the problem only happens with this particular XLSX and not with other similarly sized XLSX?
And only when a debugger is attached or does it happen without debugger too? Task Manager -> Details -> (rclick columns) -> Select Columns -> Peak Working Set gives an idea of memory usage during runtime.
One idea is the debugger probably loads far more debug symbol information into memory for an ASP.NET application than for a console application.
Looking at your repo, the ConsoleApp uses netcoreapp3.1:
https://github.com/SpiritBob/ExcelReading/blob/5088e0b03641a921fa218b20ddc7145710ca35f2/ConsoleApp/ConsoleApp.csproj#L5
While WebProject uses net5.0:
https://github.com/SpiritBob/ExcelReading/blob/5088e0b03641a921fa218b20ddc7145710ca35f2/WebProject/WebProject.csproj#L4
Does it reproduce in the ConsoleApp if using net5.0 here as well?
Can we see the list of files and the file sizes from the unzipped XLSX, e.g the output from dir /s in a cmd from the unzipped XLSX directory?

Throwing out some random thoughts here. Just to be sure, can you confirm if the problem only happens with this particular XLSX > and not with other similarly sized XLSX?

I've downloaded some public xlsx sample files, one ranging about 50 mb in size, and none of them produced the thing my original file does. To note - all files I opened, were instantly loaded up when execution passed the ExcelReaderFactory.CreateReader method call, while my file takes about ~7-8 seconds for that command to finish, which in the meantime you can see on the ASP.NET Core project how the memory slowly skyrockets to reported values, whereas on the console, it reaches a certain threshold and stays there for the remainder of the long operation.

And only when a debugger is attached or does it happen without debugger too? Task Manager -> Details -> (rclick columns) -> Select Columns -> Peak Working Set gives an idea of memory usage during runtime.
I ran both projects via their release's executable, and tracked that on the Task Manager.

Memory for ASP NET Core after reader is about 240mb,.
Memory for Console App after reader is about 120mb.

Perhaps the debugger is then reporting incorrectly? (500 mb, but in reality the process uses much less than that?)
For the above tests, I've changed the Console App project to be NET 5.0, so nothing has changed.

Can we see the list of files and the file sizes from the unzipped XLSX, e.g the output from dir /s in a cmd from the unzipped XLSX directory?

The unzipped folder containing .xlsx's files is called stumpy_g. The full path to that folder has been obfuscated.
ObfuscatedPath\\stumpy_g>dir /s
 Volume in drive C has no label.
  I suppose this is where the 7-8 seconds go, and accounts for ~600mb of allocations (~300mb XML, 16 bit per character).
F.ex another XLSX with mostly images would load faster and allocate less, since there is less of actual data in it to parse.
Could it be the website project is using a different garbage collector than the console project?
https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc

The two interesting files:
01/01/1980  12:00 AM        30,798,086 sharedStrings.xml
...
01/01/1980  12:00 AM       278,566,993 sheet1.xml

These files are pretty huge and ExcelDataReader parses both in CreateReader(). I suppose this is where the 7-8 seconds go, and accounts for ~600mb of allocations (~300mb XML, 16 bit per character).
F.ex another XLSX with mostly images would load faster and allocate less, since there is less of actual data in it to parse.
Could it be the website project is using a different garbage collector than the console project?
https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc

I can't beliee it! This has been the cause all along! It seems a Console Application uses a Workstation GC, while the ASP.NET Core project uses Server GC.
When I disabled Server GC in the Console App and in the ASP.NET Core App via
<ServerGarbageCollection>false</ServerGarbageCollection>

Now both projects are capped at 100 mb memory! If I enable it - both are capped at around 500-700 mb memory!!!
This is a shoker. I'm curious to learn why the Server GC does not lift even a finger to take care of that garbage. I made a small test - queue 14 tasks on the ThreadPool for execution that open up the same file, and monitored how the memory behaves.
When it was Workstation GC - the memory capped at about 1009 mb.
When it was Server GC - the memory reached as high as 1.3GB-1.4GB.
Frankly, I thought the Server GC would beat the Workstation GC in that scenario, but it seemed the opposite.

The Server GC trades memory in favor of throughput. Then there's the Retain VM option that also affects how .NET frees memory back to the OS.
I have an idea how we can change ExcelDataReader to allocate less but will have to test if it has any real world impact. Could be that the objects we can do something about are so shortlived that it doesn't matter.

@andersnm Could this be temporarily be fixed by not storing all the unzipped data/ XlsxSST in-memory, but instead a temporary file?
Implementation of a temporary file stream can be as simple as:
public class TempFileStream : FileStream
{
    public TempFileStream()
        : base(Path.GetTempFileName(), FileMode.Create, FileAccess.ReadWrite, FileShare.Read, 4096, FileOptions.DeleteOnClose)
    {
    }

    public TempFileStream(FileAccess access)
        : base(Path.GetTempFileName(), FileMode.Create, access, FileShare.Read, 4096, FileOptions.DeleteOnClose)
    {
    }

    public TempFileStream(FileAccess access, FileShare share)
        : base(Path.GetTempFileName(), FileMode.Create, access, share, 4096, FileOptions.DeleteOnClose)
    {
    }

    public TempFileStream(FileAccess access, FileShare share, int bufferSize)
        : base(Path.GetTempFileName(), FileMode.Create, access, share, bufferSize, FileOptions.DeleteOnClose)
    {
    }
}

@SpiritBob I suppose using a temp file will not reduce the total allocations, but rather double the number due to the added roundtrip. Alas I don't have any other good suggestions here.

@andersnm Just to confirm: This issue with memory is only prevalent for xlsx files that have a shared string table, correct? The idea here is that I'm able to limit the potential memory that is being consumed by introducing a queue for just those files.

@SpiritBob The key factor is likely the the combined size of the XMLs inside file XLSX, not necessarily only the string table.
I am assuming the memory pressure occurs in the XML parser. Every tag name, text element, attribute name, attribute value inside the XML become individually allocated strings (although quickly freed; it's SAX-based). Internally .NET uses 16 bits per character, so there is a 2x memory usage from that, and the XML is parsed twice, so there are 2x memory allocations from that.
Note that this excludes images - there can be large images in the spreadsheet, but these are ignored completely.

@SpiritBob The key factor is likely the the combined size of the XMLs inside file XLSX, not necessarily only the string table.
I am assuming the memory pressure occurs in the XML parser. Every tag name, text element, attribute name, attribute value inside the XML become individually allocated strings (although quickly freed; it's SAX-based). Internally .NET uses 16 bits per character, so there is a 2x memory usage from that, and the XML is parsed twice, so there are 2x memory allocations from that.
Note that this excludes images - there can be large images in the spreadsheet, but these are ignored completely.

From my internal testing it seems that the larger the shared string table is, the larger memory is being used. For example: a 20 mb file with a 100 mb shared string table takes up as much as 310 mb of memory, while a 50 mb file with only 30 mb shared string table ends up taking 120 mb of memory.
As for .xlsb files, which are ZIP files as well, there is no memory pressure there at all, but parsing the file takes a whooping 60-120 seconds, compared to the same file in xlsx format.

@SpiritBob I trust your testing and agree it sounds indeed like the string table is the main factor! To add, there was a report on bad XLSB perf on some .NET runtimes in #493 - alas not solved yet.

@andersnm  @appel1 ExcelDataReader loads all the shared string into RAM. for larger file, should it be avoided?
I am thinking of loading shared string to seek-able file instead of RAM. When needed we can seek to a specific record and return the content.
I have implemented it at https://github.com/ArjunVachhani/ExcelDataReader/commit/b571bda0210cd564089c3f995f58908118ba00fb
I would like you hear your thoughts.

I'll take a look but I'm not sure it is worth it because of the added complexity. Lots of pitfalls with read and write errors, performance cost of reading, string allocation increase, issues with antivirus, where to write it, configuration, cleanup and so on.
If memory usage is a real issue then maybe we could store it more efficiently in memory somehow.

@appel1 In my specific use-case, we'd like to avoid pressuring the RAM at all costs. CPU can take its time to compute, but if RAM gets overfilled, we'll be facing a serious issue of an application crash.
I've seen files that are just at about 70 mb large, and yet consume up to a gigabyte of ram. Now imagine a scenario where you have 20 concurrent such files being processed.
Is the above suggestion perfect for this case? Yes
Are there workarounds? Yes, but much more complicated than simply having it supported outside-the-box. You'd have to create your very own Semaphore (if you'd like to preserve order of the processed files) and assume how much memory at one point the application can spare for these files and go from there.

Hi - not sure loading 20 concurrent 70mb spreadsheets is necessarily something this project should aim to support out of the box.
If there is strong demand to externalize the SST storage, the one idea is to abstract it away behind an interface, and let users of the library provide their own custom SST storage implementation in the reader configuration.  Internally it can remain a Dictionary<> by default.

@kikaragyozov I am also facing memory issue when upload huge files. In my case files are more than 200MB in size.
So I have decided to a build a library to read huge Xlsx file. library is mostly ready. If you try it today it should work fine. please fell free to try and share your feedbacks and bugs.
In coming days I will push few more changes to iron out any issue and add documentation.
Link to repository https://github.com/ArjunVachhani/XlsxHelper

@kikaragyozov I am also facing memory issue when upload huge files. In my case files are more than 200MB in size.
So I have decided to a build a library to read huge Xlsx file. library is mostly ready. If you try it today it should work fine. please fell free to try and share your feedbacks and bugs. In coming days I will push few more changes to iron out any issue and add documentation.
Link to repository https://github.com/ArjunVachhani/XlsxHelper

I made a really quick test with a random .XLS file only to have it throw an exception with End of Central Directory Not Found.
@andersnm is there motivation in addressing this hopefully? The final suggestion in regards to providing more control to consumers on how the STS should be read/stored via an interface is a good start. This would avoid the turmoil of having to deal with all the added complexities @appel1 described in reading the thing lazily.
Internally, we've made a few specialized classes to help allocate a specific amount of memory in the process to be used exclusively by excel files with shared string tables, and share it fairly, along with a safe upper-bound of how large the string table is allowed to be before we outright reject such file.
ExcelDataReader is by-far the best library out there currently in terms of performance. The only factor that still in my eyes deserves an optimization to truly bring this to the next level is hopefully these costly string allocations, taking up so much memory.
The official Open XML SDK by Microsoft has a serious issue when it comes to memory as well, as they I believe open the archive in Update mode, which is known to load the entire contents of related ZipArchiveEntries in memory.


Rework structure to be more pythonic

Add setup tools, organize the code and tools better, create doc folder to add future changelog
Start using git tags instead of manual versioning

For now will not try to improve this further.
Reason: If paco is installed from pip, the tests will be in a very obscure directory, and I think it will be worse to actually interact with it.
I've tried to also just create a small cli that installs and calls francinette from the egg, but I don't think there is a way of decoupling the caller from the then francinette execution.
For now it is abandoned.


Azure tts is not working

AttributeError: 'NoneType' object has no attribute 'reason'
The error occurs when I use with self.voiceover("..."), the example scene also has this issue.

Resolved with #6


flaky eureka test

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_pilot/1069/pilot-presubmit/441/

Looks like it is missing a sort in the Services() implementation.


New lint: Inherent method #[must_use] added

After some testing it turned out it is not possible to have a single lint for both Struct and Trait methods. Here is the reason:

assume we want to support both scenarios, Struct inherent method becomes #[must_use] and Trait method becomes #[must_use] at the same time. The former method has public visibility, the latter has default, while both Struct and Trait must have public visibility.
in a single query, we have to give exact parameters to both baseline and current parts of the query. So we opt for method visibility to be either public or default, because otherwise we will miss some Struct or Trait methods.
now imagine that we have one pub Struct Foo and one pub Trait Bar, that both have a method that becomes #[must_use]. If we run the query that looks for public or default methods in both baseline and current crates, the query will find matches for all 4 possible pairings, despite there being only 2 of them sensible to the user. To avoid this, the query would have to "on Struct look for a matching Struct, then for the matching method, on Trait look for a matching Trait, then for the matching method", which even if was possible would be begging to get split into 2 separate queries (at least I see it this way).
because of this we have to precise our query to a single visibility value for either baseline or current version.

And since we decided in #268 that there is no need for a lint that detects addition of #[must_use] for a method that is also moved to Trait, we eliminated the only cross-type case from the 3 possibilities I introduced in that PR and are left with:

Struct method -> Struct #[must_use] method, with the lint inherent_method_must_use_added implemented in this PR,
Trait method -> Trait #[must_use] method, with the lint trait_method_must_use_added, on which I am still working in [insert PR once opened].

I think this implementation has an interesting bug.
The PR description describes this lint as Struct method -> Struct #[must_use] method, but that is actually too narrow. Enums and unions can also have inherent impl blocks whose methods may be marked with #[must_use]. The lint correctly anticipates this and uses the coercion ... on ImplOwner instead of ... on Struct. This means it will correctly flag enum methods (and, when implemented, union methods as well).
Except, the current and baseline branches of the query never ensure that the kind of item matches between the baseline and the current code. So I predict that the following code will cause a false-positive here:
baseline:
pub struct Foo {};

impl Foo {
    pub fn bar() {}
}

current:
pub enum Foo {
    A,
}

impl Foo {
    #[must_use]
    pub fn bar() {}
}

I predict that the current code will flag Foo::bar as having gained a #[must_use] attribute, when in fact that's a completely new method on a completely new type.
It would be great to add "true-positive" test cases for enum and union methods just like the existing ones for structs, even though the union ones won't be caught (opne an issue about this and link it in a comment, so we remember that it's okay to for that test output to change when we implement unions in the adapter).
It would also be good to have "true-negative" test cases where structs/enums/unions get each other's names like this.
With respect to fixing the lint query, there are two options:

Each vertex has a special implicit field __typename: String! that contains the exact type of the vertex: on ImplOwner it would be Struct or Enum or Union (when implemented). You could tag and filter on that value across the baseline / current branches of the query to make sure the item type matches up.
Alternatively, if you think there's an advantage to doing so, you could split the lint into separate lints for Struct / Enum / Union types and replace ... on ImplOwner with ... on Struct etc. This doesn't seem to offer any obvious advantages over the alternative, and is more complex, but I wanted to bring it up for your consideration in case I missed something.

That's a very interesting bug, I wasn't aware of the inherent_impl existing for types other than Struct. At first glance I also think it will be best to just broaden the query to handle both Struct and Enum, Union cases, but I will look deeper into this first before proceeding with changes.

Alternatively, if you think there's an advantage to doing so, you could split the lint into separate lints for Struct / Enum / Union types and replace ... on ImplOwner with ... on Struct etc. This doesn't seem to offer any obvious advantages over the alternative, and is more complex, but I wanted to bring it up for your consideration in case I missed something.

The only "advantage" I can point out for now is that this would result in multiple smaller test crates, one per lint, instead of one big test crate handling all of the cases for one, common lint. But if the test crate turns out too large, we will be able to easily split it into smaller test cases and leave the common lint unchanged (I will keep it in mind to write the tests as easily splittable because of this).

I will keep it in mind to write the tests as easily splittable because of this.

The best option was to use the hybrid approach mentioned here https://github.com/obi1kenobi/cargo-semver-checks/pull/282#issuecomment-1382352356, so I already split the tests into relevant-named files. I also added a "test file pair" against the changed type false-positive scenario which was shown above.
This is currently blocked on #149 & #292.
If the lint was split into more specialized ones for Enum, Struct& Union, this would not require implementing #149 and would allow for adding Enum & Struct lints quickly, while the Union lint would wait until #292 is complete (and could be mentioned in the issue itself to be added once Union is in the schema, among some other lints that it will enable).
@obi1kenobi considering the above, despite it requiring a bit more work with splitting the lint, I think it could in return make the workflow a lot smoother and faster (with #149 being rated as the hardest addition in #241, I am not sure if this is a good idea to block this PR on that issue). If you have nothing against it, I would proceed with this approach to thin out the #[must_use] issue as much as possible.

I don't think I see why this is blocked on #149. That issue is required for representing the type of fields, function arguments, generics, etc. I'm confused why those are involved here.
Are we not able to use the ImplOwner type's inherent_impl edge? The __typename special field on ImplOwner will be one of "Struct" / "Enum" / "Union" so it seems like we should be able to use the standard @tag + @filter combination across current and baseline to determine when the types are of the same kind (struct / enum / union).

It is indeed not blocked, my reading has just fooled me on this bit

Each vertex has a special implicit field __typename: String! that contains the exact type of the vertex: on ImplOwner it would be Struct or Enum or Union (when implemented).

I applied the "when implemented" part to the whole sentence, that is __typename and assumed it was a part of #149, while it of course was just about the Union...
With that problem out of the way, this should be working now! I also updated the issue for adding Union #292 to remind that once it is released, the output file will become invalid.


midiOff not received with midi keyboards with 62+ keys

We are having a strange issue. When we connect any midi keyboard with 62+ keys, note on is coming in properly. But note off is not being received. Everything works as expected in the very same build for smaller keyboards
The only indication i see is a update from AudioKit 4.5.6 to 4.7.2.
Is this a known issue?

I have just downgraded to 4.5.6 again (manually build with swift5) and the issue is gone.. was there a change since then?

I have tried 4.8 and the issue is also present there

This sounds like it must have to do with the change to the MIDI listener protocol. Do you have an implementation example that we could see? I'm just back at this from a few days of vacation but I can take a look soon.

This might be addressed by this change: https://github.com/AudioKit/AudioKit/commit/9dcb2f2177f9a575417484752bfb5b9bba6ddfa8

I have tested the development branch and I am getting note offs with my 88 key keyboard.  I had to fix the MIDI Utility examples to use the MIDI listener protocol correctly and I imagine you'll need to do something similar.

okay great. i will give it a try and let you know. Any any ETA for the changes above?

They are already in develop. Are you asking when it will be pushed to master or put into a CocoaPod?

Yes, i am wondering when its going to be published to cocoapods

Mh, i have build a framework from latest develop, updated the protocol handler and i am still not receiving the noteOff events.
when checking the file AudioKit.swift inside the module cache folder in DerivedData i see this

public protocol AKMIDIListener {

    /// Receive the MIDI note on event
    ///
    /// - Parameters:
    ///   - noteNumber: MIDI Note number of activated note
    ///   - velocity:   MIDI Velocity (0-127)
    ///   - channel:    MIDI Channel (1-16)
    ///
    func receivedMIDINoteOn(noteNumber: MIDINoteNumber, velocity: MIDIVelocity, channel: MIDIChannel)

    /// Receive the MIDI note off event
    ///
    /// - Parameters:
    ///   - noteNumber: MIDI Note number of released note
    ///   - velocity:   MIDI Velocity (0-127) usually speed of release, often 0.
    ///   - channel:    MIDI Channel (1-16)
    ///
    func receivedMIDINoteOff(noteNumber: MIDINoteNumber, velocity: MIDIVelocity, channel: MIDIChannel)

    /// Receive a generic controller value
    ///
    /// - Parameters:
    ///   - controller: MIDI Controller Number
    ///   - value:      Value of this controller
    ///   - channel:    MIDI Channel (1-16)
    ///
    func receivedMIDIController(_ controller: MIDIByte, value: MIDIByte, channel: MIDIChannel)

    /// Receive single note based aftertouch event
    ///
    /// - Parameters:
    ///   - noteNumber: Note number of touched note
    ///   - pressure:   Pressure applied to the note (0-127)
    ///   - channel:    MIDI Channel (1-16)
    ///
    func receivedMIDIAftertouch(noteNumber: MIDINoteNumber, pressure: MIDIByte, channel: MIDIChannel)

    /// Receive global aftertouch
    ///
    /// - Parameters:
    ///   - pressure: Pressure applied (0-127)
    ///   - channel:  MIDI Channel (1-16)
    ///
    func receivedMIDIAfterTouch(_ pressure: MIDIByte, channel: MIDIChannel)

    /// Receive pitch wheel value
    ///
    /// - Parameters:
    ///   - pitchWheelValue: MIDI Pitch Wheel Value (0-16383)
    ///   - channel:         MIDI Channel (1-16)
    ///
    func receivedMIDIPitchWheel(_ pitchWheelValue: MIDIWord, channel: MIDIChannel)

    /// Receive program change
    ///
    /// - Parameters:
    ///   - program:  MIDI Program Value (0-127)
    ///   - channel:  MIDI Channel (1-16)
    ///
    func receivedMIDIProgramChange(_ program: MIDIByte, channel: MIDIChannel)

    /// Receive a MIDI system command (such as clock, sysex, etc)
    ///
    /// - parameter data: Array of integers
    ///
    func receivedMIDISystemCommand(_ data: [MIDIByte], time: MIDITimeStamp)

    /// MIDI Setup has changed
    func receivedMIDISetupChange()

    /// MIDI Object Property has changed
    func receivedMIDIPropertyChange(propertyChangeInfo: MIDIObjectPropertyChangeNotification)

    /// Generic MIDI Notification
    func receivedMIDINotification(notification: MIDINotification)
}

The protocol seems not to be updated

i have tried 4.6.4, same issue here. noteOff is not incoming.. trying to nail down the version

well... nailed it down to be introduced in 4.6.0. the strange thing i just realized is, that we also have an AKAppleSampler attached. This device is receiving the noteOffs properly from what i hear. Just our client is not receiving them. Doublechecking the protocol

Mh, seeing this issue in 4.6.0 but not in 4.5.6. But in between there was not protocol change. so just tried the exact same version of our app, but the issue pops up with 4.6.0 but note offs are properly received with 4.5.6.
Any other ideas what the cause could be?

While I don't see it here in the log, I have an email in which an old version of the MIDI protocol listener was pasted in.  I feel like this still must be the root of the problem. Any way you can add me to the project if its a private repo so I can test for myself. Also willing to do a screen share on your computer if that is more appropriate / easier / quicker.

yep. i was suspecting this too. But using the same implementation of the protocol between version 4.5.6 and 4.6.0 leads to the same problem while the protocol was not changed in this versions right?

There were pretty extensive MIDI changes between those two versions. I'd really rather get it fixed for you on the most recent master or even develop branch (and then make a release).


[BEAM-12949] Fix DatastoreIO template compatibility

Switches hintNumWorkers to a ValueProvider in order to make this configurable from existing classic Dataflow templates.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @tvalentyn

Sorry, I found another incompatibility I need to fix. I will ping here again when the PR is ready.

PR is now ready for review.

R: @udim

@tvalentyn
Throttling on our Dataflow templates is currently broken, so I'd like to get this fix in before 2.34.0 is cut. Please take a look.

R: @pabloem

LGTM. I don't know if we have a postcommit test for this connector... I'll run Java postcommit in any case and merge after getting a passing run. FWIW, I think we should add an integration test if we don't have it.

Run Java PostCommit

Thanks! These are both good features to have

Thank you!
There are a few integration tests for reads & writes, admittedly they're pretty basic so they don't scaling behavior or things like that.


Prevent ghost-pings in docs get command

Solves #1624.
Will no longer delete the invoking message when the given symbol is invalid if the message contains user/role mentions. If it has mentions, allows deletion of error message through reactions.

the pypi command and possibly others in the future also delete invalid invocations, we should extract this to an util function under utils.messages instead of only implementing the functionality for this cog.
Also, like @Bluenix2 mentioned, please do not make unnecessary styling changes like the imports


Remove duplicated release notes for v3.5.0

The release notes appear twice, the commits list compares v3.5.0 to itself.

Thanks!


mobx@next error with latest next.

Maybe it's too early,
Marged locally latest next with feature/mobx@next:

TypeError: Cannot read property 'createHref' of undefined
at Link.render (C:***\node_modules\react-router-dom\Link.js:72:36)

Question:
Can someone please provide a short example of consuming DB data by mobx store for this project?
Edit:
Found this repo:
https://github.com/kennethtruong/react-webapp
Thanks.

@saniko You could try to upgrade your react-router-dom to stable version.

Hey, I have synced this with the next branch, so you should be good to go.
You can take a look at https://github.com/ueno-llc/starter-kit-universally if you want to see complete example of mobx store with server side rendering.

Thanks!


UWP sample is out of date and references RecognizeSpeechAsync()

The UWP portion of how to use the Speech SDK tells the reader to insert code in the RecognizeSpeechAsync() method, which isn't present in any of the provided UWP source code. It appears to exist in the C# sample code only.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e4b55e56-55a1-d635-6bea-271dc361d676
Version Independent ID: 8e9cb28f-c133-f310-3c7d-3a3b7e85f69d
Content: Quickstart: Recognize speech from a microphone - Speech service - Azure Cognitive Services
Content Source: articles/cognitive-services/Speech-Service/quickstarts/speech-to-text-from-microphone.md
Service: cognitive-services
Sub-service: speech-service
GitHub Login: @trevorbye
Microsoft Alias: dapine

@dreijer Thanks for the feedback. We are investigating the issue and will update you shortly.

@trevorbye Could you please review the following section Create a Speech configuration for Insert this code in the RecognizeSpeechAsync() method.
Here is the sample for UWP https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/samples/csharp/uwp/speechtotext-uwp/speechtotext-uwp
Could you please share any other samples available using the RecognizeSpeechAsync() method.

It looks like RecognizeOnceAsync() is the method that needs to be referenced in the documentation. @trevorbye is looking into this.

fixed the issue #please-close


0.3.0 seems not to be running

$ docker run --rm -it -v /data/scratch/beast_test:/scratch -v /data/bids/rewardBeastBIDS:/data:ro -v /data/out/beast_test/derivatives:/out poldracklab/fmriprep:0.3.0 /data /out participant --participant_label 387 -w /scratch
/usr/local/miniconda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/workflows/dmri/mrtrix/group_connectivity.py:16: UserWarning: cmp not installed
  warnings.warn('cmp not installed')
170323-19:10:33,419 interface WARNING:
	 AFNI is outdated, detected version Debian-16.2.07~dfsg.1-2~nd16.04+1 and AFNI_17.0.16 is available.
170323-19:10:34,291 interface WARNING:
	 AFNI is outdated, detected version Debian-16.2.07~dfsg.1-2~nd16.04+1 and AFNI_17.0.16 is available.
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/fsl/preprocess.py:1587: UserWarning: This has not been fully tested. Please report any failures.
  warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input convergence_threshold requires inputs: number_of_iterations
  warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input metric_weight requires inputs: metric
  warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input sigma_units requires inputs: smoothing_sigmas
  warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input concat_xfm requires inputs: in_file2
  warn(msg)
Exception occurred in traits notification handler.
Please check the log file for details.
Exception occurred in traits notification handler for object:
epi_mask = <undefined>
fmri_file = <undefined>
motion_confounds_file = <undefined>
movpar_file = <undefined>
source_file = <undefined>
t1_tpms = <undefined>
t1_transform_flags = [False, True]
, trait: trait_added, old value: <undefined>, new value: t1_transform
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
    handler( *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
    object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
170323-19:10:34,811 traits ERROR:
	 Exception occurred in traits notification handler for object:
epi_mask = <undefined>
fmri_file = <undefined>
motion_confounds_file = <undefined>
movpar_file = <undefined>
source_file = <undefined>
t1_tpms = <undefined>
t1_transform_flags = [False, True]
, trait: trait_added, old value: <undefined>, new value: t1_transform
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
    handler( *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
    object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
Exception occurred in traits notification handler for object:
epi_mask = <undefined>
fmri_file = <undefined>
motion_confounds_file = <undefined>
movpar_file = <undefined>
source_file = <undefined>
t1_tpms = <undefined>
t1_transform_flags = [False, True]
, trait: trait_added, old value: <undefined>, new value: reference_image
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
    handler( *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
    object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
170323-19:10:34,824 traits ERROR:
	 Exception occurred in traits notification handler for object:
epi_mask = <undefined>
fmri_file = <undefined>
motion_confounds_file = <undefined>
movpar_file = <undefined>
source_file = <undefined>
t1_tpms = <undefined>
t1_transform_flags = [False, True]
, trait: trait_added, old value: <undefined>, new value: reference_image
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
    handler( *args )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
    object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==99.99.99', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/run_workflow.py", line 88, in main
    create_workflow(opts)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/run_workflow.py", line 164, in create_workflow
    settings=settings, run_uuid=run_uuid)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 43, in base_workflow_enumerator
    settings=settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 71, in base_workflow_generator
    return basic_fmap_sbref_wf(subject_data, settings, name=subject_id)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 172, in basic_fmap_sbref_wf
    (t1w_pre, confounds_wf, [('outputnode.t1_tpms', 'inputnode.t1_tpms')]),
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 233, in connect
    infostr))
Exception: Some connections were not found
Module ConfoundDiscoverer has no input called inputnode.t1_transform

Module ConfoundDiscoverer has no input called inputnode.reference_image

--ignore fieldmaps
On Thu, Mar 23, 2017 at 12:14 PM, Chris Markiewicz<EMAIL_ADDRESS>
wrote:

$ docker run --rm -it -v /data/scratch/beast_test:/scratch -v /data/bids/rewardBeastBIDS:/data:ro -v /data/out/beast_test/derivatives:/out poldracklab/fmriprep:0.3.0 /data /out participant --participant_label 387 -w /scratch
/usr/local/miniconda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/workflows/dmri/mrtrix/group_connectivity.py:16: UserWarning: cmp not installed
warnings.warn('cmp not installed')
170323-19:10:33,419 interface WARNING:
AFNI is outdated, detected version Debian-16.2.07~dfsg.1-2~nd16.04+1 and AFNI_17.0.16 is available.
170323-19:10:34,291 interface WARNING:
AFNI is outdated, detected version Debian-16.2.07~dfsg.1-2~nd16.04+1 and AFNI_17.0.16 is available.
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/fsl/preprocess.py:1587: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input convergence_threshold requires inputs: number_of_iterations
warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input metric_weight requires inputs: metric
warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input sigma_units requires inputs: smoothing_sigmas
warn(msg)
/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base.py:431: UserWarning: Input concat_xfm requires inputs: in_file2
warn(msg)
Exception occurred in traits notification handler.
Please check the log file for details.
Exception occurred in traits notification handler for object:
epi_mask =
fmri_file =
motion_confounds_file =
movpar_file =
source_file =
t1_tpms =
t1_transform_flags = [False, True]
, trait: trait_added, old value: , new value: t1_transform
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
self.dispatch( handler, *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
handler( *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
170323-19:10:34,811 traits ERROR:
Exception occurred in traits notification handler for object:
epi_mask =
fmri_file =
motion_confounds_file =
movpar_file =
source_file =
t1_tpms =
t1_transform_flags = [False, True]
, trait: trait_added, old value: , new value: t1_transform
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
self.dispatch( handler, *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
handler( *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
Exception occurred in traits notification handler for object:
epi_mask =
fmri_file =
motion_confounds_file =
movpar_file =
source_file =
t1_tpms =
t1_transform_flags = [False, True]
, trait: trait_added, old value: , new value: reference_image
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
self.dispatch( handler, *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
handler( *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
170323-19:10:34,824 traits ERROR:
Exception occurred in traits notification handler for object:
epi_mask =
fmri_file =
motion_confounds_file =
movpar_file =
source_file =
t1_tpms =
t1_transform_flags = [False, True]
, trait: trait_added, old value: , new value: reference_image
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 519, in _dispatch_change_event
self.dispatch( handler, *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/traits/trait_notifiers.py", line 482, in dispatch
handler( *args )
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 784, in _set_input
object.traits()[name].node.set_input(name, newvalue)
AttributeError: 'NoneType' object has no attribute 'set_input'
Traceback (most recent call last):
File "/usr/local/miniconda/bin/fmriprep", line 11, in
load_entry_point('fmriprep==99.99.99', 'console_scripts', 'fmriprep')()
File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/run_workflow.py", line 88, in main
create_workflow(opts)
File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/run_workflow.py", line 164, in create_workflow
settings=settings, run_uuid=run_uuid)
File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 43, in base_workflow_enumerator
settings=settings)
File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 71, in base_workflow_generator
return basic_fmap_sbref_wf(subject_data, settings, name=subject_id)
File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 172, in basic_fmap_sbref_wf
(t1w_pre, confounds_wf, [('outputnode.t1_tpms', 'inputnode.t1_tpms')]),
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 233, in connect
infostr))
Exception: Some connections were not found
Module ConfoundDiscoverer has no input called inputnode.t1_transform
Module ConfoundDiscoverer has no input called inputnode.reference_image
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/407, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp3FLx7dus-ikwQR5-SFhMxOIaZEPks5rosSagaJpZM4MnO6P
.

Oh right... My brain is just shot, today, I guess.


Add runtime features to schema directives

Currently directives only act as "tags", ideally they would support custom functionality. For example, we'd like to check that a specific request header is set when calling mutations with a given directive.

I second this @lfvarela , would be great to have info be accessible from directive resolvers 👍

Great one would be Apollo's APQ. Build middleware for it 👯

@basvandriel how would that work? from what I see in those docs you add some directives and then you add a plugin. The directives don't do anything alone, right?

@patrick91 As far as I understand, query results are not cached, just the query string to improve improvements. Also has it's security reasons imo. It sends the queries in a sha256 hash. See the diagram in the link

The main use case for me is applying permissions, but have those permissions be visible to the client so that the client knows what headers to send or under what scopes they can access such mutations, without having to look at backend code.
One implementation option or additional idea could be to extend  BasePermission to be linked to a directive.

Hey guys,
Joining the conversation here to add my thoughts.
There are some interesting use cases for schema directives that have runtime functionality. For example, I expanded the current schema directive in strawberry-django-plus (https://github.com/blb-ventures/strawberry-django-plus/blob/master/strawberry_django_plus/directives.py) to be able to add runtime permission checking: https://github.com/blb-ventures/strawberry-django-plus/blob/master/strawberry_django_plus/permissions.py
One interesting feature of this approach is that they get printed to the schema, so someone consuming it can parse it to retrieve some extra info. E.g. the frontend guys that work with me parses our schema and checks for the permissions there. Knowing the current user's permissions they can add @include/@skip directives to avoid querying a field that they know they will not have permission for (and even if they did, the runtime functionality would not allow it).

Useful!


hi I've been trying to do this it lets me select the game but as soon as I click a game when it says launch and done filling everything out and crashes

What game is this?


Explore how to save scorecards definition inside a volume in a Kubernetes cluster

Let's define a new Kubernetes custom resource called scorecard and create a simple operator that will write that scorecard inside a volume in the Kubernetes cluster.

Now scorecards are saved inside the database


ProfileUpdate API fails when Contact is provided but not RawContact(s)

Problem
The following code is guaranteed to produce a failed result...
Contacts(context).profile().update().contact(contactToUpdate).commit()

There are two causes of this bug;

There is an internal API check that fails the operation immediate if at not even one RawContact is provided. In the above case, a Contact is provided but not a RawContact 🤦
There is another internal API check that fails the operation if the Contact is a profile contact instead of failing the operation if the Contact is NOT a profile contact🤦

Solution
Fix the internal API checks...

🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥
This is now release in 0.3.1!
🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥


Internal server error with ApiListener and header Message-Id

Current Behavior
My configuration has an <ApiListener> but no message log and no error storage. A HTTP POST request is sent to it with header Message-Id = 1000. The pipeline has been written to write the bode of the POST request in the database. That works fine. The issue starts when the same message is sent again. In Bruno, I see "500 Internal Server Error" in the response panel. In the Frank!Console, my adapter is still green. In the log, I see the following:
2024-09-02 10:19:22,111 INFO  [http-nio-8080-exec-7] {mid=1000, cid=1000, adapter=writeDb} receivers.Receiver - Receiver [ApiListener]  messageId [1000] correlationId [1000] businessCorrelationId [1000]
2024-09-02 10:19:22,112 WARN  [http-nio-8080-exec-7] {mid=1000, cid=1000, adapter=writeDb} receivers.Receiver - Receiver [ApiListener]  received message with messageId [1000] which has a problematic history; aborting processing
2024-09-02 10:20:18,194 INFO  [Cleaner-0] {} stream.Message - Leak detection: Message was not closed properly! Content: [Message[51117bad:CoyoteInputStream]: [org.apache.catalina.connector.CoyoteInputS]
2024-09-02 10:20:18,194 INFO  [Cleaner-0] {} stream.Message - Leak detection: Message was not closed properly! Content: [Message[677b0f37:String]: [<error>too many redeliveries or retries; exitState [S]
2024-09-02 10:20:18,195 INFO  [Cleaner-0] {} stream.Message - Leak detection: Message was not closed properly! Content: [Message[3d6dcc66:<null>]]

Expected Behavior
A success response should be returned. HTTP code 200 and something like <result><rowsupdated>1</rowsupdated></result>. This should be a cached response. Alternatively, a clear error or warning can be shown in the Frank!Console that the functionality to cache duplicates does not work without a message log.
Environment Information
FF! 8.3.0-20240830.042323: Frank2Transactions
Running on 677cd32ccafb using Apache Tomcat/10.1.28
Java Version: OpenJDK Runtime Environment (21.0.4+7-LTS)
Heap size: 113.2 MiB, total JVM memory: 214.0 MiB
Free memory: 100.8 MiB, max memory: 2.0 GiB
Free disk space: 38.6 GiB, total disk space: 58.4 GiB
Steps To Reproduce

Download Frank application from https://github.com/frankframework/frank-manual/tree/issueWithReceiverCaching/srcSteps/Frank2Transactions/v490 .
Do docker compose up.
Load the bruno files in Bruno.
Execute the write-db request twice.

PostgreSQL
What browsers are you seeing the problem on?
Microsoft Edge
Relevant Log Output
See above.

Anything else?
No response

I tried adding en JdbcErrorStorage (not on the shown branch). Then the second attempt of the request produces a record in the error storage:

So with an additional <JdbcErrorStorage>, the issue of duplicates is covered in some way.

I also saw in the Frank!Console that there were rejected messages. This also works without the error storage.

A message-id is meant to be unique. In your case because it fails it is stored in the errorstorage.
It's strongly adviced to use a messagelog with checkForDuplicates="true" on the Receiver.
See iaf-test as example.

The difference with the previously mentioned example is checkForDuplicatesMethod="CORRELATIONID".

I have a very similar configuration that behaves as I would expect. Now I have this:
<Configuration
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="../FrankConfig.xsd"
    >
    <Adapter name="writeDb">
        <Receiver checkForDuplicates="true"
            checkForDuplicatesMethod="CORRELATIONID" correlationIDXPath="/input/@correlationId">
            <ApiListener uriPattern="/write" method="POST" />
            <!-- <JdbcErrorStorage slotId="write-db"/> -->
            <JdbcMessageLog slotId="write-db"/>
        </Receiver>
        <Pipeline>
            <XmlInputValidator schema="input.xsd" root="input" throwException="true"></XmlInputValidator>
...

I do not supply headers anymore and the body of HTTP message is:
<input correlationId="1000" message="My first message" />

When I send this twice, the second response is the HTTP 304 Not Modified that I want.

I wonder whether this has to do with Ladybug issue https://github.com/wearefrank/ladybug/issues/293.

I see a reminder in my private todo list that I should test this issue. I told myself to do this when the latest source code of Ladybug will have been included in the FF!. Now I wonder who has to work on this issue right now. @nielsm5, will you work on this issue as the present assignee, or are you actually waiting for Ladybug to be updated?
I delete my reminder now and await your response or your work on this issue.


[Services Box are not coming in Proper order.]

What happened?
The services Section are not coming in proper order only three boxes are coming in a row instead all four should come .

Do assign me the role under GSSoC'23. I will fix this bug.
How can we reproduce this bug?
WE can make It Correct by placing in a row.
Desktop Information (Optional)
No response
Urgency (Optional)
Medium priority

@agamjotsingh18 sir,  please assign this issue to me under GSSoC'23. I will fix this bug with the help of HTML and CSS.

Sir, I have an idea. So whenever the width of a screen decreases the boxes are coming down. Rather than that i can just add a feature of sliding those four sections. So please assign this issue to me under GSSoC'23 if you like this idea.

Its not an issue, its wrapping for the mobile versions


Try putting the screenshot folder in git-lfs

Over time the git history is going to get heavy if you keep committing binary files to git

But they are so small it doesn't matter, kumbavu zangu


feat: Add a nicer summary view for trip planner output

Before

After

Parent Ticket: Trip Planner Preview | Output summary view
Subtask Ticket: Add mode icons (not worrying about the smushed icons for now) to the summary view

General checks

[ ] Are the changes organized into self-contained commits with descriptive and well-formatted commit messages? This is a good practice that can facilitate easier reviews.
[ ] Testing. Do the changes include relevant passing updates to tests? This includes updating screenshots. Preferably tests are run locally to verify that there are no test failures created by these changes, before opening a PR.
[ ] Tech debt. Have you checked for tech debt you can address in the area you're working in? This can be a good time to address small issues, or create Asana tickets for larger issues.

New UI, or substantial UI changes

[ ] Cross-browser compatibility is less of an issue now that we're no longer supporting IE, but changes still need to work as expected in Safari, Chrome, and Firefox.
[ ] Are interactive elements accessible? This includes at minimum having relevant keyboard interactions and visible focus, but can also include verification with screen reader testing.
[ ] Other accessibility checks such as sufficient color constrast, or whether the layout holds up at 200% zoom level.

New endpoints, or non-trivial changes to current endpoints

[ ] Have we load-tested any new pages or internal API endpoints that will receive significant traffic? See load testing docs
[ ] If this change involves routes, does it work correctly with pertinent "unusual" routes such as the combined Green Line, Silver Line, Foxboro commuter rail, and single-direction bus routes like the 170?

Assorted Implementation Notes

This does not fully solve the parent ticket - there are some elements missing, and the icons won't always be right. I'm treating those as follow-up tasks.
Smushed icons are a follow-up ticket. Showing icons side-by-side will be part of this ticket, but also a follow-up PR. This will require changing the ItineraryGroups algorithm to group the legs within itineraries as well as grouping similar itineraries themselves.
I wasn't able to use the Metro badges for this because they rendered incorrectly. Happy to fix that now if it's easy, but otherwise I don't want to block the functionality here on fixing the library, especially since the icons I had already built do do the job correctly.

I don't think any of those badge or icon components are necessary.

Can definitely use metro now to put thos SVGs into a directory and turn them into metro icons. Then put that icon into a circle badge.


Unable to run timesfm due to dependency issue

When I am trying to run timesfm on my cpu device, I am unable to install lingvo package.
Getting this error:
Could not find a version that satisfies the requirement lingvo while using pip.
Any idea on how to resolve this.

hmmmm,it seems that gup device cannot install lingvo too

Duplicate of #1

the same issue

pip install -e.
Obtaining file:///home/zd/timesfm
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Collecting einshape>=1.0.0 (from timesfm==0.0.1)
Using cached einshape-1.0-py3-none-any.whl.metadata (706 bytes)
Collecting paxml>=1.4.0 (from timesfm==0.0.1)
Using cached paxml-1.4.0-py3-none-any.whl.metadata (1.3 kB)
Collecting praxis>=1.4.0 (from timesfm==0.0.1)
Using cached praxis-1.4.0-py3-none-any.whl.metadata (1.2 kB)
Collecting jax>=0.4.26 (from timesfm==0.0.1)
Using cached jax-0.4.30-py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: numpy>=1.26.4 in /home/zd/anaconda3/lib/python3.11/site-packages (from timesfm==0.0.1) (1.26.4)
Requirement already satisfied: pandas>=2.1.4 in /home/zd/anaconda3/lib/python3.11/site-packages (from timesfm==0.0.1) (2.1.4)
Collecting absl-py (from einshape>=1.0.0->timesfm==0.0.1)
Using cached absl_py-2.1.0-py3-none-any.whl.metadata (2.3 kB)
Collecting jaxlib<=0.4.30,>=0.4.27 (from jax>=0.4.26->timesfm==0.0.1)
Using cached jaxlib-0.4.30-cp311-cp311-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting ml-dtypes>=0.2.0 (from jax>=0.4.26->timesfm==0.0.1)
Using cached ml_dtypes-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)
Collecting opt-einsum (from jax>=0.4.26->timesfm==0.0.1)
Using cached opt_einsum-3.3.0-py3-none-any.whl.metadata (6.5 kB)
Requirement already satisfied: scipy>=1.9 in /home/zd/anaconda3/lib/python3.11/site-packages (from jax>=0.4.26->timesfm==0.0.1) (1.11.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/zd/anaconda3/lib/python3.11/site-packages (from pandas>=2.1.4->timesfm==0.0.1) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /home/zd/anaconda3/lib/python3.11/site-packages (from pandas>=2.1.4->timesfm==0.0.1) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /home/zd/anaconda3/lib/python3.11/site-packages (from pandas>=2.1.4->timesfm==0.0.1) (2023.3)
Collecting absl-py (from einshape>=1.0.0->timesfm==0.0.1)
Using cached absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB)
Collecting clu==0.0.11 (from paxml>=1.4.0->timesfm==0.0.1)
Using cached clu-0.0.11-py3-none-any.whl.metadata (1.9 kB)
Collecting etils==1.7.0 (from paxml>=1.4.0->timesfm==0.0.1)
Using cached etils-1.7.0-py3-none-any.whl.metadata (6.4 kB)
Collecting flax==0.8.2 (from paxml>=1.4.0->timesfm==0.0.1)
Using cached flax-0.8.2-py3-none-any.whl.metadata (10 kB)
Collecting graphviz==0.20.1 (from paxml>=1.4.0->timesfm==0.0.1)
Using cached graphviz-0.20.1-py3-none-any.whl.metadata (12 kB)
Collecting jax>=0.4.26 (from timesfm==0.0.1)
Using cached jax-0.4.26-py3-none-any.whl.metadata (23 kB)
INFO: pip is looking at multiple versions of paxml to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement lingvo==0.12.7 (from paxml) (from versions: none)
ERROR: No matching distribution found for lingvo==0.12.7
lingvo 0.12.7没有release,装不了


PERF: Optimize Stocks get_historical_data to reduce message count in queries

As noted by @shlomikushchi in #148, HistoricalReader is not currently optimized for single day requests and for retrieving close only.
Enhancements

Add close_only parameter to get_historical_data to allow retrieval of close prices only to reduce message count (through chartCloseOnly)
Optimize single-day requests to get_historical_data such that they use date as a range and set chartByDay to True

Documentation
Clarifies documentation to specify get_historical_data as optimized and accepting a date range and close_only as parameters, while Stock.get_historical_prices is an exact mirror of the Historical Prices endpoint which accepts any parameter from the IEX docs

[ ] closes #xxxx
[x] tests added / passed
[x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
[x] added entry to docs/source/whatsnew/vLATEST.txt

cc #154


Fix get_relative_call_site after the split

get_relative_call_site needs a fix after the split, specifically get_root_path doesn't resolve to the root anymore. Instead of being "smart", let's just capture FQ path to the call site module.

specifically get_root_path doesn't resolve to the root anymore

Is that because articat cannot figure out which repository is using artecat when it is called? Or that doing so would be error prone?

@dhimmel yea, exactly. It's *definitely possible to do it in some cases, but I'm not sure if I see the value, and I actually see some benefits of recording FQ file path, specifically you can see which file exactly was used.
* we could use, git rev-parse --show-toplevel which would give us the git root (in most cases), but even then, artifact doesn't have a requirement to run within some git repo, so this would fail sometimes (which arguable we already do for git stats #11).
Any opinions?

I see value in both the fully qualified and relative paths. FQ works all the time, relative works most of the time.
The use case here that I'm most excited about is the ability to have the catalog link to source code on GitHub, and that use case is easier with a relative path.
So perhaps both could be recorded.

@dhimmel sounds good, let's open an issue to do a best effort relative path capture along the FQ path.


Should we map the '/projects/get' API

See https://cloud.google.com/dns/api/v1/projects/get
The resource consists of read-only information (quotas, mostly) about the project within DNS.
One strategy might be to add a quotas method to the client which fetched and returned the quotas sub-mapping for the project.

If it's simple to do, I see no reason not to. If it is a lot of work, we can revisit.

The Client.quotas method does expose all the relevant information.


Add information about enabling the plugins

I was wondering and searching for a long time why the S3 upload did not work. Then I found out that it should be enabled in the server.js.

Thanks!