Example for ESP32-S3 as HID Host communicate with HID device (IDFGH-10454)

I have a logger that uses HID USB to communicate with a PC via a serial cable using a proprietary protocol (raw data).
I want to us ESP32-S3 as a bridge to connect the logger to a PC via BLE.
Currently, the example for HID that is in the examples is for a keyboard and a mouse and doesn't work for raw protocol (see the description of the bug after the image).

I connected my HID device to the ESP232-S3 and run the example.
The device sends data in a loop.
In: app_main
In the “do” loop after line 468:  EventBits_t event = xEventGroupWaitBit
I added a log line: ESP_LOGI(TAG, "xEventGroupWaitBits %d",(int) event);
And what I see in the terminal: every second 1 line of:
I (525846) example: xEventGroupWaitBits 0
So I assume it didn’t see the any incoming data.

I have more info on this problem.
Now I see that when I connect the device to the ESP32 the terminal (the VS Code terminal) shows an error:
ClearCommError failed (PermissionError(13, 'Access is denied.', None, 5))
I search the code but I didn't find this error in the code.

Hi @shlomo-edu,
Thanks for the opening the ticket.
May I ask you to provide some more additional information, as:

Full debug log during the connection of your HID device to the ESP.
About the second error ClearCommError failed (PermissionError(13, 'Access is denied.', None, 5)). It seems that the problem is with connection to the serial output device (either /dev/ttyX on Unix or COMx on Win). To solve this it is better to check the correction of the port name or verify the access settings to the port.

Thanks

At the end of this post I added the log of the setup.
Please note:

When I add log lines to the hid_host_install(...) in hid_host.c (line 880)
The log lines are displayed till the: HID_ENTER_CRITICAL()
But then it shows this log lines
abort() was called at PC 0x403773a3 on core 0
0x403773a3: lock_acquire_generic at C:/Users/shlomoa/esp/esp-idf/components/newlib/locks.c:130
and many more log lines, and then it restarts and again (in endless loop of restarts).
I don't get this problem (or restarts) if I don't add the log lines.

The following lines are the setup log lines when I don't add the log lines to the hid_host_install(...) function.
PS C:\Users\shlomoa\hid> set IDF_PATH=C:\Users\shlomoa\esp\esp-idf
PS C:\Users\shlomoa\hid> C:\Users\shlomoa.espressif\python_env\idf5.0_py3.8_env\Scripts\python.exe C:\Users\shlomoa\esp\esp-idf\tools\idf_monitor.py -p COM9 -b 115200 --toolchain-prefix xtensa-esp32s3-elf- --target esp32s3 c:\Users\shlomoa\hid\build\hid.elf
--- WARNING: GDB cannot open serial ports accessed as COMx
--- Using \.\COM9 instead...
--- idf_monitor on \.\COM9 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3810,len:0x1658
load:0x403c9700,len:0xbe0
load:0x403cc700,len:0x2d9c
entry 0x403c9900
I (24) boot: ESP-IDF v5.0.2 2nd stage bootloader
I (25) boot: compile time 13:27:13
I (25) boot: chip revision: v0.1
I (27) boot.esp32s3: Boot SPI Speed : 80MHz
I (31) boot.esp32s3: SPI Mode       : DIO
I (36) boot.esp32s3: SPI Flash Size : 2MB
I (41) boot: Enabling RNG early entropy source...
I (46) boot: Partition Table:
I (50) boot: ## Label            Usage          Type ST Offset   Length
I (57) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (65) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (72) boot:  2 factory          factory app      00 00 00010000 00100000
I (80) boot: End of partition table
I (84) esp_image: segment 0: paddr=00010020 vaddr=3c030020 size=0ce2ch ( 52780) map
I (102) esp_image: segment 1: paddr=0001ce54 vaddr=3fc92200 size=023b8h (  9144) load
I (104) esp_image: segment 2: paddr=0001f214 vaddr=40374000 size=00e04h (  3588) load
I (110) esp_image: segment 3: paddr=00020020 vaddr=42000020 size=24200h (147968) map
I (144) esp_image: segment 4: paddr=00044228 vaddr=40374e04 size=0d368h ( 54120) load
I (162) boot: Loaded app from partition at offset 0x10000
I (162) boot: Disabling RNG early entropy source...
I (174) cpu_start: Pro cpu up.
I (174) cpu_start: Starting app cpu, entry point is 0x40375228
0x40375228: call_start_cpu1 at C:/Users/shlomoa/esp/esp-idf/components/esp_system/port/cpu_start.c:141
I (0) cpu_start: App cpu up.
I (188) cpu_start: Pro cpu start user code
I (188) cpu_start: cpu freq: 160000000 Hz
I (188) cpu_start: Application information:
I (191) cpu_start: Project name:     hid
I (196) cpu_start: App version:      1
I (200) cpu_start: Compile time:     Jun 19 2023 13:25:06
I (206) cpu_start: ELF file SHA256:  4cb8b40f805434a4...
I (212) cpu_start: ESP-IDF:          v5.0.2
I (217) cpu_start: Min chip rev:     v0.0
I (222) cpu_start: Max chip rev:     v0.99
I (227) cpu_start: Chip rev:         v0.1
I (231) heap_init: Initializing. RAM available for dynamic allocation:
I (239) heap_init: At 3FC95020 len 000546F0 (337 KiB): DRAM
I (245) heap_init: At 3FCE9710 len 00005724 (21 KiB): STACK/DRAM
I (252) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM
I (258) heap_init: At 600FE010 len 00001FF0 (7 KiB): RTCRAM
I (265) spi_flash: detected chip: gd
I (268) spi_flash: flash io: dio
W (272) spi_flash: Detected size(8192k) larger than the size in the binary image header(2048k). Using the size
in the binary image header.
I (286) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (306) gpio: GPIO[0]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0
I (306) example: HID HOST example

Hi @shlomo-edu,
There are couple of thing that you need to consider.

When you want to print anything from the critical section, you need to use not blocking version of ESP_LOGx which is ESP_EARLY_LOGx;. ESP_LOGx is a blocking API, so we'll usually get an assert you attempt to block in a critical section.
For the HID driver the device should have a HID class device value in USB device descriptor. Otherwise, there will be an error such as Device at addr x has no HID Interface, where x - is a device address on USB line.

To proceed, may I ask you to provide a description of USB descriptor from the device you are trying to attach?
Thanks

Hi Roma,
Thanks for helping me cross this bridge.
Yes, the device class has an HID descriptor.
Here is how it is presented in YAT (a terminl) application.
Please note that I see that "client_event_cb()" (in hist_hid.c line 550) is never called.

More on this.
I see in the code that the only functions that change the xEventGroupWaitBits, are:
test_hid_wait_connection_and_install()
and
test_hid_wait_for_removal()
So I added, log lines in the beginning of the functions.
I see that it never gets there.
I don't know what else to check!!!

Hi @shlomo-edu,
Thanks for the additional information.
Meanwhile, to proceed I need to ask you the following:

Which board do you use? Probably, the problem can be not related to the HID or USB
To verify USB connection in general, you can use the usb_host_lib example
To be more specific, it is better to have a device descriptor, which can be obtained either from system information or with any external application (for example, you can use this one: Thesycon USB Descriptor Dumper )

Thanks

ESP32-S3-DevKitM 1 v1.0
My programmer used the usb_host_lib and it didn't show anything.
Following is the description of this HID usb from the Thesycon" (tdd.exe).
This means that this program can also interface with our device (as well as YAT).
So now the question is why the ESP32-S3 can't?

Information for device EC8xx III (VID=0x10C4 PID=0x1F0C):

Connection Information:
Device current bus speed: FullSpeed
Device supports USB 1.1 specification
Device supports USB 2.0 specification
Device address: 0x0015
Current configuration value: 0x01
Number of open pipes: 2

Device Descriptor:
0x12	bLength
0x01	bDescriptorType
0x0200	bcdUSB
0x00	bDeviceClass
0x00	bDeviceSubClass
0x00	bDeviceProtocol
0x40	bMaxPacketSize0   (64 bytes)
0x10C4	idVendor
0x1F0C	idProduct
0x0200	bcdDevice
0x01	iManufacturer   "Fourtec"
0x02	iProduct        "EC8xx III"
0x03	iSerialNumber   "0E5139433135"
0x01	bNumConfigurations

Configuration Descriptor:
0x09	bLength
0x02	bDescriptorType
0x0029	wTotalLength   (41 bytes)
0x01	bNumInterfaces
0x01	bConfigurationValue
0x00	iConfiguration
0xC0	bmAttributes   (Self-powered Device)
0x32	bMaxPower      (100 mA)
Interface Descriptor:
0x09	bLength
0x04	bDescriptorType
0x00	bInterfaceNumber
0x00	bAlternateSetting
0x02	bNumEndPoints
0x03	bInterfaceClass      (Human Interface Device Class)
0x00	bInterfaceSubClass
0x00	bInterfaceProtocol
0x00	iInterface
HID Descriptor:
0x09	bLength
0x21	bDescriptorType
0x0111	bcdHID
0x00	bCountryCode
0x01	bNumDescriptors
0x22	bDescriptorType   (Report descriptor)
0x0021	bDescriptorLength
Endpoint Descriptor:
0x07	bLength
0x05	bDescriptorType
0x81	bEndpointAddress  (IN endpoint 1)
0x03	bmAttributes      (Transfer: Interrupt / Synch: None / Usage: Data)
0x0040	wMaxPacketSize    (1 x 64 bytes)
0x20	bInterval         (32 frames)
Endpoint Descriptor:
0x07	bLength
0x05	bDescriptorType
0x01	bEndpointAddress  (OUT endpoint 1)
0x03	bmAttributes      (Transfer: Interrupt / Synch: None / Usage: Data)
0x0040	wMaxPacketSize    (1 x 64 bytes)
0x20	bInterval         (32 frames)
Microsoft OS Descriptor is not available. Error code: 0x0000001F

String Descriptor Table
Index  LANGID  String
0x00   0x0000  0x0409
0x01   0x0409  "Fourtec"
0x02   0x0409  "EC8xx III"
0x03   0x0409  "0E5139433135"

Connection path for device:
USB xHCI Compliant Host Controller
Root Hub
EC8xx III (VID=0x10C4 PID=0x1F0C) Port: 3
Running on: Windows 10 or greater (Build Version 19044)
Brought to you by TDD v2.17.0, Feb 23 2021, 14:04:02

@shlomo-edu ,
Thanks for sharing additional information.

With that board there is no power supply (+5 V) for the usb otg port. To make it work you need to deliver a voltage supply of +5 V to the USB Device. It could be done with external wire.

The descriptor looks fine, it is a HID comatible device without any protocol.
The HID driver will treated such device with HID_PROTOCOL_NONE.

Both example should work (HID example then need a small change) after power will be delivered.
Thanks

Hi Roma,
Redarding the usb_host_lib
I tried to build it on my my computer and it failed because it didn't find the "usb/usb_host.h".

The idea that the problem is the power makes sence, but -
The serial port is connected to my laptop that gives a power.
I see that the software works (I see the log lines in the VSC terminal).
But, I see that the led is RED.
Maybe it is an indication of a problem!!!???

Hey @shlomo-edu,

To be able to use usb component you need to set terget to esp-idf. This can be done with idf.py set-target esp32s2 (or esp32s3, depending on a board)
I mean that there are two usb ports on a board. The one, which is used for programming is power the esp32s2 and the board itself.
But for the second port (to which the another USB Device is attaching) needs to be powered additionally.

Thanks

OK, I think that you are on the right path to solve the problem.
How do I power it?

You were right. I added 5V to the USB and now it calls is_hid_device() with dev_addr=1
But,
It fails on this line:
if (get_interface_desc_by_proto(config_desc, &offset, HID_PROTOCOL_NONE))
Maybe the reason is that this is the wrong dev_addr, because I see that in the tdd.exe application, the
Device address: 0x0015 (see in previous post).

Hey @shlomo-edu,
Glad to hear it works.
Unfortunately, the ESP32-S3-DevKitM 1 v1.0 board was designed for the USB Device examples, that is why to launch USB Host examples there is a necessary to fix it yourself.
Just in case, there are several OTG boards, that can easily be used for the USB Host examples without any additional modifications.
Back to the example.
That is exactly, what we are talked about when we discussed the first version of HID driver. The idea is that it supports only Keyboard and Mice, Boot protocols (here)
So, to make it work, please feel free to:

Remove HID_SUBCLASS_BOOT_INTERFACE comparison statement in the file hid_host.c, line 282
Change the protocol from HID_SUBCLASS_BOOT_INTERFACE to HID_SUBCLASS_NO_SUBCLASS in the file hid_host.c, line 1067
Change the comparison iface_config->proto > HID_PROTOCOL_NONE to iface_config->proto >= HID_PROTOCOL_NONE in the file hid_host.c, line 966
Comment out or remove lines from 986 to 1005 in the file hid_host.c, line 986
Install the device and register an event callback hid_host_interface_event_callback during the DEVICE_ADDRESS_MASK event in the app task
Claim the interface with hid_host_claim_interface during the hid_host_interface_event_callback. But be careful, in example there is a ternary operator to distinguish the protocol, so change the code here to claim interface with protocol you need HID_PROTOCOL_NONE

After that you should be able to get the raw data report callback with params (const uint8_t *const data, const int length), where is data - is a pointer to raw data of the input report and the length - is the number of bytes in this input report.
There still can be problems during device detaching (such as memory leakage e.t.c), because to fully support HID_PROTOCOL_NONE in the first version there little bot more things need to be done.
But they are already done in the HID driver, which will be released this week as an idf-extra-component.
Thanks

Hey @shlomo-edu ,

I don't understand what is "other OTG boards", what are they, what are the differences (between them and the S3) and why should I avoid using them.
Please elaborate on that.

There are at least two OTG boards from Espressif which easily coulb be used for OTG or USB Host example without any changes:

ESP32-S3-USB-OTG
ESP32-S3 WROOM FREENOVE

Regarding our example, I started to do the changes, but I don't understand what to do in 5 and 6.

These logic is implemented in the example:

For point 5 file hid_host_example.c, line 484
For point 6 file hid_host_example.c, line 364
Feel free to refer to it.

I don't understand the following:
"But they are already done in the HID driver, which will be released this week as an idf-extra-component."
What is the driver, how can I get it, how do I work with it (is there a sample code for it)?

Now, the idf-extra-component is in merging state and is ready to be merged. So it will be available as an idf-extra-component very soon.

Is there a complete project that I can install in VCS and use (as is)?

Not yet. After the HID driver idf-extra-component will be released, the HID Host example should be updated to use this component. So, that the next step after component release.
If there is still question, feel free to ask them.
Otherwise, probably it is better to wait the release of HID Driver idf-extra-component and example with usage of it.
Thanks

Hi Roma,
Is it better to just purchase ESP32-S3-USB-OTG or ESP32-S3 WROOM FREENOVE for my purpose.
What are the adavantages of each of them?
Do you have a sample code to work with them?
Can I use them just as I use the ESP32-S3-DevKit?
Currently I added the 5V from the PC, If I add a battery to the ESP32, how do I connect it so it will supply the 5V to the USB?
Do you have a complete board with a battery?

Hi Roma.
I am still waiting for yoour reply on the ESP32-S3_OTG and ESP32-S3-WROOM?
And also on the question regarding the board with a battery?
I rub the code with some extra log lines:
Please note that the device keep sending raw data every second.
This is the log lines that I receive:
Can you see from the log why I don't receive the chracters?
I (3491) example: app_main: event=16
I (3491) example: Interface number 0, protocol Mouse
I (3491) example: after hid_host_claim_interface
E (3551) USBH: Device 1 gone
I (3551) example: hid_host_event_callback
I (3551) example: app_main: event=a
E (3551) hid-host: hid_host_release_interface(1027):
E (3551) hid-host: hid_host_release_interface(1027):
I (3951) example: hid_host_event_callback
I (3951) example: app_main: event=16
I (3951) example: Interface number 0, protocol Mouse
I (3951) example: after hid_host_claim_interface
E (4011) USBH: Device 1 gone
I (4011) example: hid_host_event_callback
I (4011) example: app_main: event=a
E (4011) hid-host: hid_host_release_interface(1027):
E (4011) hid-host: hid_host_release_interface(1027):
I (4411) example: hid_host_event_callback
I (4411) example: app_main: event=16
I (4411) example: Interface number 0, protocol Mouse
I (4411) example: after hid_host_claim_interface
E (4471) USBH: Device 1 gone

Hi @shlomo-edu,
Sorry for the long response.

Is it better to just purchase ESP32-S3-USB-OTG or ESP32-S3 WROOM FREENOVE for my purpose.
What are the adavantages of each of them?

That is fully up on you. That is different boards for different purposes, so they have different peripheral on the board. Please, refer to their description to pick one of them that suits your requirements.
The main idea - there is nothing to modify in them, so you can use them with usb_host examples without any changes. (For the ESP32-S3_OTG board some lines to the example should be added to power up the OTG port, ESP32-S3 WROOM FREENOVE you can use without any changes in the examples code).

Do you have a sample code to work with them?

Any example from esp-idf works with these boards.

Can I use them just as I use the ESP32-S3-DevKit?

Yes, but the power supply should be added to the USB OTG port with an external wire.

Currently I added the 5V from the PC, If I add a battery to the ESP32, how do I connect it so it will supply the 5V to the USB?

The same way. The difference will be that the +5 V will be delivered not from PC USB port but from external battery.

Do you have a complete board with a battery?

I am not sure. Probably, there is some boards from third party companies, but it is better to search them by yourself, because there is a huge variety of them.

@shlomo-edu,

Can you see from the log why I don't receive the chracters?

Unfortunately, there is not enough information.
The strange thing is that after interface claiming there is a USB dev gone event, which means that the device was removed.
There should be a cause of that, but that is not how it should be.
The idf-extra-component has been released last week: https://components.espressif.com/components/espressif/usb_host_hid
The next step will be the example update in the esp-idf master repository, so this version will work with any protocol of HID compatible device and provide an option to receive the raw data input report.
By the way, do you need the possibility to send some information back to the HID device from the host?
Thanks

Thanks Roma,
I will putchase one of them and I will then add a comment to let you know how it works.
Thank you very much for your help.

Hi Roma,

I tried to find where I can purchase the  ESP32-S3 WROOM FREENOVE, but could find a relibale source.
Maybe you can help me with that.
I found the code for it in GitHub and the email of the company<EMAIL_ADDRESS>but they haven't replied. Maybe you can ask them.
It looks like this path will take a lot of time till I recieive it, and I am afraid that again it will not work as expected (this will be my fouth purchase of PCB's to solve the problem).
So maybe you can help me make the ESP32-S3-DevKit work with the hid_host_example.c?
I can provide add any log lines that you ask.

Hi @shlomo-edu,
1 and 2. For this topic is better to use official contacts: https://www.espressif.com/en/contact-us/sales-questions, or you can user esp32 forum https://www.esp32.com/ which is intended for all kind of questions, not only bugs or feature requests, as the github.

From the point that this is lead to the code anyway, it is better to use examples 'as is' to get better experience. Examples in esp-idf show a way how the esp-idf can be used and they do not cover all the scenarios. If the option to change the current example in esp-idf for HID_PROTOCOL_NONE seems not reasonable, probably I would suggest then to wait for the example update to get full support of HID devices with any protocol.

Thanks

Hi @shlomo-edu,
FYI, there was an example update: https://github.com/espressif/esp-idf/commit/5ceac3cd19f2516b73746239d13c45df30675a99
Now the example uses the idf-extra-component USB Host HID (Human Interface Device) Driver and shows how input report can be obtained even with a generic HID device.
Feel free to try it to get the data from your device. Hope it will help with your case.
In addition, feel free to ask more questions if it helps. Otherwise, if there will be no activity the issue will be closed.
Thanks

Hi Roma,
Thanks, but we can't compile it because it doesn't find the macro: pdMS_TO_TICKS

Hi @shlomo-edu,
It seems that it is still the code of the old version of HID example.
The HID example was updated in master branch and was backported to release v5.1.
To experience new version of HID example, please use one of these branches.
Thanks

Hi Roma,
Please note that we downloaded the latest brunch.
Some of the files where updated 2 days ago.
Please attach the specific file that implements the: pdMS_TO_TICKS
Maybe it was not uploaded to the GIT.

Hi @shlomo-edu,
Please, try to get the final version of the code.
Pay attention to the hid_host_example.c, the new version has a new way of polling the APP_QUIT_PIN state verification to exit the example (refer to updated file here)
If there is still problem with building, please attach the full build log output for me to be able to understand the problem.
Otherwise, the Visual Studio Code can miss some headers and marked the lines with the red underlines.
That is not affected the build process, that is only a indexing problem.

It didn't work.
Here is the log -
Creating esp32s3 image...
Merged 2 ELF sections
Successfully created esp32s3 image.
Generated C:/Users/Victoria/Desktop/ESP-PRJ/hid/build/bootloader/bootloader.bin
[110/110] cmd.exe /C "cd /D C:\Users\Victoria\Desktop\ESP-PRJ\hid\build\bootloader\esp-idf\esptool_py && C:\Users\Victoria.espressif\python_env\idf5.2_py3.11_env\Scripts\python.exe C:/Users/Victoria/esp/esp-idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x0 C:/Users/Victoria/Desktop/ESP-PRJ/hid/build/bootloader/bootloader.bin"
Bootloader binary size 0x52b0 bytes. 0x2d50 bytes (35%) free.
[941/943] Linking CXX executable hid.elf
FAILED: hid.elf
cmd.exe /C "cd . && C:\Users\Victoria.espressif\tools\xtensa-esp32s3-elf\esp-12.2.0_20230208\xtensa-esp32s3-elf\bin\xtensa-esp32s3-elf-g++.exe -mlongcalls -Wl,--cref -Wl,--defsym=IDF_TARGET_ESP32S3=0 -Wl,--Map=C:/Users/Victoria/Desktop/ESP-PRJ/hid/build/hid.map -Wl,--no-warn-rwx-segments -fno-rtti -fno-lto -Wl,--gc-sections -Wl,--warn-common -T
esp32s3.peripherals.ld -T esp32s3.rom.ld -T esp32s3.rom.api.ld -T esp32s3.rom.libgcc.ld -T esp32s3.rom.newlib.ld -T esp32s3.rom.version.ld -T memory.ld -T sections.ld @CMakeFiles\hid.elf.rsp -o hid.elf  && cd ."
c:/users/victoria/.espressif/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/bin/ld.exe: esp-idf/espressif__usb_host_hid/libespressif__usb_host_hid.a(hid_host.c.obj): in function hid_host_install': C:/Users/Victoria/Desktop/ESP-PRJ/hid/managed_components/espressif__usb_host_hid/hid_host.c:1082: multiple definition of hid_host_install'; esp-idf/hid/libhid.a(hid_host.c.obj):C:/Users/Victoria/Desktop/ESP-PRJ/hid/components/hid/hid_host.c:875: first defined here
c:/users/victoria/.espressif/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/../lib/gcc/xtensa-esp32s3-elf/12.2.0/../../../../xtensa-esp32s3-elf/bin/ld.exe: esp-idf/espressif__usb_host_hid/libespressif__usb_host_hid.a(hid_host.c.obj): in function hid_host_uninstall': C:/Users/Victoria/Desktop/ESP-PRJ/hid/managed_components/espressif__usb_host_hid/hid_host.c:1163: multiple definition of hid_host_uninstall'; esp-idf/hid/libhid.a(hid_host.c.obj):C:/Users/Victoria/Desktop/ESP-PRJ/hid/components/hid/hid_host.c:936: first defined here
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

The terminal process "C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -Command ninja " terminated with exit code: 1.

Hi @shlomo-edu,
Does the example build after cleaning the folder?
Is there any other problems?
Thanks

Hi Roma,
Thanks for your inquery.
My programmer had tested it, and now it compiles OK.
But when she run it, it "crashes" (this is what she told me).
I didn't have the time (yet) to examine it deeper.
Please note that what she does is just receive the data from the device.
The device is continuously sending data to the ESP.

@shlomo-edu ,
Thanks for the response.
Please, feel free to attach the crash report or any log during your (or her) testing. It helps me to solve the problem faster (if there is one).
Thanks

Hi Roma,
Thank you very much for your assistance.
Now I realize that it is not crash, but rather it just doesn't load the binary to the ESP32.
This is what my progammer tells me.

@shlomo-edu Closing this issue as it's more likely problem with building/programming firmware and there is no issue with getting input report with HID Host example.
Feel free to reopen if it's not.

Hi Roma,
I have received the ESP32-S3-WROOM Board from Freenove (the OTG).
But I don’t see any example how to check the connection to the device.
It does have two similar USB ports, and I can download the sketch and I can read and write from/to the Arduino Serial Monitor (using the “Serail” object).
It looks like the two ports are similar, so I assume that it can Read/Write from the device also with the “Serial” object, but how do I use two Serial objects (one for the PC and one for the device).
@.***
From: Roma Jam @.>
Sent: Monday, August 14, 2023 11:39 PM
To: espressif/esp-idf @.>
Cc: Shlomo Aran @.>; Mention @.>
Subject: Re: [espressif/esp-idf] Example for ESP32-S3 as HID Host communicate with HID device (IDFGH-10454) (Issue #11702)
@shlomo-eduhttps://github.com/shlomo-edu Closing this issue as it's more likely problem with building/programming firmware and there is no issue with getting input report with HID Host example.
Feel free to reopen if it's not.
—
Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-idf/issues/11702#issuecomment-1678026365, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT6RBWII5UCG6IKVPSAHNDDXVKEHPANCNFSM6AAAAAAZLYG6QU.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Hi Roma,
Now I see that the two ports are not similar.
I can upload the sketch only when the PC is connected to the right port :
@.***
From: Roma Jam @.>
Sent: Monday, August 14, 2023 11:39 PM
To: espressif/esp-idf @.>
Cc: Shlomo Aran @.>; Mention @.>
Subject: Re: [espressif/esp-idf] Example for ESP32-S3 as HID Host communicate with HID device (IDFGH-10454) (Issue #11702)
Closed #11702https://github.com/espressif/esp-idf/issues/11702 as completed.
—
Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-idf/issues/11702#event-10091260427, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT6RBWII5UCG6IKVPSAHNDDXVKEHPANCNFSM6AAAAAAZLYG6QU.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

Hi Roma,
I used https://components.espressif.com/components/espressif/usb_host_hid and it works. I still have little experience with the host and I can not find the functionality of how to get from the device in the program string_descriptor[] and report_descriptor[] Is there such a possibility?


⚠ Develop API Endpoint has degraded performance

In 341e0a5, Develop API Endpoint (http://develop.roblox.com/) experienced degraded performance:

HTTP code: 200
Response time: 2036 ms

Resolved: Develop API Endpoint performance has improved in c69f954 after 25 days, 11 hours, 53 minutes.


Update layout of settings schema

Hi @merdok
this PR update layout of config.schema which make the settings layout more clear and comfortable to use.
Thanks
Grzegorz

Hi,
thanks for the PR. This looks indeed much more cleaner and better. But there are some issues:

none of the buttons does not work, the values are not written to the config
if you press Enter while adding a button in any of the inputs it will add endless "Name" fields.

Could you have a look at that?
You can validate the config easily here: https://hamidihamza.com/ajsf/?framework=bootstrap-4

All fixed.

This looks already much better! Thanks! What i noticed, when i open my existing config then always only the first appButton or notificanButton etc is shown i have to press on the "Add ..." button so that the other entires will display. Not sure if this was like that?

Ok,
will check this again.

I don’t know why the array are not automatically display all configured buttons. I have tested this on my own plugins and sometimes was display all sometimes only first. May be something wrong with Config UI X.
The layout is correct configured and write and save all value to the config.json.

I really like how compact and clean the config now is. Very good job on that! But we need to figure out why only the first entries are present in arrays. I noticed basically every array is now affected of this. This might confuse people. I wonder if there are any

Yes, it is very strange, I have already tested same Schema with my denon plugin and here the array with inputs automatically displays all values. I workin on it, be patient😀

index.js has nothing to do with config.schema.json, those files are completely separate and are not in any way related. But i am also not sure what causes the issue. I do not know if this is a bug in the config.schema.json or maybe it is a bug in homebridge-config-ui-x. I also could not find any examples which use such complex properties defines as in this plugin. We need to investigate this further and find a fix, since i really like the new layout 👍

I think it is not a bug in Config-UI-X because my homebridge-denon-tv plugin uses same schema and there the Inputs array are displayed all values after expand it. I will fight with this problem, this must be working. :)

That is true, if you say that it is working on another plugin of yours then it also should work here.

Any news on this?

Nothing new, this must be issue with Config UI X. I have report this as bug in Config UI X repo and waiting on answer.

Ok...

Any news on this?


Status Bar background color remains VS Code default

Describe the bug
Status Bar background color remains VS Code default which causes text to be barely visible.
To Reproduce
Steps to reproduce the behavior:

Open VS Code with either an old project, a new project or a single file.
Take a look at status bar.

Expected behavior
Text on the Status Bar is clearly visible.
Screenshots

Desktop (please complete the following information):

macOS High Sierra
VS Code Version: 1.25.1
Material Theme Version: 2.3.0

Hi @melwas. Can you please make sure that you don't have any override in your settings (both project and global)?

Hi @equinusocio, you're right :) There were some overrides which caused the issue. Closing the issue. Thanks!


Update libz-sys and curl-sys

Fixes some build issues on windows
r? @Diggsey

@bors r+

:pushpin: Commit a6a8565 has been approved by Diggsey

:hourglass: Testing commit a6a8565ebe8675cfd09c9bc4f7afe741ba02ec48 with merge e2dfde243e6bc96c881aa45112b2f77f923de285...

:sunny: Test successful - status-appveyor, status-travis
Approved by: Diggsey
Pushing e2dfde243e6bc96c881aa45112b2f77f923de285 to master...


chore(main): bump version to 2.1.9

:robot: I have created a release beep boop
2.1.9 (2023-02-04)
Bug Fixes

http service transform response handling (a5e9556)

Chores

deps: upgrade dependency packages (660b6c1)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/kenso312/nestjs-v9-webpack-boilerplate/releases/tag/v2.1.9 :sunflower:


Create withdrawal SharedRoyaltyToken interface

Overview
As a developer implementing a shared royalty token, I want to know I'm using best practices so that I don't break users (increasing gas costs) and ensure security (no reentrancy)
Reference

Reentrancy topic
Common Patterns

Questions

Are there any known ERC standards for contract withdrawals?
Can we calculate balance at withdrawal time?
How do we handle rounding errors?

Acceptance

Documented contract withdrawal pattern is written as a contract interface with NatSpec documentation (here)
Documented questions are answered

Tasks

[ ] Research existing ERCs and prior art
[ ] Document results and proposed interface

Notes:
OpenZeppelin has an escrow contract that we can/should use to handle the escrow for the token. We can expose it's api via ours.
Rounding errors doesn't really seem to be an issue (??) as the math is handled in wei. Viewing OpenZeppelin's Payment Splitter, they don't make any special cases for handling it.
For understanding, if we can calculate balance at withdrawal time:
I wrote a tiny script to run a test scenario of splitting every payment by the number of franchisors. The gas cost per transaction follows this formula (except index 0, which spikes a bit) gasUsed = 7781 * franchisorIndex + 63135
There is a max gas limit around 8mil, that means our token can be sold a max number of ~7,000,000 times. The max cost in tx fees (assuming current USD market value and a gas price of 4Gwei) is ~$3.80.
☝ That's not as bad as I was guessing. It's not amazing if we're assuming we could have a single digital asset (an album) that gets distributed via everyone who's owned it with the rights to sell an unlimited number of times (ie, a single token representing a single album).
I'm going to take another think about HMW make transfer's a fixed gas price while withdrawals increase with the number of transactions or amount that can be withdrawn.

@breakpointer What do you think about my assumptions here?

Seem solid to me; rounding errors handled, escrow seems like the right approach here, 7 million is plenty of headroom for transactions on one asset. 👍

Bah, I did the math wrong, max of 1020. Editing original post.

I'm not looking forward to implementing this. Here's hoping some other lucky person gets to!
The payment proposed above does payment calculation and distribution at transfer time, this means gas prices for transfer are linear. Ideally, it would be fixed.
Some operation that can be broken up needs to be linear so that it's possible to have (virtually) unlimited franchisors, and all franchisors can get all of the Eth they're owed. What I believe this to mean is that the only place to do the expensive calculations is at withdraw-time and withdraw needs to be able to limit the number of withdrawals by the gas expense.
I can see two different apis to do this, one more ux friendly, the other more dev friendly. I'll document below, bear in mind that for both, the following new things must be true.

Withdrawals happen per 721, you cannot withdraw all earnings from all tokens at once, at least at the 721 level. (upstream contracts could iterate over all of them)
Every token keeps track of a stream of payments, it's own transaction history uint[] payments
Withdrawals will need to keep track of which payments have been withdrawn for each franchisor to ensure no double payments
each "style" will need to implement a paymentBalanceOf(tokenId, franchisor, paymentStart, paymentEnd) function that, for any token franchisor, and range of payments, calculates what that franchisor is owed. This will become complicated with cycles of franchisors (think a franchisor selling the token to themselves)

The two different withdrawl patterns are:

(Better for devs) withdrawals are paged by the user with a count that starts where they last withdrew from, and goes to the end. If they give too high of a count, it will revert and they'll need to try again
(Better for users) withdraw transfers the most amount it can given the gas provided. returns true if done, or false if there is still more to withdraw. This will require the function to know how much gas calculating a balance is and can use gasLeft() to make sure it has enough to send the eth after. This seems tricky and I think the top one is better for a first pass at the very least as the balance calculation could vary a lot and might not be linear.

I'm assuming a "real" app would be able to streamline this whole withdrawal thing so the user never sees it anyway. Interface coming next.

The interface! A few notes. The "escrow" contract linked in the previous comment no longer applies. I still like the pattern and a custom one could be used in an implementation (a SharedRoyaltyEscrow anyone?) but that shouldn't affect the exposed abi.
Without further ado:
  /**
   * @notice Withdraws payment accumulated from transfer of a given token from
   *  the last withdrawn payment up to a _count

   * @param _tokenId The identifier for an NFT
   * @param _count The number of payments to traverse
   */
  function withdrawPayment(uint256 _tokenId, uint256 _count) external;

  /**
   * @notice Withdraws non-withdrawn payment accumulated from transfer of a
   *   given token
   * @dev Escrow should keep track of the payments that have been processed to
   *   avoid double withdrawals
   *
   * @param _tokenId The identifier for an NFT
   */
  function withdrawPayment(uint256 _tokenId) external;

  /**
   * @notice Gets balance of non-withdrawn payment accumulated from transfer of a
   *   given token up to a number of provided payments
   * @dev Used by `withdrawPayment` to calculate how much a franchisor is owed.
   * @dev Implement this function to
   *
   * @param _tokenId The identifier for an NFT
   * @param _start The payment index to start from
   * @param _count The number of payments to traverse
   */
  function paymentBalanceOf(uint256 _tokenId, uint256 _start, uint256 _count) public;

  /**
   * @notice Gets balance of non-withdrawn payment accumulated from transfer of a
   *   given token that has not been withdrawn
   *
   * @dev Defaults to overloaded `withdrawPayment` of a token's non-withdrawn balance
   *   to the last payment
   * @dev Used by `withdrawPayment` to calculate how much a franchisor is owed.
   *
   * @param _tokenId The identifier for an NFT
   */
  function paymentBalanceOf(uint256 _tokenId) public;

@barlock WRT "withdrawals are paged" Could you provide an example of the data structure you have in mind here? I thought we could have a simple balance (single float) to store what a franchisor would be owed at any given time. During the transfer transaction when eth is distributed we would be crediting those balances with a fixed rate * sale price.

I thought we could have a simple balance (single float) to store what a franchisor would be owed at any given time.

That was my original attempt and a clean solution. With that approach, you need to calculate the balance when a transfer happens. Given our "brain dead simple" model of equal royalties (the code in the original post), it limited the number of transfers to 1k. I don't believe that to be large enough for our purposes.
My second attempt, the one here, tries to perform that calculation at withdraw time rather than deposit time. A rough attempt in code below, the data structure for the escrow will likely change but hopefully it will get the idea across.
I realized when writing this that the

struct Token {
  uint256[] payments;
  address[] franchisors;
  mapping(address => uint256) franchisorLastWithdrawIndex;
}

mapping (uint256 => Token) public tokens;

function transfer (uint256 _tokenId, address payable _newOwner) {
  Token token = tokens[_tokenId];

  token.franchisors.push(_newOwner);
  token.payments.push(msg.value);
}

function withdraw (uint256 _tokenId, uint256 _count) {
  // Payment balance of is implemented per style, so bookened would calculate this
  // sender's balance (I realized now that maybe the api should specifiy which payee, need to check prior art)
  // based on the `payments` array, start, and count
  uint256 withdrawBalance = this.paymentBalanceOf(
    _tokenId,
    token.franchisorLastWithdrawIndex[msg.sender],
    _count
  );

  msg.sender.transfer(withdrawBalance);
}

function withdraw (uint256 _tokenId) {
  Token token = tokens[_tokenId];

  this.withdraw(_tokenId, token.payments.length);
}


Paginate Problem

I have an 'index.html' file . in '/_pages' and when I surpass the pagination limit with posts (paginate: 5 and I have 6 posts) then I will get the error
Error reading file /Path/To/minimal-mistakes/index.html: No such file or directory @ rb_sysopen - /Path/To/Downloads/minimal-mistakes/index.html

I also took a screenshot of the site overview and see there are two index.html in the site
I think there is a problem in how it saves the index.html in the way I configure it or the default one
in '-/pages/index.html'
---
permalink: /index
layout: home
---

in '_config.yml'
paginate: 3 # amount of posts to show
paginate_path: /_pages/page:num/

What you can try is moving the index.html out of _pages and into the root of your project.
Jekyll's pagination plugin is extremely picky and doesn't work when you deviate from its conventions.
Closing as this isn't a theme issue. Please file a bug report with Jekyll or ask for help on their talk forum.

Your permalink is likely the problem too. Try permalink: / instead.


M3-4241 Update: password validation

Description
Update our password validation to match the API.
Includes:

Remove any validation for passwords based on length and character classes
Remove any text in the UI that implies that we're checking these
Include old zxvcbn logic and use complexity to validate passwords

Seems to be working as intended for newly-created Linodes, rebuilding Linodes, etc. Is it expected for this not to apply when changing a root password though? In that case, it accepts ones graded "Weak"

Good catch, I don't think we were ever doing client validation in this situation


MPI_COMM_SELF handling issues in AMPI

Original issue: https://charm.cs.illinois.edu/redmine/issues/1240

AMPI creates 1d chare arrays for each communicator, except for MPI_COMM_SELF, for which we avoid creating a chare (to save memory?) and instead handle as a corner case everywhere.
So far we have found issues in Intercomm_create and Comm_split, but all of the communicator and topology routines should be reviewed to make sure they handle it correctly.
This is blocking work on Intercomm_create and Comm_create_group.

Original date: 2016-11-08 23:51:41

The MPI_Comm_split failures have been observed in XPACC.

Original date: 2017-06-19 04:37:17

MPI_Comm_dup with a MPI_COMM_SELF communicator to duplicate also is not handled correctly (ampi::splitReduce expected a split contribution from every rank!). This breaks some functionality in ROMIO.

Original date: 2017-06-27 20:32:47

This patch has some improvements, but is still broken for the ROMIO MPI_COMM_SELF MPI_Comm_dup case: ~~https://charm.cs.illinois.edu/gerrit/#/c/2570/~~ https://github.com/UIUC-PPL/charm/commit/7041222a094d7caf0ab9cb38869971bf04ea0a51

Original date: 2017-06-27 20:38:00

To fix this properly, we might have to implement full support for MPI_COMM_SELF in AMPI. For example, how would MPI_Comm_set_attr() work on MPI_COMM_SELF without this support?

Original date: 2017-07-06 15:53:29

Yes, I'm working on the full support now

Original date: 2017-07-06 17:59:29

~~https://charm.cs.illinois.edu/gerrit/#/c/2753/~~ https://github.com/UIUC-PPL/charm/commit/498a9f717ecfcadd6d2c001524944593609cd20c

Original date: 2017-07-06 17:59:47

It would probably be good to get this into 6.8.0

Original date: 2017-07-06 23:32:22

Related: make collectives optimizations for MPI_COMM_SELF apply to all comms with size of 1: ~~https://charm.cs.illinois.edu/gerrit/#/c/2755/~~ https://github.com/UIUC-PPL/charm/commit/0258b73eaf92fa40cac0a4c7e151671e6f98d613


Change font color

Is this still possible? I've tried changing your uidatepicker variable but to no avail.

Hi there!
Unfortunately, the UIDatePicker used internally does not allow to change the appearance of it.
If you want to use a more customizable date picker control, you can use the RMActionController to present it in a similar manner 🙂.
Best regards,
Roland


Incremental training

Issue #, if available: related to #2244
Description of changes: POC to support re-training of already-trained models. Note that for how this works internally, the structure of the network will be validated at parameter-copy-time.
Summary of the changes:

estimators can opt-in for the feature by implementing train_from(predictor, ...)
network initialization moved out of the Trainer class; now it’s done by copying the parameters of the given model (if any, otherwise same initialization as before)
PyTorchPredictor now gets the full LightningModule (so that some parameter naming headache is solved), which in turn now must implement forward an be @validated()
add access to the input model channel in gluonts.shell (work in progress) as described in https://docs.aws.amazon.com/sagemaker/latest/dg/incremental-training.html

Example usage:
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.mx import DeepAREstimator, Trainer

estimator = DeepAREstimator(
    freq="1H",
    prediction_length=24,
    trainer=Trainer(epochs=5),
)

dataset = get_dataset("m4_hourly")

print("Training the first time...")
predictor = estimator.train(dataset.train)

# get some new data...
# ok, not really

print("Training the second time...")
other_predictor = estimator.train_from(predictor, dataset.train)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

This results from removing the trailing **kwargs from train. However, it's unused: should we just update the tests and mark this as breaking? Maybe in a separate PR
2022-09-13T11:22:58.7320002Z >       predictor = estimator.train(dataset_train, num_workers=None)
2022-09-13T11:22:58.7320252Z E       TypeError: train() got an unexpected keyword argument 'num_workers'

This results from removing the trailing **kwargs from train. However, it's unused: should we just update the tests and mark this as breaking? Maybe in a separate PR
2022-09-13T11:22:58.7320002Z >       predictor = estimator.train(dataset_train, num_workers=None)
2022-09-13T11:22:58.7320252Z E       TypeError: train() got an unexpected keyword argument 'num_workers'

I think so. Probably also shows that we should disentangle data loading more.


Tests for build_x86_tests are failing in Travis

Tests for build_x86_tests are failing
According to Travis, test failures started in build_x86_tests (llvm40) with merging bf82eb3 on 2018 June 28.
Tested locally on Ubuntu 16.04 64bit with llvm40 (default), llvm39, and llvm38 have the following test failures:

x86

CVTDQ2PDv128m64_0_1
CVTDQ2PDv128m64_4_1
CMPXCHGm8r8al_3
CMPXCHGm16r16ax_3
CMPXCHGm32r32eax_3
Error: Lifted and native states did not match.
Error: Lifted and native stacks did not match.
Error: Memory access falls outside the valid range of the stack.

llvm50 has the following test failures:

x86

CVTDQ2PDv128m64_0_1
CVTDQ2PDv128m64_4_1
CMPXCHGm8r8al_3
CMPXCHGm16r16ax_3
CMPXCHGm32r32eax_3
PSHUFDv128v128i8_03_1
PSHUFDv128v128i8_47_1
PSHUFDv128v128i8_8b_1
PSHUFDv128v128i8_cf_1
PSLLDQv128v128i8_1
Error: Lifted and native states did not match.
Error: Lifted and native stacks did not match.
Error: Memory access falls outside the valid range of the stack.

amd64

CVTPS2DQv128m128_2
CVTPS2DQv128v128_2
CVTTPS2DQv128m128_2
CVTTPS2DQv128v128_2
PCMPEQBv128v128_2
PCMPEQBv128m128_2
PCMPEQWv128v128_2
PCMPEQWv128m128_2
PCMPEQDv128v128_2
PCMPEQDv128m128_2
PSHUFDv128v128i8_03_1
PSHUFDv128v128i8_47_1
PSHUFDv128v128i8_8b_1
PSHUFDv128v128i8_cf_1
PSLLDQv128v128i8_1
Error: Lifted and native states did not match.

Additional errors in llvm50 tests appear to be prior to merging bf82eb3 so LLVM 5.0 support is likely a separate issue.

Issue was introduced when attempting to eliminate the double indirection noted in Arch.cpp#L423. Commenting out the removal of the alloca instruction in Arch.cpp#L460-L475 fixes the tests.
Note that make build_x86_tests will no longer compile with the change above unless you revert Lifter.cpp, LoadRegAddress() back to:
return new llvm::LoadInst(FindVarInFunction(block->getParent(), reg_name), "", block);

and Util.cpp, LoadProgramCounterRef() back to:
llvm::IRBuilder<> ir(block);
return ir.CreateLoad(FindVarInFunction(block->getParent(), "PC"));

and setting cond in Util.cpp, LoadBranchTaken() back to:
auto cond = ir.CreateLoad(
    ir.CreateLoad(FindVarInFunction(block->getParent(), "BRANCH_TAKEN")));

This is not a proposal to comment out or revert the elimination of the double indirection. It is only meant to highlight where the source of the test failures are so it can be addressed accordingly.

This is fixed in Linux.


deploy: remove extra prefix: csi-addons-

Unnecessary prefix: csi-addons- was added by mistake in default kustomization.yaml in below commit
https://github.com/csi-addons/kubernetes-csi-addons/commit/90c23d244b198b6df9d2350bd1ab80d8e0c1d784#diff-c006b1ef995eeeb57408cce42bd84db8dfc0cbe18df03222857665cebb1ac6fd

@Rakshith-R, remove it from here as well?
https://github.com/csi-addons/kubernetes-csi-addons/blob/3da396a242f247f8b983339a889dcaeee9c69c91/config/manager/kustomization.yaml#L9


Missing dependencies when building on 35.1.0 container images possible issues with opencv

Having issues compiling the latest jetson-utils (b4d2fef) when using the nvcr.io/nvidia/l4t-pytorch:r35.1.0-pth1.13-py3  container image. I am trying to build my own custom container using nvcr.io/nvidia/l4t-pytorch:r35.1.0-pth1.13-py3 as the base image.  As a sanity check I used  dustynv/jetson-inference:r35.1.0 container image. Both containers have similar issues when trying to build jetson-utils.
In the dustynv/jetson-inference:r35.1.0 container image I clone the latest version of jetson-utils and try to compile but I get the following errors:
In file included from /root/jetson-utils/codec/gstWebRTC.h:26,
                 from /root/jetson-utils/codec/gstWebRTC.cpp:23:
/root/jetson-utils/build/aarch64/include/jetson-utils/WebRTCServer.h:26:10: fatal error: libsoup/soup.h: No such file or directory
   26 | #include <libsoup/soup.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/jetson-utils.dir/build.make:279: CMakeFiles/jetson-utils.dir/codec/gstWebRTC.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /root/jetson-utils/codec/gstWebRTC.h:26,
                 from /root/jetson-utils/codec/gstEncoder.cpp:24:
/root/jetson-utils/build/aarch64/include/jetson-utils/WebRTCServer.h:26:10: fatal error: libsoup/soup.h: No such file or directory
   26 | #include <libsoup/soup.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
In file included from /root/jetson-utils/codec/gstWebRTC.h:26,
                 from /root/jetson-utils/codec/gstDecoder.cpp:24:
/root/jetson-utils/build/aarch64/include/jetson-utils/WebRTCServer.h:26:10: fatal error: libsoup/soup.h: No such file or directory
   26 | #include <libsoup/soup.h>

Which I fixed with:
apt-get install libsoup-gnome2.4-dev
But then I get:
In file included from /root/jetson-utils/codec/gstEncoder.cpp:24:
/root/jetson-utils/codec/gstWebRTC.h:30:10: fatal error: gst/webrtc/webrtc.h: No such file or directory
   30 | #include <gst/webrtc/webrtc.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/jetson-utils.dir/build.make:4363: CMakeFiles/jetson-utils.dir/codec/gstEncoder.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /root/jetson-utils/codec/gstWebRTC.cpp:23:
/root/jetson-utils/codec/gstWebRTC.h:30:10: fatal error: gst/webrtc/webrtc.h: No such file or directory
   30 | #include <gst/webrtc/webrtc.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/jetson-utils.dir/build.make:4391: CMakeFiles/jetson-utils.dir/codec/gstWebRTC.cpp.o] Error 1
In file included from /root/jetson-utils/codec/gstDecoder.cpp:24:
/root/jetson-utils/codec/gstWebRTC.h:30:10: fatal error: gst/webrtc/webrtc.h: No such file or directory
   30 | #include <gst/webrtc/webrtc.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/jetson-utils.dir/build.make:4349: CMakeFiles/jetson-utils.dir/codec/gstDecoder.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:205: CMakeFiles/jetson-utils.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

I have tried fixing this by running:
apt-get install libgstreamer-plugins-bad1.0-dev
Running this fixes the issue but  the command itself exits with the following error(this error would stop my docker build):
Errors were encountered while processing:
 /tmp/apt-dpkg-install-Jy26cS/0-libopencv_4.5.4-8-g3e4c170df4_arm64.deb
 /tmp/apt-dpkg-install-Jy26cS/2-libopencv-dev_4.5.4-8-g3e4c170df4_arm64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Although the command fails, it allows me to move forward in the compiling
when I try to make again I get the following error:
/root/jetson-utils/codec/gstWebRTC.cpp:29:10: fatal error: json-glib/json-glib.h: No such file or directory
   29 | #include <json-glib/json-glib.h>

Which I try to fix with:
apt-get install libjson-glib-dev
Which results in the following error:
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libgstreamer-plugins-bad1.0-dev : Depends: libopencv-dev (>= 2.3.0) but it is not going to be installed
 libjson-glib-dev : Depends: gir1.2-json-1.0 (= 1.4.4-2ubuntu2) but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

I am wondering if this is due to Jetpack 5.0.2 including OpenCV 4.5.4 but the docker image manually installs OpenCV 4.5.0 (with CUDA).
Should be I be using an older version of jetson-utils? Is that the reason jetson-inference locks jetson-utils at 8b6c5ca?
Any guidance is greatly appreciated.

Is that the reason jetson-inference locks jetson-utils at 8b6c5ca?

Yes, I would use that commit of jetson-utils that the master branch of jetson-inference is currently on.   The dev branch of jetson-inference uses jetson-utils master and I still need to sanity check the container build for that (although that dockerfile / CMakePreBuild.sh has been updated with some of these dependencies)

I see, the dev branch of jetson-inference has included many of the missing requirements. Since I am not using the new features I will use the commit of jetson-utils that the master branch of jetson-inference is currently on (8b6c5ca).
Now I am running into the following issue when running CMake:
-- detecting Python 3.7...
-- found Python version:  3.8 (3.8.10)
-- found Python include:  /usr/include/python3.8
-- found Python library:  /usr/lib/aarch64-linux-gnu/libpython3.8.so
-- CMake module path:  /jetson-utils/cuda;/jetson-utils/python/bindings
-- NumPy ver. 1.24.2 found (include: /usr/local/lib/python3.8/dist-packages/numpy/core/include)
-- found NumPy version:  1.24.2
-- found NumPy include:  /usr/local/lib/python3.8/dist-packages/numpy/core/include
CMake Error at cuda/FindCUDA.cmake:1802 (add_library):
  add_library cannot create target "jetson-utils-python-38" because another
  target with the same name already exists.  The existing target is a shared
  library created in source directory "/jetson-utils/python/bindings".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  python/bindings/CMakeLists.txt:57 (cuda_add_library)

CMake Error at python/bindings/CMakeLists.txt:59 (target_link_libraries):
  Attempt to add link library "jetson-utils" to target
  "jetson-utils-python-38" which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.

CMake Error at python/bindings/CMakeLists.txt:62 (target_link_libraries):
  Attempt to add link library "npymath" to target "jetson-utils-python-38"
  which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.

I am not sure what the problem is now, I saw a similar issue in the nvidia developer forums but that didn't help.
The Dockerfile I am running is very similar to that of the Dockerfile in jetson-inference master:
FROM nvcr.io/nvidia/l4t-pytorch:r35.1.0-pth1.13-py3

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL /bin/bash

RUN apt-get update

# install development packages and gstreamer
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    cmake \
    nano \
    lsb-release \
    gstreamer1.0-tools \
    gstreamer1.0-libav \
    gstreamer1.0-rtsp \
    gstreamer1.0-plugins-rtp \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly \
    && rm -rf /var/lib/apt/lists/*

# pip dependencies for pytorch-ssd
RUN pip3 install --upgrade pip
RUN pip3 install --upgrade Cython && \
    pip3 install boto3 pandas tensorboard

RUN apt-get update
RUN apt-get install -y dialog
RUN apt-get install -y libglew-dev glew-utils libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev
RUN apt-get install -y qtbase5-dev

# install OpenCV (with CUDA)
ARG OPENCV_URL=https://nvidia.box.com/shared/static/2hssa5g3v28ozvo3tc3qwxmn78yerca9.gz
ARG OPENCV_DEB=OpenCV-4.5.0-aarch64.tar.gz

COPY scripts/opencv_install.sh /tmp/opencv_install.sh
RUN cd /tmp && ./opencv_install.sh ${OPENCV_URL} ${OPENCV_DEB}

# Install jetson-utils requirements

# RUN TZ=Etc/UTC   \
#     apt-get update && \
#     apt-get install -y libsoup-gnome2.4-dev \  ***add
#     libgstrtspserver-1.0-dev  \
#     libglew-dev \
#     libjson-glib-dev *** add

##################################################
## CHECKPOINT: Requirements to run jetson-utils ##
##################################################

# Install jetson-utils-qm

RUN git clone https://github.com/dusty-nv/jetson-utils.git && \
    cd jetson-utils &&   \
    git checkout 8b6c5ca4b2b52d51f5415ce033ee52563ddf3f2c && \
    mkdir build && cd build &&  \
    cmake .. &&   \
    make -j$(nproc) &&   \
    make install

I see, the dev branch of jetson-inference has included many of the missing requirements. Since I am not using the new features I will use the commit of jetson-utils that the master branch of jetson-inference is currently on (8b6c5ca).
Now I am running into the following issue when running CMake:
-- detecting Python 3.7...
-- found Python version:  3.8 (3.8.10)
-- found Python include:  /usr/include/python3.8
-- found Python library:  /usr/lib/aarch64-linux-gnu/libpython3.8.so
-- CMake module path:  /jetson-utils/cuda;/jetson-utils/python/bindings
-- NumPy ver. 1.24.2 found (include: /usr/local/lib/python3.8/dist-packages/numpy/core/include)
-- found NumPy version:  1.24.2
-- found NumPy include:  /usr/local/lib/python3.8/dist-packages/numpy/core/include
CMake Error at cuda/FindCUDA.cmake:1802 (add_library):
  add_library cannot create target "jetson-utils-python-38" because another
  target with the same name already exists.  The existing target is a shared
  library created in source directory "/jetson-utils/python/bindings".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  python/bindings/CMakeLists.txt:57 (cuda_add_library)

CMake Error at python/bindings/CMakeLists.txt:59 (target_link_libraries):
  Attempt to add link library "jetson-utils" to target
  "jetson-utils-python-38" which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.

CMake Error at python/bindings/CMakeLists.txt:62 (target_link_libraries):
  Attempt to add link library "npymath" to target "jetson-utils-python-38"
  which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.

I am not sure what the problem is now, I saw a similar issue in the nvidia developer forums but that didn't help.
The Dockerfile I am running is very similar to that of the Dockerfile in jetson-inference master:
FROM nvcr.io/nvidia/l4t-pytorch:r35.1.0-pth1.13-py3

ENV DEBIAN_FRONTEND=noninteractive
ENV SHELL /bin/bash

RUN apt-get update

# install development packages and gstreamer
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    cmake \
    nano \
    lsb-release \
    gstreamer1.0-tools \
    gstreamer1.0-libav \
    gstreamer1.0-rtsp \
    gstreamer1.0-plugins-rtp \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly \
    && rm -rf /var/lib/apt/lists/*

# pip dependencies for pytorch-ssd
RUN pip3 install --upgrade pip
RUN pip3 install --upgrade Cython && \
    pip3 install boto3 pandas tensorboard

RUN apt-get update
RUN apt-get install -y dialog
RUN apt-get install -y libglew-dev glew-utils libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev
RUN apt-get install -y qtbase5-dev

# install OpenCV (with CUDA)
ARG OPENCV_URL=https://nvidia.box.com/shared/static/2hssa5g3v28ozvo3tc3qwxmn78yerca9.gz
ARG OPENCV_DEB=OpenCV-4.5.0-aarch64.tar.gz

COPY scripts/opencv_install.sh /tmp/opencv_install.sh
RUN cd /tmp && ./opencv_install.sh ${OPENCV_URL} ${OPENCV_DEB}

# Install jetson-utils requirements

# RUN TZ=Etc/UTC   \
#     apt-get update && \
#     apt-get install -y libsoup-gnome2.4-dev \  ***add
#     libgstrtspserver-1.0-dev  \
#     libglew-dev \
#     libjson-glib-dev *** add

##################################################
## CHECKPOINT: Requirements to run jetson-utils ##
##################################################

# Install jetson-utils-qm

RUN git clone https://github.com/dusty-nv/jetson-utils.git && \
    cd jetson-utils &&   \
    git checkout 8b6c5ca4b2b52d51f5415ce033ee52563ddf3f2c && \
    mkdir build && cd build &&  \
    cmake .. &&   \
    make -j$(nproc) &&   \
    make install

-- detecting Python 3.7...

Hi @franciscoQM, it shouldn't attempt to find Python 3.7 on JetPack 5.x because of this check:
https://github.com/dusty-nv/jetson-utils/blob/8b6c5ca4b2b52d51f5415ce033ee52563ddf3f2c/python/CMakeLists.txt#L8
However it appears that when jetson-utils gets built standalone, LSB_RELEASE_CODENAME is never set.
Please try adding this code block to jetson-utils/CMakeLists.txt:50 so that it gets set in the standalone build.

Thanks that was the problem. Instead of running CMake like:
 cmake ..

I ran:
 cmake -DLSB_RELEASE_CODENAME="focal" ..


Titanic on GCE

This sample shows how to generate a classifier based on the Titanic dataset using scikit-learn and on a GCE instance.

@happyhuman : Don't forget to link to this guide from the top-level README

@happyhuman : One thing that just occurred to me - you should show how to stop and/or delete the instance at the end of the guide. Otherwise a user could easily forget to do so.


Matrix store and olm account is not loaded.

Hello! I have a problem related to e2ee.
I log in using an access token. I followed the instructions and edited the config.json and compose.yaml. I initially ran the Docker image using sudo docker compose without any problems, but after I tried to add e2ee, I found the following error message:
matrix-stt-bot  | 2023-12-04 09:34:41,890 - INFO - Bot started!
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - Login via access_token
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - start import_keys process, this may take a while...
matrix-stt-bot  | Traceback (most recent call last):
matrix-stt-bot  |   File "/app/src/bot.py", line 632, in <module>
matrix-stt-bot  |     asyncio.run(main())
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
matrix-stt-bot  |     return runner.run(main)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
matrix-stt-bot  |     return self._loop.run_until_complete(task)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
matrix-stt-bot  |     return future.result()
matrix-stt-bot  |            ^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/app/src/bot.py", line 613, in main
matrix-stt-bot  |     await bot.import_keys()
matrix-stt-bot  |   File "/app/src/bot.py", line 537, in import_keys
matrix-stt-bot  |     resp = await self.client.import_keys(
matrix-stt-bot  |                  ^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/site-packages/nio/client/base_client.py", line 115, in inner
matrix-stt-bot  |     raise LocalProtocolError("Matrix store and olm account is not loaded.")
matrix-stt-bot  | nio.exceptions.LocalProtocolError: Matrix store and olm account is not loaded.

Thanks for the quick reply.
I tried to log in with user_id and password, but it was unsuccessful because of the server configuration. There is OIDC authorization and i can't use user_id and password.

Create a matrix account from matrix.org or other providers.

I can't create such accounts because I have self-hosted server with disabled federation. Maybe you can suggest what I can change to make e2ee work using access_token?

Sorry, i use matrix-nio library to develop bot. It has poorly document and i can't find a way to achive that.

Okay, thank you. I'll continue my search, and I'll let you know if I find the answer.

@Setom29 Please use the latest version and follow the instructions below to make access_token work in E2EE room.

Open a private browser window and visit your element web url, then login to your bot account via OIDC.
Get current session id and access_token from element web.
Close private browser window and open a new one, revisit your element web url and login bot account via OIDC.
Send some text in E2EE room from your normal account, if you can read the decrypted content from private browser window(bot account), then export E2EE room keys from element web(bot account).
Edit your config.json (without the <>), then docker compose up

{
    "homeserver": "https://xxxx.xxxxx",
    "user_id": "@xxxxx:xxxx.xxxxx",
    "access_token": "<access_token>",
    "device_id": "<session id>",
    "model_size": "base",
    "room_id": "<some non exist room id>",
    "import_keys_path": "element-keys.txt",
    "import_keys_password": "<E2EE room keys passphrase>"
}

Some errors can be ignored.
Wait some seconds to let the bot do some syncing, then Ctrl+C stop the container.

Remove import_keys_path and import_keys_password options,  and set a correct room_id or remove it if you hope the bot to work in the rooms it is in.

Finally, launch the container in detach mode

docker compose up -d

@hibobmaster , thank you. Session id is the thing that changes over time, as I conclude from it's name, which means that my bot is not sustainable without man's intervention, which makes it looks more like a script, not a bot...
As I can see in the docs it is not, indeed:

Megolm sessions may not be reused indefinitely. The parameters which define how often a session should be rotated are defined in the m.room.encryption state event of a room.

@alexander-potemkin  You can understand in this way. From element web, you can see how many sessions are there. All of them work unless you manually logout it. (Each session_id has a correspond access_token)

The bot is sustainable without man's intervention after you properly set it up.
Can you try https://github.com/hibobmaster/matrix-stt-bot/issues/4#issuecomment-1840398688 ?

Thank you and I'm sorry to say that it still doesn't work for me... :/
I followed all of steps down to the dot in them, but the only room it works at - it's unencrypted initial room, that is also happened to be specified in the config.
To make sure it's not some room keys issue, I created a new encrypted room, added my bot there, here is what I have on the console:
2023-12-07 21:16:18,569 - INFO - Joined !ycxAopEWqQVtJssZhx:mydomain.com
2023-12-07 21:16:18,714 - INFO - Joined !ycxAopEWqQVtJssZhx:mydomain.com
Olm event doesn't contain ciphertext for our key
Received a undecryptable Megolm event from a device with no Olm sessions: @alex:mydomain.com XFNMXWGKIM
Error decrypting megolm event, no session found with session id tzb+slR8NejmneCYjp29CQKGMeqpyg4CJK5MA5gU9QA for room !ycxAopEWqQVtJssZhx:mydomain.com
2023-12-07 21:16:32,243 - decryption_failure - ERROR - Failed to decrypt message: $Jki6KordOd2Int-cgeVWJYilGn7N0rw9ywep2WedlBE from @alex:mydomain.com             in !ycxAopEWqQVtJssZhx:mydomain.com
Please make sure the bot current session is verified
2023-12-07 21:16:32,243 - ERROR - Failed to decrypt message: $Jki6KordOd2Int-cgeVWJYilGn7N0rw9ywep2WedlBE from @alex:mydomain.com             in !ycxAopEWqQVtJssZhx:mydomain.com
Please make sure the bot current session is verified

And yes - bot's session(s) are all verified. Triple checked that from the browser (logged in as a bot) and from 'regular user' browser.

Thank you very much. I followed all of the steps you provided, but I'm still getting the error:
Received a undecryptable Megolm event from a device with no Olm sessions: @alex:mydomain.com XFNMXWGKIM
Error decrypting megolm event, no session found with session id Awrm7TMzGvFPRIqmB1Sx1a1RPW9ENEF9YXdqgYpOArw for room !GnyRxyvAEIRTVnFpJJ:mydomain.com
2023-12-09 12:21:34,712 - decryption_failure - ERROR - Failed to decrypt message: $gTT1hMxBLbmHG3yKD7R-DF28K6yRvA2nqxTM4oPh4vg from @alex:mydomain.com             in !GnyRxyvAEIRTVnFpJJ:mydomain.com
Please make sure the bot current session is verified
2023-12-09 12:21:34,712 - ERROR - Failed to decrypt message: $gTT1hMxBLbmHG3yKD7R-DF28K6yRvA2nqxTM4oPh4vg from @alex:mydomain.com             in !GnyRxyvAEIRTVnFpJJ:mydomain.com
Please make sure the bot current session is verified

I did a few times, resetting the keys, device ids, etc and the issue still persist.
Double checked - all my bot sessions are verified.
Is there probably some ways to understand what exactly is missing? What is the session id that is referred in the error message: Awrm7TMzGvFPRIqmB1Sx1a1RPW9ENEF9YXdqgYpOArw?
Or any other help! Would be much appreciated!

I use login & password for Element client and token for the bot. Guess I can’t login with token on Element, unless I’m missing something?

I mean matrix-stt-bot. You can use either password or access token in config.json but not both.
If you use password, use a device_id different from element web session list and follow the instructions in readme.
If you use access token, use the instructions above.
Notice that some errors can be ignored, make sure always send a voice to test.

I use access token, follow instructions and only sending voice messages… 🙁

Can you share me your config.json and compose.yaml with redacted info.

I'm running bot.py directly from IDE, to catch errors / exceptions and follow the logic (with all files created just next to the script in src folder).
Please, find information attached: https://gist.github.com/alexander-potemkin/17d9c86778e54372714c88086a96bc18
Please, let me know once you access / download it, so that I can remove it.

I get the same error Olm event doesn't contain ciphertext for our key which is not occured in my personal server or matrix.org server

I got it work finally.

Oh, wow! Thanks a lot! I shall test it all very thoroughly and get back to you in a little while!

Unfortunately, that didn't work... =(
I've tried it on another self-hosted server, but same thing.
One thing - sessions keys dump is getting bigger with every new session / chat in a new room and from the description I can see that they gives access to the past messages, not to the future messages. And Element client generates those sessions keys as required, which makes me think that something is wrong about the way bot handles that data.
I understand that nio documentation is almost absent and Matrix protocol is not so easy to dig into, but it feels like bot shall handle device verification and sessions keys on his own, otherwise something get lost somewhere.
btw, doesn't it feels weird, that with session keys old messages can't be decrypted, despite the fact that those keys are created to enable old messages decrypt?

As for matrix bot, we don't care old messages. What i see is when the bot join the new rooms without man's intervention, it works at my end.
If you want the bot handle device verification at the moment(emoji verification), you should login via password.

Unfortunately, that didn't work

Can you try logout your current normal account, then relogin, export its E2EE room keys and let bot import again?

@hibobmaster , I appreciate your support and prompt responses - that was really of importance for me.
During the digging into the things I found myself rewriting the functionality with simple-matrix-bot-lib and matrix-commander and I made it finally work.
During my journey I was facing similar issues, until I dropped Element client completely and started using matrix-commander to obtain token - then it was back to normal.
Once again - thank you a lot for your product, time and efforts - I borrowed a few things from there and your support was of really great help!


sniffer cleanup

Some suggested cleanup in the sniffers .

OK,  the way the format detection works right now has two main goals:

Avoid reading the whole SBOM into memory
It assumes the SBOM format can generally be found at the top of the file so it tries to optimize for less I/O by catching it early.

The proposed patch breaks # 2 above. For example, if I point the sniffer it to the Kubernetes SBOM which is SPDX, it will read the whole 18 MB needlessly and read it again with the SPDX parser to find the format.
Do you want to rework the patch to maybe unify the reads and open the PR against main? There are some more comments about string conversion, etc but let's go through them in a PR to protobom/main


docker-compose: no declaration was found in the volumes section

Hello.
Trying to spin dvol, but encounter problems:
docker-compose.yml:

version: "2"
services:
  web:
    container_name: web-dev
    build:
      context: .
      dockerfile: services/development/web/Dockerfile
    links:
      - db
      - cache
    depends_on:
      - db
      - cache

  queue:
    container_name: queue-dev
    image: queue:latest
    entrypoint: python2.7 manage.py rqworker
    links:
      - db
      - cache
    depends_on:
      - db
      - cache

  db:
    container_name: db-dev
    image: db:latest
    volumes:
      - "db:/var/lib/postgresql/data"
    volume_driver: dvol

  cache:
    container_name: cache-dev
    image: redis:latest

spinning docker-compose:
$ docker-compose up
ERROR: Named volume "db:/var/lib/postgresql/data:rw" is used in service "db" but no declaration was found in the volumes section.

$ docker volume ls
list dvol: VolumeDriver.List:

  <title>404 - No Such Resource</title>

No Such Resource

No such child resource.

DRIVER              VOLUME NAME
local               00894d48cfdd9d2493378e0b12413294756a20142cc8921b137881c5c6aecfc4
local               010b05964a81ad4219a9f3c05d55a24f9fefec4644a360498fb3ba064470290c

$ docker info
Containers: 32
 Running: 1
 Paused: 0
 Stopped: 31
Images: 195
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-254:1-1054363-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 17.87 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.271 GB
 Metadata Space Used: 20.53 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.127 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.133 (2016-08-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local dvol
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.2-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.33 GiB
Name: thingol
ID: CFYC:6PZ7:X5EQ:U6CL:UVRT:NAYB:FART:Y3HS:Z4UE:UPRU:7EM6:XP25
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 <IP_ADDRESS>/8

I'm not using compose but am having the same output from
# docker volume ls

list dvol: VolumeDriver.List:

  <title>404 - No Such Resource</title>

No Such Resource

No such child resource.

DRIVER              VOLUME NAME
local               006b52bbe8e4f33a56ab535fcf56fa2f30bf0668121519071f60b768e45e0b62
local               69f6e1bc35c3421153546c2effcf037f403a6eb44cc0ef10e4ff447fa5235076
local               8b6da2ceda666d70b67d7ae539d7073a6a2dbb4bb43a5d5e211dd54e35fe7b33
local               b823b18ed7954b65cf751c6d9a73446750070771006f9d4a33887551bf54182c

It was working fine surviving reboots.  The only subsequent docker commands I executed were to init swarm and create an overlay network.

@Edke do you encontered a solution?

@tiagotakahashi sorry no, didnt find solution, not using dvol

Have you try to make relative links (eg. ./)?

Same problem:
version: '3'

services:

  web:
    image: jonathanstray/cjworkbench
    container_name: cjw-web
    volumes:
      - data:/cjworkbench
    ports:
      - "8000:8000"
    command: /cjworkbench/start-prod.sh

gives
ERROR: Named volume "data:/cjworkbench:rw" is used in service "web" but no declaration was found in the volumes section.

Ah, I see, in both cases  above the problem is we  have no volumes section in docker-compose.yml, just as the error message suggests: https://docs.docker.com/compose/compose-file/#volume-configuration-reference

ERROR: Named volume "data:/cjworkbench:rw" is used in service "web" but no declaration was found in the volumes section.
use ./data instead of data and make sure directory ./data exists.

When you use a named volume, like "data:/cjworkbench:rw" you must declare it at the docker-compose file (at the end as a good pratice). Here's an example:
volumes:
data:
driver: local
If you just want to use the path "data" you have to specify all address of it. Here's an exemple:
./data:cjworkbench:rw/

I had the same problem and I just follow the instructions from @jstray adding the volumes section at the end of my docker-compose file.
volumes:     volume-name:         driver: local  # is already local by default

I also get the same problem. I am not sure how to debug it, but this is the yml

version: '2'

services:
    haproxy:
        image: haproxy:1.7
        ports:
            # stats
            - "8036:8036"

            # splash
            - "8050:8050"
        links:
            - splash0
            - splash1
            - splash2
        volumes:
            - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro

    splash0:
        image: scrapinghub/splash:3.0
        command: --max-timeout 3600 --slots 5 --maxrss 3000 --verbosity 1
        expose:
            - 8050
        mem_limit: 4200m
        memswap_limit: 5400m
        restart: always
        links:
            - tor
        volumes:
            - ./proxy-profiles:/etc/splash/proxy-profiles:ro
            - ./filters:/etc/splash/filters:ro

    splash1:
        image: scrapinghub/splash:3.0
        command: --max-timeout 3600 --slots 5 --maxrss 3000 --verbosity 1
        expose:
            - 8050
        mem_limit: 4200m
        memswap_limit: 5400m
        restart: always
        links:
            - tor
        volumes:
            - ./proxy-profiles:/etc/splash/proxy-profiles:ro
            - ./filters:/etc/splash/filters:ro

    splash2:
        image: scrapinghub/splash:3.0
        command: --max-timeout 3600 --slots 5 --maxrss 3000 --verbosity 1
        expose:
            - 8050
        mem_limit: 4200m
        memswap_limit: 5400m
        restart: always
        links:
            - tor
        volumes:
            - ./proxy-profiles:/etc/splash/proxy-profiles:ro
            - ./filters:/etc/splash/filters:ro

    tor:
        image: jess/tor-proxy
        expose:
            - 9050
        logging:
            driver: "none"
        restart: always

I've tried adding this code below at the end:
volumes:
    filters:
        driver: local
    proxy-profiles:
        driver: local

It doesn't work

Using ./xxxx instead of xxxx solved this issue for me.

I create a volume to my postgres container, but I cant access to it for permission ... Someone can help me?

Hi @Edke ,
You should add volume section in your .yml file, please refer below code:
version: "2"
services:
web:
container_name: web-dev
build:
context: .
dockerfile: services/development/web/Dockerfile
links:
- db
- cache
depends_on:
- db
- cache
queue:
container_name: queue-dev
image: queue:latest
entrypoint: python2.7 manage.py rqworker
links:
- db
- cache
depends_on:
- db
- cache
db:
container_name: db-dev
image: db:latest
volumes:
- "db:/var/lib/postgresql/data"
volume_driver: dvol
cache:
container_name: cache-dev
image: redis:latest
volumes:
db:

Hey everyone, dvol is no longer maintained. You might try https://github.com/dotmesh-io/dotmesh instead.

@prat3ik  I wonder where you declare the path to "db" ?

Try giving absolute path it resolves for me!

You can use relative path but make sure you prepend ./
So data/video/:/mnt/video/:ro is not working but ./data/video/:/mnt/video/:ro is OKAY.

If you want to bind a docker container folder with a host folder (like usual) then use bind instead of volume. I give you an example for postgres:
services:
  db:
    image: postgres
    volumes:
      - type: bind
        source: /host_folder/postgres_data
        target: /var/lib/postgresql/data/

Here is the official documentation example with static.

Using ./xxxx instead of xxxx solved this issue for me.

Thank you K-marad, It's worked for me, like..
api → ./api
in .yml.

I've tried adding this code below at the end:
volumes:
    filters:
        driver: local
    proxy-profiles:
        driver: local

It doesn't work

Add it at the top

Hi guys, I have something here thatmight lead to explanation.
Here is an example I just tried, using only Docker, no docker-compose :
jbl@poste-devops-typique:~/.tungunska/.awx/provision$ docker run -itd --name cerbotiii --restart unless-stopped --entrypoint sh -e path=$tungunska_path -v $data_path:$tungunska_path certbot/certbot
docker: Error response from daemon: create ./data/certbot: "./data/certbot" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.

I here mention the env. :
export data_path=./data/certbot
export tungunska_path=/etc/letsencrypt/live

jbl@poste-devops-typique:~/.pegasus/.awx/provision$ echo "docker run -itd --name cerbotiii --restart unless-stopped --entrypoint sh -e path=$pegasus_path -v $data_path:$pegasus_path certbot/certbot"
docker run -itd --name cerbotiii --restart unless-stopped --entrypoint sh -e path=/etc/letsencrypt/live -v ./data/certbot:/etc/letsencrypt/live certbot/certbot
jibl@poste-devops-typique:~/.pegasus/.awx/provision$

export domains=(google.io waffle.jp ebay.com)

I ran into this very issue myself. It is entirely possible to specify a docker volume name, rather than a path, in the short syntax of the volume definition. But if you use a volume name in such a way, you must also specify the name in a top-level volumes: mapping, so what you ultimate want is this:
version: "2"
services:
  web:
    container_name: web-dev
    build:
      context: .
      dockerfile: services/development/web/Dockerfile
    links:
      - db
      - cache
    depends_on:
      - db
      - cache

  queue:
    container_name: queue-dev
    image: queue:latest
    entrypoint: python2.7 manage.py rqworker
    links:
      - db
      - cache
    depends_on:
      - db
      - cache

  db:
    container_name: db-dev
    image: db:latest
    volumes:
      - "db:/var/lib/postgresql/data"
    volume_driver: dvol

  cache:
    container_name: cache-dev
    image: redis:latest

volumes:
  db:

Note that last top-level volumes setting which contains a mapping of a volume name that one of the services is expected to use.

db:
container_name: db-dev
image: db:latest
volumes:
- "db:/var/lib/postgresql/data"
volume_driver: dvol
your volume declaration should not be one string, try
db:/var/lib/postgresql/data
and add at the bottom of your yml file
volumes:
db:

I came here with a completely different error. But I found a solution anyway, thank you.
FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.

If you, like me, have not been helped by the setting of a relative path. Then if you look at the message above and do as you see it in the example, it all worked great. My example before and after:
before :
version: "3"
services:
  test_db:
    image: postgres:11.5-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: testdb
    volumes:
      - "./dbdata:/var/lib/postgresql/data"
# the same result if:
#   volumes:
#    - ./dbdata:/var/lib/postgresql/data

# volumes:
#    dbdata:

after :
version: "3"
services:
 test_db:
    image: postgres:11.5-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: testdb
    volumes:
      - dbdata:/var/lib/postgresql/data/

volumes:
  dbdata:

worked for me with absolute path

Using ./xxxx instead of xxxx solved this issue for me.

This solved the issue for me.

This is due to
container:
  volumes:
    <object>:<target>

It should be either:
container:
  volumes:
    ./<object>:<target>

or
container:
  volumes:
    /full/path/to/<object>:<target>

or
container:
  volumes:
    <name>:<target>
volumes:
  name:
    driver: local

If you want to use named volumes you should follow this one: https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose

Named volumes can be used by multiple services.
If u need to name it like : mongo-data:db/data
Declare it at the end of the docker-compose file like that :
`  node-app-data:
image: mongo
environment:
- MONGO_INITDB_ROOT_USERNAME=usr
- MONGO_INITDB_ROOT_PASSWORD=psswd
volumes:
- mongo-db:/data/db
volumes:
mongo-db:`

make sure that volume exist

ERROR: Named volume "data:/cjworkbench:rw" is used in service "web" but no declaration was found in the volumes section. use ./data instead of data and make sure directory ./data exists.

Example with SQLServer
version: '3'
services:
  server:
    container_name: "${LOCALSTACK_DOCKER_NAME}-server"
    image: mcr.microsoft.com/mssql/server:2017-CU31-ubuntu-18.04
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA="Y"
      - MSSQL_SA_PASSWORD=A12345!ab
    volumes:
      - "./data/yardi-mssql/data:/var/opt/mssql/data"
      - "./data/yardi-mssql/log:/var/opt/mssql/log"
      - "./data/yardi-mssql/secrets:/var/opt/mssql/secrets"
 volumes:
  data:

The folder `data' must exist within the same level

I had same error but i was able to solve it  following instructions from this thread. This is how my compose yml looks like now
version: '3.1'
services:
dbpostgresql:
image: "postgres:12"
container_name: dbpostgresql
networks:
- el-pgsql
ports:
- "5432:5432"
volumes:
- db:/var/lib/postgresql/data
environment:
POSTGRES_DB: loll
POSTGRES_PASSWORD: loll
app:
container_name: e-l
image: e-l
networks:
- el-pgsql
build: ./
ports:
- "8080:8080"
depends_on:
- dbpostgresql
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://dbpostgresql:5432/loll
- SPRING_DATASOURCE_USERNAME=loll
- SPRING_DATASOURCE_PASSWORD=loll
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
networks:
el-pgsql:
volumes:
db:


New parametr for getRoutes function

What problem does this feature solve?
Dynamic menu fetching, for example now I have all my protected routes nested in "Panel" route
What does the proposed API look like?
getRoutes(parentName?: string | symbol)

getRoutes() is intended to be very simple. Any custom transformations to be done with filter(), map(), or reduce() are to be done by the user for their specific application use cases.