OTA using Thread

Hello Community,
I want to implement OTA using thread on NRF52840dk (PCA10056) and nrf52840 dongle (PCA10059).
How am I suppose to start with OTA and multi broadcasting OTA using thread  ? (Documentation and Link)

Hi, unfortunately, we don't have an out-of-the-box example for OTA with OpenThread. For sure you'd need to enable mcuboot in your OT project and use it's API (mcuboot, flash_img) to write consecutive FW chunks recevied from the server (I'm not sure what FW delivery method are you aiming at).
Two examples of existing FOTA libraries that we currently have in Zephyr are updatehub and hawkbit. The former one uses CoAP to deliver FW, the latter uses HTTP.

Another option is SMP with UDP  as transport layer.

I have a working FOTA over OT that I'm using successfully for more than a year and half in a test bed of more than 10 FTD and SED devices. It's based on @mike-scott  work on https://github.com/foundriesio/dm-lwm2m , but not sure what its license is, and could we use it on Zephyr. If he confirm that it's OK to use it as a base I could push a PR for review here.

Hello Community,
Thanks for your reply.
I am trying to updatehub the sample demo for nrf52840dk and follow the below link.
https://docs.zephyrproject.org/latest/samples/subsys/mgmt/updatehub/README.html
the step follows.
1- build  mcuboot west build -b nrf52840dk_nrf52840 and flash using west flash.
2 -updatehub demo  west build -b  nrf52840dk_nrf52840 -- -DOVERLAY_CONFIG="prj.conf overlay-prj.conf " and flash using west flash --hex build/zephyr/app_signed.hex
3- run on terminal docker run -it -p 8080:8080 -p 5683:5683/udp --rm   updatehub/updatehub-ce:latest
I have a problem with updatehub: Cannot connect to UDP remote see the below image.

I have a working FOTA over OT that I'm using successfully for more than a year and half in a test bed of more than 10 FTD and SED devices. It's based on @mike-scott work on https://github.com/foundriesio/dm-lwm2m , but not sure what its license is, and could we use it on Zephyr. If he confirm that it's OK to use it as a base I could push a PR for review here.

Hello retfie,
can you please suggest to me how you do OTA with Openthread or you can provide some steps to follow( link or code)?
                                                                                Thanks
                                                                                Nehal

CC: @otavio

Hi @nvlsianpu , @utzig ,
I ran UpdateHub using WIFI, just to make sure everything is Ok.
Board: nrf52840dk_nrf52840
WIFI: ESP-8266 Shield
Zephyr: 89f6e24590 + my configs

Flash MCUboot

*** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.465,484] <inf> mcuboot: Starting bootloader
[00:00:00.471,862] <inf> mcuboot: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.482,147] <inf> mcuboot: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.492,553] <inf> mcuboot: Boot source: none
[00:00:00.498,016] <wrn> mcuboot: Failed reading image headers; Image=0
[00:00:00.505,523] <err> mcuboot: Unable to find bootable image

Flash initial image, version 1.0.0

*** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.426,116] <inf> mcuboot: Starting bootloader
[00:00:00.432,525] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.442,687] <inf> mcuboot: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.453,094] <inf> mcuboot: Boot source: primary slot
[00:00:00.465,362] <inf> mcuboot: Swap type: none
[00:00:00.671,875] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.679,687] <inf> mcuboot: Jumping to the first image slot

uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.251,190] <inf> wifi_esp: Waiting for interface to come up
[00:00:00.570,800] <inf> wifi_esp: ESP Wi-Fi ready
[00:00:00.577,209] <inf> main: UpdateHub sample app started
[00:00:00.577,209] <inf> main: Confirming the boot image
[00:00:00.581,604] <inf> main: Network disconnected
[00:00:06.222,717] <inf> main: Network connected
[00:00:06.222,747] <inf> main: Starting UpdateHub polling mode
[00:00:06.222,747] <inf> updatehub: SHA-256 verification on download and from flash
[00:00:08.439,239] <err> updatehub: Could not receive data
[00:01:08.982,330] <inf> updatehub: No update available
[00:02:09.403,137] <inf> updatehub: No update available
uart:~$ updatehub info
Unique device id: 3205bbf32cf69807
Firmware Version: 1.0.0
Product uid: e4d37cfe6ec48a2d069cc0bbb8b078677e9a0d8df3a027c4d8ea131130c4265f
UpdateHub Server: uhu.geo4cities.com
[00:03:09.756,195] <inf> updatehub: Probe metadata received
[00:04:57.230,255] <inf> updatehub: Firmware download complete
[00:04:58.535,461] <inf> updatehub: Image flashed successfully, you can reboot now

Reboot after receive new image, version 2.0.0

uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.260,772] <inf> mcuboot: Starting bootloader
[00:00:00.267,150] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.277,343] <inf> mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
[00:00:00.287,658] <inf> mcuboot: Boot source: none
[00:00:00.293,273] <inf> mcuboot: Swap type: test
[00:00:00.583,190] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.590,972] <inf> mcuboot: Jumping to the first image slot

uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.250,579] <inf> wifi_esp: Waiting for interface to come up
[00:00:00.570,251] <inf> wifi_esp: ESP Wi-Fi ready
[00:00:00.576,660] <inf> main: UpdateHub sample app started
[00:00:00.576,660] <inf> main: Confirming the boot image
[00:00:00.580,993] <inf> main: Network disconnected
uart:~$ updatehub info
Unique device id: 3205bbf32cf69807
Firmware Version: 1.0.0
Product uid: e4d37cfe6ec48a2d069cc0bbb8b078677e9a0d8df3a027c4d8ea131130c4265f

Details:

[00:00:06.222,747]  updatehub: SHA-256 verification on download and from flash
This means that we will run SHA-256 on downloaded data and on flash after stored all received frames

[00:04:58.535,461]  updatehub: Image flashed successfully, you can reboot now
This means that UpdateHub storaged a valid image at slot-2

[00:00:00.277,343]  mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
This confirms that, there is a valid image

Q: Why the system is not swamping image-2 on slot-1?

MCUboot does not start an upgrade unless it is requested by writing the proper metadata in the secondary slot. To do it you can build a padded image, with imgtool would be --pad and optionally use --confirm. Unfortunately this makes the image as large as the slot. A better alternative is to add support in your running firmware (updatehub library or something like that), that after downloading the full image, and optionally checking that it is valid, must call boot_request_upgrade from the dfu/mcuboot code. This should trigger a test upgrade, or pass the confirm flag to make it permanent.

MCUboot does not start an upgrade unless it is requested by writing the proper metadata in the secondary slot. To do it you can build a padded image, with imgtool would be --pad and optionally use --confirm. Unfortunately this makes the image as large as the slot. A better alternative is to add support in your running firmware (updatehub library or something like that), that after downloading the full image, and optionally checking that it is valid, must call boot_request_upgrade from the dfu/mcuboot code. This should trigger a test upgrade, or pass the permanent flag to make it, duh, permanent!

I think we are doing it, see below:
https://github.com/zephyrproject-rtos/zephyr/blob/89f6e24590cf9e9b66f123ee0ce2ae5818ce1aeb/subsys/mgmt/updatehub/updatehub.c#L928-L946
We print the message "Image flashed successfully, you can reboot now". It is the last line before step 3.

Yeah, the code looks correct; try building MCUboot will debug logging, there may be more helpful messages.

[00:00:00.260,986] <inf> mcuboot: Starting bootloader
[00:00:00.267,333] <inf> mcuboot: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.277,648] <inf> mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
[00:00:00.287,994] <inf> mcuboot: Boot source: none
[00:00:00.293,609] <inf> mcuboot: Swap type: test
[00:00:00.298,919] <dbg> mcuboot.swap_erase_trailer_sectors: erasing trailer; fa_id=2
[00:00:00.594,024] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.601,867] <inf> mcuboot: Jumping to the first image slot

Hi @nandojve
I want to use it as WIFI: nordic border -router.

[00:00:00.260,986] <inf> mcuboot: Starting bootloader
[00:00:00.267,333] <inf> mcuboot: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.277,648] <inf> mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
[00:00:00.287,994] <inf> mcuboot: Boot source: none
[00:00:00.293,609] <inf> mcuboot: Swap type: test
[00:00:00.298,919] <dbg> mcuboot.swap_erase_trailer_sectors: erasing trailer; fa_id=2
[00:00:00.594,024] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.601,867] <inf> mcuboot: Jumping to the first image slot

Hi @utzig , it is working now!
I've been doing something wrong when creating the package for UpdateHub.
uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.260,986] <inf> mcuboot: Starting bootloader
[00:00:00.267,303] <inf> mcuboot: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.277,587] <inf> mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
[00:00:00.287,902] <inf> mcuboot: Boot source: none
[00:00:00.293,518] <inf> mcuboot: Swap type: test
[00:00:00.501,190] <dbg> mcuboot.swap_erase_trailer_sectors: erasing trailer; fa_id=1
[00:00:00.592,926] <dbg> mcuboot.swap_status_init: initializing status; fa_id=1
[00:00:00.600,952] <dbg> mcuboot.boot_write_swap_info: writing swap_info; fa_id=1 off=0x66fd8 (0x72fd8), swap_type=0x2 image_num=0x0
[00:00:00.613,769] <dbg> mcuboot.boot_write_swap_size: writing swap_size; fa_id=1 off=0x66fd0 (0x72fd0)
[00:00:00.623,992] <dbg> mcuboot.boot_write_magic: writing magic; fa_id=1 off=0x66ff0 (0x72ff0)
[00:00:00.633,636] <dbg> mcuboot.swap_erase_trailer_sectors: erasing trailer; fa_id=2
[00:00:12.921,905] <dbg> mcuboot.boot_write_copy_done: writing copy_done; fa_id=1 off=0x66fe0 (0x72fe0)
[00:00:13.149,230] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:13.171,813] <inf> mcuboot: Jumping to the first image slot

uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.251,129] <inf> wifi_esp: Waiting for interface to come up
[00:00:00.570,983] <inf> wifi_esp: ESP Wi-Fi ready
[00:00:00.577,362] <inf> main: UpdateHub sample app started
[00:00:00.577,362] <inf> main: Confirming the boot image
[00:00:00.581,756] <inf> main: Network disconnected
uart:~$ updatehub info
Unique device id: 3205bbf32cf69807
Firmware Version: 2.0.0

@nehalrohit ,
I did the test using WIFI to make sure all infrastructure is OK using nrf52840dk_nrf52840 board, and it is.
We are working to enable network manager on UpdateHub and there is a branch that you can test WIFI:
https://github.com/UpdateHub/zephyr/tree/topic/uhu_thread_network_ut
In fact, that branch allows you to select any ETH/WIFI/MODEM.
I've been testing UpdateHub using https://docs.zephyrproject.org/latest/boards/shields/esp_8266/doc/index.html shield.
Currently, the https://docs.zephyrproject.org/latest/boards/shields/inventek_eswifi/doc/index.html have an UDP issue that we are working on it. The Inventek's have a far better performance.
We will add UpdateHub overlays for IEEE 802.15.4 and BLE for Zephyr 2.5. In fact, it is a matter of having the transport well configured and have internet access to enable UpdateHub on it.
If you already have a Thread Networking with Internet access we can tune UpdateHub to work.

Hi @nehalrohit
I got UpdateHub over 6lowPAN working. The below config was necessary to fix DNS error:
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_SERVER1="2001:db8::1"

CONFIG_UPDATEHUB_SERVER="2001:db8::1"

I tested on my local network using RADVD to serve IPs and CoAP block size equal to 64 CONFIG_UPDATEHUB_COAP_BLOCK_SIZE_EXP=2
As I mention, we will update the sample application with the overlays soon.
In the meantime, I hope this could help you to move forward.
[00:00:00.256,622] <inf> mcuboot: Starting bootloader
[00:00:00.263,061] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.273,315] <inf> mcuboot: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.283,782] <inf> mcuboot: Boot source: primary slot
[00:00:00.296,142] <inf> mcuboot: Swap type: none
[00:00:00.548,004] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.555,847] <inf> mcuboot: Jumping to the first image slot

*** Booting Zephyr OS build zephyr-v2.4.0-1459-g423ceeb89683  ***
[00:00:00.001,861] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
[00:00:00.011,840] <inf> main: UpdateHub sample app started
[00:00:00.011,871] <inf> main: Confirming the boot image
[00:00:00.011,932] <inf> main: Network disconnected
uart:~$ net iface

Interface 0x20000b1c (IEEE 802.15.4) [1]
========================================
Link addr : F4:CE:36:08:9E:58:3C:B7
MTU       : 125
IPv6 unicast addresses (max 3):
        fe80::f6ce:3608:9e58:3cb7 autoconf preferred infinite
        2001:db8::f6ce:3608:9e58:3cb7 autoconf preferred
IPv6 multicast addresses (max 4):
        ff02::1
        ff02::1:ff58:3cb7
IPv6 prefixes (max 2):
        2001:db8::/64
IPv6 default router :
        fe80::10a2:27ee:a8d1:c641
IPv6 hop limit           : 3
IPv6 base reachable time : 30000
IPv6 reachable time      : 37394
IPv6 retransmit timer    : 0
uart:~$ net ping 2001:db8::1
PING 2001:db8::1
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=0 ttl=3 rssi=202 time=22 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=1 ttl=3 rssi=201 time=9 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=2 ttl=3 rssi=203 time=8 ms
uart:~$ updatehub run
Starting UpdateHub run...
No update found
[00:00:37.747,619] <inf> updatehub: No update available
uart:~$ updatehub run
Starting UpdateHub run...
[00:06:17.577,697] <err> updatehub: Could not receive data
[00:06:25.283,416] <inf> updatehub: Firmware download complete
[00:06:25.642,639] <inf> updatehub: Image flashed successfully, you can reboot now
*** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.256,683] <inf> mcuboot: Starting bootloader
[00:00:00.263,122] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x1
[00:00:00.273,376] <inf> mcuboot: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
[00:00:00.283,752] <inf> mcuboot: Boot source: none
[00:00:00.289,398] <inf> mcuboot: Swap type: test
[00:00:16.691,162] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:16.699,005] <inf> mcuboot: Jumping to the first image slot

uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-1459-g423ceeb89683  ***
[00:00:00.001,831] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
[00:00:00.010,772] <inf> main: UpdateHub sample app started
[00:00:00.010,772] <inf> main: Confirming the boot image
[00:00:00.010,894] <inf> main: Network disconnected
uart:~$ updatehub info
Unique device id: 3c589e08fd6088b7
Firmware Version: 2.0.0
Product uid: e4d37cfe6ec48a2d069cc0bbb8b078677e9a0d8df3a027c4d8ea131130c4265f
UpdateHub Server: 2001:db8::1
uart:~$ net ping 2001:db8::1
PING 2001:db8::1
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=0 ttl=3 rssi=210 time=21 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=1 ttl=3 rssi=209 time=10 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=2 ttl=3 rssi=210 time=9 ms
uart:~$

Hi @nandojve,
did you use the updatehub cloud server?
can you giving me the step of how you do with 6lowpan?

Hi @nandojve,
did you use the updatehub cloud server?

No, I used a local server. You can use the cloud server if you have IPv6 on your environment.
The easy way is creating a bridge between 6lowpan0 and eth0 interfaces. If you don't have a IPv6, you need a 6to4 configuration.
I talked with @otavio and we will work to create a yocto layer to enable a 6to4 NAT.

can you giving me the step of how you do with 6lowpan?

We are documenting how to do it.

Hi @nandojve
Thanks for your reply  :)
I have a nordic border router with Internet access.
here  nrf52840dk ------> nordic border router (connected using thread)
where should I run the updatehub local server on the local machine or border router?
Here I merge updatehub+ openthread demo. and flash into nrf52840dk

what parameter i have to update in CONFIG_UPDATEHUB_SERVER="<IP_ADDRESS>"  ? is it local IP or DNS IP ?

This is my Linux config:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether <MAC> brd ff:ff:ff:ff:ff:ff
    inet <NET V4>/24 brd xxxxxxxx scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 <NET V6>/64 scope global dynamic noprefixroute
       valid_lft 2591916sec preferred_lft 604716sec
    inet6 <LINK LOCAL V6>/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
5: wpan0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 123 qdisc pfifo_fast state UNKNOWN group default qlen 300
    link/ieee802.15.4 1a:3e:98:f4:d3:17:66:e1 brd ff:ff:ff:ff:ff:ff:ff:ff
6: lowpan0@wpan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/6lowpan 1a:3e:98:f4:d3:17:66:e1 brd ff:ff:ff:ff:ff:ff:ff:ff
    inet6 2001:db8::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::183e:98f4:d317:66e1/64 scope link
       valid_lft forever preferred_lft forever

I added manually 2001:db8::1/64 (may not be the best addr) to the lowpan0 iface. This is the IP address that boards will try reach.
ip addr add 2001:db8::1/64 dev lowpan0
You can add this to your RADVD config:
interface lowpan0 {
	AdvSendAdvert on;
	MinRtrAdvInterval 3;
	MaxRtrAdvInterval 30;
	AdvCurHopLimit 3;

	route ::/0 {
	};

	prefix ::/64 {
		Base6Interface eth0;
		AdvOnLink on;
		AdvAutonomous on;
		AdvRouterAddr off;
	};
};

This means, share eth0 prefix and share any other prefix at interface to lowpan0 network.
At board using native IEEE 802.15.4 6lowPAN I got:
Interface 0x20000b54 (IEEE 802.15.4) [1]
========================================
Link addr : F4:CE:36:08:9E:58:3C:B7
MTU       : 125
IPv6 unicast addresses (max 3):
        fe80::f6ce:3608:9e58:3cb7 autoconf preferred infinite
        <INET V6 PREFIX>:f6ce:3608:9e58:3cb7 autoconf preferred
        2001:db8::f6ce:3608:9e58:3cb7 autoconf preferred
IPv6 multicast addresses (max 4):
        ff02::1
        ff02::1:ff58:3cb7
IPv6 prefixes (max 2):
        <INET V6 PREFIX>::/64
        2001:db8::/64
IPv6 default router :
        fe80::183e:98f4:d317:66e1
IPv6 hop limit           : 3
IPv6 base reachable time : 30000
IPv6 reachable time      : 25205
IPv6 retransmit timer    : 0
uart:~$ net ping 2001:db8::1
PING 2001:db8::1
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=0 ttl=3 rssi=213 time=17 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=1 ttl=3 rssi=213 time=8 ms
8 bytes from 2001:db8::1 to 2001:db8::f6ce:3608:9e58:3cb7: icmp_seq=2 ttl=3 rssi=213 time=8 ms
uart:~$

Once you can ping Linux Machine, the below config will give you access to UpdateHub docker at same machine:
# Add DNS Server
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_SERVER1="2001:db8::1"

# UpdateHub
CONFIG_UPDATEHUB_SERVER="2001:db8::1"
CONFIG_UPDATEHUB_COAP_BLOCK_SIZE_EXP=2

CONFIG_UPDATEHUB_COAP_BLOCK_SIZE_EXP=2 this will instruct CoAP to use a payload block with 64 bytes. You can try using value 1 (32-bytes) and 0 (16-bytes). I not recommend CoAP blocks >=128 on 6lowPAN networks.

If you follow the OT notes about Tayga (NAT-64) you probably will ping internet. Than, you can disable UpdateHub CE and remove CONFIG_UPDATEHUB_SERVER="2001:db8::1" to get access to "coap.updatehub.io" that is UpdateHub cloud server.
You may need test with/without CONFIG_DNS_SERVER1.  --  as you suggest I changed in the .conf file.
Turn on the updatehub
CONFIG_UPDATEHUB=y
#Minimal Heap mem pool size for the updatehub working
CONFIG_HEAP_MEM_POOL_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
CONFIG_NET_IPV4=n
CONFIG_NET_CONFIG_NEED_IPV4=n
CONFIG_NET_IPV6=n
CONFIG_NET_DHCPV4=y
#Optional if you would like test on the your server
CONFIG_SHELL=y
CONFIG_UPDATEHUB_SHELL=y
CONFIG_SHELL_STACK_SIZE=6144
Debug helpers
CONFIG_LOG=y
#CONFIG_UPDATEHUB_LOG_LEVEL_INF=y
CONFIG_UPDATEHUB_PRODUCT_UID="f9007a8c24b86aa426c4214bc05a2e931b5b53c8830ec8838e7699b68e58c666"
I still get a DNS error
I am successfully sending data to https://panel.thethings.io.

UpdateHub uses sockets so I'm assuming that works with OT.
I try compile OT from current master and got:
[1/500] cd /home/gfbudke/zephyros/mainline/zephyr/build/modules/openthread/build && echo -e O...=1"\n"NRF_802154_IFS_ENABLED=0"\n"NRF52840_XXAA"\n"MBEDTLS_CONFIG_FILE="config-tls-generic.h"
-e OPENTHREAD_CONFIG_ENABLE_BUILTIN_MBEDTLS=0
OPENTHREAD_CONFIG_ENABLE_BUILTIN_MBEDTLS_MANAGEMENT=0
PACKAGE_NAME=OPENTHREAD
PACKAGE_VERSION=gf9fbfad01
OPENTHREAD_CONFIG_THREAD_VERSION=OT_THREAD_VERSION_1_1
KERNEL
__ZEPHYR__=1
_FORTIFY_SOURCE=2
BUILD_VERSION=zephyr-v2.4.0-1763-g0452db4c7ebd
__LINUX_ERRNO_EXTENSIONS__
__PROGRAM_START
NRF_802154_USE_RAW_API=1
NRF_802154_PENDING_SHORT_ADDRESSES=16
NRF_802154_PENDING_EXTENDED_ADDRESSES=16
NRF_802154_RX_BUFFERS=16
NRF_802154_CCA_MODE_DEFAULT=NRF_RADIO_CCA_MODE_ED
NRF_802154_CCA_CORR_LIMIT_DEFAULT=2
NRF_802154_CCA_CORR_THRESHOLD_DEFAULT=45
NRF_802154_CCA_ED_THRESHOLD_DEFAULT=45
NRF_802154_CSMA_CA_ENABLED=1
NRF_802154_TX_STARTED_NOTIFY_ENABLED=1
NRF_802154_ACK_TIMEOUT_ENABLED=1
NRF_802154_INTERNAL_RADIO_IRQ_HANDLING=0
NRF_802154_FRAME_TIMESTAMP_ENABLED=0
NRF_802154_DELAYED_TRX_ENABLED=1
NRF_802154_IFS_ENABLED=0
NRF52840_XXAA
MBEDTLS_CONFIG_FILE=config-tls-generic.h
[2/500] Preparing syscall dependency handling

[495/500] Linking CXX executable zephyr/zephyr_prebuilt.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      287756 B         1 MB     27.44%
            SRAM:       95712 B       256 KB     36.51%
        IDT_LIST:         120 B         2 KB      5.86%
[500/500] Linking CXX executable zephyr/zephyr.elf
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 2b55
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 19f8
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 1a04
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 15c1
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4d0d
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 73e
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 74a
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 756
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 762
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 1a7b
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4f86
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4f93
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4fa0
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4fad
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 68e6
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 6900
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 68f3
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 691a
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 690d
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset d29b
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset d2a8
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 7193
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 2d2d

*** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.377,502] <inf> mcuboot: Starting bootloader
[00:00:00.383,880] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.394,042] <inf> mcuboot: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.404,418] <inf> mcuboot: Boot source: primary slot
[00:00:00.416,656] <inf> mcuboot: Swap type: none
[00:00:00.816,436] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.824,218] <inf> mcuboot: Jumping to the first image slot

uuart:~$
[00:00:00.002,075] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
[00:00:00.006,225] <err> os: ***** MPU FAULT *****
[00:00:00.006,225] <err> os:   Data Access Violation
[00:00:00.006,256] <err> os:   MMFAR Address: 0x200125a0
[00:00:00.006,256] <err> os: r0/a1:  0x20012594  r1/a2:  0x00000000  r2/a3:  0x20012600
[00:00:00.006,256] <err> os: r3/a4:  0x200125a0 r12/ip:  0x00000007 r14/lr:  0x0003e24d
[00:00:00.006,256] <err> os:  xpsr:  0x81000000
[00:00:00.006,256] <err> os: Faulting instruction address (r15/pc): 0x0003edec
[00:00:00.006,286] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:00.006,286] <err> os: Current thread: 0x20001ea8 (main)
[00:00:01.174,255] <err> os: Halting system

It is a fault inside memset.
Try build samples/net/sockets/echo_client and got same.
@rlubos , can you assist with this, please? What is the latest hash that you know is ok?
Zephyr bc6d16e422
OT f9fbfad0119d7b0d473eaac50c2f0a0a483c2823

@nehalrohit can you try ping coap.updatehub.io? Since you are using Tayga (NAT64) you will navigate, in fact, on IPv4, try this:
1- Get IPv4 from coap.updatehub.io
ping -4 coap.updatehub.io
PING coap.updatehub.io (<IP_ADDRESS>) 56(84) bytes of data.

2- Translate IPv4 to IPv6
https://findipv6.com/ipv4-to-ipv6/<IP_ADDRESS>
IPv4 address: <IP_ADDRESS>
IPv4 Mapped IPv6 Address: ::ffff:9f59:e141

3- Create the OT IPv6 to be translated to IPv4
Prefix: 64:ff9b::
Sufix: 9f59:e141
results: 64:ff9b::9f59:e141

4- Try ping coap.updatehub.io from a Joiner
> ping 64:ff9b::9f59:e141
Done
> 16 bytes from 64:ff9b:0:0:0:0:9f59:e141: icmp_seq=8 hlim=52 time=173ms

If this works, you can configure DNS and UpdateHub as:
# Add Google <IP_ADDRESS> DNS Server
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_SERVER1="64:ff9b::808:808"

# Add UpdateHub Cloud
CONFIG_UPDATEHUB_SERVER="64:ff9b::9f59:e141"
CONFIG_UPDATEHUB_COAP_BLOCK_SIZE_EXP=2

Since UpdateHub uses sockets and you have internet access it should work.

Hi @nandojve
I am able to ping coap.updatehub.io from a Joiner.

here this setting in prj.conf
Add Google <IP_ADDRESS> DNS Server
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_SERVER1="64:ff9b::808:808"
Add UpdateHub Cloud
CONFIG_UPDATEHUB_SERVER="64:ff9b::9f59:e141"
CONFIG_UPDATEHUB_COAP_BLOCK_SIZE_EXP=2
I am still getting a DNS resolver issue. anything else need to change?

UpdateHub uses sockets so I'm assuming that works with OT.
I try compile OT from current master and got:
[1/500] cd /home/gfbudke/zephyros/mainline/zephyr/build/modules/openthread/build && echo -e O...=1"\n"NRF_802154_IFS_ENABLED=0"\n"NRF52840_XXAA"\n"MBEDTLS_CONFIG_FILE="config-tls-generic.h"
-e OPENTHREAD_CONFIG_ENABLE_BUILTIN_MBEDTLS=0
OPENTHREAD_CONFIG_ENABLE_BUILTIN_MBEDTLS_MANAGEMENT=0
PACKAGE_NAME=OPENTHREAD
PACKAGE_VERSION=gf9fbfad01
OPENTHREAD_CONFIG_THREAD_VERSION=OT_THREAD_VERSION_1_1
KERNEL
__ZEPHYR__=1
_FORTIFY_SOURCE=2
BUILD_VERSION=zephyr-v2.4.0-1763-g0452db4c7ebd
__LINUX_ERRNO_EXTENSIONS__
__PROGRAM_START
NRF_802154_USE_RAW_API=1
NRF_802154_PENDING_SHORT_ADDRESSES=16
NRF_802154_PENDING_EXTENDED_ADDRESSES=16
NRF_802154_RX_BUFFERS=16
NRF_802154_CCA_MODE_DEFAULT=NRF_RADIO_CCA_MODE_ED
NRF_802154_CCA_CORR_LIMIT_DEFAULT=2
NRF_802154_CCA_CORR_THRESHOLD_DEFAULT=45
NRF_802154_CCA_ED_THRESHOLD_DEFAULT=45
NRF_802154_CSMA_CA_ENABLED=1
NRF_802154_TX_STARTED_NOTIFY_ENABLED=1
NRF_802154_ACK_TIMEOUT_ENABLED=1
NRF_802154_INTERNAL_RADIO_IRQ_HANDLING=0
NRF_802154_FRAME_TIMESTAMP_ENABLED=0
NRF_802154_DELAYED_TRX_ENABLED=1
NRF_802154_IFS_ENABLED=0
NRF52840_XXAA
MBEDTLS_CONFIG_FILE=config-tls-generic.h
[2/500] Preparing syscall dependency handling

[495/500] Linking CXX executable zephyr/zephyr_prebuilt.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      287756 B         1 MB     27.44%
            SRAM:       95712 B       256 KB     36.51%
        IDT_LIST:         120 B         2 KB      5.86%
[500/500] Linking CXX executable zephyr/zephyr.elf
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 2b55
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 19f8
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 1a04
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 15c1
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4d0d
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 73e
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 74a
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 756
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 762
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 1a7b
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4f86
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4f93
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4fa0
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 4fad
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 68e6
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 6900
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 68f3
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 691a
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 690d
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset d29b
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset d2a8
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 7193
/home/gfbudke/zephyr-sdk-0.12.0.b/arm-zephyr-eabi/bin/arm-zephyr-eabi-objdump: DWARF error: could not find variable specification at offset 2d2d

*** Booting Zephyr OS build zephyr-v2.4.0-1442-gbc4823a21183  ***
[00:00:00.377,502] <inf> mcuboot: Starting bootloader
[00:00:00.383,880] <inf> mcuboot: Primary image: magic=good, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.394,042] <inf> mcuboot: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[00:00:00.404,418] <inf> mcuboot: Boot source: primary slot
[00:00:00.416,656] <inf> mcuboot: Swap type: none
[00:00:00.816,436] <inf> mcuboot: Bootloader chainload address offset: 0xc000
[00:00:00.824,218] <inf> mcuboot: Jumping to the first image slot

uuart:~$
[00:00:00.002,075] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
[00:00:00.006,225] <err> os: ***** MPU FAULT *****
[00:00:00.006,225] <err> os:   Data Access Violation
[00:00:00.006,256] <err> os:   MMFAR Address: 0x200125a0
[00:00:00.006,256] <err> os: r0/a1:  0x20012594  r1/a2:  0x00000000  r2/a3:  0x20012600
[00:00:00.006,256] <err> os: r3/a4:  0x200125a0 r12/ip:  0x00000007 r14/lr:  0x0003e24d
[00:00:00.006,256] <err> os:  xpsr:  0x81000000
[00:00:00.006,256] <err> os: Faulting instruction address (r15/pc): 0x0003edec
[00:00:00.006,286] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:00.006,286] <err> os: Current thread: 0x20001ea8 (main)
[00:00:01.174,255] <err> os: Halting system

It is a fault inside memset.
Try build samples/net/sockets/echo_client and got same.
@rlubos , can you assist with this, please? What is the latest hash that you know is ok?
Zephyr bc6d16e422
OT f9fbfad0119d7b0d473eaac50c2f0a0a483c2823

This error happen with SDK 0.12.0-beta-2. When I switch back to 0.11.3 I didn't see anymore objdump: DWARF error
CC: @rlubos @galak

@nandojve

Try build samples/net/sockets/echo_client and got same.

I was not able to reproduce the crash on my side, are there any special conditions to trigger it? The radio driver was updated recently (which I'd suspect), but as said I fail to reproduce. Did you call west update? Perhaps the hal_nordic went out of sync with Zephyr on your side?

You can ignore the run crash. After switch to 0.11.3 I figure out. It was needed to increase RAM from main scope.

Hi @nehalrohit ,
By some reason the DNS name can not be resolved. It is not necessary a problem with UpdateHub.
I took a note to review that code.
I open my development branch https://github.com/UpdateHub/zephyr/tree/topic/uhu_net-management. It is working with OpenThread.
By default I'm using UpdateHub CE, so at overlay-ot.conf:
CONFIG_UPDATEHUB_CE=n
You need add your server IP:
CONFIG_UPDATEHUB_SERVER="64:ff9b::xxxx:xxxx"
Procedure o discover, example from google IPv4 DNS:
# https://www.ultratools.com/tools/ipv4toipv6Result?address=<IP_ADDRESS>
# Mapped Address: 0:0:0:0:0:ffff:808:808
# Extract value                 >808:808<
# Add prefix        64:ff9b::
CONFIG_DNS_SERVER1="64:ff9b::808:808"

For UpdateHub Cloud Server:
ping -4 coap.updatehub.io
PING coap.updatehub.io (<IP_ADDRESS>) 56(84) bytes of data.
Mapped Address: 0:0:0:0:0:ffff:9f59:e141
CONFIG_DNS_SERVER1="64:ff9b::9f59:e141"

I have a working FOTA over OT that I'm using successfully for more than a year and half in a test bed of more than 10 FTD and SED devices. It's based on @mike-scott work on https://github.com/foundriesio/dm-lwm2m , but not sure what its license is, and could we use it on Zephyr. If he confirm that it's OK to use it as a base I could push a PR for review here.

Hello retfie,
can you please suggest to me how you do OTA with Openthread or you can provide some steps to follow( link or code)?
Thanks
Nehal

HI Hehal,
sorry for late response, but here is the essential change that you need on lwm2m-client re-based on current zephyr master branch:
https://github.com/retfie/zephyr/commit/3b9dd7281845c281106b2e85eca604f2665919a6
You need mcuboot bootloader on device and a working leshan server to register on. Then use 5/0/0 or 5/0/1 resource to send zephyr.signed.bin to lwm2m_client. Then look for status (5/0/3) to became 2 (ready to apply), then start update with Exec on /5/0/2.

Hi @nandojve
I already try this configuration
CONFIG_DNS_SERVER1="64:ff9b::9f59:e141"   still not work for me.  my joiner device can ping updatehub.io cloud server but the device cant show on the server.
I open my development branch https://github.com/UpdateHub/zephyr/tree/topic/uhu_net-management. It is working with OpenThread.
if you successfully did then give me a step on how you do with nrf52840dk or code reference?

Hi @nehalrohit
I confirmed here yesterday. The UpdateHub Cloud only accept connections with DTLS. I noted that when I enabled DTLS on Zephyr, for some reason, "send" returns with error and packages have been not sent. It seems something goes wrong at DTLS challenge phase.
I'll recommend, in the mean time that we are investigating the DTLS issue, you to use UpdateHub CE. You can pull the docker container and run it in your environment or at any service provider (digital ocean, vultr, etc) with docker support. The smallest machine is enough to run the service.

hi @nandojve
I can see the device on UpdateHub CE.

but updatehub run cmd not working

Hi @nandojve
by mistake, the thread is closed my issue is still going on its not yet close

Hi @nandojve
I have successfully  seen the device on the update server

create rollout not work .


BitBucket integration not working

I've pretty much followed everything here: http://readme.drone.io/setup/config/bitbucket/
The process itself is (as it should be) trivial, however it's not working.
After logging in to Drone through BitBucket, the returned URL is "/api/auth/bitbucket.org?oauth_verifier" (with a key and token), which results in a completely blank page. Additionally /var/log/upstart/drone.log also appends an empty line with a timestamp.
I'm using the latest .deb build on Ubuntu.

I think your issue come from the URL you set in Bitbucket (Manage account -> OAuth).
You should have something like: http://url.to.your.api/api/auth/bitbucket.org

My callback URL is set to http://drone.mydomain.com/api/auth/bitbucket.org.

I had same problem with github integration. In my case, I needed to use https:// instead of http, because there was an auto-redirect to https, which broke OAuth2.

Try it.
$ sudo ntpd -q

and
$ sudo hwclock --systohc --utc

@slawus I don't have HTTPS support by default (using Digital Ocean's pre-built image)
@mopemope Nope, sorry, didn't make a difference.

We had the same problem until we realised that you must expose access to your repositories in the bitbucket permissions configuration to make it work. I think this is overly excessive, as some of our users (3rd parties, contractors, etc) are only 'watchers' and have no need to share all their code with our drone instance. I will try to look through the code and provide a PR but guidance is welcome.

@dim Care to elaborate on that? I'm the only user on my team on Bitbucket, and from what I can tell I have all the permissions necessary, still results in a white screen when trying to login using Bitbucket.

Might not be related, but I did a little bit of debugging on our side and noticed that BitBucket's v1 API requires you to grant permissions to Read your account information and Read your repositories and their pull requests, because the /v1/user endpoint which is used by Drone exposes both user and repository information. BitBucket have fixed this in their v2 API, where they have a strict separation between user and repo information.
To fix this, drone would have to migrate to use https://confluence.atlassian.com/bitbucket/version-2-423626329.html, I will create a separate ticket for it in https://github.com/drone/go-bitbucket

0.4 version of Drone has migrated to Bitbucket v2 API with Oauth2. I've been running it for the past few days and can confirm it is working well.
documentation for reference:
http://readme.drone.io/docs/setup/bitbucket.html


WebGPU update

Recently, there has been a 3rdparty addition for bgfx.
This requires adding a cmake file for 3rdparty/webgpu and linking it properly.

Are you getting compile errors when trying to use other backends? Otherwise, I'd suggest waiting a little bit before implementing this, since the WebGPU backend is recent enough that there will likely still be changes in the bgfx build system. It's also missing recompiled shaders to be usable right now.

Yes it fails to compile if updating the bgfx submodule

https://github.com/widberg/bgfx.cmake/pull/83 yea try this then.


As a developer, I can specify the rate at which automated blue/green rollouts proceed

Copied from Azure Dev Ops Backlog

Here is an article describing an approach to implementing this functionality: https://azure.microsoft.com/en-gb/blog/blue-green-deployments-using-azure-traffic-manager/

moved to entref-appservice-containers project


Write Workbook (containg two worksheets) in CSV file

I am developing an angular 6 app and I am trying to write in a csv file a workbook containg two worksheets with this code :
    const workBook: Excel.Workbook = this.setupDownload();
    workBook.csv.writeBuffer().then( data => {
      const blob: Blob = new Blob( [data], {type: 'text/plain;charset=utf-8'} );
      saveAs( blob, 'virga.csv');
    });

But I only got the first worksheet in the downloaded csv file.

+1
Same on .xlsx.writeBuffer()
All worksheets persist in workbook, but file contain only one

+1

This issue is still happening.
When creating more than 1 worksheet with csv only the 1st one is actually added

All inactive issues were closed.
About CSV, I can't imagine how it should works for many spreadsheets - csv is just simple file without features like spreadsheets. Taking only one seems be reasonable.
If you don't mind please describe me what do you expect as a result?

Join our Discord: https://discord.gg/siema


feat(Paths): Add optional anyArrayIndexAccessor

PR Checklist

[x] Addresses an existing open issue: related to

resolves: #411

[x] Steps in Contributing were taken

Overview

Adds optional configuration that allows wildcards to satisfy array index access.

@mylesj for your benefit, the testing strategy of ts-essentials is to run typechecks on multiple versions (between 4.5 and 5.5). Currently, 3 typechecks have failed on version 4.6-4.8, as GHA doesn't contain continue-on-error: true, it will stop running others. Kindly ping me when you're ready, I will accept re-running typechecks once again.

@mylesj for your benefit, the testing strategy of ts-essentials is to run typechecks on multiple versions (between 4.5 and 5.5). Currently, 3 typechecks have failed on version 4.6-4.8, as GHA doesn't contain continue-on-error: true, it will stop running others. Kindly ping me when you're ready, I will accept re-running typechecks once again.

@Beraliv So I had a look at the tests under the different versions and I'm reasonably sure this change set isn't directly related - more a consequence of my re-enabling the tests and surfacing existing errors on master. Specifically:

test/paths.ts fail on 4.6
test/xor.ts fail on: 4.6, 4.7 & 4.8

That said, I spent some time trying to understand the paths issue but can't for the moment see what the problem might be - something is truncating the paths prematurely or preventing recursion. I'd ask that you take a closer look to confirm whether this is related to recent changes.
If you agree, I've selectively disabled those specific test suites for now. Otherwise I'm going to be unavailable for the next week or so but will take another look when I'm back.

Fantastic. So I'm AFK / on vacation this week but will wrap things up at the weekend.

Ah my bad, I'm not familiar with this approach to change log. Done - thanks for sticking with me, appreciate the support!

Great, thank you @mylesj for your efforts! Changes look good, so I will merge it now


Error "413 Request Entity Too Large" while request goes between containers using service discovery.

Hi,
We are getting error "413 Request Entity Too Large" when doing HTTP post with several mbs of data in the request body. The request goes between containers using service discovery.
Based on the error, I first thought that maybe the problem is between browser and ui container and I need to add nginx.ingress.kubernetes.io/proxy-body-size annotation to the load balancer based on https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-max-body-size . This did not work since the problem seems to be in the internal load balancer for the service discovery.
We had this issue earlier from user's browser to web UI but that was fixed as we lifted the limitation of size  by changing limit to 50MB for main ingress load balancer. This problem is specific to internal container communications.

Hello, @kcgochar. questions about ingress-nginx, please use https://github.com/kubernetes/ingress-nginx/issues/new/choose

/kind support
This is not the right place to ask non-documentation questions.
/close

thank you.. @tengqm @hongchaodeng  for directing correct place of question..


refactor(transformer): deserialize BabelOptions::compiler_assumptions

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#7048  👈
#7047
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Boshen and the rest of your teammates on  Graphite


[Hub Generated] Review request for InkRecognizer to add version preview/v1.0 - Changing format of points from string to object as requested by Azure board and fleshing out properties.

If you are a MSFT employee you can view your work branch via this link.
Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-python

Can one of the admins verify this patch?

Automation for azure-sdk-for-go

Changes were requested by the Azure API Review Board. @olduroja, was the 1.0 version of the APIs/SDKs deployed publicly?

Changes were requested by the Azure API Review Board. @olduroja, was the 1.0 version of the APIs/SDKs deployed publicly?

We're still in preview and the preview has been deployed publicly.

can existing users be broken by the changes? If so, how will you deal with existing users being broken?
Generally, changes to the API are accompanied with a new API version.

can existing users be broken by the changes? If so, how will you deal with existing users being broken?
Generally, changes to the API are accompanied with a new API version.
This won't break anyone. We will continue to support the old format while we're in preview.

can existing users be broken by the changes? If so, how will you deal with existing users being broken?
Generally, changes to the API are accompanied with a new API version.

This won't break anyone. We will continue to support the old format while we're in preview.

@johanste, do you have any other concern? If no, I'll approve and merge this PR.

I'm good.


Merge tikv master

What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
merge tikv
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.9 out of 17 committers have signed the CLA.:white_check_mark: HuSharp:white_check_mark: dveeden:white_check_mark: hehechen:white_check_mark: joccau:white_check_mark: lijie0123:white_check_mark: fengou1:white_check_mark: hongyunyan:white_check_mark: lance6716:white_check_mark: SpadeA-Tang:x: JmPotato:x: BusyJay:x: v01dstar:x: tabokie:x: YuJuncen:x: LykxSassinator:x: MyonKeminta:x: bufferfliesYou have signed the CLA already but the status is still pending? Let us recheck it.


[False positive] no-async-without-await and abstract methods

Bug Report

TSLint version<EMAIL_ADDRESS>TypeScript version<EMAIL_ADDRESS>Running TSLint via: CLI and VSCode

Reproduction using TSLint Playground
I couldn't reproduce it in the Playground, however even invalid code is not detected by this rule what apparently means the rule is not present there at all (?)
TSLint Playground

TypeScript code being linted
export abstract class SomeService<T> {
  protected readonly abstract repository: Repository<T>;

  protected abstract async falsePositive(id: number): Promise<T | null>;

  public async truePositive() { /* */ }
}

with tslint.json configuration:
{
  "extends": [
    "tslint:all",
    "tslint-config-standard",
    "tslint-config-airbnb-base"
  ],
  "rules": {
    "no-async-without-await": true,
	"...": "..."
  }
}

Actual behavior
Treats abstract methods as an error
Expected behavior
Should ignore abstract methods because they don't have a body

Same for overloaded functions/methods, as described in #4776


fixed issue #6993 documenting requirement of coupling map class

Summary
Fixes #6993
Details and comments
This pull request fixes the issue #6993. The documentation of the coupling map class does not specify that the indexes must be contiguous. The constructor identifies if there is a hole and adds the nodes as isolated nodes in the coupling map. I have added this requirement to the documentation in the coupling map class.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Pico 4 headset not working with StereoKit

I cannot run StereoKit with the StereoKit_DotNet project and the StereoKit_Android project on my Pico 4.
StereoKit_DotNet (PC VR) problem
I use the Pico Streaming Assistant with a USB cable, which successfully opens and runs SteamVR. The problem is when I run StereoKit it does not recognise my headset and just opens in "flatscreen mode", even though it is working right in SteamVR.

I get this info in the console:
[SK info] Couldn't find our desired MR form factor, no MR device attached/ready? [XR_ERROR_FORM_FACTOR_UNAVAILABLE]
[SK info] MixedReality display mode failed, falling back to Flatscreen

StereoKit_Android problem
When I run StereoKit natively on the Pico 4 it is just displayed in 2D mode and I can not do anything.

I then looked at the Pico 4 documentation and it stated that I should add <meta-data android:name="pvr.app.type" android:value="vr" /> in the AndroidManifest.xml file. When I then ran the StereoKit application, it was just stuck in the Pico 4 loading screen.

On their documentation, they are stating that some things need to be loaded into OpenXR, but I don't have the knowledge of OpenXR so I have not tested adding that code.

I uploaded the issue on the main StereoKit repository instead here


Initial message fix

Last week I've been checking bidirectional examples with latest Xcode 16 beta 3 and realised it uses HBv2, which is supported only by macOS 14+, and it's not perfect. After downgrading to HBv1—same problem with returning stream until some bytes are sent appeared. Haven't figured out yet why, maybe some tech inside uses URLSession. So, this PR fixes two things:

Moved server to Vapor for two reasons: 1) anyway can't use HBv2 now and 2) it could be more familiar to others;
Moved back to URLSession for client and added initial message hack.

It's working for now. I'll be checking from time to time status of Swift Server tools, maybe we can improve it in the future.

@czechboy0 will check, last time I've tried—it was a bit hard, cause it's hard to debug "not happening" events.


Rewriting api

The current API is not very helpful and I found it a bit confusing so I made small changes to the routes.

Added getting tasks/bids specific to a user
Get bid specific to a task
Added /bids/ and users/ routes

This branch was originally #156 but there's no API available so I worked on the backend for a bit.
WARNING - API calls on the front end have not been changed (only relevant if any pages made calls to the old API)

And I already had an error when I add call api in creating new task

That means the primary key is wrong?

I actually have a question. For the task table, I don't know the task_id(which should be auto increment) when I create a new task, so does the name of the route make sense if I have to know the task_id when I send post req.

The new routes all only handle GET requests, you should be able to make new tasks by making a POST request to tasks/ because of this line
router.register(r'tasks', TasksViewSet, basename='tasks')

And I already had an error when I add call api in creating new task
That means the primary key is wrong?

How are you calling the api? I just go to the links directly and they seem to work. See http://localhost:8000/api/app/users/8/bids
and
http://localhost:8000/api/app/users/8/tasks


Use cross-spawn for psc

Will be fixed once #34 is merged and ethul/purescript-webpack-plugin#3 is fixed.

Merged #34. This issue should be tracked using ethul/purescript-webpack-plugin#3


nodejs_23: 23.1.0 -> 23.2.0

https://github.com/nodejs/node/releases/tag/v23.2.0
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

It's getting harder and harder to maintain Node.js with https://github.com/NixOS/nixpkgs/pull/344438 not landing

It's getting harder and harder to maintain Node.js with #344438 not landing

Merged. Change looked reasonable.

Merged.
Please note that the nodejs_23 on staging will require a fixup (i.e. the revert removed you added here) when master gets merged into staging (happens every few hours via some automation).
Would you mind filing a PR to resolve this as well?
Feel free to cc me to merge.

Please note that the nodejs_23 on staging will require a fixup (i.e. the revert removed you added here) when master gets merged into staging (happens every few hours via some automation).

There's going to be a git conflict to resolve, which IIUC will result in the bot giving up the merge and instead post a comment in #105153. I don't think I can do the git conflict resolution in a PR, can I?

OK interesting I would've expected git to get done with that. Anyways, going to resolve now.

Done: 85228bf93ed88b0e72add43285f7d0f8cdafb959
Nothing tbd from you here.


ReadWrite/ReadOnly/WriteOnly should allow additional range types

It would often be useful to express bit fields as closed ranges or other such range types:
@ReadWrite(22...22, as: Bool.self) var booleanLikeThing: BooleanLikeThing
or the remaining bits in a register layout:
@ReadOnly(16..., as: UInt16.self)  var remainingBits: RemainingBits

I have this mostly functional, but I'm trying to figure out if I can support ... (unbound range) without needing an additional overload which could hurt compiler performance

Got it working with help from @ahoppen (thank you!)


Logout token

Trello
We're implimenting OIDC backchannel logout, to recieve messages from digital identity
indicating the uses wishes to have their session removed everywhere.
The initial stage of this is to:

give account-api the awarenss of a LogoutToken
Validate and verify contents following the specification
Ensure the JWT can be decoded correctly and the signature verified

This PR adds a Plain old ruby class to do just that.

Gonna ask for a re-review to check that this answer reflects our Friday chat and your happy to continue on that basis.
If so, I'll merge and we can rebase the next one!


Autocompletion drives me up the wall

After installing oh-my-zsh, the new autocompletion has given me rise in blood pressure that is unsustainable.  Behold:
fiesh@wsbox ~ $ mkdir /tmp/arg && cd /tmp/arg
fiesh@wsbox /tmp/arg $ touch coin-download coin_i686-w64-mingw32.shared coin_x86_64-w64-mingw32.shared coin_x86_64-w64-mingw32.static
fiesh@wsbox /tmp/arg $ cat coin<TAB>

Now the following happens: zsh adds a hyphen, positions the cursor before the hyphen. After the first <TAB>, pressing tab again autocompletes to coin-download.  WTF. (Not as in cycling through the options, but autocompletes-and-moves-on wise.)
I suppose this is some very smart autocompletion that outsmarts me instantly.  I'd really appreciate it if there was some mentioning of it, how to use it properly, and especially how to get rid of it.
Thanks!

This is from the completion zstyle settings in lib/completion.zsh.
You can get simpler behavior by sticking this or something similar in your ~/.zshrc after the point where it loads OMZ.
# Lowercase matches uppercase but not vice-versa
zstyle ':completion:*' matcher-list '' 'm:{[:lower:]}={[:upper:]}'

You can also completely replace OMZ's completion setup with your own by adding a lib/completion.zsh file to your $ZSH_CUSTOM dir (by default, ~/.oh-my-zsh/custom), which will shadow the one in OMZ.

Awesome, thank you so much!  I think it'd be great if this was somehow documented.


:bug: Analysis details needs task obj

resolves https://issues.redhat.com/browse/MTA-219

Good catch, thanks @gildub (noticed this and was about to fix it, I need to pull main more often :) )


Fixed installation from sources on README.

Backport of #607 and #675.

Build status:

Linux
Mac
Windows
Performance


Cache look-ups for known identifiers to accelerate wikidata resolution

As more collections/datasets begin using identifiers like VIAF, HUH, etc. the processing of these will become slower. Cache these as two columns during processing: identifier and wikidata to accelerate the subsequent re-processing of these.

Note to self that this is becoming significantly more pressing because there are now 108 datasets that have content in recordedByID and identifiedByID, upward of 2M occurrence records.

Performance seems to be improved & so closing.


OAuth for Multiple Accounts

I may be missing something here...but is there a way to Oauth TWO different account? My partner and I each have Google Calendars, so I assume each would have to authenticate. But after running node authenticate.js for one...I can't seem to do it for a second account.
Sorry if this is dumb or I just overlooked an obvious solution!

Nix this. It's working


Extension uses to get stuck un "Uploading..."

Agents have to relaunch the dialog again and again.
A lot of time wasted.

Relesed version 1.9.

Testing
Getting the # of pages scraped by agent.
select p.upload_reservation_id, u.scraper_last_ping_time, u.scraper_last_ping_version, count(p.id)
--select p.*
from scr_page p
join "user" u on u.email=p.upload_reservation_id
where p.upload_end_time > current_timestamp - interval '5 minutes'
and p.upload_success = true
group by p.upload_reservation_id, u.scraper_last_ping_time, u.scraper_last_ping_version
order by p.upload_reservation_id

Getting agents online.
select u.id, u.name, u.email, u.scraper_last_ping_time, u.scraper_last_ping_version, u.scraper_share, u.scraper_stat_total_earnings, u.stealth_default_max_pages_per_hour, u.stealth_default_max_pages_per_day
from "user" u
where u.scraper_last_ping_time is not null
--and<EMAIL_ADDRESS>order by u.scraper_last_ping_time desc limit 100;


Feature/migrate nested examples

Properly migrate the examples if they are in a nested folder: firmware/examples/blink-led/blink-led.cpp ==> examples/blink-led/blink-led.cpp

Coverage increased (+0.0004%) to 99.861% when pulling 6ebc575248271e0bf9a8665bc17be161b802dcd3 on feature/migrate-nested-examples into ccd1769c4aeeca26d6bd58487eb6bbe20a0d7f67 on develop.


Static code analysis.

Some static code analysis.
try-with-resources
for-each loop
Turned a Regex that depended solely on alternation into a character class one, which behaves exactly the same and is more performant.
Also turned it into a static pattern and have the class use that to do the split.

Sorry, this project is effectively deprecated now.  This code has been merged into ebean-agent which now performs all the enhancement.

Aha, didn't know that. I'll go ahead and close this.


Thumbnail also enable Preview

Microsoft PowerToys version
0.51.1
Running as admin

[X] Yes

Area(s) with issue?
PDF Preview, PDF Thumbnail, SVG Preview, SVG Thumbnail
Steps to reproduce

Disable all preview panes
Enable only PDF and SVG thumbnails
Reboot

file-explorer-log_2021-12-05.txt
✔ Expected Behavior
Preview panes not enabled for PDF and SVG
❌ Actual Behavior
Preview panes enabled for PDF and SVG
Other Software
No response

TIL: When the preview panes are disabled and the thumbnails are enabled, Windows is using the thumbnails for the preview panel.
So the thumbnail is shown in the previewer and not the images from previewers. In the case of the SVG files, it's just the same image, the thumbnail and the preview are just the same. In case of the PDF file only first page is shown when only the thumbnails are enables. When the previewer is enabled, the first 10 pages are shown in the preview panel.

Still happening on 0.55.1
@crutkas maybe something we want to be fixed for stability?

Agreed we should fix it

Is this something you’d like to pick up?

Is this something you’d like to pick up?

Will take a look and see if this is something I can solve.
I think we are configuring too many registry keys for thumbnails.

@crutkas
I have looked into this and since registry keys looks good I have tested older PT releases and 3rd party thumbnail providers.
Looks like if there isn't a preview pane available for the file extension Windows displays the thumbnail in the preview pane.
I can't find any official reference for this behavior though.
Easy to spot with PDF.
Icon preview

Preview pane

I think we can close this.


Handle onMouseDown and onMouseUp on Body or Fixture

Hello,
I would like to know when an element is dragged and I could not find a way to retrieve these data.
Can I add an event listener with onMouseDown and onMouseUp somewhere?
Thank you

You could find what fixture/body a point (for example mouse click location) intersect with and use that to implement mouse events which is how testbed works, however planck-js is just physics and does not process ui events.


set p.Block after parsing in helm provenance type

not sure that this is an actual issue, but we should only set p.Block after ensuring parseMessageBlock returns without error.
Signed-off-by: Bob Callaway<EMAIL_ADDRESS>

Looks great, I'll hold for Andrew to give it a check too.

@bobcallaway @dlorenc Queued this up to review this evening


Add typing annotations to the repo.

We would like to add typing annotations to the repo and use pyright for static type checking.

Grabbing this one!


Hide deprecation warning with createVault constructor

This is a cleanup patch from the new constructor feature.
The KeyStore.createVault method relies on the underlying new constructor, which we have issuing a deprecation warning, so now that warning is actually firing always.
We were talking about getting rid of the new method eventually, but to actually stop using it while most of its methods are instance methods on the prototype would require a bit of a refactor.
This is a cheap solution that avoids that refactor for the moment.
In the slightly longer term, we might want to make the current KeyStore class simply return another internal class, so it can use the new constructor internally, while only throwing the deprecation warning when its own new constructor is used.
I'm opening to making that change to this PR, but wanted to open the discussion first.

Not much happening getting this merged..  is eth-lightwallet still being maintained?

@christianlundkvist

This flew under the radar a bit. Checked it out and looks good. @flyswatter can you bump the version in package.json and add a note in RELEASE-NOTES? After that I can merge and publish in npm. Thanks! 😃

Merging this now since I have some time. :) Will also update version and push to npm.

Ok done! :) Eventually I want to deprecate the old version.


Service instance purger should record a purge audit event

Previously we recorded an 'audit.service_instance.delete' event when a
service instance was purged, which makes it harder for operators to
distinguish between purge and delete.
fixes #1208
cc @georgi-lozev

SAPI team will merge this PR.


add imports

🚀 The docs preview is ready! Check it out here:     https://modal-labs-examples--frontend-preview-90bad0e.modal.run


Update http4s-circe, http4s-dsl, ... to 1.0.0-M29

Updates

org.http4s:http4s-circe
org.http4s:http4s-dsl
org.http4s:http4s-ember-client
org.http4s:http4s-ember-server

from 1.0.0-M23 to 1.0.0-M29.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.http4s" } ]

labels: library-update, semver-pre-release

Superseded by #78.


Sync eng/common directory with azure-sdk-tools for PR 5103

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/5103  See eng/common workflow

The build fails on Linke verification check:

##[error][404] broken link https://github.com/microsoft/vcpkg/blob/master/docs/users/triplets.md

This is unrelated to this change, so I am going to override.

/check-enforcer override


up/down "to" migration should be exclusive

From an English semantic viewpoint, when you run umzug.down({ to: "2" }), then it should not run down migration of task 2.
Similarly, umzug.up({ to: "2" }) should not run up migration of task 2.

After debating with the team, it seems that no one cares about what English cares about. So, I'll just update docs.
Still, uzmug's from is exclusive, which seems a bit inconsistent.

This decision is extraordinarily disappointing. The behavior is wildly inconsistent with the wording in usage, and causes completely unexpected behavior.


Offer global completions in all blocks

Test coverage from integration test SnippetTabInsertIIFE.

Without this, snippets aren't offered in (e.g.) function blocks.


Identify fingerprint as parameter.

First of all I've like to congrats you for this node wrap.
I'm working on a platform with a principal machine (server) and have some raspberries(clients) conected with my fingerprint lectors (Digital Persona U.are.U) where I'm expecting to have at least 500 different users. With the enrollStart() I'm saving the fingerprint String on a database, and when I need to Identify someone I create an Array as you have on the example. I see that the identifyStart() needs this array and when you put the finger in compares, but Is there a way with this library where I can read only the fingerprint on my client, and send it to the server and compare it there?
Thanks for your help and happy holidays!

You want a identify on the server without a fingerprintsensor connected? I would name this identifymatch where you have both the enrolled fingerprints and the captured fingerprint which should be compared.
As far as I remember, this isn't possible with the libfprint, which is used in the backend.
Please correct me if I'm wrong or if I missunderstood you.

I will have connected the fingerprintsenor, read the fingerprint and then send it to the server and do the indentify there. As far as I be researching I know that the identify is live, but I wanted to ask.

There is no serverside matching AFAIK.
closing it.


Dependence of output on batch size

The response from a convolutional neural network is sometimes dependent on the batch size.
Code to reproduce
import tensorflow as tf
import numpy as np

reuse = False
init = tf.initializers.variance_scaling()
X = tf.placeholder(
    dtype=tf.float32,
    shape=[None] + [86]*3 + [1])

with tf.variable_scope("level1"):
    net1 = tf.layers.conv3d(X, 16, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init )

with tf.variable_scope("level2"):
    net2_in = tf.layers.conv3d(net1, 32, 2, strides=2, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='ds', kernel_initializer=init )
    net2 = tf.layers.conv3d(net2_in, 32, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init )

with tf.variable_scope("level3"):
    net3_in = tf.layers.conv3d(net2, 64, 2, strides=2, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='ds', kernel_initializer=init )
    net3 = tf.layers.conv3d(net3_in, 64, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init )

with tf.variable_scope("level4"):
    net4_in = tf.layers.conv3d(net3, 128, 2, strides=2, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='ds', kernel_initializer=init )
    net4 = tf.layers.conv3d(net4_in, 128, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init)

with tf.variable_scope("level5"):
    net5 = tf.layers.conv3d_transpose(net4, 64, 2, strides=2, activation='elu',padding='same',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='trans', kernel_initializer=init)
    net5 = tf.concat([net5, tf.slice(net3,[0]+[(18-14)//2]*3+[0],[-1]+[14]*3+[64])], axis=4) #tf.slice(net8, begin, tf.shape(net18))
    net5 = tf.layers.conv3d(net5, 128, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init)

with tf.variable_scope("level6"):
    net6 = tf.layers.conv3d_transpose(net5, 32, 2, strides=2, activation='elu',padding='same',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='trans', kernel_initializer=init)
    net6 = tf.concat([net6, tf.slice(net2,[0]+[(40-24)//2]*3+[0],[-1]+[24]*3+[32])], axis=4) #tf.slice(net5, begin, tf.shape(net18))
    net6 = tf.layers.conv3d(net6, 64, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init)

with tf.variable_scope("level7"):
    net7 = tf.layers.conv3d_transpose(net6, 16, 2, strides=2, activation='elu',padding='same',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='trans', kernel_initializer=init)
    net7 = tf.concat([net7, tf.slice(net1,[0]+[(84-44)//2]*3+[0],[-1]+[44]*3+[16])], axis=4) #tf.slice(net5, begin, tf.shape(net18))
    net7 = tf.layers.conv3d(net7, 32, 3, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init)

with tf.variable_scope("out"):
    net_out = tf.layers.conv3d(net7, 2, 1, activation='elu',padding='valid',kernel_regularizer=tf.contrib.layers.l2_regularizer(0.0001),reuse=reuse,name='conv', kernel_initializer=init)

model = net_out

sess = tf.Session()
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())

batch_size = 34
full_batch = np.random.rand(154,86,86,86,1)
num_batches = (len(full_batch)//batch_size)
if len(full_batch) % batch_size: num_batches += 1
y_out_list = []
print("Batch size of 34")

for i in range(0, num_batches):
    batch = full_batch[i*batch_size:min(len(full_batch), (i+1)*batch_size)]
    batch1 = full_batch[2*i*batch_size//2:min(len(full_batch), (2*i+1)*batch_size//2)]
    batch2 = full_batch[(2*i+1)*batch_size//2:min(len(full_batch), (2*i+2)*batch_size//2)]
    y_out = sess.run(model, feed_dict={X:batch})
    y_out1 = sess.run(model, feed_dict={X:batch1})
    y_out2 = sess.run(model, feed_dict={X:batch2})
    print("Should be true if independent of batch size: ", np.all(y_out==np.concatenate([y_out1,y_out2])), " Shape: ", y_out.shape)

batch_size = 20
full_batch = np.random.rand(154,86,86,86,1)
num_batches = (len(full_batch)//batch_size)
if len(full_batch) % batch_size: num_batches += 1
y_out_list = []
print("Batch size of 20")

for i in range(0, num_batches):
    batch = full_batch[i*batch_size:min(len(full_batch), (i+1)*batch_size)]
    batch1 = full_batch[2*i*batch_size//2:min(len(full_batch), (2*i+1)*batch_size//2)]
    batch2 = full_batch[(2*i+1)*batch_size//2:min(len(full_batch), (2*i+2)*batch_size//2)]
    y_out = sess.run(model, feed_dict={X:batch})
    y_out1 = sess.run(model, feed_dict={X:batch1})
    y_out2 = sess.run(model, feed_dict={X:batch2})
    print("Should be true if independent of batch size: ", np.all(y_out==np.concatenate([y_out1,y_out2])), " Shape: ", y_out.shape)

batch_size = 42
full_batch = np.random.rand(154,86,86,86,1)
num_batches = (len(full_batch)//batch_size)
if len(full_batch) % batch_size: num_batches += 1
y_out_list = []
print("Batch size of 42")

for i in range(0, num_batches):
    batch = full_batch[i*batch_size:min(len(full_batch), (i+1)*batch_size)]
    batch1 = full_batch[2*i*batch_size//2:min(len(full_batch), (2*i+1)*batch_size//2)]
    batch2 = full_batch[(2*i+1)*batch_size//2:min(len(full_batch), (2*i+2)*batch_size//2)]
    y_out = sess.run(model, feed_dict={X:batch})
    y_out1 = sess.run(model, feed_dict={X:batch1})
    y_out2 = sess.run(model, feed_dict={X:batch2})
    print("Should be true if independent of batch size: ", np.all(y_out==np.concatenate([y_out1,y_out2])), " Shape: ", y_out.shape)

== cat /etc/issue ===============================================
Linux  4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.5 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
== are we in docker =============================================
No
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== uname -a =====================================================
Linux 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
== check pips ===================================================
numpy (1.14.3)
numpydoc (0.7.0)
protobuf (3.5.2.post1)
tensorflow (1.8.0)
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.VERSION = 1.8.0
tf.GIT_VERSION = b'v1.8.0-3-gf91bd2f'
tf.COMPILER_VERSION = b'v1.8.0-3-gf91bd2f'
Sanity check: array([1], dtype=int32)
== env ==========================================================
LD_LIBRARY_PATH :/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
DYLD_LIBRARY_PATH is unset
== nvidia-smi ===================================================
Fri Feb  8 18:17:16 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     Off  | 00000000:01:00.0  On |                  N/A |
|  0%   46C    P8    16W / 200W |    383MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   30C    P2    58W / 250W |  11750MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
== cuda libs  ===================================================
/usr/local/cuda-9.0/doc/man/man7/libcudart.7
/usr/local/cuda-9.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.0/lib64/libcudart_static.a
/usr/local/cuda-9.0/lib64/libcudart.so.9.0.176
/usr/local/cuda-8.0/doc/man/man7/libcudart.7
/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.61
/usr/local/cuda-8.0/lib64/libcudart_static.a
== cat /etc/issue ===============================================
Linux  4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.5 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
== are we in docker =============================================
No
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== uname -a =====================================================
Linux 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
== check pips ===================================================
numpy               1.15.4
protobuf            3.6.1
tensorflow          1.9.0
tensorflow-fold     0.0.1
tensorflow-gpu      1.12.0
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.VERSION = 1.12.0
tf.GIT_VERSION = v1.12.0-0-ga6d8ffae09
tf.COMPILER_VERSION = v1.12.0-0-ga6d8ffae09
Sanity check: array([1], dtype=int32)
== env ==========================================================
LD_LIBRARY_PATH :/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
DYLD_LIBRARY_PATH is unset
== nvidia-smi ===================================================
Fri Feb  8 18:18:45 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     Off  | 00000000:01:00.0  On |                  N/A |
|  0%   44C    P8    16W / 200W |    383MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   30C    P2    58W / 250W |  11750MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
== cuda libs  ===================================================
/usr/local/cuda-9.0/doc/man/man7/libcudart.7
/usr/local/cuda-9.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.0/lib64/libcudart_static.a
/usr/local/cuda-9.0/lib64/libcudart.so.9.0.176
/usr/local/cuda-8.0/doc/man/man7/libcudart.7
/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.61
/usr/local/cuda-8.0/lib64/libcudart_static.a

@plooney, I am not sure that I understand your issue. Can you provide a more detailed explanation? Do you mean to say that your convolutional network produces slightly different outputs when trained using batches of slightly different sizes? That is supposed to happen. Batch sizes affect the values obtained by trainable parameters.

@Sudeepam97 Perhaps I am missing something. In the above example there is no training. I appreciate the effect of batch size on training. This  bug sounds similar
I am using a patch based approach to do image segmentation. When I have trained my model I break my images up into patches pass these through the CNN and combine the outputs to give the final output. The same model will give a different output depending on the batch size. The answer should not depend on the batch size as far as I understand it.
The example above shows that given a model and evaluating the model the batch size of 34 is distinct to that of 17 but should be the same. 42 is different to 21 but should be the same. This only happens for certain batch sizes.
In the following code I would expect all the elements in tensors to be the same. For lower values of integers some are.
tensors = []
full_batch = np.random.rand(154,86,86,86,1)
for batch_size in range(1,50):
num_batches = (len(full_batch)//batch_size)
if len(full_batch) % batch_size: num_batches += 1
y_out_list = []
print("Batch size of ", batch_size)
for i in range(0, num_batches):
    batch = full_batch[i*batch_size:min(len(full_batch), (i+1)*batch_size)]
    y_out = sess.run(model, feed_dict={X:batch})
    y_out_list = np.concatenate([y_out_list,y_out]) if not y_out_list == [] else y_out
tensors.append(y_out_list)

for i, t in enumerate(tensors):
print(i, np.all(t==tensors[0]))
The output is
0 True
1 True
2 True
3 True
4 True
5 True
6 False
7 True
8 True
9 True
10 True
11 True
12 True
13 True
14 True
15 True
16 True
17 True
18 True
19 True
20 False
21 True
22 True
23 True
24 True
25 True
26 True
27 True
28 True
29 True
30 False
31 False
32 False
33 False
34 False
35 False
36 False
37 False
38 False
39 False
40 False
41 False
42 False
43 False
44 False
45 False
46 False
47 False
48 False

Many floating point operations are not deterministic on GPU and they also have limited precision. So it's very common to not obtain bit-wise identical results.

@ppwwyyxx I seem to get consistent and expected results for certain batch sizes but not others. Is your explanation consistent with that observation?

That's reasonable as well. Tensorflow (or cudnn) may choose to use one implementation for certain batch sizes and another implementation for others.

@ppwwyyxx Consider the following
for i, t in enumerate(tensors):
    print(i, np.mean(np.abs(t))/np.mean(np.abs(tensors[0])))

The output is
0 1.0
1 1.0
2 1.0
3 1.0
4 1.0
5 1.0
6 1.0
7 1.0
8 1.0
9 1.0
10 1.0
11 1.0
12 1.0
13 1.0
14 1.0
15 1.0
16 1.0
17 1.0
18 1.0
19 1.0
20 1.0
21 1.0
22 1.0
23 1.0
24 1.0
25 1.0
26 1.0
27 1.0
28 1.0
29 1.0
30 0.98064667
31 1.0
32 1.0
33 0.9253961
34 1.0
35 1.0
36 1.0
37 1.0
38 1.0
39 0.98133874
40 0.9965034
41 0.866809
42 1.0
43 1.0
44 1.0
45 1.0
46 0.8182643
47 1.0
48 1.0
So fairly large differences for batch sizes of 34, 42, 46

That's some very interesting results to see. Running the same code on my environment gives the ratios all between 0.9999 and 1.0.
This seems to be more of an environment issue, then. It might help to upgrade cudnn because AFAIK cudnn mentions in several release notes about improvement/bugfix of numerical issues.

Upgraded to cudnn v7.4.2 with similar results
0 1.0
1 1.0
2 1.0
3 1.0
4 1.0
5 1.0
6 1.0
7 1.0
8 1.0
9 1.0
10 1.0
11 1.0
12 1.0
13 1.0
14 1.0
15 1.0
16 1.0
17 1.0
18 1.0
19 1.0
20 1.0
21 1.0
22 1.0
23 1.0
24 1.0
25 1.0
26 1.0
27 1.0
28 1.0
29 1.0
30 0.967046
31 1.0
32 1.0
33 0.9185082
34 1.0
35 1.0
36 1.0
37 1.0
38 1.0
39 0.97963786
40 0.992207
41 0.85242534
42 1.0
43 1.0
44 1.0
45 1.0
46 0.79884666
47 1.0
48 1.0

@ppwwyyxx Can you share details on your environment?

On CPU all are true and equal to 1

I've seen the same issue in training when using  tf.data.TFRecordDataset. For certain values of batch size fails to train

Running this on docker produces all 1s

Closing this issue as it is not bug/performance or build/install type. Please post this kind of support questions in Stackoverflow. There is a big community to help answering support type questions. Thanks!

Why isn't this a build or install issue? I have seen the same issue across two other machines. I don't see this is relevant to Stack overflow since it is not a programming problem. The problem is not present in Docker so I am using that as a workaround.


Add support for attribute 'agent'

[!] unknown attribute 'agent' for Risu::Models::Plugin.
Any chance this could be implemented in risu please?

Example:
</HostProperties>
<ReportItem port="0" svc_name="general" protocol="tcp" severity="0" pluginID="19506" pluginName="Nessus Scan Information" pluginFamily="Settings">
<agent>all</agent>
<description>This plugin displays, for each tested host, information about the scan itself :

Which version of risu are you using? I think 1.7.5 has support for the agent tag.

❯ risu -v
risu: 1.7.6
Ruby Version: 2.2.5
Rubygems Version: 2.6.2

User error possible. I'm using nessus 6.5.6 and just exported this from a scan done with a previous release of nessus. (minor not major)
I'll enable debug…

I removed the few references of <agent>all</agent> but there were only 25. Same result, so I think I'm misinterpreting the error.
NOTE! I'm using a historical (i.e. older) database (sqlite) — is there perhaps a table change or something that accommodated this agent piece that my database lacks? I can create a new database and test that.

(parenthetical; this seems to be working with a new database)

Yup. It's working as expected with a new database. Sorry for clutter. Possible enhancement request to update/upgrade a database? Or is that not typical usage? I use risu to make large roll-ups of individual scans pretty often.

I typically generate a database per engagement. I am looking to add diffing/trending inside of a single database (multiple scans) but I haven't done much work on that.


Fix Azure Bastion Subnet to be /26 (instead of /27)

Fixes: #342
When this reference implementation was created, /27 was the correct size for Azure Bastion.  The latest recommendation is /26 -- luckily enough room was left :)

Updated the AzureBastionSubnet subnet size, and the related IP table
Minor formatting & link changes on the IP table

Thanks for the review, @hallihan!


Project Setup Review

https://github.com/danielochuba/Group-capstone/projects/1
We have only Two people working on this project, namely @Gardimy, and @danielochuba

Hi @danielochuba and @danielochuba ,
Good job so far!
There are some issues that you still need to work on to go to the next project but you are almost there!

Highlights

Preapred Kanban Bord  ✔
Described Each Task ✔
Added Number Of point for each task ✔

Required Changes ♻

Great Job for assigning most of the tasks, I think you forget to assign group tasks for someone so please sign group tasks for one of you to fit the requirement,    👇

Optional suggestions
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you take them into account as they can make your code better.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hi Team
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉

Highlights
Linters are passing 👍
The project is running successfully 👍
Issues are well descriptive 👍
Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me @joel-kalema in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


adds new module for basic message handler injection

Signed-off-by: Matt Raffel<EMAIL_ADDRESS>

There are no unit tests at this time.
The integration tests are in protocol-aries, which is where these tests have lived for quite awhile.   Mocking the http calls is not straight forward due to how the dependency is used with a composition type.   If we want to unit test this, we have to restructure the code into a better format.
Creating unit tests with mocks will be easier once we replace the http calls with agent interactions (https://github.com/kiva/aries-guardianship-agency/issues/172)


[NIFI-6498] Set error event listener on both the TransformerFactory a…

…nd Transformer to enable logging errors when processing transformation instructions and when running the transformation itself. Also added ability to log errors from messages that are emitted with xsl:message/.

Summary
NIFI-6498
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 8
[ ] JDK 11
[ ] JDK 17

Licensing

[x] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

@exceptionfactory Thank you for taking the time and reviewing this PR. I would just like to clarify what you would like me to remove. I understand you would like me to remove method setMessageReceiver which has the Saxon specific code. Would you like me also to remove unit tests testMessageTerminate and testMessageNonTerminate also or should I keep them as they still demonstrate behavior that TransformXml exhibits when xsl:message is used with terminate and without terminate?

Thanks for the reply @dan-s1.
Although the testMessage methods include the xsl:message examples, they do not appear to assert any particular behavior, which makes sense given that it just ends up writing messages. For that reason, it seems best to remove those test methods and related files.
Reviewing the Saxon Configuration Features there is a reference to a message emitter class named MessageWarner that notifies the standard JAXP ErrorListener. Introducing this feature would still place a direct relationship to the Saxon implementation, so it still seems like it would be best to avoid for now, since xsl:message is a defined element used for testing.

@exceptionfactory They actually do assert behavior  when terminate is used whether the transform succeeds (terminate=no) or fails (terminate=yes).

Thanks @dan-s1, I was expecting something related to the message output, but on further review, I see the success and failure output scenarios, which seem worth maintaining.

@exceptionfactory So i ended up only removing the method setMessageReceiver. With that, the only issue which remains is when there is a warning from xsl:message for some reason its not redirected to use the ErrorListener configured. I did find the following Saxon issue which details a similar issue but not the exact one where it gives a lot of history behind Saxon's implementation on handling xsl:message.

@exceptionfactory I made the changes you requested. I am not sure though what to do as two of the ci builds timed out. Please advise. Thanks!

@exceptionfactory I just learnt something about the TestRunner that we could access logging that occurred during running a processor. Do you think I should add assertions to the unit tests I added that confirms there was logging when ever a compile time or runtime error occurs when transforming XML?

Do you think I should add assertions to the unit tests I added that confirms there was logging when ever a compile time or runtime error occurs when transforming XML?

If you are inclined to add a check for a log at a given level, that would be a helpful addition. It is not necessary to check the content of the message, but check for a warning or error log as appropriate could be useful.

@exceptionfactory Ok great! I will see if I can add that.

@exceptionfactory Do you mean just asserting  for example getErrorMessages on the MockComponentLog is not empty? I thought it also may be helpful to have the contents of the message similar to in the ticket

Error on line 17 column 5 SXXP0003: Error reported by XML parser: The element type "xsl:tomplate" must be terminated by the matching end-tag "</xsl:tomplate>".

in order to demonstrate the issue was fixed.

@exceptionfactory Do you mean just asserting for example getErrorMessages on the MockComponentLog is not empty? I thought it also may be helpful to have the contents of the message similar to in the ticket

Error on line 17 column 5 SXXP0003: Error reported by XML parser: The element type "xsl:tomplate" must be terminated by the matching end-tag "</xsl:tomplate>".

in order to demonstrate the issue was fixed.

Asserting an exact message makes the unit test more brittle when it comes to library changes. In this particular case, asserting that the message contains the word tomplate seems like a good approach.

@exceptionfactory I added the assertions. Please let me know if that is enough or if there should be more. Thanks!

@exceptionfactory I believe I took care of all of your requests.


Add bclschool.txt

Ofsted url: https://reports.ofsted.gov.uk/provider/27/102171

@sheyabernstein Please provide us with a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.
You can provide us with any of the following:

A copy of instruction (a hard-copy or electronic one) given to you by the school that tells you to use an email address in this email domain
An URL of a page at the official school website that says that students use this email domain
A reference to a recognized journal where an article was published by someone from the school, with explicit affiliation to the school, and e-mail address from the email domain
A WHOIS data that link the email domain to the school somehow
A copy of official letter from the school representative saying that the school provides e-mail addresses in this domain to the students
Anything else that can prove this fact in similar way, i.e. can be verified independently.

@philipto please see the attached file
letter.pdf

@sheyabernstein Thank you for the update! Pull request merged.


Mobile Web

Hi,
Will it work when embedded into Mobile Web like Cordova or platform like Flutter for mobile?
Thanks;

Hi,
It's just like a web page. I haven't tested it but if you can open a HTML file in a "web view" it can work.