Update ENet to use Godot sockets.

This PR adds an ENet implementation using Godot Sockets.
The files changed in enet library are:

host.c: one line to allow IPv6
protocol.c: three lines to allow IPv6
enet.h: Add a function and change ENetAddress structure to allow for IPv6, include the files that implement the socket functions for the library using Godot sockets (two new files godot.cpp and enet/godot.h).

It also adds non-blocking mode for PacketPeerUDP (only available via C++) so ENet should be almost as performant as with the raw sockets.
With this PR builtin enet gets full IPv4 and IPv6 support.
When compiling with builtin_enet=no the module will still work but will be IPv4 only as per the upstream library.
Fixes #6992
@punto- you might want to have a look at this :wink:
PS: You can test it with https://github.com/Faless/gd-bomber-3 which is an ugly 3.0 ported version of the multiplayer bomber in the official demo.

How does it underperform compared to enet with raw sockets? are we missing
some socket options?
On 8 March 2017 at 12:57, Fabio Alessandrelli<EMAIL_ADDRESS>wrote:

This PR adds an ENet implementation using Godot Sockets.
The files changed in enet library are:

host.c: one line to allow IPv6
protocol.c: three lines to allow IPv6
enet.h: Add a function and change ENetAddress structure to allow for
IPv6, include the files that implement the socket functions for the library
using Godot sockets (two new files godot.cpp and enet/godot.h).

It also adds non-blocking mode for PacketPeerUDP (only available via C++)
so ENet should be almost as performant as with the raw sockets.
With this PR builtin enet gets full IPv4 and IPv6 support.
When compiling with builtin_enet=no the module will still work but will
be IPv4 only as per the upstream library.
Fixes #6992 https://github.com/godotengine/godot/issues/6992
@punto- https://github.com/punto- you might want to have a look at this
😉
PS: You can test it with https://github.com/Faless/gd-bomber-3 which is
an ugly 3.0 ported version of the multiplayer bomber in the official demo.
You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/7985
Commit Summary

Add godot socket implementation
Convert ENetAddress host to 16 bytes to accomote IPv6
Update ENet module to support custom ENet lib
Allow non blocking UDP put_packet in C++.
Cleanup unused ENet files and updated thirdparty README

File Changes

M core/io/packet_peer_udp.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-0 (7)
M core/io/packet_peer_udp.h
https://github.com/godotengine/godot/pull/7985/files#diff-1 (4)
M drivers/unix/packet_peer_udp_posix.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-2 (3)
M modules/enet/SCsub
https://github.com/godotengine/godot/pull/7985/files#diff-3 (4)
M modules/enet/networked_multiplayer_enet.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-4 (36)
M modules/enet/networked_multiplayer_enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-5 (2)
M platform/windows/packet_peer_udp_winsock.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-6 (5)
M thirdparty/README.md
https://github.com/godotengine/godot/pull/7985/files#diff-7 (13)
M thirdparty/enet/enet/enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-8 (21)
A thirdparty/enet/enet/godot.h
https://github.com/godotengine/godot/pull/7985/files#diff-9 (42)
D thirdparty/enet/enet/unix.h
https://github.com/godotengine/godot/pull/7985/files#diff-10 (47)
D thirdparty/enet/enet/win32.h
https://github.com/godotengine/godot/pull/7985/files#diff-11 (57)
A thirdparty/enet/godot.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-12 (204)
M thirdparty/enet/host.c
https://github.com/godotengine/godot/pull/7985/files#diff-13 (2)
M thirdparty/enet/protocol.c
https://github.com/godotengine/godot/pull/7985/files#diff-14 (9)
D thirdparty/enet/unix.c
https://github.com/godotengine/godot/pull/7985/files#diff-15 (616)
D thirdparty/enet/win32.c
https://github.com/godotengine/godot/pull/7985/files#diff-16 (435)

Patch Links:

https://github.com/godotengine/godot/pull/7985.patch
https://github.com/godotengine/godot/pull/7985.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGVmPblL4us4Jo_kQBN5rojgKiqm0VkOks5rjxYegaJpZM4MXTJP
.

Well, there will be a little bit of c++ overhead on top of the socket
implementation (just because c++ is slower than c).
Beside that, with non-blocking UDP put_packet for enet (which is included
in this patch), we should get the same performances.
Honestly though I couldn't think of a way to reliably test performances
(beside doing empirical tests with the firewall dropping packets) so if you
have any idea I'll gladly help.
On Mar 8, 2017 22:49, "punto-"<EMAIL_ADDRESS>wrote:

How does it underperform compared to enet with raw sockets? are we missing
some socket options?
On 8 March 2017 at 12:57, Fabio Alessandrelli<EMAIL_ADDRESS>wrote:

This PR adds an ENet implementation using Godot Sockets.
The files changed in enet library are:

host.c: one line to allow IPv6
protocol.c: three lines to allow IPv6
enet.h: Add a function and change ENetAddress structure to allow for
IPv6, include the files that implement the socket functions for the
library
using Godot sockets (two new files godot.cpp and enet/godot.h).

It also adds non-blocking mode for PacketPeerUDP (only available via C++)
so ENet should be almost as performant as with the raw sockets.
With this PR builtin enet gets full IPv4 and IPv6 support.
When compiling with builtin_enet=no the module will still work but will
be IPv4 only as per the upstream library.
Fixes #6992 https://github.com/godotengine/godot/issues/6992
@punto- https://github.com/punto- you might want to have a look at
this
😉
PS: You can test it with https://github.com/Faless/gd-bomber-3 which is
an ugly 3.0 ported version of the multiplayer bomber in the official
demo.
You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/7985
Commit Summary

Add godot socket implementation
Convert ENetAddress host to 16 bytes to accomote IPv6
Update ENet module to support custom ENet lib
Allow non blocking UDP put_packet in C++.
Cleanup unused ENet files and updated thirdparty README

File Changes

M core/io/packet_peer_udp.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-0 (7)
M core/io/packet_peer_udp.h
https://github.com/godotengine/godot/pull/7985/files#diff-1 (4)
M drivers/unix/packet_peer_udp_posix.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-2 (3)
M modules/enet/SCsub
https://github.com/godotengine/godot/pull/7985/files#diff-3 (4)
M modules/enet/networked_multiplayer_enet.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-4 (36)
M modules/enet/networked_multiplayer_enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-5 (2)
M platform/windows/packet_peer_udp_winsock.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-6 (5)
M thirdparty/README.md
https://github.com/godotengine/godot/pull/7985/files#diff-7 (13)
M thirdparty/enet/enet/enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-8 (21)
A thirdparty/enet/enet/godot.h
https://github.com/godotengine/godot/pull/7985/files#diff-9 (42)
D thirdparty/enet/enet/unix.h
https://github.com/godotengine/godot/pull/7985/files#diff-10 (47)
D thirdparty/enet/enet/win32.h
https://github.com/godotengine/godot/pull/7985/files#diff-11 (57)
A thirdparty/enet/godot.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-12 (204)
M thirdparty/enet/host.c
https://github.com/godotengine/godot/pull/7985/files#diff-13 (2)
M thirdparty/enet/protocol.c
https://github.com/godotengine/godot/pull/7985/files#diff-14 (9)
D thirdparty/enet/unix.c
https://github.com/godotengine/godot/pull/7985/files#diff-15 (616)
D thirdparty/enet/win32.c
https://github.com/godotengine/godot/pull/7985/files#diff-16 (435)

Patch Links:

https://github.com/godotengine/godot/pull/7985.patch
https://github.com/godotengine/godot/pull/7985.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGVmPblL4us4Jo_
kQBN5rojgKiqm0VkOks5rjxYegaJpZM4MXTJP>
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285180438,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnBbsaPs5JwxyzsqQje-zPnFTs8uXqUks5rjyJkgaJpZM4MXTJP
.

Honestly dont think any kind of drop is really relevant
On Mar 8, 2017 3:16 PM, "Fabio Alessandrelli"<EMAIL_ADDRESS>wrote:
Well, there will be a little bit of c++ overhead on top of the socket
implementation (just because c++ is slower than c).
Beside that, with non-blocking UDP put_packet for enet (which is included
in this patch), we should get the same performances.
Honestly though I couldn't think of a way to reliably test performances
(beside doing empirical tests with the firewall dropping packets) so if you
have any idea I'll gladly help.
On Mar 8, 2017 22:49, "punto-"<EMAIL_ADDRESS>wrote:

How does it underperform compared to enet with raw sockets? are we missing
some socket options?
On 8 March 2017 at 12:57, Fabio Alessandrelli<EMAIL_ADDRESS>wrote:

This PR adds an ENet implementation using Godot Sockets.
The files changed in enet library are:

host.c: one line to allow IPv6
protocol.c: three lines to allow IPv6
enet.h: Add a function and change ENetAddress structure to allow for
IPv6, include the files that implement the socket functions for the
library
using Godot sockets (two new files godot.cpp and enet/godot.h).

It also adds non-blocking mode for PacketPeerUDP (only available via
C++)
so ENet should be almost as performant as with the raw sockets.
With this PR builtin enet gets full IPv4 and IPv6 support.
When compiling with builtin_enet=no the module will still work but will
be IPv4 only as per the upstream library.
Fixes #6992 https://github.com/godotengine/godot/issues/6992
@punto- https://github.com/punto- you might want to have a look at
this
😉
PS: You can test it with https://github.com/Faless/gd-bomber-3 which is
an ugly 3.0 ported version of the multiplayer bomber in the official
demo.
You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/7985
Commit Summary

Add godot socket implementation
Convert ENetAddress host to 16 bytes to accomote IPv6
Update ENet module to support custom ENet lib
Allow non blocking UDP put_packet in C++.
Cleanup unused ENet files and updated thirdparty README

File Changes

M core/io/packet_peer_udp.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-0 (7)
M core/io/packet_peer_udp.h
https://github.com/godotengine/godot/pull/7985/files#diff-1 (4)
M drivers/unix/packet_peer_udp_posix.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-2 (3)
M modules/enet/SCsub
https://github.com/godotengine/godot/pull/7985/files#diff-3 (4)
M modules/enet/networked_multiplayer_enet.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-4 (36)
M modules/enet/networked_multiplayer_enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-5 (2)
M platform/windows/packet_peer_udp_winsock.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-6 (5)
M thirdparty/README.md
https://github.com/godotengine/godot/pull/7985/files#diff-7 (13)
M thirdparty/enet/enet/enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-8 (21)
A thirdparty/enet/enet/godot.h
https://github.com/godotengine/godot/pull/7985/files#diff-9 (42)
D thirdparty/enet/enet/unix.h
https://github.com/godotengine/godot/pull/7985/files#diff-10 (47)
D thirdparty/enet/enet/win32.h
https://github.com/godotengine/godot/pull/7985/files#diff-11 (57)
A thirdparty/enet/godot.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-12 (204)
M thirdparty/enet/host.c
https://github.com/godotengine/godot/pull/7985/files#diff-13 (2)
M thirdparty/enet/protocol.c
https://github.com/godotengine/godot/pull/7985/files#diff-14 (9)
D thirdparty/enet/unix.c
https://github.com/godotengine/godot/pull/7985/files#diff-15 (616)
D thirdparty/enet/win32.c
https://github.com/godotengine/godot/pull/7985/files#diff-16 (435)

Patch Links:

https://github.com/godotengine/godot/pull/7985.patch
https://github.com/godotengine/godot/pull/7985.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGVmPblL4us4Jo_
kQBN5rojgKiqm0VkOks5rjxYegaJpZM4MXTJP>
.

—
You are receiving this because you authored the thread.

Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285180438,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABnBbsaPs5JwxyzsqQje-
zPnFTs8uXqUks5rjyJkgaJpZM4MXTJP>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285201547,
or mute
the thread
https://github.com/notifications/unsubscribe-auth/AF-Z2_nRBy3jSCfNLkwQOY7iw6RHQpvrks5rjzbGgaJpZM4MXTJP
.

Oh that's fine, I don't think the c++ overhead will be a huge deal.. I was
wondering more about features we might be missing, I know we don't have all
the config options of "fcntl" and stuff like that. I know we use
the TCP_NODELAY option, but there might be others that enet is using..
On 8 March 2017 at 15:18, Juan Linietsky<EMAIL_ADDRESS>wrote:

Honestly dont think any kind of drop is really relevant
On Mar 8, 2017 3:16 PM, "Fabio Alessandrelli"<EMAIL_ADDRESS>wrote:
Well, there will be a little bit of c++ overhead on top of the socket
implementation (just because c++ is slower than c).
Beside that, with non-blocking UDP put_packet for enet (which is included
in this patch), we should get the same performances.
Honestly though I couldn't think of a way to reliably test performances
(beside doing empirical tests with the firewall dropping packets) so if you
have any idea I'll gladly help.
On Mar 8, 2017 22:49, "punto-"<EMAIL_ADDRESS>wrote:

How does it underperform compared to enet with raw sockets? are we
missing
some socket options?
On 8 March 2017 at 12:57, Fabio Alessandrelli<EMAIL_ADDRESS>wrote:

This PR adds an ENet implementation using Godot Sockets.
The files changed in enet library are:

host.c: one line to allow IPv6
protocol.c: three lines to allow IPv6
enet.h: Add a function and change ENetAddress structure to allow for
IPv6, include the files that implement the socket functions for the
library
using Godot sockets (two new files godot.cpp and enet/godot.h).

It also adds non-blocking mode for PacketPeerUDP (only available via
C++)
so ENet should be almost as performant as with the raw sockets.
With this PR builtin enet gets full IPv4 and IPv6 support.
When compiling with builtin_enet=no the module will still work but will
be IPv4 only as per the upstream library.
Fixes #6992 https://github.com/godotengine/godot/issues/6992
@punto- https://github.com/punto- you might want to have a look at
this
😉
PS: You can test it with https://github.com/Faless/gd-bomber-3 which
is
an ugly 3.0 ported version of the multiplayer bomber in the official
demo.
You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/7985
Commit Summary

Add godot socket implementation
Convert ENetAddress host to 16 bytes to accomote IPv6
Update ENet module to support custom ENet lib
Allow non blocking UDP put_packet in C++.
Cleanup unused ENet files and updated thirdparty README

File Changes

M core/io/packet_peer_udp.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-0 (7)
M core/io/packet_peer_udp.h
https://github.com/godotengine/godot/pull/7985/files#diff-1 (4)
M drivers/unix/packet_peer_udp_posix.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-2 (3)
M modules/enet/SCsub
https://github.com/godotengine/godot/pull/7985/files#diff-3 (4)
M modules/enet/networked_multiplayer_enet.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-4 (36)
M modules/enet/networked_multiplayer_enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-5 (2)
M platform/windows/packet_peer_udp_winsock.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-6 (5)
M thirdparty/README.md
https://github.com/godotengine/godot/pull/7985/files#diff-7 (13)
M thirdparty/enet/enet/enet.h
https://github.com/godotengine/godot/pull/7985/files#diff-8 (21)
A thirdparty/enet/enet/godot.h
https://github.com/godotengine/godot/pull/7985/files#diff-9 (42)
D thirdparty/enet/enet/unix.h
https://github.com/godotengine/godot/pull/7985/files#diff-10 (47)
D thirdparty/enet/enet/win32.h
https://github.com/godotengine/godot/pull/7985/files#diff-11 (57)
A thirdparty/enet/godot.cpp
https://github.com/godotengine/godot/pull/7985/files#diff-12 (204)
M thirdparty/enet/host.c
https://github.com/godotengine/godot/pull/7985/files#diff-13 (2)
M thirdparty/enet/protocol.c
https://github.com/godotengine/godot/pull/7985/files#diff-14 (9)
D thirdparty/enet/unix.c
https://github.com/godotengine/godot/pull/7985/files#diff-15 (616)
D thirdparty/enet/win32.c
https://github.com/godotengine/godot/pull/7985/files#diff-16 (435)

Patch Links:

https://github.com/godotengine/godot/pull/7985.patch
https://github.com/godotengine/godot/pull/7985.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGVmPblL4us4Jo_
kQBN5rojgKiqm0VkOks5rjxYegaJpZM4MXTJP>
.

—
You are receiving this because you authored the thread.

Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285180438,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABnBbsaPs5JwxyzsqQje-
zPnFTs8uXqUks5rjyJkgaJpZM4MXTJP>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285201547,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AF-Z2_
nRBy3jSCfNLkwQOY7iw6RHQpvrks5rjzbGgaJpZM4MXTJP>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285202176,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGVmPcTtTLmp9A291We4-lxXi26PMi1gks5rjzdVgaJpZM4MXTJP
.

I know we don't have all the config options of "fcntl" and stuff like that.

I had a quick look at what enet actually uses.
In host.c from line 63 to 66.
It seems to me that the only meaningful options we are missing are SO_RCVBUF and SO_SNDBUF.
Also, of course O_NONBLOCK/FIONBIO which I really thought we were already setting on our UDP implementation but it was actually done to TCP only.
Windows/unix version of the current UDP implementation are slightly different in terms of "blocking" socket.
I'm afraid I will need to add another commit here to get proper non-blocking sockets. I'll keep you posted

Okay, now O_NONBLOCK is implemented properly for both _poll and put_packet.
Not sure if we should also implement SO_RCVBUF and SO_SNDBUF

Does the non blocking socket also doesn't block while the TCP handshake is
happening? I remember we did some hacking to insure that, that'd be nice to
keep
On 9 March 2017 at 12:17, Fabio Alessandrelli<EMAIL_ADDRESS>wrote:

Okay, now O_NONBLOCK is implemented properly for both _poll and put_packet
.
Not sure if we should also implement SO_RCVBUF and SO_SNDBUF
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/7985#issuecomment-285468653,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGVmPSxtarwutUrKTU2bBUNHmXTSmeJjks5rkF5cgaJpZM4MXTJP
.

Does the non blocking socket also doesn't block while the TCP handshake is happening?

TCP classes are left untouched, so yes, the handshake is still non blocking.
I only slightly changed the UDP implementation to explicitly set the socket as non-blocking on Unix, and allow ENet to use non-blocking write on all platforms.
By default, and via GDScript, put_packet will keep the same blocking behavior it has now and only ENet will use this feature


Training for multiple classes

Thank you for your work! I tried using it for training for more classes ( 6 classes ) on Lyft Dataset.  I updated input/bdd_input.py file to suit the needs of Lyft dataset.  It also required changing hypes/kittiBox.json to reflect new image dimensions. (height: 1024, width: 1248)
While trying to train, I ran into a few weird issues.

The loss function suddenly explodes and the program crashes with  OverflowError: value too large to convert to int.

Here is the sample output from training (showing exploding loss):

`2020-06-30 03:21:22,589 root INFO Step 0/140000: loss = 39.73; lr = 1.00e-05; 0.104 sec (per Batch); 48.1 imgs/sec
2020-06-30 03:21:23,009 root INFO       (raw)  Acc.: 0.12, Conf: 31.51, Box: 2.22, Weight: 0.38, Delta: 2.61
2020-06-30 03:21:23,009 root INFO    (smooth)  Acc.: 0.12, Conf: 31.51, Box: 2.22, Weight: 0.38, Delta: 2.61
2020-06-30 03:21:23,915 root INFO Drawing encoded images (data utils)

2020-06-30 03:22:54,322 root INFO Step 200/140000: loss = 32.08; lr = 1.00e-05; 1.826 sec (per Batch); 2.7 imgs/sec
2020-06-30 03:22:54,481 root INFO       (raw)  Acc.: 0.76, Conf: 41.94, Box: 1.88, Weight: 0.38, Delta: 0.05
2020-06-30 03:22:54,481 root INFO    (smooth)  Acc.: 0.15, Conf: 32.03, Box: 2.20, Weight: 0.38, Delta: 2.48
2020-06-30 03:23:15,353 root INFO Drawing encoded images (data utils)

2020-06-30 03:24:26,125 root INFO Step 400/140000: loss = 772.73; lr = 1.00e-05; 1.268 sec (per Batch); 3.9 imgs/sec
2020-06-30 03:24:26,274 root INFO       (raw)  Acc.: 0.85, Conf: 28.31, Box: 1.34, Weight: 0.38, Delta: 1316.65
2020-06-30 03:24:26,275 root INFO    (smooth)  Acc.: 0.18, Conf: 31.85, Box: 2.16, Weight: 0.38, Delta: 68.19
2020-06-30 03:25:08,447 root INFO Drawing encoded images (data utils)

2020-06-30 03:25:59,540 root INFO Step 600/140000: loss = 399236.34; lr = 1.00e-05; 0.850 sec (per Batch); 5.9 imgs/sec
2020-06-30 03:25:59,712 root INFO       (raw)  Acc.: 0.85, Conf: 19.65, Box: 1.41, Weight: 0.38, Delta: 314026.66
2020-06-30 03:25:59,715 root INFO    (smooth)  Acc.: 0.22, Conf: 31.24, Box: 2.12, Weight: 0.38, Delta: 15766.12
2020-06-30 03:27:03,137 root INFO Drawing encoded images (data utils)

2020-06-30 03:27:32,453 root INFO Step 800/140000: loss = 19764840.00; lr = 1.00e-05; 0.419 sec (per Batch); 11.9 imgs/sec
2020-06-30 03:27:32,613 root INFO       (raw)  Acc.: 0.85, Conf: 23.55, Box: 1.84, Weight: 0.39, Delta: 21771650.00
2020-06-30 03:27:32,614 root INFO    (smooth)  Acc.: 0.25, Conf: 30.85, Box: 2.11, Weight: 0.38, Delta: 1103560.25

2020-06-30 03:28:57,191 root INFO Step 1000/140000: loss = 166476288.00; lr = 1.00e-05; 1.692 sec (per Batch); 3.0 imgs/sec
2020-06-30 03:28:57,340 root INFO       (raw)  Acc.: 0.85, Conf: 29.38, Box: 2.21, Weight: 0.39, Delta: 186277808.00
2020-06-30 03:28:57,341 root INFO    (smooth)  Acc.: 0.28, Conf: 30.78, Box: 2.11, Weight: 0.38, Delta: 10362273.00
2020-06-30 03:28:57,392 root INFO Drawing encoded images (data utils)`

The bounding boxes start looking like dots in images denoting something is definitely wrong.

At 1st iteration, predicted boxes look normal:
predicted_boxes

Ground_truth

At 3rd Iteration, predicted boxes start looking like dots.
predicted_boxes

Ground_truth

Have you by any chance encountered something similar while dealing with the training.
Any insights into this from your expertise would be very useful.

Maybe the issue is the format of your bounding boxes? I have not looked into the Lyft dataset, but sometimes the representation of the bounding boxes are different between different datasets. Make sure that the training data from Lyft corresponds to the Kitti format, and if not, you need to convert it to the right format (which is shown for BDD in bdd_input.py). In addition, you might need to change the eval to suit your needs (bdd_eval.py also contains a list CLASSES which contains the classes you want to evaluate).
Also, the projected ground truth boxes does not seem totally accurate, but I can not say for sure without looking at the training data. Can you verify that the ground truth boxes are projected the same way independent of this project?

It seems like the formatting for the bounding boxes is consistent with the bdd_dataset. To reduce the complexity and narrow done the source of error, I tried training KittiBox with a single class(car detector) and it works fine.
The output looks like

This makes me think, if the error is creeping not from data loading but somewhere else. Did you try training a multi-class detector using this branch? Do you think there might me some modifications needed to the file decoder/fastbox.py ? Any suggestions would be really appreciated.
I also verified the projection independent of the project and it looks okay.


New example with a physics engine like Box2D

e.g. https://github.com/ByteArena/box2d

Blocker: https://github.com/ByteArena/box2d/issues/6

you could rewrite one of these examples from my Chipmunk2D port instead: https://github.com/jakecoffman/cp/tree/master/examples

@jakecoffman Just out of curiosity, what's the difference from https://github.com/vova616/chipmunk ? Yours looks better since yours is a little more active project.

Back before I did the port I tried using vova616 to implement the tank demo in the official Chipmunk2D examples and I was getting strange collision results. The tanks would collide and then spin uncontrollably. So I think it might not be an accurate or complete project. There are some missing joints as well.
Mine has all of the examples implemented and I compared them to the C version for accuracy.

I made this if you are interested in copying it into your project as a physics demo:
https://github.com/jakecoffman/cpebiten
The gopherjs version is really slow for some reason!

That's really awesome! I think the calculation is too heavy for browsers and wasm might solve this in the future.

Is it OK to adopt Apache 2.0 License to cpebiten?
Would you mind creating a PR to add your cpebiten to Ebiten's examples? If you are busy, I'm fine with doing that instead of you.

Apache 2.0 is good. I can make a PR right now!

Close per https://github.com/hajimehoshi/ebiten/pull/572

Looks like copying objects is the biggest CPU consumer.


Integration incompatible with latest HASS stable 2023.4.6

Description
Hello,
I am trying to install this integration using HACS, but it requires minimum version 2023.5.0.
You are running Home Assistant version '2023.4.6', but this repository requires minimum version '2023.5.0' to be installed.
I see that 2023.5.0 is still in beta. Release 5.1.3 states that it is compatible with 2023.5.0, but is it expected to be a minimum requirement? Is there any way to install the latest version of the integration on 2023.4.6?
When I click download, it spins forever and nothing ever happens.
Thank you for your help!
Reproduction Steps

Go to HACS
Look for Tapo intergration and try to install it
Download takes forever and error message showing that integration is incompatible.

Expected behavior
Integration being compatible and installing properly on latest stable version
If applicable, add error logs.
You are running Home Assistant version '2023.4.6', but this repository requires minimum version '2023.5.0' to be installed.
Device Firmware
1.3.3 Build 230228 Rel.79740n(4555)
Using stream component
Yes
Does camera work via official integrations?
N/A
Camera has all attributes filled out in developer tools
Yes
HASS Environment
Hass OS
Search for similar issues
Yes
Additional information
No response

Yes that is correct. Use 5.1.2 with april release of HA.


add other tools

[ ] TPI
[ ] GTO
[ ] MLEM
[ ] CML? either as per https://github.com/iterative/setup-cml/blob/v1/src/utils.js or just download latest binary from https://github.com/iterative/cml/releases

sounds good, should we start with TPI / LEO though, first? (CML is more about CI/CD, not sure what use it would be to use in devcontainer?)

No plan to to this atm unfortunately. And it's not clear if this is needed tbh.


Fix material caching for canvases

Cesium would incorrectly use an HTMLCanvasElement as the key to the Material texture cache dictionary.  This means that if you had multiple canvas-based materials, we would only ever use the first one that was added (because the cache lookup would always succeed even if the canvases were difference).
This change uses toDataURL to store canvas elements in the cache.  This is sub-optimal since toDataURL is slow and also creates huge string, but the best we can do in the short term.  We need a good overlal long term strategy for dealing with canvases, but that's outside the scope of this PR.
Fixes #2821

Superseded by #2850


Check for app updates and notify user if update is available

overview
The app should check for updates and notify the user if an update is available
behavior
Current behavior:

The user must check if a new version of the app is available, and download manually.
They must also manually push server code to the robot.

Desired behavior:

The app should check if a new version of the app is available, and prompt the user to download and install it (including displaying a changelog). This should install the new version of the app.
When the app connects to a robot, it should check the version of the server running on the robot. If that version is lower than the version of the app, the app should prompt the user whether or not to update the software on the robot. If they say "Yes", the app should download the server code and push it onto the robot.

Waiting on #527

Do we want to consider splitting this into multiple tickets (or generating a checklist)? I think front-end vs back-end update would be reasonable. Based on when I was scoping this out, I saw the following acceptance criteria for "updates":
Front-end

App should be able to determine if itself is out of date
App should prompt the user to update itself if out of date

Prompt should include release metadata (e.g. changelog)

Back-end

API should make versions available to the app

API version
Compute version
Smoothie version

App should prompt user to update if any backend versions are out of date

Prompt should include update metadata
Could be a ticket per backend component

API should accept a new version of the API from the app to install and run
API should accept a new version of smoothie to install and run
API should allow the user to opt-in or out of a compute update

Mechanism TBD and likely needs its own ticket

@mcous I like your suggestion to split this into FE and BE stories.
I'll split out the FE work into separate ticket(s). @btmorr, as far as BE work, #641 exposes API and FW versions, but I can't find any previous updates adding a feature for the API to accept new versions of the API and/or Smoothie to install and run. Could you confirm we still need this (and I'll create a ticket)?


Port 7878 closed(???)

Hello nroi,
First, thank you for flexo, which I'm attempting to use for the first time.  I appreciate the name and hope you'll adopt a suitable cartoon icon as the project logo, assuming no copyright issues....
I am a long time Archer, but I feel like a total n00b.  I installed flexo-git from AUR.  The only flexo.toml edit I made was to set low_speed_limit.  'port = 7878' is unchanged in flexo.toml.  I systemctl started and enabled flexo, and it's running --
$ systemctl status flexo | cat
● flexo.service - central pacman cache
Loaded: loaded (/usr/lib/systemd/system/flexo.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2020-12-05 11:22:52 EST; 4s ago
Main PID: 66408 (flexo)
Tasks: 1 (limit: 35952)
Memory: 2.4M
CGroup: /system.slice/flexo.service
└─66408 /usr/bin/flexo
Dec 05 11:22:52 vagente595 systemd[1]: Started central pacman cache.
Dec 05 11:22:52 vagente595 flexo[66408]: [2020-12-05T16:22:52.049Z INFO  flexo] Will switch mirror if download speed falls below 1.20 MiB/s
But port 7878 is closed :( --
$ nmap -p 7878 localhost
Starting Nmap 7.91 ( https://nmap.org ) at 2020-12-05 11:24 EST
Nmap scan report for localhost (<IP_ADDRESS>)
Host is up (0.00010s latency).
Other addresses for localhost (not scanned): ::1
PORT     STATE  SERVICE
7878/tcp closed owms
Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
iptables, ip6tables and nftables services aren't enabled or running.  I don't have any iptables or nftables packet filtering in place --
$ sudo iptables -vS
-P INPUT ACCEPT -c 16745 3135815
-P FORWARD ACCEPT -c 0 0
-P OUTPUT ACCEPT -c 18248 1334084
$ sudo ip6tables -vS
-P INPUT ACCEPT -c 30266 49863949
-P FORWARD ACCEPT -c 0 0
-P OUTPUT ACCEPT -c 27922 2447847
$ sudo nft list ruleset
$
$ sudo nft list tables
$
To add insult to injury, I installed pacoloco-git and systemctl started that after stopping flexo, and that port is open --
$ nmap -p 9129 localhost
Starting Nmap 7.91 ( https://nmap.org ) at 2020-12-05 11:36 EST
Nmap scan report for localhost (<IP_ADDRESS>)
Host is up (0.00013s latency).
Other addresses for localhost (not scanned): ::1
PORT     STATE SERVICE
9129/tcp open  unknown
Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
... so I don't appear to have any networking misconfiguration.  This is such a beginner problem!  Why am I an idiot?  Thank you for any help you can provide :)
Clemmitt Sigler

I guess you're just a bit impatient: Flexo performs latency test at the first start, so it does not immediately listen on the port. Could you try after maybe 3 minutes or so? You should then see messages like the following in the log:
Dec 05 17:51:04 archWS flexo[21332]: [2020-12-05T16:51:04.531Z INFO  flexo] Primary mirror: "https://archlinux.mailtunnel.eu/"
Dec 05 17:51:04 archWS flexo[21332]: [2020-12-05T16:51:04.610Z INFO  flexo::mirror_flexo] Retrieved 5085 files with a total size of 18.08 GiB from local file system.

The next start of Flexo should be speedier, because after the first latency test, only a select few mirrors are tested because Flexo only considers mirrors from those locations that were fast during the first latency test.
I guess I could somehow improve this from a usability perspective. Even if it "works as intended", it's understandable that users are puzzled if nothing happens after the first startup for a minute or so.

Or maybe there's something else going on… Could you please post the output of:
pacman -Qi flexo-git

Hi,
Me, impatient? ... How did you know?   The process seems to die then auto-restart (I'm not an expert on systemd) --
$ systemctl status -n 50 flexo | cat
● flexo.service - central pacman cache
Loaded: loaded (/usr/lib/systemd/system/flexo.service; disabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sat 2020-12-05 12:14:15 EST; 404ms ago
Process: 71971 ExecStart=/usr/bin/flexo (code=exited, status=1/FAILURE)
Main PID: 71971 (code=exited, status=1/FAILURE)
This seems to happen over and over again.  After I stop flexo I see this --
Dec 05 12:15:51 vagente595 systemd[1]: Started central pacman cache.
Dec 05 12:15:51 vagente595 flexo[72177]: [2020-12-05T17:15:51.928Z INFO  flexo] Will switch mirror if download speed falls below 1.20 MiB/s
Dec 05 12:15:56 vagente595 flexo[72177]: [2020-12-05T17:15:56.066Z ERROR flexo] Unable to find remote mirrors that match the selected criteria.
Dec 05 12:15:56 vagente595 systemd[1]: flexo.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 12:15:56 vagente595 systemd[1]: flexo.service: Failed with result 'exit-code'.
Dec 05 12:15:59 vagente595 systemd[1]: flexo.service: Scheduled restart job, restart counter is at 79.
Dec 05 12:15:59 vagente595 systemd[1]: Stopped central pacman cache.
Dec 05 12:15:59 vagente595 systemd[1]: Started central pacman cache.
Dec 05 12:15:59 vagente595 flexo[72195]: [2020-12-05T17:15:59.178Z INFO  flexo] Will switch mirror if download speed falls below 1.20 MiB/s
Dec 05 12:16:03 vagente595 flexo[72195]: [2020-12-05T17:16:03.914Z ERROR flexo] Unable to find remote mirrors that match the selected criteria.
Dec 05 12:16:03 vagente595 systemd[1]: flexo.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 12:16:03 vagente595 systemd[1]: flexo.service: Failed with result 'exit-code'.
Dec 05 12:16:07 vagente595 systemd[1]: flexo.service: Scheduled restart job, restart counter is at 80.
Dec 05 12:16:07 vagente595 systemd[1]: Stopped central pacman cache.
80 restarts :\  I did nothing but use yay to build and install flexo-git, then add low_speed_limit to flexo.toml.  I didn't screw around with any file permissions under /var/cache/flexo, or anything else like that :(
I have even tried logging out and back in again in case it had to do with /etc/passwd, /etc/group.
I would be very happy to help debug, please let me know how I can help :)  And thank you for everything!
Clemmitt

Could you first upgrade Flexo and try again? I pushed a new version just a few minutes before you have opened this issue. pacman -Qi flexo-git should show the following:
Version         : 1.0.8.r0.g9a02658-1

Hi,
OK, with the new version changed flexo is no longer loop crashing.  Here's systemctl status --
$ systemctl status -n 50 flexo | cat
● flexo.service - central pacman cache
Loaded: loaded (/usr/lib/systemd/system/flexo.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2020-12-05 12:31:33 EST; 3min 37s ago
Main PID: 83760 (flexo)
Tasks: 1 (limit: 35952)
Memory: 2.3M
CGroup: /system.slice/flexo.service
└─83760 /usr/bin/flexo
Dec 05 12:31:33 vagente595 systemd[1]: Started central pacman cache.
Dec 05 12:31:35 vagente595 flexo[83760]: [2020-12-05T17:31:35.504Z INFO  flexo] No cached latency test results available. Continue to run latency tests on all mirrors.
Dec 05 12:32:06 vagente595 flexo[83760]: [2020-12-05T17:32:06.005Z INFO  flexo] Primary mirror: "https://mirror.osbeck.com/archlinux/"
Dec 05 12:32:06 vagente595 flexo[83760]: [2020-12-05T17:32:06.006Z INFO  flexo::mirror_flexo] Retrieved 0 files with a total size of 0.00 B from local file system.
TCP port 7878 is open now.  But "0 files from local file system" seems strange because my local /var/cache/pacman/pkg/ is 16 GiB in size.
I can now run pacman with the flexo proxy cache....  I've got a /var/cache/flexo/state/latency_test_results.json file, but it didn't seem to proxy cache anything although 4 pkgs were upgraded --
$ find /var/cache/flexo/ ! -type d -print | sort
/var/cache/flexo/state/latency_test_results.json
I'm afraid I'm being stupid again.  My first Server line in mirrorlist is --
Server = http://localhost:7878/$repo/os/$arch
Please let me know if PEBKAC :\
Clemmitt

Hi again nroi,
Perhaps this issue should be closed as fixed since the TCP port issue is solved.  I will flag for closing with this comment.
I will open a new issue if flexo isn't proxy caching.  Again, thank you for your timely support :)
Clemmitt

It looks like you previously ran into this bug: #23
I'm glad this works now.
I think you're mixing up two different directories: Pacman uses the CacheDir configured in /etc/pacman.conf. All files that are downloaded by pacman are stored in this directory. If a file already exists in this directory, pacman isn't going to download it. Flexo on the other hand uses the cache_directory configured in /etc/flexo/flexo.toml. It doesn't care about pacman's CacheDir.
So the message Retrieved 0 files with a total size of 0.00 B from local file system. initially appears because no files are present in Flexo's cache_directory, which is different from /var/cache/pacman/pkg.
You can try the following:

Follow the log with journalctl -f --unit=flexo
Download a package that you haven't downloaded before, e.g. pacman -S --downloadonly cowsay. If this works as intended, you should now see something like Attempt to create file: "/var/cache/flexo/pkg/extra/os/x86_64/cowsay-3.04-2-any.pkg.tar.zst" in your log. This means that Flexo has downloaded the file from a remote mirror.
Remove the file from you pacman's CacheDir (not the flexo cache directory). By default, this is /var/cache/pacman/pkg/.
Download the same file again. If this also works as intended, then you should see something like the following in your log: Request served: "extra/os/x86_64/cowsay-3.04-2-any.pkg.tar.zst", but you shouldn't see the message Attempt to create file: […] appear a second time in the log, because it has already been created. This means Flexo has served the file from its cache.

Hi one last time,
Just to follow up, flexo is now working for me as intended :)  Yes, after I posted that reply I wondered if that message was referencing files under /var/cache/flexo/pkg.  So, once I did things correctly the newly installed ffmpeg pkg is cached as /var/cache/flexo/pkg/extra/os/x86_64/ffmpeg-2:4.3.1-3-x86_64.pkg.tar.zst
Downloading that pkg was very slow :(  Perhaps I need to edit flexo.toml to set 'mirror_selection_method = "predefined"' and populate the 'mirrors_predefined = []' list(?).  I'm spoiled because I use yay with powerpill on my boxes and have an excellent mirrorlist produced via rankmirrors.
Thank you again for your nearly instantaneous support :)
Clemmitt

Downloading that pkg was very slow :(

Have you tried increasing the low_speed_limit setting?

Perhaps I need to edit flexo.toml to set 'mirror_selection_method = "predefined"' and populate the 'mirrors_predefined = []' list(?).

This could also solve your problem, but this would be more of a workaround than a fix. You would have to manually edit this list when a mirror is put offline. The  low_speed_limit is used to handle the case when you got unlucky and have selected a slow mirror, so maybe try that first.

Hi again,

The low_speed_limit is used to handle the case when you got unlucky and have selected a slow mirror, so maybe try that first.

Aha!  OK, that makes sense.  I will go forth and do :)  Again, many thanks!
Clemmitt

If the downloads are still slow even with low_speed_limit set to an appropriate value, then I consider this a bug, so feel free to open a new issue in case that happens.


Merge Sort in Python

Description
This issue will let me add a solution to important merge sort in Python.

@thesmartdeveloperr Issue created, Assign me this. You may kindly add the hacktoberfest label now, and if the PR seems to be fine to you, do add the label hacktoberfest-accepted  before merging as per the general way. Thank you.

sure @TishaJhabak1014  go ahead. label will directly be added to the PR..


no more alias_method_chain

also drop rails 2/3 support, and require ruby 2

@maneframe @roor0 review plz

+1 lgtm


Dev dataprotection microsoft.data protection 2022 03 31 preview

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[x] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific language SDKs and CLIs that must be refreshed after the swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[ ] No refresh required for updates in this PR

Contribution checklist:

[ ] I commit to follow the Breaking Change Policy of "no breaking changes"
[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[x] Check this box if any of the following apply to the PR so that label "WaitForARMFeedback" will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.

[ ] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in a stable version
[ ] Removing properties in a stable version
[ ] Removing API version(s) in a stable version
[ ] Updating API in a stable or public preview version with Breaking Change Validation errors
[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

This seems to be an extension resource. How is it created? I don't see a PUT call.
Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656, deletion_comment = False)

We currently only intend to support to list (GET) on this resource via extension resource.
Creation of this extension resource in the backend is supported via a PUT on the non-extension proxy resource in our RP.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

Is this only the scope supported? i.e. can backupInstances be created for a nested resource like Microsoft.SQL/servers/databases? If so, the path should be more generic like '/{resourceId}/providers/Microsoft.DataProtection/backupInstances'

Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656ddbaca941037dac982fd4305b78cdf634, deletion_comment = False)

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

You should support GET for an individual resource as well.

Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656ddbaca941037dac982fd4305b78cdf634, deletion_comment = False)

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

This seems to be an extension resource. How is it created? I don't see a PUT call.
Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656, deletion_comment = False)

@jorgecotillo , We currently only intend to support to list (GET) on this resource via extension resource. Creation of this extension resource in the backend is supported via a different API -
https://github.com/amchandn/azure-rest-api-specs/blob/ab02656ddbaca941037dac982fd4305b78cdf634/specification/dataprotection/resource-manager/Microsoft.DataProtection/stable/2022-03-01/dataprotection.json#L1067

The PUT in the link is for creating an extension resource for a resource group. That API cannot be used for this path, which scope is a resource not a RG. You will need to add PUT under this path as well.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

Is this only the scope supported? i.e. can backupInstances be created for a nested resource like Microsoft.SQL/servers/databases? If so, the path should be more generic like '/{resourceId}/providers/Microsoft.DataProtection/backupInstances'
Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656, deletion_comment = False)

Thanks @j5lim , I've incorporated the changes as you suggested.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

You should support GET for an individual resource as well.
Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656, deletion_comment = False)

@j5lim , we currently plan to support only list on this extension resource named 'backupInstances' so that we can author ARM policies over the root resource and take appropriate action depending on the response of the list 'backupInstances' proxy resource.
We will introduce support for PUT and GET on this extension resource in upcoming API versions.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}/providers/Microsoft.DataProtection/backupInstances": {

This seems to be an extension resource. How is it created? I don't see a PUT call.
Refers to: specification/dataprotection/resource-manager/Microsoft.DataProtection/preview/2022-03-31-preview/dataprotection.json:1016 in ab02656. [](commit_id = ab02656, deletion_comment = False)

@jorgecotillo , We currently only intend to support to list (GET) on this resource via extension resource. Creation of this extension resource in the backend is supported via a different API -
https://github.com/amchandn/azure-rest-api-specs/blob/ab02656ddbaca941037dac982fd4305b78cdf634/specification/dataprotection/resource-manager/Microsoft.DataProtection/stable/2022-03-01/dataprotection.json#L1067

The PUT in the link is for creating an extension resource for a resource group. That API cannot be used for this path, which scope is a resource not a RG. You will need to add PUT under this path as well.

@j5lim , the resource which is referred in the link that I've shared is a proxy child resource over Microsoft.DataProtection/BackupVaults tracked resource. The creation of entities in our service happens when a PUT call is made for Microsoft.DataProtection/BackupVaults/{vaultName}/BackupInstances/{backupInstanceName}.
This creation creates a mapping in the service that allows entities to be returned in the list backupInstances API for the proxy resource as well.
The reason to do this is -
we currently plan to support only list on this extension resource named 'backupInstances' so that we can author ARM policies over the root resource and take appropriate action depending on the response of the list 'backupInstances' extension resource.
We will introduce support for PUT and individual GET on this extension resource in upcoming API versions.

There are issues with your approach, RBAC being one of the most importa t ones, how do you ensure that a user doing a collection GET doesn't get access to resources he/she is not meant to?
Also, in our wiki we call out incomplete CRUD as a violation to our best practices,  proxy resources should at least model a collection GET and single GET.

{resourceId}

There are issues with your approach, RBAC being one of the most importa t ones, how do you ensure that a user doing a collection GET doesn't get access to resources he/she is not meant to?
Also, in our wiki we call out incomplete CRUD as a violation to our best practices, proxy resources should at least model a collection GET and single GET.

There are issues with your approach, RBAC being one of the most importa t ones, how do you ensure that a user doing a collection GET doesn't get access to resources he/she is not meant to?
Also, in our wiki we call out incomplete CRUD as a violation to our best practices, proxy resources should at least model a collection GET and single GET.

thanks @jorgecotillo for this feedback. We are discussing internally in our team how we can enforce proper RBAC in case of LIST on this extension resource.
We'll also discuss the feasibility of supporting single GET in this API version itself.
We have planned that when we support PUT for this resource in the future, we'll leverage LinkedAccessCheck to ensure that no RBAC violation occurs during PUT.
@j5lim / @jorgecotillo do let us know your thoughts on this.

The RP should always do RBAC check on the linked resources during PUT (e.g. via linked access check).
For the collection GET, the user will require Microsoft.DataProtection/backupInstances/read permission on the request scope, which is different than the RBAC for backupVaults/backupInstances as Jorge pointed out. So it will be the RP's responsibility to check if the user has Microsoft.DataProtection/backupVaults/backupInstances/read permission on the backupInstances being returned from the collection GET. You can't use ARM's linked access check for this, but you can do via checkaccess API (search 'checkaccess' in the ARM wiki for details).

@msyyc / @Azure/swagger-specs-pull-request-approvers / @TechTress98 , can you please merge this PR now?

after contact with go owner, will ignore the go breaking change


purge more packages that are not dependencies

I noticed that you're making exciting progress on the core image generation.
I wanted to share my purge list that I use locally.
It might need more discussion and cleanup, but code is always a good way to start the discussion around next steps.

Can you kindly rebase?
I added an extra log line to the build file, mainly to see the file size of the generated images.

The generated images are only about 15-20MB lighter. I was hoping to the difference would be wider than that.
Probably because the image is already the minimal cloud image and most of the supposedly removed packages were not present.

Thanks as usual :)


How to encode comments

We need to specify how to deal with different kinds of comments.  Here an inventory and suggestion of process:
a.	Comment on a result of analysis - may be done .note to the result?  But there are different cases possible:
a.i.	result field itself maybe comment or text (e.g. "Positive" or "No longer performed by lab; replaced by analyses XXX"
a.ii.	Comment in commentfield may be combined with number (or comment0 in result field)
b.	Comment regd. reference values (e.g. age-specifics)  – in .text to the reference values?
c.     We often created dummy test codes to contain comments.  E.g. serum aspect.  Is this allowed? There are no LOINC codes for these dummies - will we allow every lab to use their own.  Can we use this system?  If not, how do we encode such comment?
d.    Comments regarding the laboratory (not the results). E.g. status of accreditation.  We currently use dummy tests for that.  Acceptable? If not, then how?
e.    Comments regarding tests performed by 3rd parties.  We currently do that by means of flags (codes) next to results, and use a dummy test to explain the legend.  Allowed?  If not then how?
f.    Comments regarding administrative issue, e.g. missing information like gender, DOB.  Today we use dummy test codes.  Allowed? If not, then how?
g.	There is also a .conclusionCode - to use instead of dummy analyses? Or a combination?   Dummy analysis have the advantage they can be linked to a sample (useful e.g. for serum aspect) - .conclusionCodes cannot do this.
Do we want/need to agree on standards for the above? Or allwo every lab to do their own way?

I think we could add some narrative, but it would be better to add such use cases to our specification, and provide the example FHIR resource instances.
If we have some narrative text, we can explain the mechanism (this may be subject to interpretation)
If we list the use cases, we can make it much more concrete by examples
If we actually provide the examples, implementers will have more clarity.

@tomto66 Can you provide an example for each case (testing dir in project) and add a reference here?

Yes we will provide examples!
On Wed, Nov 10, 2021 at 12:40 PM Bart Decuypere (eHealth) <
@.***> wrote:

@tomto66 https://github.com/tomto66 Can you provide an example for each
case (testing dir in project) and add a reference here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hl7-be/hl7-be-fhir-laboratory-report/issues/23#issuecomment-965049476,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKWKYJTIWPKS7LOFFYPPU5DULJK4LANCNFSM5HEN3BNQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

comment by Alexis Van Zeveren:
•	For comments, the proper FHIR elements must be used (eg. notes), not "dummy results", either the "narrative". Tom/Macsys will try to implement this in examples.
This is issue #23 in Github. It seems this decision is not yet clearly documented in the comments.


Multiverse & Terra Enhancement

Normally with plugins you can do the following command to generate a world "/mv create (Worldname) NORMAL -g (Plugin)" And i think Terra should support this.

Terra already supports this.
/mvcreate world_name world_type -g Terra:CONFIG_ID


MSBuild can report the wrong assembly path when loading a task

MSBuild diagnostic logs report that a task was loaded from an assembly at a specific path, but the task can be loaded from a different assembly if that assembly is found in DEVPATH first. It appears that MSBuild is not taking DEVPATH into consideration, so it assumes the assembly location is going to be the relative to the directory of the project file that has the  declaration. I believe the exact code in question can be seen here. I ended up spending a couple hours investigating why I was getting a certain build error because I assumed that the path listed in the logs was accurate. I only discovered it was wrong after exhausting all other possibilities and turning on fusion logging to confirm.
Details/example:
I’m using a packaged version of msbuild which contains msbuild.exe and the targets/tasks. MSBuild.exe lives at E:\packages\MsBuild\v14.0\Bin\MSBuild.exe, and E:\packages\MsBuild\v14.0\Bin\Microsoft.Common.tasks contains:
<UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc" AssemblyFile="Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />

E:\packages\MsBuild\v14.0\Bin\Microsoft.Build.Tasks.CodeAnalysis.dll is the expected file where the Vbc task lives, and the MSBuild diagnostic log says that it is using the assembly from that location:
Using "Vbc" task from assembly "e:\packages\MsBuild\v14.0\bin\Microsoft.Build.Tasks.CodeAnalysis.dll".

Unfortunately, due to DEVPATH being set up in my environment, the dll is actually loaded from a different location, and msbuild ends up reporting the wrong path. Fusion logs:
*** Assembly Binder Log Entry  (4/19/2016 @ 2:59:40 PM) ***

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  e:\packages\MsBuild\v14.0\bin\MSBuild.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: Where-ref bind. Location = e:\packages\MsBuild\v14.0\bin\Microsoft.Build.Tasks.CodeAnalysis.dll
LOG: Appbase = file:///e:/packages/MsBuild/v14.0/bin/
LOG: DEVPATH = D:\customDevPath;
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = MSBuild.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: e:\packages\MsBuild\v14.0\bin\MSBuild.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Found assembly in DEVOVERRIDE path D:\customDevPath\Microsoft.Build.Tasks.CodeAnalysis.DLL
LOG: Switch from LoadFrom context to default context.

I may have a repro of this.  I want to confirm this is the same issue.
I'm trying to load a DLL which can't be found.  I enabled Fusion logging and found that it was not in any of the DEVPATH search paths.  I don't want to load it from the GAC, so I'm not concerned I can't find it there.  The missing assembly is Microsoft.VisualStudio.CodeAnalysis.Sdk.dll.  This is causing build failures when Code Analysis is enabled in the command prompt MSBuild (not in VS IDE).
The path where the DLL is located is here:
F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\Extensions\Microsoft\VisualStudio\v15.0\CodeAnalysis

I added the path where the DLL is and confirmed in the MSBuild detailed log that the property was changed:
Property reassignment: $(DEVPATH)=";F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\v15.0\bin;F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\Extensions\Override\etc\TaskCopy;F:\CoreXtCache\VisualStudio.Sdk.Corext.14.0.23107.3\IDE;F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\Extensions\Microsoft\VisualStudio\v15.0\CodeAnalysis" (previous value: ";F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\v15.0\bin;F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\Extensions\Override\etc\TaskCopy;F:\CoreXtCache\VisualStudio.Sdk.Corext.14.0.23107.3\IDE") at F:\DS_Main\Tools\scopas\Extensions\Scopas.CSharp.props (67,5)

However, the Extensions subfolder goes missing from the path by the time it gets to Fusion:
LOG: Did not find assembly in DEVOVERRIDE path F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\v15.0\bin
LOG: Did not find assembly in DEVOVERRIDE path F:\CoreXtCache\.A\MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q\Extensions\Override\etc\TaskCopy
LOG: Did not find assembly in DEVOVERRIDE path F:\CoreXtCache\VisualStudio.Sdk.Corext.14.0.23107.3\IDE
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///F:/CoreXtCache/.A/MsBuild.Corext._zhIG90VQlM6jUBQEd5j0Q/Microsoft/VisualStudio/v15.0/CodeAnalysis/Microsoft.VisualStudio.CodeAnalysis.Sdk.dll.
LOG: All probing URLs attempted and failed.

Is this related to this issue or is  this something different entirely?

@clisc, I think your issue is different. I suspect this is because you are only updating the DEVPATH property in MSBuild, not actually updating the environment variable which the CLR is looking at. You'll need to figure out which process is trying to load that assembly and then make sure that process is spawned with an environment block containing the values you want.

Thanks, this will probably help me fix the issue from another angle.  I have some questions though:

Is DEVPATH not designed to be used as an MSBUILD property?
If not, why would it still recognize my path?
Finally, since it did recognize my path, why would it modify it and cut one of the subdirectories out?

Seems like very odd behavior.
In the meantime, I’ll run with your suggestion and modify the environment variable instead.
Thanks,
Cliff
From: John Keech<EMAIL_ADDRESS>Sent: Thursday, September 27, 2018 12:14 PM
To: Microsoft/msbuild<EMAIL_ADDRESS>Cc: Cliff Schomburg<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Microsoft/msbuild] MSBuild can report the wrong assembly path when loading a task (#584)
@clischttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fclisc&data=02|01|clisc%40microsoft.com|d36d024c92794a1339b108d624943a53|72f988bf86f141af91ab2d7cd011db47|1|0|636736616384578347&sdata=nLPiZRKA7kmXCyjJFk0CuCwZJGSyipPVfaiaqxf7pgc%3D&reserved=0, I think your issue is different. I suspect this is because you are only updating the DEVPATH property in MSBuild, not actually updating the environment variable which the CLR is looking at. You'll need to figure out which process is trying to load that assembly and then make sure that process is spawned with an environment block containing the values you want.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fmsbuild%2Fissues%2F584%23issuecomment-425152650&data=02|01|clisc%40microsoft.com|d36d024c92794a1339b108d624943a53|72f988bf86f141af91ab2d7cd011db47|1|0|636736616384578347&sdata=b8brir6HsFG59joNqX61ucriINvOGtuzwY5hJxAGdI8%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAV-TayLNV48qu8qbh1-ttfiby9h0V4Cbks5ufPlDgaJpZM4ILOVK&data=02|01|clisc%40microsoft.com|d36d024c92794a1339b108d624943a53|72f988bf86f141af91ab2d7cd011db47|1|0|636736616384734596&sdata=6HcIpMNxk78EmkOwtRyBAvkBnuNPlKX4ZF6D49cP0QI%3D&reserved=0.

Great thanks!


SwarmNL milestone 2 revision

Project Abstract
This is a revision to Milestone 2 for building SwarmNl, previously stipulated here: https://github.com/w3f/Grants-Program/pull/2201.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[ ] Level 2:  Up to $30,000, 3 approvals
[x] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
[x] I understand that an agreed upon percentage of each milestone will be paid in vested DOT, to the Polkadot address listed in the application.
[x] I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[x] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @sacha-lansky:matrix.org

Thank you for taking the time to review @takahser.
The way we calculated things for Milestone 2 was indeed $2k per FTE, calculated at an hourly rate of $40 USD/hour at a 40 hour week. In this revision we have increased the hourly rate to $60 USD.
Upon completing Milestone 1, I realized we had significantly miscalculated the projected allocation of resources and cost. Namely:

I realized that the level of complexity and protocol development the work entails merits more than a $40 per hour salary. Just by looking at around at rates for similar work being anywhere between 80 and 120, feel the increase to 60 is a reasonable middle ground.
The single FTE was calculated to include both our hours, i.e. 10 hours of Project Management, writing tests, writing docs etc. + 30 hours dedicated to development. Both roles turned out to be equally important, each taking an equal amount of time, which turned out to be a lot more than 40 hour weeks. In my role for example I spent a lot more time writing code. Deji spent a lot of time on docs and guides. We basically became one unit. In this revision for Milestone 2, we are trying to be more realistic on the amount of effort we realize will be needed to deliver to the best of our abilities which is why we now have 1.5 FTE (putting my commitment at 20 hours a week and Deji's at 40 hours).
We realized the importance of research and testing and I hadn't anticipated this well. Our M1 delivery was projected on 6 weeks, but it actually took us 8. This is why in this revision we've allocated 2 weeks for research in M2.1; 2 weeks for review, testing and improvements in M2.2 + 2 more weeks for integration testing and benchmarking (which is tricky!) and 2 more weeks for testing and touching things up in the final deliverable, "Extendability of library". This is also why there's an increase in time

Overall, we adjusted the hourly rate to $60 and adjusted the projected workload to be more realistic based on the experience we had shipping M1. Hope this helps clarify.

Hi @sacha-l, thanks for clarifying. Since your initial application, we have started paying out grants with a percentage in DOT (vested over two years). Would you be comfortable with switching to 50% DOT going forward?

Hi @semuelle - can you provide more details on what that means ? What are the advantages of receiving DOT vested over two years?

It means, part of the payment is in DOT tokens and linearly unlocks over the span of two years. We have incorporated 50%+ vested DOT payments to incentivise grant recipients to participate in the ecosystem beyond the grant payout. See also https://wiki.polkadot.network/docs/learn-transactions#vested-transfers and https://grants.web3.foundation/docs/faq#what-does-it-mean-for-payments-to-be-at-least-50-dot.
In your case, if you choose 50% DOT, for M2, you would get 18,000 USD in fiat or stablecoin, and the remaining 18,000 USD as DOT, where each block on the network unlocks a small portion for two years.

Hi @semuelle - After carefully considering the locked DOT payment opportunity, we decided we would happily accept switching to this model going forward.

Hi @semuelle - After carefully considering the locked DOT payment opportunity, we decided we would happily accept switching to this model going forward.

It means, part of the payment is in DOT tokens and linearly unlocks over the span of two years. We have incorporated 50%+ vested DOT payments to incentivise grant recipients to participate in the ecosystem beyond the grant payout. See also https://wiki.polkadot.network/docs/learn-transactions#vested-transfers and https://grants.web3.foundation/docs/faq#what-does-it-mean-for-payments-to-be-at-least-50-dot.
In your case, if you choose 50% DOT, for M2, you would get 18,000 USD in fiat or stablecoin, and the remaining 18,000 USD as DOT, where each block on the network unlocks a small portion for two years.

Thank you @semuelle , we think this is a great model that works. It will also help us further build and test our next project easily on the Polkadot network which we absolutely intend to do, in great style.

Thanks for your reply @Noc2. We would be happy to discount our hours for this milestone push by 20%, which would bring the total cost to $28,800.00. With this we would request that we get the amount in USDC instead of locked DOT. Would that work on your end?

Thank you @sacha-l for outlining what would works for us. We await your favourable response @Noc2, so we can work together to solve the problems and ship something great! 🚀

Sorry for the delay here. At this stage, we need to follow the new rule: "At least 50% of any grant is paid out in vested DOT." and can not easily accept application that don't take vested DOTs.

Understood. We would be happy to stick to the original plan and get more reviews from the committee in that case. If for some reason you advise against this, I would like to understand what the committee's reasoning might be. If this might be the case and because we just really want to get to work on shipping milestone 2, we would reduce some of the scope to keep it within the 30k tier.

Thank you very much @sacha-l, we patiently await your feedback @Noc2. Thank you.

Thanks @sacha-l just to confirm, you want to use the same Polkadot address for both USDC & DOT?

Thanks @sacha-l just to confirm, you want to use the same Polkadot address for both USDC & DOT payments?

Yes we will use the same address for locked DOT payments and USDC. But we will use a different address than the last milestone payment. The address will be: 14pG7HieCjNk2hZ4X2YvxkbpAXs9oCeW8aNZcD7tD2tpaJT9
Has the committee come to a conclusion ?

Thanks @sacha-l in that case, could you please update the Polkadot address at the top of the application as part of this commit? Since the address in the application has to match the one on the invoice. After that I'm willing to approve it myself, and then I can ping the other committee members again.

Thanks @sacha-l can you also change the level from 2 to 3? Sorry I should have mentioned that also.

Hi @semuelle thanks for you comment. To clarify, we would be happy to go forward with the original plan and accept locked DOT.

Hi @semuelle thanks for you comment. To clarify, we would be happy to go forward with the original plan and accept locked DOT.

Great to hear. In that case, could you please add a DOT %: 50% (or more) under the Costs line?

Hi @semuelle thanks for you comment. To clarify, we would be happy to go forward with the original plan and accept locked DOT.

Great to hear. In that case, could you please add a DOT %: 50% (or more) under the Costs line?

Hello @semuelle, we have made the changes requested.
Thank you

@thewoodfish thx for the reminder. I just commented.

Always a pleasure ser! @sacha-l has made the change. I look forward to your response. Thank you!

Thank you very much @semuelle and @Noc2. We patiently await your approval @takahser. Thank you.


[test]add migrate test

What this PR does / why we need it:
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Codecov Report

Merging #488 (816e77f) into main (00fe5a4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #488   +/-   ##
=======================================
  Coverage   25.48%   25.48%
=======================================
  Files          19       19
  Lines         973      973
=======================================
  Hits          248      248
  Misses        695      695
  Partials       30       30

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


It is a simple hello world program

What the program do ?
hello world
In what programming language it is written?
C

This pull request do not contains files in specific folders. Please fix it and reopen


targetSdkVersion 30

when i change targetSdkVersion from 28 to 30, app not playing any music

So just use 28...

So just use 28...

i want to use latest targetSdkVersion

i want to use latest targetSdkVersion

Why?

Why?


capstone first commit

In this Milestone, i use the following:

An index.html file
A Style.css file
A Script.js file

please i'm out of time its my last review I probably can do it later please

Please i'm out of time it's my last review I probably can do it later, please

I am so sorry unfortunately if I don't follow the code review rules and consistences with the previous reviewer I will be penalized


(chores) camel-test-infra-etcd3: upgrade the container to 3.5.14

/component-test camel-etcd3

/component-test camel-etcd3

I'll test this one semi automagically.

Still fails with the updated image.

the etcd3 failures are unrelated. Let's merge this one.


Add cargo-generate pacakge installation

Closes #78
We are currently using x86_64-unknown-linux-musl for x86_64-unknown-linux-gnu arch. If cargo-generate/cargo-generate/pull/602 gets merged I will update the script

Line 413 GENERATE_VERSION is executed always even in case when nothing is requested to install. It adds a dependency on git. Isn't there a simpler way without involving git and bunch of other tools to get latest version? Even fixed version might be better approach.

I encapsulated the git command in an if statement that is only executed when installing cargo-generate, to avoid potential errors from the git command when not installing cargo-generate as we discussed.
I also update the arch being used because they merged cargo-generate/cargo-generate/pull/602 and generated a new release which now has binary packages for x86_64-unknown-linux-gnu.
Finally, I updated the ESP_IDF_VERSION checks to use -z and -n

@SergioGasquez The change looks good. Please could you resolve also the conflict so that we can merge the feature?

@SergioGasquez The change looks good. Please could you resolve also the conflict so that we can merge the feature?

Conflicts resolved!


fix(executor): Fix S3 policy based auth. Fixes #5110

Fixes #5110

Does this need to be in v3.0.0-rc2?

Yes, please. If you use non-password based S3 bucket, it is completely broken

Sure thing


Adding map for jenkins on staging

please test this stage

Test FAILed

please stage this

please stage this

please stage this

please stage test

please stage test

please stage test

please stage this

please stage this

please stage this

please stage this

please stage this

please stage this

please stage this


why not support simple python projects?

why do you need to be using semantic kernel or prompt flow?
why cant it work with a simple python script.. i must be missing something.

That's an awesome question. In general, we need to have a runtime to execute these files in code. In theory you could definitely write your own prompty loader and execute it for sure. We thought it would be easier to do that in prompt flow. Are you suggesting we should have a standalone prompty library?

absolutely. otherwise, it seems to be an "azure-only" tool - and you didn't mention autogen 😏
what would really make this sing would be a lightweight prompt library manager. seeing as it is all. md and frontmatter, i was thinking of using obsidian to manage them.... could be pretty powerful..
Sent from Surface Duo

From: Seth Juarez @.>
Sent: Saturday, May 25, 2024 2:36:51 AM
To: microsoft/prompty @.>
Cc: Nick Katsivelos @.>; Author @.>
Subject: Re: [microsoft/prompty] why not support simple python projects? (Issue #6)
That's an awesome question. In general, we need to have a runtime to execute these files in code. In theory you could definitely write your own prompty loader and execute it for sure. We thought it would be easier to do that in prompt flow. Are you suggesting we should have a standalone prompty library?
—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/prompty/issues/6#issuecomment-2130917770, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMGI3BXCJ4RLVZLUOCXKOTZEAWQHAVCNFSM6AAAAABIIE33Y6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQHEYTONZXGA.
You are receiving this because you authored the thread.Message ID: @.***>

Looks like there's some parsing code in the promptflow library
https://github.com/microsoft/promptflow/blob/1826b3e55e36132281d68e1765f43aa13b6e5241/src/promptflow-core/promptflow/core/_flow.py#L218
Any chance that we can get a local package that allows us to parse prompty files so we can write our own executors? For example, I might want to take a prompty that was meant for gpt-35-turbo but run it using a custom configuration using a foundation model I deployed on AI Studio

We are working on getting some of our internal community packages released.

We've released an early version of the prompty package on PyPI. I am working out a couple of things and am hoping to have something by EOW

We've released an early version of the prompty package on PyPI. I am working out a couple of things and am hoping to have something by EOW

I have a slightly unrelated question, but is there any way to register custom executors with Prompty while using promptflow?

Not at the moment. Currently promptflow uses its own execution mechanism and not the factory pattern we use in prompty. The prompty package lives at a lower level than promptflow. Building your own executor is not super difficult though.

Will close this - we've set up the Python runtime code in this repo and it auto deploys to PyPI when tagged. Working on docs still.


Fix display of UI and IntelliSense in Visual Studio 2017

It seems that in Visual Studio 2017 I need to explicitly add PresentationFramework <IP_ADDRESS> as a reference to Python projects to display the UI in the editor window and get IntelliSense working in XAML.
#2943

I think the WindowBase reference should be updated too, but then we'll take this 👍

Note that without the HintPath, it doesn't work.

I don't see any difference with or without the hintpath, and unfortunately we can't easily embed those in the template or preserve them in the project. All of these assemblies should be resolvable by default though.


Envoy 1.31.2 not generating cluster.outbound metrics

Title: Envoy 1.31.2 not generating cluster.outbound metrics
Description:
I am seeing that the cluster.outbound metrics that used to be generated by Envoy are now missing having upgraded from 1.30.6 to 1.31.2. The Envoy upgrade has come from upgrading Istio from 1.22.5 to 1.23.3.
Repro steps:
Deploy a container in Kubernetes with an Istio sidecar. Istio is deployed via helm charts.
Query the Envoy metrics for the pod using kubectl exec -n <NAMESPACE> <POD_NAME> -c istio-proxy -- pilot-agent request GET stats
Observe that in Envoy version 1.30.6 there were metrics for cluster.outbound
cluster.outbound|33627||remote-service-33627.com.bind_errors: 0
cluster.outbound|33627||remote-service-33627.com.ssl.connection_error: 0
cluster.outbound|33627||remote-service-33627.com.ssl.fail_verify_error: 0
cluster.outbound|33627||remote-service-33627.com.ssl.fail_verify_no_cert: 0
cluster.outbound|33627||remote-service-33627.com.ssl.fail_verify_san: 0
cluster.outbound|33627||remote-service-33627.com.upstream_cx_protocol_error: 0

In Envoy version 1.31.2 these metrics do not appear despite the whole deployment being exactly the same. The only difference being the version of Istio and Envoy.
Admin and Stats Output:
/stats - envoy-stats.txt
/clusters - envoy-clusters.txt
/server_info - envoy-server_info.txt
Config:
envoy-rev.json
Logs:
I don't have access to the separate Envoy logs but these are the logs from the istio-proxy sidecar container that contains the Envoy logs.
envoy-log.txt
Call Stack:
N/A

@rwincewicz  we have just released 1.32. I am wondering if you still see this issue with latest release

We've retested our scenario with the latest version of Istio proxy (1.24.0) which includes version 1.32.2 of Envoy (reference to Envoy commit is here and the metrics are still missing. Is there anything we can try with configuration to fix this issue or any further information you need to be able to debug this?


bundlerApp: add support for wildcards [WIP]

Motivation for this change
We currently quote the executable we link to in a bundlerApp which is neat because it supports files with spaces, but the problem is that we then cannot use wildcards in the bundlerApp.exes property.
None of the existing bundlerApp users in nixpkgs has executables with spaces in their names.
Supporting regular wildcards, means we can do stuff like this:
exes = [ "check-*" "metrics-*" ]; which is great when one is using a script to automatically package a lot of ruby plugins for sensu.
Cc: @manveru @cstrahan @lilyball
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

I’m personally -1 on this, because I don’t think a high-level Nix function should be exposing low level shell functionality. I won’t block it if somebody else thinks it’s a good idea though.

I'm not a fan of this. I agree that it's unexpected for the entries in the exes property to actually be arbitrary unquoted shell inputs.
If anything, we should be updating this code to use the library function for correctly quoting shell strings (I forget offhand what it is but it's in there) instead of just assuming a single quote is sufficient. Granted, an exe with a single quote in it would be rather surprising, but it would currently break the code.
If there's a compelling reason to want to support wildcards then we should expose that as a separate property.

For gems where the list varies a lot it's nice to be able to pattern-match. Is there any other way to implement this that doesn't rely on forwarding the logic to bash?

Maybe add an exesPrefix argument that would escape the prefix with the globbing added to the end?
${(lib.concatMapStrings (x: "ln -s '${basicEnv}/bin/${x}'* $out/bin/${x};\n") exesPrefix)}

Let me recap:

Everybody agrees that the current situation is not ideal due to quoting possibly breaking
The current code will also happily create a symlink to an executable that doesn't exist
The proposed PR doesn't cut it as we blur the lines between bash and nix by writing bash globs in nix
Instead of changing the functionality of exes, introduce a new one.

I'll address points 2 to 4 and resubmit.

I think there is also an issue if the prefix matches multiple results. It might be best to use something like this;
set -o nullglob
${(lib.concatMapStrings (x: "for file in '${basicEnv}/bin/${x}'* ; do ln -s "$file" "$out/bin/$(basename "$file")"; done;\n") exesPrefix)}

ln supports having multiple source files. Really the problem with just using the glob as-is is if it matches none at all.

I'm going to be using find .

How is that any better than the shell for loop version that @zimbatm suggested?

For 2 reasons:

We have to worry about whitespaces with a for loop, and
The nullglob option is a bash thing where as find is POSIX, so let's use the standard option.

I don't mind either way, as long as it's correct it doesn't matter that much

I'm going to go ahead and drop this.
Don't really have a need for it any longer.


Removing entities from the World

Hello,
I was wondering if there is a way to delete/remove an entity from the world.
Typically, if I want to remove an obsolete entity, I would do a entity.clear() which would effectively make the entity "invisible" to queries, if I'm not mistaken; however, the world keeps a reference to the entity which prevents it from being deleted.
Is this intentional? Doesn't it create a memory leak where creating many entities lead to an increasing amount of references to objects that are not used anymore? Or did I miss something?
Thanks.

Doesn't it create a memory leak where creating many entities lead to an increasing amount of references to objects that are not used anymore? Or did I miss something?

What you're missing is that the information needed to query the entities is itself the strong reference the world has to that entity. After entity.clear() is called then the only strong references left to that entity are either entity relations targeting that entity and any references that you've made outside of the world object.  All entities are also stored in a global weak dictionary which does not not keep them alive by itself.
entity.clear() not removing the entity as the target of other entities is something I'll likely fix.

As a simpler explanation. If you invoke entity = world.new_entity() then the name entity becomes the only strong reference to that new Entity object. The only way to keep it alive is to assign a component to it, etc.

I see... However, what I did to test (and might contain a subtlety that I did not notice) is
>>> from tcod.ecs import World

>>> world = tcod.ecs.World()
>>> entity = world.new_entity(components=[object()])
>>> uid = entity.uid
>>> entity.clear()
>>> del entity

>>> world[uid]
Out[7]: <Entity(uid=object at 0x7F09E248A260)>

Since to my knowledge the UID doesn't contain a reference to the entity, and I removed other references to the entity, shouldn't it be removed and as such not possible to access from the world? Or is the UID referencing the entity in a way that I'm not aware of?

Since to my knowledge the UID doesn't contain a reference to the entity, and I removed other references to the entity, shouldn't it be removed and as such not possible to access from the world?

This is correct, you understand how UID's work and what you think should happen did happen. The entity was very likely deleted until you reused the UID to make a new entity with an equivalent UID. I can verify that entities don't leak.
The world[uid] syntax always does either of the following:

Fetches the existing entity with that world/uid combo.
Creates a new entity with that world/uid combo if it does not exist.

You'll never get a KeyError from this syntax. This is used for entity singletons, such as using world[None] as a place to store global components, or converting a hashable object to be used as a relation target.
>>> world[None].components[str] = "Easy global components"

>>> world[None].relation_tag["Level"] = world[10]  # Allows queries by value
>>> world.Q.all_of(relations=[("Level", world[10])]).get_entities()  # Query by value
{<Entity(uid=None)>}

Note that the following two lines do the same thing:
>>> entity = world.new_entity()
>>> entity = world[object()]

No matter what you do, fetching two entities using the same uid will always return an object with the same identity.
>>> world[uid] is world[uid]
True

This last part is actually the important part and is the entire reason I made this silly setup.  If objects share the same identity than CPython can skip the __eq__ test when checking if an object exists in a set, which this library does a lot of when doing queries.

I see, that's where I lacked understanding of how things work. I did expect the world to either raise an exception, or return nothing, when asking for an UID that was not created through the typical new_entity function.
Thanks for the detailed explanation!


Handle null value that marks a tombstone

Sometimes we want to encode and decode null.
We cannot replace null with None, since the Scala Kafka Streams library is a thin wrapper around the Java one, and Java expects null.

 Thanks.


Fails to connect to Charles Schwab Bank/Charles Schwab Brokerage on python3

I have a laptop running Ubuntu 18.04 and python 2.7. On this laptop ofxclient 2.0.2 will successfully connect to Charles Schwab Bank and Charles Schwab brokerage and download my account data via OFX.
On my desktop, running NixOS unstable, ofxclient is run with Python 3.7.3 and is version 2.0.3 (latest from PyPI). Here I consistently receive the error:
  File "/<nixpath>/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(lne, buffer)
socket.timeout: The read operation timed out

It would seem that the ssl library is not properly making the connection. Note that this is not due to network errors, as the machines are on the same home network, can both ping the same hosts, and I have even run ofxclient concurrently on both.
Is there a known bug in the ssl library for python3 that is triggering this? Or is this an issue with ofxclient?

Update: it turns out that newer versions of ofxclient (>2.0.2 I think) send a user-agent header, and a comma-separated "accept" header, both of which appear to confuse Schwab's OFX servers.
I was able to run ofxclient-2.0.4 on the machine where I currently had things working with 2.0.2, and verify that it broke. Then, setting institution.client_args.accept = "application/x-ofx" and institution.client_args.user_agent = False for each of my Schwab accounts allowed me to download them with ofxclient-2.0.4.

It appears that this is addressed in #81


Errors When Importing a New Library

When importing a new library, many of the items will be unmatched. The title of the files are a perfect match and appear when searched on TPDB website.
Plugin V. <IP_ADDRESS>
[2023-05-11 11:55:02.156 -04:00] [ERR] [89] ThePornDB.Plugin: API error: "Too Many Attempts."
[2023-05-11 11:55:02.236 -04:00] [ERR] [113] ThePornDB.Plugin: API error: "Too Many Attempts."
[2023-05-11 11:55:02.238 -04:00] [ERR] [113] ThePornDB.Plugin: API error: "Too Many Attempts."
[2023-05-11 11:55:02.353 -04:00] [ERR] [113] ThePornDB.Plugin: API error: "Too Many Attempts."
[2023-05-11 11:55:02.358 -04:00] [ERR] [89] ThePornDB.Plugin: API error: "Too Many Attempts."
[2023-05-11 11:55:02.364 -04:00] [ERR] [23] ThePornDB.Plugin: API error: "Too Many Attempts."

Use dev build


⚠ Jenkins Plugins Artifactory has degraded performance

In 1a4d35d, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1166 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 23d6347 after 39 minutes.


doc: add gtk4 warning

GTK4 crash due to XCompose file beeing too long, https://github.com/rvaiya/keyd/issues/537
Add a warning in the doc

Thanks. Merged (after a rebase).


window.open() a PDF file crashes on android

I try to open a PDF file using
window.open('http://domain.com/test.pdf', '_system', 'location=no');

And my app crashes on Android, but on iOS it works fine.  I compiled the app with Android 4.2.0 Crosswalk, ARM and x86

Correction,
window.open

crashes on Android regardless of file type.

+1 getting same issue with window.open, crashing android app even when just trying to open external url for social media logins

What if you displayed the file using the embedded pdf viewer?
Usage:

Supersonic: https://github.com/AppGyver/supersonic/blob/master/testApp/app/ui/scripts/FileViewsController.coffee
Steroids: https://github.com/AppGyver/steroids-js/blob/next/testApp/app/controllers/previewfileview.coffee

I couldn't reproduce the crash using: Test on Samsung S6 (Android 5.1.1) or Nexus 3 (Android 4.2), client version 4.2.1 with Crosswalk.
Could you share the crash log from adb logcat ?


Colorization is broken in vscode

It looks as though colorization of debug logs is only partly working in vscode - seems to output okay for the ms timings but not okay for the namespace:

The configuration I'm using:
"configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "Launch Program",
            "program": "${workspaceRoot}/bin/www",
            "env": {
                "DEBUG": "ss:*",
                "DEBUG_COLORS": "true"
            },
            "cwd": "${workspaceRoot}",
            "port": 3000
        }
    ]

I've found that removing the ;1 in the formatArgs function resolves it:

I'm not au fait enough with ANSI colour codes to know the implications of this. Help greatly appreciated!

Pretty sure the ;1 is "bold". We've had reports of issues with it in the past.

@Stumblor I don't get any debug output in my VSCode console. Even though the environment variable is set correctly. Are you still getting output with the latest version?

@0x80 Yes still getting (non colored) output when using 2.6.3.

@Stumblor Weird. I don't get it. Not meaning to hijack this thread but maybe you see something funny about this:
{
  "version": "0.2.0",
  "configurations": [{
      "name": "Launch",
      "type": "node",
      "request": "launch",
      "protocol": "inspector",
      "program": "${workspaceRoot}/src/index.js",
      "stopOnEntry": false,
      "args": [],
      "cwd": "${workspaceRoot}",
      "preLaunchTask": null,
      "runtimeExecutable": null,
      "runtimeArgs": [
        "--nolazy",
        "--require", "babel-register"
      ],
      "env": {
        "NODE_ENV": "development",
        "DEBUG": "scrape-from-contentful"
      },
      "console": "internalConsole",
      "sourceMaps": true,
      "outFiles": []
    }
  ]
}

When I run this from my terminal it works fine:
DEBUG=scrape-from-contentful node --require babel-register ./src/index.js
This command gives the same output in the integrated terminal in vscode, but nothing when I start the debugger. I've also tried appending  :* like you have, but it makes no difference. I must be missing something.

I set console to "integratedTerminal" and there things work fine for me. I like it since it's a bit less magic too. You see exactly how the commands are executed.

i'm running vscode 1.14.1 and i don't get any output when using default of console: internalConsole, using integratedTerminal does show output, but it would be nice to have it work with the default setting of internalConsole

This is a bug in VSCode. Please open an issue there - we use very standard output mechanisms.