[WIP] Moving Cyphal/UDP to multicast

This MR is based on this proposol discussed in the OpenCyphal forum.
In short, the changes can be broken down into three pieces:
1. Datagram header format
Current:
uint8 version = 0
uint8 priority
void16
uint32 frame_index_eot
uint64 transfer_id
void64

Proposal:
uint8 version = 1   # UPDATE
uint8 priority
uint16 source_node_id   # NEW
uint32 frame_index_eot
uint64 transfer_id
void64

Note: version will be bumped to 1, however no backward-compatibility changes are made (since the protocol is still in development).
2. Message
Current:
    fixed         reserved
   (9 bits)       (3 bits)
   ________          _
  /        \        / \
  11101111.0ddddddd.000sssss.ssssssss
  \__/      \_____/    \____________/
(4 bits)    (7 bits)      (13 bits)
  IPv4      subnet-ID     subject-ID
multicast   \_______________________/
 prefix             (23 bits)
            collision-free multicast
               addressing limit of
              Ethernet MAC for IPv4

Proposal:
    fixed   message  reserved
   (9 bits) select.  (3 bits)
   ________   res.|  _
  /        \     vv / \
  11101111.0ddddd00.000sssss.ssssssss
  \__/      \___/      \____________/
(4 bits)   (5 bits)       (13 bits)
  IPv4     subnet-ID      subject-ID
multicast   \_______________________/
 prefix             (23 bits)
            collision-free multicast
               addressing limit of
              Ethernet MAC for IPv4

3. Service
Current: regular unicast
Proposal:
    fixed          service
   (9 bits)  res.  selector
   ________      ||
  /        \     vv
  11101111.0ddddd01.nnnnnnnn.nnnnnnnn
  \__/      \___/   \_______________/
(4 bits)   (5 bits)     (16 bits)
  IPv4     subnet-ID     node-ID
multicast   \_______________________/
 prefix             (23 bits)
            collision-free multicast
               addressing limit of
              Ethernet MAC for IPv4

TODO

[x] Datagram
[x] Message
[ ] Service
[ ] Implement Scott's changes
[ ] Add anonymous mode
[ ] Update documentation

Question: Is anonymous listening possible or not? There seems to be some contradiction:
pycyphal/transport/udp/__init__.py:20:
The concept of anonymous transfer is not defined for Cyphal/UDP

vs
pycyphal/transport/udp/__init__.py:302
>>> tr_1 = pycyphal.transport.udp.UDPTransport('<IP_ADDRESS>', local_node_id=None)  # Anonymous is only for listening.

Question: Does it make sense to move source_node_id to class Frame ( defined inpycyphal/transport/commons/high_overhead_transport/_frame.py)?

Is anonymous listening possible or not? There seems to be some contradiction:

There is no contradiction: "The concept of anonymous transfer is not defined for Cyphal/UDP" means that you can't send anonymous transfers, but you can still create anonymous nodes that cannot communicate (they can only listen).

Does it make sense to move source_node_id to class Frame ( defined inpycyphal/transport/commons/high_overhead_transport/_frame.py)?

Probably not (yet) because you would have to update other transports dependent on this class.

Ok, thanks.
"Datagram" still needs some work, will ping you when a review is needed.

Does the passing of the local_node_id as source_node_id look right? (see last commit)

I have updated the unit test, to take into account source_node_id
Functional tests (run pytest from pycyphal/transport/udp):

Integration tests (pytest -k udp):

Can you review these changes? I think the first step of implementing changes related to "Datagram" is done.

The changes look good to me. Later on, we may want to enable anonymous transfers for UDP as I wrote on the forum; it should be a cheap change to introduce. The relevant part of the source code can be found by searching for "In Cyphal/UDP, the anonymous mode is somewhat bolted-on."
The text and diagrams in the module docstring at _udp.py will also need to be updated, perhaps later.
Nice work so far ;)

Later on, we may want to enable anonymous transfers for UDP as I wrote on the forum; it should be a cheap change to introduce.

The text and diagrams in the module docstring at _udp.py will also need to be updated, perhaps later.

Added to todo-list 👍

@pavel-kirienko
Could you review these changes? I just want to make sure that this part is correct (pycyphal/transport/udp/_ip), before I start updating the larger code base/unit tests.
Also some questions:

service_data_specifier_to_multicast_group and message_data_specifier_to_multicast_group are pretty similar; does it make sense to combine them? Instead of 4 functions in _endpoint_mapping.py, this would result in 2. (Equating MESSAGE_ID and SERVICE_ID into one variable NODE_ID would also help in this regard.)

I renamed most of the constants used in _endpoint_mapping.py, let me know if this change make sense.

Nice progress but there seems to be a problem with subject-/service-/node-ID mixup. Please give another look at section 4.1.1 "Transport model" of the Specification.

To publish a message on subject S, we send a multicast datagram to the multicast group whose address is computed as:

And the destination UDP port is set to 16383.

To send a request or response on service X to node N, we send a multicast datagram to the multicast group whose address is computed as:

And the destination UDP port is set to (16384 + X * 2 + (is_response)).

Could you please update your branch to sync up with master?

Could you confirm the changes are correct? I think I have addressed the issues.

Regarding pycyphal/transport/udp/_ip/_v4.py:
class IPv4SocketFactory(SocketFactory):

  def __init__(self, domain_id: int)

  def make_output_socket(
    self, remote_node_id: typing.Optional[int], data_specifier: pycyphal.transport.DataSpecifier
  ) -> socket.socket:
    # General setup
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    s.setblocking(False)
    s.bind((str(self._local), 0))  # QUESTION: What local IP address bind to? (Does it need to be bound at all?)

    # Message
    remote_ip = message_data_specifier_to_multicast_group(self._domain_id, data_specifier)
    remote_port = SUBJECT_PORT

    # Service
    remote_ip = service_data_specifier_to_multicast_group(self._domain_id, remote_node_id, data_specifier)
    remote_port = service_data_specifier_to_udp_port(data_specifier)

    # Connect
    s.connect((str(remote_ip), remote_port))

  def make_input_socket(
    self, remote_node_id: typing.Optional[int], data_specifier: pycyphal.transport.DataSpecifier # CHANGE: need remote_node_id for service
    ) -> socket.socket:
    # General setup
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    s.setblocking(False)

    # Message
    multicast_ip = message_data_specifier_to_multicast_group(self._domain_id, data_specifier)
    multicast_port = SUBJECT_PORT

    # Service
    multicast_ip = service_data_specifier_to_multicast_group(self._domain_id_, remote_node_id, data_specifier)
    multicast_port = service_data_specifier_to_udp_port(data_specifier)

    # Bind
    s.bind((str(multicast_ip), multicast_port))

@pavel-kirienko
I think it's almost finished.
The last change that needs to be addressed, is mainly related to udp/_socket_reader.py.
class SocketReader contains the following note:

This class is the solution to the UDP demultiplexing problem.
The objective is to read data from the supplied socket, parse it, and then forward it to interested listeners.
Why can't we ask the operating system to do this for us? Because there is no portable way of doing this(except for multicast sockets). Even on GNU/Linux, there is a risk of race conditions, but I'll spare you the details. Those who care may read this: https://stackoverflow.com/a/54156768/1007777.

This seems to suggest that some significant overhaul/simplification can be done here. Can you give some pointers?
Some smaller questions:

pycyphal/transport/udp/_tracer.py:160:

if ip_destination.is_multicast:
    if udp_packet.destination_port == SUBJECT_PORT:
        # Message packet
        dst_nid = None  # Broadcast
        data_spec = multicast_group_to_message_data_specifier(ip_source, ip_destination)
    else:
        # Service packet
        data_spec = udp_port_to_service_data_specifier(udp_packet.destination_port)
        # QUESTION: Correct to use DOMAIN_ID_MASK here? (or make a seperate function for this?)
        domain_id = (int(ip_destination)&DOMAIN_ID_MASK)>>18
        dst_nid = service_multicast_group_to_node_id(domain_id, ip_destination)

tests/transport/udp/ip/v4.py:53/81:

msg_i = fac.make_input_socket(None, MessageDataSpecifier(612))
test_msg_o.sendto(b"Seagull", ("<IP_ADDRESS>", SUBJECT_PORT))
time.sleep(1) ##QUESTION: BlockingIOError: [Errno 35] Resource temporarily unavailable
rx = msg_i.recvfrom(1024)
assert rx[0] == b"Seagull"
assert rx[1][0] == "<IP_ADDRESS>"  # Same address we just bound to.

This extra sleep(1) wasn't necessary before, however after some changes it started failing with the above error. Any remarks on this?

This seems to suggest that some significant overhaul/simplification can be done here. Can you give some pointers?

Yes, indeed, as you have correctly guessed, it should be possible to get rid of the SocketReader or at least simplify it. There may be an arbitrary number of sockets connected to the same multicast endpoint and the OS should perform demultiplexing correctly for you. Maybe for now, in the interest of minimizing the scope of this changeset, we should keep the socket reader in place and consider removing it later.

QUESTION: Correct to use DOMAIN_ID_MASK here? (or make a seperate function for this?)

I am not sure what are you going to use the domain-ID here for? It doesn't seem to be needed to parse the frame unless I am missing something.

This extra sleep(1) wasn't necessary before, however after some changes it started failing with the above error. Any remarks on this?

The kernel works in mysterious ways, what else can I say? Since this is just a test, it is fine to simply keep the sleep in place.

There is some discussion on the forum that you should be aware of: https://forum.opencyphal.org/t/cyphal-udp-architectural-issues-caused-by-the-dependency-between-the-nodes-ip-address-and-its-identity/1765/41
There's nothing major but it seems like we'll have to shuffle some bits around the header and the IP address, start using one common UDP port number for all traffic and discriminate services based on a dedicated service-ID field in the header instead of UDP ports, and also possibly add a header checksum. All of these changes seem quite minor in comparison to what you've already implemented here.

First I re-wrote pycyphal/transport/udp/_socket_reader.py like this:
def _dispatch_frame(
    self, timestamp: Timestamp, source_ip_address: _IPAddress, frame: typing.Optional[UDPFrame]
) -> None:
    # Do not accept datagrams emitted by the local node itself. Do not update the statistics either.
    external = self._anonymous or (source_ip_address != self._local_ip_address)
    if not external:
        return

    # Process the datagram. This is where the actual demultiplexing takes place.
    # The node-ID mapper will return None for datagrams coming from outside of our Cyphal subnet.
    handled = False
    source_node_id = None
    if frame is not None:
        # if source_ip_address is part of our Cyphal subnet
        if (DOMAIN_ID_MASK & int(source_ip_address)) == (DOMAIN_ID_MASK & int(self._local_ip_address)):
            source_node_id = frame.source_node_id
        # if source_ip_address is not part of our Cyphal subnet, source_node_id is None
        else:
            source_node_id = None

Now I'm starting to suspect that this is not how it's meant to be. Instead it should be:

class SocketReader needs a new variable domain_id

def __init__(
    self,
    sock: socket.socket,
    local_ip_address: _IPAddress,
    domain_id: int,
    anonymous: bool,
    statistics: SocketReaderStatistics,
):
  self._domain_id = domain_id

def _dispatch_frame becomes:

def _dispatch_frame(
    self, timestamp: Timestamp, source_ip_address: _IPAddress, frame: typing.Optional[UDPFrame]
) -> None:
    # Do not accept datagrams emitted by the local node itself. Do not update the statistics either.
    external = self._anonymous or (source_ip_address != self._local_ip_address)
    if not external:
        return

    # Process the datagram. This is where the actual demultiplexing takes place.
    # The node-ID mapper will return None for datagrams coming from outside of our Cyphal subnet.
    handled = False
    source_node_id = None
    if frame is not None:
        # if source_ip_address is part of our Cyphal subnet
        if self._domain_id == (DOMAIN_ID_MASK & int(source_ip_address)):
            source_node_id = frame.source_node_id
        # if source_ip_address is not part of our Cyphal subnet, source_node_id is None
        else:
            source_node_id = None

I'm not sure if source_ip_address is the multicast address here?
Note to self: replace subnet with domain_id, to avoid further confusion.

Now I'm starting to suspect that this is not how it's meant to be

You are correct in suspecting this!

I'm not sure if source_ip_address is the multicast address here?

In this new design, unicast IP addresses are no longer relevant at all. Any node can operate on any domain-ID with any node-ID regardless of its identity on the IP layer. Parameters like the source_ip_address should no longer be used anywhere except the socket factory (where it is needed only to initialize the socket correctly, this is done once per socket). The socket reader now only needs to read the datagram and pass it along regardless of the source address or the local IP address.

@pavel-kirienko
Can you check this _socket_reader implementation?
Main changes:

Removed self._local_ip_address
Removed self._anonymous
Changed SocketReaderStatistics

# Old
accepted_datagrams: typing.Dict[int, int] = dataclasses.field(default_factory=dict)
dropped_datagrams: typing.Dict[typing.Union[_IPAddress, int], int] = dataclasses.field(default_factory=dict)

# New
accepted_datagrams: typing.Dict[typing.Optional[int], int] = dataclasses.field(default_factory=dict)
dropped_datagrams: typing.Dict[typing.Optional[int], int] = dataclasses.field(default_factory=dict)

# Keys:
# None: Invalid node-ID (for dropped_datagrams)
# Int: node-ID
# Int=0xffff: anonymous frame (for accepted_datagrams)

Concerning the keys: use 0xffff for anonymous frames or None?
(Ignore the QUESTIONs in the code.)
Unit tests (_unittest_socket_readerand _unittest_socket_reader_endpoint_reuse) are running successfully, but need some additional changes to better reflect the new class structure.

I'm having this issue in _unittest_udp_transport_ipv4:
I'm trying to send a service request from client_requester to server_listener.
At first I defined server_listener as follows:
server_listener = tr.get_input_session(
    InputSessionSpecifier(ServiceDataSpecifier(444, ServiceDataSpecifier.Role.REQUEST), None), meta # remote_node_id is None
)

This makes sense from the perspective that a service input session needs to accept calls from all nodes.
However, if the test fails because there is no subscription to the multicast address <IP_ADDRESS>, therefore client_requester is not receiving any of the sent frames.
# from pycyphal/transport/udp/_ip/_v4.py:
elif isinstance(data_specifier, ServiceDataSpecifier):
    multicast_ip = service_data_specifier_to_multicast_group(self._domain_id, remote_node_id)
    multicast_port = service_data_specifier_to_udp_port(data_specifier)
    if sys.platform.startswith("linux") or sys.platform.startswith("darwin"):

Then I though, maybe it makes sense for server_listener to be defined as follows:
server_listener = tr.get_input_session(
    InputSessionSpecifier(ServiceDataSpecifier(444, ServiceDataSpecifier.Role.REQUEST), 111), meta # remote_node_id is 111
)

111 is its own node_id, so it is basically subscribing itself to the multicast address where the frames are sent.
Then it does receive the frames, however they are discarded since the source_node_id (222 in this case) does not match the subscribed node_id 111:
# from pycyphal/transport/udp/_socket_reader.py
# Handle non-anonymous frames
if source_node_id is not None:
    # Each frame is sent to the promiscuous listener (None) and to the selective listener (source_node_id).
    # We parse the frame before invoking the listener in order to avoid the double parsing workload.
    for key in (None, source_node_id):
        try:
            callback = self._listeners[key]
        except LookupError:
            pass
        else:
            handled = True
            try:
                callback(timestamp, frame)
            except Exception as ex:  # pragma: no cover
                _logger.exception("%r: Unhandled exception in the listener for node-ID %r: %s", self, key, ex)

I'm prone to thinking the first approach is the right one. However, I'm not sure how to setup this promiscuous multicast address correctly, so it receives all the frames that are sent.
The relevant part of _v4.py:
if isinstance(data_specifier, MessageDataSpecifier):
    multicast_ip = message_data_specifier_to_multicast_group(self._domain_id, data_specifier)
    multicast_port = SUBJECT_PORT
    if sys.platform.startswith("linux") or sys.platform.startswith("darwin"):
        # Binding to the multicast group address is necessary on GNU/Linux: https://habr.com/ru/post/141021/
        s.bind((str(multicast_ip), multicast_port))
    else:
        # Binding to a multicast address is not allowed on Windows, and it is not necessary there. Error is:
        #   OSError: [WinError 10049] The requested address is not valid in its context
        s.bind(("", multicast_port))
    try:
        # Note that using INADDR_ANY in IP_ADD_MEMBERSHIP doesn't actually mean "any",
        # it means "choose one automatically"; see https://tldp.org/HOWTO/Multicast-HOWTO-6.html
        # This is why we have to specify the interface explicitly here.
        s.setsockopt(
            socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, multicast_ip.packed + self._local_ip_addr.packed
        )
    except OSError as ex:
        s.close()
        if ex.errno in (errno.EADDRNOTAVAIL, errno.ENODEV):
            raise InvalidMediaConfigurationError(
                f"Could not register multicast group membership {multicast_ip} via {self._local_ip_addr} using {s} "
                f"[{errno.errorcode[ex.errno]}]"
            ) from None
        raise  # pragma: no cover
elif isinstance(data_specifier, ServiceDataSpecifier):
    multicast_ip = service_data_specifier_to_multicast_group(self._domain_id, remote_node_id)
    multicast_port = service_data_specifier_to_udp_port(data_specifier)
    if sys.platform.startswith("linux") or sys.platform.startswith("darwin"):
        s.bind((str(multicast_ip), multicast_port))
    else:
        s.bind(("", multicast_port))
    try:
        s.setsockopt(
            socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, multicast_ip.packed + self._local_ip_addr.packed
        )
    except OSError as ex:
        s.close()
        if ex.errno in (errno.EADDRNOTAVAIL, errno.ENODEV):
            raise InvalidMediaConfigurationError(
                f"Could not register multicast group membership {multicast_ip} via {self._local_ip_addr} using {s} "
                f"[{errno.errorcode[ex.errno]}]"
            ) from None
        raise  # pragma: no cover

@maksimdrachov

I'm prone to thinking the first approach is the right one.

This is correct.

However, I'm not sure how to setup this promiscuous multicast address correctly, so it receives all the frames that are sent.

The socket factory is not involved in the distinction between selective and promiscuous sessions: regardless of the mode, the socket would be configured identically. The distinction matters here where the socket reader is configured:
https://github.com/OpenCyphal/pycyphal/blob/8273967696d9c250264a252399681ae8d725b962/pycyphal/transport/udp/_udp.py#L340
Observe the logic: when you request a new input session, the UDP transport checks if there's a socket available for this endpoint already (perhaps in a different mode; e.g. there could be a selective session while you want to set up a new promiscuous session with the same data specifier). If no such endpoint is found, a new socket reader is created, otherwise the existing socket reader is used to add a new listener to it.
What you described suggests that the socket factory is operating properly but the socket reader is, apparently, somehow mishandling promiscuous listeners. I would check that.

Looking at the latest proposals from @thirtytwobits I am actually thinking if it would be easier to modify the input logic such that there is always exactly one socket dedicated to incoming service transfers (unless the transport is configured in the anonymous mode, in which case it wouldn't be possible to receive service transfers anyway so no socket is needed). Then you add some kind of basic dispatcher that receives frames from that socket and forwards them to the appropriate listener, much like the SocketReader currently does but the listener would be identified based not only on the remote address but also on the service-ID.
Next, considering that selective listeners aren't really used with subjects --- all subjects are promiscuous (there may be some support for selective subject listeners but this is not really part of the spec so not required) --- we can apply the next obvious simplification which is to remove the SocketReader completely and simply keep one socket per UDPInputSession of a subject (we may need a new specialization for that, like UDPSubjectInputSession, because the existing subclasses do own sockets).

All of this amounts to a somewhat deeper refactoring than I would initially be comfortable with but if we accept Scott's suggestions (so far there seems to be no reason not to), then we'd have to do it anyway. If you're available, we could meet next week face2face and discuss this. I'm hoping to bring the discussion around Scott's proposals to some sort of tentative conclusion by then.

output
Call-flow:
# _udp.py
def get_output_session(
    self, specifier: pycyphal.transport.OutputSessionSpecifier, payload_metadata: pycyphal.transport.PayloadMetadata
) -> UDPOutputSession

# |
# v

# _udp.py
sock = self._sock_factory.make_output_socket(specifier.remote_node_id, specifier.data_specifier)

# |
# v

# _ip/_v4.py
def make_output_socket(
    self, remote_node_id: typing.Optional[int], data_specifier: pycyphal.transport.DataSpecifier
) -> socket.socket

For Message, remote_node_id is always None. The socket is connected to the following multicast address:
remote_ip = message_data_specifier_to_multicast_group(self._subnet_id, data_specifier)
remote_port = SUBJECT_PORT

For Service, remote_node_id can't be None. (A service request always needs to be directed towards a certain remote_node_id). The multicast address is determined as follows:
remote_ip = service_data_specifier_to_multicast_group(self._subnet_id, remote_node_id)
remote_port = service_data_specifier_to_udp_port(data_specifier)

All output sockets are stored in:
# _udp.py
self._output_registry: typing.Dict[pycyphal.transport.OutputSessionSpecifier, UDPOutputSession] = {}

(Each UDPOutputSession has it's own socket, sending is done through UDPOutputSession.send())
Removal of SocketReader does not affect this part of the implementation.
input
Call-flow (current implementation):
# _udp.py
def get_input_session(
    self, specifier: pycyphal.transport.InputSessionSpecifier, payload_metadata: pycyphal.transport.PayloadMetadata
) -> UDPInputSession:

# |
# v

# _udp.py
def _setup_input_session(
    self, specifier: pycyphal.transport.InputSessionSpecifier, payload_metadata: pycyphal.transport.PayloadMetadata
) -> None:

# |
# v

# _udp.py
self._socket_reader_registry[specifier.data_specifier] = SocketReader(
    sock=self._sock_factory.make_input_socket(specifier.remote_node_id, specifier.data_specifier),
    local_node_id=self._local_node_id,
    statistics=self._statistics.received_datagrams.setdefault(
        specifier.data_specifier, SocketReaderStatistics()
    ),
)

# |
# v

# _ip/_v4.py
def make_input_socket(
    self, remote_node_id: typing.Optional[int], data_specifier: pycyphal.transport.DataSpecifier
) -> socket.socket:

Call-flow (ported)
# _udp.py
def get_input_session(
    self, specifier: pycyphal.transport.InputSessionSpecifier, payload_metadata: pycyphal.transport.PayloadMetadata
) -> UDPInputSession:

# |
# v

# _udp.py
sock = self._sock_factory.make_input_socket(specifier.remote_node_id, specifier.data_specifier)

# |
# v

# _ip/_v4.py
def make_input_socket(
    self, remote_node_id: typing.Optional[int], data_specifier: pycyphal.transport.DataSpecifier
) -> socket.socket:
    # adds listeners for relevant subject-ID/service-ID

For Message, remote_node_id is always None. The socket is bound to the following multicast address:
multicast_ip = message_data_specifier_to_multicast_group(self._subnet_id, data_specifier)
multicast_port = SUBJECT_PORT

For Service, remote_node_id is always None. The socket is bound to the following multicast address:
multicast_ip = service_data_specifier_to_multicast_group(self._subnet_id, remote_node_id)
multicast_port = service_data_specifier_to_udp_port(data_specifier)

QUESTION 1: I guess remote_node_id doesn't make a lot of sense in the input case, since both Message and Service are assumed to be receiving from all possible sources?
All input sockets are stored in:
# _udp.py
self._input_registry: typing.Dict[pycyphal.transport.InputSessionSpecifier, UDPInputSession] = {}

(Each UDPInputSession has it's own socket, receiving is done through UDPInputSession.receive())
conclusion
Here some changes that need to be made due to removal of SocketReader:

Removal of _socket_reader_registry: sockets are now stored in UDPInputSession.

All SocketReader's functionality, is also moved into UDPInputSession:

add_listener()
remove_listener()
_dispatch_frame(): becomes much simpler due to multicast

Altogether, due to changes: input setup/operation will become much closer to output setup/operation,

Multicast addresses are:
z = subject-ID (Message)

Multicast address     |  Message, Output     Message, Input      Service, Output     Service, Input
---------------------------------------------------------------------------------------------------
remote_node_id = y    |  Not supported       Not supported       239.x.x.y           Not supported
                      |
remote_node_id = None |  239.x.x.z           239.x.x.z           Not supported       239.x.255.255
                      |

Two types of UDPInputSession:

PromiscuousUDPInputSession -> rename to ServiceInputSession?:

Service, Input
Single socket, but multiple listeners such as in SocketReader, to be able to handle different service calls.
If different subnets, we might have multiple ServiceInputSessions in _input_registry.

UDPSubjectInputSession -> rename to MessageInputSession?:

Message, Input
Single socket with single listener, each subject-ID has its own MessageInputSession
If different subnets/subject-ids, we have multiple MessageInputSessions in _input_registry.

QUESTION 2: No use of SelectiveUDPInputSession? I guess "Not supported" of "Message, Input" is SelectiveUDPInputSession?
QUESTION 3: What with SocketReaderStatistics? Not used in tests/_udp.py so not strictly necessary?
Regarding's Scott's changes, I've added to TODO, will implement after this Service part is done.

For Service, remote_node_id can't be None. (A service request always needs to be directed towards a certain remote_node_id). The multicast address is determined as follows:

remote_ip = service_data_specifier_to_multicast_group(self._subnet_id, remote_node_id)
remote_port = service_data_specifier_to_udp_port(data_specifier)

This is generally correct but the remote port is now (per Scott's proposal) fixed at 9382 (not sure how the number was chosen but there is probably no difference). It is simpler than it used to be.

For Service, remote_node_id is always None. The socket is bound to the following multicast address:
...
QUESTION 1: I guess remote_node_id doesn't make a lot of sense in the input case, since both Message and Service are assumed to be receiving from all possible sources?

This is incorrect because service listeners can be promiscuous (accept service transfers from any remote node) or selective (accept service transfers from a specific remote node); the remote_node_id is None in the former case and a valid integer in the latter.
The promiscuous kind is used with service servers (see pycyphal.presentation.Server) because a server needs to be able to receive a request from any client on the network.
The selective kind is used with service clients (see pycyphal.presentation.Client) because client instances are instantiated for a specific remote endpoint that provides the service. See Presentation.make_client() in the linked docs. The objective of this design is to provide sufficient context to the transport layer implementation to allow the transport to perform required packet filtering at the low level instead of requiring the application to do this at the high level manually.
The above also applies to QUESTION 2.

Each UDPInputSession has it's own socket, receiving is done through UDPInputSession.receive()

Good. I think we should drop a few lines in the comments explaining that the UDP demultiplexing problem does not apply in this case because it is only manifested with point-to-point connections, not multicast. I am referring to this piece:
https://github.com/OpenCyphal/pycyphal/blob/8273967696d9c250264a252399681ae8d725b962/pycyphal/transport/udp/_socket_reader.py#L58-L61

I see that you decided to keep one input socket per session. I suppose this is also fine, although some inefficiencies due to data duplication in the kernel might arise compared to the approach I suggested originally, where there would be one input socket responsible for all incoming service transfers, which are then dispatched to the correct UDPInputSession based on the service-ID and the remote node-ID. Perhaps your approach is simpler and the performance issues would never arise --- we'll see. Let's proceed as you implemented it and see if it requires changes in the future.

All SocketReader's functionality, is also moved into UDPInputSession:

If there's a dedicated socket per session, why do we need to add/remove listeners dynamically? I assume the lifetime of a socket matches that of the session that owns it, so there should be no need for dynamic listener configuration. Am I missing something here?

QUESTION 3: What with SocketReaderStatistics? Not used in tests/_udp.py so not strictly necessary?

We can safely drop this for now. This is needed only for high-level diagnostics and it does not directly affect the functionality of the library. For instance, yakut subscribe, yakut call, etc. emit this information at exit.

@maksimdrachov, please either rebase on master or merge it into your branch. There have been important changes related to the CI.

@pavel-kirienko
Can you check udp/_ip? It's changed according to Scott's post.
I added to some unit tests here:

udp/_ip/_endpoint_mapping.py
udp/_ip/_v4.py

@pavel-kirienko
Re-iterating:
UDPInputSession       |  Message                        Service
----------------------------------------------------------------------------------
remote_node_id = int  |  Not supported                  SelectiveUDPInputSession
                      |
remote_node_id = None |  PromiscuousUDPInputSession     PromiscuousUDPInputSession
                      |

_udp.py:get_input_session:

becomes much closer get_output_session
_setup_input_session/_teardown_input_session are not required anymore due to much simpler setup procedure.

To make it a bit more clear for myself, this is how I see that UDPInputSession needs to look (mirroring with UDPOutputSession):
_input.py:

UDPInputSession: new member self._sock
async def receive(self, monotonic_deadline: float) -> typing.Optional[pycyphal.transport.TransferFrom]

def _process_frame(self, timestamp: Timestamp, frame: typing.Optional[UDPFrame]) -> None

def _get_reassembler(self, source_node_id: int) -> TransferReassembler (Promiscuous)
def _get_reassembler(self, source_node_id: int) -> TransferReassembler (Selective)

self._statistics_impl = PromiscuousUDPInputSessionStatistics() (Promiscuous)
self._statistics_impl = SelectiveUDPInputSessionStatistics() (Selective)

_output.py:

async def send(self, transfer: pycyphal.transport.Transfer, monotonic_deadline: float) -> bool

def construct_frame(index: int, end_of_transfer: bool, payload: memoryview) -> UDPFrame

self._statistics = pycyphal.transport.SessionStatistics()

Is this correct?

Some smaller questions/remarks:
udp/_ip/_v4.py:
# make_output_socket
elif isinstance(data_specifier, ServiceDataSpecifier):
    s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF, self._local_ip_addr.packed)
    s.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, IPv4SocketFactory.MULTICAST_TTL)
    remote_ip = service_node_id_to_multicast_group(remote_node_id)
    remote_port = DESTINATION_PORT

# make_input_socket
elif isinstance(data_specifier, ServiceDataSpecifier):
    multicast_ip = service_node_id_to_multicast_group(remote_node_id)
    multicast_port = DESTINATION_PORT
    if sys.platform.startswith("linux") or sys.platform.startswith("darwin"):
        s.bind((str(multicast_ip), multicast_port))
    else:
        s.bind(("", multicast_port))
    try:
        s.setsockopt(
            socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, multicast_ip.packed + self._local_ip_addr.packed
        )

Add a check if ServiceDataSpecifier is Request (for make_input_socket) or Response (for make_output_socket)?

In udp/_ip/_endpoint_mapping.py, we define:
SUBJECT_ID_MASK = 2**15 - 1

however:
# transport/_data_specifier.py
@dataclasses.dataclass(frozen=True)
class MessageDataSpecifier(DataSpecifier):
    SUBJECT_ID_MASK = 2**13 - 1

    subject_id: int

    def __post_init__(self) -> None:
        if not (0 <= self.subject_id <= self.SUBJECT_ID_MASK):
            raise ValueError(f"Invalid subject-ID: {self.subject_id}")

Any comment on this apparent discrepancy?

Is this correct?

Seems so, at least on the high level. I presume the next step would be to drop the no longer needed SocketReader, right?

Add a check if ServiceDataSpecifier is Request (for make_input_socket) or Response (for make_output_socket)?

I don't see what for. The request/response distinction does not seem to be manifested at the socket level, does it?

Any comment on this apparent discrepancy?

It is simply that Scott's multicast group address format reserves more bits for the subject-ID than necessary; that is all. You can see the same discrepancy in the Cyphal/serial header format. Although it seems inconsistent, ultimately, it does not matter, and it enables separate parts of the protocol to evolve independently.

I presume the next step would be to drop the no longer needed SocketReader, right?

Sure! (Currently leaving it in for reference)

I don't see what for. The request/response distinction does not seem to be manifested at the socket level, does it?

I was thinking maybe it would be good to warn the user if he tries to instantiate an input socket for a Response (which wouldn't make sense?). I guess it really doesn't matter for the actual input socket.

It is simply that Scott's multicast group address format reserves more bits for the subject-ID than necessary; that is all.

Ok, I though maybe this was something overlooked in Scott's proposal.

I've updated the _output_session unit test, there's this small inconsistency I don't get:

in pycyphal/transport/udp/_frame.py:

assert (
    memoryview(b"\x01\x07\x06\x00\xae\x08A\xc11\xd4\x00\x00\x00\x00\x00\x00\x01\x00\x00\x80\x00\x00\x03<"),
    memoryview(b"e"),
) == UDPFrame(
    priority=Priority.OPTIONAL,
    source_node_id=6,
    destination_node_id=2222,
    snm=True,
    subject_id=None,
    service_id=321,
    rnr=True,
    transfer_id=54321,
    index=1,
    end_of_transfer=True,
    user_data=0,
    payload=memoryview(b"e"),
).compile_header_and_payload()

in tests/transport/udp/_output_session.py:

assert data_main_b == (
    b"\x01\x07\x06\x00\xae\x08A\xc11\xd4\x00\x00\x00\x00\x00\x00\x01\x00\x00\x80\x00\x00V\x03"
    + b"e"
    + pycyphal.transport.commons.crc.CRC32C.new(b"one", b"two", b"three").value_as_bytes
)

Both check out correctly, however the bytes are not exactly the same (at the end the V and <).
What am I missing? The content is the same, but the header crc is somehow different?

I was thinking maybe it would be good to warn the user if he tries to instantiate an input socket for a Response (which wouldn't make sense?). I guess it doesn't really matter for the actual socket.

Maybe a check would make sense but then I urge you to think whether you are violating the principle of separation of concerns here. The job of the socket factory is to make sockets, not to check the protocol semantics.

What am I missing?

Here you seem to be computing the header CRC from the payload, whereas it should be computed from the header itself (as the name suggests):
https://github.com/maksimdrachov/pycyphal/blob/572deedd6359d41088e5fc1f9115210bed75c0de/pycyphal/transport/udp/_frame.py#L124
First you compile all fields except for the header into one binary blob, then compute its CRC and add it at the end.
There are two other major issues:

Adding new dependencies is not permitted. Please drop the dependency on the CRC library and use this instead: https://pycyphal.readthedocs.io/en/stable/api/pycyphal.transport.commons.crc.html#pycyphal.transport.commons.crc.CRC16CCITT

Mutual exclusion within a set of values should be expressed through the type system explicitly. This is very important for building self-explanatory and safe APIs. This means that this is not a good design:

    snm: bool  # Service, Not Message
    subject_id: int | None
    service_id: int | None
    rnr: bool | None  # Request, Not Response

While this is a solid alternative:
https://github.com/OpenCyphal/pycyphal/blob/75883ec57981ec9c3d659d471a7a2bad9c7f8c4c/pycyphal/transport/serial/_frame.py#L44
The definition of the DataSpecifier is here:
https://github.com/OpenCyphal/pycyphal/blob/75883ec57981ec9c3d659d471a7a2bad9c7f8c4c/pycyphal/transport/_data_specifier.py#L10-L51

@pavel-kirienko
Could you check the current implementation of pycyphal/transport/udp/_frame.py?
I think it should be correct now.

A rough update on the current status:

The last commits is working for the tests in run-unit-tests.sh, this is testing all the UDP-related unit tests (both from pycyphal as well as tests)

Even though it is "working", still some things remain to be done:

The system_ns error still need to be fixed, so far I've commented the checks out here:

tests/transport/udp/ip/link_layer.py
tests/transport/udp/ip/v4.py
tests/transport/udp/_udp.py:374

UDPTransportStatistics should be updated, the unit tests in tests/transport/udp/_udp.py looks kind of ugly right now.
Remove unnecessary debug prints
Update documentation
Update non-UDP failing unit tests

Refactoring of UDPTransportStatistics

_input.py:
class UDPInputSessionStatistics(pycyphal.transport.SessionStatistics):
    pass

@dataclasses.dataclass
class PromiscuousUDPInputSessionStatistics(UDPInputSessionStatistics):
    reassembly_errors_per_source_node_id: typing.Dict[
        int, typing.Dict[TransferReassembler.Error, int]
    ] = dataclasses.field(default_factory=dict)
    """
    Keys are source node-IDs; values are dicts where keys are error enum members and values are counts.
    """

@dataclasses.dataclass
class SelectiveUDPInputSessionStatistics(UDPInputSessionStatistics):
    reassembly_errors: typing.Dict[TransferReassembler.Error, int] = dataclasses.field(default_factory=dict)
    """
    Keys are error enum members and values are counts.
    """

_output.py:
# this is a new class!
class UDPOutputSessionStatistics(pycyphal.transport.SessionStatistics):
    sent_frames_per_destination_node_id: typing.Dict[int, int] = dataclasses.field(default_factory=dict)
    frames_timeout_per_destination_node_id: typing.Dict[int, int] = dataclasses.field(default_factory=dict)
    """
    Keys are destination node-IDs; values are number of frames sent.
    """

_udp.py:
class UDPTransportStatistics(pycyphal.transport.TransportStatistics):
    input_sessions: typing.Dict[
        int, typing.Union[PromiscuousUDPInputSessionStatistics, SelectiveUDPInputSessionStatistics]
    ] = dataclasses.field(default_factory=dict)
    """
    Keys are source node-IDs; values are session statistics.
    """
    output_sessions: typing.Dict[int, UDPOutputSessionStatistics] = dataclasses.field(default_factory=dict)
    """
    Keys are destination node-IDs; values are session statistics.
    """

or:
@dataclasses.dataclass
class UDPTransportStatistics(pycyphal.transport.TransportStatistics):
    received_datagrams: typing.Dict[pycyphal.transport.DataSpecifier, typing.Dict[int, int]] = dataclasses.field(
        default_factory=dict
    )
    """
    Basic input session statistics: keyed by data specifier, values are dicts keyed by source node ID
    (values of second dict are number of datagrams received)
    """
    sent_datagrams: typing.Dict[pycyphal.transport.DataSpecifier, typing.Dict[int, int]] = dataclasses.field(
        default_factory=dict
    )
    """
    Basic output session statistics: keyed by data specifier, values are dicts keyed by destination node ID
    (values of second dict are number of datagrams sent)
    """

Which one of UDPTransportStatistics do you prefer?

diff --git a/pycyphal/transport/udp/_session/_input.py b/pycyphal/transport/udp/_session/_input.py
index 80c4053..e7733a0 100644
--- a/pycyphal/transport/udp/_session/_input.py
+++ b/pycyphal/transport/udp/_session/_input.py
@@ -11,6 +11,7 @@ import typing
 import select
 import asyncio
 import logging
+import threading
 import dataclasses
 import pycyphal
 from pycyphal.transport import Timestamp
@@ -86,7 +87,11 @@ class UDPInputSession(pycyphal.transport.InputSession):
         assert isinstance(self._payload_metadata, pycyphal.transport.PayloadMetadata)
         assert callable(self._finalizer)
         self._transfer_id_timeout = self.DEFAULT_TRANSFER_ID_TIMEOUT
-        self._queue: asyncio.Queue[pycyphal.transport.TransferFrom] = asyncio.Queue()
+        self._frame_queue: asyncio.Queue[typing.Tuple[Timestamp, UDPFrame | None]] = asyncio.Queue()
+        self._thread = threading.Thread(target=self._reader_thread, name=str(self),
+                                        args=(asyncio.get_running_loop(),),
+                                        daemon=True)
+        self._thread.start()

     async def receive(self, monotonic_deadline: float) -> typing.Optional[pycyphal.transport.TransferFrom]:
         """
@@ -98,62 +103,56 @@ class UDPInputSession(pycyphal.transport.InputSession):
         """
         if self._closed:
             raise pycyphal.transport.ResourceClosedError(f"{self} is closed")
-
-        transfer_ready = False
-
-        consume_success = await self._consume(monotonic_deadline=monotonic_deadline)
-        while consume_success:  # if something was consumed, try to consume more
-            transfer_ready = True
-            consume_success = await self._consume(monotonic_deadline=monotonic_deadline)
-
-        if transfer_ready:
-            _logger.debug("%s: Consumed a datagram", self)
-            loop = asyncio.get_running_loop()
+        loop = asyncio.get_running_loop()
+        while True:
+            timeout = monotonic_deadline - loop.time()
             try:
-                timeout = monotonic_deadline - loop.time()
                 if timeout > 0:
-                    transfer = await asyncio.wait_for(self._queue.get(), timeout)
+                    ts, frame = await asyncio.wait_for(self._frame_queue.get(), timeout=timeout)
                 else:
-                    transfer = self._queue.get_nowait()
+                    ts, frame = self._frame_queue.get_nowait()
             except (asyncio.TimeoutError, asyncio.QueueEmpty):
                 # If there are unprocessed transfers, allow the caller to read them even if the instance is closed.
                 if self._finalizer is None:
                     raise pycyphal.transport.ResourceClosedError(f"{self} is closed") from None
                 return None
+            if frame is None:
+                self._statistics.errors += 1
+                continue
+            if not self.specifier.is_promiscuous:
+                if frame.source_node_id != self._specifier.remote_node_id:
+                    continue
+            self._statistics.frames += 1
+            source_node_id = frame.source_node_id
+            assert isinstance(source_node_id,
+                              int) and 0 <= source_node_id <= NODE_ID_MASK, "Internal protocol violation"
+            # Anonymous - no reconstruction needed
+            if source_node_id == NODE_ID_MASK:
+                transfer = TransferReassembler.construct_anonymous_transfer(ts, frame)
+            # # Single-frame transfer - no reconstruction needed # TODO FIXME
+            # elif frame.single_frame_transfer:
+            #     transfer = TransferReassembler.construct_uniframe_transfer(timestamp, frame)
+            # Otherwise, fetch the reassembler and process the frame
             else:
-                assert isinstance(transfer, pycyphal.transport.TransferFrom), "Internal protocol violation"
-                assert (
-                    transfer.source_node_id == self._specifier.remote_node_id or self._specifier.remote_node_id is None
-                )
+                _logger.debug("%s: Processing frame: %s", self, frame)
+                transfer = self._get_reassembler(source_node_id).process_frame(ts, frame, self._transfer_id_timeout)
+            if transfer is not None:
+                self._statistics.transfers += 1
+                self._statistics.payload_bytes += sum(map(len, transfer.fragmented_payload))
+                _logger.debug("%s: Received transfer: %s; current stats: %s", self, transfer, self._statistics)
                 return transfer
-        else:  # No datagrams were consumed
-            return None
-
-    async def _consume(self, monotonic_deadline: float) -> bool:
-        """
-        This method is used to consume the datagrams from the socket.
-        The method will block until a datagram is available or the deadline is reached.
-        If a datagram is read, it will call the underlying ``_process_frame()`` method
-        of the input session (PromiscuousInputSession or SelectiveInputSession).
-        If no datagram is read and the deadline is reached, the method will return ``False``.
-        """
-        loop = asyncio.get_running_loop()

-        while self._sock.fileno() >= 0:
+    def _reader_thread(self, loop: asyncio.AbstractEventLoop) -> None:
+        while not self._closed and self._sock.fileno() >= 0:
             try:
-                _logger.debug("monotonic_deadline-loop.time()=%s", monotonic_deadline - loop.time())
-                read_ready, _, _ = select.select([self._sock], [], [], monotonic_deadline - loop.time())
+                read_ready, _, _ = select.select([self._sock], [], [], 1.0)
                 if self._sock in read_ready:
                     # TODO: use socket timestamping when running on GNU/Linux (Windows does not support timestamping).
                     ts = pycyphal.transport.Timestamp.now()
-
                     # Notice that we MUST create a new buffer for each received datagram to avoid race conditions.
                     # Buffer memory cannot be shared because the rest of the stack is completely zero-copy;
                     # meaning that the data we allocate here, at the very bottom of the protocol stack,
                     # is likely to be carried all the way up to the application layer without being copied.
-                    # await asyncio.wait_for(
-                    #     loop.sock_recv_into(self._sock, _READ_SIZE), timeout=monotonic_deadline - loop.time()
-                    # )
                     data, endpoint = self._sock.recvfrom(_READ_SIZE)
                     assert len(data) < _READ_SIZE, "Datagram might have been truncated"
                     frame = UDPFrame.parse(memoryview(data))
@@ -164,49 +163,10 @@ class UDPInputSession(pycyphal.transport.InputSession):
                         endpoint,
                         frame,
                     )
-                    self._process_frame(ts, frame)
-                    return True
-                else:
-                    return False  # there was no datagram to read
-            except (asyncio.TimeoutError):
-                return False
+                    # TODO Handle QueueFull
+                    loop.call_soon_threadsafe(lambda: self._frame_queue.put_nowait((ts, frame)))
             except Exception as ex:
                 _logger.exception("%s: Exception while consuming UDP frames: %s", self, ex)
-                time.sleep(1)
-                return False
-
-    def _process_frame(self, timestamp: pycyphal.transport.Timestamp, frame: UDPFrame) -> None:
-
-        if frame is None:
-            self._statistics.errors += 1
-            return
-        if not self.specifier.is_promiscuous:
-            if frame.source_node_id != self._specifier.remote_node_id:
-                return
-        self._statistics.frames += 1
-
-        source_node_id = frame.source_node_id
-        assert isinstance(source_node_id, int) and 0 <= source_node_id <= NODE_ID_MASK, "Internal protocol violation"
-
-        # Anonymous - no reconstruction needed
-        if source_node_id == NODE_ID_MASK:
-            transfer = TransferReassembler.construct_anonymous_transfer(timestamp, frame)
-        # # Single-frame transfer - no reconstruction needed
-        # elif frame.single_frame_transfer:
-        #     transfer = TransferReassembler.construct_uniframe_transfer(timestamp, frame)
-        # Otherwise, fetch the reassembler and process the frame
-        else:
-            _logger.debug("%s: Processing frame: %s", self, frame)
-            transfer = self._get_reassembler(source_node_id).process_frame(timestamp, frame, self._transfer_id_timeout)
-
-        if transfer is not None:
-            self._statistics.transfers += 1
-            self._statistics.payload_bytes += sum(map(len, transfer.fragmented_payload))
-            _logger.debug("%s: Received transfer: %s; current stats: %s", self, transfer, self._statistics)
-            try:
-                self._queue.put_nowait(transfer)
-            except asyncio.QueueFull:
-                self._statistics.drops += len(transfer.fragmented_payload)  # queue_overflows

     @property
     def transfer_id_timeout(self) -> float:
@@ -228,9 +188,13 @@ class UDPInputSession(pycyphal.transport.InputSession):
         return self._payload_metadata

     def close(self) -> None:
+        self._closed = True
         if self._finalizer is not None:
             self._finalizer()
             self._finalizer = None
+        # TODO See https://github.com/OpenCyphal/pycyphal/blob/6a432b10c4307b006067fd0bdc867bc17790fa67/pycyphal/transport/udp/_socket_reader.py#L152-L162
+        # (EXTREMELY IMPORTANT)
+        self._sock.close()

     @property
     def socket(self) -> socket_.socket:
diff --git a/pycyphal/transport/udp/_session/_output.py b/pycyphal/transport/udp/_session/_output.py
index 2ae5734..2540ef0 100644
--- a/pycyphal/transport/udp/_session/_output.py
+++ b/pycyphal/transport/udp/_session/_output.py
@@ -201,8 +201,6 @@ class UDPOutputSession(pycyphal.transport.OutputSession):
                     loop.sock_sendall(self._sock, b"".join((header, payload))),
                     timeout=monotonic_deadline - loop.time(),
                 )
-                _logger.debug("sent completed")
-                assert False
                 # TODO: use socket timestamping when running on Linux (Windows does not support timestamping).
                 # Depending on the chosen approach, timestamping on Linux may require us to launch a new thread
                 # reading from the socket's error message queue and then matching the returned frames with a
diff --git a/tests/transport/redundant/_redundant.py b/tests/transport/redundant/_redundant.py
index 6aa094c..89374af 100644
--- a/tests/transport/redundant/_redundant.py
+++ b/tests/transport/redundant/_redundant.py
@@ -224,7 +224,7 @@ async def _unittest_redundant_transport(caplog: typing.Any) -> None:
     tr_a.attach_inferior(udp_a)
     # tr_a.attach_inferior(serial_a)

-    # tr_b.attach_inferior(udp_b)
+    tr_b.attach_inferior(udp_b)
     # tr_b.attach_inferior(serial_b)

     # assert tr_a.protocol_parameters == ProtocolParameters(
@@ -247,7 +247,7 @@ async def _unittest_redundant_transport(caplog: typing.Any) -> None:
         Transfer(
             timestamp=Timestamp.now(), priority=Priority.LOW, transfer_id=5, fragmented_payload=[memoryview(b"uio")]
         ),
-        monotonic_deadline=loop.time() + 1.0,
+        monotonic_deadline=loop.time() + 10.0,
     )
     _logger.debug("=================pub_a.send() completed=================")
     rx = await sub_any_b.receive(loop.time() + 1.0)

Concerning last commit:
I changed to reader_thread from asyncio, to a regular thread. Is this ok?
The reason is that some unit tests would fail due to not having having any asyncio.get_running_loop() to call.
This could be fixed by doing the following (in UDPInputSession::init()):
# create asyncio EventLoop, if there is none
try:
    self._loop = asyncio.get_running_loop()
except RuntimeError:
    self._loop = asyncio.new_event_loop()

However then, some unit test would fail due to loop.call_soon_threadsafe(lambda:self._frame_queue.put_nowait((ts, frame))) never getting executed. (The frane would get read from the Socket, but never actually put in the Queue).
This is solved by using self._frame_queue.put_nowait((ts, frame)) instead.

Concering the construct_uniframe, it has been omitted: process_frame is able to handle both uni-, as multiframes.

pycyphal

[ ] application

[ ] FAILED init.py::pycyphal.application  - no running event loop, doctest

[x] dsdl
[x] presenation
[ ] transport

[ ] FAILED udp/init.py::pycyphal.transport.udp - no running event loop, doctest

tests

[ ] application

[ ] FAILED file.py::_unittest_file - AssertionError: assert (PosixPath('/...ixPath('abc')) == (PosixPath('/...ixPath('abc'))
[ ] FAILED node.py::_unittest_slow_node - AssertionError: assert 0 == 258
[ ] FAILED node_tracker.py::_unittest_slow_node_tracker - assert 1 == 2

[ ] demo

[ ] FAILED _demo_app.py::_unittest_slow_demo_app[parameters0] - assert 123 == 42
[ ] FAILED _demo_app.py::_unittest_slow_demo_app[parameters2] - assert 123 == 42
[ ] FAILED _demo_app.py::_unittest_slow_demo_app_with_plant[run_config2] - assert 23 <= 20

[x] dsdl
[x] presentation
[x] transport

still todo:

[ ] go once more over unit tests
[ ] cleanup debug prints
[ ] update documentation
[ ] remove run-unit-tests.sh

@pavel-kirienko @thirtytwobits
Reminder: choose Cyphal port?

Docs preview https://pycyphal--253.org.readthedocs.build/en/253/api/pycyphal.transport.udp.html

Regarding PyLint:

Please disable the following lints in PyLint: import-error, line-too-long (these are covered by MyPy and Black).
The issue that violates the no-else-return rule should be fixed.
Not sure why you need to access protected data members. If this is done from a test suite, then please add suppression comments, otherwise, the code requires fixing.

ok
fixed
you guessed it, added suppression comment

The problem with Windows remains.
Take the following example code (generated by ChatGPT):
import socket
import struct

MCAST_GRP = "<IP_ADDRESS>"
MCAST_PORT = 5007

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("", MCAST_PORT))

mreq = struct.pack("4sl", socket.inet_aton(MCAST_GRP), socket.INADDR_ANY)

sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

# Retrieve the membership information for the socket
local_addr, local_port = sock.getsockname()
print(f"Subscribed to multicast group: {local_addr}")

while True:
    print(sock.recv(10240))

Gives following result:

Another suggestion, using getsockopt():
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

sock.bind(("<IP_ADDRESS>", 12345))

# join the multicast group
group = "<IP_ADDRESS>"
interface = "<IP_ADDRESS>"
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, socket.inet_aton(group) + socket.inet_aton(interface))

mcast_addr = sock.getsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF)

print(f"Multicast address: {mcast_addr}")

From the jit.si OpenCyphal meeting this morning:
@maksimdrachov there might be an issue when trying to bind to all multicast groups
import socket
import struct

MCAST_GRP = "<IP_ADDRESS>"
MCAST_PORT = 5007

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((MCAST_GRP, MCAST_PORT))

mreq = struct.pack("4sl", socket.inet_aton(MCAST_GRP), socket.INADDR_ANY)
print(f"socket.inet_aton(MCAST_GRP): {socket.inet_aton(MCAST_GRP)}")
print(f"mreq: {mreq}")

sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

# Retrieve the membership information for the socket
local_addr, local_port = sock.getsockname()
print(f"Subscribed to multicast group: {local_addr}")
print(f"Local Port: {local_port}")

while True:
    print(sock.recv(10240))

This results in
socket.inet_aton(MCAST_GRP): b'\xe0\x01\x01\x01'
mreq: b'\xe0\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Subscribed to multicast group: <IP_ADDRESS>
Local Port: 5007

Not sure if the intent was to bind to all groups or just the specific multicast group. If you want to bind to all groups there might be some extra steps that need to be taken.

Looks like ChatGPT was getting its examples from: https://stackoverflow.com/questions/603852/how-do-you-udp-multicast-in-python

Thanks!
The problem is only when using Windows, you can't bind to MCAST_GRP:

It works however, if you don't bind:

(And it is really only receiving from that specific multicast address, i've checked)
Now, however there doesn't seem to be a straightforward way to find out which multicast group sock is subscribed to (without referring to MCAST_GRP).

For future reference (code example of this problem):
https://github.com/maksimdrachov/multicast-question
Retrieving membership information through getsockopt doesn't seem to be an option:
https://tldp.org/HOWTO/Multicast-HOWTO-6.html

This is easy to fix: make UDPTransport accept IPAddress|str; if the argument is a string, use ipaddress.ip_address(str) to construct IPAddress.
Aside from that (and a few related type errors), the only remaining issue is that of the statistics. https://github.com/OpenCyphal/pycyphal/issues/279

Please also bump the minor version number and add a new section to the changelog.


API CHANGE. I can’t sign any more from a new account today.

Please follow the guide below

Issues submitted without this template format will be ignored.
You will be asked some questions and requested to provide some information, please read them carefully and answer completely.
Put an x into all the boxes [ ] relevant to your issue (like so [x]).
Use the Preview tab to see how your issue will actually look like.

Before submitting an issue, make sure you have:

[X] Updated to the latest viersion
[X] Read the README and docs
[X] Searched the bugtracker for similar issues including closed ones

Purpose of your issue?

[X] Bug report (encountered problems/errors)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections requests more details for particular types of issues, you can remove any section (the contents between the triple ---) not applicable to your issue.

For a bug report, you must include the Python version used, code that will reproduce the error, and the error log/traceback.
Paste the output of python -V here:
Code:
Error/Debug Log:
2018-04-11 19:56:02,926 - INFO - Instagram error message: challenge_required
2018-04-11 19:56:02,926 - INFO - Error type: checkpoint_challenge_required
2018-04-11 19:56:02,926 - INFO - Login or password is incorrect

Describe your issue
I think that something change in Instagram API. I have like 10 Instagram accounts working good but I try to add 3 new Instagram account to Instabot and I got the same error.
I did a lot of login confirmation in my Instagram App but It doesn’t working too. There is the same problem.
I think that Instagram could ask for a SMS code when we login via API now.
Can somebody help me? Am I right?

I think That its related to cookie file and crsftoken try to rename cookie.txt and let bot create new one

Same here.
This worked  great for me
https://github.com/instagrambot/instabot/issues/491#issuecomment-380574716

Something is happening... I press "That was me" from another Instagram App and It still ask me the "challenge". I delete Cookie file and anything is working. This could be the end for this script.
I try to login from other phone, then Instagram ask me for a "challege" with SMS Code. I put the SMS Code and It works now, but only with SMS Code and not "That was me"".


Allow Errors in Queue Tasks to reject Promise

Since some promise libraries automatically convert an exception into a rejection, and because the promise returned from the queue is not the same one that is used by the queue management, it was causing the exception to be completely dropped.
First attempt to fix this was adding a second then handler, which worked, however caused a couple of new issues, because the rejection is handled in the calling context, the item that is rejected is the previous queue item.  Getting the next queue item looks to be complex and would potentially have issues with queues that run more than 1 item at a time.
So since promises already essentially wrap a try catch, we wrap this around the main calling stack in dequeue, this allows the exception to be handled in the same stack it was thrown in, everything gets called in the correct order.  This should mean the try/catch example in the README is no longer needed as it will handle it internally automatically and pass it to the reject, instead of only try/catching only when the queue was empty when it was added.
This should resolve the issue in #6

Hey, thank you for the pull request! Sorry that it took so much time to review!
<EMAIL_ADDRESS>


Transducer Beam Search Implemented original algorithm from paper

Implemented original algorithm from paper  : https://arxiv.org/pdf/1911.01629.pdf
This is a fix for the following issue : https://github.com/speechbrain/speechbrain/issues/1969

@Adel-Moumen, what should we do with this PR?


fix: null values in property display breaking search

When using displaying properties in a search, that property is unset in one of the files search, the respective search breaks, and the search is missing a few results. Also, the navigation is totally broken, until the query reduces the results to a list of files where the property is always set, effectively making the search mostly unusable in such cases.
Uncaught (in promise) TypeError: Cannot read properties of null (reading 'toString')
    at createMetaDiv (plugin:obsidian-another-quick-switcher:2853:20)
    at createElements (plugin:obsidian-another-quick-switcher:2973:73)
    at _AnotherQuickSwitcherModal.renderSuggestion (plugin:obsidian-another-quick-switcher:3351:50)
    at e.setSuggestions (app.js:1:1583474)
    at i (app.js:1:1586474)
    at t.updateSuggestions (app.js:1:1586534)
    at _AnotherQuickSwitcherModal.onOpen (plugin:obsidian-another-quick-switcher:3087:10)
    at e.open (app.js:1:1476803)
    at showSearchDialog (plugin:obsidian-another-quick-switcher:5807:9)
    at callback (plugin:obsidian-another-quick-switcher:5964:11)

I looked into it, and the cause is a missing null-check when creating the div for a property value.
This PR fixes this issue by adding the missing null check.

@chrisgrieser
Thank you for your PR :)
However, I am unable to reproduce the issue in my environment. Could you provide the steps to reproduce it in the Sandbox Vault?

I cannot reproduce it in the sandbox vault, but it does break in any other one of my vaults.
Maybe it's some sort of invalid frontmatter? in any case, the fix should be relatively straight forward, without any downsides I can see?
for (const v of [value].flat()) {
  if (!v) continue; // <- added this
  frontMatterDiv.createSpan({
    cls: "another-quick-switcher__item__meta__front_matter__value",
    title: v.toString(),
    text: v.toString(),
  });
}

Here, I can reproduce it in the sandbox with this:

Install AQS
AQS settings: enable "show frontmatter" for the "recent search"
in a file, add this as frontmatter:

---
foobar:
  - null
  - "baz"
---

4. go to another file (since AQS ignores the current file)
5. trigger "recent search"
6. the error then appears in the console.

Thank you. I have confirmed the reproduction.
However, with this change, there is a possibility that 0 or false might be excluded, so I will make some adjustments after the merge 👍

Right, good call. Thank you!

Released in v11.6.0 🚀


Separate out FAQ sections into individual subpages

I think it's better to separate the FAQ into a separate section (which includes different FAQs for each type like Blockchain, wallet... ) so that everyone can see and access it right outside of the portal.

Individual pages already have links to the faq, but i'll play with the idea of subpages under faq and see too how that feels


Change hierarchy writer to only write relationships in one direction

We now only write in the direction episode -> series -> brand only.
This is for performance reasons as we have content like flat brands
with thousands of episodes that is very slow to update if all of
these relationships have to be updated every time

lgtm


remove gulp-util using the following API replacements:

gutil.File => https://www.npmjs.com/package/vinyl
gutil.replaceExtension => The .extname property on Vinyl objects or https://www.npmjs.com/package/replace-ext
gutil.colors => https://www.npmjs.com/package/ansi-colors
gutil.date => https://www.npmjs.com/package/date-format
gutil.log => https://www.npmjs.com/package/fancy-log
gutil.template => https://www.npmjs.com/package/lodash.template
gutil.env => https://www.npmjs.com/package/minimist
gutil.beep => https://www.npmjs.com/package/beeper
gutil.noop => https://www.npmjs.com/package/through2
gutil.isStream => Use the .isStream() method on Vinyl objects
gutil.isBuffer => Use the .isBuffer() method on Vinyl objects
gutil.isNull => Use the .isNull() method on Vinyl objects
gutil.linefeed => Use the string '\n' in your code
gutil.combine => https://www.npmjs.com/package/multipipe
gutil.buffer => https://www.npmjs.com/package/list-stream
gutil.PluginError => https://www.npmjs.com/package/plugin-error

gulp-util nor gulp is not used at all in this repo. It may be a transient dependency, but that's for the packages that require it to handle.


feat(clouddns): Allow deletion of RRs

This was stubbed out before, now implemented.

Current dependencies on/for this PR:

main

PR #5   👈

This stack of pull requests is managed by Graphite.


Players cant get in creationmode

My players cant get i creations mode even though they have both hotels.create.user and hotels.createmode.user
I can do it but i also have the * permission so the plugin works just not the permissions

Please follow these steps and check in-between each step if the problem is still present:

Try revoking those permissions and seeing if there is a change.
Give those permissions back and check if the problem is still present
Also give them hotels.sign.create.user
Also give them worldedit.selection.pos

Ive still got the problem i have followed all your steps but it still dont work
Then ive tried to get them the hotels.* and then it works so the plugin works and all the other permissions than the hotels creations mode

Hmm I think I have changed that permission to hotels.creationmode but never updated the wiki

Cool it works now thanks!!


macOS Game Preview location

Operating system or device, Godot version, GPU Model and driver (if graphics related):
macOS 10.13.1
Godot 3 Alpha Build 2
Radeon Pro 460 4 GB
Intel HD Graphics 530 1536 MB
Issue description:
Working in Godot 3 , Press play to view game. the window with game appears but is not in the middle of the screen but bottom right obscuring half of it. Moving screen to center the new position is not remembered. See Image attached

Same issue when using same Spaces
Steps to reproduce:
Open project, press play

Duplicate of #7477 will be fixed by #12606


Bench against lychee

Do basic benchmark against https://github.com/hello-rust/lychee. Need to disable a lot of features in lychee to get comparable featureset, as usual.
Blocker: https://github.com/hello-rust/lychee/issues/21
So practically lychee can currently only validate external links until that issue is resolved, while hyperlink can only validate relative links

this makes no sense for as long as lychee does not recurse into directories or websites

lychee is still quite a bit slower but really just does a lot more, added to readme


Bootstrap event in app for window

If a window is created and initially shown, there need to be a bootstrap event like on_bootstrap( window_id, frontend ) so the user can do the very initial setup which is usually different form run time object creation/update.
This safes the user from doing it by himself.

This is now implemented with flag in the render data.


Disallow non-integer numbers to be passed to Int ports

This should solve the issue I reported in #1162.
It adds additional checks performed when accepting a value to an incoming port of type Int.
The check is based on a polyfill for Number.isInteger. NaN and infinite values are handled correctly (and not accepted.
The output for the test case I posted looks as follows:
Port Error:

    Regarding the port named 'int' with type:

        Int

    You just sent the value:

        3.14

    but it cannot be converted to the necessary type.
    Runtime error when sending values through a port.

    Expecting an integer but was given 3.14

Open the developer console for more details.

:+1:

Okay, needed to rebase, then cleaned up some stuff. New thing is at #1175!

Thank you!


Added edit link tag helper with readme description too

@warrenbuckley. PR updated. I have simplified the edit link code a lot, taking out most of the styling options and leaving a simple bool to say if they want to use default styles or not.
I couldn't find a way to get the edit text if used like this:
<our-edit-link content-id="@(Model.Id)">
    Get this text
</our-edit-link>

So I just made it an attribute instead called text and this value defaults to "Edit"
I've updated the readme too, with a few more examples and notes on the attributes you can change.


merge from arlac77/template-github,arlac77/template-kronos-module

package.json

chore(package): add ^3.7.1 remove ^3.7.0 (devDependencies.ava)

:tada: This PR is included in version 10.0.11 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Support project configurations in a meaningful way

We do currently support the notion of different project configurations for PTVS projects, but they're pretty useless in practice for anything other than custom build actions, because all our project properties are not config-specific. Even if you do manually edit the project file to make them config-specific, we do not actually respect this in practice.
There are several interesting scenarios for varying properties with configs. One would be to allow quickly switching interpreters in sync for several projects in the solution (by specifying different interpreters for different project configs, and bind them all to the matching solution configs). Another is to change the search path depending on target interpreter, so as to use a proper version (2.7/3.3, x86/x64) of a binary module - this is particularly useful when said modules are built from a C++ project in the same solution in mixed code scenarios.
Ideally, we should make all project properties config-dependent. The most interesting ones are:

interpreter to use
search path
startup file

In addition, we should also consider supporting config|platform combinations for all the available platforms (Win32/x64/ARM) in addition to AnyCPU. While they don't really make much sense for Python itself, they do make sense once native modules are involved - so, again, the search path between x86 and x64 could differ, for example.
(Migrated from https://pytools.codeplex.com/workitem/1502)

Many pieces of this have been added already for the IoT project support, so we should be able to pick this up more easily for other Python projects.


Restore fails with "failed to check s3 bucket" error

I am trying to restore a backup to a new cluster. Using instructions here: https://rancher.com/docs/rancher/v2.6/en/backups/migrating-rancher/.  Restore is failing with this error:
INFO[2022/03/20 16:50:50] invoking set s3 service client                s3-accessKey="base64encodedaccess key=" s3-bucketName=mybucket s3-endpoint=s3.us-west-2.amazonaws.com s3-endpoint-ca= s3-folder= s3-region=us-west-2
ERRO[2022/03/20 16:50:50] error syncing 'restore-migration': handler restore: failed to check s3 bucket:mybucket, err:400 Bad Request, requeuing

Backups are in an s3 bucket. I have verified that I am using the same credentials that were used to create the backup,  And the path to the backup is correct.  What else could be wrong?

What version was and is used (if they are different)?
Also please share the exact config used (redacted) and I assume Amazon S3 is used as storage.

The old Rancher is a docker install, Version 2.6.2. The new Cluster is a EKS cluster, K8s 1.20.  Yes, S3 is used for storage.  I did not install rancher on the new cluster.
It's not a large installation, only one downstream cluster.
What would happen if I did the following:

Install rancher on the new cluster.
Remove rancher from the downstream cluster
Import the downstream cluster into the new rancher?

What would be the down side of doing it that way?  What kinds of things would I likely loose and have to re-create?
My Config for creating the restore:
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
  name: restore-migration
spec:
  backupFilename: nightly-65eff810-5027-4527-b0ab-81ac234768f0-2022-03-20T00-00-00Z.tar.gz
  prune: false
  storageLocation:
    s3:
      credentialSecretName: s3-creds
      credentialSecretNamespace: default
      bucketName: rancher-backups
      region: us-west-2
      endpoint: s3.us-west-2.amazonaws.com

S3 Secret
apiVersion: v1
kind: Secret
metadata:
  name: s3-creds
type: Opaque
stringData:
  accessKey: ACCESSKEYBASE64ENCODED
  secretKey: SECRETKEYBASE64ENCODED

On the new cluster, I installed the backup but using the following command.
helm repo add rancher-charts https://charts.rancher.io
helm repo update
helm install rancher-backup-crd rancher-charts/rancher-backup-crd -n cattle-resources-system --create-namespace --version 2.1.0
helm install rancher-backup rancher-charts/rancher-backup -n cattle-resources-system --version 2.1.0

On the old cluster I had installed it through the ui

And the bucket and the credentials are 100% identical? Were you able to restore from it on the original setup? Seems very weird to have a difference in exactly the same code with exactly the same configuration.

Yes, I tracked it down to https://github.com/rancher/docs/pull/3594. It was generically changed to base64-encoded values but it wasn't taken into account that it depends on using data or stringData.

With the fix of the credentials I got a little further. But I still think something is missing. I am now getting this error:
I```
NFO[2022/03/31 13:58:29] Processing Restore CR restore-migration
INFO[2022/03/31 13:58:29] Restoring from backup rch2/nightly-backup-file.tar.gz
INFO[2022/03/31 13:58:29] invoking set s3 service client                s3-accessKey=REDACTED s3-bucketName=my-rancher-backups s3-endpoint=s3.us-west-2.amazonaws.com s3-endpoint-ca= s3-folder= s3-region=us-west-2
INFO[2022/03/31 13:58:29] Temporary location of backup file from s3: /tmp/nightly-vetpro-65eff810-5027-4527-b0ab-81ac234768f0-2022-03-31T00-00-00Z.tar.gz
INFO[2022/03/31 13:58:29] Successfully downloaded [rch2/nightly-backup-file.tar.gz]
INFO[2022/03/31 13:58:30] Processing controllerRef apps/v1/deployments/rancher
WARN[2022/03/31 13:58:30] Error getting object for controllerRef rancher, skipping it
INFO[2022/03/31 13:58:30] Starting to restore CRDs for restore CR restore-migration
INFO[2022/03/31 13:58:30] restoreResource: Restoring apprevisions.project.cattle.io of type apiextensions.k8s.io/v1, Resource=customresourcedefinitions
INFO[2022/03/31 13:58:30] Processing controllerRef apps/v1/deployments/rancher
WARN[2022/03/31 13:58:30] Error getting object for controllerRef rancher, skipping it
ERRO[2022/03/31 13:58:30] Error restoring CRDs restoreCRDs: restoreResource: err updating resource CustomResourceDefinition.apiextensions.k8s.io "apprevisions.project.cattle.io" is invalid: [spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required, spec.preserveUnknownFields: Invalid value: true: cannot set to true, set x-preserve-unknown-fields to true in spec.versions[*].schema instead]
ERRO[2022/03/31 13:58:30] error syncing 'restore-migration': handler restore: error restoring CRDs, check logs for exact error, requeuing

Don't know if it matters but my source cluster is running Rancher 2.6.1

This is the same as https://github.com/rancher/backup-restore-operator/issues/186, was the original cluster where the backup was taken upgraded (both k8s and rancher) in its lifetime?

Yes it was.
I looked at that article.  But I don't understand how to fix it.  There is a command listed there but the person says "i have no idea if this breaks anything".  Plus it does not specify where to run it?  Do I execute inside the docker container?
Or would it be sufficient to uninstall backup from source cluster and re-install it?

Thanks!!

Updates will be in https://github.com/rancher/backup-restore-operator/issues/186, this issue remains open to validate https://github.com/rancher/backup-restore-operator/pull/204

Root cause
Errors were unclear regarding S3 operations, although the original issue is followed up on by the docs team (incorrect way of creating secrets), the logging was not very clear either. The PR should make it more clear
What was fixed, or what changes have occurred

Ability to configure -debug or -trace to the deployment so you can configure debug or trace log level.
S3 secret logging, when providing a wrong/incomplete/badly structured secret, the logging should indicate what is wrong
Added filename and bucket to info and error log lines for context

Areas or cases that should be tested
Just regular cases should show the new logging, and when s3 secret is incorrectly structured (no accessKey or no secretKey for example)
What areas could experience regressions?
Nothing specific as it's all logs only
Are the repro steps accurate/minimal?

Add -debug or -trace to the deployment and it should log its setting the configured level
Create a badly structured s3 secret and check the logs

I upgraded to the newest version.  Now I am getting this error on the restore:  As you can see it now downloads the backup, but the restores still fails.
INFO[2022/05/01 15:26:50] Temporary location of backup file from s3: /tmp/nightly-vetpro-65eff810-5027-4527-b0ab-81ac234768f0-2022-05-01T00-00-00Z.tar.gz
INFO[2022/05/01 15:26:50] Successfully downloaded [nightly-vetpro-65eff810-5027-4527-b0ab-81ac234768f0-2022-05-01T00-00-00Z.tar.gz]
INFO[2022/05/01 15:26:51] Processing controllerRef apps/v1/deployments/rancher
WARN[2022/05/01 15:26:51] Error getting object for controllerRef rancher, skipping it
INFO[2022/05/01 15:26:51] Starting to restore CRDs for restore CR restore-migration
INFO[2022/05/01 15:26:51] restoreResource: Restoring resourcesets.resources.cattle.io of type apiextensions.k8s.io/v1, Resource=customresourcedefinitions
INFO[2022/05/01 15:26:51] Successfully restored resourcesets.resources.cattle.io
INFO[2022/05/01 15:26:51] restoreResource: Restoring projectnetworkpolicies.management.cattle.io of type apiextensions.k8s.io/v1, Resource=customresourcedefinitions
INFO[2022/05/01 15:26:51] Processing controllerRef apps/v1/deployments/rancher
WARN[2022/05/01 15:26:51] Error getting object for controllerRef rancher, skipping it
ERRO[2022/05/01 15:26:51] Error restoring CRDs restoreCRDs: restoreResource: err updating resource CustomResourceDefinition.apiextensions.k8s.io "projectnetworkpolicies.management.cattle.io" is invalid: [spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required, spec.preserveUnknownFields: Invalid value: true: cannot set to true, set x-preserve-unknown-fields to true in spec.versions[*].schema instead]
ERRO[2022/05/01 15:26:51] error syncing 'restore-migration': handler restore: error restoring CRDs, check logs for exact error, requeuing

For the preserveUnknownFields fix (https://github.com/rancher/backup-restore-operator/issues/186), you will need to install v2.1.2 RC currently as there is no official release with the fix yet.

v2.1.2 RC of the backup-operator?

Yes, v2.1.2 will be released with Rancher v2.6.5 but if you want to use it now, you have to use an RC version of the backup-restore-operator. Here is the latest RC: https://github.com/rancher/backup-restore-operator/releases/tag/v2.1.2-rc3

Verified on v2.6.5-rc6
Steps to Reproduce:

Bring up HA Rancher v2.6.5-rc6 and Install Rancher Backups 2.1.0
Create resources (user, cluster templates etc)
Create s3 backup
Create new cluster, Bring up HA Rancher v2.6.5-rc6 and Install Rancher Backups 2.1.0
Restore from s3 backup (No errors)

I am not able to update my production rancher to 2.5.6-rc.  Two reasons:  1. there is a company rule against using beta/rc products in production. 2. There is a known bug in upgrading the docker version of rancher.  I tried to do it some months ago using the workaround that was suggested, but rancher failed to come up afterwards.  So I am now in the process of building a brand new HA Rancher 2.6.4.  When its ready I will move my apps over there.

I had trouble connecting to S3 by following the docs - turns out it was an issue with the way the secret was created. After creating a generic secret using literals, it worked.


blade changes has no effect

After I've enabled htmlmin.blade I'm not able to see changes I make to blade files on browser.
I already tryied to disable opcache in php.ini, changed cache driver in .env and to delete cache (php artisan cache:clear) and views (php artisan view:clear).
I'm working on localhost, so it's not a problem related to different environment times.

@slillo Sorry to answer after so long, but how did you solve this problem?

@slillo Sorry to answer after so long, but how did you solve this problem?


Fix pointer problems in device memory builder

[ ] Added an entry to CHANGELOG_VULKANO.md or CHANGELOG_VK_SYS.md if knowledge of this change could be valuable to users
[ ] Updated documentation to reflect any user-facing changes - in this repository
[ ] Updated documentation to reflect any user-facing changes - PR to the guide that fixes existing documentation invalidated by this PR.
[x] Ran cargo fmt on the changes

Could this be a fix for #1545, at least my poor main.rs passes nicely without a segfault.

The dream of using vulkano crate release on our project is just a tiny bit closer

@hakolao Thank you for the fix! Since #[repr(C)] affecting public interface, would you mind adding an entry to changelog.

The dream of using vulkano crate release on our project is just a tiny bit closer

That's great news! Also, would be nice if you add a project to this list(just make a PR to readme.md): https://github.com/vulkano-rs/vulkano#projects-using-vulkano We would like to track the projects using Vulkano and encourage other users to use Vulkano through the showcasing of other people projects. Thanks in advance!

Added, thanks @Eliah-Lakhin. Our project is not open source yet. And is in such an early phase that it's better to add it later to that list (nothing to link to yet). But I'll keep that in mind!

@hakolao Pull Request merged. Thanks!

Thank you for fixing the problem @hakolao.
🚀🚀


Highlight selected day number by default on all months

I just noticed that pickadate highlights selected day number by default on all months.
Example: if I select the 5th day of the month, it will highlight the 5th day of all the months on the calendar.
Why this happens? Is that a feature or a bug? :)
Is there any way to disable this?
Thanks!

It is considered as a feature.
https://github.com/amsul/pickadate.js/issues/165, https://github.com/amsul/pickadate.js/issues/457.
The least intrusive way of removing that "feature" is to change its style.


Invalid Completion After Variable definition with map binding pattern

Description:
function testListBindingPattern() {
    Student student = {name: "", "science": 22};
    Student {name: sName, ...marks} = student;
    <cursor>
}

type Country record {
    string name;
    Capital capital;
};

type Capital record {|
    string name;
|};

type Student record {|
    string name;
    int...;
|};

Consider the above sample and Invalid completions are provided at the cursor point
Affected Versions:
1.0.0 at least

This issue has been fixed along with the latest revamp of the completion engine


Unable to stop --catalogs option from parsing filename

When I try to run something like:
v8r --catalogs catalog.json -- file.json
I get the help message, plus the error:
Not enough non-option arguments: got 0, need at least 1
I expected that it should be possible to put options before the filename positional argument, and indeed the following works:
v8r --catalogs catalog.json -v file.json
I suspect the error has something to do with the Yargs configuration. The array() docs state that it should be possible to indicate the end of an array of values with --, but that doesn't seem to be working. Maybe it's a bug upstream?
I also think the help message could be improved here. Most users aren't aware that -- is how most CLI frameworks have users escape array option parsing. I think that should either be mentioned, or the default changed so that you have to specify --catalogs catalog1.json --catalogs catalog2.json to use multiple catalogs, which feels like an uncommon case.

v8r file.json --catalogs catalog.json (with the positional arg first) works fine, so you shouldn't need to pass an extraneous arg like -v just to get --catalogs to work.
I would also expect v8r --catalogs catalog.json -- file.json should also work, but as you say this might be an issue with v8r or it could be a problem upstream with yargs. I'll have a look into it.

or the default changed so that you have to specify --catalogs catalog1.json --catalogs catalog2.json to use multiple catalogs

I've not checked it, but from memory I think v8r file.json --catalogs catalog1.json --catalogs catalog2.json is already valid - I think yargs allows both out of the box. I'm not sure there is a way to prevent yargs from allowing the --catalogs catalog1.json catalog2.json syntax too though.

v8r file.json --catalogs catalog.json (with the positional arg first) works fine, so you shouldn't need to pass an extraneous arg like -v just to get --catalogs to work.

The reason that works is because there's nothing else for the option to grab when you write the command that way. --catalogs eats everything including the positional arg up until another option is detected, hence the -v works.

I've not checked it, but from memory I think v8r file.json --catalogs catalog1.json --catalogs catalog2.json is already valid - I think yargs allows both out of the box. I'm not sure there is a way to prevent yargs from allowing the --catalogs catalog1.json catalog2.json syntax too though.

That's correct, it's valid.
If it can be required that way, my thought here is that might be a more clear UX than the current. Similar to how --schema works, so maybe there's a setting on string()? This would be a breaking change though, technically, if that's a factor in your decision.
Thanks for all your efforts. 👍


Events should be emitted after validation in the tx library

Packages versions
https://github.com/0xPolygonMiden/miden-base/commit/06ee35783fa08476cc467c5bf5aa4092e9727d41
Bug description
The existing emit clauses:

account_vault_remove_asset
account_vault_add_asset
incr_nonce

Are doing the emit prior to validation. This means it is possible to have side effects based on an event, prior to knowing if the event was correctly created.
This is a very low severity, because for it to be exploited, an event handler would have to be implemented with side effects, which is bad practice. However, it is a simple fix to delay the emit events to ensure they correspond to valid entries.
How can this be reproduced?

Relevant log output
No response

Closed by #660 and #708.


Code prediction fails in Windows for C++ even after using extensions

VSCode Version:1.3.1
OS Version:Windows 10 build 10586

Steps to Reproduce:

I have included a bits directory inside the Microsoft Visual Studio official include directory. This bits directory is just a copy from the minGW source.
The purpose of the directory is that it contains the useful stdc++.h header file

But after doing so even though the editor shows no problem in including the files but it is not showing the functions of vectors or other data structures. There is no suggestion for code completion.

(I have clang installed so there is no issue because of clang)
Also instead of using 'bits/stdc++.h' if I just use 'vector' the code prediction works fine.
Please look into it it's urgent.

@sgdgp please file against the C++ extension you are using.


Docker + Roots wrong composer inclusion path

Hi,
I'm trying to add this plugin in my bedrock/roots installation with no success.
The problem is that composer tries to load the main file but at the wrong place.
I'm using docker to get a development stack (PHP-APACHE-MYSQL) and the web folder in the docker is /var/www/html mapped with web of bedrock/roots
/var/www/vendor/composer/../../web/app/plugins/intervention/src/Intervention.php
should have been
/var/www/vendor/composer/../../--->html<----/app/plugins/intervention/src/Intervention.php
Have you any idea or advice to make this working?
Thanks for any help

@mrgesco I'll retest with Bedrock, but initial feeling is that it may have something do with the setup.
The path is loaded using define('INTERVENTION_DIR', dirname(__FILE__));  which runs from intervention.php within the plugin folder and should return the correct path for loading files from there. You could try echo constant INTERVENTION_DIR and see what it returns?

Thank you @darrenjacoby for your quick reply!
This is the value of INTERVENTION_DIR and the plugin is working
/var/www/html/app/plugins/intervention
But I have always the warning about composer
include(/var/www/vendor/composer/../../web/app/plugins/intervention/src/Intervention.php): failed to open stream: No such file or directory /var/www/vendor/composer/ClassLoader.php:444

@mrgesco try give rc.4 a go, tagged today and let me know if that helps.

No luck same behavior.
Getting the plugin manually works fine without the warning.
I'm pretty sure the problem is with docker but can't figure why...
Plugins required from wpackagist works fine.

Unfortunatly not. I have installed the plugin manually and not with composer

Thanks for reporting back @mrgesco. I haven't been able to replicate, so I'm going to close for now. If you discover any more information regarding the issue please do let me know.


update docs to describe supported cadence doc pragma

Update docs to describe currently supported cadence doc pragma

Thanks for the feedback, updated PR


update mapbox-earcut follow ManimCommunity/manim#2931

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

See my comment in #3.
Also, ^1.0.0 from the Manim poetry specification translates to >=1.0.0,<2.0.0


randomly crashes

it says "Blocked by ddos protection. Solve the captcha to continue.", then closes after a couple seconds, I usually get this error when the game lasts a bit long (around x10 and more)
also i'm sorry for bothering u that much, and I cant copy the error file since its too long for github idk why

This isn't caused by the length of the games as when the program checks to see if it's been blocked it hasn't even started recording the games and their crashpoints yet.

This isn't caused by the length of the games as when the program checks to see if it's been blocked it hasn't even started recording the games and their crashpoints yet.

then whats the issue

I'm not sure

Can you send errors.txt


Drop the rustup iron requirement

It seems to work just fine in Iron without it:
FROM ros:iron
SHELL [ "/bin/bash", "-c" ]
RUN apt-get update -qq && apt-get install -qqy python3-colcon-core ros-iron-demo-nodes-cpp
RUN mkdir -p /ws/src
WORKDIR /ws
RUN git clone https://github.com/ros2/rmw_zenoh /ws/src/rmw_zenoh
RUN rosdep update && rosdep install --from-paths src --ignore-src -y
RUN source /opt/ros/iron/setup.bash && colcon build
CMD [ "bash", "-c", "source /ws/install/setup.bash && \
  export RMW_IMPLEMENTATION=rmw_zenoh_cpp && \
  ros2 run rmw_zenoh_cpp rmw_zenohd & \
  ros2 run demo_nodes_cpp talker & \
  ros2 run demo_nodes_cpp listener" ]

That is...extremely surprising.  The Zenoh folks assure us that we need at least Rust 1.75.0 to build Zenoh currently, and Ubuntu 22.04 only has 1.62.  But I haven't tried this myself yet.

Well, if there is a good reason to keep it, I don't mind.
But this is easier testing :slightly_smiling_face:

Ah, found it:
$ docker run just-build-docker rustc --version
rustc 1.75.0 (82e1608df 2023-12-21) (built from a source tarball)

Yeah I'm also surprised the build passes without pinning rustc to 1.75. See https://github.com/ros2/rmw_zenoh/issues/138
@Timple can you push a change to this PR to update this line in CI to install @stable instead of @1.75.0?

Ah, found it:
$ docker run just-build-docker rustc --version
rustc 1.75.0 (82e1608df 2023-12-21) (built from a source tarball)

For systems that do not have rustc installed, running rosdep will install the apt version of rustc which @clalancette pointed out which should cause build to fail.

Sure, I can do. But why is there actually a github action for installing rust? The ROS tooling can take care of this right? Just as with the commands in the README.md

The ROS tooling can take care of this right? Just as with the commands in the README.md

The ROS tooling, ie rosdep will install the apt version of rustc. On Jammy this is 1.62 which will cause a build failure. On Noble, it will directly install 1.75 which works readily. Hope this clarifies.

Sorry for accidentally closing the PR.
Do you mind deleting the step in the CI that installs rustc? We can let rosdep install rustc for Iron on Jammy

I'm have c++ code style violations?


PXP-672: [CLI] Include .yaml file types in application init workflow list

Summary
This PR addresses an issue in the application initialization process where the CLI was not displaying all available GitHub Actions workflows during step #2. The root cause of the problem was identified as the exclusion of .yaml files when checking for workflows in the .github/workflows directory.

Ticket: PXP-672
What's Changed

Fixed

[x] Include .yaml file in workflow filter

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: mfauzaan
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: mfauzaan, onimsha
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Store person

Codecov Report

Merging #13 into main will increase coverage by 21.94%.
The diff coverage is 64.28%.

@@             Coverage Diff             @@
##             main      #13       +/-   ##
===========================================
+ Coverage   57.80%   79.74%   +21.94%
===========================================
  Files          19       20        +1
  Lines         282      316       +34
===========================================
+ Hits          163      252       +89
+ Misses        119       64       -55

Impacted Files
Coverage Δ

src/model/store.h
100.00% <ø> (ø)

src/exception/product_exception.cpp
50.00% <37.50%> (-50.00%)
:arrow_down:

src/model/store.cpp
78.88% <67.74%> (+78.88%)
:arrow_up:

src/exception/person_exception.cpp
33.33% <100.00%> (+33.33%)
:arrow_up:

src/exception/product_exception.h
100.00% <100.00%> (ø)

src/model/order.cpp
94.73% <100.00%> (+3.50%)
:arrow_up:

src/model/person.cpp
72.72% <0.00%> (+3.63%)
:arrow_up:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7d2c3b...f996346. Read the comment docs.


boto3: 1.1.4 (initial)

what happened?

@samuelrivas i've merged your changes in nixpkgs

Oh, I see.
I was confused because github claimed the pull request was closed without merging

Thanks!