Create QUIC library that can be exposed to JS and uses the Node dgram module or libuv UDP socket

Specification
We need QUIC in order to simplify our networking stack in PK.
QUIC is a superior UDP layer that can make use of any UDP socket, and create multiplexed reliable streams. It is also capable of hole punching either just by attempting to send ping frames on the stream, or through the unreliable datagrams.
Our goals is to make use of a QUIC library, something that is compilable to desktop and mobile operating systems, expose its functionality to JS, but have the JS runtime manage the actual sockets.
On NodeJS, it can already manage the underlying UDP sockets, and by relying on NodeJS, it will also ensure that these sockets will mix well with the concurrency/parallelism used by the rest of the NodeJS system due to libuv and thus avoid creating a second IO system running in parallel.
On Mobile runtimes, they may not have a dgram module readily available. In such cases, having an IO runtime to supply the UDP sockets may be required. But it is likely there are already existing libraries that provide this like https://github.com/tradle/react-native-udp.
The underlying QUIC library there is expected to be agnostic to the socket runtime. It will give you data that you need to the UDP socket, and it will take data that comes of the UDP socket.
However it does need 2 major duties:

The multiplexing and managing of streams.
The encryption/decryption TLS side

Again if we want to stay cross platform, we would not want to bind into Node.js's openssl crypto. It would require instead that the library can take a callback of crypto routines to use. However I've found that this is generally not the case with most existing QUIC libraries. But let's see how we go with this.
Additional context

https://github.com/MatrixAI/Polykey/issues/234 - This is the main issue talking about how to replace utp-native
https://github.com/napi-rs/napi-rs/tree/main/examples/napi - examples of how to use napi-rs
https://nodejs.org/api/dgram.html - dgram module in NodeJS where the QUIC data will be plumbed to
https://blogs.keysight.com/blogs/tech/nwvs.entry.html/2021/07/17/looking_into_quicpa-pUtF.html - some notes about how QUIC data can be viewed on wireshark
https://www.sitepoint.com/rust-global-variables/
https://deepu.tech/memory-management-in-rust/
https://github.com/cloudflare/quiche/issues/326#issuecomment-577281881 - Disabling TLS verification and doing it custom
Required options https://github.com/cloudflare/quiche/issues/1250
https://blog.cloudflare.com/head-start-with-quic/
https://github.com/romain-jacotin/quic/blob/master/doc/QUIC_crypto_protocol.md

QUIC and NAPI-RS

https://github.com/napi-rs/napi-rs/discussions/1373 - callback usage
https://github.com/napi-rs/napi-rs/issues/1374 - Callback usage in factory methods
https://github.com/cloudflare/quiche/issues/326#issuecomment-577281881 - custom TLS verification in quiche
https://github.com/napi-rs/napi-rs/discussions/1409 - how to deal with external object types
https://github.com/napi-rs/napi-rs/discussions/1379 - napi-rs can't export submodules in rust for now

Tasks

[x] Experiment with Neon or Napi-rs
[x] Experiment with quiche by cloudflare
[x] Create bridge code plumbing UDP sockets and QUIC functions
[x] Create self signed TLS certificate during development - https://github.com/MatrixAI/js-quic/issues/1#issuecomment-1356229150
[ ] Extract out the TLS configuration so that it can be set via in-memory PEM variable and key variable. - 2 day

This should be doable with https://docs.rs/quiche/latest/quiche/struct.Config.html#method.with_boring_ssl_ctx
This needs to construct a SSL ctx with boring ssl, and pass in PEM string and key string (or Uint8Array) to be processed by the boringssl library.

[ ] Decide whether application protocols are necessary here, or abstract the quiche Config so the user can decide this (especially since this is not a HTTP3 library). - 0.5 day
[x] Fix the timeout events and ensure that when a timeout occurs, that the connection gets cleaned up, and we are not inadvertently clearing the timeout due to null. Right now when a quiche client connects to the server, even after closing, the server side is keeping the connection alive. - 1 day
[x] We need lifecycle events for QUICConnection and QUICStream and QUICServer and QUICSocket. This will allow users to hook into the destruction of the object, and perhaps remove their event listeners. These events must be post-facto events. - 0.5 day
[ ] Test the QUICStream and change to BYOB style, so that way there can be a byte buffer for it. Testing code should be able generator functions similar to our RPC handlers. - 1 day
[ ] Complete the QUICClient with the shared socket QUICSocket. - 3 day
[ ] Test the multiplexing/demultipexing of the UDP socket with multiple QUICClient and a single QUICServer. - 1 day
[ ] Test the error handling of the QUIC stream, and life cycle and destruction routines. - 1 day
[ ] Benchmark this system, by sending lots of data through. - 1 day
[ ] Propagate the rinfo from the UDP datagram into the conn.recv() so that the streams (either during construction or otherwise) can have its rinfo updated. Perhaps we can just "set" the rinfo properties of the connection every time we do a conn.recv(). Or... we just mutate the conn parameters every time we receive a UDP packet.
[ ] Ensure that when a user asks stream.connection they can acquire the remote information and also all the remote peer's certificate chain.
[ ] Integrate dgram sending and recving for hole punching logic.

Ok I have managed to get the ability for quiche to send a packet. Actually I haven't sent the packet to the UDP socket, but I can see how quiche attempts to create an initial packet for me plumb into the UDP socket.
This involved several challenges.
The first is that self.0.send takes in a reference to a mutable buffer. There's a preference to avoid copying in Rust lang similar to C and C++.
This buffer can actually be taken from the outside back in JS land. This might be a better idea, as it minimises the amount of copy pasting going on.
Alternatively we can create the array in Rust, then return as a Buffer. However the send call returns the number of bytes written.
Therefore the buffer is pre-allocated to be MAX_DATAGRAM_SIZE and it is subsequently written to.
This means later the array needs to be sliced according the to the write size and then returned as a buffer.
This means we generally 2 styles of calling the Rust functions. We can call them the way rust expected to be called, by passing in a buffer as a parameter and returning the write length as well. It turns out that socket.send actually easily supports taking a buffer, offset and the length to be used: https://nodejs.org/api/dgram.html#socketsendmsg-offset-length-port-address-callback. In terms of speed, it would make use to re-use the same buffer over and over.
The other way is more idiomatic JS, which would mean that the function should return the buffer to be sent that is properly sliced up.
I think we should preserve the rust style, and then do any JS-specific idiomatic abstractions on the JS side.
At the same time, the send can also return an error which indicates that it is Done. If it is done, then there are no more packets to send. Using a JS style call, we can just return undefined in that situation. However I'm not entirely sure if this is actually possible using napi-rs. It seems you have return a Rust value, and that it automatically gets converted to JS via the napi-rs macros.
With rust style calls, the done is returned as an exception which is kind of strange, but also doable. But this is very much different from how JS should be used, and it means we also have to catch non-done exceptions.
Another thing to realise is that when you have situations where you have a buffer of some fixed size, and a then write length indicating how much of that buffer was used. This will always eventually lead to a slicing operation.
When doing any slicing, the function's stack will end up with a pointer (reference) to the slice, it won't do any copying.
Rust does not have dynamically allocated arrays that sit on the stack, so you have to use vectors instead. Vectors are heap-allocated arrays that can be dynamically changed. String is also heap allocated string relative to &str. At the same time, Rust manages the memory of vectors and knows when to deallocate them automatically.
So if we wanted to slice out from the array and also return it as a Buffer to JS, then it has to be turned into a vector first, and that vector will be dynamically allocated simply through the !vec macro.
Finally another problem is sending structs back to JS as objects. This requires hte usage of #[napi(object)] macro. However this only works on structs have all the fields public.
The send info contains information that is necessary to know how to send the packet data to the UDP socket.
It however ends up containing std::net::SocketAddr and std::time::Instant both which contain private fields. So these types cannot be easily automatically converted to JS object. It could probably be done if I implement some sort of interface for the SocketAddr. Like https://docs.rs/napi/latest/napi/trait.NapiRaw.html and https://docs.rs/napi/latest/napi/trait.NapiValue.html.
Alternatively I created my own Host struct that just contains a String and u16. This results in a conversion from the SocketAddr to Host and wrapping std::time::Instant into External::new() because it's supposed to be opaquely compared at the rust-level.
This led to a few other issues:

You cannot declare nested structs in Rust, you have to give each one a separate name.
There does not appear to be a way to send Rust tuples to JS, napi-rs does not understand how to deal with tuples. I would have thought that they just become arrays on the JS side... but no.
Rust enums cannot be represented either, they become objects on the JS side. This is because Rust enums are actually ADTs. They have a tag, and then the subsequent value/values.

So with this all in mind, I ended up creating another custom struct just so that so I can get the value out of conn.send.
This reveals a structure that always contains the initial packet. This is probably the packet that leads to connection negotiation. If you call it again right now, you get a JS error with Done as the message.
Now we can just plumb this into the dgram module to send.

When should the send method be called? Apparently it should be triggered under an under a number of different conditions:
https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.send
And it should be looped in fact, until it is done.
The conditions are:

The application should call send() multiple times until Done is returned, indicating that there are no more packets to send. It is recommended that send() be called in the following cases:

When the application receives QUIC packets from the peer (that is, any time recv() is also called).
When the connection timer expires (that is, any time on_timeout() is also called).
When the application sends data to the peer (for example, any time stream_send() or stream_shutdown() are called).
When the application receives data from the peer (for example any time stream_recv() is called).

Once is_draining() returns true, it is no longer necessary to call send() and all calls will return Done.

So I can imagine attaching callback to things like on_timeout to trigger "sending", and any time we receive packets, and send data to the peer or receive data to the peer.
We could call this "socketFlushing", it will loop until done. The socket flushing would be triggered every time any of the above things happen... which in turn will flush all the packets to the socket.... will it wait for the socket to have actually sent it by waiting on the callback? I'm not sure if that's necessary...
I guess it depends. On the QUIC stream level, it should be considered independent, you don't wait for the socket to be flushed before returning to the user. But one could await a promise will resolve when they really want to have the entire socket flushed.

Ok I can see that there is no support for tuples at all here: https://github.com/napi-rs/napi-rs/tree/main/examples/napi. Also not for tuple-structs which I use in a newtype style pattern.
The only thing closest is to create an array like so:
#[napi]
fn to_js_obj(env: Env) -> napi::Result<JsObject> {
  let mut arr = env.create_array(0)?;
  arr.insert("a string")?;
  arr.insert(42)?;
  arr.coerce_to_object()
}

On the generated TS it looks like a object.
I reckon it would be dynamically deconstructable as an array. Pretty useful here, to avoid having to define one-off structs.

I've settled on just returning an Array. It beats having to redefine the output type all the time.

The streams in quiche doesn't have its own data type. Instead they are identified by a stream ID. The connection struct provides the methods https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.stream_recv and you have to keep track of the stream IDs somehow.
So we would expect that upon creating an RPC call, it would call into the network to create a new stream for a connection.
It seems creating a new stream is basically performing this call: https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.stream_send
You select a stream ID like 0, and you just keep incrementing that number to ensure you have unique stream IDs. The client-side just increments the number, the server side just receives the streams.
Note that once we have a stream, the stream is fully duplex (not half-duplex), in-order and reliable.

The quiche::connect and quiche::accept should be both factory method of creating the Connection class.
However in the case of the server side, you can't create a new connection until you've taken a specific packet from the UDP socket and parsed it first. You have to check the internal packet information to decide whether to accept a new connection, or to simply plumb it in as if you received the data for a given stream.
So the order of operations:

Receive packet from UDP socket.
Check if the packet is a new connection, if so, create the new connection, otherwise retrieve an existing connection.
Call conn.recv() to plumb the packet data into some relevant stream.

This does mean we maintain clients hashmap of ConectionId => Connection.

I created a function to randomly generate the connection ID.
It's a random 20 bytes.
Instead of using Rust's ring library to get the random data. I'm just calling into a passed in JS function to acquire the random data. This unifies the random data usage directly from the Node runtime (and reduces how much rust code we are using here).
/// Creates random connection ID
///
/// Relies on the JS runtime to provide the randomness system
#[napi]
pub fn create_connection_id<T: Fn(Buffer) -> Result<()>>(
  get_random_values: T
) -> Result<External<quiche::ConnectionId<'static>>> {
  let scid = [0; quiche::MAX_CONN_ID_LEN].to_vec();
  let scid = Buffer::from(scid);
  get_random_values(scid.clone()).or_else(
    |err| Err(Error::from_reason(err.to_string()))
  )?;
  let scid = quiche::ConnectionId::from_vec(scid.to_vec());
  eprintln!("New connection with scid {:?}", scid);
  return Ok(External::new(scid));
}

However I realised that since it's just a 20 byte buffer. Instead of passing an External<quiche::ConnectionId<'static>> around.
We can just take a Buffer directly that we expect to be of length MAX_CONN_ID_LEN.
Meaning the connection ID is just made in JS side, and then just passed in.
On the rust side, they can use quiche::ConnectionId::from_ref(&scid) which means there's no need to to keep any memory on the Rust side. It's all just on the JS side.
This basically means we generally try to keep as much data on the JS side, and manage minimal amounts of memory on the Rust side.

The <'static> is used because it's sort of a dummy lifetime, because the ConnectionId is actually an enum of either an owned vector or a reference to some memory that exists elsewhere. The owned vector doesn't have a lifetime annotation.

A better alternative to just Buffer would be our own ConnectionId newtype.
But I can't seem to create good constructor for it that would take a callback. Problem in https://github.com/napi-rs/napi-rs/issues/1374.

On the connecting side, there's an option for the server name.
If QUIC uses the standard TLS verification, we could imagine that this will not work for us. I'm not entirely sure.
We need to provide our custom TLS verification logic. I haven't found how to do this yet with QUIC but it should be possible.

Found it: https://github.com/cloudflare/quiche/issues/326#issuecomment-577281881
It is in fact possible.

The stream methods are mostly done on the rust side.
I've found that the largest type I can make use of is i64 not u32. This gets closer to what JS numbers are capable of, although not all of i64 can be represented in JS. but those are the edges, is unlikely to reach those.
So I've converted it to those. I'm still using as instead of the "safer" variants of try_into or try_from because it's just more convenient to coerce the numbers.
I've also found how to write the FromNapiValue trait. Basically it ends up calling NAPI-related C++ functions that convert the JS value (known as NAPI value) into a C-like value, and that value can be stored in Rust.
So basically whenever napi-rs can't automatically derive the marshalling code, we have to implement them like FromNapivalue and ToNapiValue.
I think the napi macros also expose ways of writing out exactly what the generated types should be. I'm still not entirely sure we want to do use the generated types from napi-rs because we can do it like in js-db where we just write out our TypeScript types manually.

On the JS side, I'm hoping to manage most of the state there. That means things like a map of connection IDs to connection objects, and maps of stream IDs to stream data.
That way the Rust part stays as pure as possible. We would only have 1 UDP socket for all of this. So that would simplify the the network management.

It turns out in order to send dgrams, you need to have a connection object created. However if we aren't calling conn.send, we aren't sending the initial packet for handshaking purposes.
This means, we can use conn.dgram_send for the hole punching packets first, before performing the conn.send which would initiate the handshaking process.
One thing I'm not sure about is how the TLS interacts with the dgram send. For hole punching packets we don't care encrypting the data that much, but if we can encrypt the hole punching packets that would nice. However this would at the very least require some handshake for the TLS to work... and if we are hole punching then we cannot have already done the TLS handshake.

I've started modularising the rust code for consumption on the TS side.
Some things I've found out.
For enums, there are 2 ways to expose third party enums to the TS side.
The first solution is to something like this:
use napi::bindgen_prelude::{
  FromNapiValue,
  sys
};

#[napi]
pub struct CongestionControlAlgorithm(quiche::CongestionControlAlgorithm);

impl FromNapiValue for CongestionControlAlgorithm {
  unsafe fn from_napi_value(env: sys::napi_env, value: sys::napi_value) -> Result<Self> {
    let value = i64::from_napi_value(env, value)?;
    match value {
      0 => Ok(CongestionControlAlgorithm(quiche::CongestionControlAlgorithm::Reno)),
      1 => Ok(CongestionControlAlgorithm(quiche::CongestionControlAlgorithm::CUBIC)),
      2 => Ok(CongestionControlAlgorithm(quiche::CongestionControlAlgorithm::BBR)),
      _ => Err(Error::new(
        Status::InvalidArg,
        "Invalid congestion control algorithm value".to_string(),
      )),
    }
  }
}

The basic idea is the new type pattern, but then we have to implement the FromNapiValue trait, and marshal numbers from JS side to the equivalent Rust values.
The problem here is that the generated code results in class Shutdown {} which is just a object type.
On the TS side, there's no indication of how you're supposed to construct this object.
We would most likely have to create a constructor or factory method implementation that enables the ability to construct CongestionControlAlgorithm like CongestionControlAlgorithm.reno() or CongestionControlAlgorithm.bbr().
However this is kind of verbose for what we want.
Another alternative is to use the https://napi.rs/docs/concepts/types-overwrite on the method that uses CongestionControlAlgorithm.
#[napi(ts_arg_type = "0 | 1 | 2")] algo: CongestionControlAlgorithm,

However I found another way.
Redefine the macro:
/// Equivalent to quiche::CongestionControlAlgorithm
#[napi]
pub enum CongestionControlAlgorithm {
  Reno = 0,
  CUBIC = 1,
  BBR = 2,
}

Then this is converted to TS like:
/** Equivalent to quiche::CongestionControlAlgorithm */
export const enum CongestionControlAlgorithm {
  Reno = 0,
  CUBIC = 1,
  BBR = 2
}

Then we use it, we use the match to map all the value from our enum to the third party enum.
  #[napi]
  pub fn set_cc_algorithm(&mut self, algo: CongestionControlAlgorithm) -> () {
    return self.0.set_cc_algorithm(match algo {
      CongestionControlAlgorithm::Reno => quiche::CongestionControlAlgorithm::Reno,
      CongestionControlAlgorithm::CUBIC => quiche::CongestionControlAlgorithm::CUBIC,
      CongestionControlAlgorithm::BBR => quiche::CongestionControlAlgorithm::BBR,
    });
  }

This application of the match can be a bit verbose... so if we want to do multiple times, it can be abstracted to a internal function next to the enum.
Maybe we can provide a Into trait implementation? https://doc.rust-lang.org/rust-by-example/conversion/from_into.html That might help automate this process.

By doing this:
/// Equivalent to quiche::CongestionControlAlgorithm
#[napi]
pub enum CongestionControlAlgorithm {
  Reno = 0,
  CUBIC = 1,
  BBR = 2,
}

impl From<CongestionControlAlgorithm> for quiche::CongestionControlAlgorithm {
  fn from(algo: CongestionControlAlgorithm) -> Self {
    match algo {
      CongestionControlAlgorithm::Reno => quiche::CongestionControlAlgorithm::Reno,
      CongestionControlAlgorithm::CUBIC => quiche::CongestionControlAlgorithm::CUBIC,
      CongestionControlAlgorithm::BBR => quiche::CongestionControlAlgorithm::BBR,
    }
  }
}

impl From<quiche::CongestionControlAlgorithm> for CongestionControlAlgorithm {
  fn from(item: quiche::CongestionControlAlgorithm) -> Self {
    match item {
      quiche::CongestionControlAlgorithm::Reno => CongestionControlAlgorithm::Reno,
      quiche::CongestionControlAlgorithm::CUBIC => CongestionControlAlgorithm::CUBIC,
      quiche::CongestionControlAlgorithm::BBR => CongestionControlAlgorithm::BBR,
    }
  }
}

We get bidirectional transformation of the enums and we can use algo.into() generically.

The pacing at gives us an Instant. Which is not consistent across platforms. On non-macos, non-ios and non-windows, this can be converted into a timespec.
#[cfg(not(any(target_os = "macos", target_os = "ios", target_os = "windows")))]
fn std_time_to_c(time: &std::time::Instant, out: &mut timespec) {
    unsafe {
        ptr::copy_nonoverlapping(time as *const _ as *const timespec, out, 1)
    }
}

#[cfg(any(target_os = "macos", target_os = "ios", target_os = "windows"))]
fn std_time_to_c(_time: &std::time::Instant, out: &mut timespec) {
    // TODO: implement Instant conversion for systems that don't use timespec.
    out.tv_sec = 0;
    out.tv_nsec = 0;
}

A time spec is basically a timestamp with seconds component and a nano seconds component.
There isn't any portable rust functions right now that convert a std::time::Instant to something like milliseconds where I could just turn it into a JS timestamp that being of Date.
On the other hand it is intended to be an opaque type, so that's why I have it as an External, however this doesn't help us figure out how to pace the sending of the packet data.
One idea is to just do what the ffi.rs does and convert it to a timespec  and then extract out the underlying second and nanosecond data and convert to milliseconds or Date object on JS.
However there's no information how to do this for the macos, ios and windows case. So it's kind of useless atm.

I think pacing isn't necessary since it isn't used in the examples, so we can leave it out for now. But the generated types won't be usable as it is referring to Instant.
I'm not entirely sure how generated types are supposed to work with External. Unless you're supposed to just create a dummy class type for that type that it is returning.
I'm likely end up writing our own TS file instead of relying on the generated code, but the generated code is useful for debugging.

Ok so I'm starting to experiment with the rust library by calling it from the JS.
One thing I'm unclear about is that this library being @matrixai/quic will essentially be about exposing the quiche library directly, or is intending to wrap it into its own set of abstractions.
In this library do we also create the socket and thus expose a socket runtime, such as creating a class Server and thus class Client.
It seems so, because quiche would expose quiche::accept to create a server connection, and quiche::connect to create a client connection.
We know that in both cases, they can use the same socket address, thus using 1 socket address to act as both client and server.
If we do this, we might just create a class Connection. However we already have this as provided by the native library.
So if we just expose the QUICHE design, then the actual socket management will be in PK.

Ok so then only in the tests would we be making use of the sockets.

Ok as I have been creating the QUIC server, I've found that there are many interactions with the socket during the connection establishment. It makes sense that this library would handle this complexity too. Things like address validation, stateless retries and other things all happen during connection establishment.

Ok I'm at the point where it's time to deal with streams.
So therefore this library does have to do:

Handle the interaction between the connection establishment and socket.
Convert the QUIC streams into actual Web/DOM streams

So there's quite a bit of work still necessary on the JS side.

The ALPNs may not be relevant unless we are working with HTTP3. It also seems we could just refer to our own special strings, since our application level protocol isn't HTTP3.

We could do something like polykey1 as the ALPN. Where 1 would be the application layer protocol which would be RPC oriented...? I'm not sure if this is even needed.

The above structure also implies that upon timeout, and readable socket events, that we also start to send packets out. The // Now we process things to send out. That works because it is assumed that handling writable and readable streams ends up calling streamSend which then puts data in quiche's buffer that needs to be sent out.
However in JS again, this doesn't seem idiomatic. Instead we should have an event emitter that emits events upon any streamSend occurring, that indicates there's data to be sent out. After all any data we buffer into quiche should eventually be flushed.
Upon doing so, we would want to then perform the conn.send() which plumbs the data out and then socketTo().
Note how it iterates over all connections in an attempt to find if there's any data to send. It would only make sense to send data from connections that have stream sends.
Unless... it's possible that that connections have data to be sent even if no stream has sent anything. For example things like keepalive packets. I'm not sure. But in that case, shouldn't that be done on the basis of timeouts?
All of this requires some prototyping.

I started working on the web stream abstraction.
In the web streams there's no default DuplexStream to construct. It has to be done by yourself.
But this ends up being easy to do:
class MyStream<R = any, W = any> implements ReadableWritablePair<R, W> {
  public readable: ReadableStream<R>;
  public writable: WritableStream<W>;
  public constructor() {
    this.readable = new ReadableStream();
    this.writable = new WritableStream();
  }
}

In our case, we have created a QUICStream that just uses Uint8Array as the R and W.
Now for the readable side it has to be constructed while "wrapping" a push-based data source.
Quiche's QUIC streams are in fact push-based... this is because we start at the UDP socket, where we attach an event handler for the message event. This triggers quiche processing, which eventually leads us to iterating over connection.readable() stream IDs which tells us all the quic streams IDs that have data that is readable. At this point we would have to do connection.streamRecv and plumb this data into the ReadableStream<Uint8Array> above.
As a quick prototype, I setup a readable stream like this:
this.readable = new ReadableStream({
  type: 'bytes',
  start(controller) {
    events.addEventListener('data', (event) => {
      controller.enqueue(event.detail);
      if (controller.desiredSize <= 0) {
        events.pause();
      }
    });
    events.addEventListener('end', () => {
      controller.close();
    }, { once: true });
    events.addEventListener('error', (event) => {
      controller.error(event.detail);
    }, { once: true });
  },
  pull() {
    events.resume();
  },
  cancel() {
    events.destroy();
  }
});

Note that the default queuing strategy is CountQueuingStrategy with a high water mark of 1. However I was not able to use node's new CountQueuingStrategy, I think there's a bug in NodeJS. The reason we stick with the default anyway is because the flow control would already be configured on quiche, and there's no need to do further complicated byte length flow control on the web streams, instead it will be 1 chunk at a time being passed to the reader of the readable stream.
Now what is events. It is custom EventTarget that will emit events. I like to think of it as a "derived event emitter". Remember that eventemitters are not as flexible as subjects/observables. If we think of the socket as an "event emitter"/push-based data source, this implies that the "push-flow" is preserved through quiche processing which then translates to event emitters for every stream that is readable.
Socket Events -> Stream 1 Events -> Web Stream 1
                 Stream 2 Events -> Web Stream 2
                 Stream 3 Events -> Web Stream 3
                 ...             -> ...

So 1 single socket event, could translate to multiple stream events, and each stream event is pushed to the each web stream.
I considered using 1 event emitter for all stream events, and indexing each event by the stream ID, but it seemed more idiomatic to just create separate EventTarget object for each stream, and then the event names become easier to work with.
Now the UDP socket is an EventEmitter in NodeJS. But quiche's streams are not actually event emitters. So we have to create an events object for each stream to bridge this data flow.
If everything was an observable, this could probably be done in a higher order way... transforming socket events into an observable..., then passing it into a transformer/demuxer pipeline which then goes to multiple web streams. But I'm going to avoid using any rxjs magic here and just use plain old JS stuff to be portable.

For the writable side, it's a bit simpler. We only need to provide the write handler which will plumb the data back into a QUIC stream with connection.streamSend.
this.writable = new WritableStream({
  write(chunk) {
    // Replace this with `streamSend`
    console.log(chunk);
  }
});

Finally the entire QUICStream object can be constructed in one of 2 ways:

Upon receiving a connection - and the creation of new streams
When starting a new stream to specific connection

Because 1. is a "push based dataflow", it would make sense to provide a higher-order handler that is called upon the creation of a new stream.
// Receiving a new stream
quic.on('stream', handleStream);

// Start a new stream to some connection
const stream = quic.streamStart();

Interestingly push-based dataflow ultimately look like event handlers.

Thinking about the event architecture more https://github.com/MatrixAI/js-quic/issues/1#issuecomment-1343746712, there are more events than just the UDP socket event. We will need to attach an event handler for all of those possible events together. To do that we should be able to "create" a QUICServer instance that internally attaches event handlers to all those possible events.

Important to remember that event emitters and event targets are fully synchronous. They are not async pub/sub. Which can be different from observables.

Discovered a bug with napi-rs: https://github.com/napi-rs/napi-rs/issues/1390. Need to make sure where we have factory methods, we don't use [#napi(constructor)] macro on top of the struct itself.

I've made a start on QUICServer class. It currently extends the EventTarget so I can make it similar to other nodejs objects that emit events, but I didn't want to use EventEmitter so it can be portable to other runtimes later.
This class will have a number of protected handlers, each which will be attached to all the possible events that may occur in the process of running QUIC.
Furthermore, the binding to a host is worked out, by using ip-num to determine whether the host is IPv4 or IPv6 and defaulting to IPv4 if it is a host name. The binding may still fail, so we need to race towards a possible failure.
Then we get the resolved host and port as properties.
That's when we can then start handling messages. In a way that starts all the other possible events because we have to have at least 1 connection before we can have any of the other connection events.
I imagine the handleMessage will translate to further events like stream events. If it is a new stream, a new QUICStream object will be created and then emitted as an event on the QUICServer. It is up to the user to register a handler for quicServer.addEventListener('stream', (stream) => { ... }).
I wonder what we would do if we have no handlers for the streams? We would still need to register it as state, and eventually garbage collect them? I'm not sure. Perhaps without a stream handler, all new streams would be immediately closed. Would need to check what exactly is supposed to happen with respect to stream lifecycles.

I asked ChatGPT what happens if you create an HTTP server with http.createServer without a request listener...

If you create an HTTP server using http.createServer() without passing a request listener function, the server will not have any way to handle incoming HTTP requests. As a result, any HTTP requests sent to the server will simply be ignored and will hang until they time out.

Therefore we can have a similar behaviour, if we don't have anything that is handling the streams, then they get created... but they should eventually timeout.
However due to https://github.com/cloudflare/quiche/issues/908 there is no such thing as "stream timeouts" in the QUIC spec. And the config.set_max_idle_timeout is only for the entire connection: https://datatracker.ietf.org/doc/html/draft-ietf-quic-transport-34#section-10.1
Therefore timing out streams is an application-level concern, it can be considered that one could have an infinite stream....
But if we leave infinite streams around along with the possibility of not having a on-stream handler, then we have a possible resource-leak here.
So either we:

Setup per-stream timeouts just like how the HTTP server has timeouts for requests/responses
Ensure that when we construct the QUIC server, that a stream handler is provided.

Note that there is still limits of how many streams can exist for any given connection and if the connection timesout, it ends up closing all the streams anyway.
In fact... technically the server side for HTTP servers doesn't actually timeout the request anyway. It waits for the client to time it out. So yes there can be dangling resources in NodeJS too. Which means we could just accumulate stream objects internally... although without any handler, then I'm guessing any created QUICStream objects would just be GCed. So maybe this is not a problem.

Furthermore the HTTP server has this https://nodejs.org/api/http.html#servertimeout which sets the timeout for all requests. The default is 0 which means there's actually no timeouts for requests. So we can replicate this model.

This means HTTP server's "request/response" transaction is equivalent to QUIC server's "stream".

I'm using the cargo run --bin quiche-client -- 'http://<IP_ADDRESS>:55555' for testing the QUIC server. I'm not sure how to run quiche's example code, but the app code tries to perform an HTTP3 request. On our quic server perspective, any data started is just another QUIC stream. Would you be able to create an HTTP3 server? Sure... you could, but that's not what we care about. That would have to done on top of the streams.

I'm actually running the QUICServer now and I've gotten to the stateless retry. And I can see the protocol flow now:
┌────────┐                                ┌────────┐
│ Client │                                │ Server │
└───┬────┘                                └────┬───┘
    │          ┌───────────────────┐           │
  1.├─────────►│Initial            ├──────────►│
    │          │version:<PHONE_NUMBER>│           │
    │          │token: []          │           │
    │          │dcid: A            │           │
    │          │scid: B            │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
  2.│◄─────────┤Version Negotiate  │◄──────────┤
    │          │version: 1         │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Initial            │           │
    │          │version: 1         │           │
  3.├─────────►│token: []          ├──────────►│
    │          │dcid: A            │           │
    │          │scid: B            │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Retry              │           │
  4.│◄─────────┤token: [A]         │◄──────────┤
    │          │scid: C            │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Initial            │           │
    │          │version: 1         │           │
  5.├─────────►│token: [A]         ├──────────►│
    │          │dcid: C            │           │
    │          │scid: B            │           │
    │          └───────────────────┘           │
    │                                          │
    │                                          │

Note that after the stateless retry, it passes a signed token containing the original DCID, but it also passes the key-derived connection ID from the original DCID as the new_scid.
The client is supposed to use the new SCID and pass back that token.
One thing that is confusing to me is that the subsequent initial packet has the new SCID appear in header.dcid and not header.scid so I asked this here: https://github.com/cloudflare/quiche/issues/1370
At any case, after stage 5., server now has both the original DCID odcid AND the originally key-derived connection ID.

It appears that QUIC server requires a TLS certificate and TLS key even if we are not verifying the peer. If we don't load the TLS certificate or TLS key, when we do conn.recv() we will get a TlsFail error.
[2022-12-17T11:48:19.531713597Z ERROR quiche_server] d9fb4a3b84d6d01156eaa167bd6028c3a527ba60 recv failed: TlsFail

So even if you do config.verify_peer(false) you still need to provide a TLS certificate and key because TLS is something that is built into the protocol. Still it seems kind of confusing here, because then a random certificate and key should be sufficient too if we are not verifying peers. Of course the issue is you don't know whether the client will want to verify or not.
We can think of the TLS certificate and key as something provides the server an "identity". It seems it must always be generated.

And with respect to https://github.com/cloudflare/quiche/issues/1250. Let's assume then certs and keys are necessary.
Then we can use step to generate a self-signed certificate for development purposes. Let's generate it for localhost and <IP_ADDRESS>.
step certificate create \
  localhost localhost.crt localhost.key \
  --profile self-signed \
  --subtle \
  --no-password \
  --insecure \
  --force \
  --san <IP_ADDRESS> \
  --san ::1 \
  --not-after 31536000s

The above generates a cert like:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:<PHONE_NUMBER>62316474925316115295518905462 (0x4f1a0c7a47d87b139dac28b12e0cd476)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=localhost
        Validity
            Not Before: Dec 17 12:11:00 2022 UTC
            Not After : Dec 17 12:11:00 2023 UTC
        Subject: CN=localhost
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    6c:ca:34:fd:43:9e:a3:c4:96:07:3a:aa:d2:83:37:
                    bd:99:15:df:34:4f:ff:e9:be:e3:f8:4d:85:7c:73:
                    2a:91
                Y:
                    b1:5b:10:fc:22:83:a3:19:9a:80:a6:8d:36:0e:8d:
                    73:78:86:25:58:67:26:b2:60:f3:c4:97:aa:7b:11:
                    98:50
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Subject Key Identifier:
                81:56:BA:8B:E2:DF:0E:73:1E:5F:E4:82:61:43:4C:09:4F:D4:47:98
            X509v3 Subject Alternative Name:
                IP Address:<IP_ADDRESS>, IP Address:::1
    Signature Algorithm: ECDSA-SHA256
         30:45:02:21:00:d9:de:15:52:d2:8d:c1:a9:b2:a1:39:ff:c2:
         4b:8d:33:75:63:f5:02:d5:25:90:c7:ec:e6:09:41:76:70:69:
         b0:02:20:4e:87:92:67:8b:78:6f:e5:2d:9a:7b:b2:48:82:31:
         93:3d:34:fc:34:61:83:1c:2b:04:d8:3c:d1:51:f5:61:45

In production usage, we would probably want to do this without having to load from a file. However QUIC config only provides ways to load from file, or to take a boringssl context object. I'm not even sure if that would work here.
So that could mean PK may need to produce a temporary file and then call these functions to load. This library could facilitate this, produce a temporary file somehow and then just load it from there, then delete the temporary file.

And we have a successful connection!
To test:
npm run napi-build
npm run ts-node -- ./quicserver.ts

Then in quiche project:
cargo run --bin quiche-client -- --no-verify 'http://<IP_ADDRESS>:55555'

This is using the certificates that have been put into ./tmp/localhost.crt and ./tmp/localhost.key.
Without the --no-verify the client will fail immediately after the server first responds properly with conn.send, it probably checks the contents of the cert and decides this is invalid, and sends a Handshake packet which probably indicates an error or a closing of the connection.
Furthermore I can see that after the first conn.recv() the TLS handshake hasn't even started. So the connection is not in early data nor established. After sending the response with conn.send which is 1200 bytes.
  WE GOT A CONNECTION! Connection {}
  Connection Trace ID b263221dc542ab564ed3dddcf1c1d3adfb34a367
  RECV INFO {
    to: { addr: '<IP_ADDRESS>', port: 55555 },
    from: { addr: '<IP_ADDRESS>', port: 50194 }
  }
  CONNECTION processed this many bytes: 1200
  NOT In early data or is established!
  SENDING loop
    SENDING 1200 {
      from: { addr: '<IP_ADDRESS>', port: 55555 },
      to: { addr: '<IP_ADDRESS>', port: 50194 },
      at: [External: 37621b0]
    }
    SENDING 0 null
    NOTHING TO SEND

The next packet that the client sends which is still an Initial packet, and I'm guessing this is where the config.enableEarlyData() comes in, with actual data.
In this case the connection is now in early data or established, and there is both a writable and readable stream:
  EXISTING CONNECTION
  Connection Trace ID b263221dc542ab564ed3dddcf1c1d3adfb34a367
  RECV INFO {
    to: { addr: '<IP_ADDRESS>', port: 55555 },
    from: { addr: '<IP_ADDRESS>', port: 50194 }
  }
  CONNECTION processed this many bytes: 1350
  WRITABLE stream 0
  READABLE stream 0
  SENDING loop
    SENDING 516 {
      from: { addr: '<IP_ADDRESS>', port: 55555 },
      to: { addr: '<IP_ADDRESS>', port: 50194 },
      at: [External: 38f6be0]
    }
    SENDING 0 null
    NOTHING TO SEND

At this point we can see that a stream can be writable... and the stream also has data to be read.

At this point we need to hook the stream events into the dom/web streams we were prototyping earlier. I've believe we don't require any kind of partial data objects, it should all be managed as web stream backpressure.

Right now the above all happens in one handler handleMessage, as we mentioned earlier, getting a message is just 1 event. We should handle multiple events and emit other events that can trigger other handlers. So handling message event could be split into other events once we have a connection or not. We can also factor out the connection setup into its own function...

I'm trying to connect the QUIC stream to the WritableStream.
So in our QUICStream class we have both a readable and writable properties. The writable is a WritableStream. This stream ultimately connects the RPC writer to the QUIC stream.
RPC Writer -> WritableStream -> QUIC stream

It is the latter half that is confusing to implement.
In the rust quic server example, they manage this by storing a map of partial responses. The handle_writable first looks up to see if there's any partial responses. If there is, it will perform conn.stream_send. This may send less bytes than what is needed to be sent. In that case, the remaining response is saved to be used again later.
Now it turns out the reason why you cannot just keep pushing bytes into conn.stream_send even if you are flushing the QUIC connection to the UDP socket is because the other peer's send buffer may still be full.
This means we have a few complications:

The WritableStream has to have a backpressure that slows down the RPC writer, the RPC writer can just wait for the ready promise before doing anything further.
This backpressure mechanism has to be derived from any situation where the conn.streamSend returns a sent length less than the input chunk.
If the write() method of the WritableStream is a promise, only when the promise is resolved will the WritableStream proceed to continue writing.
It seems that if write() can be made to only resolve when the chunk is fully sent out, then that would be a sufficient backpressure.
The problem is that for a chunk to be fully sent out, you have to wait for next "event" before attempting another conn.streamSend. This "event" could another UDP socket message, or it could be a QUIC connection timeout event.
Therefore, how does the write() promise wait for such a thing in an asynchronous way?

When we iterate over conn.writable(), what it is telling us is that these streams have capacity to be written to. Ultimately the fact that a stream is writable is an push-flow event that is pushed from the UDP socket datagram being received.
This can be translated to an event being emitted like events.dispatchEvent(new QUICStreamWritableEvent({ detail: streamId })).
Could this event somehow result in the write() promise end up resolving?
For that to occur, we would need the write() promise to await on an event handler to be called, and where the event results in the remaining chunk to be completely sent to the QUIC stream.

That may look like something like:
async function streamSendFully(chunk) {
  let sentLength;
  try {
    sentLength = conn.streamSend(streamId, chunk);
  } catch (e) {
    throw e;
  }
  if (sentLength < chunk.length) {
    await new Promise((resolve) => {
      streamEvents.addEventListener(
        'writable',
        () => {
          resolve();
        }
      );
    });
    return await streamSendFully(
      chunk.subarray(sentLength)
    );
  }
}

new WritableStream({
  async write(chunk: Uint8Array, controller) {
    await streamSendFully(chunk);
  }
});

Notice that the streamSendFully is essentially awaiting on the writable event before it attempts to finish the streamSendFully.
This writable event will need to come from an EventTarget for a specific stream.
In our QUICServer we are going to have a map of QUICConnection (which itself wraps the JS bridged object from rust), which must then maintain QUICStream objects.
Since our QUICStream is going to wrap ReadableStream and WritableStream. Then we can potentially have this event just be emitted against the same QUICStream object.
Thus:
async function streamSendFully(chunk) {
  const sentLength = conn.streamSend(streamId, chunk);
  if (sentLength < chunk.length) {
    await new Promise((resolve) => {
      this.addEventListener(
        'writable',
        () => {
          resolve();
        },
        { once: true }
      );
    });
    return await streamSendFully(
      chunk.subarray(sentLength)
    );
  }
}

Some things above still need consideration:

How does stream closing affect this?
How do we deal with fin? Or 0-length chunks?
Should there be a situation where the internal promise can be rejected?
Do you need to do return await streamSendFully(...) at the end or can we just do return streamSendFully?

Also note that the above snippet was iterated from a recursive callback function.

The result here is that we have a promise that only resolves when the chunk is fully sent onto the stream, which itself depends on a writable event being emitted to the QUICStream object, which can only occur when conn.stream() iteration occurs which derives this "event" from a receiving a UDP socket message or a QUIC connection timeout.

The mixing of promises and event handlers is always going to be a bit complicated because promises are pull-oriented API while event handlers is push-oriented API.
And in this case, it looks like we have situations where a write() promise pulled to be resolved can only be resolved while waiting for push-events... and may not be resolved if those push-events never occur. Failure scenarios could mean the stream gets closed or there is an error.

Note that in order to run the quiche/apps:
cd quiche/apps
cargo run --bin quiche-server
cargo run --bin quiche-client -- --no-verify https://<IP_ADDRESS>:4433

By extending QUICStream with EventTarget, we get something like this:
class QUICStream extends EventTarget implements ReadableWritable<Uint8Array, Uint8Array> { ... }

QUIC streams in quiche doesn't have any kind of object. They are just identified by the streamId.
So this object is what will be exposed to the end user when they get new streams to work with.
We can see that this is a duplex stream, so web-stream functions can work on it.
In its constructor, we construct the this.readable = new ReadableStream and this.writable = new WritableStream.
Now as per the above dicussion, I'm prototyping the method streamSendFully that is an asynchronous function that only resolves when all the chunks are fully sent. Right now there's no specialised error handling.
It does this by checking that if the sent length is less than the input data, it will proceed to wait for the writable event to be emitted before it proceeds. This is our "plug"/lock.
The idea is that the QUIC connection would emit a writable event to the QUIC stream to unplug it.
However using events here might leak too much implementation details. Alternatives might be to use a function and a lock from async-locks, but this should be sufficient to prototype this.
Actually we should revisit the plug/unplug pattern: https://github.com/MatrixAI/Polykey/blob/91287ab57f9cc34c337f1dc9f7523ea122aa96e5/src/nodes/Queue.ts#L76-L99
Furthermore the writable stream:

start doesn't do anything atm..., the assumption is that the stream does already exist. This may be more relevant later when one starts a QUIC connection/stream from the client-side.
write just calls await this.streamSendFully.
close just calls await this.streamSendFully(Buffer.from([]), true) which sends a 0-length buffer and switches fin to true
abort calls this.conn.streamShutdown(this.streamId, quic.Shutdown.write, reasonToCode(reason)). This would send a RESET_STREAM frame indicating an abrupt termination of a stream with an application supplied error code, where 0 is the general code in case we don't understand the reason.

On the readable side of QUIC stream, we are dealing with a push-flow source.
In our start method, we register a handler for the readable event. We will use the readable event every time we know that a QUIC stream is readable.
Upon being readable we go through this logic:

Is it paused? If so, do nothing and return here.
Allocate a buffer of 1024 and read from the stream.
If we get an exception of Done, then there's nothing to do, we just return. (Here I have reversed the change of Done to 0-length reads/writes because it does seem to indicate something different from having an empty buffer).
If we get a different exception, we send this to the controller with controller.error(e); and we return here`.
Now we enqueue the buffer with the received length.
If the fin is true, then we do controller.close() and return.
If the desired size is not null, and it is <= 0 then we pause.

For the pull() method, we will:

Unpause the recv plug
Dispatch a new readable event, which will trigger another read of the handleReadable.

For the cancel() method, we will:

Remove the event listener handleReadable.
Run this.conn.streamShutdown to shutdown the read side with the reason mapped to a code.

Some possible improvements:

Using byob seems it might be faster.
How exactly the fin and Done works will need to be experimented with.
How exactly the desiredSize is null needs to be considered. It seems that users can set the stream's desiredSize to be null to avoid any blocking at all, and this would just cause the controller the buffer up all the data... basically removing any kind of backpressure and having an unbounded queue.

It turns out that plugging the receive was a simple matter of switching a boolean. No promises or events required.
On the other hand for the send side where we wait for promise to resolve based on a writable event, an alternative is to use a plug, and expect the emitter of writable to explicit unplug however this is not symmetric.
By symmetry I mean that the QUICStream right now receives 2 internal events: readable and writable. Where both events are ultimately determined by whether the QUIC stream from quiche is in fact readable or writable, and these events being derived from every UDP socket message and timeout event.

If we are going to expose the Done exception, we will need to do that for the other functions too, not just stream_recv and stream_send.

Just a note, when we have the QUICStream later. We can do things like: await quicStream.writable.getWriter().ready;.
This would essentially be waiting for the stream to be ready to write. This is how the backpressure works at the writer to web stream stage.

Since we are using EventTarget right now, we need to consider error handling. The EventTarget does not have any special handling for exceptions or errors. That means we should not use async functions as event handlers for EventTarget. Any rejections would be considered an uncaughtException and will go to process.on('uncaughtException', () => { }).
This means any errors on socket sending or other callbacks must be done with callbacks that then emit the event to the error handler.
A default error handler should be available... that will then throw the event detail as an uncaught exception if no custom error handler is made available.

So the command:
napi build --platform --js ./src/native/index.js --dts ./src/native/index.d.ts

Will compile a single binary index.linux-x64-gnu.node while putting the index.js and index.d.ts into the same directory.
As long as this is the case, TSC will realise that index.d.ts is the types for index.js.
Now I can choose to name the index.linux-x64-gnu.node differently, this is controlled by the napi config inside package.json.
The generated JS file does do the equivalent of node-gyp-build in js-db. It figures out which platform we are on, and then loads the appropriate dependency.
Interestingly it does in fact support:

Different platform triples that are in the local directory
The importation of "optional dependencies" such as @matrixai/quic-android-arm64.

I'm wondering how to control these names. It seems to automatically derive as suffixes of the main package name being @matrixai/quic.
So we would end up with something like:
@matrixai/quic
@matrixai/quic-android-arm64
@matrixai/quic-android-arm-eabi
@matrixai/quic-win32-x64-msvc
@matrixai/quic-win32-ia32-msvc
@matrixai/quic-win32-arm64-msvc
@matrixai/quic-darwin-x64
@matrixai/quic-darwin-arm64
@matrixai/quic-freebsd-x64
@matrixai/quic-linux-x64-musl
@matrixai/quic-linux-x64-gnu
@matrixai/quic-linux-arm64-musl
@matrixai/quic-linux-arm64-gnu
@matrixai/quic-linux-arm-gnueabihf

If we go with the optional package route, we will need to have a multi-package repo, since each separate platform will need its own directory with a package.json.
Also it seems once we create optional dependencies, we can apply constraints to each optional dependency using the os and cpu keys:

https://docs.npmjs.com/cli/v9/configuring-npm/package-json#os
https://docs.npmjs.com/cli/v9/configuring-npm/package-json#cpu

This should be able to add constraints to which ones you are installing for.
However there's another issue. Suppose you creating a cross-platform native package that depends on another cross-platform native package. In that scenario, if you are only allowed to install linux because you're on linux, it can make it more difficult to distribute your own set of binaries, unless you are also changing the entire OS via a CI/CD. For example things like android above, who would be building using an android OS.
Atm, it seems like all optional packages would be installed without the constraints.

https://github.com/npm/rfcs/pull/519
https://github.com/napi-rs/package-template
https://napi.rs/docs/deep-dive/release#3-the-native-addon-for-different-platforms-is-distributed-through-different-npm-packages

I've moved all the rust code into src/native/napi. The Cargo.toml has been updated accordingly.
We are probably going to create a subdirectory called packages and actually make use of optionalDependencies in this main package.
The packages directory then requires each of the built binaries to be put into the directory during the CI/CD build process and then published.
The os/arch constraint will then be applied.

Ok one of things that is different between EventEmitter and EventTarget is the handling of errors. Here's a demo:
const et = new EventTarget();
et.dispatchEvent(new Event('error');

Nothing happens. No handler for error event, no special handling.
Now in EventEmitter which is how the TCP server behaves along with other node constructs:
import events from 'events';
const ee = new events.EventEmitter();
ee.emit('error');

Immediately we get:
Error [ERR_UNHANDLED_ERROR]: Unhandled error. (undefined)

This is the case with TCP servers:
import net from 'net';

async function main() {
  const server = net.createServer((conn) => {
    console.log(conn);
  });
  server.listen(55555, () => {
    server.emit('error');
  });
}

void main();

This results in the same error. Only when we add an event handler for error does it not end up breaking the entire node runtime.
So if we want to replicate the behaviour of TCP server and TCP connections, we would need to do something similar with EventTarget. To do so, we would need to add a "default" error handler to the error event, and then check that the error event hasn't previously been handled, so it's the handler of last resort.
Alternatively we can overload the addEventListener so that if the error handler is being assigned, we remove the default error handler.
Note that it's possible the error handler is removed, so as long as there's no handlers, the throwing the exception would need to be ensured.
There's another thing, it would be important to ensure that if errors are just ignored, that the server/connection can continue to work, especially given the node runtime will continue to run.
But one could also differentiate between recoverable errors, and unrecoverable errors.

The main top level classes should avoid to use as much nodisms as possible. So I'll need that defaulting behaviour to the classes extending EventTarget.
There's some additional complexity. The role of the QUICServer and the QUICConnection. In TCP there's just the server and the socket. That's it. The socket itself can have errors too, but that may also result in errors that exist the node system. In QUIC we would have 3 concepts, server (wrapping UDP socket, and propagating udp errors), connection, stream.
I need to check what happens if the an error is emitted to the tcp connection (while the server is running), if the same thing happens. And I just realised that I need to propagate the UDP socket error to the QUICServer too.

Actually I cannot do the above error handling with EventTarget very easily. This is because event target does not allow the same listener instance to be used multiple times unless the capture is toggled.
We don't know how many listeners there would be an error handler. We would end up having to keep track of these listener instances... Which just overcomplicates the situation.
Options are:

Don't replicate behaviour of event emitter, without an error handler, errors just go to nowhere.
Use EventEmitter, but then we have to figure out a solution in the future for non-node environments, there's some libraries like eventemitter3, but I suspect not very lightweight - consider that the UDP socket dgram module is already node-specific, I imagine that further abstraction would be necessary in the future for non-node environments anyway
Bring in a library like eventemitter3. However it turns out even this library doesn't actually have a default behaviour of throwing the exception just like event target.

I think the best solution right now is 1., to not replicate the behaviour, users can still decide what to do with their addEventListener('error').

Regarding garbage collection.
For streams, stream close happens in 2 separate ways:

Read side
Write side

For the read side, a stream could be closed when we receive fin packet. This is indicated by the fin boolean through connection.streamRecv().
In this case, it would mean that we should close the stream. Because this means the QUIC stream is closed, and therefore our web stream is closed too.
Alternatively it is possible that, our web stream is cancelled, and thus stream.cancel() is called. In this case, we perform connection.streamShutdown on the read side.
I'm not entirely sure what would happen if there's an error during streamRecv, if that means we should attempting to shutdown the stream, or whether that would the case already. The examples don't show what happens.
Right now I just propagate the error using controller.error(e);.
However I imagine that if there is in fact a error here... we should attempt to shutdown the stream. I'm trying to see if this is a problem.
For the write side, we have both stream.close and stream.abort. The stream.close sends a fin packet, whereas stream.abort immediately shutsdown.
So because these streams are duplex (and we are only dealing with duplex streams atm in this library), then a QUICStream is only truly closed when both sides are closed. And right now half-open state is possible.
So upon any of the above closing of the readable and writable, we call a function right now called gcStream that checks that both read and send sides are closed, if so, it proceeds to delete the QUICStream from the parent streams map.
This means stream lifecycle can be determined by the user of the QUICStream, and by the QUICConnection. It's also possible that QUICConnection also proceeds to explicitly stop the readable and writable side with QUICConnection.stop. This proceeds to cancel on the readable, while doing writable.close because the writable side should be gracefully done.
This seems to make sense for QUICStream.

Now for QUICConnection, we have a separate problem. A connection can also be closed by outside events, primarily due to recv, but also could happen during an error with send.
In the case of recv, an error is dispatched. But in the examples, they just continue the read loop. They don't do anything to the connection.
I believe this is because:

On success the number of bytes processed from the input buffer is returned. On error the connection will be closed by calling close() with the appropriate error code.

Which implies that the recv itself will end up calling the close() if there was an error. So we don't need to do this on our end.
Ok I can see it:
                    // In case of error processing the incoming packet, close
                    // the connection.
                    self.close(false, e.to_wire(), b"").ok();

So connection.recv does in fact automatically close the underlying connection.
On the send side, if the send fails, we have to explicitly call connection.close to indicate the fact that we failed to send a packet. We have to choose whether it is an application error or a library error, the error code and error message.
The problem is that connection.close() does not mean connection.isClosed() is true. It's a lazy operation, meaning there are some stuff that still needs to be done.

Note that the connection will not be closed immediately. An application should continue calling the recv(), send(), timeout() and on_timeout() methods as normal, until the is_closed() method returns true.

And because there's callbacks in quiche library. We have to actually poll the quiche library to know when a connection is actually closed. And only then can we proceed to remove the connection from the connections map in QUICServer.

The other issue is that I'm not sure how closing a connection affects all of its existing underlying streams.
It would appear that closing a connection should also mean all its underlying streams are closed. But if that's the case, how does our QUICStream get knowledge about this?
It's possible we get a memory leak here, since if the streams are all closed, they could potentially not be readable or writable anymore, and in that case, we never get a quic stream error, and can therefore not propagate such errors to the web stream.
One alternative is to proceed to do an explicit stop on all our streams if there's a connection error. Actually we need to do it probably non-gracefully since if we cannot send data on the connection, it's rather useless to attempt to write an explicit stream close.

So a couple problems here:

How to propagate the connection close to the QUICStream object so that it can be properly garbage collected. Beware of how the underlying stream works, and how QUICStream object itself would be aware or not aware of its underlying object state.
How to deal with the fact that QUICConnection itself cannot just be closed synchronously. We have to poll whether it is closed, before we can remove it from the connections map. The polling seems to occur on every event either when we received a UDP socket message, or when there's a timeout event.

For 2., I'm just doing it at the end of QUICServer handleMessage and handleTimeout.
It's time to do the next practical test and map out what happens with the new class structure. And then from this point onwards we need to iterate on the class structure as there's still unknown unknowns and known unknowns. Difficult translating a loopy example codebase in quiche to a nodejs evented codebase.

Do note that the in order to pass in in-memory private key and certificates we will need to use boringssl's builder for the config struct. https://docs.rs/quiche/0.16.0/quiche/struct.Config.html#method.with_boring_ssl_ctx
It appears to support building up things like: https://docs.rs/boring/2.1.0/boring/pkey/struct.PKey.html#method.private_key_from_pem
And then you use functions like https://docs.rs/boring/2.1.0/boring/ssl/struct.SslContextBuilder.html#method.set_private_key which sets it up.
The builder is consumed and returns the ssl context: https://docs.rs/boring/2.1.0/boring/ssl/struct.SslContextBuilder.html#method.build.
After building this, more config settings can be set. This can done as a separate issue.

Ok I'm able to use wireshark to inspect the protocol using the filter udp.port == 55555 && not icmp.
Here we only want to see the interaction between the client and server.
One issue is that we do need to decrypt the TLS, apparently I need to make use of https://docs.rs/quiche/0.1.0/quiche/struct.Config.html#method.log_keys. That can dump the keylog file that can be loaded by wireshark.

In the initial packet the client sends to the server, it has both the DCID and SCID.
The DCID is what the client uses to identify the server. The SCID is what the client uses to identify itself.
Now why are we using dcid in our code to identify the connection? Because as chatgpt says:

The reason that the DCID is typically used to identify the connection in Quiche (and other Quic implementations) is because the DCID is included in every packet that is sent over the connection, while the SCID is only included in the initial packet sent by the client. This means that the DCID is more readily available for use as a connection identifier, as it is present in every packet.

And this is in fact true according to wireshark, there are QUIC protected paylaod packets that only contain the DCID and no longer the SCID. Because they use a short header.

Tracing the QUIC implementation is tough without some sort of tracing system for the logs. And this is a pending thing to do later.

I just realised a new logging standard that might be useful:
Receive X   <-- start
Receiving X <-- only use this if you intend 1 log message
Received X  <-- stop

We currently using ing when we start, but really we should the ing only when we intend to use 1 log message like an "event" compared to a trace which has a start event and stop event. I think in opentracing this is called a log? Not sure about the terminology there.

For 2., I'm just doing connection iteration at the end of QUICServer handleMessage and handleTimeout.
It's time to do the next practical test and map out what happens with the new class structure. And then from this point onwards we need to iterate on the class structure as there's still unknown unknowns and known unknowns. Difficult translating a loopy example codebase in quiche to a nodejs evented codebase.

With respect to the timeout. It turns out that for a new connection that was just accepted, calling timeout() returns null. This must imply that the timeout() is intended be "polled", that is we must call into the quiche library on different events to find out whether the connection has a timeout that we must setup. In the rust examples, these appear to occur on every new message sent. Because otherwise how would we ever know if a connection ever needs to set a timeout or not? We may be able to spread out checks for this depending on various state changes on the connection but this requires further experimentation.

We might to start working on the QUICClient class to fully work out the protocol... the server is getting to the point where the streams are just waiting for work to be done. And the example quiche client is using HTTP3, and we also need to somehow deal with the connection streams too...
So when we create a quic server, we need to have an event handler for new connections, but for each connection a handler for new streams.

We need to start building out the QUICClient in order to prove that all these requirements can be met: https://github.com/MatrixAI/Polykey/issues/234#issuecomment-1123133027.
Specifically:

The ability to multiplex separate connections into 1 socket, and therefore that means 1 port. This is necessary for NAT busting. A connection is between node to node. This means the same port/socket is used for multiple connections to different nodes.

Right now quiche doesn't do anything with sockets, so we are just receiving messages on the UDP socket and then processing it using the header parse. But we need to see how this would be done on the client side.

When using the same socket, it is likely we would use the same dgram socket for multiple client connections to different servers and also for the server.
We need to also be able to identify whether the received packet is intended for which client connection.

The client.rs in quiche apps is more comprehensive than the example client.rs it also demonstrates how connection migration would be done on the client side. Only clients can do connection migration right now.

In terms of identifying packets, we can work with parsing their connection IDs, or their remote addresses. If we identified packets as coming from a certain address we could route to the right client. But we would have to first see if it is intended for the server, and not for the clients.
Also discussion on long and short headers:

Packets with long headers include Source Connection ID and Destination Connection ID fields. These fields are used to set the connection IDs for new connections; see Section 7.2 for details.
Packets with short headers (Section 17.3) only include the Destination Connection ID and omit the explicit length. The length of the Destination Connection ID field is expected to be known to endpoints. Endpoints using a load balancer that routes based on connection ID could agree with the load balancer on a fixed length for connection IDs or agree on an encoding scheme. A fixed portion could encode an explicit length, which allows the entire connection ID to vary in length and still be used by the load balancer.

But the problem even with addresses, we would need to know if these addresses are legitimate, which is why the server does the stateless retry.

This version of QUIC uses the long packet header during connection establishment; see Section 17.2. Packets with the long header are Initial (Section 17.2.2), 0-RTT (Section 17.2.3), Handshake (Section 17.2.4), and Retry (Section 17.2.5). Version negotiation uses a version-independent packet with a long header; see Section 17.2.1.
Packets with the short header are designed for minimal overhead and are used after a connection is established and 1-RTT keys are available; see Section 17.3.

In reviewing Polykey's usage of UTP, it does seem that UTP was made for this usecase. Where we have the ability to create a UTP object when doing UTP.createServer(), and then subsequently it was possible to create UTP connections on the same UTP object. Thus being able to act like a server, and the ability to create multiple clients.
It seems that quiche should be capable of doing this. There's no restriction on sharing the same UDP socket after doing bind, we can have multiple QUICClient and a QUICServer all using the same UDP socket.
The only issue is that all the examples show that the clients and server all depend on handleMessage but we just need a way of parsing these UDP messages and identifying what kind of packet it is... and then directing it. Although I'm also confused that it's possible for there to be coalesced QUIC packets in 1 UDP datagram. If so, then how does the Header.fromSlice work? Maybe it only reads 1 QUIC packet out of the UDP datagram, meaning its possible for there to be more than one?

The client's INITIAL packet sent to the server contains a randomly set SCID and DCID.

When an Initial packet is sent by a client that has not previously received an Initial or Retry packet from the server, the client populates the Destination Connection ID field with an unpredictable value. This Destination Connection ID MUST be at least 8 bytes in length. Until a packet is received from the server, the client MUST use the same Destination Connection ID value on all packets in this connection.
The Destination Connection ID field from the first Initial packet sent by a client is used to determine packet protection keys for Initial packets. These keys change after receiving a Retry packet; see Section 5.2 of [QUIC-TLS].
The client populates the Source Connection ID field with a value of its choosing and sets the Source Connection ID Length field to indicate the length.

Ok so I think the idea here is that if we are to combine clients and server, we then have to have a combined handleMessage demuxer. It must then analyse the packet header to get the connection ID. If the UDP datagram has multiple coalesced QUIC packets, then it is assumed that all these QUIC packets would be intended for the same connection. If any of the packets deviates from the first packet's connection IDs, then that will result in a processing error later by quiche.
Then inside this demuxer it has to refer to a ConnectionMap. This map is basically something that will need to be shared across QUICClient and QUICServer because it has to be used to identify QUIC packets intended for client-side connections or server-side connections.
Further prototyping required on this front.

I'm not entirely sure if NodeJS supports dual stack properly where ipv4 mapped ipv6 addresses are possible: https://stackoverflow.com/questions/61741547/udp-client-server-mix-ipv4-and-ipv6
See https://blog.apify.com/ipv4-mapped-ipv6-in-nodejs/
Suppose we use the rinfo, then if we were bound to ::, would we then get ipv4 packets as ::ffff:<IP_ADDRESS>?
And also this only works if we bind to :: and not any other IPv6 addresses. This will require further testing later.

I got a confirmation that using connection IDs is how we would demux packets for shared UDP socket.
So I'm creating a QUICSocket to encapsulate the management of a shared socket object, since creating an appropriate UDP socket is a bit complex.

Now that we know that some packet headers are short-form and thus do not have an SCID, therefore when we do quiche.header.fromSlice, what happens to the scid property? Well it still exists, but the Rust code uses ConnectionId::default() which will produce an empty byte array.

Updated handshake diagram with DCID and SCID.
The SCID is the source connection ID, and it is meant to be chosen by the peer to represent the ID for itself.
The DCID gets "agreed" upon using the below protocol.
Retry and version negotiation packets and packets with short header cannot be coalesced with other packets in the same datagram. This means they are always by themselves.
Notice that in the retry packet, a new SCID S2 is used. This is derived from the S1 via HMAC signing. The HMAC signing is not part of the QUIC specification, however it is security feature to ensure the integrity and prevent replay attacks. It is just chosen by the quiche implementation to use in its examples.
The client has to change its DCID in response to this retry packet. But it is possible for the server to change its SCID again in its next initial packet. In which case the client has to change the DCID again. There's no specific reason why this might occur.
Note that at this point in time, the QUIC spec only supports client-side connection migration. It's not expected that the server would have its address change. Although this would be the case in the future. Connection migration just means that the client can change its address (possibly because it changed networks on a mobile network).
In PK's P2P case, any change in address would affect both client side and server side. So it would require performing a new handshake and reconnecting all connections.
┌────────┐                                ┌────────┐
│ Client │                                │ Server │
└───┬────┘                                └────┬───┘
    │          ┌───────────────────┐           │
  1.├─────────►│Initial            ├──────────►│
    │          │version:<PHONE_NUMBER>│           │
    │          │token: []          │           │
    │          │dcid: S1           │           │
    │          │scid: C1           │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
  2.│◄─────────┤Version Negotiate  │◄──────────┤
    │          │version: 1         │           │
    │          │dcid: C1           │           │
    │          │scid: S1           │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Initial            │           │
    │          │version: 1         │           │
  3.├─────────►│token: []          ├──────────►│
    │          │dcid: S1           │           │
    │          │scid: C1           │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Retry              │           │
  4.│◄─────────┤token: [S1]        │◄──────────┤
    │          │dcid: C1           │           │
    │          │scid: S2           │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Initial            │           │
    │          │version: 1         │           │
  5.├─────────►│token: [S1]        ├──────────►│
    │          │dcid: S2           │           │
    │          │scid: C1           │           │
    │          └───────────────────┘           │
    │                                          │
    │          ┌───────────────────┐           │
    │          │Initial            │           │
    │          │version: 1         │           │
  6.│◄─────────┤token: []          │◄──────────┤
    │          │dcid: C1           │           │
    │          │scid: S3           │           │
    │          └───────────────────┘           │

In terms of demuxing the handleMessage. It is also possible to register multiple handlers for the message event. However this is not efficient because every time an message event occurs on the dgram socket, it ends up calling every single handler.
If we use event.stopImmediatePropagation() we can cancel the handling of subsequent handlers. But again it's still possible with a 100 QUIC clients, it would check all 100 clients before hitting the server.
So our demuxing logic instead happens within a single message handler, that instead checks a shared connection map to decide what to do depending on whether a server is registered or not, or whether the connection exists or not.

So now I'm at the point where the QUICSocket can call into the QUICServer and ask it to handle a new connection, but only if it is registered as the server for the socket. Otherwise it just discards those kinds of packets.
It can also acquire an existing QUICConnection from the connection map if the connections DCID matches one of them.
However some decisions has to be made now.

Should the QUICServer or the QUICSocket be the one to put the connection into the map?
How should the quic socket push data into the connection? Should it handle it at the QUICSocket handleMessage, or is that something we delegate that to the QUICClient or QUICServer? I'm still not entirely sure what the QUICClient really does besides bootstrap a QUICConnection.
When identifying the QUICConnection, there doesn't seem to a way to identify whether it is a client or server connection, and if we identify the client, how to then access the client object too. Not sure if we even need this.

In the case of 1., we could say that neither should, that is instead the construction of a QUICConnection leads it to be put into the map, and correspondingly the destruction of the connection takes it out of the map. Creation and destruction can be approached with a CreateDestroy pattern where, asynchronous creation and asynchronous destruction.

For the QUICConnection, I've made the 2 methods recv and send represent:

recv means bridging socket -> connection -> streams.
send means bridging socket <- connection <- streams.

This means QUICConnection.recv is supposed to be called by the socket, since that's where the data flows from.
The QUICConnection.send on the other hand is supposed to be called by the stream, since that's where the data flows from.
HOWEVER the QUICConnection.send may also have data flowing from other sources, such as just handshake protocol, so even in in the handle message of the socket, in can trigger QUICConnection.send.
So the control flow can come from other sources.
This means the send actually loops over the connection object and flushes all data to the socket.
This also means, there's some tight coupling between all these objects, since the socket injects itself into the server/client which itself injects it to the connection object when it creates it... etc. I've realised that there's no other way to do it, it has to be somewhat tightly coupled mechanism.
Finally the problem is errors, where do they go? Should they go to the caller? Or in the case where the caller isn't available or it's all just event driven, it seems errors has to goes to a generic error event (that is like a parent object listening for the error event). But in that case it goes against how we normally write things with promise based methods, where errors flow back up the control flow.
I'm still not entirely sure how to do it, so right now I'm doing both and going to figure out which is the best soon.

This paper https://www.dsi.fceia.unr.edu.ar/downloads/informatica/info_III/eventexcep.pdf talks about exactly the issue I'm dealing with right now. How to manage errors from methods are basically "event handlers". The composition of these objects is not strictly hierarchical. The event invoker isn't actually really more capable of handling these errors, and especially with a promise-API, having the invoker discard errors results in ugly line noise of await p.catch(() => {}); (and in some cases, there should be exceptions where something was written incorrectly.

Ok the QUICConnection now works against 3 possible events: send, recv and timeout. These 3 events drive all interactions that could occur with a connection. The send results in a UDP socket event. The recv results in stream events. The timeout ends up closing the connection eventually due to draining timer, idle timer or path timer.
I think on server side created connections, they don't have a timeout until the first recv call occurs (which happens immediate for newly created server-side connections).

Cool, it is sort of working right now. I can see the timeouts being set, but it's not really cleaning up the connection just yet.
INFO:QUICServer:Starting QUICServer on <IP_ADDRESS>:55555
INFO:QUICSocket:Starting QUICSocket on <IP_ADDRESS>:55555
INFO:QUICSocket:Started QUICSocket on <IP_ADDRESS>:55555
INFO:QUICServer:Started QUICServer on <IP_ADDRESS>:55555
DEBUG:QUICServer:QUIC packet version is not supported, performing version negotiation
DEBUG:QUICServer:Send VersionNegotiation packet to <IP_ADDRESS>:34101
DEBUG:QUICServer:Sent VersionNegotiation packet to <IP_ADDRESS>:34101
DEBUG:QUICServer:Send Retry packet to <IP_ADDRESS>:34101
DEBUG:QUICServer:Sent Retry packet to <IP_ADDRESS>:34101
DEBUG:QUICServer:Accepting new connection from QUIC packet
INFO:QUICConnection ad0c415a56fd5c885fd2a35a2fa9f2580932122a:Creating QUICConnection
EVER TIMEOUT null
INFO:QUICConnection ad0c415a56fd5c885fd2a35a2fa9f2580932122a:Created QUICConnection
got the connection QUICConnection {}
EVER TIMEOUT 4999
EVER TIMEOUT 998
EVER TIMEOUT 4997
EVER TIMEOUT 28
EVER TIMEOUT 4999
EVER TIMEOUT 4999
EVER TIMEOUT null
^CINFO:QUICServer:Stopping QUICServer on <IP_ADDRESS>:55555
INFO:QUICConnection ad0c415a56fd5c885fd2a35a2fa9f2580932122a:Destroying QUICConnection
INFO:QUICConnection ad0c415a56fd5c885fd2a35a2fa9f2580932122a:Stopped QUICConnection
INFO:QUICSocket:Stopping QUICSocket on <IP_ADDRESS>:55555
INFO:QUICSocket:Stopped QUICSocket on <IP_ADDRESS>:55555
INFO:QUICServer:Stopped QUICServer on <IP_ADDRESS>:55555

Important that this library will only focus on QUIC, and not HTTP3.

We need to investigate how the timers work and ensure that they are actually being timed out properly and then it should proceed to close the connection if there's no response on the other side.

@tegefaulkes regarding 14.

[ ] Propagate the rinfo from the UDP datagram into the conn.recv() so that the streams (either during construction or otherwise) can have its rinfo updated. Perhaps we can just "set" the rinfo properties of the connection every time we do a conn.recv(). Or... we just mutate the conn parameters every time we receive a UDP packet.

We need to understand that under QUIC, it's possible for the same connection to have different remote host and remote port.
See: https://www.rfc-editor.org/rfc/rfc9000.html#name-connection-migration
Now I'm actually not entirely clear atm.
It's possible that when the client migrates to a new network path, that the dcid or scid changes.
I haven't handled this case yet in the QUICConnection class. That is, it would mean maintaining the same QUICConnection object instance (because we don't want to lose the streamMap state), and transitioning the connection ID.
At any case, this mean the remoteHost and remotePort could change at any point in time when querying the connection.
It's even possible that there could be multiple valid concurrent remote hosts and ports that are going to different streams on the same connection, I haven't confirmed this case yet.
Furthermore you're handling events on the stream itself, and you want to give it an object of remote host/port information at the point of handling that stream.
My current solution is to:
We put in the properties localHost, localPort, remoteHost and remotePort into QUICConnection, however the remoteHost and remotePort can be change at any time, so the information you are passing in the handler is only valid at that specific point in time where you are constructing the object. This may be sufficient for your usecase, since you are just using it for logging and nothing else.

So now there is:
QUICConnection.localHost
QUICConnection.localPort
QUICConnection.remoteHost
QUICConnection.remotePort

The remoteHost and remotePort can change on every QUICConnection.recv invocation.
This still needs to be tested with connection migration. Atm only client connections can migrate, servers cannot migrate according to the QUIC spec.
Since every PK agent is both client and server, this makes it bit weird.

Migrating a connection to a new server address mid-connection is not supported by the version of QUIC specified in this document. If a client receives packets from a new server address when the client has not initiated a migration to that address, the client SHOULD discard these packets.

That means if a PK agent were to be migrating to new IP/port due to disconnection, then the servers would have to restart. This would be important on mobile networks, it's something we will think about when we get PK on the mobile phone. We don't actually need to do "live migration" for our PK agent, we can rely on our kademlia system to do this, but indeed all client connections would have to automatically disconnect and retry on the new address upon detecting this on the node graph.

The config.setMaxIdleTimeout does control the initial timeout.
So on the server side, upon constructing the QUICConnection, the initial call to conn.timeout() gives back null.
However after the first conn.recv(), the next call gives back 6000 ms. (Actually 5999).
What's a bit confusing is that after calling conn.send(), the next call changes this to 1000ms (actually 998).
The max idle timeout parameter is explained as:
    /// Sets the `max_idle_timeout` transport parameter, in milliseconds.
    ///
    /// The default value is infinite, that is, no timeout is used.
    pub fn set_max_idle_timeout(&mut self, v: u64) {
        self.local_transport_params.max_idle_timeout = v;
    }

This parameter is actually exchanged with the client in order to get the lowest possible max idle timeout.
On the second time and third time we call it, we are checking if we are draining. In both cases this is false (probably because the connection isn't closed).
There's a check as to the lowest loss detection timer, and again compared with the idle timer.
This means on the second time it is called, the idle timer must win, and there probably isn't a loss detection timer. And thus we get 5000ms.
Then on the third time, there must be a loss detection timer set, and therefore the 1000ms is lower, and this is returned instead.
The config does not have anyway of setting this loss detection timer, this must be set automatically.
                .filter_map(|(_, p)| p.recovery.loss_detection_timer())

The loss detection timer is set inside the Recovery struct.
It appears to be set by a function Recovery::set_loss_detection_timer.
This feels like this is called by the Connection::send. Note that the loss_detection_timer is a Option<Instant>.
I suspect this loss detection timer has something to do with: https://www.rfc-editor.org/rfc/rfc9000.html#name-loss-detection-and-congesti.
This would make sense that the loss detection time may be something that is lower than the idle timeout, since we are more concerned if some packet was lost, and therefore probably needs to be resent.
From ChatGPT:

The draining timer is used to ensure that all data has been sent on a connection before it is closed. The idle timer is used to close a connection if no data is sent or received for a certain period of time. The loss detection timer is used to detect when packets have been lost in transit and retransmit them if necessary. All 3 timers are used to manage the state of a connection and ensure reliable communication.

It does make sense that we would reset the timer upon additional events.

Ok so when the connection times out, and it does eventually. The handler is in fact called.
Now I put in:
          console.log('draining', this.conn.isDraining());
          console.log('closed', this.conn.isClosed());
          console.log('timed out', this.conn.isTimedOut());
          console.log('established', this.conn.isEstablished());
          console.log('in early data', this.conn.isInEarlyData());
          console.log('resumed', this.conn.isResumed());

Before and after calling this.conn.onTimeout().
I can see something like this:
TIMEOUT HANDLER CALLED
draining false
closed false
timed out false
established true
in early data false
resumed false
AFTER ON TIMEOUT
draining false
closed true
timed out true
established true
in early data false
resumed false

It is interesting here that upon calling onTimeout() the connection is closed. But the draining remains false.
This can mean that if it is already closed, the draining does not remain true even if draining were to be the case after which a timeout occurs and then the closing happens.
Anyway this leads us to call this.send() again just in case timing out means that we are draining now.
In the send, the finally clause kicks in and here it goes to run setTimeout again.
At this point we do re-run the setTimeout before then checking if the connection is closing or draining.
Now 2 things happen here, the next call to the this.conn.timeout is null, this is because the connection is already closed, therefore no timeout is further necessary. However I imagine that if instead the connection was draining, another timeout is necessary.
The problem was that I was checking that the status is not in destroying and that the connection was isClosed() && isDraining(), that's obviously not possible now that we know what happens inbetween onTimeout().
So I've changed this to isClosed() || isDraining().

On solution is something like this:
      if (
        this[status] !== 'destroying' &&
        (this.conn.isClosed() || this.conn.isDraining())
      ) {
        await this.destroy();
      } else if (
        this[status] === 'destroying' &&
        (this.conn.isClosed() && this.resolveCloseP != null)
      ) {
        // If we flushed the draining, then this is what will happen
        this.resolveCloseP();
      }

But I need more testing on flushing situations so we can see how it behaves.

Tagging @tegefaulkes to keep up with this progress.

@tegefaulkes I'm testing the handling of streams atm.
It looks like:
const writer = stream.writable.getWriter();

for await (const read of stream.readable) {
  console.log(read);
}

await writer.ready;
await writer.write(Buffer.from('Hello World'));

await writer.ready;
writer.releaseLock();

await stream.destroy();

Now I have some questions...
It seems that a "writer" is a locked reference to the writable stream. It's a way of maintaining control/ownership of the stream, such that only 1 writer can be the one using the stream.

When we call await writer.close(), and later inside stream.destroy it calls await this.writable.close(), there's an error TypeError [ERR_INVALID_STATE]: Invalid state: WritableStream is closed. How are we supposed to close a writable stream, if there's a writer that still exists? Are we meant to check that if the writer is still locked (that is with an active writer), therefore it cannot be destroyed?
In your RPC handlers where you use the async generator syntax, what do you actually do when you finish handling? In my case, I'm thinking that this.readable.cancel() then await this.writable.close() is the right order of events. But now the issue is that how do you co-ordinate that with reader/writer objects?
What does it really mean to do writer.releaseLock()? Is there actually meant to be multiple potential writers to the same stream?

So a stream can only have 1 reader or writer.
If you call getWriter() twice, you'll get a TypeError.
The only way to "destroy" the first writer is by calling writer.releaseLock().
Now if you were to do writer.close(), the stream is now closed, so that actually means you cannot get a second writer with stream.getWriter().
This means this is the only way to get 2 writers:
const writer1 = stream.getWriter();
writer1.releaseLock();
const writer2 = stream.getWriter();

After calling writer1.releaseLock it should be assumed that this writer is no longer usable, and should be left to be garbage collected.

I don't know the full details but if something is consuming a stream then it is locked. there can only be one consumer of a stream. in this case if if you take a writer of a writable with const writer = writableStream.getWriter() then ultimate control of that writable is done through that writer. So if you call await writer.close() then what will close out the writer and work down the connected chain of streams closing everything. hence the error when you close the writer and then the writableStream.

If you wanted to close the writeable stream then you'd have to check if it's locked first. Alternatively if this is part of clean up then you could throw an error into the writable stream and I think everything chained to it should end up closing and throwing if they're still being used.

When consuming the readableStreams I either read it to completion or close it early with readableStream.close() If it was consumed with a for await (const thing of thing){} loop then closing is handled automatically even if it ends early. For the writable I have to call writer.end() in some way. I think the only way to end a stream if you're not holding onto the end of the chain is to make it throw an error.

I know it's possible to end a stream without closing the whole chain. there's an option preventClose that does it, releaseLock() likely does the same. I'll need it for the raw handlers to work. I haven't looked at it recently to recall the exact usage.

Therefore while handling a stream in the RPC handler, it would be important that the handler would release the lock to any writer it has.
However this is only possible if the RPC handler were a raw handler of RWP and were to acquire a writer with stream.writable.getWriter().
If they don't release their writer with writer.releaseLock() there's no standard way of force closing the writable stream. This means if a raw RPC handler doesn't release its writer lock, then the stream cannot be closed.
We can however extend the getWriter method such that we keep a reference to any writer that was acquired, and then explicitly release the lock and then close the stream. I'm not sure if this is a good idea, and primarily would only be used to force close the streams even in the case where the raw handlers are written incorrectly or they are still running.

@tegefaulkes here's a demonstration of how the locking works in the presence of derived streams:
import { ReadableStream, WritableStream, TransformStream } from 'stream/web';

const inputStream = new ReadableStream({
  start(controller) {
    controller.enqueue('Hello ');
    controller.enqueue('World!');
    controller.close();
  }
});

const transformStream = new TransformStream({
  transform(chunk, controller) {
    controller.enqueue(chunk.toUpperCase());
  }
});

console.log('Input Stream is locked', inputStream.locked); // false

const outputStream = inputStream.pipeThrough(transformStream);

console.log('Input Stream is locked', inputStream.locked); // true
console.log('Output Stream is locked', outputStream.locked); // false

const reader = outputStream.getReader();

console.log('Input Stream is locked', inputStream.locked); // true
console.log('Output Stream is locked', outputStream.locked); // true

reader.releaseLock();

console.log('Input Stream is locked', inputStream.locked); // true
console.log('Output Stream is locked', outputStream.locked); // false

Here you can see that as soon as the input stream is composed, it becomes locked. However the output stream is not locked, not until I get a reader. At that point both input and output streams are locked.
Then if I release the reader, the input stream is still locked, while the output stream is now unlocked.
This is actually quite important, it means that even if say the raw RPC handler ends up releasing the readers/writers, the original QUICStream will still end up being locked, which could prevent garbage collection here if I'm trying to see if the stream is closed yet.
There's one issue though, it appears "cancelling" the reader does not actually release the reader. Note that readers can only be cancelled and not closed, but cancelling is the equivalent of closing the reader. This technically means you still have to call reader.releaseLock() to ensure that the reader is no longer locked.
With this example:
import { ReadableStream, WritableStream, TransformStream } from 'stream/web';

async function main () {

  const inputStream = new ReadableStream({
    start(controller) {
      controller.enqueue('Hello ');
      controller.enqueue('World!');
      controller.close();
    },
    cancel: (reason) => {
      console.log('INPUT STREAM CANCELLED');
    }
  });

  const transformStream = new TransformStream({
    transform(chunk, controller) {
      controller.enqueue(chunk.toUpperCase());
    }
  });

  console.log('Input Stream is locked', inputStream.locked); // false

  const outputStream = inputStream.pipeThrough(
    transformStream,
    {
      preventClose: false,
      preventAbort: false,
      preventCancel: false
    }
  );

  console.log('Input Stream is locked', inputStream.locked); // true
  console.log('Output Stream is locked', outputStream.locked); // false

  const reader = outputStream.getReader();

  console.log('Input Stream is locked', inputStream.locked); // true
  console.log('Output Stream is locked', outputStream.locked); // true

  await reader.cancel();
  await reader.closed;
  reader.releaseLock();

  console.log('Input Stream is locked', inputStream.locked); // true
  console.log('Output Stream is locked', outputStream.locked); // false

}

void main();

Notice that the input stream is still locked even when the output stream's reader is cancelled, closed and released...
This breaks our assumptions for how to destroy the QUICStream, if the input stream is always locked due to composition even if the end result is that is is closed... then we can't really rely on the locked boolean to know whether we can proceed to closing or not.

I've tried a few things. You cannot call inputStream.cancel on a stream that is already locked. And in this case the inputStream stays locked when it is piped to the output stream.
Basically this means once a stream is composed, it stays locked forever even if the output stream is unlocked.
So our problem now is that with respect to QUICStream.destroy, there's no way for the QUICStream to know whether the resulting derived stream is locked or not. Because its own readable and writable will basically be locked forever because it is assumed it would be composed.

Now with respect to QUICStream.destroy there's a choice we have to make here.
If the streams are locked, but the this._recvClosed or this._sendClosed is not true, then we can either:

Ignore them, since they are locked, we assume owership/control of the lifecycle is another place's responsibility, then we just delete the stream and assume they don't exist.
Throw an error, if they were locked and closed, that's ok, no error, if they are not locked and we can close it, that's ok, no error, but if they are locked AND not closed, that means they are in-use and the other system hasn't shut them all down, thus we must prevent our own destruction.

I think I'm going with 2., in the sense that the streams are locked (responsibility has shifted) but also they aren't closed. I'll adapt the ErrorQUICStreamLocked erro and change it to ErrorQUICStreamLockedAndNotClosed.

So now the QUICStream.destroy works like this:
  /**
   * Explicit destruction of the stream
   */
  public async destroy() {
    this.logger.info(`Destroy ${this.constructor.name}`);
    // If the streams are locked, this means they are in-use
    // or they have been composed with `pipeThrough` or `pipeTo`.
    // At this point the management of their lifecycle is no longer
    // `QUICStream`'s responsibility.
    // If they not already closed, we cannot proceed with destroying
    // this `QUICStream`.
    if (
      (this.readable.locked && !this._recvClosed)
      ||
      (this.writable.locked && !this._sendClosed)
    ) {
      // The `QUICConnection` may then result in a partial destruction,
      // some of its `QUICStream` may be destroyed
      // This means the destruction may need to be retried.
      // However a proper destruction should destroy the users of
      // the `QUICStream` first before destroying the `QUICConnection`.
      throw new errors.ErrorQUICStreamLockedAndActive();
    }
    // If the streams are not locked, and they haven't been closed yet,
    // we can close them here.
    if (!this.readable.locked && !this._recvClosed) {
      this.readable.cancel();
    }
    if (!this.writable.locked && !this._sendClosed) {
      await this.writable.close();
    }
    this.streamMap.delete(this.streamId);
    this.dispatchEvent(new events.QUICStreamDestroyEvent());
    this.logger.info(`Destroyed ${this.constructor.name}`);
  }

Which means during the PK destruction it's important to ensure that the RPC client and RPC server is destroyed first before destroying the QUIC socket.
And also means NodeConnection would have to encompass the RPC destruction first.
Also the RPCServer must then attempt to destroy its streams if they are still there if RPCServer.destroy is called.
The only possibility is with a raw handler that takes the RWP and forgets to close things.
It does also appear that you must call releaseLock even after calling cancel or close. Otherwise the locks won't be released. Using raw stream handlers is therefore dangerous as it can leak to stream leaks.

One other thing, the for await on the readable stream does not auto cancel the readable stream. This make sense, since there could be more data on the stream arriving later. It does however auto release the internal reader.

Another interesting thing is that if you do controller.close() on a readable stream, the readable stream can no longer be cancelled because it is already closed.

Ah I see, so if a stream is locked because it is composed then we have no control over it anymore. We can't force close it in anyway. Not from the outside at least. That does mean it's up to the final part of the stream chain to handle the life-cycle. of the streams.
This is not ideal IMO since we do in the worst case need to force close the streams to avoid hanging the program. The handlers to take an abort signal but it's advisory, it can just ignore any signal and keep running. Or the handler can be badly implemented and never consume the reader or write thus just hanging.
It may still be possible to force close a stream by using a transformation stream. The transformation controller has the following interface
    interface TransformStreamDefaultController<O = any> {
        readonly desiredSize: number | null;
        enqueue(chunk?: O): void;
        error(reason?: any): void; // Signals to both the readable and writable side that an error has occurred while processing the transform data, causing both sides to be abruptly closed.
        terminate(): void; // Closes the readable side of the transport and causes the writable side to be abruptly closed with an error.
    }

Either the error() or terminate() method will cause the stream to end early. It can be called at any time. This may be what we need. I'll prototype it to see how it works.

OK I've tested it. Using a  transform stream this way it is possible to force close the stream. If we use error(reason) it closes the stream and any attempt to write to or read the stream results in the reason being thrown. Using terminate() will cause the stream to close, the reading side to end early without an error and the writer to throw TypeError: Invalid state: TransformStream has been terminated. I think this is a reasonably way to force close the stream if needed.

If it is possible to refer to the methods of the controller to force close the stream, then this can be done at the QUICStream level not at some arbitrary point in the transformation stack.
I can copy over the methods of the readable and writable controller to instance properties of QUICStream.

I've managed to have a working QUICSocket that doesn't die upon HTTP3 queries.

If it is possible to refer to the methods of the controller to force close the stream, then this can be done at the QUICStream level not at some arbitrary point in the transformation stack.
I can copy over the methods of the readable and writable controller to instance properties of QUICStream.

@tegefaulkes to make this happen on QUICStream, there is difference between readable and writable.
On the readable stream we have controller.error and controller.close.
On the writable stream we only have controller.error.
I think if we are forcing a shutdown and thus breaking the downstream ownership, we have to use controller.error for both ends.
I have to extract the logic that shuts down the quic stream and then error out on the QUICStream readable and writable objects.

Ok the QUICStream now has:

read and write - the read converts an external push into an internal pull via boolean-skips, the write converts an internal push to an external pull via promise plugs
streamRecv and streamSend this is the actual business logic to interact with the quic streams and web streams, both now handle RESET_STREAM and STOP_SENDING frame error codes
closeRecv and closeSend this is where closing logic occurs, they can close by removing state, but if there is an "error", they will actually close by sending an error by doing a stream shutdown, so basically if a shutdown is needed, that means an error is required as this usually means a shutdown is occurring without completing regular stream request/response cycle, but if there's no error, then likely the stream is already closed and we just need to do QUICStream cleanup
destroy - now has the force boolean, however we are unlikely to use this much, we would want to PK to gracefully close the RPC server and have properly implemented RPC handlers, then to destroy the QUICSocket, but I suppose if a regular SIGTERM doesn't complete, then we could do a little delay before finally sending a force signal, but that's a PK application design issue

@tegefaulkes the QUICStream is pretty good structure right now.
When you integrate this into the RPC system, you need the rpc/utils to have reasonToCode and codeToReason mappers (for unexpected rpc handler and client errors).
The websocket/TCP socket will already have an event interface/node stream, that means its backpressure mechanism may be simpler than what we are doing with QUIC stream. You need to check the way they do backpressure.
Finally to get hole punching in, this could be at just the UDP level before QUICConnection, but it could also be done at the QUICConnection with dgram, but using useless random dcid and scid at the beginning.

The QUICServer creates a new QUICConnection on a method newConnection.
The QUICClient by comparison would create a new QUICConnection at the beginning in its own creation.
The QUICServer returns this back to the caller, which in this case is the QUICSocket, it then decides to call recv and send on it.
When the QUICClient is creating, it would create a new QUICConnection, and upon creation, it would be set in the socket's connection map.
This technically means we have no need of using registerClient, since connections would be part of the connection map already.
However what about lifecycle issues?
Well a client or server could encapsulate the socket is the socket is not shared. Thus one only destroys the client/server.
However if the socket is shared, then this could mean that socket is what manages the lifecycle of clients and servers. However I think this should instead turn into a sort of cross reference. That is the socket cannot be destroyed if there are still clients and servers running.
It is the user's responsibility to then shutdown their clients and server before shutting down their shared socket.
Like a foreign key with restrict constraint.

The QUICClient can also destroyed by a failure in the QUICConnection. Remember that recv, send and timeout can cause the connection to be destroyed. Unlike QUICServer, if the QUICConnection is destroyed, the client must also be destroyed. In that sense, the QUICClient seems mostly a factory for creating the QUICConnection, the QUICSocket primarily just keeps track of it in the connection map.
The QUICClient may then expose the ability to start streams, which is different from the QUICConnection usage on the server side, which is primarily receiving stream events.
However it should make sense that it is possible to start new streams even from the server side. So I imagine that each connection object must have the ability to start a stream.

Considering that the server side, we expose the connection object through the connection event, it should mean that QUICClient should just expose the connection property directly to the end user so they can use it create streams if necessary.
We may want to hide the internal methods though. This could be done with the @internal doc flag.

This would then enable on the RPC server to be able to call RPC methods back to the caller much more easily. The server is just starting a stream back to the client.

For the QUICConnection.streamNew to be implemented we need a way of creating the new stream ID.
As we discovered before the only way to start a stream is to use stream_send but with a unique stream ID.
However I don't see anyway to prevent stream IDs from conflicting from streams being received.
I checked the example code, and I can see that for the HTTP3 example, it uses the src/h3/mod.rs and in the send_request function it uses:
let stream_id = self.next_request_stream_id;

It later does something like:
self.next_request_stream_id = self.next_request_stream_id.checked_add(4).ok_or(Error::IdError)?;

The send_request call ends up returning the new stream_id.
It adds 4 to it because of this commit message:

Instead of increasing the next available stream ID counters, wait until data (e.g. HEADERS or some control data) has been actually written to the stream, this way we avoid burning through a bunch of stream IDs in case of errors (e.g. stream blocked, or insufficient stream credits).
This also fixes a potential integer overflow when 4 is added to the current stream ID (the if would only check that the previous ID was below std::u64::MAX, but the result of the sum could go over the limit), though it's unlikely anyone can actually hit that limit.

So instead of just creating a new stream ID, a stream ID is used and some data is sent over (in the case of HTTP3, these are the headers) and only if this succeeds, do we end up incrementing the stream ID.
Now again there's no where in the code that indicates whether it's possible for the stream ID received may overlap with stream IDs that we may create here.
I think this is because from the client-perspective, there's no expectation of streams being created as an event, so any streams created would be of the client's volition.
But if we want to be able to create streams on the server side back to the client, then we would want to avoid to overlap with the stream IDs that may be received from the client.

This makes me wonder if it is even possible for the QUICConnection when accepted is capable of sending on a stream ID that is not already existing.

Turns out there's a pattern/rules to stream ID generation described here: https://datatracker.ietf.org/doc/html/draft-ietf-quic-transport-25#section-2.1

Therefore in terms of bits:
00 - client initiated, bidirectional
10 - server initiated, bidirectional
01 - client initiated, unidirectional
11 - server initiated, unidirectional

Within each type, the streams are then just numerically increased. It has to be increased by 1, if you use a stream ID out of order, it results in all the lower numbered stream IDs also opened. But I think this explains why it adds by 4 on the client side.
This means there's no chance of overlap for the stream ID for server initiated streams.
However this also means when we create a new stream, we need to give it a type... well the type would depend on whether the QUICConnection was created via an accept (server-initiated) or via a connect (client-initiated).
Right now all of our streams will be bidirectional so we will ignore the unidirectional form.

The initial client stream is supposed to start as 0. Then the next client stream is 0+4.
4 is 100. This preserves the 00 indicating a client initiated bidirectional stream.
We just keep adding 4 to get the next stream.
00
100 - 4
101
110
111
1000 - 8
1001
1010
1011
1100 - 12

The initial server bidirectional stream starts at 1 which is 01.
Then we add 4 as usual.
10 - 1
11
100
101
110 - 5
111
1000
1001
1010 - 9

So QUICStream.streamNew just needs to know how the stream was created and use 0 or 1 and then increment by 4 to get new stream IDs.

Note that you must release the lock of the writer even if the writer has an error:
import { ReadableStream, WritableStream, TransformStream } from 'stream/web';

async function main () {
  const writableStream = new WritableStream({
    write: async (chunk: Uint8Array) => {
      throw new Error('fail');
    },
  });
  const writer = writableStream.getWriter();
  try {
    await writer.write(new Uint8Array(0));
  } catch (e) {
    console.log(writableStream.locked);
    writer.releaseLock();
    console.log(writableStream.locked);
  }
}

void main();

@tegefaulkes this will be important for any raw handlers and other places where you are acquiring the writer/readers. Always release the lock regardless of error.

Due to the last condition that opening stream IDs out of order will result in lower streams also being opened it is necessary that streamNew calls are seralised. We need to have a lock here to ensure that each call to streamNew is blocking each other.

Done with async-locks, adding in a streamIdLock. We don't have DB transactions here, so it's not attached to a transaction.
While bidi and uni can be in parallel, we don't support uni streams yet, since that would have to create a different kind of QUICStream and right now QUICStream is always ReadableWritablePair.
I'm not sure if we would ever support unidirectional streams yet..., every RPC call involve bidirectionality so we don't ever need unidirectional streams for our usecase in PK.

We can now proceed with:

[ ] Complete the QUICClient with the shared socket QUICSocket. - 3 day
[ ] Test the multiplexing/demultipexing of the UDP socket with multiple QUICClient and a single QUICServer. - 1 day
[ ] Test the error handling of the QUIC stream, and life cycle and destruction routines. - 1 day

In the QUICClient.createQUICClient, we will also do QUICConnection.connectQUICConnection.
Now this doesn't immediately create a stream, because we don't need to.
But in order to actually "start" the connection, it's not enough to just create the connection object, because no data is actually being sent on the socket.
It also makes sense to actually start the packet/frame sending so that some handshake can actually be completed. This means triggering the QUICConnection.send.
I believe upon doing so, that "ignites" the interaction loop. So we just need to do at least one await connection.send(). It will trigger the socket to send UDP datagrams, and any received datagrams will then be routed to the connection.
We also need add event handlers to the connection on failure and correspondingly destroy the QUICClient too.

@tegefaulkes I've just assigned you to this issue which you can work on after the websocket transport has been merged.
There's no PR for this, as I've just been pushing WIP commits to master. We can continue to do this until we squash everything into 1 commit, or slice up several different sections of the repository.

In terms of tasks to be done @tegefaulkes, the major ones are already documented above in the issue task list.
To be more precise, I was in the middle of working on QUICClient.
This class wraps around the QUICConnection. Take note that for P2P purposes, both the server and client can share the same QUICSocket instance. This is what enables us to do hole punching.
Thus these are the immediate todos: https://github.com/MatrixAI/js-quic/issues/1#issuecomment-1413231540 before going into the rest of the tasks in the issue.
So in the QUICClient it has to perform QUICConnection.connectQUICConnection, which is one of the smart constructors of QUICConnection. Which is opposed to QUICConnection.acceptQUICConnection.
Once a connection object is created, it doesn't mean that any data has actually been sent. Because it is UDP at the bottom, the thing is connection-less. There's no state maintained until you send some data. This is why I proposed https://github.com/MatrixAI/js-quic/issues/1#issuecomment-1414692561 where we have to do a quicConnection.send() in order to start the process. But I'm not sure if this is supposed to send an empty frame first or if that's meant to be done on the stream.
Another thing is that on the client-side, there's 3 layers of abstraction:

QUICClient
QUICConnection
QUICStream

It's 1 to 1 for QUICClient and QUICConnection. This means we don't replicate all the methods from the QUICConnection to the QUICClient. Instead we can just expose the QUICConnection as a connection property. That basically means that the client is just a thin wrapper that deals with all the bootstrapping and teardown logic.
So you'd do quicClient.connection to deal with the connection directly. Subsequently when you actually want to communicate something, it all starts with streams. So you have to create a new stream from your connection.
The lifecycle of connections is managed by the socket. The lifecycle of streams is managed by the connection. So the same socket could have multiple client connections, and multiple server-side connections. While each connection may have multiple streams.
We still haven't implemented the dgram features of quic connections. This would be added as methods to the QUICConnection. This should be explored in the context of hole punching, but we need to experiment with this to see if it is relevant.
Important regarding errors: https://github.com/MatrixAI/js-quic/issues/1#issuecomment-1406002244
There are "connection errors" and there are "stream errors" these are independent of each other. Connection errors can be protocol-level errors, or application errors, it's possible to distinguish with a special code. I think that's the case with stream errors too.
To work on this you actually need a few pieces of documentation:

The QUIC RFC spec is essential to understand the flow
The quiche rust library docs is necessary to understand what the underlying library is doing

To run and build this system. Use nix-shell, then npm run napi-build. This will build the rust library. Then we also have an example server in src/bin/server.ts. This is executed with npm run server. We don't have an example client yet, but we have been using the quiche's own client to test against the server. But now we are building our own client with QUICClient, so you shouldn't need to do that anymore.
 PK isn't a PKI yet... as we don't have secret "generation" but we can bring this into the future, so we don't need to use step-cli.

In order to make it easier for me to review and coordinate with you @tegefaulkes you should start a new PR for any changes on master, so we can see. I should be getting my new laptop on Tuesday (Sydney time), so I can doing some actual work on it.

In order to test, we need to have a tests/utils.ts that exports:
export {
  generateKey,
  sign,
  verify,
  randomBytes
};

I'm using webcrypto as it is native to nodejs and portable to test which doesn't require external libraries.
It does end up being a bit inefficient due to the import/export between CryptoKey and ArrayBuffer.
But in production in PK we use libsodium so this is fine.
Note that js-db still uses node-forge in its testing which we shouldn't be using anymore since it is super slow.