Proposal: Go streaming API

#3 lays out some ideas for how Twirp could support streaming RPCs on the wire. Equally important is how we support them in generated code.
This issue is for designing the generated Go code. When we've landed on a proposal that seems pretty good, I'll edit this top text with what we've got, and then we can look at implementing it.
Let's use this service to discuss what things would look like:
syntax = "proto3";

service Streamer {
  rpc Transaction(Req) returns (Resp);
  rpc Upload(stream Req) returns (Resp);
  rpc Download(Req) returns (stream Resp);
  rpc Bidirectional(stream Req) returns (stream Resp);
}

message Req {}
message Resp {}

A few weeks ago I took spenczar's experimental work on the v6_streams_alpha branch and have gotten a protobuf proof-of-concept up and running with a streaming download endpoint. Based on my experience, I have a few suggestions to discuss and points to bring out, but first the codes:
https://github.com/twitchtv/twirp/pull/110/files
Main takeaways from my foray:

Twirp streams are/will be awesome — with proper tuning, a twirp stream client is able to receive more than a million Hats per second on my 2013 MacBook Pro (caveat: see the flushing notes at the end of this comment)

How protobuf errors could work (and are implemented in my PR):

If the server's RPC implementation returns an error directly (instead of a RespStream) a standard JSON-encoded twirp error is returned and the HTTP status code gets set accordingly (internally, uses the same writeError function as unary RPCs). This makes things easy from a client implementation point-of-view: any non-200 status code is a JSON-encoded twirp error.
If the server's RPC implementation returns a RespStream…

The HTTP status code is always 200 and Content-Type is application/protobuf
Errors from the RespStream's Next function are encoded as the "trailer" field of the stream: trailer-field-tag + length-delimited-protobuf-string

If Next returns an io.EOF error, the trailer string is just "EOF"
For any other error, the trailer's protobuf string is a JSON-encoded twirp error

Channel-based stream interface helper

The initial proposal of using a chan *Resp doesn't provide any way to handle mid-stream errors. In rigging up my own RespStream, I wrote myself into an annoying anti-pattern using a chan *Resp and a separate chan error — code comments like "// resps channel closes before error channel, so calling Next here to let errs signal the stream's completion" were indicatave of some deep problems with a two-stream approach. Thus I've got another proposed implementation:

Use a chan *RespOrError where RespOrError looks like type RespOrError struct{resp *Resp, err error} to encapsulate Next's two return values in a single channel.
Without a good implementation in the generated code, devs implementing their own twirp stream endpoints are likely end up in territory similar to my two-channel antipattern, particularly when trying to push data to the RespStream, as the Next interface is a pulling pattern. To me, chan *RespOrError heads in the direction of providing a clean push interface to RespStreams.
Relevant sections in the PR:

example/cmd/server/main.go
example/cmd/server/hat_stream_sender.go

And now the trickiest point for last: flushing the response writer. To reliably get messages over the wire as they come in, http.Flusher's Flush function has to be used to force the sending of a packet. My naive implementation is flushing after every message (provided the response writer implements the Flusher interface), but in high-load scenarios this dramatically reduces throughput — flushing every message gives <10% throughput in my initial tests. The sweet-spot really depends on the use-case, and coming up with a good solution for when to flush the response writer is going to be tricky!
There have been some grpc discussions (here and here) about flushing, it sounds like both grpc and go's http2 server will flush "only when a packet is full or there's nothing else to send", but I haven't yet tracked down where this flushing functionality is implemented — it would be great to see both to compare/contrast. You can see that some users want to opt out of the batching or take direct control of the flushing, but grpc doesn't provide the hooks and adding them would be difficult architecturally — food for thought!
And those are my thoughts 🖖

@fritzherald, thank you so much for this dive! Your hands-on experience here is hugely valuable.
First, I think chan *RespOrError seems like a solid advance. I have nitpicks - maybe it should be chan RespOrError? - but the core idea of a single channel which returns a union-type-ish value seems solid. I'm especially convinced by the trickiness of maintaining two channels. Let's do what you suggest.
Your proposals on errors and the trailer format seem equally good, to me.
Flushing is hard, for sure. I think that grpc-go#1242 presents the hard part most clearly, as there's a tradeoff between latency and throughput here: if you don't flush on every message, then you're necessarily letting messages sit in a buffer for a bit. But if you do flush on every message, then you're using your TCP connection very inefficiently, especially for small messages.
Adding a smart TCP scheduler to Twirp to handle this seems very difficult and complex. I am pretty sure we shouldn't do that. Instead, we should try to find a way to let picky users choose their flushing behavior, while taking a middle route as the default.
Hypothetically, users can control things by providing their own http.ResponseWriters to Twirp servers which flush after every call to Write. For example:
func MinimizeLatency(h http.Handler) http.Handler {
  return http.HandlerFunc(w http.ResponseWriter, r *http.Request) {
    if flusher, ok := w.(responseFlusher); ok {
      w = lowLatencyResponseWriter{base: flusher}
    }
    h(w, r)
  }
}

type responseFlusher interface {
  http.ResponseWriter
  http.Flusher
}

type lowLatencyResponseWriter struct {
  base responseFlusher
}

func (w lowLatencyResponseWriter) Write(p []byte) (int, error) {
  n, err := w.base.Write(p)
  if err != nil {
    return n, err
  }
  w.base.Flush()
  return n, nil
}

func (w lowLatencyResponseWriter) Flush() { w.base.Flush() }

This could be used as middleware like so:
server := MinimizeLatency(NewHaberdasherServer(app, hook))
http.ListenAndServe(":8080", server)

Obviously, it's easy to think of more complex variations. A ResponseWriter could count some number of bytes and then flush, or guarantee that it never holds a message for longer than a millisecond, etc. We can promise that we won't call the ResponseWriter's Write method from multiple goroutines to make this more straightforward.

This control-flushing-via-response-writer strategy would work, I think, but it's definitely a little complex. I think that's okay, if we can get enough simplicity for the common case for people that most people don't have to bother with making their own ResponseWriter. Ideally, only people with very special needs will mess with this.
Would a default strategy of "never call Flush(), just expect net/http to handle this for us" be good enough for most users?  We should investigate that.

@spenczar the control-flushing-via-response-writer strategy seems like a great approach for enabling customized flushing strategies. As for "never call Flush()", I just ran a quick test and the results indicate there would probably need to be some flushing strategy provided by twirp:
Test with no flushing
Configuration: Using the Repeat rpc defined here, the client requests a stream of responses with a 100ms delay between each response:
// error handling omitted for clarity
startedAt = time.Now()
repeatRespStream, err := client.Repeat(req)
log.Printf("Received response stream after %v\n", startedAt)
for {
	repeatResp, err := repeatRespStream.Next(context.Background())
	log.Printf("Received resp after %v\n", startedAt)
}

Results:

First ~12 seconds: client waits for Repeat to return
@ 13 seconds: Repeat returns the RepeatRespStream
@ 26 seconds: Next returns for the first time, followed immediately by another 126 messages
@ 39 seconds: messages 128 through 252 arrive
@ 51 seconds: messages 253 through 376 arrive
...

Increasing the delay 10x to one second results in a ~10x multiplier to all of the times above: i.e. ~130 seconds before the rpc call returns the RespStream.

I'm having trouble imagining how Twirp could provide a default flushing strategy that knows to get out of the way of a custom writer without some sort of option flag to disable the built-in flusher. An alternative could be for the twirp package to define one or multiple handlers like your MinimizeLatency above, but it would mean that users who aren't aware of the need for a flushing handler could easily be confused by clients that are hanging on the initial rpc call while the server's Next implementation is happily chugging away.
I guess another possibility could be to add a FlushHandler field to the ServerHooks struct and for twirp to rig up the handler internally, using a default implementation if no handler is provided by the user.

Okay, that experiment makes it clear that we need to have a default flushing strategy, and we need to let users override the default. I think adding this to ServerHooks would be a mistake, as it really messes with the purpose of ServerHooks.
The natural thing here, if we had nothing in place already, would be to accept the flushing strategy as a parameter when you construct the server handler (in NewHaberdasherServer, in other words). I think it would probably be done through a config struct, like this:
type ServerConfig struct {
  Hooks *ServerHooks
  Flush *FlushStrategy
}

type FlushStrategy struct {
  // If at least MaxLatency time has elapsed since the last flush, and unflushed bytes
  // are available, flush.
  MaxLatency       time.Duration
  // If at least ByteThreshold bytes have been written since the last flush, flush.
  ByteThreshold    int
  // If at least MessageThreshold bytes have been written since the last flush, flush.
  MessageThreshold int
}

// generated code:
func NewHaberdasherServer(svc Haberdasher, conf twirp.ServerConfig) TwirpService

This, I think, feels extremely clear... but it's a compatibility-breaker, once code is regenerated by users. On the other hand, a major version bump is a reasonable time to break compatibility, and we'd be doing a major version bump for streaming anyway, so this might be the right time.
The other downside of switching to a config struct is that it makes it much more appealing to add tons of config junk in. I think one of Twirp's strengths is the how little configuration it takes currently, and if we could keep that, I'd be happier.
Anyway, I'd like to find an alternative, but I think it'll be hard to find something much better than this config struct style.

Oh! I have an important alternative that strikes me right after posting that last comment( of course) which is a Set method on the generated code.
Generated servers could have an additional method, func (h *haberdasherServer) SetFlushStrategy(maxLatency time.Duration, byteThreshold, msgThreshold int). This would maintain compatibility. It's a little odd, but it would work.

SetFlushStrategy seems like a good approach. What would be the process for disabling built-in flushing when using a custom flushing response writer? Related: what do zero values imply? I'm inclined to think they should indicate that the parameter has no effect on flushing, so SetFlushStrategy(time.Millisecond, 0, 0) would imply that byte and msg thresholds are ignored.
I could also see a FlushStrategy config struct as the arg to SetFlushStrategy working well, something like...
type FlushStrategy struct {
	maxLatency time.Duration
	byteThreshold, msgThreshold int
}

// Not strictly necessary, but could be handy to have pre-defined vars for common values:
var (
	FlushStrategyDisabled = FlushStrategy{} // zero val means no flushing
	FlushStrategyDefault = FlushStrategy{/* whatever defaults are settled on */}
	FlushStrategyEveryWrite = FlushStrategy{byteThreshold: 1}
)

Then twirp.FlushStrategyDefault could be used in the generated .twirp.go file as the default if SetFlushStrategy is never called, and calling SetFlushStrategy would look like...
// zeros for byteThreshold and msgThreshold imply no flushing based on bytes/msgs
server.SetFlushStrategy(twirp.FlushStrategy{maxLatency: time.Millisecond})

// or...
server.SetFlushStrategy(twirp.FlushStrategyDisabled) // for use with custom `responseFlusher`-style response writer

// revert to default flushing...
server.SetFlushStrategy(twirp.FlushStrategyDefault)

@fritzherald Nice, I like this a lot!

What would be the process for disabling built-in flushing when using a custom flushing response writer?

I think this would be server.SetFlushStrategy(FlushStrategyDisabled), so that the custom ResponseWriter can have control.

what do zero values imply? I'm inclined to think they should indicate that the parameter has no effect on flushing, so SetFlushStrategy(time.Millisecond, 0, 0) would imply that byte and msg thresholds are ignored.

That's my thinking too.

something like... [use a struct]

Yes, I like this proposal too. An additional advantage is that we could reuse those shapes for describing flushing in clients when they are uploading streams.
Good! I think we're ready to roll on an implementation of this!

One last thing I've been thinking about. I wonder if your chan RespOrError can be used in place of my <Msg>Stream type everywhere.
The generated interface would look like:
type Streamer interface {
  Transaction(context.Context, *Req) returns (*Resp, error)
  Upload(context.Context, <-chan ReqOrError) returns (*Resp, error)
  Download(context.Context, *Req) returns (<-chan RespOrError, error)
  Bidirectional(context.Context, <-chan ReqOrError) returns (<-chan RespOrError, error)
}

This seems like it might be easier for users to wrap their heads around.
I will write up a thorough walk-through of this design, following the pattern in https://github.com/twitchtv/twirp/issues/70#issuecomment-361365458, in the next hour or so.
@sdboyer-stripe hey there 🤓

Okay, I like the channel of pairs. I think it's better.

The generated interface would use <-chan ReqOrError and <-chan RespOrError types:
type Streamer interface {
  Transaction(context.Context, *Req) returns (*Resp, error)
  Upload(context.Context, <-chan ReqOrError) returns (*Resp, error)
  Download(context.Context, *Req) returns (<-chan RespOrError, error)
  Bidirectional(context.Context, <-chan ReqOrError) returns (<-chan RespOrError, error)
}

Since the parameters are specified as <-chans, they are all read-only. This saves us from worrying about users closing the channels or inserting values from the wrong side.
These channeled types are struct pairs, generated for each type:
type <Type>OrError struct {
  Msg *<Type>
  Err error
}

Exactly one of Msg or Err will be non-nil in a valid value of this struct.
As the recipient:
You iterate over the channel. You check whether the error is non-nil; if so, the stream has a problem and won't continue. Otherwise, you can use the Msg field.
When the stream is terminated, the channel will be closed.
To shut down the stream early as the recipient, cancel the context. Otherwise, you must read from the channel until it is closed. Failure to do one of these two things will leave the HTTP connection hanging open.
The channel is unbuffered. If you need to buffer, you can do it yourself by copying into a buffered channel.
As the sender:
It's your responsibility to provide an instance of the channel. The generated Twirp code will be reading from the channel until it is closed, or until the context is canceled, or until a non-nil error value is emitted. If you put in a non-nil error value, you should close the channel right afterwards.
We don't need to provide a constructor for this - it's obvious how you can set up buffering.

For the 4 unidrectional cases:
As a unidirectional client-sender:
You construct a message channel stream, and then you call Upload(ctx, stream) on the generated client.
The generated client code immediately opens an HTTP request to the server. It then iterates over the stream, checking in each loop iteration for context cancelation too. It writes messages to the wire, and for a non-nil error, it writes the error as a trailer and ends the stream.
for {
  var (
    m   *Req
    err error
  )
  select {
  case <-ctx.Done():
    err = twirp.NewError("context canceled", twirp.Canceled)
  case v, ok := <-stream:
    if !ok {
      err = twirp.NewError("stream complete", twirp.NoError)
    }
    if v.Err != nil {
      err = v.Err
    } else {
      m = v.Msg
    }
  }

  if m != nil {
    writeMsgToWire(m)
  }
  if err != nil {
    writeErrToWireAsTrailer(err)
    terminateWireStream()
    break
  }
}

Then, it reads the server's response, parses it, and returns it.
If the server recipient cancels the stream early, then the generated client will immediately return a nil value for *Resp, and an error explaining that the recipient aborted the stream.
For example:
client := NewStreamerClient("http://<IP_ADDRESS>", http.DefaultClient)
ch := make(chan ReqOrError, 5)
go func() {
  for i := 0; i < 100; i++ {
    ch <- ReqOrError{
      *Req: &Req{},
    }
  }
  close(ch)
}()
resp, err := client.Upload(context.Background(), stream)

As a unidirectional server-recipient:
An incoming HTTP request comes in for the Upload API, and we construct a chan ReqOrError to describe the stream. You should read from the channel until it is closed. If the context is canceled, the generated code will close the channel, so you don't need to check for a context cancelation too.
To abort, return a nil *Response and a twirp.Error with a code of twirp.Canceled.
Once you return, the generated HTTP handler will call req.Body.Close() and send your response.
For example:
type *streamerImpl struct {...}

func (s *streamerImpl) Upload(ctx context.Context, s <-chan ReqOrError) (*Resp, error) {
  for v := range s {
    if v.Err != nil {
      // Oh no!
      logCanceledStream(v.Err)
      return nil, twirp.NewError("i give up", twirp.Canceled) // ???
    }

    err = handle(ctx, v.Msg)
    if err != nil {
      return nil, err
    }
  }
  return &Resp{}, nil
}

As a unidirectional server-sender:
An incoming HTTP request triggers a call to Download(ctx, req). Your code should construct a message channel and return it. If you don't like the request you received for some reason, you can return an error and a nil RespStream.
The generated code will read from the stream repeatedly until it is closed, or the context is canceled.
If the client cancels the stream early, then we'll trigger the Error hook on *ServerHooks.
For example:
func (s *streamerImpl) Download(ctx context.Context, req *Req) (RespStream, error) {
  if !valid(req) {
    return nil, errInvalid
  }
  ch := make(chan RespOrError, 10)
  go func () {
    for i := 0; i < 100; i++ {
      ch <- RespOrError{Msg: &Resp{}}
    }
    close(ch)
  }()
  return NewRespStream(ch), nil
}

As a unidirectional client-recipient:
You call Download(ctx, req), receiving a <-chan RespOrError. You should read from it repeatedly until you receive nil, <error>. You can abort by canceling the context.
For example:
func doDownload(handle func(*Resp) error) {
  client := NewStreamerClient("http://<IP_ADDRESS>", http.DefaultClient)
  ctx, cancel := context.WithCancel(context.Background())
  respCh, err := client.Download(ctx, &Req{})
  if err != nil {
    return err
  }

  for v := range respCh {
    if v.Err != nil {
      // Oh no!
      return v.Err
    }
    err = handle(v.Msg)
    if err != nil {
      cancel()
      return err
    }
  }
  return nil
}

Advantages of this design
No special constructors are necessary.
Channels don't require as much documentation as the interface types. Concepts map more cleanly: You close(ch) a channel to close the stream, for example.
User code is a bit simpler, to my eyes.
Generated code will be easier to write (especially in its interactions with context cancelations).
If messages are being transmitted too slowly, senders always have access to a context they can cancel to abort, which is very flexible.
It's obvious how recipients can add buffering on the read side: they copy from the unbuffered channel that comes from generated code into a buffered channel of their own.
Disadvantages of this design
Canceling as the recipient of a stream can be a little clumsy, and is not symmetric: servers return a special error, while clients cancel a context.
We have to demand users set one of Err and Msg in the struct union type-ish thing, but can't enforce this.

Rad!! Love it. I'll try to update my PR next week with an implementation of the <-chan RespOrError API for the Download rpc

I just wanted to say I am using the branch v6_streams_alpha_downloads and it works amazingly thanks! Hope to see this feature soon :)

Just my two cents, which I'm sure is a minority view but alas: I actually really like that Twirp doesn't have streaming. When you deploy streaming capability across a large org, it can result in issues - as streaming is implemented now, it inherently pushes RPC framework concerns to the application level, which results in inconsistent implementations and behavior, which results in lower reliability. It would be great if the Twirp protoc plugins required you to opt-in to streaming with an option, or at the least had an option to explicitly opt-out (i.e. --twirp_out=streaming=false:.) so that the plugin errored if it found a streaming endpoint. But as I said, just my two cents, it's cool to see Twirp continuing to grow.

@peter-edge my understanding is streaming uses a separate API, Twirp doesn't convert all RPC calls into a streaming model underneath, you have to explicitly craft your code to support it, and you can mix endpoints that use stream semantics and others that are regular RPC calls.

That's what I'd expect, I'm saying that one of the really nice things about Twirp (in my opinion) is that it doesn't support streaming - if developers have the option to use streaming, they will use it, and my experience with that at an org-wide level hasn't been the greatest is all. By crafting your code, I think you just mean having stream methods in your Protobuf definition, and I'd prefer if there was an option at protoc invocation time where this would result in an error. It would be pretty easy to add, let me know if I can help.

I'm sorry for the long silence on this topic. Part of the reason I've gone quiet is just that I've been focused on other work, but part of it is that I have always had my doubts about the suitability of streaming in Twirp, and have been grappling with that uncertainty, trying to find a design that maintains simplicity.
Streams are a dangerous feature for users
@peter-edge has expressed my deepest concern extremely well. One of Twirp's best features today is that it is very hard to use it badly. You kind of can't screw things up too much if you hand it to an inexperienced developer as a tool kit. And while you can make mistakes in API design and message design, those mistakes stay local: they largely aren't architectural or infrastructural.
Streams are different. They imply expectations about how connection state is managed. Load balancing them is really hard, which means that a decision to use streams for an API has ramifications that ripple out to the infrastructure of a system. So, streams introduce some architectural risk when recommending Twirp.
How plausible is it that users trip on that pothole? Unfortunately, I think it is quite likely. Streaming is the sort of feature whose downsides can be hard to see at first, but it sounds great at first. "Lower latency and the ability to push data to the client? Sign me up!" But people could easily reach for it too early in order to future-proof their APIs, "just in case we need streams later," and walk themselves into an architectural hole which is very difficult to get out of.
Streams are complex and hard to implement well
In addition, streams add significant complexity to the Twirp project. The alpha branch that implements them has to parse protobuf message binary encoding directly - it can't really lean on the proto library for fiddly details of the encoding, but relies on a few low-level utility functions. This kind of terrifies me. I'm worried about maintaining something like that, for the health of the project.
It also imposes a much, much heavier burden on implementers in other languages. One of the best things about Twirp has been how quick and simple it is to write a generator in new languages. We saw Ruby, JavaScript, Java, and Rust implementations appear in a matter of days of the project being first released! I doubt we would have seen any third party implementations if they needed to do low-level manipulation of byte streams to pull out binary-encoded protobuf tag numbers.
The complexity also translates into a clumsy API. It's difficult to do this in a general way that feels really ergonomic while covering the many ways a streaming connection can break. I am still not thrilled with the Go API we designed, and expect that other language implementations would be just as difficult to get really right.
Streams are required only rarely
This is all risk associated with implementing streams. What is the reward, on the other side?
Frankly, at Twitch we have hundreds of Twirp services, and have not once found a compelling need for streams. I can think of a small number of backend systems that do use streaming communication between a client and server, but all of them have extra requirements which make Twirp unusable: some talk to hardware appliances or stream video data, which cannot be represented in Protobuf in any reasonable way. Some have extreme performance requirements which Twirp doesn't aim to meet. None of our designs would work for those niche applications.
Meanwhile, most simple streaming RPCs can be implemented in terms of request-response RPCs, so long as they dosn't have extreme demands on latency or number of open connections (although HTTP/2 request multiplexing mostly resolves those issues anyway!). Pagination and polling are the obvious tools here, but even algorithms like rsync are surprisingly straightforward to implement in a request-response fashion, and probably more efficient than you think if you're using http/2, since the transport is streaming anyway.
Sometimes you really do need streams for your system, like if you are replicating a data stream. Nothing else is really going to work, there. But Twirp does not need to be all things for all use-cases. There is room for specialized solutions, but streams are a special thing, perhaps too specialized for Twirp, and they have resounding architectural impact.
Maybe we shouldn't do streams
Given the above, I worry about adding streams to Twirp. I think it would be a step in the wrong direction.
 I'd like to better understand these use cases. What am I missing in the above analysis?
(if you'd like to reach me in private on this topic, you can also reach me by email at<EMAIL_ADDRESS>I'm happy to discuss)

@spenczar I've got some experience in doing streaming over HTTP/1.1 and there are two ideas I'd like to share. Before I get into it, I do think that for most RPC needs, streaming isn't needed or can be solved in another way.
If streaming is needed, then in most cases it is in the form of server side streaming, such as:
syntax = "proto3";

service Streamer {
  rpc Download(Req) returns (stream Resp);
}

If that is the one use case Twirp should support, then something like the server side of SSE can be used. I'm not certain about the semantics of retries upon reconnects in gRPC. If that's supported, I've seen SSE and HTTP Range headers used together to obtain those kinds of semantics.
The other idea is that websockets could be used to provide semantics for client side or bidirectional streaming. Depending on whether connection retries should be supported or not, some protocol semantics might need to get implemented on top of it, and some form of SSE and range headers might help there.
I personally think I'd rather go down this path than implementing something like an iterator on top of HTTP requests.

@gudmundur makes a really good point in regards to websockets. Websockets are more widely available than http/2, but are just as much of a standard. @spenczar you make some good points against making a streaming protocol at all, but use cases do exist for them. One hypothetical use case would be an email client program, connected to a server program. With streaming, the server can notify the client once a new email is available, rather than the client polling for changes, which would cut down on bandwidth considerably.
(It may be argued that for email this is a solved problem with things like IMAP IDLE, but that comes with it's own set of issues, such as needing an IDLE connection per folder.)
How clunky would it be to make an external library to handle streaming with websockets, layered on top of Twirp? Off the top of my head, it would be able to pass off incoming streaming requests on to Twirp to handle and send any response back through, but outgoing streaming requests would have to be sent directly to the external library, because Twirp has nothing in place for this.

While I think @spenczar and @peter-edge are right about a lot of the reasons to be cautious about introducing streaming into an architecture, I also would gently push back on the idea that streams are a special case that is rarely required. While that may be the case for cloud-based backend deployments, microservice architectures are being adopted across a wide variety of environments with constraints and concerns that can be very different from those of large-scale cloud service architectures. Streaming data and events are ubiquitous in many use-cases, and relying on developers to deconstruct/reconstruct their streams on both ends of the wire strikes me as a recipe for just a different flavor of instability and headache.
My team has been using this thread's implementation of streaming twirp endpoints for almost a year now along with our own Java, Swift, and JavaScript/React-Native client implementations, and I recently sent out a survey internally to get the perspectives of the developers who have been using it. Our experience strongly supports the decision to not adopt this thread's approach for streaming twirps, but it also points to some of the value/necessity of streaming APIs for us. A summary of the big takeaways:

Concerns about the difficulty of implementing streaming clients are well-founded

Bugs around streaming HTTP implementations stuck around and were difficult to corner (e.g. streams going dead without closing the connection after long idle periods). Streaming HTTP requests/responses aren't that widely adopted and support/documentation varies greatly across different languages

Some environments (e.g. JS, Swift) require special handling when protobuf messages get split between response packets, can result in difficult-to-debug edge cases

Protobuf library implementations have varied support for decoding raw protobufs (e.g. not supported in Swift)
Responses to the survey question "What's the worst thing about streaming twirps? What would we miss the least if we had to get rid of them?":

"Quite a few unknowns in terms of malfunctions, memory overhead/leaks, and system resources in general. It's dark in here and I'm scared."
"Rare but challenging stability issues."
"Unexplainable behaviors, unexpected crashes, random disconnections."

Despite those harrowing accounts, half of our developers still think we made the right choice going with twirp streams, and nobody thinks we should abandon them at this point.

Everyone really likes the APIs.

"They've made it easier for us to break up monolithic Go services — what used to be an internal chan-based API became a twirp stream API with minimal impact to the surrounding code."
For non-Go clients, ReactiveX Observables have mapped really well to this thread's Go API and have provided a powerful common interface across multiple languages

"The auto-generated Rx observable clients for streaming endpoints are awesome… paired with redux-observables on the frontend made for some rad hacking"

For the half of developers that think twirp streams were a bad idea, their preference for what we should have used instead is split between "grpc", "websockets", and "unsure"

"Building a protobuf-based api feels right, but part of me wonders if we could have avoided some of the stability issues with a more widely supported and tested approach."

Nobody sees completely abondoning streaming as a viable option

There is, however, a growing approach to avoiding streaming in favor of non-streaming solutions when feasible, largely due to the stability issues noted above
Given that we can't avoid streaming altogether, protofile definitions for our streaming APIs are seen as critical for maintaining sanity

All of this has me now leaning toward a websocket-based approach as @gudmundur and @qskousen have pointed to. There are a whole bunch of specific implementation questions that follow (e.g. Should a service mux multiple RPCs over the same websocket connection? [Probably yes?]), but if a streaming solution is unlikely to be mainlined into this repo then I think the question for this discussion becomes if/how twirp can work nicely with external implementations of streaming RPCs. It's not too hard to imagine an external implementation that wraps/falls-back-to twirp for non-streaming endpoints, similarly conceivable to imagine twirp supporting some sort of plugin interface where it hands off streaming requests to a registered handler. Either way, consistent errors and support for ServerHooks would seem important for a good experience.
That's about as far as I've thought it through, I'm curious to hear what others would hope for in a twirp-compatible streaming implementation. Feel free to message me on Gophers Slack if you'd like to collaborate or want to discuss more nuances off-thread: @mikey 🖖

Closing for inactivity. The more general streaming discussion can continue in #3
New implementations or alternatives can be handled on their own issues or pull-requests.


Trouble Public MOT20 Results

Thanks for sharing your work.
I modified your track.py script to run on the detection files from MOT20-train (public detector protocols). Your code ran fine, but the tracking results seem very poor (combined HOTA = 48.5, combined MOTA = 49.62 on train set of MOT20, Evaluate with TrackEval). In general I would like to use your code with a MOT-style det.txt file, but I am concerned that it is not currently working right.
added code :
detections = []
for _,li in enumerate(lines):
      new = [float(x) for x in li.split(",")][:6]
      new = new + [1] + [1] + [0]
      new[4] = new[2] + new[4]
      new[5] = new[3] + new[5]
      if int(new[0]) == (frame_id):
          detections.append(new[2:])
detections = np.array(detections)
trackerTimer.tic()
online_targets = tracker.update(detections, img_info["raw_img"])
trackerTimer.toc()

I really appreciate it if you answer these questions. Thanks very much.

I am using the same detector and pass the detections to both StrongSORT and BoTSORT.
I see no big difference between StrongSORT and BoTSORT with default  parameters.
https://github.com/NirAharon/BoT-SORT/issues/42

Hi @Mr-Akbari, thank you for trying this.
I will try to run BoT-SORT and BoT-SORT-ReID with the public detections, and get back to you with an answer.

Thank you very much.

Hi @Mr-Akbari, thank you for trying this. I will try to run BoT-SORT and BoT-SORT-ReID with the public detections, and get back to you with an answer.

Hi @NirAharon, did you find an answer?


⚠ Bofrost.be has degraded performance

In d1471e2, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1128 ms

Resolved: Bofrost.be performance has improved in ba2b040.


IndexError: list index out of range,怎么解决，求指点

Traceback (most recent call last):
File "main.py", line 157, in
main()
File "main.py", line 74, in main
train_iter, dev_iter, test_iter, alphabet, alphabet_static = load_data(config=config)
File "/home/zgy/wll/tibetan_Segand-POS-Tag/DataUtils/mainHelp.py", line 242, in load_data
train_iter, dev_iter, test_iter, alphabet, alphabet_static = preprocessing(config)
File "/home/zgy/wll/tibetan_Segand-POS-Tag/DataUtils/mainHelp.py", line 168, in preprocessing
train_iter, dev_iter, test_iter = create_iter.createIterator()
File "/home/zgy/wll/tibetan_Segand-POS-Tag/DataUtils/Batch_Iterator.py", line 84, in createIterator
self._convert_word2id(self.data[id_data], self.operator, self.operator_static)
File "/home/zgy/wll/tibetan_Segand-POS-Tag/DataUtils/Batch_Iterator.py", line 112, in _convert_word2id
pos = inst.pos[index]
IndexError: list index out of range

Hi,
Too little information to see anything, debug carefully by yourself, maybe it is data problem.
Best wishes~


BinaryReader ints

78187493520 turns into 878082192

maybe fixed already?

This behavior was still broken, but has been fixed by #277.  Closing.


Simple subclasses of FowardingBakedModel will always be unwrapped during item rendering

NeoForge patches the ItemRenderer to allow replacing the model based on transforms: https://github.com/neoforged/NeoForge/blob/1.21.1/patches/net/minecraft/client/renderer/entity/ItemRenderer.java.patch#L17 (Note that Forgified Indigo's ItemRenderer hook comes after this patch)
The default implementation of IBakedModelExtension#applyTransform returns itself as the target model: https://github.com/neoforged/NeoForge/blob/1.21.1/src/main/java/net/neoforged/neoforge/client/extensions/IBakedModelExtension.java#L67
Forgified Fabric API's ForwardingBakedModel forwards that result: https://github.com/Sinytra/ForgifiedFabricAPI/blob/1.21.1/fabric-renderer-api-v1/src/client/java/net/fabricmc/fabric/api/renderer/v1/model/ForwardingBakedModel.java#L120-L123, meaning the final model used by the renderer will be the wrapped model of the ForwardingBakedModel instead of the ForwardingBakedModel itself.
This breaks Continuity's emissive item model textures.

Replacing Forgified's ForwardingBakedModel#applyTransform implementation with the following code will fix the Continuity issue in the common case where the wrapped model returns itself, but will still break with custom NeoForge models that change that behavior.
BakedModel result = wrapped.applyTransform(transformType, poseStack, applyLeftHandTransform);

if (result == wrapped) {
    return this;
}

return result;

Fixing the edge case likely requires duplicating this via unsafe and swapping out the value of the wrapped field for result.


This is might a bit off-topic, but let's discuss the transition of Open XML SDK to Rust

This is might a bit off-topic, but let's discuss the transition of Open XML SDK to Rust
Rust became a good enough language for development and backend services. And It is fast.
So, I put forward the idea of transitioning Open XML SDK to Rust.
There might be pros and cons, but I'd like to hear the community and contributors' responses on that.

why? there's no way we would "transition" to rust - that implies getting rid of the .net version. Feel free to start a project to enable it in rust :)

Maybe a little outdated, but recently I made some experiments called ooxmlsdk.
Similar to this project it takes data json for codegen. currently xml parsing/writing is ok, still no parts, validation, particle and of course no linq.
Feel free to check it if anyone is interested.


Graph subsystem optimizations and refactor

This refactor makes it more efficient to serialize and store graph nodes as well as the splitting of the BinaryNode interface into BinaryNode and HasBinaryChildren interfaces which simplifies leaf node implementations.
The storage efficiency comes primarily from eliminating unused class properties.

Unfortunately, since some class properties of various leaf nodes have been removed, the classes will now hash to a different revision number than ones that were saved with a previous version of the library. For this reason, I'm targetting this at the 2.1 release, rather than the current bugfix release as to not "shock" people that aren't expecting their models not to work after upgrading.

Thanks for the review @DrDub! I added some more tests and fixed a missing exception class import.


Markdown and Python3

With python2, markdown errors are processed and corrected as described in https://github.com/ableev/Zabbix-in-Telegram/wiki/Markdown-and-HTML
In log:
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Original message has been fixed due to https://github.com/ableev/Zabbix-in-Telegram/wiki/Markdown-and-HTML

but with python3 we can see cyclic error:
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
zbxtg_group.py: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 85
...

@XaTTa6bl4 are you using emojis? If yes - I probably can fix it. If not - please, provide input and output, not just errors.

Yes, I'm using emojis. Example:
/usr/lib/zabbix/alertscripts/zbxtg_group.py 'TGroup' '{{OK}} RECOVER: *Unavailable by ICMP ping*, IP:<IP_ADDRESS>' '_SwitchName_12_ zbxtg;markdown'

@XaTTa6bl4 thanks.
Let's hope that users won't be using more than one emoji.
Dirty hack applied.
Please, test it.


WIP: Create PR on main website when SPEC repo is updated

See https://github.com/scientific-python/scientific-python.org/issues/408
The purpose of this PR is to autogenerate PRs for https://github.com/scientific-python/scientific-python.org to update the SPEC submodule when changes appear on main. I am trying to follow the instructions found here:

https://github.com/peter-evans/create-pull-request/blob/main/docs/concepts-guidelines.md#creating-pull-requests-in-a-remote-repository

It says you need a "a repo scoped Personal Access Token (PAT) ," but I am using a "Fine-grained personal access tokens" with " Read and Write access to pull requests" to "scientific-python/scientific-python.org".
I haven't used peter-evans/create-pull-request before and would love detailed feedback or suggestions on this.

I wonder whether dependabot could be used for this, too rather than a new thing? afaik it can handle git submodule updates, too, and we already use it.

I wonder whether dependabot could be used for this, too rather than a new thing? afaik it can handle git submodule updates, too, and we already use it.

I was considering that as well. If we do that I think we may to use a cron job to check whether this repo has been updated. I thought it made more sense to make a PR whenever we updated this repo instead. But I don't have a strong opinion. If we use dependabot on the "scientific-python/scientific-python.org" repo, I suppose we should do it everyday.

Closing as we preferred this approach: https://github.com/scientific-python/scientific-python.org/pull/448


Example

Hi @DonnYep
Thank you so much for awesome Framework, The examples list are very easy to run
When I try to run 08_ObjectPool, I see the null error here, Please check

NullReferenceException: Object reference not set to an instance of an object
Cosmos.Test.MyObjectSpawner+<Start>d__13.MoveNext () (at Assets/Examples/ExampleScripts/08_ObjectPool/MyObjectSpawner.cs:36)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/UnitySynchronizationContext.cs:153)
UnityEngine.UnitySynchronizationContext:ExecuteTasks() (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/UnitySynchronizationContext.cs:107)

Thank you so much!

Hi @codingwatching
Thank you very much for your attention to my framework, this issue will be fixed immediately. If you find any other issues, please give me feedback and I will respond immediately.
Thank you very much.

Hi @DonnYep
Thank you! Have a nice day!

Thank you for the quick update, it works like a charm now!

Hi @codingwatching
Glad to help you and look forward to more feedback from you!
Thank you!


Add possibility to translate "cancel" and "upload" in effects bar

Is your feature request related to a problem? Please describe.
Currently those labels are not translated at all.
Describe the solution you'd like
Translation for those labels.

Thanks! I’ll also browse through the code and see if I missed any other labels.

Found another one! Added these translations:

cancel
upload
close

This is deployed in v2.7.1.


feat(guardian): import pre existing access

@ravisuhag @mabdh @haveiss applied the change here: https://github.com/odpf/guardian/pull/309


Warning message after first logging in - User.php line 1204

Warning: count(): Parameter must be an array or
an object that implements Countable in
/var/www/vhosts/shinobichronicles.com/httpdocs/classes/User.php
on line 1204
https://github.com/levimeahan/shinobi-chronicles/blob/f2d5de82c1850305d85ac2c91c3f5f7764da86d1/classes/User.php#L1204

I believe the issue is the blacklist is a default of null, instead of an empty array upon registration. I'll do some quick testing to confirm.


Sub-transaction for market_feed.sql Frame 1

In reference to the nested while loop in the main transaction loop
https://github.com/osdldbt/dbt5/blob/d9170b8be92624024fa1f65cfad765602da4915c/storedproc/pgsql/pgsql/market_feed.sql#L71

@MaheshGouru Please expand with the details like we discussed this past Monday regarding where the transaction statements go and converting the function to a procedure.

https://github.com/MaheshGouru/dbt5/blob/market_feed_conversion/storedproc/pgsql/pgsql/market_feed.sql
The sub-transaction control was added by reformating the function to a stored procedure convention along with adding COMMITs for DML calls and ROLLBACKS for fetching

I'd like this to be in a slightly different perspective.  For example, it should be describe the work you think you need to do, as opposed to describe work you have done first.  Then it would look more like (not reference is going to :
... adding COMMIT before the END LOOP on line 101
What is still meeting are:

where to start the sub transaction
updating the CREATE FUNCTION with CREATE PROCEDURE
any parameter changes to handle the return value, if applicable
where in the C++ code changes need to go to call the procedure

f0a34cc74e2ef94ce62e3a15b702530430d1f776


[REVIEW] arpanaditya

GitHub Profile Link
https://github.com/arpanaditya

🚀

Thank you so much @eddiejaoude for your suggestions. I have modified my readme accordingly :)


CRAB Client does not need SiteDB Service

not urgent !!

Thanks Stefano, I will include in this tag.

@belforte btw, this and the other TW PR were added to 1.1.17.pre2


Reading nested TTree TBranchElement

I am new to uproot so I may be missing something, but from documents i think the following should work. I have a ROOT TTree made by saving a c++ class as a TClonesArray into a TTree. The ttree.show() method seems to understand the relevant data inside the tree but I do not know how to read it out.
below is with python 3.6, I also tried python 2.7 with the same result.
>>> pairs = uproot.open("rootfiles.root")
>>> tree = pairs["PairDst"]
>>> tree
<TTree b'PairDst' at 0x000110503d30>
>>> tree.show()
Pairs                      TStreamerInfo              astable(asdtype("[(' fBits', '>u8'), (' fUniqueID', '>u8'), ('mVertexZ', '>f4'), ('mDeltaVertexZ', '>f4'), ('mGRefMult', '>u2'), ('mChargeSum', 'i1'), ('mPt', '>f4'), ('mEta', '>f4'), ('mPhi', '>f4'), ('mMass', '>f4'), ('mRapidity', '>f4'), ('d1_mYLocal', '>f4'), ('d1_mZLocal', '>f4'), ('d1_mTof', '>f4'), ('d1_mMatchFlag', 'i1'), ('d1_mLength', '>f4'), ('d1_mPt', '>f4'), ('d1_mEta', '>f4'), ('d1_mPhi', '>f4'), ('d1_mId', '>u2'), ('d1_mNHitsFit', 'i1'), ('d1_mNHitsMax', 'i1'), ('d1_mNHitsDedx', 'u1'), ('d1_mNSigmaPion', '>f4'), ('d1_mNSigmaKaon', '>f4'), ('d1_mNSigmaProton', '>f4'), ('d1_mNSigmaElectron', '>f4'), ('d1_mDCA', '>f4'), ('d1_mPid', '>f4'), ('d2_mYLocal', '>f4'), ('d2_mZLocal', '>f4'), ('d2_mTof', '>f4'), ('d2_mMatchFlag', 'i1'), ('d2_mLength', '>f4'), ('d2_mPt', '>f4'), ('d2_mEta', '>f4'), ('d2_mPhi', '>f4'), ('d2_mId', '>u2'), ('d2_mNHitsFit', 'i1'), ('d2_mNHitsMax', 'i1'), ('d2_mNHitsDedx', 'u1'), ('d2_mNSigmaPion', '>f4'), ('d2_mNSigmaKaon', '>f4'), ('d2_mNSigmaProton', '>f4'), ('d2_mNSigmaElectron', '>f4'), ('d2_mDCA', '>f4'), ('d2_mPid', '>f4')]", "[('mVertexZ', '<f4'), ('mDeltaVertexZ', '<f4'), ('mGRefMult', '<u2'), ('mChargeSum', 'i1'), ('mPt', '<f4'), ('mEta', '<f4'), ('mPhi', '<f4'), ('mMass', '<f4'), ('mRapidity', '<f4'), ('d1_mYLocal', '<f4'), ('d1_mZLocal', '<f4'), ('d1_mTof', '<f4'), ('d1_mMatchFlag', 'i1'), ('d1_mLength', '<f4'), ('d1_mPt', '<f4'), ('d1_mEta', '<f4'), ('d1_mPhi', '<f4'), ('d1_mId', '<u2'), ('d1_mNHitsFit', 'i1'), ('d1_mNHitsMax', 'i1'), ('d1_mNHitsDedx', 'u1'), ('d1_mNSigmaPion', '<f4'), ('d1_mNSigmaKaon', '<f4'), ('d1_mNSigmaProton', '<f4'), ('d1_mNSigmaElectron', '<f4'), ('d1_mDCA', '<f4'), ('d1_mPid', '<f4'), ('d2_mYLocal', '<f4'), ('d2_mZLocal', '<f4'), ('d2_mTof', '<f4'), ('d2_mMatchFlag', 'i1'), ('d2_mLength', '<f4'), ('d2_mPt', '<f4'), ('d2_mEta', '<f4'), ('d2_mPhi', '<f4'), ('d2_mId', '<u2'), ('d2_mNHitsFit', 'i1'), ('d2_mNHitsMax', 'i1'), ('d2_mNHitsDedx', 'u1'), ('d2_mNSigmaPion', '<f4'), ('d2_mNSigmaKaon', '<f4'), ('d2_mNSigmaProton', '<f4'), ('d2_mNSigmaElectron', '<f4'), ('d2_mDCA', '<f4'), ('d2_mPid', '<f4')]"))

>>> tree["Pairs"].array()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jdb/miniconda3/lib/python3.6/site-packages/uproot/tree.py", line 1269, in array
    basket_itemoffset = self._basket_itemoffset(interpretation, basketstart, basketstop, keycache)
  File "/Users/jdb/miniconda3/lib/python3.6/site-packages/uproot/tree.py", line 1227, in _basket_itemoffset
    numitems = interpretation.numitems(key.border, self.basket_numentries(i))
  File "/Users/jdb/miniconda3/lib/python3.6/site-packages/uproot/interp/objects.py", line 144, in numitems
    return self.content.numitems(numbytes, numentries)
  File "/Users/jdb/miniconda3/lib/python3.6/site-packages/uproot/interp/numerical.py", line 165, in numitems
    assert remainder == 0
AssertionError

I also expected something like:
tree["Pairs"]["mMass"]

or
tree["Pairs.mMass"]

to be possible, but that simply gives a KeyError

also I should note that I am using uproot 3.3.0.

The assertion error is internal; you're doing the right thing.
It's not tree["Pairs"]["mMass"] or tree["Pairs.mMass"] because mMass is part of a leaf-list: it's not possible to read separately from the rest of its branch. (Same is true in ROOT: in a leaf-list data structure, all leaves must be read. If you want to get mMass without also reading everything else, you'd have to put them in separate branches.)
Without the assertion error, you could access the mass as tree["Pairs"].array()["mMass"], but that's reading everything and picking out the mass.
Could you send me your file so that I can debug the assertion error? Somehow, the bytes aren't adding up to an integer multiple of the size of a single row.

Thanks for the quick response.  The data itself is pure simulation so there is no problem making it public . The files are produced with c++ compiled code (compiled against ROOT 6.14/04) using structure here: FemtoPair.h
Here is a small sample (3Mb file) of the TTree you can get from my dropbox:
PairDst_uproot.root
If you find it helpful it is fine with me to use it for unit tests, I can also make you an even smaller one if that helps.
The way the tree was saved results in two keys with the same name in the file with the ";1" and ";2" suffixes, I am not sure if that matters.  but maybe it is confusing the way the data is read out. Thanks for this project, I started something like this (pure python ROOT I/O) but never got this far.
Let me know if you need any additional info. Thanks

Okay, I can help you get started with your work, but I can't make it look like a solution you could have come up with on your own! To get your data, do this:
import uproot
tree = uproot.open("PairDst_uproot.root")["PairDst"]
interpretation = uproot.asdtype([(' cnt', '<u4'), (' vers', '<u2')] + eval(str(tree["Pairs"].interpretation.content.todtype)))
data = tree["Pairs"].array(interpretation)

Horrible, right?
The issue is that you have simple, easy to interpret data inside a class. Classes have headers, but these headers aren't always written. The thing I don't know is why these headers are sometimes written and sometimes not. I have a standard set of rules that were derived from the ROOT files I've seen, and according to those heuristics, I expected TObject headers but not byte count and version. You have byte count and version, but not for both the FemtoPair and its FemtoDstBranch base class, just one of these and not the other.
The horrible code above adds the two fields in; I should probably create an easier way to manipulate this.
tree["Pairs"].interpretation

gives the standard interpretation, which is missing two fields, and
tree["Pairs"].interpretation.content.todtype

gets down to the Numpy dtype, which has many fields. (It's a record array.) To add the two extra fields at the beginning, I do
[(' cnt', '<u4'), (' vers', '<u2')] + eval(str(tree["Pairs"].interpretation.content.todtype))

and then wrap it up as a new asdtype interpretation.
interpretation = uproot.asdtype([(' cnt', '<u4'), (' vers', '<u2')] + eval(str(tree["Pairs"].interpretation.content.todtype)))

When we read an array, the first optional argument is a custom interpretation, designed to let you assert that you know how your data are laid out:
data = tree["Pairs"].array(interpretation)

If you find that your whole dataset is too big to load into one array and want to use uproot.iterate or tree.iterate, you can still use this interpretation, but you have to associate it with the "Pairs" branch:
for arrays in tree.iterate({"Pairs": interpretation}):
    do_something_for_subset_of_events(arrays)

I'm going to keep studying your file to see how it's different from the others, to see what hint the metadata can give me that these two fields are needed. But meanwhile, you can get started with your work.

Oh, you do data["mMass"] to get just the mass. This is a Numpy thing (see that link I gave you).


chore: bound the parallelism

Add the linter rule that checks for bounded parallelism, and mark the 2 locations that do Promise.all as bounded.

Should have been a chore, thanks

💚 All backports created successfully

Status
Branch
Result

✅
v2-main

Questions ?
Please refer to the Backport tool documentation and see the Github Action logs for details


Login via api

What is done
This pull request essentially modifies the manage_login_modal callback in main.py, so as to check the credentials entered by the user through the login route of the API instead of the encrypted login correspondences file. As a consequence, all references to the simple-crypt library in the Dockerfile, the requirements, etc can be eliminated. I also added requests to the requirements.
What remains to be done
I could not find a way to retrieve the group_id of the logged-in user. I tried the users/me or simply users routes in a notebook, which work just fine, but I could not find any group_id field in the output. I added placeholders in the code to fetch the group id properly via the API (right now, it is set arbitrarily to 1) and I also added a small group_correspondences dictionary, that will give for each group around what point to center the map, as well as the zoom level to use.
Maybe I was looking in the wrong direction for the group_id 🤔

Thanks for this PR, should we wait for the group_id answer from @frgfm before merge ?

Yes, I guess so!
And otherwise, I have pushed a new commit to correct for the backdrop issue noticed by @chloeskt and to add some comments in the new pieces of code.