Feature/89 volunteers documentation

In this PR, I have implemented the volunteer form for its documentations. To test it, follow this steps:

Make sure that you are running the latest version of branch develop of the backend and that you have a user admin on the api you can access with (If not follow the steps using the auth/users) and
Go to http://localhost:3000/registrarse and complete the form selecting "Registrarse como voluntario" (use a valid email because you'll need it to approve and ignore the toast error if they are not too explicit because it's been implemented in another pr).
Once the form has been completed correctly, a toast success will appear telling you to check your email (you can check that the user has been created on http://localhost:8000/api/user/ with 'role' VOLUNTARIO, 'volunteer' as null and 'is_enabled' as false)
Go to your email and you'd have received an email from<EMAIL_ADDRESS>with a link
Click on the link and click on the POST button. Now the account has been enabled (you can check on http://localhost:8000/api/user/ that 'is_enabled' is now true)
Go to http://localhost:3000/iniciar-sesion and log in with the email and password you have created the user.
Now, http://localhost:3000/voluntario/formulario should appear and you'll need to complete the form with the information asked. The download buttons are used for helping the user get the sexual offenses certificate and obtaining the proper documents it will need to upload signed, but for now you don't need to sign them, just upload a pdf file. Notice that if the birthdate introduced is under 18, 2 extra file inputs will appear. (you can check that the volunteer has been created on http://localhost:8000/api/volunteer/ and that the user in http://localhost:8000/api/user/ has attribute 'volunteer' with the id from the /api/volunteer)
Once the form has been completed correctly, http://localhost:3000/voluntario/espera should appear with a message and the status of it's request as PENDIENTE.
Now, open a new window and go to http://localhost:3000/iniciar-sesion. Log in with your admin profile and go to Voluntarios > Solicitudes. And you should see the user name and surname you have created.

If you accept the request, the page of the user will automatically go to http://localhost:3000/voluntario/perfil.
If you decline the request, the page of the user will still be on the http://localhost:3000/voluntario/espera, but status will have changed to RECHAZADO

Everything works as intended when I create and reject a volunteer, but I get this error when I try to accept one.
Uncaught runtime errors:
×
ERROR
Request failed with status code 404
AxiosError: Request failed with status code 404
at settle (http://localhost:3000/static/js/bundle.js:113075:12)
at XMLHttpRequest.onloadend (http://localhost:3000/static/js/bundle.js:111737:66)
at Axios.request (http://localhost:3000/static/js/bundle.js:112238:41)
at async handleAceptar (http://localhost:3000/static/js/bundle.js:2323:20)

thanks for fixing the issue so fast, works as intended, good work!

The funcionality works as expected and the changes introduced are nice. Just two tiny things I noticied.

In the volunteer form more Toas message with an error could be add to indicate that not file is given/or the formart it is not accept it
The status message should be "Estado" in Spanish instead

I expect that in the following week we'll develop more specific toast.error that's why I have not taken much detail on them, as we might even need a new component for them because it's repeated code on many diferent files

The funcionality works as expected and the changes introduced are nice. Just two tiny things I noticied.

In the volunteer form more Toas message with an error could be add to indicate that not file is given/or the formart it is not accept it
The status message should be "Estado" in Spanish instead

I expect that in the following week we'll develop more specific toast.error that's why I have not taken much detail on them, as we might even need a new component for them because it's repeated code on many diferent files

You are misunderstaing some concepts I think. The toast message with the message are created directly on the frontend(you can see how in the branch I am working). For example if you go to network when not uploading a documment the validation error from the back appears so from the frontend whtat we have to do it is to show them

The funcionality works as expected and the changes introduced are nice. Just two tiny things I noticied.

In the volunteer form more Toas message with an error could be add to indicate that not file is given/or the formart it is not accept it
The status message should be "Estado" in Spanish instead

I expect that in the following week we'll develop more specific toast.error that's why I have not taken much detail on them, as we might even need a new component for them because it's repeated code on many diferent files

You are misunderstaing some concepts I think. The toast message with the message are created directly on the frontend(you can see how in the branch I am working). For example if you go to network when not uploading a documment the validation error from the back appears so from the frontend whtat we have to do it is to show them

I didn't misunderstand, I know what it's suppose to show but in front we'll need to implement personalized messages because backend shows the message in english. What I'm trying to say is that we have many pages using toast errors messages and most of them are with the same message, so in the end codacy will complain about that and we'll have to create a component calling all the different toast errors. That's why at the moment I have created generic toast errors and I have implemented some more after your request

One question, have you done that the post you are making to the api, includes the headers?

If you look at the code, you can clearly see that it's using the headers. Otherwise, the post should not be working


Properly handle merged stories

https://lobste.rs/s/n0mcqh/r

Seems like there's no way to see merged stories from the JSON endpoint :(

This is technically infeasible, closing.


are there themes for prompt?

I would like to change the text and style of the prompt error messages (like "Please enter Y or N"), I guess the text is not possible without hacking the source, but I wonder if I could change the default prompt.invalid theme. There is the possibility to change the default console theme with Console(theme=xx) but that seems not to work for Prompt()

You should be able to set a theme for prompts. What have you tried?

You may have missed that you need to tell the Prompt class which console to use.
theme = Theme({"prompt": "bold green blink"})
console = Console(theme=theme)
if Confirm.ask("does this work?", console=console):
    console.print(":thumbs_up:")


net: add mechanism to wait for readability on a TCPConn

EDIT: this proposal has shifted. See https://github.com/golang/go/issues/15735#issuecomment-266574151 below.
Old:
The net/http package needs a way to wait for readability on a TCPConn without actually reading from it. (See #15224)
http://golang.org/cl/22031 added such a mechanism, making Read(0 bytes) do a wait for readability, followed by returning (0, nil). But maybe that is strange. Windows already works like that, though. (See new tests in that CL)
Reconsider this for Go 1.8.
Maybe we could add a new method to TCPConn instead, like WaitRead.

/cc @ianlancetaylor @rsc

read(2) with a count of zero may be used to detect errors.  Linux man page confirms, as does POSIX's read(3p) here.  Mentioning it in case it influences this subverting of a Read(0 bytes) not calling syscall.Read.

I found a way to do without this in net/http, so punting to Go 1.9.

Actually, the more I think about this, I don't even want my idle HTTP/RPC goroutines to stick around blocked in a read call. In addition to the array memory backed by the slice given to Read, the goroutine itself is ~4KB of wasted memory.
What I'd really like is a way to register a func() to run when my *net.TCPConn is readable (when a Read call wouldn't block). By analogy, I want the time.AfterFunc efficiency of running a func in a goroutine later, rather than running a goroutine just to block in a time.Sleep.
My new proposal is more like:
package net

// OnReadable runs f in a new goroutine when c is readable;
// that is, when a call to c.Read will not block.
func (c *TCPConn) OnReadable(f func()) {
   // ...
}

Yes, maybe this is getting dangerously into event-based programming land.
Or maybe just the name ("OnWhatever") is offensive. Maybe there's something better.
I would use this in http, http2, and grpc.
/cc @ianlancetaylor @rsc

Sounds like you are getting close to #15021.
I'm worried that the existence of such a method will encourage people to start writing their code as callbacks rather than as straightforward goroutines.

Yeah. I'm conflicted. I see the benefits and the opportunity for overuse.

If we do OnReadable(f func()), won't we need to fork half of standard library for async style?
I don't see any way to push something asynchronously into e.g. gzip.Reader. Does this mean that I have to choose between no blocked goroutine + my own gzip impl and blocked goroutine + std lib?

Re 0-sized reads.
It should work with level-triggered notifications, but netpoll uses epoll in edge-triggered mode (and kqueue iirc). I am concerned if cl/22031 works in more complex cases: waiting for already ready IO, double wait, wait without completely draining read buffer first, etc?

@dvyukov, no, we would only use OnReadable in very high-level places, like the http1 and http2 servers where we know the conn is expected to be idle for long periods of time. The rest of the code underneath would remain in the blocking style.

This looks like a half-measure. An http connection can halt in the middle of request...

@dvyukov, but not commonly. This would be an optimization for the common case.

An alternative interface can be to register a channel that will receive readiness notifications. The other camp wants this for packet-processing servers, and there starting a goroutine for every packet will be too expensive. However, if at the end you want a goroutine, then the channel will introduce unnecessary overhead.
Channel has a problem with overflow handling (netpoll can't block on send, on the other hand it is not OK to lose notifications).
For completeness, this API should also handle writes.

We need to make sure that this works with Windows IOCP as well.

Not obvious to me why the API has to handle writes. The thing about reads is that until the data is ready for reading, you can use the memory for other work. If you're waiting to write data, that memory is not reusable (otherwise you'd lose the data you are waiting to write).

@rsc If we do just 0-sized reads, then write support is not necessary. However, if we do Brad's "My new proposal is more like": func (c *TCPConn) OnReadable(f func()), then this equally applies to writes as well -- to avoid 2 blocked goroutines per connection.

If memory usage is the concern, it is possible to make long parked G use less memory instead of changing programming style? One main selling point of Go to me is high efficiency network servers without resorting to callbacks.
Something like shrink the stack or move the stack to heap by the GC using some heuristics, that will be littile different from spinning up a new goroutine on callback memory usage wise, and scheduling wise a callback is not much different than goready(). Also I assume the liveness change in Go1.8 could help here too.
For the backing array, if it is preallocated buffer, than a callback doesn't make much different than Read(), maybe it will make some different if it is allocated per-callback and use a pool.

How about a epoll like interface for net.Listener:
type PollableListener interface {
   net.Listener
   // Poll will block till at least one connection been ready for read or write
   // reads and writes are special net.Conn that will not block on EAGAIN
   Poll() (reads []net.Conn, writes []net.Conn)
}

Then the caller of Poll() can has a small number of goroutines to poll for readiness and handle the reads and writes. This should also works well for packet-processing servers.
Note that this only needs to be implemented in the runtime for those Listeners that multiplexed in the kernel, like the net.TCPListener. For other protocol that multiplex in the userspace and doesn't attached to the runtime poller directly, like udp listener or multiplexing streams in a tcp connection, can be implemented outside the runtime. For example, for multiplexing in a tcp connection, we can implemented the epoll like behavior by read from/write to some buffers then poll from them.

I'm fairly new to Go but seeing the callback interface is a little grating given the blocking API exposed everywhere else. Why not expose a public API to the netpoll interfaces?
Go provides no standard public facing event loop (correct me if I'm wrong please). I have need to wait for readability on external FFI socket(s) (given through cgo). It would be nice to re-use the existing netpoll abstraction to also spawn FFI sockets onto rather than having to wrap epoll/IOCP/select. Also I'm guessing wrapping (e.g) epoll from the sys package does not integrate with the scheduler which would also be a bummer.

For a number of my use cases, something like this :
package net

// Readable returns a channel which can be read from whenever a call to c.Read
// would not block.
func (c *TCPConn) Readable() <-chan struct{} {
        // ...
}

.. would be nice because I can select on it. I have no idea whether it's practical to implement this though.
Another alternative (for some of my cases at least) might be somehow enabling reads to be canceled by using a context.

type ReadableSig struct {
    C *TCPConn,
    I interface{}
}
// Readable marks connection as waiting for read.
// It will send connection to channel when it becomes readable (or closed by peer)
//  together with additional information passed as 'i' argument.
// Note: if channel blocks, it will create goroutine.
func (c *TCPConn) Readable(i interface{}, ch chan<- ReadableSig) {
}

Or even better:
type ReadableNotify struct {
    Conn *TCPConn,
    Ctx context.Context
}
// Readable marks connection as waiting for read and It will send connection
// to channel when it becomes readable (or closed) together with context.
// Internally it may spawn goroutine. It will certainly spawn goroutine if channel
// blocks.
// Note: connections always waits for readability, but if context is closed then
// notification may be omitted. If notification recieved, then Context should be
// checked manually.
func (c *TCPConn) Readable(ctx context.Context, ch chan<- ReadableNotify)

btw, can runtime spawn utility goroutine with less stack than user-faced goroutine?

Hi! Great idea 🎆
In our go server with >2millions of alive WebSocket connections we were used epoll to bring the same functionality.
epoll.Handle(conn, EPOLLIN | EPOLLONESHOT, func(events uint) {
    // No more calls will be made for conn until we call epoll.Resume().
    ...
    epoll.Resume(conn)
})

That is, it could be useful to make callback call only once after registration.
In our case we've used EPOLLONESHOT to make this work.

I think you'll have to add support for Registered IO in windows, instead of IO Completion Ports

I want API for nonblocking Read, or nonblocking Readable test.
Neither blocking wait nor callback don't satisfy my use case.
My use case is DB driver (go-sql-driver/mysql).
Basically, it is request-response protocol.  But sometime connection is closed by server or someone in the middle.
If we send query, we can't retry safely because query is not idempotent.  So we want to check connection
is closed or not before sending query.  Pseudo code is here.
func (mc mysqlConn) Query(query string, args... driver.Value) error {
    if mc.conn.Readable() {
        // Connection is closed from server.
        return mc.handleServerGone()
    }

    mc.sendQuery(query, args...)
    return mc.recvResult()
}

Adding one or more goroutine makes the driver more complicated and slower.

@methane I'm sorry, I don't understand your example: if Readable returns true, I would not expect the connection to be closed.  And, even if you omitted a !, if Readable returns false, that could just mean that the other side hasn't written any data; it doesn't mean that the connection is closed.  Determining whether it is possible to read from a socket is not the same as determining whether the socket is closed.

@ianlancetaylor Note that how we get EOF.  We need to call conn.Read().
As I said above, what I want is "nonblocking Read, or nonblocking Readable test."

Won't write on a closed/broken connection return an error? If yes, then you could just write the request always. If write succeeded, request was sent. If write failed, request wasn't send.
This has obvious races, but it's the same as for your code.

@dvyukov No.  Even though remote peer closed connection, conn.Write() will success.
It's because TCP allows "half close".  (I hadn't tested it on Unix socket yet)

@dvyukov DB closes full connection at socket API level.  But it doesn't affect TCP.
Until client closes connection (including half close of write side), Write() will success.

@dvyukov Please try this code:
https://play.golang.org/p/ElUBU_l8QC2

Yup, write fails as expected:
$ go run /tmp/test.go
0 Read 0 (EOF)
0 Written 6 (<nil>)
1 Read 0 (EOF)
1 Written 0 (write tcp <IP_ADDRESS>:42246-><IP_ADDRESS>:39591: write: broken pipe)
2 Read 0 (EOF)
2 Written 0 (write tcp <IP_ADDRESS>:42246-><IP_ADDRESS>:39591: write: broken pipe)

I would expect it to be the other way around. Readability test should always check local read buffer first. So if some data is already received, but remote end is closed, readability test will always succeed, but write will fail.

Yup, write fails as expected:

0 Read 0 (EOF)
0 Written 6 (<nil>)

You send 6 bytes, even after you received EOF.

Why does kernel do this? Is it a kernel bug? The connection is already in CLOSE_WAIT state. This makes read fail, but not write...

Because there are no difference between half close (CloseWrite) and full close on TCP layer.
Both send just one FIN packet.
So client don't know remote peer will receive data or not.
When client send data packet, server will return RST packet.
That's why second send returns EPIPE.  Here is tcpdump of the sample program.
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
19:45:09.516488 IP (tos 0x0, ttl 64, id 21231, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.33704 > localhost.11451: Flags [SEW], cksum 0xfe30 (incorrect -> 0x751c), seq 30788114, win 43690, options [mss 65495,sackOK,TS val<PHONE_NUMBER> ecr 0,nop,wscale 7], length 0
19:45:09.516497 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.11451 > localhost.33704: Flags [S.E], cksum 0xfe30 (incorrect -> 0x0a34), seq<PHONE_NUMBER>, ack 30788115, win 43690, options [mss 65495,sackOK,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>,nop,wscale 7], length 0
19:45:09.516505 IP (tos 0x0, ttl 64, id 21232, offset 0, flags [DF], proto TCP (6), length 52)
    localhost.33704 > localhost.11451: Flags [.], cksum 0xfe28 (incorrect -> 0xdcb8), ack 1, win 342, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 0
19:45:09.516529 IP (tos 0x0, ttl 64, id 36746, offset 0, flags [DF], proto TCP (6), length 52)
    localhost.11451 > localhost.33704: Flags [F.], cksum 0xfe28 (incorrect -> 0xdcb7), seq 1, ack 1, win 342, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 0
19:45:09.616632 IP (tos 0x2,ECT(0), ttl 64, id 21234, offset 0, flags [DF], proto TCP (6), length 58)
    localhost.33704 > localhost.11451: Flags [P.], cksum 0xfe2e (incorrect -> 0x9869), seq 1:7, ack 2, win 342, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 6
19:45:09.616641 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    localhost.11451 > localhost.33704: Flags [R], cksum 0x4929 (correct), seq<PHONE_NUMBER>, win 0, length 0

Interesting. Thanks.
But readability is also not what you want, right? Readability will say OK is connection is half closed the other way, or fully closed but have local read data queued.

"Readability" meant "conn.Read() will return without block".
MySQL protocol is basically "request-response" protocol.  Until we send a query, server don't send
anything in most case.  But there are exceptions: server shut down, server (or Linux or router) closed idle connection, etc.
In such cases, we don't want send query because query may be not idempotent.
That's why I want non-blocking Read.

If your query is not idempotent, you are sunk.  The connection might
disappear after the server has received the query and committed the result,
but before it can respond.  You just don’t know.
You can have at-least-once, or at-most-once.  Exactly-once has been proven
impossible.  It simply can’t be done, unless the recipient keeps a record
of which messages it has received, a sequence number, or something similar.
On Nov 30, 2018 8:01 AM, "INADA Naoki"<EMAIL_ADDRESS>wrote:
"Readability" meant "conn.Read() will return without block".
MySQL protocol is basically "request-response" protocol. Until we send a
query, server don't send
anything in most case. But there are exceptions: server shut down, server
(or Linux or router) closed idle connection, etc.
In such cases, we don't want send query because query may be not idempotent.
That's why I want non-blocking Read.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/15735#issuecomment-443196520, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGGWB3Z1d2aNcFdZQoAxJXgT_63pwjq6ks5u0SwWgaJpZM4Ihu7E
.

@DemiMarie It's off topic.

What if instead of an explicit callback API we use something channel based?  We could add a method NotifyWhenReadable to os.File and net.TCPConn, etc.
func (t *TCPConn) NotifyWhenReadable(c chan<- interface{}, val interface{})

When the connection is readable, val is sent to c.  This would happen once.  If you wanted another notification, you would have to call NotifyWhenReadable again.  Naturally the channel send would be non-blocking, and the channel should be buffered.
This is a little bit like signal.Notify.
This permits a callback API but doesn't guide people toward it.  A goroutine could wait for its connection to be readable easily enough via
    ch := make(chan interface{}, 1)
    conn.NotifyWhenReadable(ch, nil)
    <-ch

or you could have a single server goroutine
    for f := range ch {
        go f.(func())()
    }

and each separate connection would
    conn.NotifyOnRead(ch, func() { /* do the next thing */ })
    return

Naturally the channel send would be non-blocking

How are lost values supposed to be handled? If we drop a notification, part of the program will stay deadlocked forever?
Here @prattmic says the reason gvisor avoids standard network polling is not memory consumption, but rather latency:
https://github.com/golang/go/issues/29734#issuecomment-457303059
So I think we need to consider this aspect as well and if possible not introduce inherent latency penalty.

Since a notification is sent at most once per call to NotifyWhenReadable, it's easy to give the channel a large enough buffer to never lose any notifications.
I think gVisor is kind of a special case.  I think most large servers care more about memory.  But of course if we can fix both that would be better.

Since a notification is sent at most once per call to NotifyWhenReadable, it's easy to give the channel a large enough buffer to never lose any notifications.

You mean for the channel per connection scheme? If a single channel is used for all connections, it would require knowing some hard upper bound on number of connections.
If one of the recommended schemes is to use a single channel, I suspect it will become a scalability bottleneck. Of course users can start partitioning the channel into N independent channels, but then it will have a problem with load balancing: either some partitions will be overloaded and/or bad locality and/or preallocataing N times more memory for channel buffers.
I also wonder if we will be able to do this "exactly one non-spurious notification". I think C networking servers will do own polling, but then always followed by non-blocking reads/writes. If we leave reads/writes blocking, then it will bring very unique requirements for the polling mechanism.

E.g. the current linux polling scheme is always armed edge-triggered epoll. It by design generates spurious notifications.

Why don’t just add callback interface? That is, with the same semantics as @ianlancetaylor suggested – with re-register of the callback for the next readable event after one occurred. Inside callback user could define its own strategy of handling event. Probably it will dispatching task between some free goroutines.
Also regarding this comment from @dvyukov:

If we do OnReadable(f func()), won't we need to fork half of standard library for async style?

I think this may not affect any wrappers around raw conn in general – we could subscribe for readable event on conn and read from wrapper, hoping that it would not block on split cases, when available data is not sufficient for wrapper purposes of decoding/processing data.

@gobwas A callback interface is what @bradfitz suggested above.  My main concern with a callback interface is, simply, it's not typical Go style.  I was hoping to explore alternatives.

And to be clear to others (Ian knows this), in the same comment I suggested that above, I wrote:

Yes, maybe this is getting dangerously into event-based programming land.

Which is another way of saying "not typical Go style"

unfortunately, "not typical Go style" is the thing people have to fall in when they want top perofrmance and/or millions of connections per process.
https://medium.freecodecamp.org/million-websockets-and-go-cc58418460bb
https://github.com/eranyanay/1m-go-websockets

typical Go style is quiteconvenient for high-level nd medium-levellogic, and quite inefficient at low level.

@ianlancetaylor as exploration, maybe option of exporting network poller API would be better idea than a callback? I mean something like this:
import "net/poll"

p, err := poll.Create(...) // epoll_create() on Linux.
if err != nil {
    // handle error
}
go func() {
    for ev := range poll.Wait() { // epoll_wait on Linux.
        ev.Conn.Read(...)
        ev.Data // User data bound within Notify() call.
    }
}()
conn, err := net.Dial(...)
if err != nil {
    // handle error
}
if err := p.Notify(poll.Read, conn, data); err != nil { // epoll_ctl on Linux.
    // handle error
}

Thus, we may leave general poller untouched (and non-blocking), but provide additional polling mechanism which will block when poll.Wait() stopped being drained.
Also Notify() api may differ, e.g. accepting different channels to write event to, as you suggested, but in blocking manner – without events loss and with strong notice in method comment =)

I also wonder if we will be able to do this "exactly one non-spurious notification". I think C networking servers will do own polling, but then always followed by non-blocking reads/writes. If we leave reads/writes blocking, then it will bring very unique requirements for the polling mechanism.

@dvyukov, would it work to do a non-blocking read of a byte or two internally to verify readability before raising a notification? That adds at least one syscall to the overhead, but for a network or storage situation, maybe that's OK.
In the spirit of epoll, it would be helpful to get notifications for an arbitrary set of io.Reader objects via a single channel, since we cannot select on a runtime-defined set of channels.

would it work to do a non-blocking read of a byte or two internally to verify readability before raising a notification?

This looks dirty and slow and in the end there can be just 1 byte. So it was readable, but not now anymore ;)

In the spirit of epoll, it would be helpful to get notifications for an arbitrary set of io.Reader objects via a single channel, since we cannot select on a runtime-defined set of channels.

I think this is supported by the Ian's proposal, no?

Then let r.Read() be non-blocking after r.NotifyWhenReadable() and possibly return error io.WouldBlock?

@networkimprov JFYI there is an ability to Select() on runtime defined set of channels via reflect api.

Do you have a plan to deal with the issue Dmitry raised?

@networkimprov Which issue?  If you mean the problem with possibly losing a channel notification, then of course he is quite right, and the channel send must be a blocking send.  Depending on the implementation, this may require spinning up a goroutine to do the send.

No, I was referring to

I also wonder if we will be able to do this "exactly one non-spurious notification". I think C networking servers will do own polling, but then always followed by non-blocking reads/writes. If we leave reads/writes blocking, then it will bring very unique requirements for the polling mechanism. E.g. the current linux polling scheme is always armed edge-triggered epoll. It by design generates spurious notifications.

How do you deal with this without letting r.Read() be non-blocking after r.NotifyWhenReadable() and possibly returning error io.WouldBlock?

The problem I was trying to address was "large server with a lot of goroutines burns a lot of memory in buffers waiting to read."  For that purpose it's OK if a spurious notification leads to a blocking Read, as long as spurious notifications are rare.  But maybe I'm missing something.

Brad was trying to get rid of the idle goroutines: https://github.com/golang/go/issues/15735#issuecomment-266574151
If that's not a requirement, an alternate read API could internally poll, malloc/free, and do non-blocking reads, and return a new buffer on success. An internal buffer pool would avoid thrashing the allocator due to fruitless reads after spurious poll events.
func (r *T) ReadSize(maxLen int) (byte[], error)

My channel suggestion also permits getting rid of the idle goroutines, for people who want to write a more complex program.

But it does incur a read-buffer + idle goroutine for every spurious poll event, which could be costly.
@dvyukov, any idea as to the proportion of spurious epoll wakeups in Linux?
More on epoll difficulties (from 2y ago):
https://news.ycombinator.com/item?id=13736674

@dvyukov, any idea as to the proportion of spurious epoll wakeups in Linux?

No, I don't.
But if a solution relies on absence of spurious wakeups for correctness/performance, I guess we need these numbers for all OSes before making a decision.

So if I understand correctly, I am now stuck: I cannot read a 0 byte string to detect if the connection is closed and I cannot read a 1 byte because my server never writes anything and the read is blocking?

FYI, we implemented this in mysql driver.
https://github.com/go-sql-driver/mysql/blob/master/conncheck.go
Note that it works only on Unix, and it caused several allocations.
I really want this is implemented in stdlib in efficient (and probably cross platform) way.

It's highly useful to reduce overhead when the go program holds a lot of connections (connection pool) and now I'm using the epoll mechanisms do this which is not idiomatic.
Looking forward to land in Go, many thanks:)

@ianlancetaylor a typical problem I have in an http proxy is, that connection spikes can create spikes in memory usage. I think this can be fixed with using epoll and I hope your approach will cover the problem. We would need to be able to set the max concurrency level for the goroutine calls, that will read and write from/to sockets.
We have an internal protection mechanism to avoid this problem, but you can see the memory spike in in bufio:

I need readability notification badly in my project, as I have to pre-allocate 4k buffer per connection.
https://github.com/xtaci/kcptun/blob/master/client/main.go#L56

So there is RawConn now which has an interface very much like what @bradfitz was proposing here. However, it calls the read callback before calling wait for read. It must do this, as the net poller uses edge-triggered events - they won't fire if there is already data on the socket.
One workaround is to use a small stack buffer for the initial Read, and then when that reads some data, allocate and copy it to a real buffer, and then call Read again. That helps, but you'll still have the goroutine's 4KB stack overhead.
Another option is use RawConn.Control with unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.TCP_DEFER_ACCEPT, 1) for connections like HTTP where the client must send data first, it won't create a goroutine in the first place until there's data buffered, but only helps for that initial Read. It can be combined with the stack buffer approach for long-lived connections.
I'd still like to see an approach (maybe an async OnReadable callback method on RawConn, like bradfitz is proposing here?) to avoid having the goroutine overhead. Mail.ru avoids the net poller entirely and manually manages epoll for precisely this reason, as 4KB per WebSocket connection with millions of open but mostly idle WebSockets waiting for new mail is just too much overhead.

I'd still like to see an approach (maybe an async OnReadable callback method on RawConn, like bradfitz is proposing here?) to avoid having the goroutine overhead. Mail.ru avoids the net poller entirely and manually manages epoll for precisely this reason, as 4KB per WebSocket connection with millions of open but mostly idle WebSockets waiting for new mail is just too much overhead.

I don't think OnReadable callback is a good idea. If I have 2 handlers to toggle based on incoming data, then it's not impossible to code this kind of nested callbacks which reference to each other.
For such reason, in order to writing comprehensive logic we have to copy the buffer from callback out to another goroutine, in such case, the memory usage will be out of control, as we lost the ability to back-pressure the congestion signal to senders. (Before, we won't start next net.Read() until we've processed data.)
Even for a callback like func() { Read);Process();Write() } scenario, if Write() blocks, this callback still holds 4KB-buffer waiting to be sent.
In all these cases above, we still have inactive 4KB-buffers somewhere.

I wrote a library based on ideas above, golang async-io
https://github.com/xtaci/gaio

For my particular use case having a non-blocking Read is highly desirable. An chat service aiming to manage millions of connections per instance. Blocking interface forces to have an alive goroutine per connection, which makes this goal totally unrealistic.

I wrote a library based on ideas above, golang async-io
https://github.com/xtaci/gaio

And it works great - it uses an epoll event handler and callback to solve the readability and writeability issues that are discussed here (if I'm not mistaken ... lol)
I use it instead of a multiplexer for a message proxy component which handles one to many client connections for inbound and outbound traffic.
It has a batch style of message processing which requires a message reader and frame size delimited message protocol. That is, the message reader compiles a complete message from 1 or more partial messages provided in each event result buffer

I wrote another async-io lib nbio to avoid using two goroutines per connection and reduce memory usage.
It supports http 1.x and is basically compatible with net/http. Many net/http-based web frameworks can be easily converted to use the async-io network layer.
It also supports tls and websocket.
I am trying to increase the load capacity of a single hardware, 1m/1000k-connections would not be that hard anymore for golang.
https://github.com/lesismal/nbio

In addition to the array memory backed by the slice given to Read, the goroutine itself is ~4KB of wasted memory.

If the problem is the size of the idle goroutine stack, would it make sense to have the runtime recognize goroutines that are likely to block ~immediately and allocate a much smaller initial stack for them?

@bcmills, maybe that could be a new proposal?

In addition to the array memory backed by the slice given to Read, the goroutine itself is ~4KB of wasted memory.

If the problem is the size of the idle goroutine stack, would it make sense to have the runtime recognize goroutines that are likely to block ~immediately and allocate a much smaller initial stack for them?

Problems are both goroutine stack and buffer for Read. Especially when TLS is involved.

At least, if we had goroutines memory pressure under control the read buffer issue may be tackled by extending net.Conn API:
conn := getTLSConn()

if err := conn.WaitForContent(); err != nil {
    return err
}
rdBuff := grabReadBuffer()
defer releaseReadBuffer(rdBuff)

n, err := conn.ReadAvailableContent(rdBuff)
if err != nil {
    return err
}
handleContent(rdBuff)
return nil

Problems are both goroutine stack and buffer for Read. Especially when TLS is involved.

At least, if we had goroutines memory pressure under control the read buffer issue may be tackled by extending net.Conn API:
conn := getTLSConn()

_, err := conn.WaitRead() // block until there's available content
if err != nil {
    return err
}
rdBuff := grabReadBuffer()
defer releaseReadBuffer(rdBuff)

n, err := conn.ReadAvailable(rdBuff)
if err != nil {
    return err
}
handleContent(rdBuff)
return nil

I wrote an async-io lib nbio and forked std tls and rewrote it to support non-blocking.
Here are examples that save more memory:
https://github.com/lesismal/nbio/pull/62#issuecomment-881221338
In my simple echo test with100k websocket tls connections, compared with std based frameworks:

solution
qps
cpu
memory

nbio
avg:110-120k, running fine
around 300%
1.3G

std
avg: 60-80k, with obvious stw
around 300%
3.3G

If I may voice a contrary opinion, I would argue that there are other ways to obtain the same benefits (minimize space wasted on parked goroutine stacks and on buffers that are waiting for data to be written into them) without unleashing callbacks and/or readability signals.
One such way could be to have buffers be decommitted from memory before submitting them for a blocking read (I haven't tested this, but AFAIK it should work). Right now this would require an additional syscall (e.g. madvise) before the read, but once/if #31908 comes around that won't be required anymore (as the madvise and read could be submitted in a single syscall). Given page-aligned buffers of size greater than a page, this should transparently achieve the goal at least for the buffers.
For the goroutine stack, something similarly transparent could also be achieved: as part of GC we already shrink the stacks of goroutines, if they are too big. It is not impossible to imagine to extend this mechanism to detect goroutines that have been parked for a while waiting for I/O and that would "freeze" their stacks, aggressively packing together the contents of the frozen stacks (without maintaining the minimum 4KB size), and freeing them. When the goroutine needs to be unparked a new stack of the appropriate size would be allocated, the frozen stack contents would be copied into the newly allocated stack and the goroutine would resume. This should transparently achieve the goal for the stacks of goroutines blocked on I/O.
I won't deny that there is a lot of handwaving in this comment: in my defense, I am not trying to provide a fully fleshed-out design. I'm just trying to point out that there may be other possibilities to achieve the same goals besides the mechanisms being discussed in the rest of this issue. Because these mechanisms, as has been pointed out already, are pretty non-go-like I would hope that any alternative that may achieve transparently the desirable goals set out in this issue is fully exhausted before more extreme options are implemented.

There are several potential issues with madvise:

It conflicts with large 2M pages.
The buffers are not necessary page-aligned.
The contents are not necessary discard-able (for a short read the rest of buffer must be left intact).

(sorry for being off topic, I will not add further comments in this issue; if needed I will update this comment to address further replies to avoid polluting the discussion with design considerations of a potential counterproposal)
IIUC 1 and 2 have multiple potential solutions (document that only pagesize-aligned buffers can benefit from this optimization, teach the memory allocator to handle pagesize-sized allocations especially, provide an allocator function specialized for this purpose, suggest to use off-go-heap memory, ...) so possibly these are not a showstopper (unless I'm missing something).
3 OTOH could be a showstopper (for the buffer part), but I'm confused. io.Reader docs seems to indicate that is not the case:
type Reader interface {
	Read(p []byte) (n int, err error)
}

[...] Even if Read returns n < len(p), it may use all of p as scratch space during the call. [...]

It only says that the space beyond n can be used as scratch space, but it does not say whether the contents are restored after the area has been used as scratch space. I always interpreted that to mean that data in p[n:] may not contain the original contents when Read returns, otherwise to be able to use all of p as scratch space Read would always require yet another scratch space to temporarily store a copy of p just in case of a short read (but then what would be the point of using p as the scratch space?). Did I get this wrong all this time?

[...] Even if Read returns n < len(p), it may use all of p as scratch space during the call. [...]

I think you are right. I just assumed it should not change what's not returned following the principle of the least surprise.

I am surprised to find out that there is no non-blocking way to read network connections in stdlib while Go is the most network-centric language out there. How come there is no progress since 2016 and one has to look for 3rd party libraries(none of which work on windows)?

@ivanjaros The Go runtime uses non-blocking I/O internally, so that Go programs don't have to.
Note that this issue is not about non-blocking I/O.  It's about a way for a goroutine to wait for a connection to be readable without having to actually read from it.

@ianlancetaylor my point was that this issue has been ongoing for years but we still have to result to using 3rd party libraries when we want to seek the bleeding edge of performance over DX through any of:
https://github.com/tidwall/evio
https://github.com/panjf2000/gnet
https://github.com/hslam/netpoll
https://github.com/xtaci/gaio
https://github.com/Allenxuxu/gev
https://github.com/pigogo/netgo
https://github.com/npat-efault/poller
https://github.com/alberliu/gn
https://github.com/smallnest/1m-go-tcp-server
On the other hand, since there are these options, the entire discussion might have become obsolete 🤷♂

@ivanjaros
It seems only my lib nbio supports async-io and async parser for tls/http/websocket by now but wasn't listed in your list and few people know it...

Note that this issue is not about non-blocking I/O. It's about a way for a goroutine to wait for a connection to be readable without having to actually read from it.

@ianlancetaylor If go support a readable callback, the conn.Read and conn.Write should not blocked any more, else users still need to use at least one goroutine to handle Read/Write logic, then it would not be different from the current std solution on reducing the cost from lots of goroutines.

Sounds like you are getting close to #15021.
I'm worried that the existence of such a method will encourage people to start writing their code as callbacks rather than as straightforward goroutines.

@ianlancetaylor
Definitely agree that async-io does need to be used callback in parsing logic, in which we also needs to consider the memory optimization of the half-packet bytes cache.
But only the part of parsing needs to use callbacks, such as the http parser.
The application-layer users can still keep the synchronous style as std does: async-io and callback-parser parse the complete package and pass them to a size-limited-goroutine-pool for business logic processing.
Actually, my lib has already implemented it like this and basically compatible with net/http. A simple hello-world server as below:
package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/lesismal/nbio/nbhttp"
)

func HelloWorld(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "hello world")
}

func main() {
	router := &http.ServeMux{}
	router.HandleFunc("/", HelloWorld)
	// also can use gin
	// router := gin.New()
	// router.GET("/", ...)
	// or echo
	// router := echo.New()
	// router.GET("/hello", ...)

	svr := nbhttp.NewEngine(nbhttp.Config{
		Network: "tcp",
		Addrs:   []string{"localhost:8080"},
		Handler: router,
	})

	err := svr.Start()
	if err != nil {
		log.Fatalf("nbio.Start failed: %v\n", err)
		return
	}
	defer svr.Stop()

	<-make(chan int)
}

And even can be used with other std based frameworks like gin or echo, more examples:

gin-http-and-websocket-server
echo-http-and-websocket-server

I am sorry if it is not appropriate to mention my own lib here, but I think this [async-io] + [callback parsing] + [goroutine pool] solution can really allow us both to get the resource cost under control and to write synchronous logic.
We have the cake and eat it, too.
It has been verified in load testing and in some of my projects.

There is waiting for a TCP connection to have data available, which is this issue.  And there is non-blocking I/O.  They are not the same thing.

There is waiting for a TCP connection to have data available, which is this issue. And there is non-blocking I/O. They are not the same thing.

We know that it is blocking io, but the currently provided interfaces such as conn.Read/Write are non-blocking interfaces for the application layer.
If the non-blocking Read/Write interface is not provided with Readable together, for the application layer, it is the same as the current std.

@lesismal can you provide the data, maybe it's more convincing to show data.
@ianlancetaylor  If we would consider to have a fixed size goroutine pool (as suggested in https://github.com/golang/go/issues/15735#issuecomment-1235331181) for handlers that can be set by some configuration, it would also fix the unbounded memory usage issue in https://github.com/golang/go/issues/35407.

@szuecs I've provided some test results on the previous floor, please check it or better to try the code in yourself's env:
https://github.com/golang/go/issues/15735#issuecomment-917435376

We know that it is non-blocking io, but the currently provided interfaces such as conn.Read/Write are blocking interfaces for the application layer.

As you probably know, they block the goroutine that calls them.  They don't block any threads.  Goroutines are cheap.  Programs that need to avoid blocking goroutines are a special case.  And as noted above, it is already possible to use non-blocking I/O in Go.

As you probably know, they block the goroutine that calls them. They don't block any threads. Goroutines are cheap. Programs that need to avoid blocking goroutines are a special case. And as noted above, it is already possible to use non-blocking I/O in Go.

Yes, I know that they don't block any threads.
But when using std and there are millions of connections, the huge num of goroutines are not as cheap as there are only thousands of connections. That's why there are many third libs to do it as @ivanjaros listed:
https://github.com/golang/go/issues/15735#issuecomment-1235148890

In a 1m connections test:
Golang needs at least 8G memory(8k*1m), and the CPU cost for both schedule and GC is also huge, so we can't deploy this service on a VM/hardware with low-middle level specifications, else we come across both OOM and STW problems.
Compared to other languages in this test, Golang performs even worse than Java-Netty and Nodejs, with multiple memory costs, that's such unexpected amazing.
As a cloud-native language, Golang will handle more and more connections as basic frameworks. The more we can reduce the num of goroutines, the more hardware, energy, and money we can save.
Not joking, but sincerely:joy::
For environmental protection!
For the earth!

My opinion is that if provide Readable, non-blocking Read/Write interface should be provided together, else Readable will not be useful.

When PHP 7, i think, came out couple of years ago, Rasmus(PHP author) had a talk on some PHP conference where he showed how much electricity the better performance of this new version saved due to lower demand on hardware and hence lower economic costs in general. So this is not as outlandish as it might seem. As pointed above, the cost-savings are already impressive https://github.com/golang/go/issues/15735#issuecomment-917435376 and with stdlib implementation we might get even better CPU performance.

All of the implementations listed above use mostly a reactor pattern, not a pro-actor pattern. See proactor pattern. (boost::asio is a pro-actor lib).
If I am not mistaken the proposal here is to add readability which would be implemented using callbacks, which would make Go support some kind of semi-proactor interfaces. In my opinion this is fine, many people have been looking forward to this implementation, but I think the main problem is going to be the callbacks.
In a high-frequency environment where we call OnReadable more than 1M times per second, we are going to create a function object every time (because closures are structs), which will be allocated in the heap because it'll escape the stack.
I have implemented myself a library that uses a true pro-actor pattern (not open-sourced yet), and it performs well, but not better than Go. Mainly because the heap allocations increase and decrease per second (given the above callback problem).
Libraries that implement a pro-actor pattern might improve the TLS & HTTP/2 performance given that the tls package has locks everywhere and HTTP/2 in Go is full of channels and mutexes (not criticism, both libraries are very well done).
Summary: If this issue goes forward and a PR is presented, are we going to see closures allocated in the stack? In C++, lambdas are just objects (like in Go) but they are moved from stack to stack, afaik they are never allocated in the heap (unlike Go).
I know that callbacks might not allocate actual heap, they just reuse memory because if the object is less than 32KiB mallocgc takes memory that is available, but still reducing performance.

I don't think we need callbacks. I think the following would do:
type HServer struct {
	addr string
	conns map[net.Conn]struct{}
	done context.Context
	addQueue chan net.Conn
	closeQueue chan net.Conn
}

func (s *HServer) Run() {
	lis, _ := net.Listen("tcp", s.addr)

       go s.loop()

	for {
		if conn, err := lis.Accept(); err != nil {
				s.addQueue <- conn
		}
	}
}

func (s *HServer) loop() {
	for {
		select {
			case <-s.ctx.Done():
				for conn := range s.conns {
					conn.Close()
				}
				s.conns = nil
				return
			case conn := <-s.addQueue:
				s.conns[conn] = struct{}{}
			case conn := <-s.closeQueue:
					conn.Close()
					delete(s.conn, conn)
			default:
				for conn := range s.conns {
					if conn.CanRead() { <----- this is what it's all about, right?
						go s.handle(conn)
					}
				}
		}
	}
}

func (s *HServer) handle(conn net.Conn) {
	buf := bytes.NewBuffer(nil)
	if _, err := buf.ReadFrom(conn); err != nil {
		println(err)
		s.closeQueue <- conn
		return
	}
	// ... do something with the data
}

I don't think we need callbacks. I think the following would do:
...

Why would you do that in an example? You don't want to condition your handle. In your code:
for conn := range s.conns {
	if conn.CanRead() { <----- this is what it's all about, right?
		go s.handle(conn)
	}
}

The conn.CanRead will condition the handling, so, you don't handle the conn? Ah no, you handle it but after some loops... Wasting CPU polling the channels and then falling through the default.
But that's why Go is slow. If you use one channel is fine, if you are polling multiple channels and expecting the software to be latency sensitive you are not going to achieve your goals. That's why libraries like fasthttp or evio were created. fasthttp uses standard Go but replaces the http package, and evio is just a replacement of net (with a reactor pattern). The problem that people want to solve is handling as many connections as possible with the lowest latency possible. And in standard Go that's not possible, mainly because you have a coroutine per client (maybe 2, one for reading and another for writing).
The idea is to have a reactor patter, so:
func broadcast(msg []byte, conns []net.Conn) {
  for _, c := range conns {
    c.OnWritable(func() {
      c.Write(msg)
    })
  }
}

But the problem with the above is that c.Write might block. Because let's say msg is 65536 bytes, and the OS buffer is only able to hold 1024 bytes now. We'll write 1024 bytes and then... what? Some Go libraries might fail with an ErrShortWrite error (see bufio). Therefore the c.Write(msg) must block. UNLESS you can specify to not block with something like c.WriteNonblock(msg), but that would require A LOT of changes.
Given the above, and my previous comment, I think implementing this in std Go would be difficult. Not difficult, but it would require a lot of changes to the interfaces and existing data structures. It'd be nice to have a reactor patter library supported by Go std that uses Go data structures because they can make use of compiler directives to not be stored in the heap (See this). So suggesting a package like net/reactor wouldn't be that crazy, I think. I'd happily work on it

You can simply but continue into the default statement to check the channels again once you handle all connections. But that was just an example. As for handling - the main point is to use goroutine only if there is data on the connection to be handled and not to have a dedicated goroutine per each connection. so if you have 100k connections but only 100 are active at any one moment, that is savings of 99 900 goroutines * 2kB(minimum) = 199 800 kB or roughly 200 MB memory they would take + context switching.

@ivanjaros
If conn.CanRead() is blocking interface, one connection's blocking makes all the other connections wait for long in the loop, even if they have been readable already.
If conn.CanRead() is non-blocking interface, that for loop will cause cpu 100%.

true

..although you could have have a connection collection primitive that would handle this internally and would work in a channel-like manner. so handing a single connection could work as blocking Read() and non-blocking CanRead(), neither would break existing api, and this new collection would be a new primitive extending the net package and which would allow to add and remove connection and block until one of the connections is readable.

..although you could have have a connection collection primitive that would handle this internally and would work in a channel-like manner. so handling a single connection could work as blocking Read() and non-blocking CanRead(), neither would break existing api, and this new collection would be a new primitive extending the net package and which would allow to add and remove connection and block, with timeout, until one of the connections is readable, in which case it would return it for reading. ...just thinking out loud here.

Too complex. You don't want some things explicitly blocking and others implicitly, although the doc makes a distinction. IMO what people want is 2 different approaches, one more pro-actor that allows you to have one goroutine to handle multiple connections, and another (that already exists) to handle a connection per goroutine (go net). But the problems presented above might be a blocker.

Either way, i just wanted to keep the conversation going since I'd like to write a chat server in near future and this type of functionality will be a must so I will have to use one of the libraries I listed above, but would prefer this to be in the stdlib.
Also I bet proxy like Traefik would get a nice performance gain from this as well. I mean, gnet equals in performance the fastest code out there in c/cpp/rust/...  so the performance is there, in Go. It just needs to be unlocked.

I mean, gnet equals in performance the fastest code out there in c/cpp/rust/... so the performance is there, in Go

@ivanjaros
That's not right.
For HTTP, gnet uses a simple parser that does not implement full features of the HTTP protocol, so its testing code cost much less of cpu than a full-featured HTTP server.
As we know, golang can't get the same performance as c/cpp/rust but goroutine and chan make us to write code easier.

Also I bet proxy like Traefik would get a nice performance gain from this as well. I mean, gnet equals in performance the fastest code out there in c/cpp/rust/... so the performance is there, in Go. It just needs to be unlocked.

Yeah, but the problem with frameworks like gnet or evio is that they are not usable in production. There might be cases where you can use it, even with codecs. They are too complicated to be used in serious environments. A pro-actor pattern library would be more suitable (like boost::asio).
Standard Go is quite performant, just take a look at fasthttp. It is built using standard Go. The only problem with Go is the amount of sync mechanisms that it requires. That's why I said that a pro-actor pattern might be faster for TLS & HTTP/2. Or any protocol that uses streams instead of one single stateless connection.

As we know, golang can't get the same performance as c/cpp/rust in most scenarios, but can get near to c/cpp/rust in some scenarios such as IO, and the most important is: goroutine and chan make us write code easier.

I mean...  I benchmarked fasthttp vs boost::beast quite a lot (in AWS 2xc5n.4xlarge, client and server in a placement group) and Go is able to handle 200K QPS below 5ms (100th percentile) and boost::beast was able to do 200K sometimes below 4.8ms other times below 5.5ms. So there's a lot of variance in boost::beast, whilst in Go I got the same result in all the benchmarks.
Now, the only problem with Go is that the more the connections the less performant the I/O becomes. And that's a limitation you should be aware of while building a system.
So it depends on how your Go program is structured. If it has locks, it doesn't. If uses channels, etc... In my benchmarks I had no locks, just plain HTTP (no TLS) with some caching system in both fasthttp & boost::beast.
I wouldn't say that Go is less performant than C++ and Rust. It also depends on the library that you use. boost::beast is ok, it does scale well, but it is not as latency sensitive as you might think (truth is that you can easily plug in solarflare's onload in boost::asio and get a performance improvement), and Rust is not as different from Go. They also use coroutines and they also need to lock in some scenarios but they have the advantage of not having a GC ("advantage").
Bechmarks are mostly a lie. People prepare their programs for the benchmark in question (like gnet's case). Production ready environments don't need to lie in their benchmarks (fasthttp)

@d2gr
I think proactor is not the key point, but are the num of goroutines, and blocking or non-blocking.
I've tried a lot about a non-blocking based HTTP server, I think we can't gain both high performance and high online together using golang by now:

For std-based frameworks that use net.Conn(blocking IO interface), including fasthttp, the cost of hardware grows fast as the num of connections increases, because they all use at least one goroutine for each connection, including fasthttp. It's hard to reduce the cost for gc, memory and schedule.
For non-blocking frameworks, no matter whether it's reactor or proactor, we need to handle IO and logic in different goroutine pools. That needs more heap escape, more complex async parser, and we can not optimize the buffers as fasthttp does.

@lesismal
I mention pro-actor because it's the easiest way to handle async (unlike async/await, or reactor). I agree that we need less heap and more stack-based structures. I don't know what do you mean with async parser.

@ivanjaros That's not right. For HTTP, gnet uses a simple parser that does not implement full features of the HTTP protocol, so its testing code cost much less of cpu than a full-featured HTTP server.

you are mixing apples and oranges here. nobody is talking about http servers here. gnet is merely networking framework(like ALL the projects mentioned before). what you build on top of it is up to you.

@d2gr
We need to cache half-packet bytes because we can't use ReadFull. The parser and buffer usage logic are much more complex than net.Conn based frameworks.
@ivanjaros Please refer to the reasons I've mentioned here and in previous comments.
Here are some benchmark reports, you can run the test in your own env:
https://github.com/lesismal/go-net-benchmark/issues/1
I get the same level of performance between nbio and gnet, but I support a lot more in nbio than gnet. I tried a lot to optimize the performance and reduce the cost, but I can achieve only balance between performance and cost reduce, can't gain both of them together.
That's not only about  HTTP!
For simple IO logic, we do gain a good performance, but for product env, for the complex logic, the reasons I mentioned make the performance down.
You can try it, I will be glad to see if you can find some way to make more promotions.

@lesismal, @ivanjaros, @d2gr: it isn't clear to me how the above discussion relates to the feature proposed in this issue. For off-topic performance discussions, please start a thread on the golang-dev mailing list or a similar venue outside of the issue tracker.

@bcmills Sorry for spamming a bit, but my comments where related to the issue. At least this one https://github.com/golang/go/issues/15735#issuecomment-1244473297

@bcmills
Sorry about that.
But I think the CanRead does related to non-blocking interfaces and performance. If the new feature doesn't consider these points, the new feature will not be useful and should not be added.
If the new feature provides only a CanRead but still blocking Read/Write, there will be problems like @ivanjaros 's for-loop or gobwas/ws.
My previous related comments:

My opinion is that if provide Readable, non-blocking Read/Write interface should be provided together, else Readable will not be useful.

If conn.CanRead() is blocking interface, one connection's blocking makes all the other connections wait for long in the loop, even if they have been readable already. If conn.CanRead() is non-blocking interface, that for loop will cause cpu 100%.

@bcmills
Actually, CanRead interface is just the same thing that 1m-go-websockets and gobwas/ws did. That's the smallest import for event-driven on std's TCPConn, but as we've discussed, it leads to the problem:
https://github.com/gobwas/ws/issues/143
That's the same paroblem as the for-loop-block we discussed in previous comments.
To solve this problem in gobwas/ws, we still need to serve each connection with at least 1 goroutine, which come back to the solution of the current std but is more complex. Then, there's no benefit, seems and performs even worse.
So, if Read/Write interfaces are still blocking mod, I would prefer you maintainers just keep it not changed rather than add a new CanRead interface.
Also that's why I hope if you add CanRead, please add non-blocking Read/Write interfaces together. And then, one more thing, the non-blocking interfaces and seperate goroutine pool make the performance worse than current std if there are not a lot of connections, I think that's should be also considered before this proposal is accepted.
But all right, I will stop discussing performance, I just focus on the CanRead, blocking or non-blocking.

what if the only addition into the stdlib for net.Conn would be read method that would behave like a channel? that way you can decide to wait for data or move elsewhere. so to use my previous example:
type HServer struct {
	addr string
	conns map[net.Conn]struct{}
	done context.Context
	addQueue chan net.Conn
	closeQueue chan net.Conn
}

func (s *HServer) Run() {
	lis, _ := net.Listen("tcp", s.addr)

       go s.loop()

	for {
		if conn, err := lis.Accept(); err != nil {
				s.addQueue <- conn
		}
	}
}

func (s *HServer) loop() {
	for {
		select {
			case <-s.ctx.Done():
				for conn := range s.conns {
					conn.Close()
				}
				s.conns = nil
				return
			case conn := <-s.addQueue:
				s.conns[conn] = struct{}{}
			case conn := <-s.closeQueue:
					conn.Close()
					delete(s.conns, conn)
			default:
				for conn := range s.conns {
                                        select {
                                             case data, err := <-conn.ChannelRead():
                                                delete(s.conn, conn) // remove from list
						go s.handle(conn, data) // handle connection with first chunk of data
                                             default:
                                             // continue to next connection
                                        }
				}
		}
	}
}

func (s *HServer) handle(conn net.Conn, data []byte) {
	buf := bytes.NewBuffer(data)
	if _, err := buf.ReadFrom(conn); err != nil {
		println(err)
		s.closeQueue <- conn
		return
	}
	// ... do something with the data
        conn.Write() // send some response
        // return connection back to "queue"
       s.addQueue <- conn
}

just add those functions to net.Conn:
ReadAvailable() bool
WriteAvailable() bool
ReadNonBlocking(b []byte) (n int, err error)
WriteNonBlocking(b []byte) (n int, err error)

@AnimusPEXUS Thanks, but that doesn't address this issue.  This issue is about putting a goroutine to sleep until there is something to read.

@ianlancetaylor determining "if ther's something to read" is system-to-system specific question. for NIX* it's 'select' function. maximum you can do here for nix is make callback or push signal after select returned and if we want select non-blocking, then we have to put socket into non-blocking state. for non-socket/non-unix there only two options: endless loop, constantly checking readiness, or, again, callback if peer can somehow separately indicate 'I have something to read'

I don’t think Go as it stands today is the best choice for this kind of code.
Optimized Rust and C++ servers use a combination of compiler-generated state machines (async/await and co_async/co_await, respectively) and manual memory management to achieve very high performance.  Go does not support either of these features, so achieving similar performance in Go would require essentially writing C with Go syntax.

@AnimusPEXUS There is a reason that this issue is still open.


Refused to display in a frame because it set 'X-Frame-Options' to 'deny'.

Hi,
I've cloned this repository and changed the settings to my Azure AD B2C. It opens the popup but it seems that during the process of acquiring the token, it's raising the error:
Refused to display 'https://login.microsoftonline.com/te/x.onmicrosoft.com/b2c_1_jms-su-si/oauth2/v2.0/authorize?response_type=token&scope=https%3A%2F%2Fx.onmicrosoft.com%2Fdemoapi%2Fdemo.read openid profile&client_id=d4ea3a10-bf58-43c7-9a0d-189a1cacee07&redirect_uri=http%3A%2F%2Flocalhost%3A62592%2F&state=b500b7af-6aee-4d32-8e34-5ec1ead1fa84&nonce=41fff769-56e9-41cf-a559-778dc10026e4&client_info=1&x-client-SKU=MSAL.JS&x-client-Ver=0.1.1&client-request-id=ea9d0c0b-38fd-49cc-921e-4e21176aaede&prompt=none&domain_req=534fe24d-0068-4fe9-84a2-7872f971da75&login_req=0f09664c-2e62-48c2-b26c-adc5acf9debf-b2c_1_jms-su-si&domain_hint=organizations' in a frame because it set 'X-Frame-Options' to 'deny'.
Is that because Chrome updated it's settings or it's a wrong configuration on my side?

I've replaced the scope with the GUID and it's working now.

What is a GUID for b2c?

For anyone else struggling with this...
This iframe is often produced from the url itself producing an error. In my case it was navigating back to my SPA from another url ie. myapp.com/some/other/path, you still need to hard-set your redirectUri to myapp.com.

@ashpr  Did you mean your page that's calling msal.js login functionality was at "myapp.com/some/other/path", but that you needed to set the redirectUri in Azure portal to be your root domain "myapp.com" only?

@blinkybill  It's the redirectUri in your msal configuration because by default if you don't provide a redirectUri, it defaults to window.location.href which will include the /path/.

Just in case it helps someone or maybe someone can correct me:
I had issues with this when changing authority like when using loginRedirect() to trigger edit_profile or reset_password flows. When coming back to starting page the cached (localStorage) access token and idToken had wrong authority values and that caused the access token not beeing fetched with the error like in the title. I resolved it by clearing those wrong values from localStorage... Hackintosh  way.
I would like for someone to set me back on the right path.

Just in case it helps someone or maybe someone can correct me:
I had issues with this when changing authority like when using loginRedirect() to trigger edit_profile or reset_password flows. When coming back to starting page the cached (localStorage) access token and idToken had wrong authority values and that caused the access token not beeing fetched with the error like in the title. I resolved it by clearing those wrong values from localStorage... Hackintosh way.
I would like for someone to set me back on the right path.

We had a similar issue, and the answer was not to change your current UserAgentApplication authority, but instead create a new UserAgentApplication with the new authority.

Just in case it helps someone or maybe someone can correct me:
I had issues with this when changing authority like when using loginRedirect() to trigger edit_profile or reset_password flows. When coming back to starting page the cached (localStorage) access token and idToken had wrong authority values and that caused the access token not beeing fetched with the error like in the title. I resolved it by clearing those wrong values from localStorage... Hackintosh way.
I would like for someone to set me back on the right path.

We had a similar issue, and the answer was not to change your current UserAgentApplication authority, but instead create a new UserAgentApplication with the new authority.

Thank you for this. Actually I had a dream this night to try creating a new instance for edit profile and reset the password :).
I actually started coding and separating different stuff in different components this morning.
One thing I think might help (or even is mandatory) is changing storage for the other instance, but then again having written that maybe the session ID might be enough for msal to differentiate between the two different UserAgentApplication instances in the same storage type.
Also maybe I'll need different redirect pages for login stuff vs edit profile vs reset password, just to supply a different redirect handler because of the mentioned session ID, or maybe I won't again.
Trial and error to win this game. I appreciate the response again, it did shed some light and encouraged me to try the mentioned approach, Thanks!

Just in case it helps someone or maybe someone can correct me:
I had issues with this when changing authority like when using loginRedirect() to trigger edit_profile or reset_password flows. When coming back to starting page the cached (localStorage) access token and idToken had wrong authority values and that caused the access token not beeing fetched with the error like in the title. I resolved it by clearing those wrong values from localStorage... Hackintosh way.
I would like for someone to set me back on the right path.

We had a similar issue, and the answer was not to change your current UserAgentApplication authority, but instead create a new UserAgentApplication with the new authority.

Thank you for this. Actually I had a dream this night to try creating a new instance for edit profile and reset the password :).
I actually started coding and separating different stuff in different components this morning.
One thing I think might help (or even is mandatory) is changing storage for the other instance, but then again having written that maybe the session ID might be enough for msal to differentiate between the two different UserAgentApplication instances in the same storage type.
Also maybe I'll need different redirect pages for login stuff vs edit profile vs reset password, just to supply a different redirect handler because of the mentioned session ID, or maybe I won't again.
Trial and error to win this game. I appreciate the response again, it did shed some light and encouraged me to try the mentioned approach, Thanks!

Here's the code we use for the password reset application.
import { UserAgentApplication } from 'msal';
import { ADB2C_CLIENT_ID, ADB2C_PASSWORD_RESET_AUTHORITY } from '../constants';
import { HubMsalConfig } from '../msal-config'; // Our default MSAL Config.

export const msalPasswordChangeApplication = new UserAgentApplication(
      ADB2C_CLIENT_ID,
      ADB2C_PASSWORD_RESET_AUTHORITY,
      () => {},
      {
        validateAuthority: false,
        redirectUri: HubMsalConfig.redirectUri
      })
);

Using it:
    msalPasswordResetApplication.loginRedirect(ADB2C_SCOPES);

I know what you mean, its been a bumpy ride for us but we're finally there. I've been meaning to blog about all this at some point as I feel the documentation really is lacking.

Just in case it helps someone or maybe someone can correct me:
I had issues with this when changing authority like when using loginRedirect() to trigger edit_profile or reset_password flows. When coming back to starting page the cached (localStorage) access token and idToken had wrong authority values and that caused the access token not beeing fetched with the error like in the title. I resolved it by clearing those wrong values from localStorage... Hackintosh way.
I would like for someone to set me back on the right path.

We had a similar issue, and the answer was not to change your current UserAgentApplication authority, but instead create a new UserAgentApplication with the new authority.

Thank you for this. Actually I had a dream this night to try creating a new instance for edit profile and reset the password :).
I actually started coding and separating different stuff in different components this morning.
One thing I think might help (or even is mandatory) is changing storage for the other instance, but then again having written that maybe the session ID might be enough for msal to differentiate between the two different UserAgentApplication instances in the same storage type.
Also maybe I'll need different redirect pages for login stuff vs edit profile vs reset password, just to supply a different redirect handler because of the mentioned session ID, or maybe I won't again.
Trial and error to win this game. I appreciate the response again, it did shed some light and encouraged me to try the mentioned approach, Thanks!

Here's the code we use for the password reset application.
import { UserAgentApplication } from 'msal';
import { ADB2C_CLIENT_ID, ADB2C_PASSWORD_RESET_AUTHORITY } from '../constants';
import { HubMsalConfig } from '../msal-config'; // Our default MSAL Config.

export const msalPasswordChangeApplication = new UserAgentApplication(
      ADB2C_CLIENT_ID,
      ADB2C_PASSWORD_RESET_AUTHORITY,
      () => {},
      {
        validateAuthority: false,
        redirectUri: HubMsalConfig.redirectUri
      })
);

Using it:
    msalPasswordResetApplication.loginRedirect(ADB2C_SCOPES);

I know what you mean, its been a bumpy ride for us but we're finally there. I've been meaning to blog about all this at some point as I feel the documentation really is lacking.

Thanks for your code. However, I can see that you're using <=0.2.4 version, I am trying to set up using 1.2.0 version. I seems to be even bumpier now :). Your redirect callback is empty, I am trying to handle successful and unsuccessful returns and cover also cancel and hopefully not logOut the user after changing password just update all one the fly. And do the same for edit profile.
A blog is a great ides, I planned on spending 1-2 days tops on this, and after 3 it's still in progress. I tried various react libraries and opted out to do my own msal integration yet still issues.  THANKS


Add R4 copy of EXM124 module

No changes. Just a simple copy to place future changes in.

Codecov Report

Merging #11 into abacus will increase coverage by 5.39%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             abacus      #11      +/-   ##
============================================
+ Coverage     75.98%   81.37%   +5.39%
- Complexity     1972     2048      +76
============================================
  Files            72       72
  Lines         13010    13010
  Branches       1794     1794
============================================
+ Hits           9885    10587     +702
+ Misses         2626     1908     -718
- Partials        499      515      +16

Impacted Files
Coverage Δ
Complexity Δ

...org/mitre/synthea/export/ClinicalNoteExporter.java
72.97% <0%> (-5.41%)
12% <0%> (-1%)

.../main/java/org/mitre/synthea/export/FhirDstu2.java
84.05% <0%> (ø)
97% <0%> (ø)
:arrow_down:

src/main/java/org/mitre/synthea/engine/State.java
91.69% <0%> (+0.17%)
17% <0%> (ø)
:arrow_down:

...c/main/java/org/mitre/synthea/export/Exporter.java
43.63% <0%> (+0.36%)
46% <0%> (ø)
:arrow_down:

...in/java/org/mitre/synthea/world/agents/Person.java
85.08% <0%> (+0.4%)
106% <0%> (+1%)
:arrow_up:

.../main/java/org/mitre/synthea/engine/Generator.java
91.94% <0%> (+0.42%)
55% <0%> (-1%)
:arrow_down:

...ava/org/mitre/synthea/modules/LifecycleModule.java
91.97% <0%> (+1%)
126% <0%> (+3%)
:arrow_up:

...c/main/java/org/mitre/synthea/export/FhirStu3.java
89.54% <0%> (+1.17%)
142% <0%> (+3%)
:arrow_up:

...ain/java/org/mitre/synthea/export/CSVExporter.java
94.78% <0%> (+1.95%)
69% <0%> (+2%)
:arrow_up:

src/main/java/org/mitre/synthea/export/FhirR4.java
83.82% <0%> (+2.14%)
170% <0%> (+9%)
:arrow_up:

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f809681...38e2cb9. Read the comment docs.


fix: sender signAndSendTransaction response type change

Description
The PR is to fix the incompatible signAndSendTransaction response type with Sender v1.0.0 and later version.
Closes #392
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings

Type of change. This type of change is the main reason for the PR.

[x] FIX - a PR of this type patches a bug.
[ ] FEATURE - a PR of this type introduces a new feature.
[ ] BUILD - a PR of this type introduces build changes.
[ ] CI - a PR of this type introduces CI changes.
[ ] DOCS - a PR of this type introduces DOCS improvement.
[ ] STYLE - a PR of this type introduces style changes.
[ ] REFACTOR - a PR of this type introduces refactoring.
[ ] PERFORMANCE - a PR of this type introduces performance changes.
[ ] TEST - a PR of this type adds more tests.
[ ] CHORE - a PR introduces other changes than the specified above.

Breaking changes

[x] BREAKING CHANGE - SPECIFY: The change will work for Sender versions after v1.0.0, but is not compatible with versions before 1.0.0.
[ ] NO BREAKING CHANGE - this PR doesn't contain any breaking changes and it's backwards compatible

@AmazingBeerBelly please also help review the code from Sender side.

Looks good to me!

Hey, @think-in-universe thank you for submitting this PR.
Can you please make this PR in the dev branch?

Can you please make this PR in the dev branch?

sure. submitted here: https://github.com/near/wallet-selector/pull/398
feel free to close the this PR.


Add PartiQL support for DynamoDB

Add PartiQL support for DynamoDB

Thanks, @donatoaz  Appreciated!

just warming up... ;)
Donato Azevedo
On Tue, Nov 24, 2020 at 10:48 AM Victor GRENU<EMAIL_ADDRESS>wrote:

Thanks, @donatoaz https://github.com/donatoaz Appreciated!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zoph-io/awscon-onepager/pull/3#issuecomment-732983287,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA7EJ76KRYMMDLGLK25SCDSRO2R7ANCNFSM4UA5BJ5Q
.


Diagram paths corrected

closes #169

Hi Erwan,
1. Is it possible to add indicative parameters to the tables?

2. Is it possible to increase the fond of the text (parameters,etc.) in the block diagrams? It appears too small in my case.

Dear Lampros,
Your comments pertain to the branch 236 (https://github.com/CRESYM/colib0.github.io/pull/243).

The present pull request deals only with the paths to the block diagrams, which are currently not visible on the website.


Fix warning App_name

Overview
App_name is declared twice.
If .xcodeproj is not found, end with error
Related tickets

LGTM 🏖


feat: fixed class vs className incorrect use

class prop bad use fix

fixed class vs className props incorrect use in the nextjs templates's files

changed class to className prop in:

templates/frontend/next-app/src/app/hello-components/page.js
templates/frontend/next-page/src/pages/hello-components/index.js
templates/frontend/vanilla/src/components.js files

Note: Sorry for the double quotes it's my lint doing some messy trick I will try to fix it

I don't understand why Actions are not being trigered, but I am sure this will break the Check-Lint Action

Stagnated and solved on #2133


Bug fix for import('vscode-uri').default

Fix for issue #81 and
#88
Fresh wdio-vscode-service project is not starting because of import('vscode-uri').default
Error message:
TypeError: Cannot destructure property 'URI' of '(intermediate value).default' as it is undefined

The solution has used already at @wdio/utils/build/utils.js:204
The import that was changed here has been working without .default 11 month ago,
before the change 1562cd1

Fixed failing tests

hey @christian-bromann could you please approve the workflow, the CI tests have been fixed and should pass now

The failure on CI for test:e2e:run seems to doesn't relate to my changes
2023-11-16T01:08:24.540Z INFO @wdio/cli:launcher: Run onPrepare hook
2023-11-16T01:08:24.543Z INFO wdio-vscode-service/launcher: Fetch releases from https://update.code.visualstudio.com/api/releases/stable
2023-11-16T01:08:24.628Z DEBUG @wdio/cli:utils: Finished to run "onPrepare" hook in 87ms
2023-11-16T01:08:24.629Z INFO @wdio/local-runner: Shutting down spawned worker
2023-11-16T01:08:24.880Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2023-11-16T01:08:24.881Z INFO @wdio/local-runner: shutting down
SevereServiceError in "onPrepare"
SevereServiceError:
A service failed in the 'onPrepare' hook
SevereServiceError: Couldn't fetch latest VSCode: read ECONNRESET
    at VSCodeServiceLauncher._fetchVSCodeVersion (file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/dist/launcher.js:196:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async VSCodeServiceLauncher._setupChromedriver (file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/dist/launcher.js:130:25)
    at async VSCodeServiceLauncher._setupVSCodeDesktop (file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/dist/launcher.js:110:72)
    at async VSCodeServiceLauncher.onPrepare (file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/dist/launcher.js:47:17)
    at async file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/node_modules/@wdio/cli/build/utils.js:[43](https://github.com/webdriverio-community/wdio-vscode-service/actions/runs/6884575889/job/18728289538?pr=93#step:6:44):17
    at async Promise.all (index 0)
    at async Launcher.run (file:///Users/runner/work/wdio-vscode-service/wdio-vscode-service/node_modules/@wdio/cli/build/launcher.js:94:13)

when running locally it downloads the versions:
INFO wdio-vscode-service/launcher: Download Chromedriver (v114.0.5735.90)
INFO wdio-vscode-service/launcher: Download VSCode binary (1.84.2)

I guess on CI it tries to download the same

The issue relates to the fact that Chrome moved to Chrome for Testing and has changed the way how driver are supposed to be downloaded. This service should rely on the wdio-chromedriver-service anymore and instead should rely on WebdriverIO to download the right driver. I haven't had time to make that change yet.

Thank you @christian-bromann !


allowedAttributes option has wrong TypeScript typings

The documentation says to set allowedTag and allowedAttributes to empty arrays ([]) if we don't want to allow any tags. But the @types/sanitize-html declaration is:
  interface IOptions {
    allowedAttributes?: { [index: string]: string[] } | boolean;
    // remaining props...
  }

So according to the types declaration, the value for allowedAttributes should be an empty object.
So, if I'm not mistaken, the IOptions interface declaration should be:
  interface IOptions {
    allowedAttributes?: { [index: string]: string[] } | boolean | [];
    // remaining props...
  }

Good catch on the docs, fixed that in the README for the next release.
On Fri, Oct 19, 2018 at 11:27 AM wujekbogdan<EMAIL_ADDRESS>wrote:

The documentation says to set allowedTag and allowedAttributes to empty
arrays ([]) if we don't want to allow any tags. But the
@types/sanitize-html declaration is:
interface IOptions {
allowedAttributes?: { [index: string]: string[] } | boolean;
// remaining props...
}
So according to the types declaration, the value for allowedAttributes
should be an empty object.
So, if I'm not mistaken, the IOptions interface declaration should be:
interface IOptions {
allowedAttributes?: { [index: string]: string[] } | boolean | [];
// remaining props...
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/punkave/sanitize-html/issues/259, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB9fawWWrVy6TaAADCL29u_nM6iSGFAks5ume9egaJpZM4XwlWe
.

--
Thomas Boutell, Chief Software Architect
P'unk Avenue |<PHONE_NUMBER>  |  punkave.com


DBSCAN

This PR wraps the DBSCAN clustering algorithm from sklearn into our code and address issue #25.
Note that I moved DBSCAN into the extrapolation folder since I believe it's conceptually similar to KMeans i.e. create clusters with the intention of holding out some clusters to evaluate validation and testing performance. Technically, these clusters could also be used to create interpolative splits by greedily sampling one data point from each cluster, but I think that's conceptually similar to kennard stone and empirically, random splits often achieve very even distributions between train, val, and test even without the explicit requirement that it be exactly equal (greedy) sampling.
test_molecules.py is failing locally. I tried debugging but haven't gotten to the bottom of that one just yet. I hypothesize the QM9 molecules are too similar (or at least the ones that are downsampled during the test) and/or perhaps the default DBSCAN hyperparameters don't work well for these molecules and/or featurization so it seems to not create the proper number of clusters or properly sized clusters...feel free to take a stab at it. I'll be in the office this afternoon too.
Looking forward to you feedback!

Thanks for reviewing, fixing, and merging!


Refactor to make Analysis-tab and Data treatment-tab independent of each other

ACGui-class is a mess, and this can be helped by splitting the two tabs into separate classes

Pass the main window as a variable to each of the tabs, so that they can access each other...


Disable CGO for all platforms build (fixes #26)

Set the environment variable CGO_ENABLED=0  to make the compiled binaries free of GLIBC dependencies.
This fixes: #26

If this actually fixes the mentioned issue, I'd love to see this merged.
I am currently facing the linked issue.


VCD support for signal generator

Current signal generator is hard coded to PRBS-31, we should support arbitrary waveforms

Closing as refactoring made this redundant: IBISDriverFilter can now be applied to arbitrary digital waveforms including but not limited to a PRBS or a CSV/VCD import.


Add angular mock passThrough extern

$httpBackend.RequestHandler.passThrough is available in angular mock 1.4
but never made it to this extern file for some reason. This commit takes
care of that.
Refer to
https://code.angularjs.org/1.4.0/docs/api/ngMockE2E/service/$httpBackend
for more details.

@mprobst LGTM

LGTM.

@selkhateeb Thanks for the PR!


Update Dependencies & Documents 📖

Hello 👋
I think this is a reliable library that is still working.
Please update the project dependencies (compose 1.2.1) and write document for the public functions. At first, I did confused.
Recent updates have stabilized the lazyVerticalGrid, so the demo is outdated.

Hi I'm making a simpler implementation of this library please check my repositories I'm still working on it but the target is to make a more customizable transition to support any kind of element using it's modifiers, for now we can only perform the offset transition the next step will be the scale and then a progress, to manually perform the transition, and so on, also I have a navigation for the shared element transitions but I didn't implemented on the new yet! Maybe tomorrow...

Hello, I don't see your shared-elements on your github, are they still in development?

Hello, will the shared element be updated to have material3 support for transition?


complete getUser w/ robots and sessions gives duplicate GQL fragments error

getUser does not currently populate it's .robots of .robots.sessions fields.  But when doing the fix in users.toml:
query QUERY_GET_USER (
  $userLabel: EmailAddress!
  ) {
    users (where: {label: $userLabel}) {
    ...FRAGMENT_USER
    robots {
      ...FRAGMENT_ROBOT
      sessions {
        ...FRAGMENT_SESSION
      }
    }
  }
}

A new duplicate fragments error with gql occurs.  See error stack below...
The error relates to BLOBENTRY GQL fragment being loaded multiple times in the nested user-robot-sessions GQL above.  There is chatter on this issue for Python's GQL in general, e.g.:

Likely solved similar to JS: https://github.com/dotansimha/graphql-code-generator/issues/4684#issuecomment-864523128
Dropping interpolated for symbolic nesting only: https://github.com/dotansimha/graphql-code-generator/pull/6018#issuecomment-848557286
Next best solution: https://gist.github.com/lanwin/bdc33d0bf291851bda48d17f98b98ff2
Original solution: https://github.com/apollographql/graphql-tag/pull/278
More info: https://github.com/dotansimha/graphql-code-generator/issues/3063

Note the query above works in playground, and also already released via SDK.jl.

In : usr = await getUser(ngc, userLabel)
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/__main__.py", line 58, in runcode
    return future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
  File "/home/dehann/software/NavAbility/NavAbilitySDK.py/src/navability/services/user.py", line 34, in getUser
    resp = await navAbilityClient.query(
  File "/home/dehann/software/NavAbility/NavAbilitySDK.py/src/navability/entities/navabilityclient.py", line 81, in query
    result = await client.execute(options.query, options.variables)
  File "/home/dehann/.local/lib/python3.8/site-packages/gql/client.py", line 408, in execute
    result = await self._execute(document, *args, **kwargs)
  File "/home/dehann/.local/lib/python3.8/site-packages/gql/client.py", line 393, in _execute
    self.client.validate(document)
  File "/home/dehann/.local/lib/python3.8/site-packages/gql/client.py", line 117, in validate
    raise validation_errors[0]
graphql.error.graphql_error.GraphQLError: There can be only one fragment named 'FRAGMENT_BLOBENTRY'.

GraphQL request:13:10
12 |
13 | fragment FRAGMENT_BLOBENTRY on BlobEntry {
   |          ^
14 |   id

GraphQL request:49:10
48 |
49 | fragment FRAGMENT_BLOBENTRY on BlobEntry {
   |          ^
50 |   id

Just just looks to be a duplicate of #112


Updated comment to include GitHub issue # in postActivity() method of…

Changelog Entry
Updated comment to include GitHub issue # in postActivity() method of DirectLineSpeech class, by @stevkan in PR #3542
Description
'TODO' comment doesn't include the Github issue #. PR adds the issue # to the comment.
Design

Specific Changes
Added GitHub issue number to comment on line 74, in DirectLineSpeech.js file.
Review Checklist

This section is for contributors to review your work.

[ ] Accessibility reviewed (tab order, content readability, alt text, color contrast)
[ ] Browser and platform compatibilities reviewed
[ ] CSS styles reviewed (minimal rules, no z-index)
[ ] Documents reviewed (docs, samples, live demo)
[ ] Internationalization reviewed (strings, unit formatting)
[ ] package.json and package-lock.json reviewed
[ ] Security reviewed (no data URIs, check for nonce leak)
[ ] Tests reviewed (coverage, legitimacy)

@stevkan sorry for the trouble, could you remove the CHANGELOG entry? We've started trying to not include 'internal' updates that won't affect customers.

@corinagum, are you referring to the CHANGELOG.md entry in the PR description, in the actual PR, or both?

Just the CHANGELOG.md

ping

Closing due to inactivity


[stable/minio] Support for Minio GCS gateway

Allows to use this chart optionally as Minio GCS gateway for Google Cloud Storage. This is similar to the already provided Azure gateway configuration parameter.

/lgtm

/assign


[CP][stable channel] revert previous cherry-pick for DDC

commit(s) to merge: TBD (https://dart-review.googlesource.com/c/sdk/+/165873 in the CQ, which reverts 53895b2)
merge instructions: expect a clean merge
What is the issue: A fix of a previous issue that we cherry-picked introduced additional problems.
What is the fix: Revert the previous cherry-pick (#43453) and work on an alternative solution to the original problem.
Why cherrypick: We think removing the old cherry-pick is a safer path forward.
The bug in question has existed for years, only recently a flutter package with unique code-generation patterns triggered the bug (https://github.com/flutter/flutter/issues/64011). The fix revealed new failures like (https://github.com/flutter/flutter/issues/66859) that affect simpler packages and we are afraid the scope of the failures could potentially be a lot wider. To prevent further regressions and disruptions, we prefer to remove the earlier "fix" until we find a proper solution.
Risk: Low, this brings the behavior back to what it was before.
Link to original  issue(s):
https://github.com/flutter/flutter/issues/64011 - the original problem
https://github.com/flutter/flutter/issues/66859 - an example new problem from the earlier fix
The intent is to include this in the next 2.10.0 stable hotfix release
/cc @kevmoo @mit-mit @whesse @athomas @vsmenon @franklinyow @Markzipan

Is this a revert of #43453 ?
@sigmundch Please fix the title.  I can approve this after the CL merged

Precisely.
Sorry I missed the title, I just updated it.

lgtm

Approved

close


mention in docs that we cannot send notifications to users, until status bar is enabled.

Lack of information in docs, it would be helpful if you could mention that enable status bar to send user notifications.

⚠ Do not edit this section. It is required for imodeljs.github.io ➟ GitHub issue linking

LINK: Notifications-and-Messages---iTwin.js-link-to-docs
TAG: imodeljs-docs-question
SOURCE: https://www.itwinjs.org/learning/ui/appui-react/notifications/

@YashodipD Could you elaborate on what you're looking for?
@raplemie

I would like to see some note somewhere in notifications page which says without enabling status bar, we cannot show user notifications. this note might be trivial, but it will save some time of beginner developers.


Detect Light/Dark Theme

Add support for automatically switching between light and dark modes based on preferred colour scheme as reported by browser.
Resolves #133.
Behaviour:

default to dark theme with no automatic switching
the scheme switcher cycles between light, dark, and auto modes
the scheme switcher shows which mode is active by displaying a sun (light mode), moon (dark mode) or adjust (auto mode) icon

Testing:
I've "tested" this by making the appropriate changes in the Firefox developer tools... Is it even possible to run a local instance without the secrets?
Next steps:
It might be worth checking the browser support for the MediaQuery API. Specifically, MediaQueryList.addEventListener isn't supported in IE and would need to be replaced by MediaQueryList.addListener, which is deprecated in most browsers.

Yeah, that would just be a matter of changing one boolean. I did it this way to make the default behaviour the same as before but I would also prefer auto add default.

For now, I wrote something similar into a userscript: https://greasyfork.org/en/scripts/444353-tum-live-auto-dark

Hi @just-max
I'm super sorry I let this go stale :/ I'd still like to merge this feature but I'm afraid this PR is a little outdated by now. I had a look at https://tailwindcss.com/docs/dark-mode#toggling-dark-mode-manually, imo their code is very clean and i love the look of the preference switch in the top right corner.

Also they recommend putting the theme related code into the DOM to avoid flickering when the page is loaded.
If you want you can pick this up, otherwise I'll implement it :)

Hey! No worries, it was just a feature I found myself missing :)
I could put something together.

For the dropdown I would adapt the notification dropdown (for consistency)
FA Icons: I would use sun, moon, and desktop
I like how simple the implementation is too, but it means that anyone who has used the site before will not benefit from the automatic theme changing. I would suggest changing the local storage key to something else (theme) and migrating appropriately.

Sounds great! :100: