Protocol error: Max outbound streams is 500, 500 open

Hello. We are using hyper to send push messages using Apple Push Notification Service. Some times we get TooManyStreamsError: Max outbound streams is 500, 500 open when sending HTTP2 request.
The code we use is quite simple:
class APNS(object):
    def __init__(self, cert, key, dev=False, alt_port=False):
        self.server = 'api.development.push.apple.com' if dev\
            else 'api.push.apple.com'
        self.port = 2197 if alt_port else 443

        self._ssl_context = init_context(cert=(cert, key))
        self._conn = HTTP20Connection(
            host=self.server,
            port=self.port,
            force_proto='h2',
            ssl_context=self._ssl_context)

    def send(self, push_token, payload,
             id_=None, expiration=None, priority=None, topic=None):
        headers = HTTPHeaderMap()
        if id_:
            headers['apns-id'] = id_
        if expiration:
            headers['apns-expiration'] = expiration
        if priority:
            headers['apns-priority'] = priority
        if topic:
            headers['apns-topic'] = topic

        stream = self._conn.request(
            'POST',
            '/3/device/' + push_token,
            headers=headers,
            body=json.dumps(payload, cls=PayloadEncoder)
        )
        return stream

    def feedback(self, stream_id):
        with self._conn.get_response(stream_id) as resp:
            if resp.status != 200:
                body = json.loads(resp.read())
            else:
                body = None
            return (
                resp.status,
                resp.headers.get('apns-id', [None])[0],
                body)

We use gevent celery worker to send pushes with maximum of 50 greenlets, so as I understand, there should not be more then 50 opened streams in every moment because we are closing stream after getting response.
Please explain what does this error mean and how we can work around it?

So there are two possibilities. One is that, while you might only allow 50 greenlets, the send greenlets might be starving out the feedback greenlets. This admittedly only happens if the size of the responses is > 64kB.
The other risk is that the non-200 errors aren't having their bodies consumed. Generally you should either read() the stream or reset it if you don't want it: otherwise, there's a risk of leaving the stream open.
However, at this point I think turning on logging is likely to be the most initially useful first pass. Do you mind enabling debug logging for me and seeing what the results produce?

(Side note: this is some extremely well-factored code, I approve highly!)

Thank you for reply and for hyper )

So there are two possibilities. One is that, while you might only allow 50 greenlets, the send greenlets might be starving out the feedback greenlets. This admittedly only happens if the size of the responses is > 64kB.

currently, we executing send() and feedback() in single celery task sequentially, so number of sends and feedbacks should be equal

However, at this point I think turning on logging is likely to be the most initially useful first pass. Do you mind enabling debug logging for me and seeing what the results produce?

I enabled debug for hyper and will post logs here when error will be reproduced

(Side note: this is some extremely well-factored code, I approve highly!)
Thank you!

TooManyStreamsError was not reproduced yet, but I have some debug logs. There are a lot of messages like this:
2016-08-03 13:05:00,770:DEBUG:hyper.http20.connection:recv for stream 0 with set([1, <many values from 1 to 16293> ])  already present

The last number (16293) and "recv for stream" number (0) are not changed within a number of last messages in logs. Is it normal situation?
I uploaded the whole log to dropbox, it's quite big (500 kb).
https://www.dropbox.com/s/a9rtr511dthfxbc/pcelery_async-stdout---supervisor-eUkFtI.log.gz?dl=0

That's somewhat interesting, but not that important in this case I think.
Those messages appear if a frame has been received on a given stream, but that stream has not yet attempted to read its frames. It exists as a guard against the possibility of having a stream stuck behind a read lock.
However, if a stream never has to emit a read: that is, if all the data that it was expecting got pushed to it by other streams reading, then it will never get evicted from that set. Its mere presence in that set is not normally that useful a debugging piece of information. It looks like we have a bug where we don't evict members from that set if a stream is cleanly closed, which would be good to fix (see #265). However, it's unrelated to this problem.
The fact that this is not reproduced regularly suggests, however, that it's a somewhat intermittent problem. Can you try regularly logging out the value of len(HTTP20Connection.streams)? That will help me get a feel for how many streams we think are open at one time.

We enabled logging of streams count and currently maximum what we have is 12 streams:
2016-08-04 13:39:43,959:DEBUG:backend.push.push_apns2:Have 12 opened streams

It seems strange because it rather far from 500 ).
Is it possible that TooManyStreamsError shows incorrect number of max_outbound_streams?

@binrush I doubt it. A quick glance at the relevant code doesn't seem wrong.
I wonder if this was only encountered in a weird error condition scenario.

We decided to decrease grenlets number to 10. Currently we have following in logs:
2016-08-07 00:54:07,361:ERROR:celery.worker.job.on_timeout:Hard time limit (900s) exceeded for backend.tasks.async.send_push[378a60a9-54aa-460b-be66-79c875ff6239]
2016-08-07 00:54:07,647:ERROR:backend.push.push_apns2.apns_send_msg:Too many opened streams, 168
2016-08-07 00:54:15,376:ERROR:celery.worker.job.on_timeout:Hard time limit (900s) exceeded for backend.tasks.async.send_push[0e46e1c8-65fb-4d40-83ad-7fc9cfb4f0ef]
2016-08-07 00:54:15,683:ERROR:backend.push.push_apns2.apns_send_msg:Too many opened streams, 169

So, looks like streams number is much greater then greenlets number

@binrush It's certainly possible. Do you have your "have opened streams" log for this as well?

@Lukasa here it is https://www.dropbox.com/s/ly57q0vfq8r5jtt/Начало работы с Dropbox.pdf?dl=0
But seems there is no any messages from hyper except for one I mentioned above

Uh, I don't think that link is right.

@Lukasa sorry, here is right one https://www.dropbox.com/s/4ibew32vllgqmsv/lastlog.tar.gz?dl=0

Yup, timeouts do seem to be the problem here: it does seem like celery is timing those requests out and they never end up closed. It would be interesting to see why 900s are being taken for some of these requests.

BTW, in celery task we have something like this:
            stream_id = connection.send(token, pl, topic=topic)
            status, apns_id, body = connection.feedback(stream_id)

So seems like execution stucks on getting feedback.
Do you have any suggestions on how we can futher debug it?

@binrush Ideally it would be good to see a traceback from the point where celery killed the worker. Is that going to be available?

@Lukasa currently, no. I'm trying to figure out how to get it and will write here if I have any success with it

Unfortunately, I still did not fugure out, how to get stack trace for Hard time limit (900s) exceeded error, so I made a trace dump of all greenlets in moment when streams count became more then greenlet count. Most greenlets have following trace:
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent
/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/celery/concurrency/gevent.py", line 33, in apply_timeout
    propagate=(Timeout, ), **rest)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/celery/concurrency/base.py", line 36, in apply_target
    ret = target(*args, **kwargs)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task
    return _tasks[task].__trace__(uuid, args, kwargs, request)[0]
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/prince/proj/python/prince-backend/backend/api/api.py", line 256, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/prince/proj/python/prince-backend/backend/tasks/async.py", line 36, in send_push
    extra={'prince': msg.body})
  File "/home/prince/proj/python/prince-backend/backend/push/push_apns2.py", line 86, in apns_send_msg
    stream_id = connection.send(token, pl, topic=topic)
  File "/home/prince/proj/python/prince-backend/backend/push/apyns.py", line 38, in send
    body=json.dumps(payload, cls=PayloadEncoder)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 250, in request
    with self._write_lock:
  File "/usr/lib/python2.7/threading.py", line 174, in acquire
    rc = self.__block.acquire(blocking)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/hub.py", line 606, in switch
    return greenlet.switch(self)

Full log attached
pcelery_async-stdout---supervisor-fWMgHq.log.gz

@Lukasa do you have any suggestions on how we can deal with it?

So right now it seems that we're getting stuck on the write lock. This raises an important question: is it possible that a thread has died holding the lock. Or, worse, timed out? Can you try setting socket.setdefaulttimeout(30) and see if that changes anything?

@Lukasa with this setting, we are getting the following:
Traceback (most recent call last):
 File "/home/prince/proj/python/prince-backend/backend/push/push_apns2.py", line 92, in apns_send_msg
   status, apns_id, body = connection.feedback(stream_id)
 File "/home/prince/proj/python/prince-backend/backend/push/apyns.py", line 43, in feedback
   with self._conn.get_response(stream_id) as resp:
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 293, in get_response
   return HTTP20Response(stream.getheaders(), stream)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/stream.py", line 223, in getheaders
   self._recv_cb(stream_id=self.stream_id)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 744, in _recv_cb
   self._single_read()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 644, in _single_read
   self._sock.fill()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/common/bufsocket.py", line 167, in fill
   count = self._sck.recv_into(self._buffer_view[self._buffer_end:])
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/_sslgte279.py", line 470, in recv_into
   return self.read(nbytes, buffer)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/_sslgte279.py", line 314, in read
   self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/_socket2.py", line 179, in _wait
   self.hub.wait(watcher)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/hub.py", line 627, in wait
   result = waiter.get()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/hub.py", line 875, in get
   return self.hub.switch()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/gevent/hub.py", line 606, in switch
   return greenlet.switch(self)
SSLError: ('The read operation timed out',)

Hrm. That traceback is not terribly helpful, really, though it suggests that we're expecting a HEADERS frame and not getting one within at least 30 seconds. Specifically, in fact, that the connection is dead for 30 seconds. Do you have or can you obtain a tcpdump that shows no receive action on the socket for that 30 second period?

@Lukasa here is tcpdump for single connection
dump.zip

Hrm. Sadly, nothing leaps out from that packet capture. =( This is beginning to get quite frustrating: we may need to start adding some kind of substantial logging to the codebase if we can't get a stacktrace from celery. =(

@Lukasa maybe we can fork hyper, add some debug messages and use forked version in our project?

That certainly works for me. In the first instance the most useful debug logs would be ones that dump out stream creation/destruction information (one log before this line dumping out newly created stream ID, one in each branch of this except block noting what stream ID is getting deleted and whether it's already been deleted), and then a few scattered around this somewhat complex method to see if there are bugs floating around in it.

@Lukasa is this ok https://github.com/binrush/hyper/commit/987c7200fbaa0f750051e5d0c65c77906f5c0244 ?

Some notes left inline in that commit diff.

@Lukasa btw, is it ok to install from development branch? I have some problems installing:
rush@rush-laptop:~$ ~/venv/bin/pip install git+https://github.com/binrush/hyper.git@development
Collecting git+https://github.com/binrush/hyper.git@development
  Cloning https://github.com/binrush/hyper.git (to development) to /tmp/pip-McAADa-build
    Complete output from command python setup.py egg_info:
    error in hyper setup command: Invalid environment marker: python_full_version < "2.7.9"

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-McAADa-build/

@binrush You probably need to update the pip in your virtualenv!

@Lukasa here is log with debug messages we added
pcelery_async-stdout---supervisor-fWMgHq.log.gz

Thanks @binrush, I'll try to deal with this later today.

So the logs paint a bit of a picture here. If we start from timestamp 2016-08-18 06:36:31,134, we have these logs from start of connection to connection failure:
2016-08-18 06:36:31,134:ERROR:backend.push.push_apns2:NEW APNS CONNECTION DEV=False, PID=3264
2016-08-18 06:36:31,308:DEBUG:hyper.http20.connection:Created stream with id 1
2016-08-18 06:36:31,681:DEBUG:hyper.http20.connection:Selected NPN protocol: h2
2016-08-18 06:36:31,682:DEBUG:hyper.http20.connection:recv for stream 0 with set([]) already present
2016-08-18 06:36:31,683:INFO:hyper.http20.connection:Received unhandled event <RemoteSettingsChanged changed_settings:{1: ChangedSetting(setting=1, original_value=4096, new_value=4096), 3: ChangedSetting(setting=3, original_value=None, new_value=500), 5: ChangedSetting(setting=5, original_value=16384, new_value=16384), 6: ChangedSetting(setting=6, original_value=None, new_value=8000)}>
2016-08-18 06:36:31,685:DEBUG:hyper.http20.connection:recv for stream 1 with set([]) already present
2016-08-18 06:36:31,800:INFO:hyper.http20.connection:Received unhandled event <SettingsAcknowledged changed_settings:{2: ChangedSetting(setting=2, original_value=1, new_value=0)}>
2016-08-18 06:36:31,801:INFO:hyper.http20.connection:Received unhandled event <SettingsAcknowledged changed_settings:{}>
2016-08-18 06:36:31,802:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:31,803:DEBUG:hyper.http20.connection:recv for stream 1 with set([]) already present
2016-08-18 06:36:33,913:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[8a722ae5-b58c-4a79-910f-77c2cf2cf306]
2016-08-18 06:36:33,948:INFO:backend.push.push_apns2:Have 1 opened streams
2016-08-18 06:36:33,949:DEBUG:hyper.http20.connection:Created stream with id 3
2016-08-18 06:36:34,184:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:34,186:DEBUG:hyper.http20.connection:recv for stream 1 with set([3]) already present
2016-08-18 06:36:34,865:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[00d42c24-5b01-4c3d-88bc-85f8475d1bc5]
2016-08-18 06:36:35,000:INFO:backend.push.push_apns2:Have 2 opened streams
2016-08-18 06:36:35,001:DEBUG:hyper.http20.connection:Created stream with id 5
2016-08-18 06:36:35,276:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:35,278:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5]) already present
2016-08-18 06:36:37,374:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[c3671e48-6ba8-4304-bc04-4b570ddc2b63]
2016-08-18 06:36:37,466:INFO:backend.push.push_apns2:Have 3 opened streams
2016-08-18 06:36:37,467:DEBUG:hyper.http20.connection:Created stream with id 7
2016-08-18 06:36:37,742:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:37,744:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7]) already present
2016-08-18 06:36:40,850:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[1e138de0-463c-44c2-ada8-4a7039aa0b88]
2016-08-18 06:36:40,876:INFO:backend.push.push_apns2:Have 4 opened streams
2016-08-18 06:36:40,877:DEBUG:hyper.http20.connection:Created stream with id 9
2016-08-18 06:36:41,152:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:41,154:DEBUG:hyper.http20.connection:recv for stream 1 with set([9, 3, 5, 7]) already present
2016-08-18 06:36:41,673:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[6e8cb49c-a0bc-46b5-a02d-44481ee50ab8]
2016-08-18 06:36:41,721:INFO:backend.push.push_apns2:Have 5 opened streams
2016-08-18 06:36:41,722:DEBUG:hyper.http20.connection:Created stream with id 11
2016-08-18 06:36:41,951:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[10151ee2-0268-423b-8d6c-ee937b05077d]
2016-08-18 06:36:41,992:INFO:backend.push.push_apns2:Have 6 opened streams
2016-08-18 06:36:41,993:DEBUG:hyper.http20.connection:Created stream with id 13
2016-08-18 06:36:41,996:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:41,998:DEBUG:hyper.http20.connection:recv for stream 1 with set([11, 9, 3, 5, 7]) already present
2016-08-18 06:36:42,113:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:42,115:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13]) already present
2016-08-18 06:36:42,439:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[c4406cb7-d281-4875-9a9e-1d55baefec05]
2016-08-18 06:36:42,461:INFO:backend.push.push_apns2:Have 7 opened streams
2016-08-18 06:36:42,462:DEBUG:hyper.http20.connection:Created stream with id 15
2016-08-18 06:36:42,736:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:42,737:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15]) already present
2016-08-18 06:36:49,408:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[2c7e6254-e958-4f14-b393-6ee500435582]
2016-08-18 06:36:49,433:INFO:backend.push.push_apns2:Have 8 opened streams
2016-08-18 06:36:49,434:DEBUG:hyper.http20.connection:Created stream with id 17
2016-08-18 06:36:49,579:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[6180d497-b262-41e4-8770-8fb22d2bd804]
2016-08-18 06:36:49,606:INFO:backend.push.push_apns2:Have 9 opened streams
2016-08-18 06:36:49,608:DEBUG:hyper.http20.connection:Created stream with id 19
2016-08-18 06:36:49,709:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:49,711:DEBUG:hyper.http20.connection:recv for stream 1 with set([17, 3, 5, 7, 9, 11, 13, 15]) already present
2016-08-18 06:36:49,825:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:49,827:DEBUG:hyper.http20.connection:recv for stream 1 with set([19, 17, 3, 5, 7, 9, 11, 13, 15]) already present
2016-08-18 06:36:57,330:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_silent_push[38587f04-29ab-4cf7-97ac-c578c17e7d93]
2016-08-18 06:36:57,332:INFO:backend.push.push_apns2:Have 10 opened streams
2016-08-18 06:36:57,333:DEBUG:hyper.http20.connection:Created stream with id 21
2016-08-18 06:36:57,608:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:36:57,610:DEBUG:hyper.http20.connection:recv for stream 1 with set([19, 17, 3, 5, 7, 9, 11, 21, 13, 15]) already present
2016-08-18 06:37:00,764:INFO:backend.push.push_apns2:Have 11 opened streams
2016-08-18 06:37:00,765:DEBUG:hyper.http20.connection:Created stream with id 23
2016-08-18 06:37:01,000:INFO:backend.push.push_apns2:Have 12 opened streams
2016-08-18 06:37:01,003:DEBUG:hyper.http20.connection:Created stream with id 25
2016-08-18 06:37:01,042:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:01,047:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23]) already present
2016-08-18 06:37:01,158:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:01,161:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25]) already present
2016-08-18 06:37:01,550:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[04f210bd-2395-43b2-8c9b-9b98a8f25bd0]
2016-08-18 06:37:01,686:INFO:celery.worker.job:Task backend.tasks.async.send_push[04f210bd-2395-43b2-8c9b-9b98a8f25bd0] succeeded in 0.13423680095s: ObjectId('57b58fcd2077e40ce566b172')
2016-08-18 06:37:02,784:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[0110de0c-9b68-4320-bef7-980d42f6ec00]
2016-08-18 06:37:02,888:INFO:backend.push.push_apns2:Have 13 opened streams
2016-08-18 06:37:02,889:DEBUG:hyper.http20.connection:Created stream with id 27
2016-08-18 06:37:03,164:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:03,165:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27]) already present
2016-08-18 06:37:09,817:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_silent_push[12904cf8-5209-48db-a2de-8dbcae394af0]
2016-08-18 06:37:09,819:INFO:backend.push.push_apns2:Have 14 opened streams
2016-08-18 06:37:09,820:DEBUG:hyper.http20.connection:Created stream with id 29
2016-08-18 06:37:10,095:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:10,097:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29]) already present
2016-08-18 06:37:11,849:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_silent_push[52cf2ba0-590d-4e58-af84-9d07fee0c6ed]
2016-08-18 06:37:11,852:INFO:backend.push.push_apns2:Have 15 opened streams
2016-08-18 06:37:11,853:DEBUG:hyper.http20.connection:Created stream with id 31
2016-08-18 06:37:12,129:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:12,131:DEBUG:hyper.http20.connection:recv for stream 1 with set([3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]) already present
2016-08-18 06:37:17,342:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[57302fa4-3dd3-410e-834d-751fda0dae4b]
2016-08-18 06:37:17,364:INFO:backend.push.push_apns2:Have 16 opened streams
2016-08-18 06:37:17,365:DEBUG:hyper.http20.connection:Created stream with id 33
2016-08-18 06:37:17,640:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:17,642:DEBUG:hyper.http20.connection:recv for stream 1 with set([33, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]) already present
2016-08-18 06:37:19,929:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[b6992a57-8989-498e-add7-13958de2eb6e]
2016-08-18 06:37:20,085:INFO:backend.push.push_apns2:Have 17 opened streams
2016-08-18 06:37:20,086:DEBUG:hyper.http20.connection:Created stream with id 35
2016-08-18 06:37:20,136:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[8441e6b9-e3b3-4b29-a5a1-3364e7935bef]
2016-08-18 06:37:20,603:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:20,605:DEBUG:hyper.http20.connection:recv for stream 1 with set([33, 3, 5, 35, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]) already present
2016-08-18 06:37:20,732:INFO:backend.push.push_apns2:Have 18 opened streams
2016-08-18 06:37:20,733:DEBUG:hyper.http20.connection:Created stream with id 37
2016-08-18 06:37:21,008:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:21,010:DEBUG:hyper.http20.connection:recv for stream 1 with set([33, 3, 5, 35, 7, 9, 11, 13, 15, 17, 37, 19, 21, 23, 25, 27, 29, 31]) already present
2016-08-18 06:37:24,896:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[f7c26994-7d3b-4fb5-b647-33b274c0190e]
2016-08-18 06:37:24,930:INFO:backend.push.push_apns2:Have 19 opened streams
2016-08-18 06:37:24,932:DEBUG:hyper.http20.connection:Created stream with id 39
2016-08-18 06:37:25,207:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:25,208:DEBUG:hyper.http20.connection:recv for stream 1 with set([33, 3, 5, 35, 7, 9, 11, 13, 15, 17, 37, 19, 21, 23, 25, 27, 29, 39, 31]) already present
2016-08-18 06:37:34,120:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[a5295650-d994-44ab-8e70-0b3f3a10bb87]
2016-08-18 06:37:34,599:INFO:backend.push.push_apns2:Have 20 opened streams
2016-08-18 06:37:34,599:DEBUG:hyper.http20.connection:Created stream with id 41
2016-08-18 06:37:34,875:DEBUG:hyper.http20.connection:_single_read(): success
2016-08-18 06:37:34,877:DEBUG:hyper.http20.connection:recv for stream 1 with set([33, 3, 5, 35, 7, 9, 41, 11, 13, 15, 17, 37, 19, 21, 23, 25, 27, 29, 39, 31]) already present
2016-08-18 06:37:39,004:INFO:celery.worker.strategy:Received task: backend.tasks.async.send_push[156fb258-23fd-4fba-8308-ff133a518119]
2016-08-18 06:37:39,029:ERROR:backend.push.push_apns2:Too many opened streams, 21

What seems notable to me here is that we're repeatedly spinning trying to read for stream 1, but never actually finding data for it. However, none of the reads for the other streams progress: the data is read out for them and passed to the various objects but the response for them never actually gets read. In the meantime, further tasks keep coming in and sending their request out, eventually leading to too many concurrently open streams being available.
So, let's try some things.
Firstly, can you add logging to your code to note every time you get a response, and what stream ID it's for? I'm interested to see if we're ever getting a response for stream 1 in this kind of situation. I should note that this doesn't always happen: sometimes things seem to move forward.
Secondly, can you really confirm that too many greenlets aren't running? We're ending up with 21 open requests even though none ever completed as far as I can see, which strongly suggests that your rate limiting of greenlets isn't working. The logs seem to suggest that as soon as a task shows up it immediately starts executing. If we can fix that then we'll probably be a lot closer to getting things to behave properly.

@Lukasa thank you for spending time on this

Firstly, can you add logging to your code to note every time you get a response, and what stream ID it's for? I'm interested to see if we're ever getting a response for stream 1 in this kind of situation. I should note that this doesn't always happen: sometimes things seem to move forward.

I've added log message after calling feedback()

Secondly, can you really confirm that too many greenlets aren't running? We're ending up with 21 open requests even though none ever completed as far as I can see, which strongly suggests that your rate limiting of greenlets isn't working. The logs seem to suggest that as soon as a task shows up it immediately starts executing. If we can fix that then we'll probably be a lot closer to getting things to behave properly.

Sorry, I forget to note. Currently, we have celery worker with 50 greenlets running, and when opened stream count reaches 21, we creating new connection. This is a kind of workaround to make it work without manual restarts. Mybe we can temporary disable this if it can help debugging.

@binrush I see, that's interesting. Are you catching the exception to detect that case?

you mean "Too Many Streams" case? No, we just check if conn.streams is greater then 20

@binrush Interesting. So why do we still see the problem in the logs?

@Lukasa what exact problem do you mean? Too Many Connection? It's just debug message, actually currently we do not reach real TooManyStreams exception because we are recreating connection when streams count become growing
    if len(connection._conn.streams) > 20:
        logger.error("Too many opened streams, %s", len(connection._conn.streams))
        connection = get_connection(force=True)
    else:
        logger.info("Have %s opened streams", len(connection._conn.streams))

Oh, I see. =) That makes more sense.

@Lukasa sorry for mislead you )

That's fine. =D So yes, in that case we're mostly just interested in what is going on with our apparently stuck stream.

Here is new log with debug message you asked. Opened streams number starts growing at 2016-08-29 03:00:01,179:
2016-08-29 03:00:01,178:INFO:backend.push.push_apns2:Have 0 opened streams
2016-08-29 03:00:01,179:DEBUG:hyper.http20.connection:Created stream with id 1279
2016-08-29 03:00:01,180:DEBUG:hyper.http20.connection:recv for stream 1279 with set([1, 1091, 3, 5, 7, 9, 11, 13, 15, 1041, 1271, 19, 21, 1047, 25, 27, 29, 31, 33, 1059,
37, 1063, 41, 1031, 45, 1071, 49, 51, 1077, 55, 1161, 57, 1083, 61, 1093, 1087, 1061, 65, 1035, 1025, 69, 1095, 73, 75, 77, 79, 1105, 83, 85, 1111, 89, 1267, 1039, 93, 95
, 1207, 1121, 99, 1125, 103, 105, 107, 109, 111, 113, 115, 1163, 117, 119, 121, 123, 125, 127, 1027, 129, 131, 133, 1217, 1057, 137, 139, 143, 145, 147, 1173, 151, 153, 1
179, 157, 159, 1185, 1051, 165, 167, 1203, 169, 171, 173, 175, 177, 179, 181, 183, 185, 187, 1029, 189, 191, 193, 195, 197, 199, 201, 203, 205, 207, 209, 211, 213, 1239,
217, 219, 221, 223, 1147, 227, 229, 1233, 233, 1259, 1167, 1261, 239, 1099, 243, 245, 1139, 247, 249, 251, 1079, 253, 255, 257, 259, 261, 263, 265, 1195, 269, 271, 273, 2
77, 279, 281, 283, 285, 287, 289, 1237, 291, 293, 295, 297, 301, 1101, 303, 305, 1255, 307, 1033, 309, 313, 315, 317, 319, 321, 323, 1075, 325, 327, 331, 333, 335, 337, 3
39, 341, 343, 347, 1253, 1205, 355, 1171, 1137, 361, 1103, 363, 365, 1085, 369, 371, 375, 1251, 379, 381, 383, 1191, 385, 387, 389, 391, 395, 399, 401, 403, 405, 407, 409
, 1241, 1263, 413, 415, 419, 421, 425, 1037, 429, 431, 435, 437, 1097, 441, 443, 445, 1269, 449, 451, 455, 1187, 457, 459, 1127, 463, 465, 1277, 469, 1243, 471, 473, 475,
 1175, 477, 1141, 481, 1107, 483, 1073, 487, 489, 491, 493, 495, 497, 1201, 499, 1067, 507, 509, 1109, 513, 515, 517, 519, 521, 523, 525, 527, 529, 531, 1211, 1123, 535,
1177, 537, 539, 541, 543, 545, 547, 549, 551, 553, 555, 557, 559, 561, 563, 565, 567, 569, 571, 1209, 573, 575, 577, 579, 583, 585, 587, 589, 1247, 591, 1213, 595, 597, 5
99, 601, 603, 605, 607, 1043, 609, 611, 613, 615, 617, 619, 621, 623, 625, 627, 629, 631, 633, 637, 641, 1131, 645, 649, 651, 1215, 655, 659, 661, 1113, 663, 665, 1135, 1
045, 669, 673, 675, 677, 679, 681, 683, 685, 687, 689, 691, 697, 701, 703, 705, 707, 709, 711, 713, 1143, 1183, 1149, 721, 1115, 723, 1081, 1145, 731, 739, 743, 747, 751,
 753, 755, 759, 1151, 765, 767, 771, 773, 1219, 1153, 777, 779, 781, 1117, 783, 785, 787, 1197, 789, 791, 793, 795, 797, 799, 1069, 807, 809, 811, 815, 817, 819, 821, 823
, 825, 827, 829, 831, 833, 1221, 835, 837, 839, 841, 1119, 843, 845, 847, 849, 851, 853, 855, 857, 859, 861, 863, 865, 1133, 1199, 873, 877, 881, 883, 885, 889, 1257, 893
, 895, 897, 899, 1155, 901, 903, 905, 907, 1053, 909, 911, 913, 915, 917, 919, 921, 925, 1249, 927, 929, 931, 1189, 933, 935, 937, 1129, 941, 1181, 945, 947, 949, 951, 12
25, 955, 957, 959, 1157, 961, 963, 965, 1089, 967, 1055, 969, 1275, 971, 975, 977, 979, 983, 987, 989, 991, 993, 995, 997, 1003, 1005, 1007, 1009, 1013, 1227, 1193, 1065,
 1017, 1021]) already present
2016-08-29 03:00:01,186:INFO:backend.push.push_apns2:Have 1 opened streams
2016-08-29 03:00:01,187:DEBUG:hyper.http20.connection:Created stream with id 1281
2016-08-29 03:00:01,188:INFO:backend.push.push_apns2:Have 2 opened streams
2016-08-29 03:00:01,189:DEBUG:hyper.http20.connection:Created stream with id 1283
2016-08-29 03:00:01,228:INFO:backend.push.push_apns2:Have 3 opened streams
2016-08-29 03:00:01,229:DEBUG:hyper.http20.connection:Created stream with id 1285
2016-08-29 03:00:01,235:INFO:backend.push.push_apns2:Have 4 opened streams
2016-08-29 03:00:01,236:DEBUG:hyper.http20.connection:Created stream with id 1287
2016-08-29 03:00:01,240:INFO:backend.push.push_apns2:Have 5 opened streams
2016-08-29 03:00:01,242:DEBUG:hyper.http20.connection:Created stream with id 1289
2016-08-29 03:00:01,262:INFO:backend.push.push_apns2:Have 6 opened streams
2016-08-29 03:00:01,263:DEBUG:hyper.http20.connection:Created stream with id 1291
2016-08-29 03:00:01,265:INFO:backend.push.push_apns2:Have 7 opened streams
2016-08-29 03:00:01,266:DEBUG:hyper.http20.connection:Created stream with id 1293
2016-08-29 03:00:01,267:INFO:backend.push.push_apns2:Have 8 opened streams
2016-08-29 03:00:01,267:DEBUG:hyper.http20.connection:Created stream with id 1295
2016-08-29 03:00:01,269:INFO:backend.push.push_apns2:Have 9 opened streams
2016-08-29 03:00:01,270:DEBUG:hyper.http20.connection:Created stream with id 1297
2016-08-29 03:00:01,272:INFO:backend.push.push_apns2:Have 10 opened streams
2016-08-29 03:00:01,272:DEBUG:hyper.http20.connection:Created stream with id 1299
2016-08-29 03:00:01,282:INFO:backend.push.push_apns2:Have 11 opened streams
2016-08-29 03:00:01,282:DEBUG:hyper.http20.connection:Created stream with id 1301

Hrm, it doesn't look like we're receiving any frames for stream 1 here. I think we need a log after this line. Something like: log.debug("Received frames for streams %d", stream_ids).

Here's new log wish message we needed. Connection is created at 2016-08-30 03:25:04,355, and there is message:
2016-08-30 03:25:05,198:DEBUG:hyper.http20.connection:Received frames for streams set([1])

Problems start at 2016-08-31 03:00:02,085 when streams number starts growing.
logs.zip

Hrm. If we look there and rewrite some of the logs, I see that we create 21 streams in about 4 seconds. In that time we have received very little back: we received and closed the first stream, but none of the others.
Is it possible for us to use a gevent semaphore for this? Is there any reason not to gate your maximum streams in this way? That way we can hide the "max streams" things (which are fundamentally a bit misleading) and try to catch a stream that gets stuck instead.

Sorry, I don't understand you completely. Do you mean we should not recreate new connection when getting 21 streams and wait for actual TooManyStreams error?

@binrush In production you are welcome to continue your current system. However, I was suggesting a slightly more adaptive system where you use a semaphore that allows you to queue requests behind the semaphore. That allows us to identify which requests are timing out which, combined with this log, gives me a bit more understanding of what's going on.

@Lukasa do you mean something like this?
# module level
semaphore = gevent.lock.Semaphore()
# ...
            with semaphore:
                stream_id = connection.send(token, pl, topic=topic)
                status, apns_id, body = connection.feedback(stream_id)

I do, but with value=some_maximum_outstanding_stream_count

So, I finally used value=10 and everything works fine, no fails today. Attaching a log - at 2016-09-02 03:00:00 (this time we are sending many pushes) streams number is growing to 11 and then starts decreasing at 2016-09-02 03:03:13,262.
We are planning to increase semaphore value to 30 and see what will happen

With value=20, we are getting problems. Streams count starts growing and finally, workers start dying with Hard time limit (900s) exceeded. Please take a look at this log from 2016-09-04 13:00:31,962
logs.zip

For now, we switched from gevent celery workers to 'prefork', so each worker is separate process that has its own connection. And we still have problems with timeouts. With prefork model, celery prints traceback of where program stucks, here it is
Traceback (most recent call last):
 File "/home/prince/proj/python/prince-backend/backend/tasks/async.py", line 40, in send_push
   extra={'prince': msg.body})
 File "/home/prince/proj/python/prince-backend/backend/push/push_apns2.py", line 91, in apns_send_msg
   status, apns_id, body = connection.feedback(stream_id)
 File "/home/prince/proj/python/prince-backend/backend/push/apyns.py", line 43, in feedback
   with self._conn.get_response(stream_id) as resp:
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 312, in get_response
   return HTTP20Response(stream.getheaders(), stream)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/stream.py", line 230, in getheaders
   self._recv_cb(stream_id=self.stream_id)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 775, in _recv_cb
   self._single_read()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 666, in _single_read
   self._sock.fill()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/common/bufsocket.py", line 167, in fill
   count = self._sck.recv_into(self._buffer_view[self._buffer_end:])
 File "/usr/lib/python2.7/ssl.py", line 771, in recv_into
   return self.read(nbytes, buffer)
 File "/usr/lib/python2.7/ssl.py", line 641, in read
   v = self._sslobj.read(len, buffer)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/billiard/pool.py", line 235, in soft_timeout_sighandler
   raise SoftTimeLimitExceeded()
SoftTimeLimitExceeded: SoftTimeLimitExceeded()

Ok, that's very interesting. This is still us getting stuck waiting for a response that seems not to be coming. For whatever reason, it seems like the socket timeout itself isn't firing, which suggests that there is still data coming in on the connection, but for whatever reason the total time spent processing the response exceeds the soft time out.
Is it possible for you to play about with this timeout level a bit to see if it's just perhaps a bit too aggressive? Try doubling it, for example, and seeing if that reduces the rate of errors?

@Lukasa actually, there was no socket timeout set, i've missed it at some moment, sorry. After returning it back, I get following instead of SoftTimeLimit:
2016-09-28 03:15:00,143:ERROR:backend.push.push_apns2.apns_send_msg:Try 0, 1 streams opened
Traceback (most recent call last):
 File "/home/prince/proj/python/prince-backend/backend/push/push_apns2.py", line 91, in apns_send_msg
   status, apns_id, body = connection.feedback(stream_id)
 File "/home/prince/proj/python/prince-backend/backend/push/apyns.py", line 43, in feedback
   with self._conn.get_response(stream_id) as resp:
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 312, in get_response
   return HTTP20Response(stream.getheaders(), stream)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/stream.py", line 230, in getheaders
   self._recv_cb(stream_id=self.stream_id)
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 775, in _recv_cb
   self._single_read()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 666, in _single_read
   self._sock.fill()
 File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/common/bufsocket.py", line 167, in fill
   count = self._sck.recv_into(self._buffer_view[self._buffer_end:])
 File "/usr/lib/python2.7/ssl.py", line 771, in recv_into
   return self.read(nbytes, buffer)
 File "/usr/lib/python2.7/ssl.py", line 641, in read
   v = self._sslobj.read(len, buffer)
SSLError: ('The read operation timed out',)

So that suggests that we're looking for a response for whatever stream ID we're reading from, and we're never even receiving the headers for that response.
There are fundamentally two ways that can happen. The first way is that the server can really be never sending a HEADERS frame. We should be able to detect that by simply printing out every h2 event we see: if ResponseReceived is never printed for the stream in question, then we know that we're waiting for a response that will never come.
The second way this can happen is a hyper bug. A quick code read doesn't suggest it's there, but it's possible that we're somehow losing the headers in the response. Somehow I don't think that's happening.
Are you interested in chasing the first case down?

@Lukasa sure. Should I add some mode debug messages to hyper?
And returning to root problems, if server really does not send HEADERS frame for some streams, is it possible that in this case stream remains opened and id causes TooManyStreamsError?

@binrush In answer to your question, yes, if no HEADERS frame is ever received then we'll consider the stream still to be open.
Yup, you'll want to add debug messages in _single_read(). If you add a logging statement just inside the for event in events loop that prints out event, that'd be fine I think.

Sorry for delay. Finally I've added debug messages you asked and got logs. If I understand correctly, stream 12797 is created, but no data recieved for it:
2016-10-17 03:07:21,331:DEBUG:hyper.http20.connection:Created stream with id 12797
2016-10-17 03:07:21,333:DEBUG:hyper.http20.connection:recv for stream 12797 with set([]) already present

Then in a minute I get Read Operation Timed Out when reading response for this stream:
2016-10-17 03:08:21,393:ERROR:backend.push.push_apns2:Try 0, 1 streams opened, current stream 12797
Traceback (most recent call last):
  File "/home/prince/proj/python/prince-backend/backend/push/push_apns2.py", line 97, in apns_send_msg
    status, apns_id, body = connection.feedback(stream_id)
  File "/home/prince/proj/python/prince-backend/backend/push/apyns.py", line 43, in feedback
    with self._conn.get_response(stream_id) as resp:
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 312, in get_response
    return HTTP20Response(stream.getheaders(), stream)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/stream.py", line 230, in getheaders
    self._recv_cb(stream_id=self.stream_id)
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 776, in _recv_cb
    self._single_read()
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/http20/connection.py", line 666, in _single_read
    self._sock.fill()
  File "/home/prince/.virtualenvs/prince/local/lib/python2.7/site-packages/hyper/common/bufsocket.py", line 167, in fill
    count = self._sck.recv_into(self._buffer_view[self._buffer_end:])
  File "/usr/lib/python2.7/ssl.py", line 771, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python2.7/ssl.py", line 641, in read
    v = self._sslobj.read(len, buffer)
SSLError: ('The read operation timed out',)

Full log is attached
logs.zip

@binrush I'm beginning to wonder if the best thing for us to do here is actually to dump out all sent and received data so that I can analyse it. Are you happy enough to do that from a privacy standpoint?

@Lukasa actualy, there are some private information that we can not share. How are you going to make a dump? Is it possible for us to somehow 'cut' sensitive information from it?

I was going to propose literally dumping the TCP streams from both sides, though in hindsight that's somewhat problematic. Ideally we'd have a pcap file but that's going to be quite a lot of work to obtain. So probably we're not well equipped to dump the data. Nevermind. =(

So, here's the question: is there anything special about the failing streams? Now that you can reliably determine which streams are failing, you should take a look at those requests to see if there's anything in common about them.
If there isn't, then the problem is probably in the APNS backend and we should consider whether we need to make code changes to allow you to time out individual stream reads more gracefully than this. If there is something in common there, we may be able to move forward.

I started experiencing this also using requests + hyper.contrib.HTTP20Adapter.  The scenario is similar to @binrush, but using hyper via requests.Session() in a celery task with gevent workers.
The requests.Session is long lived attached to the celery.Task on instantiation.
In my case, my destination is Google's HTTP2 enabled load balancer with flask backend.  The endpoints are REST APIs authenticated via Authorization header JWT.  Sample response headers(no body).
POST /create
{
    b'alt-svc': b'clear',
    b'via': b'1.1 google',
    b'server': b'nginx/1.11.8',
    b'date': b'Tue, 21 Feb 2017 03:56:56 GMT',
    b'content-length': b'0',
    b'content-type': b'text/html; charset=utf-8',
    b'location': b'https://example.com'
}

python-3.5.3

# pip freeze
requests==2.13.0
hyper==0.7.0
hyperframe==3.2.0
h2==2.6.0.dev0
hpack==2.3.0
gevent==1.2.1
greenlet==0.4.12

# celery.py
class ContextTask(TaskBase):
  abstract = True
  _session = None

  @property
  def session(self):
      if self._session is None:
          session = requests.Session()
          session.mount('https://example.com', HTTP20Adapter(
              backoff_factor=0.5,
          ))
          self._session = session
      return self._session

# tasks.py
@celery.shared_task(bind=True)
def get_account(task, account_id):
    response = task.session.get(
        'https://example.com',
    )
    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        if response.status_code in [502, 503]:
            raise task.retry(countdown=5, exc=err)
        raise err
    return response.json()

TooManyStreamsError: Max outbound streams is 100, 100 open
  File "flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "newrelic/packages/wrapt/wrappers.py", line 556, in __call__
    args, kwargs)
  File "newrelic/hooks/framework_flask.py", line 103, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 33, in reraise
    raise value
  File "flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "newrelic/packages/wrapt/wrappers.py", line 517, in __call__
    args, kwargs)
  File "newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "<decorator-gen-46>", line 2, in get_user_profile
  File "<decorator-gen-44>", line 2, in _authenticated
  File "flask_api/decorators/authentication.py", line 81, in _authenticated
    return f(*args, **kwargs)
  ...
  File "celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  ...
  File "newrelic/packages/wrapt/wrappers.py", line 553, in __call__
    return self._self_wrapper(wrapped, instance, args, kwargs)
  File "newrelic/hooks/application_celery.py", line 80, in wrapper
    return wrapped(*args, **kwargs)
  File "celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  ...
  File "requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "raven/breadcrumbs.py", line 297, in send
    resp = real_send(self, request, *args, **kwargs)
  File "newrelic/packages/wrapt/wrappers.py", line 553, in __call__
    return self._self_wrapper(wrapped, instance, args, kwargs)
  File "newrelic/api/external_trace.py", line 124, in dynamic_wrapper
    return wrapped(*args, **kwargs)
  File "requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "hyper/contrib.py", line 78, in send
    request.headers
  File "hyper/common/connection.py", line 103, in request
    method=method, url=url, body=body, headers=headers
  File "hyper/http20/connection.py", line 281, in request
    self.endheaders(message_body=body, final=True, stream_id=stream_id)
  File "hyper/http20/connection.py", line 555, in endheaders
    stream.send_headers(headers_only)
  File "hyper/http20/stream.py", line 98, in send_headers
    conn.send_headers(self.stream_id, headers, end_stream)
  File "h2/connection.py", line 803, in send_headers
    (max_open_streams, self.open_outbound_streams)

Hrm, it's another instance of weirdly seeming to have a stream left alive.
Out of interest, can you count how many exceptions you handle before you hit this error? I wonder if there's an issue with leaking them.

Gonna chime in here. I am getting this error with the following configuration:

Heroku cedar-14
Python 2.7.13
Celery 4.0.2
django-push-notifications 1.5.0
apns2 0.2.1
hyper 0.7.0

2017-05-16T17:48:32.868406+00:00 app[worker.1]: [2017-05-16 12:48:32,868: INFO/MainProcess] Received task: messages.messenger._task_send_apns[cdda25a1-e90d-4a83-8ad5-2e0f8456d536]
2017-05-16T17:48:32.888843+00:00 app[worker.1]: [2017-05-16 12:48:32,888: INFO/PoolWorker-3] Received unhandled event <SettingsAcknowledged changed_settings:{ChangedSetting(setting=SettingCodes.ENABLE_PUSH, original_value=1, new_value=0)}>
2017-05-16T17:48:32.889073+00:00 app[worker.1]: [2017-05-16 12:48:32,888: INFO/PoolWorker-3] Received unhandled event <SettingsAcknowledged changed_settings:{}>
2017-05-16T17:48:32.961707+00:00 app[worker.1]: [2017-05-16 12:48:32,961: INFO/PoolWorker-5] Received unhandled event <RemoteSettingsChanged changed_settings:{ChangedSetting(setting=SettingCodes.HEADER_TABLE_SIZE, original_value=4096, new_value=4096), ChangedSetting(setting=SettingCodes.MAX_CONCURRENT_STREAMS, original_value=None, new_value=1000), ChangedSetting(setting=SettingCodes._max_frame_size, original_value=16384, new_value=16384), ChangedSetting(setting=SettingCodes.MAX_HEADER_LIST_SIZE, original_value=None, new_value=8000)}>
2017-05-16T17:48:32.961919+00:00 app[worker.1]: [2017-05-16 12:48:32,961: INFO/PoolWorker-5] Connected to APNs
2017-05-16T17:48:32.962188+00:00 app[worker.1]: [2017-05-16 12:48:32,962: INFO/PoolWorker-5] Connected to APNs
2017-05-16T17:48:32.962351+00:00 app[worker.1]: [2017-05-16 12:48:32,962: INFO/PoolWorker-5] APNs set max_concurrent_streams to 1000
2017-05-16T17:48:32.962478+00:00 app[worker.1]: [2017-05-16 12:48:32,962: INFO/PoolWorker-5] Sending to token <redacted>
2017-05-16T17:48:32.963602+00:00 app[worker.1]: [2017-05-16 12:48:32,963: INFO/PoolWorker-3] Got response for <redacted>: Success
2017-05-16T17:48:32.963917+00:00 app[worker.1]: [2017-05-16 12:48:32,963: INFO/PoolWorker-5] Finished sending all tokens, waiting for pending requests.
2017-05-16T17:48:32.967672+00:00 app[worker.1]: [2017-05-16 12:48:32,967: INFO/PoolWorker-2] Received unhandled event <RemoteSettingsChanged changed_settings:{ChangedSetting(setting=SettingCodes.HEADER_TABLE_SIZE, original_value=4096, new_value=4096), ChangedSetting(setting=SettingCodes.MAX_CONCURRENT_STREAMS, original_value=None, new_value=1000), ChangedSetting(setting=SettingCodes._max_frame_size, original_value=16384, new_value=16384), ChangedSetting(setting=SettingCodes.MAX_HEADER_LIST_SIZE, original_value=None, new_value=8000)}>
etc...

Should those logs be showing the error?

Apologies, these log outputs are new to me as of upgrading to apns2, so I'm not sure if they are the cause of our memory issues. Basically I get the Received unhandled event <SettingsAcknowledged changed_settings logs over and over and over, and the memory usage spikes to 100% and then Heroku kills our Celery worker process. I found this issue by searching for strings in these logs and assumed it was related, but I guess I was mistaken.

Those should be benign. 👍


Adds missing options to the "Quick API Reference"

What kind of changes does this PR include?

Minor content fixes (broken links, typos, etc.)

Description
Adds missing options to the "Quick API Reference" section.

Thank you for this, @ixkaito!  :raised_hands:


hey

I have to clean it, after this the work is read

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 8b2cb61f-1605-70ba-c5fb-cc4a9d9f2c8d
Version Independent ID: 6eaec3e9-0ed9-f3ec-7516-032d57246715
Content: Compiler Error CS0103
Content Source: docs/csharp/language-reference/compiler-messages/cs0103.md
Product: dotnet-csharp
Technology: csharp-diagnostics
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Hi @Astrabacus
Can you supply more details on this? I'm not sure what you're asking us to do.


failed while pushing the image to the private registry repo.

Hi
Sealer is an amazing tool , and  I am trying to use it to  build my own cluster deployment .
when pushing the image to private registry , i met with the error as following
manifest invalid: manifest invalid
I have tried the docker push .... on the same machine ,it is ok .
I am not sure that the problem is the bug of the sealer .
however ,could  you give  me some advice?
thank you .
Environment:

sealer version 0.6.1
OS ubuntu Ubuntu 18.04.5 LTS
Kernel Linux T-SH-GITE015 4.15.0-136-generic
Install tools:  docker version: 20.10.12
Others: registry repo: jfrog artifactory 6.12.2

could you pls share your push cmd ?

/invite


Modifying the power utilities to support more flexible hypothesis tests

This commit introduces the possibility to specify more flexible hypothesis tests
for each of the power utilities. The calculations used to default to the following:
H0: epsilon = 0 vs H1: epsilon != 0, with epsilon the difference in means between
treatment and control. Now, via the introduction of the alternative and mu
parameters, you can speficiy hypothesis tests such as:

(alternative='greater', mu=delta) => H0: epsilon <= delta vs epsilon > delta
(alternative='lower', mu=defta)   => H0: epsilon >= delta vs epsilon < delta
(mu=delta)                        => H0: epsilon = delta vs epsilon != delta

I also introduced several simplification for consistency:

Across all utilities I defaulted to using a normal approximation for
the test statistics (instead of the exact t-distribution). Indeed, `solveforsample_{T,G}test
was already making this assumption (by fixing the degrees of freedom to a large number)
so the simplification should make the utilities more consistent
In solveforpower_Gtest I opted for a test with unequal variances and didn't
fallback to the equal variance assumption when mu=0 (for simplicity)

There was also a mistake in solveforeffectsize_Ttest where we would return the square
of the correct value.
The commit also introduces a tentative test suite which is far from being exhaustive
but which should get us started. Specifically I test each utility by using their
output as parameters for simulated experiments. I then compare the observed power
obtained with the theoretical power doing an equivalence test with a 0.01 margin
(a two one-sided test). The margin is chosen so that the equivalence test has
adequate power (0.999) and false positive rate (0.001) without being computationally
prohibitive (30000 experiments, as given by -in R with the TOSTER library-:
ceiling(powerTOSTone.raw(alpha = 0.001, statistical_power = 0.999, sd = sqrt(0.8*0.2),
low_eqbound = -0.015, high_eqbound = 0.015)))
I test most of the utilities by simulating bernoulli distributed metrics (except
the ones which are radically different for binary vs continuous metrics) instead
of normally distributed ones for speed (one draw of binomial accounts for a whole
experiment)

Holding this up as we still need UI changes and force mu to 0 when we are not using "non inferiority" test


Update Chromedriver to v105

What is the context of this PR?
Updates Chomedriver to v105
How to review
Tests pass
Checklist

[ ] New static content marked up for translation
[ ] Newly defined schema content included in eq-translations repo

 Update Chromedriver to 105


test -d to make sure awscli is actually installed

brew --prefix shows the location that a package is or would be
installed:
https://github.com/Homebrew/homebrew/blob/a8582c58/share/man/man1/brew.1#L448-L449

@quodlibetor Is this still needed on the current version?
Also, it looks like if you squash these three commits the first two are actually useless.

Seems like latest plugin version is sane in the face of non-brew, but I am spending most of my days on OSX these days so I can't actually check.


EntropySGD: gradient updates for Langevin dynamics

Hi,
I've been attempting to implement the EntropySGD in this paper (https://arxiv.org/pdf/1611.01838.pdf). However the inner loop of the optimizer requires L gradient updates to estimate the exponential decayed mus which are used to update the parameters on one batch of data. It looks like  'get_updates' is called from training.py. My question is:

Is there a way to update the network params, compute a new loss and get new gradients on the same batch of data in the optimizer class to run the L iterations for the Langevin dynamics?
If not, can this be implemented in other ways or can this be added as a future feature?

Thanks

Being implemented https://github.com/fchollet/keras/issues/6563


Bacteria names and info from accession #

Convert RefSeq accession numbers into bacteria names. This could all go in a reference file with several columns: accession number, bacteria name (genus and species), strain identifier, optional other information we might want. Issue #2 describes work related to fetching information from NCBI using accession numbers; the function acc2gb.py might be modified for this purpose.
Example reference file:

accession number, bacteria name (genus and species), strain identifier
NC_000853,Thermotoga maritima, MSB8

The CRISPRdb website has a list that sort of seems like the right thing, except they're using the NCBI taxon ID instead of the actual sequence accession number.
Example from CRISPRdb file:
243274|Thermotoga maritima MSB8
What we want:
NC_000853,Thermotoga maritima, MSB8

The 'Comments' section of the Genbank file often has other information, such as where the organism was isolated, Gram staining, etc. It would be good to have the option to pull other fields if they exist (they may often not).
It will be helpful to be able to access Genbank for other information in general, so I think my previous comment about CRISPRdb can be ignored.

@mbonsma I'd be interested in working on this.

@darwinhang Awesome! That would be fantastic. I think there are a few different routes this could go depending on what you're interested in and feel like doing. This issue is closely related to #47 (Extract data from Python object generated with NCBI Entrez), and there's a twin issue for a different database (Issue #52), so if you want you could check out all three.
Also I just made a flow chart about the project, which hopefully makes it a bit clearer how the pieces fit together.
Let me know if there's anything I can do to smooth your path.

@mbonsma Thanks for sharing the flow chart. It does seem like it would make sense to work on this issue and #47  at the same time.

@darwinhang - I made bac_info_parser.py that uses Bio.SeqIO.parse to read in Genbank files. It could probably be improved to be more versatile - right now I have it just extracting the bacteria name and taxonomy and making a dictionary.
I think this can close issue #47 though. Bio.SeqIO.parse is the same as Bio.SeqIO.read mentioned in that issue; the only difference is that parse handles multiple records per file while read can only do single records.

Related - organism_name_update fetches just the organism name from the NCBI accession # and adds it to the crispr.sqlite database.


add support for Configuration API

The Configuration API feature has been implemented in Dapr runtime: https://github.com/dapr/dapr/pull/3606
It needs to be accessible via the SDK.

/assign

Good day all, I would like to take this on. I observe that the methods that need to be implemented are GetConfigurationItem, GetConfigurationItems, SubscribeConfigurationItems, UnsubscribeConfigurationItems.

@amulyavarote has already begun implementation on this, but if the two of you would like to collaborate, or trade work I'll leave that up to you.

No problem. I can take on this issue  instead.

Thanks for taking care of this. @amulyavarote any update on the state of the configuration API implementation?


Question: Can we send traceid or span info when encountered error ?

Started using this library as this
As far as I understand, it collects traces and send to specified Sumo Logic Collector.
and if collectErrors:true is set then it collects and send uncaught exceptions,errors to the Collector.
But, is there any way we can also send the "traceId" or "span" info associated with the particular error ?

cc: @TRybakSUMO ?


GitCheatFinder (Android App)

I want to add the GitCheatFinder project which is an Android App that helps you to find easily the right Git Commands that you want to use.
Tech used in Kotlin.
Made in Android Studio.
Can you please assign me?
Repository link: https://github.com/aritra-tech/GitCheatFinder

I want to add the GitCheatFinder project which is an Android App that helps you to find easily the right Git Commands that you want to use. Tech used in Kotlin. Made in Android Studio. Can you please assign me?

Make sure to get started from the Contributing.md 🚀

I want to add the GitCheatFinder project which is an Android App that helps you to find easily the right Git Commands that you want to use. Tech used in Kotlin. Made in Android Studio. Can you please assign me?
 Make sure to get started from the Contributing.md 🚀

Ok, sure I will go through the Contributing.md file


nixos/network-interfaces-scripted: fix a container networking bug

Motivation for this change
Fixes #47210:  When a bridge interface was reconfigured, running containers using this bridge lost connectivity: restarting network-addresses-brN.service triggered a restart of network-setup.service via a partOf relationship introduced in 07e0c0e0a2f237639600f2a0d62f6eac748b1e6e. This in turn restarted brN-netdev.service.
The bridge was thus destroyed and recreated with the same name but a new interface id, causing attached veth interfaces to lose their connection.
This change removes the partOf relationship between network-setup.service and network-addresses-brN.service for all bridges so they can be reconfigured without being destroyed and recreated.
I don't see any negative side effects atm, but would kindly ask the the authors of of 07e0c0e0a2f237639600f2a0d62f6eac748b1e6e (which introduced the bug while fixing https://github.com/NixOS/nixops/issues/640) to have a look:  cc @nh2, @domenkozar, @fpletz, @aszlig, @basvandijk
There's also an (uglier) alternative solution without changing the partOf relationships: modify the start/stop scripts to skip deleting and recreating a bridge while there's still a veth interface attached.
Things done
NixOS tests:

[x] containers-restart_networking now passes (which failed before due to the bug)
[x] all other container tests still pass
[x] all networking tests still pass

cc @srhb

I also found that despite our apparent fix, the original problem we intended to fix reappeared: https://github.com/NixOS/nixops/issues/833
We don't know yet why it is.

If that's the case, maybe we could just revert 07e0c0e0a2f237639600f2a0d62f6eac748b1e6e  ?

Well, it did fix some of the occurrences (we verified that), but apparently not all of them.

Perhaps we'll figure it out for good at NixCon? That'd be awesome.

Let's do that :smile:

Your analysis sounds and looks very correct, thanks a lot for that @xeji ! I will also happily wait for some more feedback from the original authors. :)

Huh, containers test was failing for one year :)
I admit I don't see the full picture of networking dependencies and what should trigger a restart given changed configuration, but exclusion of bridges does sound like a step forward.

I'll merge this now as there were no objections. It's probably not the best possible solution but works for now. We should try to refactor and simplify the networking dependencies eventually.
An alternative would be to just let systemd-networkd handle container networking (it's designed to work well with systemd-nspawn containers out of the box) and drop the scripted setup for containers.

backported to 18.09: 589d270ce44

@xeji  This commit actually causes bridges not to appear after nixos-rebuild switch.  I think it should be reverted and we need to think of another way to fix this. #50208

Actually, #50208 was a duplicate of https://github.com/NixOS/nixpkgs/issues/42828, it seems this broke a lot in bridge networking. How should we move forward with this?


optimize HPA controller in autoscaling

I found a parameter to adjust the HPA controller interval that would help to make it reacts quickly to the function's load. I tested it with 5s and the result was reasonable - The function was scaled up and down quickly. There might be harming the system by putting HPA under stress but that would just be a suggestion for optimization.

/lgmt


(GUIDE) Procedure for running on Windows 10/11 for NVidia GPU

Hey all. I had to go through a bit of extra to get this to run on Windows. I noticed a few issues (possibly #30, maybe #29) which are likely to be solved by a "clean/isolated quasi-Proper" install guide. Here's what worked for me. (I'm not covering installing ffmpeg -- I used chocolatey as linked from the main post and it was annoying, but I literally used the instructions at chocolatey.org's Install page)
Notes/justifications:

I'm writing this using a venv. This prevents python packages from clobbering each-other's dependencies with updates/downgrades. This prevents version conflicts, as only the things needed per-application are installed, per-application and loaded independently. You'll need to pick a location and remember it; this is where the app "lives" and you'll need to reference it. This folder can be changed to anyplace on your machine which Python can reach and execute from. You will need to re-activate the venv every time you run the application, unless you create a bat script to do so for you.
If you don't like it, I don't want to hear about it -- write your own guide. If you need help or clarification, or something doesn't work for you, let me know and I'll do what I can to help.

To start, you need Python. I'm using 3.10.6 but most versions should work, so long as they've been around a few weeks. If you don't have it, install it from the official Windows download site -- you can install it via Microsoft Store, but I am fundamentally against cli-from-Store apps for several reasons.
I'll describe the procedure three times; first by description. Second, by sequence of commands. Third, by screenshots.
By description:
Open a command prompt. Create your venv directory. Have Python populate it. I'm using \venv\auto-subtitle and my Windows drive is C: (as is most peoples' I'd imagine) -- I have multiple venv installations, so this is convenient for me.
Install torch as indicated here -- this is the PyTorch "Get Started Locally" guide, which describes how to install Torch for locally-run AI workloads. This entire guide pivots around that page.
Torch is the part of the Python stack which does the "AI work" and having a Torch version which supports your GPU is CRITICAL to making these things work. This process works with nearly any application which uses Torch.
Anyway -- from that page, select Stable, Windows, Pip, Python, CUDA 11.7, and run what "Run this command" tells you.
From there, install/run auto-subtitles. Deactivate the venv if you need the cli for other Python things, otherwise just close it.
By commands:
FIRST TIME:
Start>Run>cmd
mkdir \venv\auto-subtitle
cd \venv\auto-subtitle
python -m venv .

Tried this on Fedora 38 with ROCm beta 5.6, works fine. Might need to wait a bit for the Windows version though.
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.6

That's correct - the Linux version of ROCm has been known to work (obviously not with this project lol) since the ROCm bindings were first released for PyTorch 1.8 in early 2021.
As a comparison, PyTorch 0.3.0 from 2018 supports NVidia on Windows.
The Windows version of ROCm has not had PyTorch bindings, an announced ETA, or even announced specific plans to support Windows+ROCm+PyTorch, during that two and a half year period. In fact, the Windows version of ROCm hasn't been released at all yet; it's still in beta. The PyTorch people can't really even start working on it until AMD gets around to finalizing Windows support for ROCm.
You will need to wait indefinitely for the Windows version. Don't plan on it being short. Regarding Windows, AMD has yet to finish/release;

ROCm Porting tools
ROCm Debugger
ROCm Communication Libraries
ROCm AI Libraries
some management tools
ROCm AI frameworks
cmake for HIP
HIP ray tracing support
ROCm 5.6 itself (it's Linux-only so far)
Lots of their source code required for manual builds of various libraries and tools on Windows

These things could be almost done, or they could be nowhere. AMD failed to invest resources in the Windows side of this market when it was early and growing. They didn't put a foot in the door at all until recently. They're very far behind, as they undervalued this market considerably.
If you're building a system and care at all about using your GPU for anything "AI", machine learning, or GPGPU related, reconsider AMD or plan to use real Linux. (Real Linux as in not-WSL -- using WSL does not improve this situation. I'm not knocking WSL, I'm a huge fan.)
If you already have a built system and wish to use your AMD GPU for this stuff, your only route now-or-soon is to set up Linux on a second drive, or set up a fast USB boot device with persistent storage and dual-boot. I recommend using the tool Rufus to create said medium, as it supports persistent partition creation.
Good luck!

Not sure what you meant by "not with this project."
To clarify, I was able to get this project to work GPU accelerated (6900XT) using Fedora 38 and ROCm 5.6, using your guide above, tweaking for linux.
I monitored CPU and GPU usage using radeontop and bpytop, CPU usage was low and GPU usage was 97%.
Hopefully this helps others with AMD GPU.


Create task merge to main

Requesting another review after changing to column name and setting it as an optional field


Recommended way of adding build process for CSS

I'd like to use SCSS in an Arch project, but seeing as it hides the Webpack config from you, I'm not sure what the best way of handling this would be.

@chrisvfritz In the works. I have a working solution which reads your local webpack config from your project and extends the arch default. Atm I'd use gulp or something else to bundle your styles, which is what we do where we use arch right now.

Sort of related: https://github.com/arch-js/arch/issues/21

:smiley: That sounds rad! I'll use gulp in the meantime.

I'd say that is the long term approach anyway. I'm really against exposing the Arch build step and mixing it with app concerns, because it'll in turn make it way harder for us to iterate on the details of the build step.
Say we started with browserify and then webpack came out with react hot loader. We'd want to use that, but it would be a huge breaking change and really tricky to do. This can and will happen again in the future...

In short - use your own build step as you would otherwise. We should make the fact the dist folder is publicly served more obvious in the docs or generator-arch.

I can definitely see the hesitation in exposing the Arch build process. Until inline CSS with React is more reliable though, I think it does make sense for a default configuration to exist, supporting at least one popular CSS preprocessor. It looks like at Red Badger, you use SASS, Less, and Stylus. So here's a proposal:
Default builds would be set up for SASS, Less, and Stylus, watching the following folders:

app/sass
app/scss
app/less
app/stylus

If any of the above folders exist and contain a main.extension file, then in the first matching folder, that main.extension file is built into dist/css/main.css. I think this would cover most use cases, be relatively simple to set up and maintain in Webpack, and you could switch to Broccoli in a month, have it do the same thing, and still not mess anyone up.
But... if someone wants to do something more complicated, then they can configure their own setup outside of Arch. What do you think?

I suppose we should have an example with how you'd go about setting such a thing up. I added it to the ideas in #69. For Arch, I'd say this is out of scope, until we decide to include support for CSS in JS.
I believe you don't even need webpack for this, just plain gulp would work, right?

Yeah, setting this up outside of Arch is pretty trivial. It just means a little extra configuration whenever starting a new project that includes external CSS and a preprocessor - which I'd guess would be most projects. And then it's also necessary to run another watcher alongside arch-cli s -w. Just a little annoying. :-/


Repository not found for thirdweb sdk

I am trying to do the basic steps in your tutorials, e.g. here: https://portal.thirdweb.com/guides/mint-nft-using-nextjs#3-client-side-react-components-to-render-list-of-nfts-using-react-sdk-and-a-mint-button
However, every time I try to install the necessary packages I get an error saying the repo where your package is hosted cannot be found:
npm ERR! code 128
npm ERR! An unknown git error occurred
npm ERR! command git --no-replace-objects ls-remote<EMAIL_ADDRESS>npm ERR! ERROR: Repository not found.
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.

I can install other packages just fine; it only affects this one.
Is this an issue with v2? I was also able to instal the old sdk packages (@3rdweb/hooks, for example) with no problems.

hey @crevulus, thanks for the report. it's odd that in the npm install it tries to do anything with github at all. I'm assuming your following the install steps from the guide you linked?
npm install @thirdweb-dev/react thirdweb-dev/sdk ethers

If so I can try to reproduce it.
It might also be helpful if you could provide your npm version npm -v and node version node -v

@jnsdls
yarn: 1.22.10
npm: 8.1.0
node: v16.13.0

thanks! I'll try it out shortly and see if I can reproduce it on my end

Actually, I think I've just spotted it. You're missing an @ in the command for the sdk package name. Should be @thirdweb-dev/sdk. I tried editing it myself but couldn't find your docs, only your example code. @jnsdls

hey @crevulus, sorry about that just found the error (and fixing it in the guide itself) it's the missing @ in the thirdweb-dev/sdk

Actually, I think I've just spotted it. You're missing an @ in the command for the sdk package name. Should be @thirdweb-dev/sdk. I tried editing it myself but couldn't find your docs, only your example code. @jnsdls

hah! yup exactly that should be @thirdweb-dev/sdk we'll fix it shortly!

will keep this open until we've fixed the guide(s)

Thank you for the report @crevulus! We just fixed the error in the guide:
https://github.com/thirdweb-dev/portal/commit/0ef8af2835331b0f5f986aa6c95f9b83fb1028ec


DOC: Mention quadratic_kappa metric in eval_metric docs

Issue #, if available:
Quadratic Kappa is an important evaluation metric used in lots of kaggle competitions. This was added in https://github.com/autogluon/autogluon/pull/1104 but the documentation doesn't mention its availability as one of the eval_metric options inside AutoGluon.
Description of changes:
Add quadratic_kappa to the doc list of eval_metric options for classification.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Previous CI Run
Current CI Run

@Innixma little reminder for this small change. Thanks for review!


Use CONFIGURE_DEPENDS in file(GLOB*

ONNX RT makes heavy use of globbing to collect source files in the CMake scripts. This can lead to breaks when doing incremental builds where files are removed/added without otherwise touching the CMake files. This issue can be avoided either by explicitly listing all source files, or by using CONFIGURE_DEPENDS which re-runs glob during build to check whether CMake has to be re-configured.
Note that this is only an issue when not using the tools/ci_build/build.py script (which by default would do an --update that re-configures CMake each time) and instead running CMake directly. Using CMake directly is useful when consuming onnxruntime via add_subdirectory from another project.

For the two options, I'd prefer explicitly listing them, as it always works.

I put the other alternative in the issue title as I had the feeling that the dev team prefers to touch CMake files as infrequently as possible, and listing all source files explicitly would mean more changes to CMake files when adding/removing/moving files. Before jumping to a PR for explicitly listing all files it would be good to get agreement from other devs as well, to avoid big surprises at PR time and potentially rejecting it at that point.

May I know how much does it bother you? It's not a problem for me because whenever I sync the code with remote or switch a branch, I always delete the whole build directory. CONFIGURE_DEPENDS will make incremental builds better, but, still, they are not reliable. I still need to empty the build dir frequently, then what's the difference?

It's not a big problem, but still, would be nice to improve dev experience. Why do you think CONFIGURE_DEPENDS is not enough to make builds reliable? If there are other issues, then they should be fixed as well.

CONFIGURE_DEPENDS seems like a good solution to me.

@letmaik if this solution works for you feel free to contribute. Thanks!


Create initial implementation of Spring Cloud Deployer SPI

As a developer, I'd like to refactor the current Kubernetes deployer implementation, so I can prep the deployer to consume spring-cloud-deployer's AppDeployer specs.

Closed by f2083091ad518faf6d87d085913e2531d1cd85dc


make weather file directory when it doesnt exist

tiny fix to make the weather model directory; if user calls processWM directly it may not have already been created

@bbuzz31 the same two tests are failing for you that are failing for me in #601. Any ideas what's going on?

@bbuzz31 the same two tests are failing for you that are failing for me in #601. Any ideas what's going on?

I think you can still merge this though; clearly this is not the reason those tests are failing.


Exploring ethernet on Arty A7

Hi @WillGreen
First of all thank you for your blog and github repos. They have so far been instrumental in my FPGA learning process. I want to get familiar with ethernet and create a project whereby I would blink leds every time the board clocks an ethernet packet with certain content.
Do you have any recommendations as to where to start? Would you be interested in publishing such a walkthrough on your blog and providing supplementary github materials?
Ed

Hello Ed, and thank you for your support.
I don't have any plans to cover Ethernet in the immediate future.
However, I have three suggestions to get you going:

Most like a Project F approach is the fpga4fun.com Ethernet receiver series: https://www.fpga4fun.com/10BASE-T0.html, which doesn't use an external PHY IC or even require an RJ45 connection.

The Arty A7 has an Ethernet PHY and RJ45 jack, so an alternative approach would be to use the on-board PHY as described by https://digilent.com/reference/programmable-logic/arty-a7/reference-manual#ethernet_phy and the Texas Instruments datasheet: https://www.ti.com/product/DP83848J

Finally, www.fpga-cores.com has an Ethernet core in VHDL that's free for non-commercial or evaluation purposes and has instructions for the Arty A7: https://www.fpga-cores.com/tutorials/ethernet-on-arty-a7-board/ - I've not used this core, so can't comment or how easy-to-use or capable it is.

I hope this helps, and good luck with your hardware design,
Will

Hi @eliquinox, I'll close this issue shortly unless you have any questions about my suggestions.


Neovim fails to open file with spaceVim

Describe the bug
The team of SpaceVim says that:

It seems a bug of neovim package in nixos, I can not fix it, there is no file name in v:argv, btw, it use --embed which mean, neovim is running in backgroud, and a gui/tui ui is used. you need to build a native nvim yourself.

Refering to https://github.com/SpaceVim/SpaceVim/issues/4882#issue-1778012305
Steps To Reproduce
Steps to reproduce the behavior:

Use NixOS.
run sudo nixos-rebuild switch --upgrade
run `nix-shell '' -A neovim
run `nix-shell '' -A gnumake
install SpaceVim
Apply patch https://github.com/SpaceVim/SpaceVim/issues/4845#issuecomment-1516608105
run nvim some_file

Expected behavior
For SpaceVim is covered here https://github.com/SpaceVim/SpaceVim/issues/4882#issue-1778012305
For what SpaceVim Expects, check with SpaceVim developer.
Additional context
Trying to solve an error with SpaceVim. I checked and is no directly with neovim, because the file opens correctly without SpaceVim.
Notify maintainers
@teto
@linsui
@ivanbrennan
@pacien
@pennae
@thiagokokada
@ck3d
@ncfavier
@dasj
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.35, NixOS, 23.05 (Stoat), 23.05.1353.35130d4b4f0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos-23.05"`
 - channels(user): `"home-manager-23.05.tar.gz, release-23.05.tar.gz-23.05.tar.gz"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

False alarm, just update SpaceVim


filter using predicate whether List contains some string.

Goals

I have a realm object that has List
and I want to filter using predicate whether List contains some string.
but I cannot find solution
Expected Results

We can filter List using NSPredicate
Actual Results

Steps to Reproduce

Code Sample

Version of Realm and Tooling

Realm framework version: ?
Realm Object Server version: ?
Xcode version: ?
iOS/OSX version: ?
Dependency manager + version: ?

I am having the same problem too.
let predicateNumber = NSPredicate(format: "ANY numbers CONTAINS[c] '\(keyword)'")
numbers is List

We don't support that yet I'm sorry to say.


WIP Add additional special case for template uninstall

Summary
Added additional qualification for uninstalling a template that was installed from a path to a local nupkg file.
While not directly related (sorry), I noticed a nearby [!NOTE] block wasn't rendering correctly due to leading whitespace and fixed that.

@dsplaisted @KathleenDollard who would be a good person to review this?

@seancpeters @vijayrkn

@patridge can you make sure you merge the latest from master to this PR when you address the comments? Please let me know if you'd like me to do that for you. The topic has a new section for 2.1 and so some of this content shows up in multiple places.

I don't know what happened to master on this repo, but whatever it was seems to have been a bit insane. I can't merge the upstream latest into my local repo without it failing spectacularly, yet I have no changes to master on my local machine. It should just be a simple fast-forward but it's not letting that happen.
I'm going to have to recreate my changes on a new branch and force-push them over my fork's branch to get the latest.

@mairaw I've reworded the uninstall note addition to be more generic now, per the comments by @seancpeters.
Looks like the add-on whitespace fix was also fixed elsewhere and is no longer a change here, simplifying the PR changes nicely to just the relevant uninstall lines.

FYI: The uninstall output is going to change in the relatively near future, per this PR https://github.com/dotnet/templating/pull/1572 from @mlorbetske


Update to ClrMD version 4.0.0-beta.24314.3

Enable ClrMD to authenticate to a symbol server if required.
Update ClrMD API calls.

Thanks everyone for reviewing!


Shorter syntax for declaring variables

Golang made a genius move by removing the 'var' in favor of :=
It makes all code shorter and more python-y
I propose that Cone makes the same move and does something like this:
imm x = 5 becomes x ::= 5
mut x = 5 becomes x := 5
It's not just the amount of characters that is at play here. Reading a file from top to bottom is much easier when the name of variables is right there on the leftmost side. Consider this: imm and mut are in reality metadata for the compiler. Sure, it's useful to know this when reading code but the real information we usually want to know as programmers when we scan over some code visually with our eyes is not all this metadata stuff. It's the actual names of variables, functions, constructs. This syntax makes that scanning easier and faster for humans.

Thanks for the suggestion. In a language without permissions, I too see benefits to the := syntax. It is something I have thought about.
My largest concern here is that 'imm' and 'mut' are not the only permissions, opening up the need for variations on := for each of them, and then having people keep straight which is which. Because of this problem, it is more likely I would want to swap the permission and variable, sticking the permission after the colon and before any type, if present.
Making this change creates yet another challenge: parsing determinism. Variable declarations need to be statements and never expressions, and without some recognized leading token, the non-backtracking parser cannot enforce this. So, before making this design change, I would need to work out some other way to enforce this.

First off let me just say this language looks really great, nice job!
I was just about to make this suggestion as made by Meai1, except I would have had it the other way around, with := for imm and ::= for mut, since most will want immutability by default (having functional inclinations).
My reasons for wanting the change are not tied to Go though, rather I wish for a language where the variable names are always first for readability sake (and a more declarative programming experience, see Red lang).
However, keywords after the variable name will also work for me.
Sadly I do not have a solution for the parsing determinism issue, but I will point you to some interesting languages that might have solved the issue since they have variables without leading keywords:
moonscript: https://moonscript.org/
crystal: https://crystal-lang.org
red: https://www.red-lang.org/
jai: https://github.com/MichaelSDavid/Cupcake-Compiler/blob/master/modules-source/Basic/procedures.jai
pointless: https://ptls.dev/
agda? : https://agda.readthedocs.io
Either way, keep up the good work.

I think this is a matter of personal preference.  As personal preference goes, I actually like imm and mut being on the leftmost side - it draws my eyes to variable declarations.  It actually really bugs me that go has both var AND the := operator.  Python suffers from the problem of not being able to declare a variable without assignment; golang supports unassigned declaration, but because of the infix declaration/assignment operator, there's now two ways to declare a variable.  I much prefer the simplicity of always having the declaration prefixed by a common keyword that IDEs know how to highlight.  Just my two cents.


Beginning Long and Lat incorrect for london

The provided starting centre of the map is incorrect for the mentioned location.
Line 5 of app.js " center: { lat: 52.632469, lng: -1.689423 },"
Clicking on this menu item will open a new tab in your web browser with the content of the HTML served from the simple Python HTTP server. If everything went well, you should see a map centered on London, England.
Those coordinates actually centre you in Tamworth see: https://www.google.com/maps/place/52°37'56.9"N+1°41'21.9"W/@52.6324722,-1.6916117,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x0!8m2!3d52.632469!4d-1.689423

Mind submitting a pr with the correct center?

Of course,
I suggest using "51.507413, -0.127703" for the coordinate points.


Remove IE6-9 conditional comments from project template

I'd like to propose removing the shims for IE6-9 from the project template.
The rationale for this is the same as for dropping support for EOL version of Django, etc. - i.e., these versions of Internet Explorer are no longer supported by the vendor (the last version of Windows that you cannot upgrade to use IE10 or higher was Windows XP, which reached end of life in 2014). The Wagtail documentation itself says that IE10 and lower are not supported.
On that basis I think that Wagtail should not include these shims by default - projects that for whatever reason still need to support these browsers can easily add them to their templates.

Thank you @solarissmoke, that seems very sensible to me. I also removed the <meta http-equiv="X-UA-Compatible" content="IE=edge" />, which is useless for recent versions of IE, and MS Edge. And reflected those changes on the separate 500.html template.


trouble building HEIR: timeout running git reset

Hello
I am trying to build HEIR following the Getting Started docs.
This is on a university Intel Xeon cluster running Rocky Linux. I'm using bazel 7.1.2.
Happy to reply with other env/hardware if needed. Thanks!
jmatter heir $ bazel build @heir//tools:heir-opt
WARNING: Output base '/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9' is on NFS. This may lead to surprising failures and undetermined behavior.
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server (1765298) after 10 seconds ...
INFO: Repository llvm-raw instantiated at:
  /home/jmatter/heir/WORKSPACE:43:12: in <toplevel>
  /home/jmatter/heir/bazel/import_llvm.bzl:12:23: in import_llvm
Repository rule git_repository defined at:
  /home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git.bzl:181:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'llvm-raw':
   Traceback (most recent call last):
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git.bzl", line 175, column 35, in _git_repository_implementation
		update = _clone_or_update_repo(ctx)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git.bzl", line 36, column 20, in _clone_or_update_repo
		git_ = git_repo(ctx, directory)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 91, column 12, in git_repo
		_update(ctx, git_repo)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 110, column 10, in _update
		reset(ctx, git_repo)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 155, column 9, in reset
		_git(ctx, git_repo, "reset", "--hard", git_repo.reset_ref)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 179, column 15, in _git
		_error(ctx.name, start + list(args), st.stderr)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 200, column 9, in _error
		fail("error running '%s' while working with @%s:\n%s" % (command_text, name, stderr))
Error in fail: error running 'git reset --hard 392ca64893dc77f823cb4734238f1ba1d271b997' while working with @llvm-raw:
Timed out
ERROR: /home/jmatter/heir/WORKSPACE:43:12: fetching git_repository rule //external:llvm-raw: Traceback (most recent call last):
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git.bzl", line 175, column 35, in _git_repository_implementation
		update = _clone_or_update_repo(ctx)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git.bzl", line 36, column 20, in _clone_or_update_repo
		git_ = git_repo(ctx, directory)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 91, column 12, in git_repo
		_update(ctx, git_repo)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 110, column 10, in _update
		reset(ctx, git_repo)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 155, column 9, in reset
		_git(ctx, git_repo, "reset", "--hard", git_repo.reset_ref)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 179, column 15, in _git
		_error(ctx.name, start + list(args), st.stderr)
	File "/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9/external/bazel_tools/tools/build_defs/repo/git_worker.bzl", line 200, column 9, in _error
		fail("error running '%s' while working with @%s:\n%s" % (command_text, name, stderr))
Error in fail: error running 'git reset --hard 392ca64893dc77f823cb4734238f1ba1d271b997' while working with @llvm-raw:
Timed out
ERROR: Error computing the main repository mapping: no such package '@llvm-raw//utils/bazel': error running 'git reset --hard 392ca64893dc77f823cb4734238f1ba1d271b997' while working with @llvm-raw:
Timed out
Loading:
jmatter heir $

Hey! Do you know if it's because git itself on your machine is timing out? (see https://groups.google.com/g/bazel-discuss/c/XQpMONOIHsQ)
It seems like you might be able to increase that timeout (https://github.com/bazelbuild/bazel/issues/6029) as well. Could you check to see if running bazel clean --expunge and then retrying might work?
If not, I think we could add shallow_since to our new_git_repository and maybe that saves some time? But I'm not totally sure it will or that it's a good idea, since bazel says it is buggy

WARNING: Output base
'/home/jmatter/.cache/bazel/_bazel_jmatter/261dd8aa36a61e57ee04e376d53b37d9'
is on NFS. This may lead to surprising failures and undetermined behavior.

This seems like a suspicious warning.
Try the output_base flag and cf the note about NFS here:
https://bazel.build/docs/user-manual#output-base
On Wed, Jun 5, 2024, 3:36 PM asraa @.***> wrote:

Hey! Do you know if it's because git itself on your machine is timing out?
(see https://groups.google.com/g/bazel-discuss/c/XQpMONOIHsQ)
It seems like you might be able to increase that timeout (
bazelbuild/bazel#6029 https://github.com/bazelbuild/bazel/issues/6029)
as well. Could you check to see if running bazel clean --expunge and then
retrying might work?
If not, I think we could add shallow_since
https://bazel.build/rules/lib/repo/git#new_git_repository-shallow_since
to our new_git_repository
https://github.com/google/heir/blob/main/bazel/import_llvm.bzl#L12C5-L12C23
and maybe that saves some time? But I'm not totally sure it will or that
it's a good idea, since bazel says it is buggy
—
Reply to this email directly, view it on GitHub
https://github.com/google/heir/issues/718#issuecomment-2151072568, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAS2PKWEC3C23TW4RQHET4TZF6HE7AVCNFSM6AAAAABI3SIANSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJRGA3TENJWHA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Try the output_base flag and cf the note about NFS here: https://bazel.build/docs/user-manual#output-base
I'm also new to bazel so thanks for the tip! I'm not sure what other filesystems I have access to—I'll talk to our IT folks.

For some reason, I stopped getting a timeout error! It's failing now with gcc: error trying to exec 'cc1': execvp: No such file or directory. I think that's just a bazel/environment thing so I won't take up your time troubleshooting that :)
For future reference (mine and others') bazel apparently has a flag that might help with this kind of thing:
--experimental_scale_timeouts=<a double> default: "1.0"
Scale all timeouts in Starlark repository rules by this factor. In this way, external repositories can be made working on machines that are slower than the rule author expected, without changing the source code

- Command-Line Reference | Bazel

Hey! Just looking at that last error coming from gcc, we normally build with clang, so to change that you can set your CC and CXX env vars to clang/clang++. That being said normally I can build with gcc too so it might just be a missing package