Watchers not getting updates

etcd Version: 3.2.4
Git SHA: c31bec0
Go Version: go1.8.3
Go OS/Arch: linux/amd64
Client version: 8ba2897a21e4fc51b298ca553d251318425f93ae
We are currently running a 5 node etcd cluster with 200k+ watchers, we have been rarely writing to the cluster. Today when we updated the value of a key, less than 1% of all the watchers did not receive the update, this is not the first time we observed this issue.
Some more debugging on a problematic application process shows this:
goroutine 79 [select, 4361 minutes]:
internal_code/vendor/github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc4203615e0, 0xc420aa0000, 0xc42055ec00)
    /internal_code/vendor/github.com/coreos/etcd/clientv3/watch.go:591 +0x526
created by internal_code/vendor/github.com/coreos/etcd/clientv3.(*watchGrpcStream).run
    /internal_code/vendor/github.com/coreos/etcd/clientv3/watch.go:434 +0x47b

and we found this in the etcd log:
Nov 18 01:54:51 {etcd_hostname} etcd[53042]: transport: http2Server.HandleStreams failed to read frame: read tcp {etcd_ip}:2379->{application_ip}:44460: read: connection timed out
Not sure if this is red herring but the time of the error event does kinda match the waiting time on the goroutine. We do see this waiting time on a well-behaved box to be a few minutes and we think it's because we had WithProgressNotify option enabled by default, can you verify this theory?
We also see read: connection timed out and read: connection reset by peer pretty often in the logs.
We are not able to reproduce this issue, but restarting the problematic application could fix the issue. There are still a few applications running in the problematic state, they all seem to have connections to every etcd box, just won't be able to receive any updates. Is there any thing we could try to debug this?

We know there were some issues with clientv3 balancer that might cause the clients to be pending on bad connections.
We just released 3.2.10 to fix the client side issues we found.
However, we are not sure why you see read: connection timed out or read: connection reset by peer.
My suggestion is:

try hard to reproduce the problem
try to upgrade to 3.2.10 to see if it would solve the issue

Not sure if this is red herring but the time of the error event does kinda match the waiting time on the goroutine. We do see this waiting time on a well-behaved box to be a few minutes and we think it's because we had WithProgressNotify option enabled by default, can you verify this theory?

maybe the connection set a timeout? and when notified, the read/write timeout already hit? what is the client configuration you use? i think you can easily reproduce this issue with a simple server and client.

Thank you for the suggestions. We could try upgrading the clients to 3.2.10. Would it be necessary to upgrade the servers to 3.2.10 as well?
The read: connection reset by peer error is a red herring, it's primarily because client processes restarting.
I don't think we explicitly set a timeout on the connections, otherwise I'd imagine the client should get a watch error when the connection times out, in which case the goroutine watching the grpc stream shouldn't being blocked for 3 days, but I can double check.

Just checked and verified that the reason why the goroutine was only blocked for a few minutes in the clients that did get the update was because we enabled "WithProgressNotify", so the well-behaved clients got notified approximately every 10 minutes even though there was no key/value updates in etcd.
It seems the ill-behaved clients didn't get any notification updates since the etcd server it watches got the `http2Server.HandleStreams: ... read: connection timed out" error for the client, which led me to suspect that perhaps the GRPC watch stream was somehow leaked either on the client side or the server side for those ill-behaved clients, because no updates would be sent to those streams the clients were watching.

Thank you for the suggestions. We could try upgrading the clients to 3.2.10. Would it be necessary to upgrade the servers to 3.2.10 as well?

It is mostly a client-side issue. You also need to update gRPC dependencies to 1.7.3. It would be great if you can update the server though.

It seems the ill-behaved clients didn't get any progress notification updates since the etcd server it watches got the `http2Server.HandleStreams: ... read: connection timed out" error for the client, which led me to suspect that perhaps the GRPC watch stream was somehow leaked either on the client side or the server side for those ill-behaved clients, because no updates would be sent to those streams the clients were watching.

Right. There was a problem in the previous gRPC balancer implementation. I am not 100% sure the exact problem you hit is resolved. Thus, if you can reproduce it, we would like to take a look.
One thing I would suggest you look at is the slow watcher and watcher count metrics. If the slow watcher metrics are normal (at low count), we can probably rule out etcd server side issue (but it still can be a gRPC server side problem).

Cool, we'll try upgrading the client and the server to 3.2.10 and perform the same operation that triggered this issue. I don't know how difficult it is to repro this locally since I'm not sure what condition triggered the server-side "read: connection timed out" error, which appeared to be the direct cause of this issue (?), and since not all clients are affected, i'd think it might be non-deterministic and thus difficult to repro locally, but I can give it a shot.
FYI, the slow watcher count is very low (mostly 0, occasionally 1).

@xichen2020 Great. We will also look into the problem in the next couple of days after thanks giving.

I think the ill-behaved clients are still connected to the etcd servers (at least the connections are established in netstat). If it's the stream that's leaked but both sides are still connected, I don't think keepalives are going to help us in this particular case.

If the stream is leaked but the connection is not closed

Possibly. It is also possible that the connection, the gRPC stream are all closed correctly, but the logical stream in client library fails to get closed.

@cw9 @xichen2020
One quick question: are all the watchers on the same client blocked forever or just some of the watchers in the same client?

All the watchers on the same client are blocked forever. No key value updates can be propagated to that client.

cc @xiang90 @cw9
Want to provide some update on this: we upgraded both our clients and our server to 3.2.10 and it seems to have helped. We performed the this operation that triggered this issue and this time it appears that there was one client that didn't get the latest update from etcd.
We saw the same issue with this ill-behaving client, i.e., the watchers on the client were blocked. However in this case, it appears that the client thinks it is still connected to all etcd servers (netstat on client side), but none of the etcd servers are actually connected to the client (netstat on server side). Looking further, in the server logs, all the servers got:
transport: http2Server.HandleStreams failed to read frame: read tcp server_ip:server_port->client_ip:client_port: read: connection timed out

around the same time (a few minutes apart), which appeared to be when the watchers in the client stopped getting notification updates. I'm guessing this is a case where connections are closed on the server side but not on the client side, and since the client does not have keepAlives enabled, the client doesn't know the connection has been closed by the server and is therefor stuck forever.
I'll try enabling KeepAlives on the client side and see if that fixes the issue, unless you have other suggestions?

@xichen2020
Please enable keepalive and give it another try.

We performed the this operation

Did you find a way to reproduce the problem? We would like to know how to reproduce so that we can cover this case as an integration test. Thanks.

I tried reproducing it locally but so far no luck, it appears to be somewhat nondeterministic. Will look into it a bit further.

@xichen2020
I looked into the error you posted a little bit deeper today.
The error message is directly returned by the recv syscall (ETIMEDOUT as error 110: see https://github.com/golang/go/blob/964639cc338db650ccadeafb7424bc8ebb2c0f6c/src/syscall/zerrors_linux_386.go#L1488).
Recv itself should not cause ETIMEDOUT apparently. Either a failure to send or a failure of TCP keepalive makes the server realize that connection is broken.
If the client side however, I assume, only receives on the broken connection. So if the connection is not closed by the client-side (as you proved with netstats), the client side wont know its close anyway. Enabling client side keepalive will help on this case.
I am not entirely sure what caused the connection to be broken. However, since the error is returned from the kernel, I think there was something below etcd/gRPC layer.

Possibly, I'm guessing some temporary network blip on the client-server link could have caused the broken connections but not 100% sure.
Just out of curiosity, it seems the keepAlives are done by sending periodic probes inside a goroutine rather than setting the keepAlive options on the TCP socket, any idea why?

Just out of curiosity, it seems the keepAlives are done by sending periodic probes inside a goroutine rather than setting the keepAlive options on the TCP socket, any idea why?

This is a decision gRPC team made. So I can only speak of the reasoning behind it from my own experience.
Normally speaking, application layer keepalive is more reliable than TCP keepalive:

TCP keepalive behaviors differently across different platforms
TCP keepalive is not easy to config correctly
TCP keepalive cannot detect application layer "deadlock" or misbehavior; as long as the kernel still behaviors, the connection wont be closed

However, application layer keepalive might use more resources than TCP keepalive. Too frequent ping has to be avoided.

That's fair, and thanks for the explanations. Seems the keepAlives are done per underlying connection rather than per gRPC stream, so the overhead is higher than socket-level probing but probably still manageable with not too frequent pings, although the clients probably need to add some jittering to avoid all clients sending keepAlive probes at the same time.

Just to close the loop here: we performed a few more tests that previously triggered this issue after upgrading both the client and the server to 3.2.10 and adding client-side keepAlives, and did not see this issue come up since the change. As such, we consider this issue resolved with the version upgrade and the client-side keepAlives.

@xichen2020 Awesome. Thanks. I am closing this for now. If the same issue poped up again, please reopen. We will add more testing around watchers in the near future.


Conda install (@ 2020, Apr. 13) automatically install version 0.04 (the oldest version)

Describe the bug
Installation Version Error
To Reproduce
On a CentOS 7.6 x64 system, with newly installed Anaconda3, install the package following the instruction will lead to version 0.04 installed.
This could be an Anaconda (Cloud) setting problem.
(Time: 2020, Apr 13)

This can totally be due to the new ambermini restrictions introduced in https://github.com/conda-forge/ambertools-feedstock/pull/29.
Can you paste the output of conda list?

The current list is twisted by me ...
Please note that I firstly installed the ambertools via Anaconda (python 3.7.6) binary,
then RDKit,
then OpenMM.
The openforcefield was installed after the above packages.
(There seems to be no "ambermini" in my installation)
I thought the package dependencies could be solved automatically.
*
Here is my solution, and it works for the runing of ethanol and cyclohexane case:
Download version 0.6.0 tar
Download the versioin 0.6.0 openforcefield and install it locally.

Here is the output of conda install openforcefield again (just print the solved environment)
$ conda install openforcefield
Collecting package metadata (current_repodata.json): done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - <unknown>/linux-64::openforcefield==0.6.0=py37_1                                                         done

## Package Plan ##

  environment location: /home/kael/anaconda3

  added / updated specs:
    - openforcefield

The following packages will be SUPERSEDED by a higher-priority channel:
openforcefield     ::openforcefield-0.6.0-py37_1 --> omnia::openforcefield-0.0.4-py37_1

I see! I am adding tests for this scenario in #577.
In the meantime, I could create a new environment that correctly installs v0.6:
$ conda create -n openff576 openforcefield

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/jaime/.conda/envs/openff576

  added / updated specs:
    - openforcefield

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ambertools-19.11           |   py37h00db77c_5        74.5 MB  conda-forge
    boost-1.72.0               |   py37h9de70de_0         316 KB  conda-forge
    boost-cpp-1.72.0           |       h8e57a91_0        21.8 MB  conda-forge
    cairo-1.16.0               |    hcf35c78_1003         1.5 MB  conda-forge
    glib-2.58.3                |py37he00f558_1004         3.3 MB  conda-forge
    libtiff-4.1.0              |       hc7e4089_6         668 KB  conda-forge
    libwebp-base-1.1.0         |       h516909a_3         845 KB  conda-forge
    llvm-openmp-10.0.0         |       hc9558a2_0         2.8 MB  conda-forge
    lz4-c-1.9.2                |       he1b5a44_0         223 KB  conda-forge
    mistune-0.8.4              |py37h8f50634_1001          53 KB  conda-forge
    nbconvert-5.6.1            |   py37hc8dfbb8_1         487 KB  conda-forge
    netcdf-fortran-4.5.2       |nompi_h45d7149_104         1.2 MB  conda-forge
    pillow-7.1.1               |   py37h718be6c_0         654 KB  conda-forge
    pycairo-1.19.1             |   py37h01af8b0_3          77 KB  conda-forge
    rdkit-2020.03.1            |   py37hdd87690_1        24.7 MB  conda-forge
    scipy-1.4.1                |   py37ha3d9a3c_3        18.8 MB  conda-forge
    urllib3-1.25.8             |   py37hc8dfbb8_1         160 KB  conda-forge
    yaml-0.2.3                 |       h516909a_0          82 KB  conda-forge
    zstd-1.4.4                 |       h6597ccf_3         991 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       153.0 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_llvm
  ambertools         conda-forge/linux-64::ambertools-19.11-py37h00db77c_5
  attrs              conda-forge/noarch::attrs-19.3.0-py_0
  backcall           conda-forge/noarch::backcall-0.1.0-py_0
  bleach             conda-forge/noarch::bleach-3.1.4-pyh9f0ad1d_0
  boost              conda-forge/linux-64::boost-1.72.0-py37h9de70de_0
  boost-cpp          conda-forge/linux-64::boost-cpp-1.72.0-h8e57a91_0
  bson               conda-forge/noarch::bson-0.5.9-py_0
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_2
  ca-certificates    conda-forge/linux-64::ca-certificates-2<IP_ADDRESS>-hecc5488_0
  cairo              conda-forge/linux-64::cairo-1.16.0-hcf35c78_1003
  certifi            conda-forge/linux-64::certifi-2<IP_ADDRESS>-py37hc8dfbb8_0
  cffi               conda-forge/linux-64::cffi-1.14.0-py37hd463f26_0
  chardet            conda-forge/linux-64::chardet-3.0.4-py37hc8dfbb8_1006
  cryptography       conda-forge/linux-64::cryptography-2.8-py37hb09aad4_2
  curl               conda-forge/linux-64::curl-7.69.1-h33f0ec9_0
  cycler             conda-forge/noarch::cycler-0.10.0-py_2
  cython             conda-forge/linux-64::cython-0.29.16-py37h3340039_0
  decorator          conda-forge/noarch::decorator-4.4.2-py_0
  defusedxml         conda-forge/noarch::defusedxml-0.6.0-py_0
  entrypoints        conda-forge/linux-64::entrypoints-0.3-py37hc8dfbb8_1001
  fftw               conda-forge/linux-64::fftw-3.3.8-nompi_h7f3a6c3_1110
  fftw3f             omnia/linux-64::fftw3f-3.3.4-2
  fontconfig         conda-forge/linux-64::fontconfig-2.13.1-h86ecdb6_1001
  freetype           conda-forge/linux-64::freetype-2.10.1-he06d7ca_0
  gettext            conda-forge/linux-64::gettext-<IP_ADDRESS>-hc5be6a0_1002
  glib               conda-forge/linux-64::glib-2.58.3-py37he00f558_1004
  hdf4               conda-forge/linux-64::hdf4-4.2.13-hf30be14_1003
  hdf5               conda-forge/linux-64::hdf5-1.10.5-nompi_h3c11f04_1104
  icu                conda-forge/linux-64::icu-64.2-he1b5a44_1
  idna               conda-forge/noarch::idna-2.9-py_1
  importlib-metadata conda-forge/linux-64::importlib-metadata-1.6.0-py37hc8dfbb8_0
  importlib_metadata conda-forge/noarch::importlib_metadata-1.6.0-0
  ipykernel          conda-forge/linux-64::ipykernel-5.2.0-py37h43977f1_1
  ipython            conda-forge/linux-64::ipython-7.13.0-py37hc8dfbb8_2
  ipython_genutils   conda-forge/noarch::ipython_genutils-0.2.0-py_1
  jedi               conda-forge/linux-64::jedi-0.16.0-py37hc8dfbb8_1
  jinja2             conda-forge/noarch::jinja2-2.11.1-py_0
  jpeg               conda-forge/linux-64::jpeg-9c-h14c3975_1001
  jsonschema         conda-forge/linux-64::jsonschema-3.2.0-py37hc8dfbb8_1
  jupyter_client     conda-forge/noarch::jupyter_client-6.1.2-py_0
  jupyter_core       conda-forge/linux-64::jupyter_core-4.6.3-py37hc8dfbb8_1
  kiwisolver         conda-forge/linux-64::kiwisolver-1.2.0-py37h99015e2_0
  krb5               conda-forge/linux-64::krb5-1.17.1-h2fd8d38_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.34-h53a641e_0
  libblas            conda-forge/linux-64::libblas-3.8.0-16_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-16_openblas
  libcurl            conda-forge/linux-64::libcurl-7.69.1-hf7181ac_0
  libedit            conda-forge/linux-64::libedit-3.1.20170329-hf8c457e_1001
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-7.3.0-hdf63c60_5
  libgomp            conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2
  libiconv           conda-forge/linux-64::libiconv-1.15-h516909a_1006
  liblapack          conda-forge/linux-64::liblapack-3.8.0-16_openblas
  libnetcdf          conda-forge/linux-64::libnetcdf-4.7.4-nompi_h9f9fd6a_101
  libopenblas        conda-forge/linux-64::libopenblas-0.3.9-h5ec1e0e_0
  libpng             conda-forge/linux-64::libpng-1.6.37-hed695b0_1
  libsodium          conda-forge/linux-64::libsodium-1.0.17-h516909a_0
  libssh2            conda-forge/linux-64::libssh2-1.8.2-h22169c7_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2
  libtiff            conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
  libwebp-base       conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3
  libxcb             conda-forge/linux-64::libxcb-1.13-h14c3975_1002
  libxml2            conda-forge/linux-64::libxml2-2.9.10-hee79883_0
  llvm-openmp        conda-forge/linux-64::llvm-openmp-10.0.0-hc9558a2_0
  lz4-c              conda-forge/linux-64::lz4-c-1.9.2-he1b5a44_0
  markupsafe         conda-forge/linux-64::markupsafe-1.1.1-py37h8f50634_1
  matplotlib-base    conda-forge/linux-64::matplotlib-base-3.2.1-py37h30547a4_0
  mistune            conda-forge/linux-64::mistune-0.8.4-py37h8f50634_1001
  msgpack-python     conda-forge/linux-64::msgpack-python-1.0.0-py37h99015e2_1
  nbconvert          conda-forge/linux-64::nbconvert-5.6.1-py37hc8dfbb8_1
  nbformat           conda-forge/noarch::nbformat-5.0.4-py_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  netcdf-fortran     conda-forge/linux-64::netcdf-fortran-4.5.2-nompi_h45d7149_104
  networkx           conda-forge/noarch::networkx-2.4-py_1
  notebook           conda-forge/linux-64::notebook-6.0.3-py37_0
  numpy              conda-forge/linux-64::numpy-1.18.1-py37h8960a57_1
  olefile            conda-forge/noarch::olefile-0.46-py_0
  openforcefield     omnia/linux-64::openforcefield-0.6.0-py37_1
  openforcefields    omnia/linux-64::openforcefields-1.1.0-py37_0
  openmm             omnia/linux-64::openmm-7.4.1-py37_cuda101_rc_1
  openssl            conda-forge/linux-64::openssl-1.1.1f-h516909a_0
  packaging          conda-forge/noarch::packaging-20.1-py_0
  pandas             conda-forge/linux-64::pandas-1.0.3-py37h0da4684_0
  pandoc             conda-forge/linux-64::pandoc-2.9.2-0
  pandocfilters      conda-forge/noarch::pandocfilters-1.4.2-py_1
  parso              conda-forge/noarch::parso-0.6.2-py_0
  pcre               conda-forge/linux-64::pcre-8.44-he1b5a44_0
  perl               conda-forge/linux-64::perl-5.26.2-h516909a_1006
  pexpect            conda-forge/linux-64::pexpect-4.8.0-py37hc8dfbb8_1
  pickleshare        conda-forge/linux-64::pickleshare-0.7.5-py37hc8dfbb8_1001
  pillow             conda-forge/linux-64::pillow-7.1.1-py37h718be6c_0
  pip                conda-forge/noarch::pip-20.0.2-py_2
  pixman             conda-forge/linux-64::pixman-0.38.0-h516909a_1003
  prometheus_client  conda-forge/noarch::prometheus_client-0.7.1-py_0
  prompt-toolkit     conda-forge/noarch::prompt-toolkit-3.0.5-py_0
  pthread-stubs      conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
  ptyprocess         conda-forge/noarch::ptyprocess-0.6.0-py_1001
  pycairo            conda-forge/linux-64::pycairo-1.19.1-py37h01af8b0_3
  pycparser          conda-forge/noarch::pycparser-2.20-py_0
  pygments           conda-forge/noarch::pygments-2.6.1-py_0
  pyopenssl          conda-forge/noarch::pyopenssl-19.1.0-py_1
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pyrsistent         conda-forge/linux-64::pyrsistent-0.16.0-py37h8f50634_0
  pysocks            conda-forge/linux-64::pysocks-1.7.1-py37hc8dfbb8_1
  python             conda-forge/linux-64::python-3.7.6-h8356626_5_cpython
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  python_abi         conda-forge/linux-64::python_abi-3.7-1_cp37m
  pytz               conda-forge/noarch::pytz-2019.3-py_0
  pyyaml             conda-forge/linux-64::pyyaml-5.3.1-py37h8f50634_0
  pyzmq              conda-forge/linux-64::pyzmq-19.0.0-py37hac76be4_1
  rdkit              conda-forge/linux-64::rdkit-2020.03.1-py37hdd87690_1
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  requests           conda-forge/noarch::requests-2.23.0-pyh8c360ce_2
  scipy              conda-forge/linux-64::scipy-1.4.1-py37ha3d9a3c_3
  send2trash         conda-forge/noarch::send2trash-1.5.0-py_0
  setuptools         conda-forge/linux-64::setuptools-46.1.3-py37hc8dfbb8_0
  six                conda-forge/noarch::six-1.14.0-py_1
  smirnoff99frosst   omnia/linux-64::smirnoff99frosst-1.1.0-py37_1
  sqlite             conda-forge/linux-64::sqlite-3.30.1-hcee41ef_0
  terminado          conda-forge/linux-64::terminado-0.8.3-py37hc8dfbb8_1
  testpath           conda-forge/noarch::testpath-0.4.4-py_0
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  toml               conda-forge/noarch::toml-0.10.0-py_0
  tornado            conda-forge/linux-64::tornado-6.0.4-py37h8f50634_1
  traitlets          conda-forge/linux-64::traitlets-4.3.3-py37hc8dfbb8_1
  urllib3            conda-forge/linux-64::urllib3-1.25.8-py37hc8dfbb8_1
  wcwidth            conda-forge/noarch::wcwidth-0.1.9-pyh9f0ad1d_0
  webencodings       conda-forge/noarch::webencodings-0.5.1-py_1
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  xmltodict          conda-forge/noarch::xmltodict-0.12.0-py_0
  xorg-kbproto       conda-forge/linux-64::xorg-kbproto-1.0.7-h14c3975_1002
  xorg-libice        conda-forge/linux-64::xorg-libice-1.0.10-h516909a_0
  xorg-libsm         conda-forge/linux-64::xorg-libsm-1.2.3-h84519dc_1000
  xorg-libx11        conda-forge/linux-64::xorg-libx11-1.6.9-h516909a_0
  xorg-libxau        conda-forge/linux-64::xorg-libxau-1.0.9-h14c3975_0
  xorg-libxdmcp      conda-forge/linux-64::xorg-libxdmcp-1.1.3-h516909a_0
  xorg-libxext       conda-forge/linux-64::xorg-libxext-1.3.4-h516909a_0
  xorg-libxrender    conda-forge/linux-64::xorg-libxrender-0.9.10-h516909a_1002
  xorg-libxt         conda-forge/linux-64::xorg-libxt-1.2.0-h516909a_0
  xorg-renderproto   conda-forge/linux-64::xorg-renderproto-0.11.1-h14c3975_1002
  xorg-xextproto     conda-forge/linux-64::xorg-xextproto-7.3.0-h14c3975_1002
  xorg-xproto        conda-forge/linux-64::xorg-xproto-7.0.31-h14c3975_1007
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_0
  yaml               conda-forge/linux-64::yaml-0.2.3-h516909a_0
  zeromq             conda-forge/linux-64::zeromq-4.3.2-he1b5a44_2
  zipp               conda-forge/noarch::zipp-3.1.0-py_0
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006
  zstd               conda-forge/linux-64::zstd-1.4.4-h6597ccf_3

How are your channels defined? Mine are:
$ conda config --show

[...]
channel_priority: flexible
channels:
  - conda-forge
  - defaults
  - omnia
[...]

channel_alias: https://conda.anaconda.org
channel_priority: flexible
channels:
  - conda-forge
  - omnia
  - defaults
client_ssl_cert: None
client_ssl_cert_key: None

Try putting omnia under defaults.  You should see these contents in ~/.condarc.

No, it didn't work. Here is the conda config section after the reorder:
channel_alias: https://conda.anaconda.org
channel_priority: flexible
channels:
  - conda-forge
  - defaults
  - omnia
client_ssl_cert: None

here is the output of the conda install openforcefield:
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - <unknown>/linux-64::openforcefield==0.6.0=py37_1                                             done

## Package Plan ##

  environment location: /home/kael/anaconda3

  added / updated specs:
    - openforcefield

The following packages will be SUPERSEDED by a higher-priority channel:

  openforcefield     <unknown>::openforcefield-0.6.0-py37_1 --> omnia::openforcefield-0.0.4-py37_1

Can you try creating a new environment instead of installing the package on the base one? That will make it easier for the dep solver:
conda create -n openforcefield openforcefield
conda activate openforcefield

Here is the output of conda create -n openforcefield openforcefield
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/kael/anaconda3/envs/openforcefield

  added / updated specs:
    - openforcefield

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |           1_llvm           5 KB  conda-forge
    ambertools-19.11           |   py37h00db77c_5        74.5 MB  conda-forge
    backcall-0.1.0             |             py_0          13 KB  conda-forge
    bleach-3.1.4               |     pyh9f0ad1d_0         111 KB  conda-forge
    boost-1.72.0               |   py37h9de70de_0         316 KB  conda-forge
    boost-cpp-1.72.0           |       h8e57a91_0        21.8 MB  conda-forge
    bson-0.5.9                 |             py_0          14 KB  conda-forge
    bzip2-1.0.8                |       h516909a_2         396 KB  conda-forge
    ca-certificates-2<IP_ADDRESS> |       hecc5488_0         146 KB  conda-forge
    cairo-1.16.0               |    hcf35c78_1003         1.5 MB  conda-forge
    certifi-2<IP_ADDRESS>         |   py37hc8dfbb8_0         151 KB  conda-forge
    cffi-1.14.0                |   py37hd463f26_0         221 KB  conda-forge
    chardet-3.0.4              |py37hc8dfbb8_1006         169 KB  conda-forge
    cryptography-2.8           |   py37hb09aad4_2         617 KB  conda-forge
    curl-7.69.1                |       h33f0ec9_0         137 KB  conda-forge
    cycler-0.10.0              |             py_2           9 KB  conda-forge
    cython-0.29.16             |   py37h3340039_0         2.2 MB  conda-forge
    decorator-4.4.2            |             py_0          11 KB  conda-forge
    entrypoints-0.3            |py37hc8dfbb8_1001          12 KB  conda-forge
    fftw-3.3.8                 |nompi_h7f3a6c3_1110         5.8 MB  conda-forge
    fontconfig-2.13.1          |    h86ecdb6_1001         340 KB  conda-forge
    freetype-2.10.1            |       he06d7ca_0         877 KB  conda-forge
    gettext-<IP_ADDRESS>           |    hc5be6a0_1002         3.6 MB  conda-forge
    glib-2.58.3                |py37he00f558_1004         3.3 MB  conda-forge
    hdf4-4.2.13                |    hf30be14_1003         964 KB  conda-forge
    hdf5-1.10.5                |nompi_h3c11f04_1104         3.1 MB  conda-forge
    icu-64.2                   |       he1b5a44_1        12.6 MB  conda-forge
    idna-2.9                   |             py_1          52 KB  conda-forge
    importlib-metadata-1.6.0   |   py37hc8dfbb8_0          43 KB  conda-forge
    importlib_metadata-1.6.0   |                0           3 KB  conda-forge
    ipykernel-5.2.0            |   py37h43977f1_1         161 KB  conda-forge
    ipython-7.13.0             |   py37hc8dfbb8_2         1.1 MB  conda-forge
    ipython_genutils-0.2.0     |             py_1          21 KB  conda-forge
    jedi-0.16.0                |   py37hc8dfbb8_1         778 KB  conda-forge
    jpeg-9c                    |    h14c3975_1001         251 KB  conda-forge
    jsonschema-3.2.0           |   py37hc8dfbb8_1          89 KB  conda-forge
    jupyter_client-6.1.2       |             py_0          74 KB  conda-forge
    jupyter_core-4.6.3         |   py37hc8dfbb8_1          71 KB  conda-forge
    kiwisolver-1.2.0           |   py37h99015e2_0          87 KB  conda-forge
    krb5-1.17.1                |       h2fd8d38_0         1.5 MB  conda-forge
    ld_impl_linux-64-2.34      |       h53a641e_0         616 KB  conda-forge
    libblas-3.8.0              |      16_openblas          10 KB  conda-forge
    libcblas-3.8.0             |      16_openblas          10 KB  conda-forge
    libcurl-7.69.1             |       hf7181ac_0         573 KB  conda-forge
    libedit-3.1.20170329       |    hf8c457e_1001         172 KB  conda-forge
    libffi-3.2.1               |    he1b5a44_1007          47 KB  conda-forge
    libgcc-ng-9.2.0            |       h24d8f2e_2         8.2 MB  conda-forge
    libgfortran-ng-7.3.0       |       hdf63c60_5         1.7 MB  conda-forge
    libgomp-9.2.0              |       h24d8f2e_2         816 KB  conda-forge
    libiconv-1.15              |    h516909a_1006         2.0 MB  conda-forge
    liblapack-3.8.0            |      16_openblas          10 KB  conda-forge
    libnetcdf-4.7.4            |nompi_h9f9fd6a_101         1.3 MB  conda-forge
    libopenblas-0.3.9          |       h5ec1e0e_0         7.8 MB  conda-forge
    libpng-1.6.37              |       hed695b0_1         308 KB  conda-forge
    libsodium-1.0.17           |       h516909a_0         330 KB  conda-forge
    libssh2-1.8.2              |       h22169c7_2         257 KB  conda-forge
    libstdcxx-ng-9.2.0         |       hdf63c60_2         4.5 MB  conda-forge
    libtiff-4.1.0              |       hc7e4089_6         668 KB  conda-forge
    libuuid-2.32.1             |    h14c3975_1000          26 KB  conda-forge
    libwebp-base-1.1.0         |       h516909a_3         845 KB  conda-forge
    libxcb-1.13                |    h14c3975_1002         396 KB  conda-forge
    libxml2-2.9.10             |       hee79883_0         1.3 MB  conda-forge
    llvm-openmp-10.0.0         |       hc9558a2_0         2.8 MB  conda-forge
    lz4-c-1.9.2                |       he1b5a44_0         223 KB  conda-forge
    markupsafe-1.1.1           |   py37h8f50634_1          26 KB  conda-forge
    matplotlib-base-3.2.1      |   py37h30547a4_0         7.1 MB  conda-forge
    mistune-0.8.4              |py37h8f50634_1001          53 KB  conda-forge
    msgpack-python-1.0.0       |   py37h99015e2_1          90 KB  conda-forge
    nbconvert-5.6.1            |   py37hc8dfbb8_1         487 KB  conda-forge
    ncurses-6.1                |    hf484d3e_1002         1.3 MB  conda-forge
    netcdf-fortran-4.5.2       |nompi_h45d7149_104         1.2 MB  conda-forge
    networkx-2.4               |             py_1         1.2 MB  conda-forge
    numpy-1.18.1               |   py37h8960a57_1         5.2 MB  conda-forge
    olefile-0.46               |             py_0          31 KB  conda-forge
    openforcefield-0.6.0       |           py37_1        29.7 MB  omnia
    openssl-1.1.1f             |       h516909a_0         2.1 MB  conda-forge
    pandas-1.0.3               |   py37h0da4684_0        11.1 MB  conda-forge
    pandoc-2.9.2               |                0        16.8 MB  conda-forge
    pandocfilters-1.4.2        |             py_1           9 KB  conda-forge
    parso-0.6.2                |             py_0          66 KB  conda-forge
    pcre-8.44                  |       he1b5a44_0         261 KB  conda-forge
    perl-5.26.2                |    h516909a_1006        15.4 MB  conda-forge
    pexpect-4.8.0              |   py37hc8dfbb8_1          79 KB  conda-forge
    pickleshare-0.7.5          |py37hc8dfbb8_1001          13 KB  conda-forge
    pillow-7.1.1               |   py37h718be6c_0         654 KB  conda-forge
    pip-20.0.2                 |             py_2         1.0 MB  conda-forge
    pixman-0.38.0              |    h516909a_1003         594 KB  conda-forge
    prompt-toolkit-3.0.5       |             py_0         232 KB  conda-forge
    pthread-stubs-0.4          |    h14c3975_1001           5 KB  conda-forge
    ptyprocess-0.6.0           |          py_1001          15 KB  conda-forge
    pycairo-1.19.1             |   py37h01af8b0_3          77 KB  conda-forge
    pycparser-2.20             |             py_0          89 KB  conda-forge
    pygments-2.6.1             |             py_0         683 KB  conda-forge
    pyopenssl-19.1.0           |             py_1          47 KB  conda-forge
    pyparsing-2.4.7            |     pyh9f0ad1d_0          60 KB  conda-forge
    pyrsistent-0.16.0          |   py37h8f50634_0          89 KB  conda-forge
    pysocks-1.7.1              |   py37hc8dfbb8_1          27 KB  conda-forge
    python-3.7.6               |h8356626_5_cpython        52.9 MB  conda-forge
    pyyaml-5.3.1               |   py37h8f50634_0         185 KB  conda-forge
    pyzmq-19.0.0               |   py37hac76be4_1         503 KB  conda-forge
    rdkit-2020.03.1            |   py37hdd87690_1        24.7 MB  conda-forge
    readline-8.0               |       hf8c457e_0         441 KB  conda-forge
    requests-2.23.0            |     pyh8c360ce_2          47 KB  conda-forge
    scipy-1.4.1                |   py37ha3d9a3c_3        18.8 MB  conda-forge
    send2trash-1.5.0           |             py_0          12 KB  conda-forge
    setuptools-46.1.3          |   py37hc8dfbb8_0         634 KB  conda-forge
    six-1.14.0                 |             py_1          13 KB  conda-forge
    smirnoff99frosst-1.1.0     |           py37_1          72 KB  omnia
    sqlite-3.30.1              |       hcee41ef_0         2.0 MB  conda-forge
    terminado-0.8.3            |   py37hc8dfbb8_1          23 KB  conda-forge
    tk-8.6.10                  |       hed695b0_0         3.2 MB  conda-forge
    toml-0.10.0                |             py_0          16 KB  conda-forge
    tornado-6.0.4              |   py37h8f50634_1         639 KB  conda-forge
    traitlets-4.3.3            |   py37hc8dfbb8_1         133 KB  conda-forge
    urllib3-1.25.8             |   py37hc8dfbb8_1         160 KB  conda-forge
    wcwidth-0.1.9              |     pyh9f0ad1d_0          20 KB  conda-forge
    webencodings-0.5.1         |             py_1          12 KB  conda-forge
    wheel-0.34.2               |             py_1          24 KB  conda-forge
    xorg-kbproto-1.0.7         |    h14c3975_1002          26 KB  conda-forge
    xorg-libice-1.0.10         |       h516909a_0          57 KB  conda-forge
    xorg-libsm-1.2.3           |    h84519dc_1000          25 KB  conda-forge
    xorg-libx11-1.6.9          |       h516909a_0         918 KB  conda-forge
    xorg-libxau-1.0.9          |       h14c3975_0          13 KB  conda-forge
    xorg-libxdmcp-1.1.3        |       h516909a_0          18 KB  conda-forge
    xorg-libxext-1.3.4         |       h516909a_0          51 KB  conda-forge
    xorg-libxrender-0.9.10     |    h516909a_1002          31 KB  conda-forge
    xorg-libxt-1.2.0           |       h516909a_0         374 KB  conda-forge
    xorg-renderproto-0.11.1    |    h14c3975_1002           8 KB  conda-forge
    xorg-xextproto-7.3.0       |    h14c3975_1002          27 KB  conda-forge
    xorg-xproto-7.0.31         |    h14c3975_1007          72 KB  conda-forge
    xz-5.2.5                   |       h516909a_0         430 KB  conda-forge
    yaml-0.2.3                 |       h516909a_0          82 KB  conda-forge
    zeromq-4.3.2               |       he1b5a44_2         668 KB  conda-forge
    zipp-3.1.0                 |             py_0          10 KB  conda-forge
    zlib-1.2.11                |    h516909a_1006         105 KB  conda-forge
    zstd-1.4.4                 |       h6597ccf_3         991 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       375.8 MB

The following NEW packages will be INSTALLED:

It seems that the pkgs are correctly solved.
Do I need to continue to download the pkgs for your further diagnosis?
(It does not trouble much ...)

Never mind, The pkgs are being downloaded. I will test the case after the new environment is completed.

Though, it seems that the major difference here, is my ambertools pkg.
my ambertools is from the source of ambermd, with size of about 150M, much larger than that on the above list.

ambermd's ambertools might have a different build process than the one in conda-forge. It bundles several libraries instead of adding them as conda dependencies, for example. That might explain the file size difference.

CondaError: Downloaded bytes did not match Content-Length
  url: https://conda.anaconda.org/conda-forge/linux-64/python-3.7.6-h8356626_5_cpython.tar.bz2
  target_path: /home/kael/anaconda3/pkgs/python-3.7.6-h8356626_5_cpython.tar.bz2
  Content-Length: 55424091
  downloaded bytes: 24127919

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/linux-64/rdkit-2020.03.1-py37hdd87690_1.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/omnia/linux-64/openforcefield-0.6.0-py37_1.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.10-hed695b0_0.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

(base) [kael@ecs-s6-large-2-linux-20200412093116 ~]$
Socket error Event: 32 Error: 10053.
Connection closing...Socket close.

Connection closed by foreign host.

That's very odd, @Kael-the-Mad. I also had some weird behavior from some conda channels earlier today, but it seems to have cleared up now. Are you still getting that error?

I've seen the solver try to "upgrade" to 0.0.4 before -- It's a much earlier version of the toolkit, so it had far fewer dependencies. Usually it means that dependencies from conda-forge and omnia can't be found, so the only OpenFF Toolkit version that can have its dependencies satisfied is 0.0.4.

After that failure, I have tried conda install -c omnia openforcefield on a totally different CentOS server.
Yet, the output is even worse:
$ conda install -c omnia openforcefield
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.                                                            failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

It seems that the only successful try to install openforcefield into the 'base' env is to manually install the ***v0.6.0 tar.gz locally.
Creating new environment seems to require too much downloading to the (commercial) cloud servers: once get stuck by several pkgs, the full process could be ruined somehow. (Or I should try my luck later ...)

Maybe the openforcefield pkg is being maintained. Even creating new environment $ conda create -n openforcefield openforcefield failed...
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - openforcefield

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

The openforcefield project really makes the simulation involving small molecules much more convenient.
Hopefully this could be fixed soon!

In this last attempt you are missing the needed channels. Correct command if those channels are not part of your permanent configuration (~/.condarc)  is:
conda create -n openforcefield -c conda-forge -c omnia openforcefield

Thanks for the reminder. You are right.
I will try to create a new env on a third server node one more time.
nohup conda create -n openforcefield -c conda-forge -c omnia -y openforcefield &
I will check it and report in time.

...
pandoc-<IP_ADDRESS>       | 18.7 MB   | ########## | 100%

==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.8.3

Please update conda by running

    $ conda update -n base -c defaults conda

CondaError: Downloaded bytes did not match Content-Length
  url: https://conda.anaconda.org/conda-forge/linux-64/rdkit-2020.03.1-py37hdd87690_1.tar.bz2
  target_path: /home/kael/anaconda3/pkgs/rdkit-2020.03.1-py37hdd87690_1.tar.bz2
  Content-Length: 25866956
  downloaded bytes: 20781501

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/omnia/linux-64/openforcefield-0.6.0-py37_1.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

There is no new env called 'openforcefield'.
Is there a way to 'continue' the download, to only download the previously unavailable pkgs?
By simply running again the nohup conda create -n openforcefield -c conda-forge -c omnia -y openforcefield &, could conda 'remember' that it has downloaded almost all the pkgs, but a few exceptions?

could conda 'remember' that it has downloaded almost all the pkgs, but a few exceptions?

That has been my experience -- I have several packages automatically cached on my computer, so new installs only need to download a few others. And you're using an up-to-date version of conda (4.8) so I'd expect it to have that behavior for you as well.
Thanks so much for digging into this. If you're free to continue, I'd love to figure out if there's a way to make these installations work reliably. However my best guess is that the internet is really saturated right now, and you are far from the package server.
In case you're unable to get the OFF Toolkit installed on a computer where you need it, an alternative is that we provide stable offline installers (which include all dependencies) with each release. For example you can get a "miniconda"-style installer for our latest release in the Assets section of the 0.6.0 GitHub release.

Inspired by your experience, I continued the conda create -n openforcefield -c conda-forge -c omnia  openforcefield. As you expected, the installation has been interrupted by bad internet several times. But the trials got normal conda responses. Finally, the installation is finished (as shown below.)
Proceed ([y]/n)? y

Downloading and Extracting Packages
openforcefield-0.6.0 | 29.7 MB   | ################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate openforcefield
#
# To deactivate an active environment, use
#
#     $ conda deactivate

I will test the examples (with the *.xml replaced), and report later.
I think this has proved that at least the conda create -n env ... would work.
Besides, the commercial cloud servers are located in Beijing, China, and owned by Huawei and Tencent companies. Linux release is CentOS Linux release 7.6.1810 (Core). Downloaded Anaconda installer is 'Anaconda3-2020.02-Linux-x86_64.sh'.
plus
All I need to pay is a little patience for the internet, then I could use an open-source and free of charge tool. Thank you!
I am learning to pack the newly created environment, then transfer it to the other servers.
If you want to add some new tests, please leave a message here.

I'm very happy to see that this works. I'll close this issue for now, since it seems to have been due to internet connectivity, but please feel free to re-open it if you have additional questions.

I am learning to pack the newly created environment, then transfer it to the other servers.

If you are looking for an installation that you can easily transfer to other servers, you should consider using our single-file installers. They work like Miniconda (so they "just install" on basically any system), and once installed, you can add/update conda packages and environments. Full instructions for using these builds are here.


bump 4.1.0-alpha

What have you changed?
This is not an actual release, it just bump up the master version so that it's larger than any known release.
What is the type of the changes?

Bugfix

How is the PR tested?

Manual test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No.
Does this PR affect tidb-ansible?
No.

Did this cause any problem? I'm afraid it's not a good idea since we will use fast-forward in release-4.0 branch.

Did this cause any problem?

It breaks the version checks. Master branch is considered more update to date than any releases.

we will use fast-forward in release-4.0 branch.

You have already bumped 4.0.0-beta on release-4.0 branch, therefore it can't be fast forward.

@yiwu-arbug PTAL


Zulip integration is missing critical error-handling

The matterbridge Zulip integration does not follow our API guidelines for the get_events request.  See https://zulipchat.com/api/get-events-from-queue.  Specifically, it ignores the BAD_EVENT_QUEUE_ID error condition, and thus if the event queue is ever garbage-collected, matterbridge will run in an infinite loop hitting the GET /events API with invalid input, which can contribute to server load.
Matterbridge code here: https://github.com/42wim/matterbridge/blob/master/bridge/zulip/zulip.go#L103
The fix is to check the events output as described.  You can see a sample implementation here: https://github.com/zulip/python-zulip-api/blob/master/zulip/zulip/init.py#L643

From the little I've been able to gather this is actually an issue with the matterbridge/gozulipbot dependency (which hasn't been updated in 2+ years and neither have any forks). It's code does not appear able to handle the BAD_EVENT_QUEUE_ID condition.

@Helcaraxan matterbridge uses the work branch of matterbridge/gozulipbot
I've been adding extra stuff there, also a fix for this now :-)

@42wim good to know. Would be handy if we could document that somewhere in a code-comment or likewise. 🙂


Add portuguese language

Add portuguese language to the bot

@umutxyp, recreated the pull request

@umutxyp, recreated the pull request

added. THANKS


RuntimeError: DataLoader worker (pid 34569) is killed by signal: Bus error. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.

您好，我在训练mmdetction的时候，每次都是训练600~1000steps之后会报错？数据加载的python进程的内存占比会不断升高，直到报错？我是用的slurm集群训练的。我安装的pytorch版本是0.4.1，用的是pip安装的方式。
``    return self.data_queue.get()
File "/mnt/lustre/liudonghui/anaconda3/envs/venv_mmdet/lib/python3.6/multiprocessing/queues.py", line 342, in get
res = self._reader.recv_bytes()
File "/mnt/lustre/liudonghui/anaconda3/envs/venv_mmdet/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/mnt/lustre/liudonghui/anaconda3/envs/venv_mmdet/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/mnt/lustre/liudonghui/anaconda3/envs/venv_mmdet/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "/mnt/lustre/liudonghui/anaconda3/envs/venv_mmdet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 227, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 34569) is killed by signal: Bus error. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.

hi, when i build docker image, and run it, the error come like : RuntimeError: DataLoader worker (pid 147) is killed by signal: Bus error.

Traceback (most recent call last):   File "/tmp/src/pytorch/image_detect/mmdetection_gcc540//tools/train_douyu.py", line 324, in <module>     main()   File "/tmp/src/pytorch/image_detect/mmdetection_gcc540//tools/train_douyu.py", line 320, in main     logger=logger)   File "/tmp/src/pytorch/image_detect/mmdetection_gcc540/mmdet/apis/train.py", line 60, in train_detector     _non_dist_train(model, dataset, cfg, validate=validate)   File "/tmp/src/pytorch/image_detect/mmdetection_gcc540/mmdet/apis/train.py", line 222, in _non_dist_train     runner.run(data_loaders, cfg.workflow, cfg.total_epochs)   File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 358, in run     epoch_runner(data_loaders[i], **kwargs)   File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 260, in train     for i, data_batch in enumerate(data_loader):   File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 804, in __next__     idx, data = self._get_data()   File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 771, in _get_data     success, data = self._try_get_data()   File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 724, in _try_get_data     data = self.data_queue.get(timeout=timeout)   File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler     _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 147) is killed by signal: Bus error.


Version 1.1.0

This will be a major update with better design and better performance.
Personal Rating is now on this repo. I am going to control when to update my app. If I have enough funds, I will host a server in Japan for faster access for users in China.

The main focus of 1.0.7 is adding IAP. Therefore, I have to decide which features should be locked. I have come up with some of them.

WoWs RS
Ship filter
Detailed stats for players and ships

It will be 1.49AUD for the first year and 2.99AUD for further subscriptions.

Version 1.0.7
For larger devices (No longer a priority)

[ ] Menu | Search
[ ] Wiki | Detail
[ ] Player ships | Ship detail

The ratio will be 1:2
For all devices

[ ] Pro version
[ ] Ask user to write a review and share with friends
[ ] Fancy pro version button
[ ] Polish UI
[ ] Code refactor
[ ] Show SAP

Refactor or update first and then refactor?

Outdated


Not able to start services with volumes in rancher-nfs using"driver_opt"

Rancher Versions: v1.6.0-rc3
Cattle environment with 3 hosts
Deploy rancher-nfs - rancher/storage-nfs:v0.8.0
Steps to reproduce the problem:
Create a service with volumes in rancher-nfs using"driver_opt"
version: '2'
volumes:
  yello123:
    driver_opts:
      export: /var/nfs
      host: <IP_ADDRESS>
    driver: rancher-nfs
services:
  nginx5:
    image: nginx
    volumes:
    - yello123:/data1

service fails to start with error - Error (/VolumeDriver.Attach 'testmultiplenfs_yello123_b7391' (driver 'rancher-nfs') returned status 500: 500 Internal Server Error)
Errors seen in nfs-nfs-driver logs:
5/1/2017 4:58:59 PMtime="2017-05-01T23:58:59Z" level=info msg="Running on host ip-172-31-35-191.us-west-2.compute.internal(1h18) with driver rancher-nfs(1sd3)"
5/1/2017 4:58:59 PMtime="2017-05-01T23:58:59Z" level=info msg="Starting plugin for rancher-nfs"
5/1/2017 5:14:56 PMtime="2017-05-02T00:14:56Z" level=info msg="container a280ec9eb4cb183185f7beb935c1124664015a7945a053294c4c842139a21750 destroyed"
5/2/2017 9:20:45 AMtime="2017-05-02T16:20:45Z" level=info msg=create.request name="testmultiplenfs_yello123_b7391" options=map[export:/var/nfs host:<IP_ADDRESS>]
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=info msg="Updating volume testmultiplenfs_yello123_b7391 (1v786) rancher-nfs:1sd3 on 1h18, state inactive => updating-inactive" err=<nil>
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=info msg=create.response
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=info msg=attach.request name="testmultiplenfs_yello123_b7391"
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=error msg=attach.response error="No such volume"
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=info msg=attach.request name="testmultiplenfs_yello123_b7391"
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=error msg=attach.response error="No such volume"
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=info msg=attach.request name="testmultiplenfs_yello123_b7391"
5/2/2017 9:20:47 AMtime="2017-05-02T16:20:47Z" level=error msg=attach.response error="No such volume"

@sangeethah this is fixed in master now.

Tested with Rancher Versions: v1.6.0-rc4
Cattle environment with 3 hosts
Deploy rancher-nfs - llparse/storage-nfs:dev
Create a service with volumes in rancher-nfs using"driver_opt"
version: '2'
volumes:
yello123:
driver_opts:
export: /var/nfs
host: *******
driver: rancher-nfs
services:
nginx5:
image: nginx
volumes:
- yello123:/data1
Able to start service successfully and the see the nfs volume getting created in host provided in driver_opts.


Problem releasing game to appstore

I'm developing a game with this awesome game engine and I want to release my game to appstore. My game uses FB SDK and AdMob and I managed to compile the Godot source for iOS. The game runs fine on simulator and device, But when I archived it in xcode and send it to appstore there is some problems that should be fixed. This is the app store connect error email:
ITMS-90683: Missing Purpose String in Info.plist - Your app's code references one or more APIs that access sensitive user data. The app's Info.plist file should contain a NSBluetoothPeripheralUsageDescription key with a user-facing purpose string explaining clearly and completely why your app needs the data.
And Same for NSCameraUsageDescription and NSBluetoothAlwaysUsageDescription. My game doesn't use camera or bluetooth at all and i think it's for arkit usage in my app. I removed the privacy strings from info.plist but it didn't fix it. How can i disable using bluetooth and camera for my iOS game?

I believe this is an export template issue that will need to be resolved by removing the required APIs and frameworks from the export template. This will require significant effort, and may be easier to simply supply these strings in the short term.
These strings will appear in the permission dialog if you attempt to use bluetooth or camera apis. since you're likely not using either APIs you shouldn't see these strings appear.

@SinaZK  Can you (or anyone else) still reproduce this bug in Godot 3.2.3 or any later release?
If yes, please ensure that an up-to-date Minimal Reproduction Project (MRP) is included in this report (a MRP is a zipped Godot project with the minimal elements necessary to reliably trigger the bug). You can upload ZIP files in an issue comment with a drag and drop.

@SinaZK  Can you (or anyone else) still reproduce this bug in Godot 3.2.3 or any later release?
If yes, please ensure that an up-to-date Minimal Reproduction Project (MRP) is included in this report (a MRP is a zipped Godot project with the minimal elements necessary to reliably trigger the bug). You can upload ZIP files in an issue comment with a drag and drop.

You know I am watching you.
On Thursday, 14 January 2021, Hugo Locurcio<EMAIL_ADDRESS>wrote:

@SinaZK https://github.com/SinaZK Can you (or anyone else) still
reproduce this bug in Godot 3.2.3
https://godotengine.org/article/maintenance-release-godot-3-2-3 or any
later release?
If yes, please ensure that an up-to-date Minimal Reproduction Project
(MRP) is included in this report (a MRP is a zipped Godot project with the
minimal elements necessary to reliably trigger the bug). You can upload ZIP
files in an issue comment with a drag and drop.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/33281#issuecomment-760213633,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALMZECVCYZ5CCGBKFCAJ4YTSZ32FXANCNFSM4JILJRFA
.

--
nicole.gmail.org

You know I am watching you.
On Thursday, 14 January 2021, Hugo Locurcio<EMAIL_ADDRESS>wrote:

@SinaZK https://github.com/SinaZK Can you (or anyone else) still
reproduce this bug in Godot 3.2.3
https://godotengine.org/article/maintenance-release-godot-3-2-3 or any
later release?
If yes, please ensure that an up-to-date Minimal Reproduction Project
(MRP) is included in this report (a MRP is a zipped Godot project with the
minimal elements necessary to reliably trigger the bug). You can upload ZIP
files in an issue comment with a drag and drop.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/33281#issuecomment-760213633,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALMZECVCYZ5CCGBKFCAJ4YTSZ32FXANCNFSM4JILJRFA
.

--
nicole.gmail.org


feat: implement json helpers

Closes #232 and updates the inlayHint code to support the data property.  #231 must be merged first.

This is now ready for review, thanks.

Thanks!


Fix suffix generating and secret recreate UI state changes

Changes proposed in this pull request:

When adding new service binding, both name and secret name have random suffix added
Fix binding name field to not be disabled for all instances when switching from an instance that has this field disabled
When recreating secret for given Service Binding, suggest Secret Name with the same name as binding

Related issue(s)

See #442

/hold

/unhold


msCentipede for atac seq

For atac seq, do I need to pre-shift the bam files? or it has been taken into account.
thanks

Sorry for the late response. For atac seq, you do not need to pre-shift the bam files. This has been taken into account.


Bumped up versions for main Dockerfile

Bumped up

nodejs 14.x -> 16.x (Current LTS version)
npm  6.x -> 8.x

Thanks!


@ConfigProperty not injected in a RouteBuilder

Probably caused by how we instantiate RouteBuilders.
Stumbled across this when adding tests to the twitter test project where we have this
https://github.com/apache/camel-quarkus/blob/f3e7dbcad37aec1ae54b2dfb5d05d2ff462b6e49/integration-tests/twitter/src/main/java/org/apache/camel/quarkus/component/twitter/CamelRoute.java#L26-L27
I guess the users should expect @ConfigProperty and @Inject to work in RouteBuilders?

you can also use properties placeholders in the uri definition with {{...}}
On Thu, 15 Aug 2019 at 16:08, Peter Palaga<EMAIL_ADDRESS>wrote:

Probably caused by how we instantiate RouteBuilders.
Stumbled across this when adding a tests to the twitter test project where
we have this
https://github.com/apache/camel-quarkus/blob/f3e7dbcad37aec1ae54b2dfb5d05d2ff462b6e49/integration-tests/twitter/src/main/java/org/apache/camel/quarkus/component/twitter/CamelRoute.java#L26-L27
I guess the users should expect @ConfigProperty and @Inject to work in
RouteBuilders?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel-quarkus/issues/136?email_source=notifications&email_token=AAOIJBPVXNA62BCCOSVR2QLQEVPO3A5CNFSM4IL6SDUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HFOBVGA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOIJBISWD4XX4S3NJQQKSLQEVPO3ANCNFSM4IL6SDUA
.

--
Luca Burgazzoli

Thanks, {{camel.component.twitter-directmessage.user}} works.
@ConfigProperty and @Inject do not work because we do not let Arc to instantiate them https://github.com/apache/camel-quarkus/blob/c74d4a2846c5bd5b6c2963e9b2d8ccf78fadd4a9/extensions/core/deployment/src/main/java/org/apache/camel/quarkus/core/deployment/CamelInitProcessor.java#L88 The question is whether we should fix this?

there’s a bean post processor in camel that usually take cares of doing
injection so we perhaps need to implement a dedicated one for arc
On Thu, 15 Aug 2019 at 17:45, Peter Palaga<EMAIL_ADDRESS>wrote:

Thanks, {{camel.component.twitter-directmessage.user}} works.
@ConfigProperty and @Inject do not work because we do not let Arc to
instantiate them
https://github.com/apache/camel-quarkus/blob/c74d4a2846c5bd5b6c2963e9b2d8ccf78fadd4a9/extensions/core/deployment/src/main/java/org/apache/camel/quarkus/core/deployment/CamelInitProcessor.java#L88
The question is whether we should fix this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel-quarkus/issues/136?email_source=notifications&email_token=AAOIJBMXZPNKBQIL3AZ3RVTQEV2ZNA5CNFSM4IL6SDUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MFTVI#issuecomment-521689557,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOIJBNQCFLB7KGZNCT4JULQEV2ZNANCNFSM4IL6SDUA
.

--
Luca Burgazzoli

I have primitive PoC how this could work.
Plz. assign to me.


Update PostgreSQL documentation

base64 commands: please use long options so they work on Mac as well
port-forward to Service does not work unless one has made that permission explicitly available
"In a different console, run the PostgreSQL client:" doesn't work unless you also have the environment variables from the previous step
The database and application database creation step does not work, it complains about a database that does not exist (which is what we are trying to create in the first place!), because it seems you have to connect to the postgres database initially: https://elastisys.io/compliantkubernetes/user-guide/additional-services/postgresql/#create-an-application-user
We apparently require SSL traffic, but then, where does one get the certificate information from? The documentation doesn't even mention this or make it clear. This may be required to configure in th application, e.g., https://node-postgres.com/features/ssl

Will a NodeJS application that just sets ssl: true work, or does it need any particular CA information etc? https://stackoverflow.com/questions/22301722/ssl-for-postgresql-connection-nodejs

@llarsson In case you need an urgent answer, our on-boarding process is as follows:

The PostgreSQL admins create a super user Secret of this shape which contains the correct PGSSLMODE variable. This is usually require, i.e., require SSL but don't check the CA. We reserve the right to change this value at any points :smile:, so the super user is expected to follow the instructions and not hard-code the value.

The super user is expected to set the same PGSSLMODE for the application. (See here.)

Usually, all PostgreSQL clients obey the standard PG environment variables, including PGSSLMODE, and the NodeJS client seems to be no different.
Hope this helps.

I'm looking to script the steps so we have a simple script to call for "connect me as admin to my PostgreSQL", since all those instructions are difficult for users, but would be lovely to just have a script do.
I'll add it to this issue as soon as some access things have been figured out. :heart:

The scripts, currently untested due to lacking access permissions and totally in a WIP state, are here:
https://github.com/elastisys/compliantkubernetes/compare/main...postgres-script
Will update them when I get proper access.

The idea is that you should be able to source $(./postgres-details.sh) whenever you need the PG... environment variables set, so you can more easily do both the "connect as admin step" (for which there's a script and you should be able to feed a HEREDOC as stdin to it, with any "create database" commands) and the "generate an application secret" step (no script yet).

Updated the scripts now that I got connection info :)

That did not work, sadly, @cristiklein -- will have to ask someone more knowledgeable regarding the PostgreSQL stuff.


Using React Router for navigation in labextension

Is your feature request related to a problem? Please describe.
Navigation with modals and pop-ups is slow and troublesome for users.
Describe the solution you'd like
Navigation is handled by React Router which renders the paths in an outlet with breadcrumbs for global navigation.
Overview of the tasks at hand

All components will be changed to work with the new router

Components to be changed/updated

CourseManageComponent
LectureListComponent

Big changes
Change to dataloading model for the larger components, these will now use the ReactRouter format, the smaller internal components will continue to pass props.

@meffmadd FYI, I just updated the branch pointer associated with the issue, I merged last weeks changes into "migration/grader-labextension/upgrade-to-jupyterlab-4" and started a new branch: feature/react-router-for-dataloading-components to handle the rest of the TODOs

Marking this as closed, further changes have been picked up in feature/react-router-for-dataloading-components


Would there be code released?

I notice that camera-ready day of this workshop has been passed, would there be code released?

+1.

+1.

+1


Inconsistent tensor sizes error with own data

I tried out my own dataset in data/mydata (with the A and B folders) but I get the following error:
~/DiscoGAN-pytorch$ python main.py --dataset=mydata --num_gpu=1
[*] MODEL dir: logs/mydata_2017-03-21_14-49-25
[*] PARAM path: logs/mydata_2017-03-21_14-49-25/params.json
Traceback (most recent call last):
  File "main.py", line 41, in <module>
    main(config)
  File "main.py", line 33, in main
    trainer.train()
  File "/home/bart/DiscoGAN-pytorch/trainer.py", line 161, in train
    valid_x_A, valid_x_B = self._get_variable(A_loader.next()), self._get_variable(B_loader.next())
  File "/home/bart/anaconda3/envs/Python36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 174, in __next__
    return self._process_next_batch(batch)
  File "/home/bart/anaconda3/envs/Python36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 198, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
  File "/home/bart/anaconda3/envs/Python36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 32, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/bart/anaconda3/envs/Python36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 68, in default_collate
    return torch.stack(batch, 0)
  File "/home/bart/anaconda3/envs/Python36/lib/python3.6/site-packages/torch/functional.py", line 56, in stack
    return torch.cat(list(t.unsqueeze(dim) for t in sequence), dim)
RuntimeError: inconsistent tensor sizes at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.10_1488755368782/work/torch/lib/TH/generic/THTensorMath.c:2548

What is the size and number of images for each dataset?

24467 images in A and 1784 images in B.
On 21 March 2017 at 15:59, Taehoon Kim<EMAIL_ADDRESS>wrote:

What is the size and number of images for each dataset?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/carpedm20/DiscoGAN-pytorch/issues/1#issuecomment-288105678,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGgWsWBXZIU6qNcITCvnsNz6rwoYOwtks5rn-XDgaJpZM4Mj6-n
.

What about the resolution? Are they constant for each dataset?

I did not change anything in the source so it should scale the datasets to
64x64 I suppose. The input images in A and B are of varying size.
On 21 March 2017 at 17:24, Taehoon Kim<EMAIL_ADDRESS>wrote:

What about the resolution? Are they constant for each dataset?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/carpedm20/DiscoGAN-pytorch/issues/1#issuecomment-288134335,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGgWulmX7ebSb3YTVbr2znTfaGSp5kuks5rn_mxgaJpZM4Mj6-n
.

So pytorch will make a tensor first and then will proceed a scaling. To build a tensor, bached images should be concatenated into a single tensor which makes an error due to different sizes.

Using tools like imagemagick will do.
In my case, *mogrify -resize 256x256! -quality 100 -path ../A .jpg (imagemagick command) solved the issue.

Yes I updated README.md to contain this issue. Thanks @dandelin!

Thanks, for scale + center crop:
mogrify -resize 256x256^ -gravity center -crop 256x256+0+0 -quality 100 -path ../A *.jpg


Skip indexes are not in use

Hey, I am trying to tame columnar store in our enterprise solution and faced this behaviour
select org, sum(saleit), sum(costit)
from sale_agg_cs_top
where
dt = '2021-01-01'::date
and it = 0
group by org
execution plan would be
GroupAggregate  (cost=214852.39..214859.59 rows=270 width=68) (actual time=596.516..597.755 rows=981 loops=1)
Group Key: org
Buffers: shared hit=22080
->  Sort  (cost=214852.39..214853.17 rows=315 width=13) (actual time=596.500..596.593 rows=995 loops=1)
Sort Key: org
Sort Method: quicksort  Memory: 71kB
Buffers: shared hit=22080
->  Foreign Scan on sale_agg_cs_top  (cost=0.00..214839.32 rows=315 width=13) (actual time=1.652..595.635 rows=995 loops=1)
Filter: ((dt = '2021-01-01'::date) AND (it = 0))
Rows Removed by Filter: 4362836
CStore File: /var/lib/pgsql/10_5432/data/cstore_fdw/16409/197166589
CStore File Size: 263228026
Buffers: shared hit=22080
Planning time: 0.126 ms
Execution time: 598.100 ms
but when I involve a join
with fact as (
select org, sum(saleit), sum(costit)
from sale_agg_cs_top
where
dt = '2021-01-01'::date
and it = 0
group by org
)
select f.org, ka."JustDescription", f.*
from
fact as f
join
"TableDescription" as ka
on ka.org = f.org
I encounter a great degradation
Nested Loop  (cost=214859.86..215160.24 rows=270 width=101) (actual time=1799.279..1804.731 rows=981 loops=1)
Buffers: shared hit=25023
CTE fact
->  GroupAggregate  (cost=214852.39..214859.59 rows=270 width=68) (actual time=1799.247..1800.919 rows=981 loops=1)
Group Key: sale_agg_cs_top.org
Buffers: shared hit=22080
->  Sort  (cost=214852.39..214853.17 rows=315 width=13) (actual time=1799.213..1799.333 rows=995 loops=1)
Sort Key: sale_agg_cs_top.org
Sort Method: quicksort  Memory: 71kB
Buffers: shared hit=22080
->  Foreign Scan on sale_agg_cs_top  (cost=0.00..214839.32 rows=315 width=13) (actual time=6.196..1798.565 rows=995 loops=1)
Filter: ((dt = '2021-01-01'::date) AND (it = 0))
Rows Removed by Filter: 13497703
CStore File: /var/lib/pgsql/10_5432/data/cstore_fdw/16409/197166589
CStore File Size: 263228026
Buffers: shared hit=22080
->  CTE Scan on fact f  (cost=0.00..5.40 rows=270 width=68) (actual time=1799.253..1801.392 rows=981 loops=1)
Buffers: shared hit=22080
->  Index Scan using "pКонтрагент" on "Контрагент" ka  (cost=0.28..1.09 rows=1 width=33) (actual time=0.003..0.003 rows=1 loops=981)
Index Cond: ("@Лицо" = f.org)
Buffers: shared hit=2943
Planning time: 1.960 ms
Execution time: 1805.475 ms
As you can see first time it scaned 4362836 and second time 13497703. The table sale_agg_cs_top has 13498698 rows. That's why I think skip indexes are not in use.
All tables were analyzed.
Versions
PostgreSQL 10.15 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit
cstore_fdw 1.7
CentOS 7.7.1908

Citus extension has recently started supporting columnar tables please see https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-compression-for-postgres
It uses PostgreSQL's recent table access method features and have better performance than the architecture cstore_fdw was build on.
We strongly recommend switching to citus columnar tables.

Murat,
Thank you very much. I really appreciate it.
Will try cirus extension out when our Postgres is upgrated.
Best regards.
On April 27, 2021 at 12:46 AM, Murat Tuncer @.***> wrote:
Citus extension has recently started supporting columnar tables please see https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-compression-for-postgres
It uses PostgreSQL's recent table access method features and have better performance than the architecture cstore_fdw was build on.
We strongly recommend switching to citus columnar tables.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


Fix regressions in PNG filter

Fixes #305.  This partially reverts #298, but keeps the correct parts.

Please remove the assertion and the clones. To patch the issue order of the two for loops in the Avg and  Paeth case has to be inverted. Then everything should be fine.

Please remove the assertion and the clones. To patch the issue order of the two for loops in the Avg and Paeth case has to be inverted. Then everything should be fine.

Done.

Thanks!


unexpected presence of GO terms

The Artificial Intelligence Ontology is considering importing some terms from SWO

https://bioportal.bioontology.org/ontologies/AIO
https://arxiv.org/abs/2404.03044

In fact, our earliest releases have imported all of SWO, but the presence of terms like 'kinase activity' is at least aesthetically concerning. We have also unsuccessfully tried to extract subsets of SWO with ODK.

This kind of thing is common in OBO ontologies that make Full the default. We are recommend everyone makes a Base ontology release. This is easy with ODK
https://oboacademy.github.io/obook/explanation/owl-format-variants/

Can you make
http://purl.obolibrary.org/obo/swo/swo-base.owl
redirect to this (or the github raw version of it).
We can do this PR if you like
https://github.com/OBOFoundry/purl.obolibrary.org/blob/master/config/swo.yml
(while I'm looking at this including an .obo version will increase uptake among many users)

Sounds like a good plan. I'm in the ELIXIR AHM today (virtually) so it's likely I won't get started until later, FYI.

I'm afraid that I didn't get the chance, as planned, to do this work yesterday, though it remains at the top of my list. I do hope that you are able to use the base file in the meantime.
Thanks!


Add .npmignore to exclude development files from the npm package

I copied .gitignore as .npmignore and added a few development files and folders to be excluded from the npm package as they're unnecessary. Also removed node_modules as it's unnecessary as per npm's documentation.

This isn't necessary. The package.json defines a files array, which acts as a whitelist for what can go into the package.

Sorry, didn't notice that.

Please refrain from spamming projects with PRs like this. It would be desirable to make a PR to add a npmignore if if there was something that was actually in the published tarball that needed to be removed.


[BUG] Results loader impossible to load Equal length and No missing values dataset results

Describe the bug
using get_estimator_results when setting the datasets list parameter to the univariate list in tsc_datasets to get the 128 UCR results. will always give 112
because in tsc_datasets the dataset names are for example "AllGestureWiimoteX" but in the csv files of the results on tsc.com is "AllGestureWiimoteXEq" for example so cant load them
the dataset list should allign with the bake off csv files
Steps/Code to reproduce the bug
from aeon.benchmarking.results_loaders import get_estimator_results
import pandas as pd
import csv
from aeon.datasets.tsc_datasets import univariate

cls = ["TS-CHIEF","MR-Hydra","InceptionTime","H-InceptionTime","HC2"]
results=get_estimator_results(estimators=cls, datasets=univariate)

Expected results
get the 128 datasets
Actual results
getting the 112
Versions
No response

This on me, will look when I get back

urghh, this is tedious, so either I change the website results on tsc.com or add some exception clauses in the code in aeon. Any preference @hadifawaz1999 ?

urghh, this is tedious, so either I change the website results on tsc.com or add some exception clauses in the code in aeon. Any preference @hadifawaz1999 ?

I think an easier solution is a code work in aeon, and leave the website as is

I think only the 112 datasets are uploaded in the first place, so there are no results to load regardless of the name. I think this is more of an issue with the dataset list than the function, if its asked to load AllGestureWiimoteX it should load that and not AllGestureWiimoteXEq.

I think only the 112 datasets are uploaded in the first place, so there are no results to load regardless of the name. I think this is more of an issue with the dataset list than the function, if its asked to load AllGestureWiimoteX it should load that and not AllGestureWiimoteXEq.

any ideas @TonyBagnall  ?


IconToggle: Scrolling does not cancel onPress

If you place an IconToggle inside a ScrollView, the provided onPress handler will fire even if the user begins to scroll. This happens because the IconToggle's onPress handler is called within a child TouchableWithoutFeedback component's onPressOut handler which does not cancel for scroll events like its onPress handler would.
From https://facebook.github.io/react-native/docs/touchablewithoutfeedback#onpress

onPress
Called when the touch is released, but not if cancelled (e.g. by a scroll that steals the responder lock).

The offending implementation is at: https://github.com/xotahal/react-native-material-ui/blob/3dbf5cedbce84a36307a87c67b783bf62dc507af/src/IconToggle/IconToggle.react.js#L186-L203
Reproducible demo: https://snack.expo.io/@twelfley/icontoggle-bug

:tada: This issue has been resolved in version 2.0.0-alpha.1 :tada:
The release is available on:

npm package (@develop dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


listen for key events globally

I am extremely new to druid, so I apologize if this isn't the right place for this.
The motivation is this issue.
Is global/unfocused key event capturing something that's supported in druid? I looked at the event_viewer example and it looks like it only works if the window (and more specifically the widget the controller is wrapped around) is in focus. it also appears that media key event capturing is spotty in Linux. (on my machine media keys aren't captured at all, on some machines it seems that the events are firing some times, but not always?)
Edit: After a quick glance at the gtk implementation for window, it looks like we're connecting key_press/key_release events at the window level, meaning we would only receive keyboard events while the window is in focus. Assuming that's true, I'm hoping this could be a place to discuss a design that supports for listening for events globally.

So long-term yes I would like some concept of 'ambient focus' where we will treat key events more like normal events, and just send them down the tree until handled.
Currently, this doesn't exist; you need to do something a bit hacky, like have a custom Controller at the root of the tree that takes focus, and that then sends the events where you want them. Unfortunately it can't send them using the normal Event mechanism, since druid will check to see if a widget has focus before it delivers a key event. 😒

Global key (device event) capture should be optional, that shouldn't be enabled by default. It also may be detected as key logger by anti-virus programs.
It can also cause hog unneccessary CPU in Windows. e.g. winit does that, and it eats CPU even though window is not visible or minimized..

yes you're right I've misread the issue, listening to key events globally is definitely not a case I've been thinking hard about, and certainly wouldn't be on the near-term roadmap. I could imagine some mechanism where druid apps could register platform-wide hotkeys, but I'd prefer to see some other mechanism for that..

Closing this for now.


Add skip item location validation tests

This change is

Changes Unknown when pulling fef5f7af5b85636867a31f54f119b222e83af133 on kukirokuk:a225674966381388_time_dependent_long_lat_tests into ** on openprocurement:a200894405797627_validation_for_deliveryLocation**.


Detroit - User Export

User Story:
As a jurisdictional admin OR an admin in partners portal,
I want to be able to export my jurisdiction’s user information in CSV format,
So that I can check on user engagement.
Acceptance criteria:
Given that I am an admin looking at the users page in partners portal:

When I am looking above the user table, then I should see a button indicating that I can export to CSV.
When I click the “export to csv” button, then my user information should download as a CSV.
When I click the “export to csv” button, then my user export will download in a timely fashion.
When I click the “export to csv” button, then my page shows an indication that the export is in progress.
When my user export downloads successfully, then I will see a message indicating the download was completed.
When there is an issue with my download, then I will see an error message on the page.

Given that I am an admin looking at my user export:

When I open the user information CSV, then I will see the columns listed in user details tab.
When I look through my user export files, then I will see data that reflects the users in the portal at time of download.
When I look through my user export files, then I will see user information for both partners and admins.

Implementation Notes:

Please refer to the design asset for export buttons and error/success messages.
Please review the Technical Design Doc and Product Brief

QA Notes:
Go to the detroit dev partners portal and click "export". https://detroit-partners-dev.netlify.app/users. CSV should be downloaded with appropriate columns

This looks good to me!  Thanks a ton, Morgan.


Adding class on filtered item w/o hiding them

Hello.
May be I missed somthing... but I didn't find a way to prevent hiding with the online "style='display:none'" on filtered item.
I think it could be usefull let visible filtered items and just identify this items with the mixitup.State (or class).
Thanks about your plugin, it's quite awesome

Hello,
I get this question a lot, and I usually tell people that you don't need MixItUp do this. MixItUp is a fairly complex library designed to smoothly animating elements between hidden and shown states. If you just want to add some class names to items matching your selector string, you can just do something like this:
Array.from(container.children).forEach(el => (el.matches('.my-category') ? el.classList.add('matching') : el.classList.remove('matching')));

I understand your point. But I'm using your plugin because it's a powerfull tool to filter and toogleproduct on my product list page. Callbacks are completly usefull too.
In order to made transformation like styling (w/o modify display of all item) it could be a good a functionnaly no?

That's good to hear. I appreciate there's a lot more functionality in MixItUp at this point than there was early on, so I'll definitely give it some thought.
I've been thinking about decoupling the filter/toggle/ui functionality from the DOM manipulation/animation functionality in a future refactor, so that would probably be the time to add this kind of thing.

@patrickkunka - So is there any way to access / target the elements which have been hidden after a filtering option is selected?
I have a unique case where I have limited control over the output HTML structure of the target items. In my case, for the elements which have the correct classes to use for filtering, they are each wrapped inside of a parent div. So while the filtering hides the contents, the parent div of each item is not hidden, so there is blank white space showing.
I can not figure out a workaround for this based on the documentation. Am I SOL here?
In my mind I was trying to figure out how to target the parents of each hidden element using onMixEnd callback, or something similar but I don't see a way to target them.

I personally would need to mark some way filtered elements. That's because I need to alternate-colour them, but without any class to use to get only visible items, I cannot maintain alternate colours upon filtering. Any idea?


Update txfunc tests data

This PR updates the data used for 3 of the tests in test_txfunc.py.  These updates are to be consistent with a new default start year.

Codecov Report

Merging #535 into master will increase coverage by 0.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #535      +/-   ##
==========================================
+ Coverage   76.44%   76.45%   +0.01%
==========================================
  Files          47       47
  Lines        6214     6213       -1
==========================================
  Hits         4750     4750
+ Misses       1464     1463       -1

Impacted Files
Coverage Δ

ogusa/tests/test_txfunc.py
66.66% <0%> (+0.56%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 955e694...96cadfc. Read the comment docs.


Domain Renewal email

Figma link : https://www.figma.com/file/S1UKYgWewNqNHZFAaBUilG/Starknet-ID-design?type=design&node-id=1565%3A660&mode=design&t=GGc5OHbzXM1MzXMK-1

Here is an example of an email page that a user receives when their domain is about to expire. A link represented by a button allows them to quickly renew their domain with a 20% discount.

By clicking on the 'Renew my domain' button, the user is redirected to the renewal page with various input fields. The discount is displayed next to the total due.

Ajouter un titre sur la page de paiement :


Issue with aggregation for high throughput volume metrics - resulting in incorrect aggregate value

Problem

We have being observing tremor events with incorrect count value after aggregation when throughput hits above 10k/10s

version='0.11.5'
Not seeing any bottleneck with resources - Memory, CPU
We have been observing the right number of events ingress to tremor and egress events from tremor aggregation but the aggregated value after reaching a throughput of 10k per 10sec resulting incorrect aggregated values
The data discrepancy is directly proportional to increasing in aggregation interval (from 10s , 1m and 1h)
Steps

Use the trickle script for 10s aggregation
Check the result with increase in throughput incremental way at onramp and observe the output aggregated value from stdout.
When you hit a measurement throughput of 10k, you will start seeing incorrect aggregate value

Trickle script being used for 10s interval aggregation

#!config metrics_interval_s = 300

use std::type;

use std::record;

define script time_to_ns
script
    use std::range;
    use std::math;

    # ---------- convert timestamp to nano second -----------
    for range::range(1,19) of
        case(i,e) =>
            match math::min(event.timestamp*10,9223372036854775806) of # max unix timestamp
                case<PHONE_NUMBER>854775806 => emit
                default => let event.timestamp = event.timestamp * 10
            end
    end;
    emit;
end;
create script time_to_ns;

define script round_up_timestamp
script
    # ---------- round up the timestamp to next 10th second ------
    let window_in_ns = 10 *<PHONE_NUMBER>;
    let time_mod = event.timestamp % window_in_ns;
    match time_mod of
        case 0 => emit
        default => let event.timestamp = event.timestamp - time_mod + window_in_ns
    end;
    # -------------------------------------------------------

    emit;
end;
create script round_up_timestamp;

define tumbling window ten_secs
with
    interval = core::datetime::with_seconds(10),
script
    event.timestamp
end;

create stream normalize;
create stream aggregate;

define generic::batch operator batch
with
count = 10000,
timeout = 5
end;
create operator batch;

select event from in into time_to_ns;
select event from time_to_ns into round_up_timestamp;

select {
"measurement": event.measurement,
"tags": event.tags,
"field": group[2],
"value": event.fields[group[2]],
"timestamp": event.timestamp,
}

from round_up_timestamp
group by set(event.measurement, event.tags, each(record::keys(event.fields)))
into aggregate
having type::is_number(event.value);

select
{
"measurement": event.measurement,
"tags": event.tags,
"stats": aggr::stats::dds(event.value, [ "0.9", "0.95", "0.99" ]),
"field": event.field,
"timestamp": aggr::win::last(event.timestamp), # we can't use min since it's a float
}
from aggregate[`ten_secs`]
group by set(event.measurement, event.tags, event.field)
into normalize;

select {
"measurement": event.measurement,
"tags": event.tags,
"fields": {
"count_#{event.field}": event.stats.count,
"min_#{event.field}": event.stats.min,
"max_#{event.field}": event.stats.max,
"mean_#{event.field}": event.stats.mean,
"sum_#{event.field}": event.stats.count*event.stats.mean,
"p90_#{event.field}": event.stats.percentiles["0.9"],
"p95_#{event.field}": event.stats.percentiles["0.95"],
"p99_#{event.field}": event.stats.percentiles["0.99"]
},
"timestamp": event.timestamp,
}

from normalize
where event.stats.count > 0
into batch;

select {
"measurement": event.measurement,
"tags": event.tags,
"fields": {
"count_#{event.field}": 0, # count is integer
"min_#{event.field}": 0.0,
"max_#{event.field}": 0.0,
"mean_#{event.field}": 0.0,
"sum_#{event.field}": 0.0,
"p90_#{event.field}": 0.0,
"p95_#{event.field}": 0.0,
"p99_#{event.field}": 0.0
},
"timestamp": event.timestamp,
}

from normalize
where event.stats.count == 0
into batch;

select event from batch into out;

Possible Solution(s)

Seems likely there is a bug in bucketing or some limitation with memory allotment during the aggregation process.
Notes
Output of rustup --version:
Output of rustup show:
Output of tremor --version:

One mitigation we can make here is to add metrics to make events dropped due to cardinality limits visible although its likely not an issue in this case. Thanks for reporting this issue @dullasuresh37 🚀  - we're looking into it!

Hi @dullasuresh37 there are three things you could try to help to narrow it down, ideally separately:

use a count instead of DDS for counting (this would rule out an issue with DDS counting)

select
{
"measurement": event.measurement,
"tags": event.tags,
"stats": aggr::stats::dds(event.value, [ "0.9", "0.95", "0.99" ]),
"count": aggr::stats::count(),
"sum":: aggr::stats::sum(event.value),
"field": event.field,
"timestamp": aggr::win::last(event.timestamp), # we can't use min since it's a float
}
from aggregate[`ten_secs`]
group by set(event.measurement, event.tags, event.field)
into normalize;

select {
"measurement": event.measurement,
"tags": event.tags,
"fields": {
"count_#{event.field}": event.count,
"min_#{event.field}": event.stats.min,
"max_#{event.field}": event.stats.max,
"mean_#{event.field}": event.stats.mean,
"sum_#{event.field}": event.sum,
"p90_#{event.field}": event.stats.percentiles["0.9"],
"p95_#{event.field}": event.stats.percentiles["0.95"],
"p99_#{event.field}": event.stats.percentiles["0.99"]
},
"timestamp": event.timestamp,
}

from normalize
where event.stats.count > 0
into batch;

expand the window dimension limit, as darach suggested you might create a cardinality that triggers the cardinality overflow protection you can do that by:

define tumbling window ten_secs
with
    interval = core::datetime::with_seconds(10),
    max_groups = 10000000, # this is usually a really bad idea!
script
    event.timestamp
end;

(the easiest one) I noticed that up to 5000 the count equals the messages for 7500 it is a 10x increase, this seems odd is that a typo or is something else going on? Is there an anomaly in the data? Is it the same data (just replayed faster) or different datasets being used?

Yes - setting a low cardinality limit is typically not advisable but it can be useful to explore behaviour under highly variant streaming conditions. So max_groups should be used with care and always documented if ever used ( your future self will thank you later ). We'll see if something else comes to mind as we go through the case.

Using count,min,max,sum functions instead DDS fixed the issue

select
{
"measurement": event.measurement,
"tags": event.tags,
"stats": aggr::stats::dds(event.value, [ "0.9", "0.95", "0.99" ]),
"count": aggr::stats::count(),
"sum": aggr::stats::sum(event.value),
"min": aggr::stats::min(event.value),
"max": aggr::stats::max(event.value),
"mean": aggr::stats::mean(event.value),
"field": event.field,
"timestamp": aggr::win::last(event.timestamp), # we can't use min since it's a float
}
from aggregate[`ten_secs`]
group by set(event.measurement, event.tags, event.field)
into normalize;

select {
"measurement": event.measurement,
"tags": event.tags,
"fields": {
"count_#{event.field}": event.count,
"min_#{event.field}": event.min,
"max_#{event.field}": event.max,
"mean_#{event.field}": event.mean,
"sum_#{event.field}": event.sum,
"p90_#{event.field}": event.stats.percentiles["0.9"],
"p95_#{event.field}": event.stats.percentiles["0.95"],
"p99_#{event.field}": event.stats.percentiles["0.99"]
},
"timestamp": event.timestamp,
}

from normalize
where event.stats.count > 0
into batch;

Yes, you're right
that's a typo - actual and expected is 7.5K throughput /10 sec which is good

Seems like the bug is with DDS

Perfect! Thanks! And yes :) this narrows it down to DDS, which is a fairly small surface area compared to the rest, we'll hopefully be able to track this down quickly with this info!


51/process messages from client

#51

Really beautiful.
But what about tests?

There are two places that have not been tested:

handle function
new and from_str functions from /api/*

About handle
I don't know how to test it using unit tests (I suppose there is a way by imitating request from server to websocket)
But most importantly that all this staff with switch function will be removed very soon and we will have game instance shared between all connections and this instance will process commands. Thus tests which we'll write now becomes invalid
About other little functions
There is .unwrap almost everywhere and this is something we don't want in production. I think we have to work on error handling and after that write tests for this folks


8/20 Hackey Hour

Last time we met, we decided that if we had things to talk about, why not try to make this happen more frequently than not.
Last time we all tried out a code review type approach on understanding @monprins genewordsearch package. We even submitted multiple pull requests with either suggestions or bug fixes!
edit: Due to the BICB Industry Sympoisum last week, the 8/20 hackey hour was postponed.
Next hackey hour (8/27) will be focused on a web based tools designed by Justin, in the Myers lab. I think he is working on getting the code up on github, so a link to the repo will follow.
Happy Hacking!
Rob

FYI, the next hackey hour is 8/27!

How do we subscribe to email alerts for the group? Do we star this repository?

I think it you just hit the eyeball and watch to Repo, you should get email when people comment and such.

Perfect, that appears to have done the trick.


Additional JS solutions

I also fixed an existing file from .txt -> .java

I had to rebase after changing my commit typo which was a big problem because I had pulled in your latest changes to keep my repo up to date. Got pretty messy but I think I sorted it all out.