[CI][Docker] xgboost 1.0.1 causes segfault on test_autotvm_xgboost_model.py

With the release of XGBoost 1.0.x (i.e xgboost-1.0.1-py3-none-manylinux1_x86_64.whl), it seems that installing TVM from scratch (rebuilding Docker containers) makes tests/python/unittest/test_autotvm_xgboost_model.py to fail with a segfault.
Investigating it a bit further, if I manually revert it to xgboost-0.90 it works fine. Using xgboost-1.0.1, this is the message I see:
tests/python/unittest/test_autotvm_xgboost_model.py::test_fit Fatal Python error: Segmentation fault

Thread 0x00007f4f98de4700 (most recent call first):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 379 in _recv
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407 in _recv_bytes
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250 in recv
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f4f905e3700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/queue.py", line 164 in get
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f4f8fde2700 (most recent call first):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f4fb514c700 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/xgboost/core.py", line 1248 in update
  File "/usr/local/lib/python3.6/dist-packages/xgboost/training.py", line 74 in _train_internal
  File "/usr/local/lib/python3.6/dist-packages/xgboost/training.py", line 209 in train
  File "/workspace/python/tvm/autotvm/tuner/xgboost_cost_model.py", line 272 in fit_log
  File "/workspace/tests/python/unittest/test_autotvm_xgboost_model.py", line 35 in test_fit
  File "/usr/local/lib/python3.6/dist-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/python.py", line 1445 in runtest
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 210 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 237 in from_call
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 185 in call_and_report
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/usr/local/lib/python3.6/dist-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 247 in _main
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 197 in wrap_session
  File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.6/dist-packages/_pytest/config/__init__.py", line 93 in main
  File "/usr/local/lib/python3.6/dist-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib/python3.6/runpy.py", line 85 in _run_code
  File "/usr/lib/python3.6/runpy.py", line 193 in _run_module_as_main
./tests/scripts/task_python_unittest.sh: line 27: 24582 Segmentation fault      (core dumped) TVM_FFI=ctypes python3 -m pytest -v tests/python/unittest

@tqchen, I didn't see any PR or discussion about it, but are you aware about any ongoing initiative to move TVM to XGBoost 1.0.x, or shall we pin xgboost to be 0.90, to prevent the error to happen? (note: I'm happy to send a patch to pin the version)

@leandron can you create a minimum reproducible example? e.g. pickle the data that causes segfault in XGBoost. Then we can start to bring attention of the XGBoost dev community. In the meanwhile, we can pin xgboost to 0.9

cc @merrymercy @hcho3

It would be nice if we can get a reproducible example. We are currently working on the patch release 1.0.2 and I want to get a patch to fix this issue.

@leandron @hcho3 please followup :)

@leandron can you please comment about the current state?

@leandron Thanks for pointing out which part of TVM test is failing. Not sure if running in debug mode would help, since XGBoost is crashing with segfault here. I will take a look some time this week.

I compiled TVM from source and tried running the test tests/python/unittest/test_autotvm_xgboost_model.py::test_fit and I cannot reproduce the issue. Do I need a specific Docker container to reproduce the problem?

I also tried building ci-cpu Docker image from scratch and running the unit test myself. The test runs without crashing.

We applied a workaround, pinning the xgboost version to be 0.90. Which XGBoost version you see in the image you created from scratch?

@leandron I checked out commit 8502691b5b7ca152da9eb626529070db53d479c8 so that XGBoost 1.0.x is used.

@leandron can you also provide a bit more details
e.g. does directly run tests/python/unittest/test_autotvm_xgboost_model.py fails or do we need to run the entire unittest. It would also be nice if you can send a CI binary hashtag(perhaps in docker hub) to confirm the problematic issue.

I see this reliably with a virtualenv on bionic on AWS.
environment:
ami:<PHONE_NUMBER>77/ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408
python version
Python 3.7.5

installed python packages:
antlr4-python3-runtime==4.8
Cython==0.29.16
decorator==4.4.2
psutil==5.7.0
pylint==2.4.4
  - astroid [required: >=2.3.0,<2.4, installed: 2.3.3]
    - lazy-object-proxy [required: ==1.4.*, installed: 1.4.3]
    - six [required: ~=1.12, installed: 1.14.0]
    - typed-ast [required: >=1.4.0,<1.5, installed: 1.4.1]
    - wrapt [required: ==1.11.*, installed: 1.11.2]
  - isort [required: >=4.2.5,<5, installed: 4.3.21]
  - mccabe [required: >=0.6,<0.7, installed: 0.6.1]
pytest==5.4.1
  - attrs [required: >=17.4.0, installed: 19.3.0]
  - importlib-metadata [required: >=0.12, installed: 1.6.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - more-itertools [required: >=4.0.0, installed: 8.2.0]
  - packaging [required: Any, installed: 20.3]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.14.0]
  - pluggy [required: >=0.12,<1.0, installed: 0.13.1]
    - importlib-metadata [required: >=0.12, installed: 1.6.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - py [required: >=1.5.0, installed: 1.8.1]
  - wcwidth [required: Any, installed: 0.1.9]
tornado==6.0.4
xgboost==1.0.2
  - numpy [required: Any, installed: 1.18.3]
  - scipy [required: Any, installed: 1.4.1]
    - numpy [required: >=1.13.3, installed: 1.18.3]

backtrace:
~/ws/tvm$ gdb python
GNU gdb (Ubuntu 8.1-0ubuntu3.2) <IP_ADDRESS>80409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run tests/python/unittest/test_autotvm_xgboost_model.py
Starting program: /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/bin/python tests/python/unittest/test_autotvm_xgboost_model.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4104700 (LWP 11905)]
[New Thread 0x7ffff3903700 (LWP 11906)]
[New Thread 0x7fffef102700 (LWP 11907)]
[New Thread 0x7fffec901700 (LWP 11908)]
[New Thread 0x7fffea100700 (LWP 11909)]
[New Thread 0x7fffe78ff700 (LWP 11910)]
[New Thread 0x7fffe50fe700 (LWP 11911)]
[New Thread 0x7fffe28fd700 (LWP 11912)]
[New Thread 0x7fffe00fc700 (LWP 11913)]
[New Thread 0x7fffdd8fb700 (LWP 11914)]
[New Thread 0x7fffdb0fa700 (LWP 11915)]
[New Thread 0x7fffd88f9700 (LWP 11916)]
[New Thread 0x7fffd60f8700 (LWP 11917)]
[New Thread 0x7fffd38f7700 (LWP 11918)]
[New Thread 0x7fffd10f6700 (LWP 11919)]
[Thread 0x7fffe50fe700 (LWP 11911) exited]
[Thread 0x7fffd88f9700 (LWP 11916) exited]
[Thread 0x7fffd10f6700 (LWP 11919) exited]
[Thread 0x7fffd60f8700 (LWP 11917) exited]
[Thread 0x7fffdb0fa700 (LWP 11915) exited]
[Thread 0x7fffdd8fb700 (LWP 11914) exited]
[Thread 0x7fffe00fc700 (LWP 11913) exited]
[Thread 0x7fffe28fd700 (LWP 11912) exited]
[Thread 0x7fffe78ff700 (LWP 11910) exited]
[Thread 0x7fffea100700 (LWP 11909) exited]
[Thread 0x7fffec901700 (LWP 11908) exited]
[Thread 0x7fffef102700 (LWP 11907) exited]
[Thread 0x7ffff3903700 (LWP 11906) exited]
[Thread 0x7ffff4104700 (LWP 11905) exited]
[Thread 0x7fffd38f7700 (LWP 11918) exited]
[New Thread 0x7fffd10f6700 (LWP 11936)]
[New Thread 0x7fffd38f7700 (LWP 11937)]
[New Thread 0x7fffd60f8700 (LWP 11938)]
[Thread 0x7fffd10f6700 (LWP 11936) exited]
[Thread 0x7fffd60f8700 (LWP 11938) exited]
[Thread 0x7fffd38f7700 (LWP 11937) exited]
[New Thread 0x7fffd38f7700 (LWP 11955)]
[New Thread 0x7fffd60f8700 (LWP 11956)]
[New Thread 0x7fffd10f6700 (LWP 11957)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > >(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
(gdb) bt
#0  0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > >(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#1  0x00007fffb5b970b7 in xgboost::GenericParameter::ConfigureGpuId(bool) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#2  0x00007fffb5bb2cd1 in xgboost::LearnerImpl::Configure() ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#3  0x00007fffb5bad69d in xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#4  0x00007fffb5ab0639 in XGBoosterUpdateOneIter ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#5  0x00007fffce06adae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#6  0x00007fffce06a71f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#7  0x00007fffce27e944 in _ctypes_callproc () from /usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#8  0x00007fffce27efb3 in ?? () from /usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#9  0x00000000005d96db in _PyObject_FastCallKeywords ()
#10 0x000000000054aa51 in ?? ()
#11 0x0000000000551c08 in _PyEval_EvalFrameDefault ()
#12 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#13 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#14 0x000000000054de9e in _PyEval_EvalFrameDefault ()
#15 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#16 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#17 0x000000000054a880 in ?? ()
#18 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#19 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#20 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#21 0x000000000054a880 in ?? ()
#22 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#23 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#24 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#25 0x000000000054a880 in ?? ()
#26 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#27 0x00000000005d88dc in _PyFunction_FastCallKeywords ()
#28 0x000000000054dd08 in _PyEval_EvalFrameDefault ()
#29 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#30 0x000000000054d803 in PyEval_EvalCode ()
#31 0x00000000006308e2 in ?? ()
#32 0x0000000000630997 in PyRun_FileExFlags ()
#33 0x000000000063160f in PyRun_SimpleFileExFlags ()
#34 0x000000000065450e in ?? ()
#35 0x000000000065486e in _Py_UnixMain ()
#36 0x00007ffff7a05b97 in __libc_start_main (main=0x4b84d0 <main>, argc=2, argv=0x7fffffffdfc8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffdfb8) at ../csu/libc-start.c:310
#37 0x00000000005df80a in _start ()
(gdb)

pytest log:
~/ws/tvm$ TVM_FFI=ctypes python3 -m pytest -s -v tests/python/unittest -k 'test_autotvm_xgboost_model'
================================================================ test session starts =================================================================
platform linux -- Python 3.7.5, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/bin/python3
cachedir: .pytest_cache
rootdir: /home/ubuntu/ws/tvm
collecting 88 items                                                                                Testing using contexts: [cpu(0)]
collected 562 items / 560 deselected / 2 selected

tests/python/unittest/test_autotvm_xgboost_model.py::test_fit Fatal Python error: Segmentation fault

Thread 0x00007f6c59dd9700 (most recent call first):
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 379 in _recv
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 407 in _recv_bytes
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 250 in recv
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 470 in _handle_results
  File "/usr/lib/python3.7/threading.py", line 870 in run
  File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f6c5cddb700 (most recent call first):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 422 in _handle_tasks
  File "/usr/lib/python3.7/threading.py", line 870 in run
  File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f6c5c5da700 (most recent call first):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 413 in _handle_workers
  File "/usr/lib/python3.7/threading.py", line 870 in run
  File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
  File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f6c7f460740 (most recent call first):
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/core.py", line 1249 in update
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/training.py", line 74 in _train_internal
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/training.py", line 209 in train
  File "/home/ubuntu/ws/tvm/python/tvm/autotvm/tuner/xgboost_cost_model.py", line 272 in fit_log
  File "/home/ubuntu/ws/tvm/tests/python/unittest/test_autotvm_xgboost_model.py", line 35 in test_fit
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/python.py", line 184 in pytest_pyfunc_call
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/python.py", line 1479 in runtest
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 135 in pytest_runtest_call
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 100 in runtestprotocol
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib/python3.7/runpy.py", line 85 in _run_code
  File "/usr/lib/python3.7/runpy.py", line 193 in _run_module_as_main
Segmentation fault (core dumped)

Let me try again with TVM_FFI=ctypes environment variable set. What does this do?

The trace might offer some insights, @hcho3 , couldit caused by ConfigureGpuId?
0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > >(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
(gdb) bt
#0  0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > >(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
   from /home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#1  0x00007fffb5b970b7 in xgboost::GenericParameter::ConfigureGpuId(bool) ()

Is it possible that somehow XGBoost linked a wrong dmlc static library?

i don't know unless we go and dig deeper, but if the bug is reproducible, then it should not be hard to find the cause

tried to reproduce with xgboost built from source (at HEAD/e4f5b6c8 and v1.0.2), no luck (test_autotvm_xgboost_model passes). if I reinstall the pip package (1.0.2), I can get it to reproduce again.
I built xgboost with this config:
cmake -GNinja .. -DUSE_CUDA=ON -DUSE_NCCL=ON -DOPEN_MP:BOOL=ON
any other suggestions to get it to build or install like the pypi package? might it be related to building the package on centos?

@areusch Are you still running bionic when installing from pip?

yes

There is something I wanted to point out, which is an insight after @areusch's comment (thanks for that!).
The VM I'm running this test, does not have a GPU. However, the same test used to pass on this very same machine, with xgboost<1. Is that the case for you @areusch?
@hcho3 do you think this is something caused by a change in behaviour after xgboost>=1.0 ?

no GPU on my instance (it is c5.4xlarge)

I can reproduce it on bionic.  Here is what I have found so far:

It's only reproducible with bionic, I have both 19.04 (EOL) and 19.10 tested.
It's only reproducible with the tvm test, so normal training with XGBoost doesn't have the issue.
It's only reproducible with pip installation.  Building from source works fine, which is annoying.
From valgrind, it has 1 byte invalid read, which is not presented if built from source.
Valgrind also reported multiple invalid reads from tvm. (built from source without any cmake flag).

@hcho3  It would be of great help if I can obtain a debug build or RelaseWithDebugInfo build.

I am still unable to reproduce it on my Bionic machine. @areusch Can you share the content of your config.cmake?
@trivialfis I'll try to build a wheel using CentOS Docker image.

I am still unable to reproduce it on my Bionic machine. @areusch Can you share the content of your config.cmake?

@hcho3 You have to install the package on pip to reproduce it.  Building from source works fine.

@hcho3 Here is config.cmake from tvm.

@trivialfis I rebuilt the 1.0.2 wheel with debug symbol enabled: https://drive.google.com/file/d/1cELaBb_rnmb9y8irSwEBQ5AIvaDQgDcF/view?usp=sharing. Feel free try it.
@areusch Since you are using AWS, can you make AMI from your EC2 instance and share it with me?

@hcho3 I found the cause:
The dmlc error is thrown from libtvm.so instead of libxgboost.so, which can not be caught by XGBoost.  I believe there's some mixed up in the built binary.
#0  0x00007fffc821853f in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x00007fffc821a098 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x00007ffff7b15168 in __GI___backtrace (array=<optimized out>, size=<optimized out>) at ../sysdeps/x86_64/backtrace.c:111
#3  0x00007fffc8b324fd in dmlc::StackTrace[abi:cxx11](unsigned long, unsigned long) () from /home/fis/Workspace/XGBoost/incubator-tvm/build/libtvm.so
#4  0x00007fffc8b32dbc in dmlc::LogMessageFatal::~LogMessageFatal() () from /home/fis/Workspace/XGBoost/incubator-tvm/build/libtvm.so
#5  0x00007fffb971b4c5 in dh::ThrowOnCudaError (code=cudaErrorInsufficientDriver, file=0x7fffb9a82278 "/workspace/src/common/common.cu", line=14) at /workspace/src/c_api/../data/../common/common.h:41
#6  0x00007fffb972ac75 in xgboost::common::AllVisibleGPUs () at /workspace/src/common/common.cu:14

But I don't understand how it happens.

Somehow the dmlc::LogMessageFatal constructor is a plt function, but the one in tvm is not plt, and calling it in XGBoost got resolved into the one in tvm.
Thread 1 "python" hit Breakpoint 1, 0x00007fffb952ac60 in dmlc::LogMessageFatal::LogMessageFatal(char const*, int)@plt ()
   from /home/fis/Workspace/XGBoost/lib/python3.6/site-packages/xgboost/./lib/libxgboost.so
(gdb) bt
#0  0x00007fffb952ac60 in dmlc::LogMessageFatal::LogMessageFatal(char const*, int)@plt ()
   from /home/fis/Workspace/XGBoost/lib/python3.6/site-packages/xgboost/./lib/libxgboost.so
#1  0x00007fffb971b49c in dh::ThrowOnCudaError (code=cudaErrorInsufficientDriver, file=0x7fffb9a82278 "/workspace/src/common/common.cu", line=14)
    at /workspace/src/c_api/../data/../common/common.h:41
#2  0x00007fffb972ac75 in xgboost::common::AllVisibleGPUs () at /workspace/src/common/common.cu:14
#3  0x00007fffb961496e in xgboost::gbm::GBTree::Configure (this=0xc41b70, cfg=std::vector of length 12, capacity 12 = {...})
    at /workspace/src/gbm/gbtree.cc:54
#4  0x00007fffb9643985 in xgboost::LearnerImpl::ConfigureGBM (this=0x1397db0, old=..., args=std::vector of length 12, capacity 12 = {...})
    at /workspace/src/learner.cc:925
#5  0x00007fffb963aeaf in xgboost::LearnerImpl::Configure (this=0x1397db0) at /workspace/src/learner.cc:252
#6  0x00007fffb9641649 in xgboost::LearnerImpl::UpdateOneIter (this=0x1397db0, iter=0, train=0x12282e0) at /workspace/src/learner.cc:722
#7  0x00007fffb9561414 in XGBoosterUpdateOneIter (handle=0x1397db0, iter=0, dtrain=0x13856e0) at /workspace/src/c_api/c_api.cc:501
#8  0x00007ffff5092dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#9  0x00007ffff509271f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#10 0x00007ffff52a65c4 in _ctypes_callproc () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#11 0x00007ffff52a6c33 in ?? () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so

Somehow the dmlc::LogMessageFatal constructor is a plt function in XGBoost, but the one in tvm is not plt, and calling it in XGBoost got resolved into the one in tvm.

Looks promising!
Naive question: based on what you see here, is it possible to correlate with the original stack trace/error described in the first report? https://github.com/apache/incubator-tvm/issues/4953#issue-572158142

@leandron I believe it's the root cause of this issue.  I just don't know how to fix it.  Maybe it's a bug in system linker or compiler used to build to pip package?  Or a convention in hairy C++ ABI I'm not aware of?  Or there's a wrong CMake flag that makes the function plt somehow while it's not supposed to?

@hcho3
Could you please create another build with position independent code of dmlc-core disabled?
set_target_properties(dmlc PROPERTIES
  CXX_STANDARD 11
  CXX_STANDARD_REQUIRED ON
  POSITION_INDEPENDENT_CODE OFF)  # OFF -> ON
list(APPEND LINKED_LIBRARIES_PRIVATE dmlc)

I wonder if that was due to inconsistency between dmlc-core of tvm and xgb. https://github.com/apache/incubator-tvm/pull/5401 updated the logging to latest, please check again.

@trivialfis Here it is: https://drive.google.com/file/d/13WZRRaUPKil4rwH2avgUix_xO5LpIYs5/view?usp=sharing

@hcho3 do you still need an AMI from me? I think you can repro by using the AMI I mentioned earlier
ami:<PHONE_NUMBER>77/ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408
it should be enough to just try and run the tvm test using a pip installed xgboost. I can build you another if it would help.

@hcho3 One last request, otherwise I'm running out of ideas.
Patch both xgboot and rabit's C API macro:
For xgboost:
diff --git a/include/xgboost/c_api.h b/include/xgboost/c_api.h
index f9c0a0ff..baaaeb43 100644
--- a/include/xgboost/c_api.h
+++ b/include/xgboost/c_api.h
@@ -20,7 +20,7 @@
 #if defined(_MSC_VER) || defined(_WIN32)
 #define XGB_DLL XGB_EXTERN_C __declspec(dllexport)
 #else
-#define XGB_DLL XGB_EXTERN_C
+#define XGB_DLL XGB_EXTERN_C __attribute__ ((visibility ("default")))
 #endif  // defined(_MSC_VER) || defined(_WIN32)

 // manually define unsigned long

For rabit:
diff --git a/include/rabit/c_api.h b/include/rabit/c_api.h
index 0a96ef7..47c5735 100644
--- a/include/rabit/c_api.h
+++ b/include/rabit/c_api.h
@@ -18,7 +18,7 @@
 #if defined(_MSC_VER) || defined(_WIN32)
 #define RABIT_DLL RABIT_EXTERN_C __declspec(dllexport)
 #else
-#define RABIT_DLL RABIT_EXTERN_C
+#define RABIT_DLL RABIT_EXTERN_C __attribute__ ((visibility ("default")))
 #endif  // defined(_MSC_VER) || defined(_WIN32)

 /*! \brief rabit unsigned long type */

Build XGBoost with following flags appended:
-DCMAKE_CXX_FLAGS='-fvisibility=hidden' -DCMAKE_C_FLAGS='-fvisibility=hidden'

@trivialfis Did you update TVM to latest?

@hcho3 I tried master branch and the commit before:

I wonder if that was due to inconsistency between dmlc-core of tvm and xgb. #5401 updated the logging to latest, please check again.

@trivialfis And you ran git submodule update --init --recursive?

@hcho3 Yes.
fis@fis-Standard-PC-Q35-ICH9-2009:~/Workspace/XGBoost/incubator-tvm$ git status
HEAD detached at 56941fb9d
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	.gdb_history

nothing added to commit but untracked files present (use "git add" to track)

@areusch Can you try out the latest TVM master on your end? I'm still having trouble reproducing.

Finally, I reproduced it. Yes!

@hcho3 Could you try applying the patches I posted above and use the corresponding cmake flags?

@trivialfis I applied your patch and changed CMake flags. And now the unit test does not crash any more.
You should try it too. Get the wheel at https://xgboost-wheels.s3-us-west-2.amazonaws.com/xgboost-1.0.2-py3-none-manylinux1_x86_64.whl.

@hcho3 Yup.  It works fine on my machine too

@hcho3 tested with your new wheel on my aws instance and the test now passes!

@trivialfis Can you elaborate what your patch does? Does it hide certain symbols?

It hides all the symbols, except for C APIs.  So if anyone's using C++ header, it might generate a lots of linker errors.

@trivialfis I think we can hide all C++ symbols when building Python wheels. WDYT?

@hcho3 Yup.  Good idea.  We can make it a CMake option:
https://stackoverflow.com/questions/17080869/what-is-the-cmake-equivalent-to-gcc-fvisibility-hidden-when-controlling-the-e

@trivialfis Let me file a pull request. We'll include the fix as part of the upcoming 1.1.0 release.

We need to be careful about this.  Rabit and dmlc core are independently built, I'm not sure what will happen if they throws an error, as hiding symbols means exception can not be propagated out.

Not entirely sure in the context of static linking.

Got it. How about compiling the wheel using latest Ubuntu (not CentOS) and put it in a S3 bucket? The TVM CI can pull from this bucket instead of PyPI.
Current build environment for the Pip wheel is quite old: CentOS 6 + devtoolset-4 (GCC 5.x). When we drop CUDA 9.0, we can upgrade to CentOS 6 + devtoolset-6 (GCC 7.x). Upgrade may fix the issue.

Before upgrading libc dependency, we can try to add a test that forces rabit to throw an error, see if it crashes XGBoost with segfault.  An uneven all reduce can do.
Or a test with dmlc core, a file nonexist error seems to be simple.

Just make sure error is not thrown in header.

I believe if it works it will be a net gain for XGBoost, hiding symbols is a good practice for shared libraries.

With https://github.com/dmlc/xgboost/pull/5590, I can now run tests/python/unittest/test_autotvm_xgboost_model.py::test_fit without crashing.

@leandron @areusch @tqchen I've put up RC1 for the upcoming XGBoost 1.1.0 release. Feel free to try it:
python3 -m pip install xgboost==1.1.0rc1

The unit test should not crash.

Thanks @hcho3 @areusch @leandron @trivialfis for resolving this problem.


docker-compose PHP error

Hello there, I'm trying to set up this app in my local machine and got an error with following message
"Attempted to load class "MakerBundle" from namespace "Symfony\Bundle\MakerBundle". Did you forget a "use" statement for another namespace?"
How can I solve this?

Thank you.

Hi @Ahmard,
I think you are running the application in DEV-Mode.
MakerBundle should not be loaded in prod.

Okay, I will try running it in prod mode and see.
Thanks.

I had the same issue, turns out the normal .env file was overriding the .env.docker file


Use TUF mirror to distribute long lived keys

Description
I would like to use a TUF mirror to distribute the (long-lived) public signing key used in the cosign verify* commands, and have cosign use the public key directly from the TUF root. I imagine some sort of syntax like this:
cosign verify --key <tuf-root-target>

Ideally, there is an option to provide multiple public keys in the TUF root. This would more easily allow a standard key rotation procedure. cosign would then try each key.
Considerations:

Should the public key be available on its own TUF root?

I'm not sure this should live in Cosign, though I'm happy to be convinced otherwise. I think Cosign, and other sigstore clients, should simply take in a verification key and it's up to the verifier to load it from a secured source. https://github.com/sigstore/cosign/issues/2908 is a similar issue.

Closing out, this is not something on our roadmap, but feel free to propose a change if it's something you need.


Add profiling example

import sciris as sc
import starsim as ss

pars = dict(
    start = '2000-01-01',
    stop = '2020-01-01',
    diseases = 'sis',
    networks = 'random'
)

# run sim
sim = ss.Sim(pars)
sim.run()

# check how long each step took
sim.loop.plot_cpu()

# profile slow step
s2 = ss.Sim(pars)
sc.profile(s2.run, follow=ss.SIS.step)

Also tidy up plotting if there are lots of labels, and fix the labeling of sim.loop.cpu_df


按照步骤，环境和组件都装成功了，但是启动显示报错，找不到mmcv?

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
Traceback (most recent call last):
File "D:\AI\ComfyUI\nodes.py", line 1879, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\AI\ComfyUI\custom_nodes\ComfyUI-MuseTalk_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS
File "D:\AI\ComfyUI\custom_nodes\ComfyUI-MuseTalk\nodes.py", line 26, in
from musetalk.utils.preprocessing import get_landmark_and_bbox,read_imgs,coord_placeholder
File "D:\AI\ComfyUI/custom_nodes/ComfyUI-MuseTalk\musetalk\utils\preprocessing.py", line 10, in
from mmpose.apis import inference_topdown, init_model
File "D:\AI\ComfyUI.ext\Lib\site-packages\mmpose_init_.py", line 2, in
import mmcv
ModuleNotFoundError: No module named 'mmcv'
Cannot import D:\AI\ComfyUI\custom_nodes\ComfyUI-MuseTalk module for custom nodes: No module named 'mmcv'

Did you have installed mmcv successfully?

sane here,  tried manually pip install mmcv, but build wheel error


can't get contract to appear on conductor

Stephanie accidentally decided to remove a contract from conductor.  When I go to scout and find the contract, the check mark is still checked for conductor though.  I need that contract to go back in conductor but can't get it to appear back.
spec 7096.
It seems like the 'visible on conductor' box in Admin:Scout means nothing and is controlling nothing.

Which do you want? Do you want to undo the removal and have the child contract back into the "Department Review Sent?" If so, you can go here and un-archive that child contract.
If you want, I can also delete that contract and you'll be able to start over from the beginning.

I've made some changes to the admin to make this a bit easier to deal with -- I'll push them up and merge them. You'll need to use the Admin/Conductor/Contract bit to get it to work. Here's what it will look like:


Define support level for pkg-config based builds.

The current CMake configuration supports using pkg-config to discover the flags for our dependencies.  Basically the CMake file calls pkg-config(1) to discover what compiler and linker flags are needed to find libcurl, or OpenSSL, or protobuf.
In some Linux distribution it seems like this is the preferred way to configure dependencies (e.g. Gentoo packages google-cloud-cpp using pkg-config(1).
I think we have basically three options:

Drop the support for pkg-config(1) completely, we remove the CMake code that uses pkg-config(1) to discover the dependencies.

Support pkg-config(1) as "contributed", meaning we accept patches to improve it, but do not create a build that runs it or fix the configuration ourselves (this is basically the current support).

Support pkg-config(1) for "realsies", meaning we create at least one build with it, and keep it working.

/cc: @devjgm @frankyn @houglum
Please discuss below.  My opinion is we should support this for realsies.

@remyabel I effectively implemented this suggestion:

Personally, I think that you should provide your own FindGrpc module, since CMake doesn't seem to have one yet.

I think we can remove pkg-config as an option.

Yup, thank you for your hard work. We should send them a patch.


Authentication logs

We'd like to access our team's Slack authentication logs using the Python SDK.
The Slack SDK version
I am using the latest Python SDK
Python runtime version
Python 3.10.9
OS info
ProductName:		macOS
ProductVersion:		13.3.1
ProductVersionExtra:	(a)
BuildVersion:		22E772610a
Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:58 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6020
Steps to reproduce:
I'd like to access the authentication logs from a start to an end date
Expected result:
Once I have access to the logs, I will work on them
Actual result:
I started with using this link.
https://slack.dev/python-slack-sdk/audit-logs/index.html
Requirements
I agree with the rules.

Hey @arijit32 👋  Accessing authentication logs between a start and end date can be done by adding filters to the logs method of the audit log client. A list of all available filters can be found here, so to filter within a specified date range, the following snippet might be useful:
import os
from slack_sdk.audit_logs import AuditLogsClient

client = AuditLogsClient(token=os.environ["SLACK_ORG_ADMIN_USER_TOKEN"])

api_response = client.logs(oldest=1645568542, newest=1684433923)
print(api_response.typed_body)

More information on the values of api_response.typed_body can be found here. Hopefully this helps with accessing these logs, but feel free to follow up with more questions if you have any!


Focus on highlighted list item

Hi, I'm trying to focus on the highlighted list item when pressing the down or up arrow key. I use to have a long list of users to mention and I want to auto scroll the list down next to the highlighted item. I've tried to add this quill keyboard handler for down arrow but it didn't work:
down: {
        key: 40,
        handler: function(range: any, context: any) {
          const mentionListItem: any = document.querySelector('.ql-mention-list-item.selected');
          const focusList: any = findDOMNode(mentionListItem).nextSibling;
          // TODO: know how to focus list item
          mentionListItem.focus();
          return true;
        }
      }

I guess because we already return false to the downHandler function of quill-mention module? Is there a way we focus the item to scroll it down to the remaining items of the list? Like in this example .

You need to set the correct scrollTop value on the ql-mention-list-container.

Thank you for the response @MadSpindel but when should I set this property?

I think the best way is to make a pull request. So probably in highlightItem() you need to check if its scrollable and then set the correct scrollTop value on this.mentionContainer

Okay, I'll try to push a PR today. Thanks!

@MadSpindel I already have updates in my local but I can't seem to push a branch in this repo. I'm getting a permission denied.

@abelski21 You need to fork the branch first. Then you need to make a pull request from your branch to this master branch.

Got it. Thanks! Here's my pull request #8


Sticky Contexts

Adds sticky contexts feature.
Example:

new face StickyContexts, defaults to default+Fbuid.
new option lsp_sticky_contexts_skip, defaults to 0.
new commands lsp-sticky-contexts-(enable|disable) similar to existing commands.

UPDATE:

added additional option lsp_sticky_contexts_max, defaults to 4.

The README for this feature also should be updated.

I've been using this for a few days and it's pretty great! My only gripes are the lack of highlighting (impossible to fix?) and the fact that it's not synchronous with the viewport. Neither are show-stoppers though so I will continue to use this in my daily work. Thank you!

@raiguard Thanks for the feedback!
Agree it would be nice to have a syntax highlighting, and I am not really sure to achieve it, unfortunately, maybe @krobelus has some ideas.

and the fact that it's not synchronous with the viewport

Could you elaborate? I didn't fully get it.

The ranges update on idle, so the context lines don't always stay exactly at the top of the viewport - they jump around, especially if you're moving very quickly.

I did not look at the code yet but tried it briefly.
Apparently this uses the full symbol range, so comments are included, like
/// Apply text edits to the file pointed by uri either by asking Kakoune to modify corresponding

probably selectionRange gives better results. Or only show the function name? That would be more robust (multi-line functions).

Agree it would be nice to have a syntax highlighting, and I am not really sure to achieve it

Highlighting information is not really exposed I don't think. (On a related note I'd probably not make it bold by default but Default or dim)
But maybe Kakoune could have a custom always-on-top highlighter to "transplant" a given line to the top of the viewport.
That would solve the glitches when scrolling mentioned above.
When pressing <c-d> repeatedly to scroll, the cursor will be on the first line, hiding the sticky context, which is unfortunate.
Not sure how to fix this even with a custom highlighter. Would be great to fix but not a blocker.
Nesting doesn't seem to work properly. If I move around in apply_text_edits_to_file_impl, it keeps displaying the parent apply_text_edits_to_file.
I remember it sometimes displayed two lines, which was a bit too much I think.

Apparently this uses the full symbol range, so comments are included, like

Indeed, didn't notice that. Using selection_range now for that.

Or only show the function name?

Yeah, that would work, but it's not always a function, sometimes it's a package/module/class/trait. Didn't try, but feels like keywords like fn, class, package, etc. are important to show. It'd would be difficult to do that for each language.

On a related note I'd probably not make it bold by default but Default or dim

Removed the boldness and switched to @Default.

But maybe Kakoune could have a custom always-on-top highlighter to "transplant" a given line to the top of the viewport.
That would solve the glitches when scrolling mentioned above.

Yeah, it's not something we can do in this PR, I think.

Nesting doesn't seem to work properly. If I move around in apply_text_edits_to_file_impl, it keeps displaying the parent apply_text_edits_to_file.
I remember it sometimes displayed two lines, which was a bit too much I think.

Yes, actually, it was intentional - to stack contexts on each other. E.g. if cursor in the nested function of another function, and both signatures are outside of the window position, then those 2 signature lines will be rendered.
I've added another option called lsp_sticky_contexts_max to limit max amount of sticky contexts. Combined with lsp_sticky_contexts_skip it makes it flexible to choose what do you want to see.

When pressing  repeatedly to scroll, the cursor will be on the first line, hiding the sticky context,

actually we can do set scrolloff 1,0. Or `"%opt{lsp_sticky_contexts_max},1" if sticky context takes more lines.
I think lsp-sticky-context-enable should set scrolloff to that if the top-scrollof one is still at the default 0.
Another glitch is that when scrolling the viewport with the mouse such
that the cursor goes off-screen, then sticky context will go stale. I don't
off-hand know a solution but it's not a big deal.

it makes it flexible to choose what do you want to see.

I feel strongly that configuration options are not a substitute for figuring out a good default behavior.
It's fine to take them for now but in future I'd probably opt to kill your darlings and replace them with a more robust UI, see below.

declare-option int lsp_sticky_contexts_max 4

My feeling is that 4 lines is way too much noise for the default behavior.
Even 2 lines I don't really like though I didn't try for long, maybe I'm wrong.
My use case for this feature is to quickly see which function I'm viewing.
One thing that helps a lot is if it's always in the same place.
Take the modeline as example; there is a static location I can glance at to find out how many selections I have etc.
So I'd prefer one line with minimal clutter, like:
... → subnamespace → someclass → somefunction() → innerfunction()

which I believe is what VSCode does.
I'd like to try putting the sticky context in the modeline (which shows up in the terminal title). That would happen to work well with my setup. No color support but I don't care about that.
It's probably fine to merge these features in a less-than-perfect state but
I'd really like to avoid what happened with inlay hints, they had horrible
glitches for years until we got a very simple patch from upstream adding
flag-lines -after which fixed it.

Didn't try, but feels like keywords like fn, class, package, etc. are important to show. It'd would be difficult to do that for each language.

in the projects I work on it's obvious from the naming convention whether something is a type or a function, and the remaining symbol kinds are probably insignificant. If needed we can add the LSP-provided kind like Function and wait for more standardization.

@krobelus if I'm not mistaken then you're referring to this one?

@krobelus Pushed a new commit to replace sticky contexts with breadcrumbs in the modeline. It's a quick prototype to see how it works.
What perhaps nice to have:

Highlight the last element using markup, e.g. {Information}
Render on the left side?

Linked to this: https://github.com/mawww/kakoune/issues/4507

There are two options on what two show:
a. the sticky line, possibly with some markup
b. breadcrumbs
and several options how to show it:

top line using replace-ranges
modeline (which shows up as terminal title)
top line using a new highlighter

I think breadcrumbs with modeline or new highlighter have potential to be enabled by default.
I propose we merge modeline breadcrumbs enabled by default and watch out for any problems[*].
Does that sound good to you?
Some comments on the breadcrumbs MVP:

the modeline change should be window-scoped, not buffer scoped because you can have two clients showing a buffer with different cursor position.
I prefer > to ->, to save space

Render on the left side?

The problem is that that would be ambiguous with echo. Not sure if that's a problem in practice... maybe something to try for the future though I wouldn't bet on it.
[*]: If anyone actually needs it we can also add the sticky line using replace-ranges but with plans to replace it with a new highlighter to get rid of glitches.
The new highlighter would also be useful for breadcrumbs -- modeline breadcrumbs work best in envs like i3 where the terminal title is clearly visible; a custom in-buffer line would be better for tmux.

@krobelus Thanks for the feedback!
I would enable breadcrumbs in the modeline by default, yes. It's simple and useful.
Also I think that even in the current implementation sticky contexts also provide a value, but they aren't that stable, and indeed the best thing to do is have highlighters fixed to provide stable, non-glitchy and a pleasant experience.
I don't really know what is the best course of actions here, as I want both to merge it right away as they still seem to be useful in the current form, but also it would be even better if those de-synchronizations would go away. And even better, would be if that highlighter could "copy" the highlighting of the source lines as @raiguard mentioned - that would simply be a perfect solution.
It's a somewhat complex decision, but I would like to avoid the situation of merging them, and then they would be in a good enough to ignore the problem and not to improve state.
I'm also concerned if the code isn't getting too bloated and repetitive where each feature is implemented "in itself" without regards with the rest of the codebase.
So, I don't have a strong opinion over sticky contexts.Would be really cool if we would have improved highlighters in Kakoune right away, but yeah, they need to be implemented first.

If we merge the sticky contexts, I suggest we don't document it and make it a hidden command; then it's fine to remove it without warning.

I'm also concerned if the code isn't getting too bloated

Adding more requests doesn't really cost us anything.
Though it can be a problem if we add competing features and let one of them bitrot.
(Oh and we do have bloat - our debug binary is 177M (!) should look into why)
I'd say don't merge sticky contexts until someone has used them for a while and vouches for them.

@krobelus Yeah, let's keep sticky contexts and indent guides open.
I've extracted the breadcrumbs PR: https://github.com/kakoune-lsp/kakoune-lsp/pull/719

Closing in favour of https://github.com/kakoune-lsp/kakoune-lsp/pull/721.


Prevent multi-character short options

Throws a command authoring error if the developer attempts to use a multi-character short option.

This checklist is used to make sure that common guidelines for a pull request are followed.

[N/A] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[X] I adhere to the Command Guidelines.

View a preview at https://prompt.ws/r/Azure/azure-cli/6429
This is an experimental preview for @microsoft users.

Codecov Report

Merging #6429 into dev will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff         @@
##           dev   #6429   +/-   ##
===================================
  Coverage    0%      0%
===================================
  Files       11      11
  Lines      133     133
  Branches     9       9
===================================
  Misses     133     133

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c17d31...3fbce3b. Read the comment docs.


Roll engine 0d749933bcc6..e69319216729 (11 commits)

git@github.com:flutter/engine.git/compare/0d749933bcc6...e69319216729
git log 0d749933bcc6..e69319216729 --no-merges --oneline
2019-10-09<EMAIL_ADDRESS>[web_ui] Check if a pointer is already down for the specific device (flutter/engine#12470)
2019-10-09<EMAIL_ADDRESS>Roll src/third_party/dart bb04f145b2..41d13f0ffe (24 commits)
2019-10-09<EMAIL_ADDRESS>Roll fuchsia/sdk/core/mac-amd64 from u0JpE... to pczjK... (flutter/engine#13025)
2019-10-09<EMAIL_ADDRESS>Add missing focusable testing info (flutter/engine#13013)
2019-10-09<EMAIL_ADDRESS>Roll src/third_party/skia 54548626a977..e27a503a0a21 (1 commits) (flutter/engine#13024)
2019-10-09<EMAIL_ADDRESS>Refactor: FlutterDartProject (flutter/engine#13006)
2019-10-09<EMAIL_ADDRESS>Roll src/third_party/skia 77dde599c98a..54548626a977 (1 commits) (flutter/engine#13023)
2019-10-09<EMAIL_ADDRESS>Roll src/third_party/skia 2b1a25a4d324..77dde599c98a (1 commits) (flutter/engine#13021)
2019-10-09<EMAIL_ADDRESS>Roll src/third_party/dart f4a72bfc64..bb04f145b2 (18 commits)
2019-10-09<EMAIL_ADDRESS>Roll fuchsia/sdk/core/linux-amd64 from q_uYX... to cknsi... (flutter/engine#13019)
2019-10-09<EMAIL_ADDRESS>Roll fuchsia/sdk/core/mac-amd64 from wuAtw... to u0JpE... (flutter/engine#13018)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

Auto-roller completed checks. About to merge.

Since this commit dart:html is not found in the Firebase-Dart package.
Here is the issue

I'm not using firebase and it fails in my project also since this morning on 1.10.15. Rollbacking to 1.10.14 works.
Error compiling dartdevc module:live_app|lib/util/webrtc/webrtc_web.ddc.js

packages/live_app/util/webrtc/webrtc_web.dart:75:12: Error: Method not found: 'promiseToFuture'.
    return promiseToFuture(enumerateDevices());
           ^^^^^^^^^^^^^^^
packages/live_app/util/webrtc/webrtc_web.dart:79:30: Error: Method not found: 'promiseToFuture'.
    Future<dynamic> future = promiseToFuture(getDisplayMedia({}));
                             ^^^^^^^^^^^^^^^

@obshksw how to rollback to 1.10.14


[Port] Fix HDFS node to only show up for BDC connections (#12714)

Fixes #12713
When I refactored the node logic to delay creating the HDFS connection until the folder was expanded the logic that short-circuited the creation of the session node was also removed. I'm still leaving that part as it is but now just doing a simple check on the connection before creating the session node.
cherry-picked from 572310b9067368be952368b951dea60adb746ddc

Pull Request Test Coverage Report for Build 291903316

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 38.841%

Totals

Change from base Build 290526992:
0.01%

Covered Lines:
19239

Relevant Lines:
45186

💛  - Coveralls


enhancement: CoreTooltip documentation

Modify CoreTooltip documentation

Closing as done

Some enhancements are required as listed below:

[ ] Add expandedCode and remove unnecessary code from code prop

-Fix

[ ] Customization

[ ] Custom child element

[ ] Distance from Anchor alternative example missing

[ ] Variable width

[ ] Transitions

[ ] Virtual element

Change Description

[ ] Change First title and import example missing
[ ] Change title in Showing and hiding


Feature Request: Add ability to split versions apart

While playing with the new overlay feature I recognized I would like to split versions of the same movie apart through PMM rather than doing it manually in Plex GUI.

add split_duplicates: true to you library setup in your config.yml to split all duplicates at the beginning of your run.


RHEL 7, CentOS 7, SLES repos: [FileNotFound] Install ASP.NET Core 2.2 Runtime (v2.2.8) on Linux

Dependencies Resolved

===========================================================================================================================================
 Package                                Arch                  Version                     Repository                                  Size
===========================================================================================================================================
Installing:
 aspnetcore-runtime-2.2                 x86_64                2.2.8-1                     packages-microsoft-com-prod                 30 M
Installing for dependencies:
 dotnet-host                            x86_64                3.0.1-1                     packages-microsoft-com-prod                 39 k
 dotnet-hostfxr-2.2                     x86_64                2.2.8-1                     packages-microsoft-com-prod                196 k
 dotnet-runtime-2.2                     x86_64                2.2.8-1                     packages-microsoft-com-prod                 27 M
 dotnet-runtime-deps-2.2                x86_64                2.2.8-1                     packages-microsoft-com-prod                2.8 k
 libicu                                 x86_64                50.2-3.el7                  base                                       6.9 M

Transaction Summary
===========================================================================================================================================
Install  1 Package (+5 Dependent packages)

Total size: 64 M
Total download size: 30 M
Installed size: 81 M
Is this ok [y/d/N]: y
Downloading packages:
aspnetcore-runtime-2.2.8-rh.rh FAILED
https://packages.microsoft.com/rhel/7/prod/aspnetcore-runtime-2.2.8-rh.rhel.7-x64.rpm: [Errno 14] HTTPS Error 404 - Not Found --:--:-- ETA
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

Error downloading packages:
  aspnetcore-runtime-2.2-2.2.8-1.x86_64: [Errno 256] No more mirrors to try.

We are having problems with the Microsoft linux repos right now, but this is a new one, thanks for the report:

https://packages.microsoft.com/rhel/7/prod/aspnetcore-runtime-2.2.8-rh.rhel.7-x64.rpm: [Errno 14] HTTPS Error 404 - Not Found --:--:-- ETA

@leecow, do you know if this is a consequence of how we're fixing https://github.com/dotnet/core/issues/3863? Are we putting the repo in a bad state? (Temporarily? Maybe it needs to reindex?)

This has been seen on CentOS 7 as well, now:

https://packages.microsoft.com/centos/7/prod/aspnetcore-runtime-3.0.1-rh.rhel.7-x64.rpm:  [Errno 14] HTTPS Error 404 - Not Found

For now, it's possible to work around this issue by taking the URL that hit the 404, removing the rh.rhel7- part of the filename, then running yum install against it:
# CentOS example:
$ sudo yum install https://packages.microsoft.com/centos/7/prod/aspnetcore-runtime-3.0.1-x64.rpm
$ sudo yum install dotnet-sdk-3.0

You can also download it (wget) and install it using rpm -i. https://github.com/dotnet/core/issues/3863#issuecomment-557164394

RPM repos should be cleaned up and good to go now.

Still seeing this on a CentOS 7 install:
https://packages.microsoft.com/centos/7/prod/dotnet-runtime-deps-3.0.1-rhel.7-x64.rpm: [Errno 14] HTTPS Error 404 - Not Found

it's still not working
https://packages.microsoft.com/centos/7/prod/dotnet-runtime-deps-2.2.8-rhel.7-x64.rpm: [Errno 14] HTTPS Error 404 - Not Found
Will it be fixed ?

@leecow, are you working on this? I think we need to delete and reupload to force the index to refresh?
An interesting note, the ...primary.xml file from e.g. https://packages.microsoft.com/sles/15/prod/repodata/ has entries for both the rh.rhel.7 and "normal" packages. The rh.rhel.7 one is listed first, so maybe that's why it's winning. And I suppose the tooling doesn't handle removal properly, and that's why we're having trouble here. Really a shame the backend infra is so unknown.

Yeah, clearing yum's cache sorted that one out for me.

@leecow has this been resolved?


Unable to use multiple descriptor sets in graphics pipeline

I tried to extend the bindless-example and also bind storage buffers to be used by the pipeline.
The source-code can be seen in this gist. I copied in the BufferUploader helper class that I started using to bulk upload buffers to device-local/unmappable memory.
(If it seems interest I can create a PR.)
When I run the code, the application crashes with the following error:
# ... lots lines about creating the images and copying buffers
2023-01-13T13:43:16.016728Z TRACE screen_13::driver::device: waiting...
2023-01-13T13:43:16.017945Z TRACE screen_13::driver::graphic: create
2023-01-13T13:43:16.019378Z  INFO screen_13::driver::shader: vertex attribute 0.0: R32_UINT (offset=0)
2023-01-13T13:43:16.019454Z TRACE screen_13::driver::shader: binding chunk_buffers: 2.0 = UniformBuffer[0]
2023-01-13T13:43:16.019554Z TRACE screen_13::driver::shader: binding : 1.0 = StorageBuffer(ReadWrite)[1]
2023-01-13T13:43:16.019668Z TRACE screen_13::driver::shader: binding textures: 0.0 = CombinedImageSampler[0]
2023-01-13T13:43:16.019725Z DEBUG screen_13::driver::shader: image binding textures using default sampler
2023-01-13T13:43:16.020901Z TRACE screen_13::driver::graphic: detected write attachment 0
2023-01-13T13:43:16.022442Z DEBUG screen_13::event_loop: first frame dt: 0.016666668
2023-01-13T13:43:16.022508Z TRACE screen_13::event_loop: 🟥🟩🟦 Event::RedrawRequested
2023-01-13T13:43:16.028956Z TRACE screen_13::event_loop: received 8 events
2023-01-13T13:43:16.029998Z DEBUG screen_13::driver::swapchain: Swapchain image count: 3
2023-01-13T13:43:16.030097Z DEBUG screen_13::driver::swapchain: Presentation mode: FIFO
2023-01-13T13:43:16.033281Z  INFO screen_13::driver::swapchain: Swapchain dimensions: 512x512
2023-01-13T13:43:16.035246Z TRACE screen_13::event_loop: ✅✅✅ render graph construction: 6159 μs (0% load)
2023-01-13T13:43:16.035303Z TRACE screen_13::display: present_image
2023-01-13T13:43:16.036115Z DEBUG screen_13::graph::resolver: schedule: [0: Test]
2023-01-13T13:43:16.036212Z TRACE screen_13::graph::resolver: leasing [0: Test]
2023-01-13T13:43:16.038265Z TRACE screen_13::driver::render_pass: create
2023-01-13T13:43:16.038444Z TRACE screen_13::graph::resolver: recording pass [0: Test]
2023-01-13T13:43:16.039113Z TRACE screen_13::graph::resolver:   writing 3 descriptors (65 buffers, 64 images)
2023-01-13T13:43:16.039433Z  INFO screen_13::driver::instance: VUID-VkWriteDescriptorSet-descriptorType-00330
2023-01-13T13:43:16.039456Z ERROR screen_13::driver::instance: 🆘 vkUpdateDescriptorSets() pDescriptorWrites[0] failed write update validation for VkDescriptorSet 0x2158a00000000fc[] with error: Write update to VkDescriptorSet 0x2158a00000000fc[] allocated with VkDescriptorSetLayout 0x7675b400000000eb[] binding #0 failed with error message: Attempted write update to buffer descriptor failed due to: Buffer (VkBuffer 0xc4f3070000000021[]) with usage mask 0x22 being used for a descriptor update of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER does not have VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT set.. The Vulkan spec states: If descriptorType is VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER or VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC, the buffer member of each element of pBufferInfo must have been created with VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT set (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkWriteDescriptorSet-descriptorType-00330)
2023-01-13T13:43:16.039491Z DEBUG screen_13::driver::instance: 🛑 PARKING THREAD `main` -> attach debugger to pid 1567066!

I tried to dig through Screen 13's source, but am unsure why the chunk_buffers is logged as UniformBuffer. I suspect this is reflected incorrectly, but am yet to fully understand src/driver/shader.rs.
Any help would be appreciated. I ported the code from a prior experiment with Vulkano, so I either used Screen 13 wrong or there is indeed a bug.

Unfortunately the reported error is not always the same, but can also be:
2023-01-13T18:52:08.665297Z ERROR screen_13::driver::instance: 🆘 vkUpdateDescriptorSets() pDescriptorWrites[0] failed write update validation for VkDescriptorSet 0x2158a00000000fc[] with error: Attempting write update to VkDescriptorSet 0x2158a00000000fc[] allocated with VkDescriptorSetLayout 0x71a35600000000e9[] binding #0 with type VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER but update type is VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER. The Vulkan spec states: descriptorType must match the type of dstBinding within dstSet (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkWriteDescriptorSet-descriptorType-00319)

I did some debugging and set a breakpoint here: https://github.com/attackgoat/screen-13/blob/master/src/graph/resolver.rs#L2641
I double-checked the sets and bindings and the ids are correct and the application goes through each one exactly once before crashing. I did not double-check the descriptor type.
In another debugging session I set a breakpoint here just before updating the descriptor sets: https://github.com/attackgoat/screen-13/blob/master/src/graph/resolver.rs#L2830
rust-gdb was hesitant to let me inspect descriptors properly, but what was a surprise to me with my limited knowledge of Screen 13 is, that L2830-breakpoint is hit twice before crashing. I was expecting a single descriptor update.

I added a debug!-log to Screen 13 and it looks like the bindless storage buffer array is not properly reflected as STORAGE_BUFFER but UNIFORM_BUFFER, but even so the above errors indicate that the DescriptorSets are mismatched, For the time being I added UNIFORM_BUFFER usage to the relevant buffers. This now also adds a new error to the list (the above errors still happen, just random which one I get):
2023-01-13T19:45:19.507570Z ERROR screen_13::driver::instance: 🆘 vkCreateShaderModule(): The SPIR-V Capability (StorageBuffer16BitAccess) was declared, but none of the requirements were met to use it. The Vulkan spec states: If pCode declares any of the capabilities listed in the SPIR-V Environment appendix, one of the corresponding requirements must be satisfied (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkShaderModuleCreateInfo-pCode-01091)

While I do not fully understand the code in src/driver/mod.rs this feature seems to be not available in Screen 13, yet.
Unfortunately my experience with Vulkan and Rust is limited so at this point I will halt my investigation.

This weekend I should be able to learn much more about this and the other issues; for 16bit buffers there is a device feature that needs to be enabled; I ran into that on a private branch and have a fix that I am ready to push up but the fallout changes are pretty large on device creation (user standpoint it gets easier) - this will be v0.9.

Looking forward to v0.9. :+1:
There seems to be two issues I am experiencing right now (excluding 16bit support):

When I use a single DescriptorSet rather than 3, I get rid of the vkUpdateDescriptorSets errors, so there seems to be an issue with multiple using multiple DescriptorSets.
When I then remove the use of 16bit, I get the following error:

2023-01-14T06:00:47.429667Z ERROR screen_13::driver::instance: 🆘 Type mismatch on descriptor slot 0.2 (expected `VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER_DYNAMIC`) but descriptor of type VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER The Vulkan spec states: layout must be consistent with all shaders specified in pStages (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-VkGraphicsPipelineCreateInfo-layout-00756)

Which brings me back to the binding that is reflected as UNIFORM_BUFFER, but should be a STORAGE_BUFFER:
2023-01-14T06:00:47.411752Z TRACE screen_13::driver::shader: binding chunk_buffers: 0.2 = UniformBuffer[0]
2023-01-14T06:00:47.411770Z TRACE screen_13::driver::shader: binding : 0.1 = StorageBuffer(ReadWrite)[1]
2023-01-14T06:00:47.411785Z TRACE screen_13::driver::shader: binding textures: 0.0 = CombinedImageSampler[0]

When I temporarily patch Screen 13 to return DescriptorInfo::StorageBuffer(...) instead, the code runs without errors: https://github.com/attackgoat/screen-13/blob/master/src/driver/shader.rs#L528
Well, I still see only a single triangle instead of a quad, but no validation errors, so most likely something I did wrong.

I fixed issues in my code and with the hacky patch mentioned above, the following code creates the same output as the bindless example: https://gist.github.com/trevex/6dd09a9a97a6a3eb8a091ba44c424fad
Regarding the second issue I suspect this might be a spirq issue, so I will try to reproduce this

Well was not a spirq issue but my fault. I did not specify the Vulkan version using inline-spirv (which I did when I previously used shaderc directly).
Related issue: https://github.com/PENGUINLIONG/spirq-rs/issues/78

So three issues - and I think we have solutions for all now:

16-bit access: 5ec3644 enables the features needed; nothing is required from the gist code in order to work although it should check device.vulkan_1_1_features.storage_buffer16_bit_access before using it (gpuinfo.org indicates 96% support)
SPIR-V 1.3+: Yes for this GLSL to compile as expected you want to specify vulkan1_2 when using inline_spirv!. I believe what led you astray is most screen-13 examples do not show this syntax because they do not require it - the ray tracing ones do. One fix is to jam the vulkan1_2 tag onto every example. For now I have added SPIR-V 1.5 as a sane default to screen-13-hot.
Multiple descriptor sets: The issue here is that driver::PipelineDescriptorInfo stores two members for the Vulkan data; one map of sets-to-layouts and another of sets-to-pool sizes; each was a HashMap and the code in graph::Resolver::write_descriptor_sets was iterating layouts is HashMap::values order (L1654) and later indexing based on descriptor set index (L2681, etc). I pushed a change to use BTreeMap so the order is predictable.

Both gists run without errors for me, and I can use multiple descriptor sets with different combinations. Let me know if you still have issues or see anything else - these are fantastic reports you're making!

I copied in the BufferUploader helper class that I started using to bulk upload buffers to device-local/unmappable memory.
(If it seems interest I can create a PR.)

Certainly! If we can keep the code as unopinionated as possible, to allow for all types of Vulkan/graphics usage, then it would fit into the main crate. If not or it seems like it might cause trade-offs when selecting this type for a project we could give it a home in the contrib/ directory.
I don't currently publish the contrib/ crates, but this summer I hope to get to a beta stage where I start to do that.

Thanks you so much!
I am currently traveling, but will have a look ASAP and close this issue afterwards. I really like my Screen 13 experience so far, I really like how intuitive and nimble (for a lack of a better word) the API is.
Also kudos for tracking down the DescriptorSet issue.
Next on my todo-list is mipmapping and once I have that implemented I will draft up a PR. As it will use a compute shader it might be a nice extension to screen-13-fx.

Thanks works as expected both 16bit support and the DescriptorSet fix (tested on laptop with integrated AMD GPU and desktop pc with NVIDIA)


How to add borders on every page excluding the first page?

I want to apply border settings from the second page, not the first page. How can I achieve that?

Did you find any solution?

Just add empty div at the top of your html with ids such as pageHeader-first/pageHeader-2
Then in config define the contents of the header (see footer example in readme with default and specific config)
Note: height will be the same across the pages, you can only configure contents.
I wanted to add an image as header on the first page and for the rest of the pages just empty margin.
I added the image on pageHeader-first and configured default as empty string.
Note: image on headers only work if the image is locally available, remote url doesn't work.
Hope that helps

"header": {
                contents: '

'
            },

i tried this  in my config but it did not work.

config = {
        header: {
              height: '70px',
              contents: ``,
        },

try this

Did not work.
var options = {
format: 'A4',
"base": "file:///home/xxx/Projects/xxxx/public/",
header: {
height: '70px',
contents: ``,
},
"border": {
"right": ".3in",
"left": ".5in"
},
    }

I have a banner image on first page. problem is it does not take full width and height as border is there

Don't apply border, just apply header and footer. And in config leave the content empty.

But if don't apply border, how do I set margin for other pages.

When you set format='A4', it sets default margin

`<div id="pageHeader"></div>`

do this, so that image is applied on first-page only and set the image height as header height in your config.

When you set format='A4', it sets default margin

it is not doing in my case. I get all pages with any margin. can i set it using marging or some css property?


Create Table with FLOAT and DECIMAL types (now it's an alias for double)

[#267]

Coverage decreased (-0.1%) to 52.766% when pulling 68c7eba83c0e2bebba6cfb399caf2e84bada41e7 on MattPerron:267 into fb6856a5d08be56b56ed670ff230b4a04f244eb0 on cmu-db:master.


Smart key hook 🔑

Background
I'm not yet comfortable with smart locks. I'm also quite particular about following a routine. In my world, if the key is in the door then the likely hood of the door being open is high. I also don't have the greatest memory recently and I've left the key in the door lock too many times recently. I've often let my mind run off with how I could solve this. Whatever the solution it had to look good, be unobtrusive and not change the behaviour of a lifetime. After various far-out ideas in my head, one evening it just clicked. I could use a door contact sensor and leverage a magnet on the keys to replicate the work of a hook and make a smart key hook which could notify me if the keys were on the hook or not. My intention is to have one of these for each of the keys to the external doors as well as for each family members set of house and car keys.
With the addition of sensors I already have on doors and windows, I am also able to send a text-to-speech message if a door closed and the keys weren't placed on the hook after a few minutes.

Objective
Share my automation and build instructions for my smart key hook.
Ingredients

Aqara door/window contact
Magnetic key ring
Magnet to glue onto door contact
Super glue

Shared on Twitter.

Todo

[ ] Automation with TTS
[ ] Lovelace and customise changes
[ ] Blog post/video


User sign up/creation improvements

Users can provide a name when signing up
All fields are now updatable in the admin panel

It also fixes #69

It also addresses #63


Scroll events and bounce rate

Scroll events are passed as interaction hits to Google Analytics. This means that if a visitor scrolls at least 25%, your bounce rate will be affected.
https://www.analyticsmania.com/post/komito-analytics

Added nonInteraction parameter to tracking options.


Fix: wrong group name

Typo in group names:
PMenuSel -> PmenuSel
PMenu -> Pmenu

I actually  would not notice it either, if it were not for the neovim fork I was making. I have already written an e-mail to you about it, but I guess it must have disappeared in the flood e-mails you are getting. Here it is nord.nvim. In the e-mail, I was asking whether you would maybe want to adopt it as a neovim fork, since the colour groups of neovim against vim are slowly diverging. It is an exact copy apart from it including the additional groups.

Sorry for not replying, I've moved your email into the "queue" directory, but the time to process it is rare 😐
I know there is currently some movement regarding the Neovim v0.5 release and its long-awaited features like TreeSitter. There are contributions like arcticicestudio/nord#157 and yours, but I need to understand whats the differences and the actual advantages between a "normal" Vim color scheme plugin and a one that is scoped to Neovim. When there are no real advantages but only satisfying the "because its new and Lua", creating an additional Neovim theme to the official Nord ports will add mostly +99,8% maintenance overhead.
Anyway, I'll try to find some time this weekend to post a reply in all the issues & PRs regarding this topic and also finally implement support for TreeSitter.


Shift Click "clear history"

Ability to Shift click on all the chat bubbles to use the option clear history quickly.
This would be useful for streamers that have to avoid TOS comments made in game.

Will say at the very least, the Chat Filter plugin can be used to remove known words/phrases that you don't want ever showing up. I can still see the use in this for ""imaginative"" stuff that gets around it.


[New Feature]: Winget could add DisplayVersion if the installer does not

Description of the new feature/enhancement
Hey there. Recently I noticed that winget does work correctly as expected when I manually set the DisplayVersion registry value of a setup that doesn't - so why not make winget set it? As it does a hash check, winget verifies the correct version number of the software it installs (if the manifest is correct).
For the setup I tested, a re-install did even remove the DisplayVersion value, so if the user (or anything else) does an update, the value is removed (and not wrong)
Even if there was a setup that does not delete the DisplayVersion, the worst thing that can happen is that winget installs the same version again - which is exactly what happens now.
Proposed technical implementation details (optional)
No response

Duplicate of winget-cli#2294

Close with reason: See above;


Webhook Error: No signatures found matching the expected signature for payload.

How to do ?
Installing the Vercel template
https://github.com/vercel/nextjs-subscription-payments
The problem and logs :

Check this:
https://github.com/vercel/nextjs-subscription-payments/pull/293/files

im still a tad lost, what do you save the env key as here ? it seems different from the webhook api secret

Run into the same issue while testing the app deployed on Vercel, and that what I did wrong (probably as it's not clearly described in Readme):

❌ I used whsec_xxx that I saw when I was adding events, but that webhook secret is for Local testing, not for testing on Vercel

So what you need to do is:

finish selecting event and hit "Add events" button
get redirected to the Webhook page
✅ use Signin secret from the Webhook page as your Production STRIPE_WEBHOOK_SECRET env variable

redeploy the latest build on Vercel in order to apply the updated env variable


Add --no-preview-dart-2 when in dart1 mode.

This is follow-up to https://github.com/flutter/flutter/pull/17682 to fix dart1 tests.

LGTM, thanks for correcting my breakage! The failing device lab test executions were lagging behind by ~10 hours :-/


Follow up and test on bug filed on codespace

https://github.com/microsoft/vscode-maven/issues/504

This is tracked in CSE feedback at https://csefy19.visualstudio.com/CSE Feedback/_workitems/edit/365941


fix: remove outdated .characters for text

$0.characters.count is out of date and make compile error.

thanks for your great works! 👍


cannot use string literal in expression

<test>

test expression

</test>

will get error: Uncaught SyntaxError: Unexpected token }
riot version: 2.0.9
to repro and test:  http://plnkr.co/edit/zSEGhY?p=preview

color=='red' works with no spaces

@rsbondi cool, thanks for telling me this!
This is an easy workaround, so this bug is low pri if people know the trick.

Definitely not good.

Fixed. Thanks!

'
' does not work,
https://muut.com/riotjs/guide/#class-shorthand

Seems it had a problem with JS keywords like new. Totally missed this in tests.
Fixed now: https://github.com/muut/riotjs/commit/d456611c0c450e56afb7ce731a49da192c913910
Thank you @tanzhihang !


Develop

Major update

transaction
sumeragi
json parse

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 14
- Added 20

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 5

See the complete overview on Codacy


The reason why using theshold (0.5) in cal_CIoU

Hi!
Why do you use the theshold (0.5) in cal_CIoU, although the training doesn't give any information about the 0.5? In other words, is it just from the hyp-param tunning, or reasoned from mathematical properties behind the contrastive loss ?

The reason what I'm asking is that recent papers Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning , A Closer Look at Weakly-Supervised Audio-Visual Source Localization use relative prediction, which always choose the 50% region as the prediction results, without any thresholdm so I just become curious :)

Hi!
Why do you use the theshold (0.5) in cal_CIoU, although the training doesn't give any information about the 0.5? In other words, is it just from the hyp-param tunning, or reasoned from mathematical properties behind the contrastive loss ?
The reason what I'm asking is that recent papers Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning , A Closer Look at Weakly-Supervised Audio-Visual Source Localization use relative prediction, which always choose the 50% region as the prediction results, without any thresholdm so I just become curious :)

Hi, I think i can answer your question, you can find the define of ciou in this paper: "Learning to Localize Sound Source in Visual Scenes", they wrote in Results and analysis as:


cp: --reflink with no argument should be assumed --reflink=always

The --reflink command line option to cp takes an optional argument (always, auto, or never, with always being the default) but it can be used without an argument. uutils cp does not allow this type of usage, but should
GNU cp v8.30:
$ touch f && cp --reflink f g
cp: failed to clone 'g' from 'f': Operation not supported

uutils cp:
$ touch f && ./target/debug/cp --reflink f g
./target/debug/cp: invalid argument 'f' for 'reflink'

This is similar to an issue with mktemp --tmpdir, described here: https://github.com/uutils/coreutils/blob/4b517a361de7dfda90f6d131b822ef0704d1e2b9/src/uu/mktemp/src/mktemp.rs#L172-L178

Can confirm. Working on this.

Hm, so I can add  .default_missing_value("auto") but then it still tries to take the file name as an argument. But I can also then set  .require_equals(true), however it will fail if nothing is specified because = is required. It looks like there's no softer version of require_equals in clap.

Maybe this can be of help? This is how we did the --color argument for ls.
https://github.com/uutils/coreutils/blob/043c009a413edeb45013a73817dd5d303a162635/src/uu/ls/src/ls.rs#L1428-L1436

@tertsdiepraam that works, thanks! Will make a PR now.


ARIA-17: watchdog for mozart job timedout

git push --set-upstream origin ARIA-17-es7

a lot of formatting issues and unused variables (etc. url), but develop-es1 is deprecated for the most part so it LGTM 👍


ARROW-14711: [Javascript] Fixed TextDecoder for old NodeJS versions

A fix was proposed for the issue stated at https://github.com/apache/arrow/issues/11662.
This fix would allow the developer to use older NodeJS versions for running Arrow. An expert review is required for testing the fixed usability.

@sisco0 Thanks for the pull request. Possibly @domoritz  can review?

I'm not very familiar with the node APIs but it seems like this change will break in browsers.

Please take a look to the answer in the original issue https://github.com/apache/arrow/issues/11662#issuecomment-985806777

I'm not very familiar with the node APIs but it seems like this change will break in browsers.

Since this PR is the answer for issue #11662 (Node environment issue) the browser environment will not see the change as require check, used to determine if global object should be augmented, is undefined in the browser.
Additional/alternative check could be typeof window === 'undefined' to ensure this will not break into browser env.
However I think both solutions will not be helpful as the issue is found in test environment (when running unit tests react-testing-library) and this test environment contains both window object and require function so the global object is not augmented - issue persists.

We don't have these checks for node anywhere else since they are brittle. Instead, it would be better to use different files such as https://github.com/apache/arrow/blob/master/js/src/Arrow.node.ts and https://github.com/apache/arrow/blob/master/js/src/Arrow.dom.ts.
Also, could you clarify which node version is considered old here? It's important to know so we can remove the patch when node deprecates the old version.


Update RELEASE.md to record deployment block #

Checklist:

[x] Code contains relevant tests for the problem you are solving
[x] Ensure all new and existing tests pass
[x] Update any relevant READMEs and docs
[x] Submit to the develop branch instead of master

Description:
Please explain the changes in this PR:
Update RELEASE.md with a note to record the block # contracts were deployed at. This allows us to know where in the blockchain events for a given version of the contracts occurred.

@ambertch where should these be recorded?

The truffle run logs containing the addresses are currently recorded in the release notes, alongside those could be a good place to store the information

@ambertch -- Can you help me understand the motivation for this again? Is it specifically for our Indexing service, so when finds Events on the blockchain it knows what version it's dealing with? I'm okay with this, but it seems brittle. Is there no way to trace the event back to the contract that emitted it, and thus derive the version from the contract address?

@wanderingstan yes that's true. The events are tagged with the source address as well: https://api-ropsten.etherscan.io/api?module=logs&action=getLogs&fromBlock=0&toBlock=latest&topic0=0xd8562534ed9a3ce564a595a66dbcc871a37cff70205ea8bc8d0276dcaa4d1009&apikey=9BG9HPBASP3X95Z5ERF6TY99AXAE1XXSME&address=0x2861f28756e14cf0733383be8aa66ba3b65f1b4e
Mainly I just thought that might be a good-to-have, the block number can be determined from the address as well (from the contract creation transaction).

Added manually in https://github.com/OriginProtocol/origin-js/commit/4d6530a37f5b790878b82ad26ea113eb5480791d to get it on the release branch.


Add Sasha to jhu-biobank-nf-project.yaml

FYI @ajs3nj

@thomasyu888 sorry for the confusion. This is an old defunct workspace that we are reviving. I changed the name to avoid the exact kind of confusion that this PR has caused I think.

This one is not time sensitive (the other one name "ntap-add5-project" is the one that is time sensitive)
The runs on this one are old test runs and ok to be deleted
If the workspace gets deleted and new one gets created that will be ok. Currently, neither I nor Sasha have read-write access to this workspace.


Exclude specific struct properties from Serialization

Please explain the suggested feature in detail.
Currently all properties inside a struct are serialized and synchronized via the network. By adding the [NonSerialized] header attribute to a property, it could be excluded from that process. Other network libraries allow that, this is sometimes also known as "skip serializing field".
How exactly does this keep you from releasing your game right now?
Just a performance improvement as it would allow to cache certain things locally in the struct.
Can you suggest a possible solution?
If its not possible (due to fixed size of structs or whatever) it would also be possible to synchronise those properties with 0, "" (nullstring) or NULL instead. Especially in the case of a string it would reduce size.

fixed in 7f8733ce6a8f712c195ab7a5baea166a16b52d09


Results page in README is 404

The README.md has a link to a page describing the results.
It points to: https://sonobuoy.io/docs/results.md
Which is 404

The page should be https://sonobuoy.io/docs/v0.16.5/results/
or even https://sonobuoy.io/docs/master/results/ for master


Fix documentation in readme.

[x] Correct the documentation in readme.md to show what each of the parameters do, but make it clearer as to which parameter belongs to which function.
[x] Finish writing the parameter descriptions for the inputcolour and testcolour functions.

Updated the readme which includes the missing parameters and added sub-headings for each of the functions to make it clearer which parameters are required.


No IPv6 support

First of all, thanks for this great plugin!
In the past I was using the custom executor, a self-written python script and a non-proxmox hypervisor to run my Gitlab-runner jobs. I noticed a while ago that Gitlab introduced the new fleeting system, but as my Go skills are shit I wasn't able to get it to work.
Your plugin looks great for my use-case - there's just one problem (and a couple questions) that I have - my VMs are running in IPv6-only mode and thus do not have any local IPv4 address. Gitlab handled that just fine in the past, but it looks like your plugin has a hardcoded dependency when querying the qemu agent to only return IPv4 adresses. That leads to the following error
ERROR: connection preparation failed                err=rpc error: code = Unknown desc = failed to determine IPv4 address for instance instance=100 runner=xxxxxxxx subsystem=taskscaler took=179.669512ms
since my VMs don't have IPv4 adresses.
Is there a particular reason for this restriction, and can it be removed? Ideally it would prefer IPv6 by default (since that's current-gen internet) but have a config entry to revert back to IPv4 if necessary.
Also, a related question: With the above error and the VM being unreachable, the plugin seems to create new VMs over and over and over, hoping that one will finally come online. Is there some kind of timeout / max attempt counter I can set? I don't really want the plugin to keep making new VMs over and over if there's some kind of error like this one. I checked the plugin settings but didn't find a setting like "max_attempts" or similar.
Is there a setting so that the plugin only tries to set up a new VM once or twice and then just returns an error to Gitlab if the VM's still not up? Is that even something that can be controlled from within the plugin or is that something I'd need to configure in or report to Gitlab?

Quick update: There doesn't seem to be a reason for that restriction, if I modify the "ConnectInfo" function and remove the block for non-ipv4-adresses, it works just fine and Gitlab connects to my machine.
I will try to make a merge request that adds an option to choose between auto, ipv4 and ipv6, if that's okay.

Is there a particular reason for this restriction, and can it be removed?

The only reason is that I used IPv4 in my environment, it can be changed to restrict to IPv6 instead. With removal I'm not sure if it would deterministically return the same address every time without additional logic

I will try to make a merge request that adds an option to choose between auto, ipv4 and ipv6, if that's okay.

Sure! Just please set ipv4 as the default so that it is not a breaking change. For auto it should probably check them in order to make it deterministic.

Is there a setting so that the plugin only tries to set up a new VM once or twice and then just returns an error to Gitlab if the VM's still not up?

I think that this is controlled by GitLab Runner but I will look into this when I have some free time

Thanks for the response. I've opened MR #12, there's now a new parameter that defaults to ipv4 (old behavior) that can be set to ipv4, ipv6 or any. When set to any, it will always use an IPv6 address if the VM has one, and an IPv4 if it doesn't, so it should still be deterministic.