Illegal instruction (core dumped) when running PopPUNK

Hi,
I've been trying to get PopPUNK running on a server to no avail - with the main error being:
PopPUNK (POPulation Partitioning Using Nucleotide Kmers)
        (with backend: sketchlib v1.5.3
         sketchlib: /home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/pp_sketchlib.cpython-38-x86_64-linux-gnu.so)
Mode: Creating clusters from assemblies (create_db & fit_model)
Illegal instruction (core dumped)

all of this being on a newly made conda environment
I've seen #106 had a similar problem and tried following the steps that were mentioned there, but none of them have solved the issue.
Looking at the apport.log file from the dumped core it only states:
ERROR: apport (pid 25164) Thu Nov 19 21:51:23 2020: writing core dump to core (limit: -1)
ERROR: apport (pid 26346) Thu Nov 19 21:54:28 2020: called for pid 26323, signal 4, core limit<PHONE_NUMBER>3709551615, dump mode 1
ERROR: apport (pid 26346) Thu Nov 19 21:54:28 2020: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 26346) Thu Nov 19 21:54:28 2020: script: /home/ubuntu/anaconda3/envs/ppunk/bin/poppunk, interpreted by /home/ubuntu/anaconda3/envs/ppunk/bin/python3.8 (command line "/home/ubuntu/anaconda3/envs/ppunk/bin/python3.8 /home/ubuntu/anaconda3/envs/ppunk/bin/poppunk --easy-run --r-files reference_list.txt --output lm_example/ --threads 8 --plot-fit 5 --min-k 13 --full-db")
ERROR: apport (pid 26346) Thu Nov 19 21:54:28 2020: executable does not belong to a package, ignoring

I've tried compiling pp-sketchlib from scratch as I've read that might cause some problems (especially regarding the CPU) but it presents its own problem when trying to build it.
Running  python3 setup.py install in the cloned pp-sketchlib repository yields:
running install
running bdist_egg
running egg_info
writing pp_sketchlib.egg-info/PKG-INFO
writing dependency_links to pp_sketchlib.egg-info/dependency_links.txt
writing entry points to pp_sketchlib.egg-info/entry_points.txt
writing requirements to pp_sketchlib.egg-info/requires.txt
writing top-level names to pp_sketchlib.egg-info/top_level.txt
reading manifest file 'pp_sketchlib.egg-info/SOURCES.txt'
writing manifest file 'pp_sketchlib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/pp_sketch
copying pp_sketch/matrix.py -> build/lib.linux-x86_64-3.8/pp_sketch
copying pp_sketch/__main__.py -> build/lib.linux-x86_64-3.8/pp_sketch
copying pp_sketch/__init__.py -> build/lib.linux-x86_64-3.8/pp_sketch
running build_ext
-- The C compiler identification is GNU 5.5.0
-- The CXX compiler identification is GNU 5.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/linuxbrew/.linuxbrew/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/linuxbrew/.linuxbrew/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /home/ubuntu/anaconda3/envs/ppunk/bin/python3 (found version "3.8.6")
-- Found PythonLibs: /home/ubuntu/anaconda3/envs/ppunk/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/ubuntu/anaconda3/envs/ppunk/include (found version "2.6.1" )
-- Found Armadillo: /home/ubuntu/anaconda3/envs/ppunk/include (found version "9.900.4")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Found OpenMP: TRUE (found version "4.0")
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- CUDA not found, compiling CPU code only
-- Configuring done
CMake Warning at CMakeLists.txt:51 (add_library):
  Cannot generate a safe runtime search path for target pp_sketchlib because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libgomp.so.1] in /home/linuxbrew/.linuxbrew/Cellar/gcc/5.5.0_4/lib may be hidden by files in:
      /home/ubuntu/anaconda3/envs/ppunk/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8
/home/ubuntu/anaconda3/envs/ppunk/bin/cmake -S/home/ubuntu/pp-sketchlib -B/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8 --check-build-system CMakeFiles/Makefile.cmake 0
/home/ubuntu/anaconda3/envs/ppunk/bin/cmake -E cmake_progress_start /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8//CMakeFiles/progress.marks
/usr/bin/make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
/usr/bin/make  -f CMakeFiles/pp_sketchlib.dir/build.make CMakeFiles/pp_sketchlib.dir/depend
make[2]: Entering directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
cd /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8 && /home/ubuntu/anaconda3/envs/ppunk/bin/cmake -E cmake_depends "Unix Makefiles" /home/ubuntu/pp-sketchlib /home/ubuntu/pp-sketchlib /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8 /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8 /home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles/pp_sketchlib.dir/DependInfo.cmake --color=
Dependee "/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles/pp_sketchlib.dir/DependInfo.cmake" is newer than depender "/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles/pp_sketchlib.dir/depend.internal".
Dependee "/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8/CMakeFiles/pp_sketchlib.dir/depend.internal".
Scanning dependencies of target pp_sketchlib
make[2]: Leaving directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
/usr/bin/make  -f CMakeFiles/pp_sketchlib.dir/build.make CMakeFiles/pp_sketchlib.dir/build
make[2]: Entering directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
[  7%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/dist/dist.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/dist/dist.cpp.o -c /home/ubuntu/pp-sketchlib/src/dist/dist.cpp
[ 15%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o -c /home/ubuntu/pp-sketchlib/src/sketchlib_bindings.cpp
[ 23%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/dist/matrix_ops.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/dist/matrix_ops.cpp.o -c /home/ubuntu/pp-sketchlib/src/dist/matrix_ops.cpp
[ 30%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o -c /home/ubuntu/pp-sketchlib/src/reference.cpp
[ 38%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/sketch/seqio.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/sketch/seqio.cpp.o -c /home/ubuntu/pp-sketchlib/src/sketch/seqio.cpp
[ 46%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/sketch/countmin.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/sketch/countmin.cpp.o -c /home/ubuntu/pp-sketchlib/src/sketch/countmin.cpp
[ 53%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/sketch/sketch.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/sketch/sketch.cpp.o -c /home/ubuntu/pp-sketchlib/src/sketch/sketch.cpp
[ 61%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/database/database.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/database/database.cpp.o -c /home/ubuntu/pp-sketchlib/src/database/database.cpp
[ 69%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/api.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/api.cpp.o -c /home/ubuntu/pp-sketchlib/src/api.cpp
[ 76%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/dist/linear_regression.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/dist/linear_regression.cpp.o -c /home/ubuntu/pp-sketchlib/src/dist/linear_regression.cpp
[ 84%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/random/rng.cpp.o
/home/linuxbrew/.linuxbrew/bin/c++ -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/home/ubuntu/pp-sketchlib/src -isystem /home/ubuntu/anaconda3/envs/ppunk/include -isystem /home/ubuntu/anaconda3/envs/ppunk/include/python3.8 -isystem /home/ubuntu/anaconda3/envs/ppunk/include/eigen3 -DVERSION_INFO=\"1.5.4\" -march=native -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fopenmp -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/random/rng.cpp.o -c /home/ubuntu/pp-sketchlib/src/random/rng.cpp
In file included from /home/ubuntu/pp-sketchlib/src/random/rng.cpp:1:0:
/home/ubuntu/pp-sketchlib/src/random/rng.hpp:17:17: error: 'size_t' does not name a type
         typedef size_t result_type;
                 ^
/home/ubuntu/pp-sketchlib/src/random/rng.hpp:18:26: error: 'size_t' does not name a type
         static constexpr size_t min() { return std::numeric_limits<uint64_t>::min(); }
                          ^
/home/ubuntu/pp-sketchlib/src/random/rng.hpp:19:26: error: 'size_t' does not name a type
         static constexpr size_t max() { return std::numeric_limits<uint64_t>::max(); }
                          ^
CMakeFiles/pp_sketchlib.dir/build.make:214: recipe for target 'CMakeFiles/pp_sketchlib.dir/src/random/rng.cpp.o' failed
make[2]: *** [CMakeFiles/pp_sketchlib.dir/src/random/rng.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
CMakeFiles/Makefile2:97: recipe for target 'CMakeFiles/pp_sketchlib.dir/all' failed
make[1]: *** [CMakeFiles/pp_sketchlib.dir/all] Error 2
make[1]: Leaving directory '/home/ubuntu/pp-sketchlib/build/temp.linux-x86_64-3.8'
Makefile:105: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 100, in <module>
    setup(
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in setup
    return distutils.core.setup(**attrs)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 175, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 161, in call_command
    self.run_command(cmdname)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 92, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.

This is all with the command:
poppunk --easy-run --r-files reference_list.txt --output lm_example --threads 8 --plot-fit 5 --min-k 13 --full-db
Running this locally on my computer does not seem to give the same error on the other hand when following all the same steps - so I'm stumped for now on what the problem could be

Hi,
Thank you for the very detailed error report. As you note, this has been seen before on some machines, though for me it went away and I wasn't sure why. I think there are two possible culprits:

pp-sketchlib being compiled with the wrong options
a problem with graph-tool

My feeling is it's the latter, but if we can get pp-sketchlib to compile for you locally we can rule that out.
I had missed a header in the failing rng.hpp file which affects some systems, and gave the error above. I have now added this in on the master branch of pp-sketchlib.
If you are able to, could you clone that and try installing again as you have done above? I hope that should work now.
If you still get the error I think it's likely graph-tool's fault, and I can look into that instead.

No problem, thank you for making this tool.
I've cloned the master branch for pp-sketchlib and it completed fine with no error messages. I've tried running the same poppunk command as above but it still seems to be giving the exact same error of (Illegal instruction (core dumped)).
I've tried running the command
poppunk_sketch --sketch --rfile reference_list.txt --ref-db listeria --sketch-size 156 --cpus 4 --min-k 15 --k-step 2
which results in the same Illegal instruction error. I hope that helps.

Ok, does seem like a sketchlib issue then. Can you run poppunk_sketch --version, just to make sure that's picking up the copy you installed? It should show v1.5.4.

Running poppunk_sketch --version oddly enough gives me poppunk_sketch 1.5.3.  Though when looking at the output from python3 setup.py install for building pp-sketchlib does indeed state the correct version (1.5.4). Looking inside the file ~/anaconda3/envs/ppunk/bin/poppunk_sketch states version 1.5.4.

Great, thanks. And you are still getting the Illegal Instruction error?
Sorry if this is a pain, but would it be possible to try the following:

Clean your sketchlib install.
Reinstall with python3 setup.py install --debug (you should see -O0 and -g options in the compile lines generated by cmake if this worked)
Run gdb -ex r --args python3 ~/anaconda3/envs/ppunk/bin/poppunk_sketch.py --sketch --rfile reference_list.txt --ref-db listeria --sketch-size 156 --cpus 4 --min-k 15 --k-step 2
In the gdb debugger this should automatically run the program until the error (if it doesn't, type the command run)
This should get to the error line. Type bt to get a backtrace. Post the results of this backtrace here.

I would greatly appreciate it if you could help with this, as I've been unable to replicate the problem myself!

I just ran it again, and the command
poppunk_sketch --sketch --rfile reference_list.txt --ref-db listeria --sketch-size 156 --cpus 4 --min-k 15 --k-step 2
seems to be running fine and outputting a listeria.h5 file.
I tried running
poppunk --easy-run --r-files reference_list.txt --output lm_example --threads 8 --plot-fit 5 --min-k 13 --full-db
on the other hand results in a different error.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/ppunk/bin/poppunk", line 33, in <module>
    sys.exit(load_entry_point('poppunk==2.3.0', 'console_scripts', 'poppunk')())
  File "/home/ubuntu/anaconda3/envs/ppunk/bin/poppunk", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/PopPUNK/__main__.py", line 19, in <module>
    from .models import *
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/PopPUNK/models.py", line 22, in <module>
    from .plot import plot_scatter
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/PopPUNK/plot.py", line 28, in <module>
    from .utils import isolateNameToLabel
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/PopPUNK/utils.py", line 18, in <module>
    import h5py
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/h5py/__init__.py", line 33, in <module>
    from . import version
  File "/home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/h5py/version.py", line 15, in <module>
    from . import h5 as _h5
  File "h5py/h5.pyx", line 1, in init h5py.h5
ImportError: /home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/h5py/defs.cpython-38-x86_64-linux-gnu.so: undefined symbol: H5Dget_num_chunks
```

Ok, so perhaps there was an instruction in the conda version that was unsupported by your CPU. Good that the manual install resolved that – it may have to be the official work-around for this issue.
The new error looks like a problem with your h5py installation, the error is just coming from import h5py, not the PopPUNK code. Do you have libhdf5-dev installed on ubuntu? Or are you using the conda installation? Can you post the result of conda list?

These are the results from conda list. Doing import h5py in a normal python shell seems to work okay.
For reference, I have:
h5py                      2.10.0          nompi_py38hfb01d0b_104    conda-forge
hdf5                      1.10.6          nompi_h3c11f04_101    conda-forge

It might first be instructive to run:

ldd /home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/h5py/defs.cpython-38-x86_64-linux-gnu.so to check the HDF5 version is compatible
nm -gDC /home/ubuntu/anaconda3/envs/ppunk/lib/python3.8/site-packages/h5py/defs.cpython-38-x86_64-linux-gnu.so to get the symbol list

Another suggestion would be that perhaps you could try downgrading to this version of h5py with conda install h5py==2.10? But let's see the results of the above first

Yep, the python in the conda environment has it, but when poppunk tries to run it it gives an error. I noticed that even when though I have h5py through python it did not show up in the conda list.

It looks like your environment is inconsistent, and has packages from conda, pip and homebrew. The supported install is everything via conda, so I'd suggest starting a fresh environment and making sure everything is being pulled in from there. I would suggest conda create -n pp_py38 poppunk==2.2.0 pp-sketchlib==1.5.3.
I guess given the above discussion this will give an illegal instruction error. You will then need to reinstall as before. Check your conda list at this point to check you've got pp-sketchlib 1.5.4 (listed under pip), and h5py and hdf5 from conda-forge.
Make sure echo $LD_LIBRARY_PATH is empty.
Not sure how you ended up with PopPUNK v2.3.0, as that's only on the master branch and hasn't been released yet. In that new version --easy-run has been removed (and is a script). You can run in three steps --create-db, --fit-model dbscan, --fit-model refine. But as we haven't updated the docs yet that might be a bit tricky, and you should probably stick with v2.2.0.

So after running brew upgrade and having that go - I ran conda create -n pp_py38 poppunk==2.2.0 pp-sketchlib==1.5.3. Afterwards to test out, I ran the
 poppunk --easy-run --r-files reference_list.txt --output lm_example --threads 8 --plot-fit 2 --min-k 13 --full-db
command just to check if it still gives me the same error. Surprisingly it did not give me the Illegal Instruction error but instead:
Graph-tools OpenMP parallelisation enabled: with 8 threads
PopPUNK (POPulation Partitioning Using Nucleotide Kmers)
        (with backend: sketchlib v1.5.3
         sketchlib: /home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/pp_sketchlib.cpython-38-x86_64-linux-gnu.so)
Mode: Creating clusters from assemblies (create_db & fit_model)
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pp_py38/bin/poppunk", line 10, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/PopPUNK/__main__.py", line 363, in main
    createDatabaseDir(args.output, kmers)
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/PopPUNK/sketchlib.py", line 82, in createDatabaseDir
    knum = ref_db['sketches/' + sample_name].attrs['kmers']
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/h5py/_hl/attrs.py", line 56, in __getitem__
    attr = h5a.open(self._id, self._e(name))
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 80, in h5py.h5a.open
KeyError: "Can't open attribute (can't locate attribute: 'kmers')"

This is all without messing with pp-sketchlib or any other package (as I had to do before). In conda list, both hdf5 and h5py are present - with PopPUNK 2.2.0 and poppunk_sketch 1.5.3.
Sorry for the many issues!

Ok, hopefully looks like the environment is sorted now!
Feels like we're getting there.
What's in lm_example? Could it be a stale/corrupt .h5 file from before giving the error?

I might have spoken too early. After removing a leftover lm_example.h5 file it went back to the Illegal Instruction error so I compiled pp-sketchlib from scratch again. Afterwards running the usual poppunk command I get:
Graph-tools OpenMP parallelisation enabled: with 8 threads
PopPUNK (POPulation Partitioning Using Nucleotide Kmers)
        (with backend: sketchlib v1.5.4
         sketchlib: /home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/pp_sketchlib-1.5.4-py3.8-linux-x86_64.egg/pp_sketchlib.cpython-38-x86_64-linux-gnu.so)
Mode: Creating clusters from assemblies (create_db & fit_model)
Sketching 2 genomes using 2 thread(s)
Writing sketches to file
Problem processing h5 databases during QC - aborting
Unexpected error: <class 'KeyError'>
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pp_py38/bin/poppunk", line 10, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/PopPUNK/__main__.py", line 364, in main
    seq_names = constructDatabase(
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/PopPUNK/sketchlib.py", line 365, in constructDatabase
    filtered_names = sketchlib_assembly_qc(oPrefix,
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/PopPUNK/utils.py", line 462, in sketchlib_assembly_qc
    seq_length[dataset] = hdf_in['sketches'][dataset].attrs['length']
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/ubuntu/anaconda3/envs/pp_py38/lib/python3.8/site-packages/h5py/_hl/attrs.py", line 56, in __getitem__
    attr = h5a.open(self._id, self._e(name))
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 80, in h5py.h5a.open
KeyError: "Can't open attribute (can't locate attribute: 'length')"

with a single file created in lm_example/.

Ah, great! Sorry it was a bit of a long route to get it working!
300 Staph, if you're using a few cores, should really only be of the order of a few minutes. The main computation step is --create-db, all the model fitting is very fast

Absolutely no problem. Thank you so much for helping throughout the whole ordeal!
Ah alright  - I've started running it 2 days ago (with 8 cores) and still seems to be on the step Trying to optimise score globally, which I'm guessing then might not be normal? The genomes being from ~5 different staph species.

That's probably having trouble fitting the model in that case.
Can I suggest you start with two separate runs, one with --fit-model gmm --K 5 --output staph_gmm --ref-db <name> one with --fit-model dbscan --output staph_dbscan --ref-db <name> and look at the plots in the output, before trying a refined fit?
(if on poppunk 2.2.0 or lower, the first command becomes --fit-model --K 5 --output staph_gmm --ref-db <name> --distances <name/name.dists>, the second --fit-model --dbscan --output staph_dbscan --ref-db <name> --distances <name/name.dists>)
It might be worth having a look through the (long) tutorial on the docs to get an idea of how to deal with different fits, but feel free to post some of your output plots here and I can take a look

Sorry for taking some time to go through it!
I've tried running it with --K 5 as well as some other --K values  (there were a total of 7/8 species instead*). It works great and all of them seem to give me somewhat similar plots with some slight variation
The results from --K 5 were below, with the other values giving similar amounts.
Network summary:
        Components      95
        Density 0.0501
        Transitivity    0.9941
        Score   0.9443

The DPGMM graph I got from --K 5 is below:

And the db_scan:

From what I can tell the fit still needs to be improved (although the network summary does show good values?).
I've tried to go through the tutorial and tune some parameters to see if I can get a better fit, but due to not much experience in it, it has been a bit more difficult for me to understand.
Thank you for the all the help!

Both of those fits look ok to me, the DBSCAN fit more specific, the GMM fit broader. I'd say the DBSCAN fit looks best, but I'd run with --microreact to check you're happy with your results on the tree.
You could also try running --refine-fit on your GMM model, which I'd expect to give something similar to the DBSCAN model.

Closing due as appears to be solved. Please do open another issue, or email me, if you want to discuss your fitting further.
The updated documentation may also be helpful: https://poppunk.readthedocs.io/en/latest/model_fitting.html


[Bug]: getAllProducts() returns an array instead of a paginated result

What happened?
The API only returns the first 10 results, see https://docs.lemonsqueezy.com/api#pagination
Instead the API Product::getAllProducts() should return a ProductPage extends PaginatedResult.
Where PaginatedResult is an abstract class with a couple of methods:

countPages()
nextPage()
previousPage()
firstPage()
lastPage()
getResults()

Additionally getAllProducts(PageQuery $query) method should accept a query object with params:

pageSize
number
...

How to reproduce the bug
See above
Package Version
x
PHP Version
8.2
Which operating systems does with happen with?
macOS
Notes
No response

To explain why I changed this from a bug to an enhancement, Pagination was defined in the initial Roadmap, and it's not set as Done. I was expecting to work on this by the end of the development, but as you require this, I will make sure to include this after the discounts.


upon exetsnion being added users should be prompted to add there leetcode info in so that they sign in prior ro using the app

As the title states for easier usage, also i think the extension itself should have a toggle to turn off or on. Idk if soemone else stated this though!

Working on some of these features soon, thanks for opening the issue!


Use yaml.safe_load() instead of yaml.load() in IDA cookiecutter.

Description:
fix: Use yaml.safe_load() instead of yaml.load(), which errors in pyyaml>=6.
Reference PRs:
https://github.com/openedx/blockstore/pull/146
https://github.com/openedx/blockstore/pull/152
Merge checklist:

[ ] All reviewers approved
[ ] CI build is green
[ ] Changelog record added
[ ] Documentation updated (not only docstrings)
[ ] Commits are squashed

Thanks for making this fix!


States solution iterations print

Closes #381

Codecov Report

Merging #386 (c7e4666) into main (7904999) will decrease coverage by 1.16%.
The diff coverage is 30.61%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main     #386      +/-   ##
==========================================
- Coverage   86.92%   85.76%   -1.16%
==========================================
  Files          43       43
  Lines        1568     1588      +20
==========================================
- Hits         1363     1362       -1
- Misses        205      226      +21

Impacted Files
Coverage Δ

src/StructuralSolvers/Solutions.jl
30.92% <3.22%> (-8.04%)
:arrow_down:

src/StructuralSolvers/StructuralSolvers.jl
72.22% <77.77%> (-1.39%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


replace file

I would like to load a JSON file from an external server and replace the one that already exists, because I mainly want to use sleekDB's search & filter. Is there the possibility to replace a file in sleekDB?
I tried that, but a new file with a new _id is always created?
 $databaseDirectory = $_SERVER['DOCUMENT_ROOT'] . "/data/test";

            $storeConfiguration = [
                "auto_cache" => true,
                'timeout' => false,
            ];
            $userStore = new \SleekDB\Store("customers", $databaseDirectory, $storeConfiguration);

            $newUsers = [
                [
                    "username" => "Lisa",
                    "age" => 17,
                    "address" => [
                        "street" => "up street",
                        "streetNumber" => 48,
                        "postalCode" => "1822",
                    ],
                ],
                [
                    "username" => "Bob",
                    "age" => 20,
                    "address" => [
                        "street" => "down street",
                        "streetNumber" => 12,
                        "postalCode" => "8174",
                    ],
                ],
            ];

            // Update all existing documents with the new data
            $userStore->updateOrInsert($newUsers);

The first thought is, that you need to delete all the old users.
Perhaps I get you wrong. As long, you won't provide an _id, sleek will create new users. For updating, the _id is required. A workaround could be this:

$databaseDirectory = $_SERVER['DOCUMENT_ROOT'] . "/data/test";

$storeConfiguration = [
    "auto_cache" => true,
    'timeout' => false,
];
$userStore = new \SleekDB\Store("customers", $databaseDirectory, $storeConfiguration);

$newUsers = [
    [
        "username" => "Lisa",
        "age" => 17,
        "address" => [
            "street" => "up street",
            "streetNumber" => 48,
            "postalCode" => "1822",
        ],
    ],
    [
        "username" => "Bob",
        "age" => 20,
        "address" => [
            "street" => "down street",
            "streetNumber" => 12,
            "postalCode" => "8174",
        ],
    ],
];

$oldUsers = $userStore->findAll();

$updateOrInsertData = [];

foreach ($newUsers as $user) {
    // Check if there is already a user with that username
    if ($oUser = $userStore->findOneBy(['username', '=', $user['username']])) {
        // Already inside database, add _id to the new element
        $updateOrInsertData[] = array_merge($user, ['_id' => $oUser['_id']]);
    } else {
        // Not inside database, just add it
        $updateOrInsertData[] = $user;
    }
}

// Update all existing documents with the new data
$userStore->updateOrInsert($newUsers);


[Feature] Support batch read table without datalake enabled

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Currently, if a table is not with datalake enabled, Fluss won't support batch read the table. Also, although the table is datalake enabeld, it the lake snapshot doesn't exist, the batch read will also fail which will force users to wait the snapshot  available.
We should resolve these two cases for better user experience
Solution
1: In the case that the table is not with datalake enabeld, if it's primary key table, we should mrege ssts and change logs to read the full data, if it's a log table, we can just read the logs upto the latest offset.
2: In the case that the snapshot is not available for datalake enabeld, we can just consider it as a table without datalake enabeld and then batch read
Anything else?
No response
Willingness to contribute

[ ] I'm willing to submit a PR!

I'd like to do it.


Getting TypeError: 'NoneType' in the create_snipe_model(newmodel) function

Hi!
I have a fresh install of SnipeIT and jamf2snipe (from the repo) and when I run it, I am getting the following error:
Traceback (most recent call last):   File ".\jamf2snipe", line 617, in <module>     create_snipe_model(newmodel)   File ".\jamf2snipe", line 417, in create_snipe_model     modelnumbers[jsonresponse['payload']['model_number']] = jsonresponse['payload']['id'] KeyError: 'payload'
I tried in settings.conf to add https to the URL and I get a different error:
Traceback (most recent call last):   File ".\jamf2snipe", line 617, in <module>     create_snipe_model(newmodel)   File ".\jamf2snipe", line 417, in create_snipe_model     modelnumbers[jsonresponse['payload']['model_number']] = jsonresponse['payload']['id'] TypeError: 'NoneType' object is not subscriptable
I have this working on another SnipeIT server that I had running as a POC and the script runs fine. Any thoughts on why I am seeing this error?

This is really odd... looks like it's receiving a 200 OK code from Snipe but there's nothing actually in the response... Or at least that is the case for the second error you have there.
Would be great to find out what's in the response that Snipe IT is sending you. Could find out by pasting this line after the request, like this:
    response = requests.post(api_url, headers=snipeheaders, json=payload, verify=user_args.do_not_verify_ssl, hooks={'response': request_handler})
    print(json.dumps(response.json(), indent=4, sort_keys=True))

That will show us what we're getting from Snipe IT, and should make it easier to figure out what's happening.

I would also check that the user that created the API has the correct permissions to edit objects? (Seen that before where it got over looked.)

Hi @ParadoxGuitarist & @christovic - First of all...so sorry for the delay here. I got completely sidetracked but was finally able to get back to this. I was able to print that info from the console and it reported it was not able to find a valid Category or Manufacturer. I went ahead and created those, set the right params in settings.conf and it works! Thanks a ton guys. Very helpful.

I also had same problem
Traceback (most recent call last):   File "/Users/me/.virtualenv/jamf2snipe/jamf2snipe", line 728, in <module>     create_snipe_model(newmodel)   File "/Users/me/.virtualenv/jamf2snipe/jamf2snipe", line 479, in create_snipe_model     modelnumbers[jsonresponse['payload']['model_number']] = jsonresponse['payload']['id'] TypeError: 'NoneType' object is not subscriptable
By using debug mode (-d), I could see this line of code above the error:
The payload for the POST request is:{'category_id': '2', 'manufacturer_id': '2', 'name': '', 'model_number': 'Mac14,14'}
(We are using jamfschool, so this is probably different for Jamf Pro)
I used postman to query all of the devices (https://OURINSTANCE.jamfcloud.com/api/devices)
Sure enough the 'Mac14,14' model did not have a model name. (I do not if this is an issue with Jamf School...) I created the model in Snipe and used 'Mac14,14' as the Model No.


Refactoring the data source before unnest

Motivation
There are a number of use cases that require “flattening” records when processing. The need for this is exemplified in the existed of MVDs, which basically have a native “flattening” built-in. Other SQL systems can also do similar behaviors, they expect an ARRAY type of data and then you put that into a FLATTEN or UNNEST or other operator and it “explodes” out one new row for every value in the array.
We need to implement a similar operator for Druid. This dove-tails a bit with some of the recent work done for handling arrays in Druid. Essentially, the operator would take in an array and then “flatten” the array into N (N=number of elements in the array) rows where each row has one of the values from the array.
Related Work
The inspiration for using unnest as a data source comes from (Work with arrays  |  BigQuery  |  Google Cloud )  where unnest has been used as a data source. Clickhouse also has a flatten functionality (Array Functions | ClickHouse Docs ) but that does not transform a dataset by adding more rows. Since for our use case, unnest can be used with dateExpand functionality (coming after unnest) we model unnest as a data source similar to Bigquery.
Methodology
Refactoring
The underlying principle here is an operation on a data source that works on a segment and creates additional rows. Joins have a similar principle where the number of rows can be more than the input table after the join operation. The current framework supports that in the following way:
Having a join data source, alongside a factory and then a specialized wrapper (JoinableFactoryWrapper.java ) around it which creates a function to transform one segment into another by the notion of a segment function.
Have a separate implementation of segment reference through  HashJoinSegment.java which uses a custom storage adapter to access a cursor to the segment for processing.
The goal here is to move out the creation of the segment map function from outside the wrapper to individual datasources. In cases where the segment map function is not an identity function (in case of join and also for unnest) the segment functions can be created accordingly for each data source. This makes the abstraction generic and readily extendable to other datasources we might create in future (e.g flatten)
The changes here:

Creation of a segment map function moved into each data source
Code refactored from JoinableFactoryWrapper to JoinDataSource
InputNumberDataSource has a dependency on broadcastJoinHelper which was moved into the data source
PlannerContext was updated with an additional JoinableFactoryWrapper object during this process
Several test cases were updated with the change to the JoinDataSource
Some Guice injector things were updated as a part of the process

New changes (will be in a followup)
After the new abstraction is in place, the following needs to be done for unnest

Have a separate UnnestDataSource class that deals with unnest
This should have it’s own segment map function
A separate UnnestSegment with a custom storage adapter to gain access to cursor in each segment
Design of the native query for Unnest considering we also need to  pass the column(s) to unnest
Finally, the SQL query that maps to the native query

Each of these development phases should be backed by unit tests/ integration tests as applicable
This PR has:

[x] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

Travis is failing from a branch coverage from the changes to BaseLeafFrameProcessor. While I am working on that I am setting this PR up for review now.

Guice issues in Integration tests. Taking a look

@imply-cheddar , @clintropolis I have addressed the comments and also resolved the conflicts. The thing remaining to address it use of a set in GuiceInjectableValues so the nullable check can be avoided

@clintropolis @imply-cheddar @abhishekagarwal87 this is ready for review. Can you please take a look

The conflicts have been resolved. Some additional refactoring introduced due to change in underlying test framework in #12965


Update .ocamlformat to the latest

Bump ocamlformat version to 0.24.1

Codecov Report

Merging #2194 (62d95a8) into main (1a74027) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 62d95a8 differs from pull request most recent head 63b6787. Consider uploading reports for the commit 63b6787 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main    #2194   +/-   ##
=======================================
  Coverage   68.13%   68.13%
=======================================
  Files         135      135
  Lines       16255    16255
=======================================
  Hits        11075    11075
  Misses       5180     5180

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Remove a stray bit of markup

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/python/managing-python-on-azure-app-service.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Thanks Brett!


Update google-cloud-nio to 0.123.17

Updates com.google.cloud:google-cloud-nio from 0.123.14 to 0.123.17.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.google.cloud", artifactId = "google-cloud-nio" } ]

labels: library-update, early-semver-minor, semver-spec-patch

Superseded by #799.


Update webauthn4j version in webauthn4j-spring-security

Update webauthn4j-spring-security to the latest version of webauthn4j published on maven central.
The update from 0.20.6 to 0.20.7 was surprisingly tricky due to the use by spring 2.7.x of a version 1.7.x of SL4J
(with the related 1.2.x version of logback)
For the time being mark as draft PR :

the solution to correct the problem with the tests is only a workaround, there is maybe a better solution, and in any case, the problem may need further discussion.
For now the version 0.20.8 of WebAuthn4j is not on Maven central for the first version of this PR but it will maybe come soon, and thus we can potentially directly update to the version 0.20.8.

Thank you for your suggestion.
It seems it's root cause is webauthn4j-core and webauthn4j-util specify logback version unnecessarily in their poms. (webauthn4j-core and webauthn4j-util use logback  in test module only.)
I'll fix it in the next webauthn4j version, but I'll merge this patch into webauthn4j-spring-security.


desired_accounts.py logging.info is too chatty in syslog

The code "logging.info('Google SSH key schema identifier found.')" will result in two entries per minute in the syslog which is too spammy like and makes remote-monitoring more complex.
» 12:15:43.273 rubystackbox-xxx accounts-from-metadata: INFO Google SSH key schema identifier found.
» 12:15:43.400 rubystackbox-xxx accounts-from-metadata: INFO Google SSH key schema identifier found.
» 12:16:43.323 rubystackbox-xxx accounts-from-metadata: INFO Google SSH key schema identifier found.
» 12:16:43.450 rubystackbox-xxx accounts-from-metadata: INFO Google SSH key schema identifier found.

This got released in our images dated 20150309 and compute-image-packages version 1.2.3. Thanks!


Crash when clicking on equalizer when track is playing

Describe the bug
When I click in the PlayerVIew on the equalizer button while a track is playing, the app crashes. When the track is paused and I click the button, it does not crash.
Relevant Logcat:
[01-31 13:20:06.159 543:543 E/AndroidRuntime]
FATAL EXCEPTION: main
Process: com.sahdeepsingh.Bop, PID: 543
java.lang.UnsupportedOperationException: AudioEffect: invalid parameter operation
	at android.media.audiofx.AudioEffect.checkStatus(AudioEffect.java:1273)
	at android.media.audiofx.BassBoost.setProperties(BassBoost.java:285)
	at com.sahdeepsingh.Bop.equalizer.a.f(Unknown Source)
	at androidx.fragment.app.c.k(Unknown Source)
	at androidx.fragment.app.j.a(Unknown Source)
	at androidx.fragment.app.o.a(Unknown Source)
	at androidx.fragment.app.o.a(Unknown Source)
	at androidx.fragment.app.o.a(Unknown Source)
	at androidx.fragment.app.j.a(Unknown Source)
	at androidx.fragment.app.j.b(Unknown Source)
	at androidx.fragment.app.j.g(Unknown Source)
	at androidx.fragment.app.j$1.run(Unknown Source)
	at android.os.Handler.handleCallback(Handler.java:751)
	at android.os.Handler.dispatchMessage(Handler.java:95)
	at android.os.Looper.loop(Looper.java:154)
	at android.app.ActivityThread.main(ActivityThread.java:6186)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)

Smartphone (please complete the following information):

Device: Moto G4
OS: Android 7.1 (Lineage OS 15.1)
Version 1.0 (FDroid)

I don't have any other devices currently, but it might have to do with using a custom rom: https://stackoverflow.com/a/51918073
This fixed it, but I'm no Android developer, so I don't know if it should always be called OnCreate, or if it will affect other phones differently. Here is my branch where I changed it: https://github.com/RomanKarwacik/Bop/commit/cbbdf65a1a2c4e611239130e7154ac1d1389c8d8

I have tested your branch and its working fine and not effecting other devices. Also as answer suggested to enable the custom equalizer first, i have commented on your commit about it.
You can make PR with and i will merge it.


time: optimize Parse and Time.Format for RFC3339

RFC3339 is the most common format to use (according to prior analysis, it accounts for 57% of formats; and that's only explicit calls to Format or Parse. It doesn't cover all the implicit cases that occur through MarshalText and UnmarshalText). We should optimize Parse and Time.Format to do a quick check for the RFC3339Nano or RFC3339 constants and hardcode the logic for that format. This would avoid the double parsing that currently occurs where we parse the format string to functionally run a virtual machine that then parses the input value.

cc @ianlancetaylor @rsc

Is it appropriate for me to offer my contribution here?

Sure, your contribution is welcomed. See https://go.dev/doc/contribute for how to contribute to Go. Thanks.

Happy to review.

Sure.
thanks @dsnet  @cherrymui
I’ll be back shortly with initial implementation.


Updates to furthest point sampling, plotting.

The resampling option in further points sampling did not reuse points. This lead to sparser graphs. This is now fixed and does not affect the results of the paper.
After further points sampling, the sample_ind attribute is now sorted.
plotting.embedding() now allows plotting trajectories and color gradients to signify the arrow of time. To use this, one needs to add a 'labels' field in construct_dataset() containing the time indices of the data points.

You did not update the tests so they crash (click on the red cross). I could try to fix it later if you want


Investigate low verification completion rate in BR

relevant slack thread
According to the Valora dashboards, Brazil has an extremely low verification rate. We cannot verify this against our own dashboards at the moment seemingly due to missing (or corrupted data) eksportisto 14 data.
TODOs for this ticket:

[x] identify what is going on with missing eksportisto data -- does migrating to the maintained eksportisto solve these issues? if so, prioritize this ticket
[ ] can we reproduce the numbers seen in Valora? if not, how large is the discrepancy and where does it come from?
[ ] come up with sensible next steps regarding BR attestation rates -- what further data explorations would be helpful? Can we come up with a list of questions/possible factors to look at and see if we find anything useful here?
[ ] make sure this investigation is well-documented so others can follow/understand stuff afterwards as well

Indeed looks like an issue with eksportisto 14 not receiving data properly. Initial work on migrating the attestation service views to the new, maintained eksportisto seems to resolve the issue at the very least. Prioritizing #9051 to fix step one.

Brazil rates investigation is documented here for future reference

Ticking off the todos in the description at the top

Added this ticket #9180 for future reference.
------Copy of ticket-------
Mila pointed out that one possible reason for our lower success rate in Brazil could be due to misrecognised phone numbers. According to Mila (wikipedia below), the number of digits in a phone number increased from 8 to 9 to expand the set of phone numbers.
From wikipedia:

Brazilian mobile phone numbers always have nine digits now, but seven digits were usual in the first years, then eight digits became the standard for several years

[ ] Check whether this could be a reason for lower success rates (relevant Slack thread)

Closing this ticket because followed up on Slack thread


DUPLO-26299 Build fix

Describe Changes
Build fix by removing any potential cyclic dependencies in docker-compose.yaml.
Link to Issues
DUPLO-26299
Related: docker/compose 12235

Task linked: DUPLO-26299 duploctl build issue


Questions about the new data types

Note: this is a copy of a forum reply , posted here so it can be addressed at some point.
I was looking at the docs for sn_data_types to see what might have changed and they seem different to the model presented in the OP. Immutable data is clear enough (PrivateBlob and PublicBlob) but looking at SeqMap and UnseqMap I'm not sure of their correspondence with the OP, or how data that is unpublished becomes published.
I speculate that an UnseqMap is always private (as for Mutable Data in the OP as it is not versioned).
I am though having trouble reconciling the other new types with the descriptions of the types in the OP, so maybe a new overview in a new topic is needed?
For example, the docs day that SeqMap is for unpublished data, unlike the AppendOnly Data of the OP which can be published or unpublished. So there seem to be differences and I can't see how things become published in the API (but maybe this is still to be implemented).
I'm interested because I'm wondering how to implement a publicly writeable data structure to handle submission of issues by non-owners to a decentralised bug tracker (cf. git-bug). I'm looking for something like a public noticeboard or inbox, which can be polled for new issues by any client checking the status of a remote code repository.
One of the questions I was trying to answer is whether I can check the creator of each entry in such a publicly writeable "append only" style data structure? This would help discourage spam, by making it easier for a client to skip over entries created by identities from a blacklist.

@happybeing I'm closing this as it's been unactioned for a long time and this repo is being archived - can be recreated in safe_network if still relevant


[no-duplicates] invalid fix when there are 3 imports with named imports and dangling commas

Input:
import {A1,} from 'foo';
import {B1,} from 'foo';
import {C1,} from 'foo';

import {
  A2,
} from 'bar';
import {
  B2,
} from 'bar';
import {
  C2,
} from 'bar';

Expected:
Error and fixed to
import {A1,B1,C1,} from 'foo';
import {
  A2,
  B2,
  C2,
} from 'bar';

(or something similar)
Actual:
Error and fixed to
import {A1,B1,,C1,} from 'foo';
              ^ SYNTAX ERROR
import {
  A2,

  B2,
,
^ SYNTAX ERROR
  C2,
} from 'bar';

Interestingly this bug requires 3 (or more) duplicate declarations to trigger.
I haven't looked yet - but given it happens for 3 and not 2 - I'd assume that ESLint is merging the distinct fixes poorly - leading to the syntax error.

facing this bug too, can confirm it only occurs with 3 or more duplicate import paths

Tests pass on latest master with this snippet, so I'll close this shortly when I push those up.

@ljharb the ESLint rule tester might not pick this up? Depends how it composites fixes together I guess. Might be worth a quick whirl as a manual test against a file with the CLI to triple check.

I’ve not run into any cases like that with the rule tester; could you make a small repro repo?

Hmm - investigating more this appears to be working fine now yeah.
Some combination of the latest ESLint version and the latest plugin version appears to have fixed it.


Sanityze the base image name and remote the digest if any

Task name: docker-common
Description: This change is going to remove the digest when is included as part of the base image name.
Documentation changes required: N
Added unit tests: N
Attached related issue: https://github.com/microsoft/azure-pipelines-tasks/issues/15112

/azp run


Provide way to set breakpoint via GUI

Atom 1.3.2, dartlang 0.5.4, SDK 1.13.2 running on macos.

will be fixed by https://github.com/dart-atom/dartlang/pull/799


INT-1049 getId improvements

remove username/password from URL
remove connection options

@Machyne mind adding a few tests for these new cases you're handling?

Added some tests! :)

@Machyne thx! :shipit:


Internal file link opened in browser

Loving this plugin so far, incredibly powerful feature addition to Obsidian! When I click on a link that directs to a local file, the browser tries to open it as a web page. The internal file link is formatted as follows (without any of the spaces):
[ link_name ] ( < file:///C:path > )
It would be nice if the plugin could recognize this type of link and allow Obsidian to open the local file instead of searching it online.

Great! Interestingly when I run Obsidian in Linux this is not an issue, internal and external links are correctly distinguished. Not sure if relevant but thought I would mention it :)

On linux, do external links such as [Google](https://google.com) correctly open within the Obsidian browser?

Yeah, both internal and external links work on Linux it seems!

That's very interesting. I wouldn't expect that 🤔

This should be fixed thanks to #24. If it isn't, please let me know (a new beta will be out later today with this update).


[Feature]: Add Input Select for Source to Media card

Requirements

[X] I have updated Mushroom to the latest available version
[X] I did a search to see if there is a similar issue or if a pull request is open.

Is your feature request related to a problem?
Mini Media player does it right, Mushrrom lacks one key component ... selecting the source
Describe the solution you'd like
Now we have a mushroom select card (great!) how about adding similar functionality to the mushroom media card?
Describe alternatives you've considered
No response
Additional context
No response

See https://github.com/piitaya/lovelace-mushroom/discussions/966

I hope this can be added soon (given there is an open PR for it) 😄 For me it would be very useful.


[Bug]: Firmware updates time out if the target device is not directly connected to the controller

Checklist

[X]  I am not using Home Assistant. Or: a developer has told me to come here.
[X] I have checked the troubleshooting section and my problem is not described there.
[X] I have read the changelog and my problem is not mentioned there.

Deploy method
Docker
Zwavejs2Mqtt version
6.15.2
ZwaveJS version
9.6.2
Describe the bug
I've experienced an issue while attempting to update firmware on multiple Zooz ZEN77 (dimmer switches) and ZSE44 (temperature/humidity sensor) devices, but only when they are not connected directly to the controller (these devices are connected via a "1 hop" connection -- that is, there is one neighbor node between the device and the controller).
The devices start a firmware update as would be expected, albeit slower than you might expect, but eventually, the firmware update times out without an error being presented in the UI, and the "FW" column within the Z-Wave JS list display reverts from a progress bar to the FW/SDK versions without any error showing either in the UI or in the logs. The only "error" I see is a "firmware update finished" event in the "events" tab of the device with a "-1" argument.
I've been experiencing this issue ever since I found new firmware updates for my devices on July 30. Over the past month, I've updated zwavejs2mqtt multiple times, healed my network to try and get the devices a better connection to the controller, and ultimately upgraded from a Aeotec Z-Stick Gen5+ to a Zooz 700-series Z-Wave USB stick, none of which helped fix the issue directly.
That being said, after upgrading to a Zooz 700-series Z-Wave USB stick from the Aeotec 500-series controller, I did notice that, after healing my network, many more of the devices on the network were able to communicate directly with the controller. When the devices in question were connected to my 500-series Z-Wave stick via a 1-hop connection, their firmware updates failed, but after upgrading to the Zooz 700-series controller and healing the network, all but 3 devices in my network were connected directly to the controller.
After this upgrade, every single one of the devices that previously had a firmware update fail on the 500-series controller (while connected via a "1 hop" connection) that were now directly connected to the controller were able to complete the firmware update expeditiously and without issue. However, the two Zooz devices (out of the three total nodes that are still connected on a "1 hop" connection) that are still not directly connected to the controller are still experiencing the same behavior, where the firmware update fails partway through.
For reference, I've carried out the same update procedures on every device (whether battery or mains-powered), and haven't seen any change in behavior unless the connection to the node swaps from a "1 hop" connection to a direct connection.
While attempting to export debug logs, I did not see any logs produced from the firmware update, but I did see "events" showing within the device in the UI pertaining to the firmware update. I've attached a snippet of the last few events when the firmware update was cancelled. For reference, node 11 is the one depicted in the image, but I've also experienced an issue with updating node 43, as it is also 1 hop away from the controller.
Node dump:
nodes_dump.zip
Node events:

Appreciate any assistance that you all able to provide! Please let me know if any more information would be helpful in debugging this issue. Thanks!
To Reproduce

Connect a device to the controller in a manner where it connects with a "1 hop" connection -- meaning one and only one neighbor between the device and the controller.
Attempt a firmware update on the device by selecting the "begin firmware update" option within the Z-Wave JS UI, and select a valid firmware file for the device.
The firmware update begins and the device starts reporting progress.
The firmware update cancels without error after progressing anywhere between 3-30% through the firmware update (in my experience).

Expected behavior
A device should successfully be able to complete a firmware update regardless of how it is connected to the controller.
Additional context
I've experienced this issue so far with Zooz devices -- I have other devices on my network, but none have needed firmware updates so far. I've corresponded with Zooz support over the past month in order to try and figure out this issue in the event that it was a hardware/firmware issue with the devices, but I have not yet been able to ascertain any other cause for failed firmware updates aside from the devices only failing to update whenever they are not directly connected to the controller.

@bradreardon This seems like you were having connectivity issues before changing sticks. Unlike most other communication, firmware updates are driven by the devices. After zwave-js initializes the upgrade, the devices request each fragment in sequence and zwave-js answers with the requested fragment.
When these requests stop for more than 30s, zwave-js considers the upgrade aborted/timed out. It seems like your devices were having trouble requesting the next fragment after a while (or the responses got lost).
We have a couple of write-ups related to detecting and fixing connectivity issues in the documentation. It might be worth working through those before attempting further firmware update troubleshooting:

Controller placement: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/connectivity-issues?id=general-troubleshooting
Making sure the network isn't flooded: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/network-health?id=optimizing-the-reporting-configuration
Measuring the connection strength: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/network-health?id=testing-the-connection-strength - you'll find the health check on the network map in zwavejs2mqtt when you click a node

Hey @AlCalzone, thanks for the information. That's helpful in understanding how these firmware updates might be failing. It sounds like I do have a bit more investigation to do on my end.
As for the issue of controller placement, I did try using a USB extension cord to connect my 500-series controller in the past, but strangely I saw worse connectivity with the extension cord in place. I'll give it another shot with the new controller and see if that helps reduce interference and get everything connected directly to the controller.
The note about network flooding is also helpful -- I do have a device that's sending power usage reports for 2 sockets on a 15-20 second interval currently, so I'll have to look into whether or not that node reporting power usage is correlated at all to the firmware updates failing.
Finally, for the network health suggestion -- I have tried using this tool before, but didn't know how to make sense of the information within it, so I appreciate you sending that page over.
I have a busy week so I'm not sure I'll be able to play around with this again until the weekend, but I'll be sure to report back with my findings. Thanks for the help!

Hey @AlCalzone, thanks for the information. That's helpful in understanding how these firmware updates might be failing. It sounds like I do have a bit more investigation to do on my end.
As for the issue of controller placement, I did try using a USB extension cord to connect my 500-series controller in the past, but strangely I saw worse connectivity with the extension cord in place. I'll give it another shot with the new controller and see if that helps reduce interference and get everything connected directly to the controller.
The note about network flooding is also helpful -- I do have a device that's sending power usage reports for 2 sockets on a 15-20 second interval currently, so I'll have to look into whether or not that node reporting power usage is correlated at all to the firmware updates failing.
Finally, for the network health suggestion -- I have tried using this tool before, but didn't know how to make sense of the information within it, so I appreciate you sending that page over.
I have a busy week so I'm not sure I'll be able to play around with this again until the weekend, but I'll be sure to report back with my findings. Thanks for the help!

Apologies: accidentally posted a duplicate response! Here's my resolution:
--
After doing some tests with my controller connected via a USB extension cord, I've confirmed that all nodes in my network are able to maintain a direct connection to the controller. I was able to complete firmware updates for the last two switches that I was experiencing issues with as a result.
Still quite interesting that the update fails for cases where the node is not directly connected! Thanks for the help in troubleshooting!

My guess is that it's caused by the interference from the USB, which in turn garbles up the radio data, and not because of a bad connection (between the nodes) per se.
I once tried to stack a PoE HAT with a Z-wave Razberry controller board on top, onto my Pi. It couldn't even communicate with a device that was just next to the controller, cause the signals got garbled.


use_state_machine should be static

Sorry if I am missunderstanding something here, but I feel like the function use_state_machine() should be static, as you use it to create the state machine in the first place. Of course it is possible to circumvent this, by directly calling StateMachine.create() directly.
Doing so without adjusting it, gives me an error:
"Cannot call non-static function "use_state_machine()" on the class "Wisp" directly. Make an instance instead."

#3

reopening this issue because it is relevant for godot 4, will add a godot 4 branch soon


Prevent Upload if Worksheet already exists

Prevents two runs from being unintentionally merged (e.g when worksheet gets repeated)

Just to add, only prevent upload when the run IDs are different - there are quite common use cases where there are samples uploaded and the worksheet already exists, e.g. new referral types, training cases


There is no SAX interface to incremental parsing of large files

A SAX-like interface to process large files is very important to enable get access to data that is not able to fit in memory.

Grouping related large file read issues into #61

A SAX-like interface to process large files is very important to enable get access to data that is not able to fit in memory.

hello,How do you deal with this problem？


'ToastStyle' cannot be constructed because it has no accessible initializers

'ToastStyle' cannot be constructed because it has no accessible initializers
Xcode 7.3

Same here
Screenshot:

Sorry about that guys. This is fixed in 1.3.0 and in master.


Update managed-dependencies.md

Removed bolt on hyperlink, it seemed more in line with other docs to not bolt this link.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Is Mendix employee. Merging.


typeahead: Redesign the function compare_people_for_relevance.

typeahead: Redesign the function compare_people_for_relevance.
We make the following changes:

Remove the dm partnership check before calling the comparator passed by the caller, and instead have the more thorough compare_by_pms be the final tie breaker. This deprioritization of dms fixes the bug where dm partners would be prioritized even if they had not participated in a stream conversation over recent senders who are not dm partners.

Make the passed comparator optional, since the tie breaker is same as the comparator that was passed so far for dms, and only pass it in case of a stream conversation.

tests: Add missing setup to typeahead_helper tests passing as a fluke.
typeahead: Add check for all time dm partners to compare_by_pms.
For the rare case where there is no recent recipient count for a pair of users when sorting, we now check if either has been a dm conversation partner ever before.
Fixes: Issue reported on CZO here
Screenshots and screen captures:
(conversation participants who are not dm partners at top of suggestions:)

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

This seems like a great checkpoint, merged, thanks for the thoughtful work on this @N-Shar-ma!


feat: add hatoo/oha

hatoo/oha: Ohayou(おはよう), HTTP load generator, inspired by rakyll/hey with tui animation
$ aqua g -i hatoo/oha

How to confirm if this package works well
Reviewers aren't necessarily familiar with this package, so please describe how to confirm if this package works well.
Please confirm if this package works well yourself as much as possible.
Command and output
$ oha --version
oha 0.5.8

Thanks!

From the next time, please use --deep option.
$ aqua-registry scaffold --deep hatoo/oha

Hmm. It doesn't work in my laptop (darwin/arm64)...
$ aqua which oha
/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/github_release/github.com/hatoo/oha/v0.5.8/oha-macos-arm64/oha-macos-arm64

$ oha --help
zsh: killed     oha --help

The prebuilt binary for darwin/amd64 works by Rosetta2.
$ git diff pkgs
diff --git a/pkgs/hatoo/oha/registry.yaml b/pkgs/hatoo/oha/registry.yaml
index 78f0c75ad..32719737c 100644
--- a/pkgs/hatoo/oha/registry.yaml
+++ b/pkgs/hatoo/oha/registry.yaml
@@ -4,6 +4,7 @@ packages:
     repo_name: oha
     description: Ohayou(おはよう), HTTP load generator, inspired by rakyll/hey with tui animation
     asset: oha-{{.OS}}-{{.Arch}}
+    rosetta2: true
     format: raw
     replacements:
       darwin: macos

$ oha --help
INFO[0000] download and unarchive the package            aqua_version=2.5.0 env=darwin/arm64 exe_name=oha package=hatoo/oha package_name=hatoo/oha package_version=v0.5.8 program=aqua registry=standard
Ohayou(おはよう), HTTP load generator, inspired by rakyll/hey with tui animation.

Usage: oha [FLAGS] [OPTIONS] <url>

Arguments:
  <URL>  Target URL.

Options:
  -n <N_REQUESTS>                    Number of requests to run. [default: 200]
  -c <N_WORKERS>                     Number of workers to run concurrently. You may should increase limit to number of open files for larger `-c`. [default: 50]
  -z <DURATION>                      Duration of application to send requests. If duration is specified, n is ignored.
                                     Examples: -z 10s -z 3m.
  -q <QUERY_PER_SECOND>              Rate limit for all, in queries per second (QPS)
      --latency-correction           Correct latency to avoid coordinated omission problem. It's ignored if -q is not set.
      --no-tui                       No realtime tui
  -j, --json                         Print results as JSON
      --fps <FPS>                    Frame per second for tui. [default: 16]
  -m, --method <METHOD>              HTTP method [default: GET]
  -H <HEADERS>                       Custom HTTP header. Examples: -H "foo: bar"
  -t <TIMEOUT>                       Timeout for each request. Default to infinite.
  -A <ACCEPT_HEADER>                 HTTP Accept Header.
  -d <BODY_STRING>                   HTTP request body.
  -D <BODY_PATH>                     HTTP request body from file.
  -T <CONTENT_TYPE>                  Content-Type.
  -a <BASIC_AUTH>                    Basic authentication, username:password
      --http-version <HTTP_VERSION>  HTTP version. Available values 0.9, 1.0, 1.1, 2.
      --host <HOST>                  HTTP Host header
      --disable-compression          Disable compression.
  -r, --redirect <REDIRECT>          Limit for number of Redirect. Set 0 for no redirection. [default: 10]
      --disable-keepalive            Disable keep-alive, prevents re-use of TCP connections between different HTTP requests.
      --ipv6                         Lookup only ipv6.
      --ipv4                         Lookup only ipv4.
      --insecure                     Accept invalid certs.
      --connect-to <CONNECT_TO>      Override DNS resolution and default port numbers with strings like 'example.org:443:localhost:8443'
      --disable-color                Disable the color scheme.
      --unix-socket <UNIX_SOCKET>    Connect to a unix socket instead of the domain in the URL. Only for non-HTTPS URLs.
  -h, --help                         Print help
  -V, --version                      Print version

https://github.com/aquaproj/aqua-registry/pull/11997

Released. https://github.com/aquaproj/aqua-registry/releases/tag/v4.1.0

From the next time, please use --deep option.
$ aqua-registry scaffold --deep hatoo/oha

Thanks for your advice. OK.


Fix #1728, -Ddubbo.service.delay not work

What is the purpose of the change
Fixed issue #1728.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Also pls to fix the unit test.
Your pr may have a problem.
If you directly override the delay here, the delay attribute in the XML file will never take effect.

That is the reason that why is the ut failed.
:)

I think we need to refactor this to fix the bug, instead of adding getGlobalServiceDelay. It's not the best way.


Remove input and output files after zipping

My attempt at solving issue #2.

Hi! Thanks for PR.
Make sure you have checked points in PR Template correctly.

Which template?

Oh, Sorry haven't added PR Templates yet 😓

Oh, Sorry haven't added PR Templates yet 😓

Give new line at the end of everyfile and also try this on local and share a screenshot

Feel free to star this repo and also respond Have you checked it locally?

Yes, l just need to add the newlines and take the screenshots.

I just added the newlines. Here's a gif of running the Python generator, it also works with the others.


Disable escape

I want to use html entities inside text node:
{
    "type": "text",
    "value": "↑"
}

But remark.stringify() repalces & to &. How I can disable escape?

You cannot, only decoded entities (↑) can occur in the AST!


Scope Request in websocket

I want to use this to build multi-tenant application with web socket that is original to use the Scope. Request for verifying the user in a socket on connection, but how I can get the request for module
async handleConnection(socket: Socket) {
const contextId = ContextIdFactory.getByRequest(???);
  const userService = await this.moduleRef.resolve(
    UsersService,
    contextId,
    { strict: false },
  );

}
My users service
@Injectable({ scope: Scope.REQUEST })
export class UsersService {
}
Again, great library~

@chungchi300 I am sorry at this time, you cant use websocket with apollo federated services. However I am rewriting the services to use GRPC internally and just have one graphql api. It will be out soon. It comes with support for Consul and consul managed configurations

@chungchi300 it is now possible with the next ntanch


Suggestion: Reference the model "Prusa MINI Silicone Bed Leveling Mod - bed tightening jig" to help adjustments

The model is on printables here: https://www.printables.com/model/43629-prusa-mini-silicone-bed-leveling-mod-bed-tightenin
Without it, I don't think I would have guessed how much to turn each adjustment very accurately - so it is definitely a help in my view.
I would suggest adding a link to it in the Requirements section - as an optional tool.

Feel free to submit a PR with the change :)

This is the very first time I have done a Pull Request on Github - I hope it's done the right way.

Perfect! Thank you.


Add Simplified Chinese translation

Hi Joe,
I tried my best to update the translation based on my daily use of jwm. I think it's time to submit to to upstream.
Thanks.

Thanks!


file formats for digitization points in BIDS-MEG specification

There are two issues regarding file formats in the BIDS-MEG specification:

Currently, BIDS-MEG only "recommends" certain file formats in Appendix VI. This does not necessarily restrict the file formats in the sense that it is not a "must" or a "should". Should this be turned into a "must" to be consistent with validator?
The headshape file formats are supposed to be in the "specific format of the 3-D digitizer’s manufacturer", but this information is completely missing from Appendix VI. What are the file formats that we should support here? One suggestion is to start from the list here

Relevant discussion thread: https://github.com/bids-standard/bids-validator/pull/585
cc @chrisfilo @teonbrooks @sappelhoff @robertoostenveld @monkeyman192
Thoughts welcome!

+1 for specifying a limited number of well-described file formats. Allowing any file format makes it impossible to write software compatible with BIDS inputs.

+1 for specifying a limited number of well-described file formats.

+1 but we need input from many people to get a representative list of well-described formats. I have questions about such formats out there on several google doc comments but so far - no responses :-)

[no solution here, just background info]
Many labs are combining the primary measurement device (the MEG system) with secondary measurement devices (such as the 3D digitizer) from another company. The number of hardware companies is still relatively limited, but the api of the (OEM) hardware is (reasonably) open, resulting in other companies making products that include hard+software, and hence a file format. Most formats I am aware of are ascii with some form of tabular content, but that is not sufficiently restrictive to implement a parser.
The issue not only applies to MEG headshapes, but also to EEG electrode positions.

Maybe it will be easier to come to a conclusion if we split the issue into
two: raw data and headshapes.
On Tue, Oct 2, 2018, 4:02 AM Robert Oostenveld<EMAIL_ADDRESS>wrote:

[no solution here, just background info]
Many labs are combining the primary measurement device (the MEG system)
with secondary measurement devices (such as the 3D digitizer) from another
company. The number of hardware companies is still relatively limited, but
the api of the (OEM) hardware is (reasonably) open, resulting in other
companies making products that include hard+software, and hence a file
format. Most formats I am aware of are ascii with some form of tabular
content, but that is not sufficiently restrictive to implement a parser.
The issue not only applies to MEG headshapes, but also to EEG electrode
positions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bids-standard/bids-specification/issues/20#issuecomment-426233329,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp0fClZfuG_AZCN_MmCbeMi6vx8t2ks5ug0fhgaJpZM4XDU2I
.

I agree. I think we can have a stronger assertion for the data format, and we should likely recommend the headshape.

So, is the consensus to have one format or many but limited? In either case, this would entail conversion on the end of the user -- potentially harmonizing the coordinate system and the units. I do not know of any tools that can currently convert from one headshape format to the other (unless @teonbrooks already knows something). That is not to say it cannot be created :)

I would say that headshapes recorded with the original MEG system manufacturer's software (i.e. limited set) should be allowed, whereas others (i.e. potentially unlimited set) should not be allowed. I only know the case for Elekta (stored in the fif file) and CTF (stored in a *.pos file).
I checked the present bids-examples and all seem to comply, except for https://github.com/bids-standard/bids-examples/blob/master/ds000117/sub-01/ses-meg/meg/sub-01_ses-meg_headshape.pos which is a pos file that accompanies an elekta fif dataset. It is not in CTF pos format, so I don't know which software created the file.

I would say that headshapes recorded with the original MEG system manufacturer's software (i.e. limited set) should be allowed, whereas others (i.e. potentially unlimited set) should not be allowed.

+1 ... are the MEG system manufacturers' file formats well enough reported, so that other file formats can be reformatted to comply with these new BIDS requirements? If not, can we somehow put documentations out there?
On the same note but regarding EEG:
And then we do not accept headshape files for EEG? Currently, they are still in our BEP, see here.
for iEEG, headshape files will not be included, as @DoraHermes told me.

okay, so here is what I found on carefully reading the spec + associated links. Does this align with what people expect (feel free to edit or comment)?
To recall, the original reason we had this issue was that @monkeyman192 found .elp and .hsp files for KIT data. Is this only an issue with KIT systems or do we have such mismatches for other systems too? @teonbrooks you have some experience working with KIT systems. Could you comment?

For this issue, I think we should create a separate issue and pr for hardware data format and lead the discussion there. @jasmainak, I would suggest that this issue be renamed to digitization data.
for .elp and .hsp, these are not native KIT format files as @robertoostenveld mentioned here. It is a format that has been supported in tandem with KIT in mne-c.
KIT does not have a native format for headshape data; it requires third-party hardware vendors.
Oftentimes it relies on the Polhemus hardware for head scanning. For the KIT systems at NYU, the protocol is to export the data from Polhemus as txt files: one with the headshape points in the native device space (in MNE, it's referred to headshape/hsp), and also the coregistration electrode positions in native device space (in MNE, electrode/elp).
In the MNE pipeline, we have a head-centric analysis flow: the head space is the common space that transforms at on. that is, we generate a dev_head_t to take our MEG sensor space from their native space to the participant's head space. This requires the head position indicator (hpi) points generated from the MEG device in the device space and the corresponding electrodes (elp) from the scanner in the native head. The transformation is generated from there.
For source transformation, we do a head_mri coregistration transformation. This requires the fiducials (fid) that are marked when scanning the head in the native head space and fiducials marked on the MRI. we sometimes use the headspace points here as well to adjust the fit of the coregistration transformation.
for the MNE pipeline to work from sensor to source space on non-neuromag (FIF) systems, we need to include external files for the following:
native head space points: the headshape points, the electrode points, the fiducials points
native device points: the hpi points
native MRI points: the fiducials
We need to decide how to best support pipelines where the data files are not bundled as a monolithic file. this is tough because this requires knowing 1) if the point files is in its native space or another space 2) what unit is in (mm vs m). I think this could be handled in a sidecar file but we need some limit on the file formats themselves. are they all tsv or csv? we can't support every niche format.

For this issue, I think we should create a separate issue and pr for hardware data format and lead the discussion there. @jasmainak, I would suggest that this issue be renamed to digitization data.

okay done

We need to decide how to best support pipelines where the data files are not bundled as a monolithic file. this is tough because this requires knowing 1) if the point files is in its native space or another space 2) what unit is in (mm vs m). I think this could be handled in a sidecar file but we need some limit on the file formats themselves.

@teonbrooks this seems to be already specified by [_coordsystems.json] in the current specification.

@robertoostenveld I don't know what generated the .pos from Wakeman/Henson Elekta dataset. I asked them.
On the Brainstorm side, we use the format that the Polhemus software used to generate, with the same .pos extension. Example: https://github.com/bids-standard/bids-examples/blob/master/ds000246/sub-0001/meg/sub-0001_headshape.pos
This file format allows the distinction between named digitized points (electrodes), unnamed digitized points (head shape) and other typical landmarks (anatomical landmarks, HPI coils, with multiple repetitions for higher precision):
This is the file format used by the Brainstorm digitizer (used by some centers as a standalone Polhemus driver - no need to use Brainstorm for the analysis):
https://neuroimage.usc.edu/brainstorm/Tutorials/TutDigitize
On a standard CTF study (at the MNI and possibly in some other places), these .pos files are placed in the .ds folder and read as part of the CTF dataset. They include both the anatomical landmarks (Nasion,Left,Right) and the head positioning coils (NAS,LPA,RPA) so that we can mark the anatomical landmarks instead of the coils in the MRI for the registration.

About the Wakeman/Henson dataset (https://github.com/bids-standard/bids-examples/blob/master/ds000117/sub-01/ses-meg/meg/sub-01_ses-meg_headshape.pos), here is Rik's answer:

It's a long time ago, but I'm fairly sure I read the Neuromag *.fif file into Matlab (using SPM/fieldtrip) and the head-position points were automatically read from the header - so I just saved them as text files from Matlab. (They were originally created by a Polhemus digitizer, which is linked to the Neuromag acquisition software, so they end up in the *.fif file of each recording).

The .pos files from ds000117 are coming from in-house code, not from a proper academic or commercial software. Therefore, you can ignore the file extension.

@robertoostenveld @ftadel what would be great is to have a table which specifies for each manufacturer what is the preferred file format for storing the digitized points (named and unnamed) along with a link to the specification of this file format.

MEG-specific:

Brainstorm digitizer / CTF-Polhemus digitizer (can be used independently from a CTF MEG system):
.pos extension, possibly within the CTF .ds folder
tab-separated file
1 line header with a number I never knew what it was used for + N lines with the following format
[\t] [] \t   \t   \t
Example: CTF-Brainstorm.pos.txt
KRISS MEG Digitizer.txt: Digitizer.txt
Elekta/Neuromag: The positions are embedded in the binary .fif file, no standard format for ascii exports
4D: The positions are embedded in binary files, no standard format for ascii exports
Yokogawa/Ricoh: The positions are embedded in the binary .con/.ave/.sqd files, no standard format for ascii exports

General purpose digitizers than may be used with MEG:

ANT Xensor .elc: XensorTest.elc.txt
ANT EETrak .elc: eetrak124.elc.txt
EGI/BESA .sfp: GSN256.sfp.txt
Polhemus .elp: electrocap72.elp.txt
Zebris .sfp: Zebris.sfp.txt

There are other formats used by other popular EEG manufacturers, but maybe we don't want to discuss this here.

FWIW, I dislike seeing a loose ".txt" in the filenames with data.
And it feels from the previous description that .txt to examples was added solely to conform the IMHO "hole" in BIDS allowing for an arbitrary _digitizer.txt file?  or is that how original software stores it?
I would have slided with @chrisgorgo from 2 years above and restrict to a set of formats/extensions here or even better go for just one.  What is the benefit of a standard if there is no standard form(at)? IMHO ideally one, most expressive/supported in tools format should be chosen and then raw files converted into it (like is done e.g. for neuroimaging data itself - from .par/.rec or DICOMs or whatever -> .nii.gz). Support tools could convert into other formats if so desired or better tools just start support it directly.


Update the agent HBase plugin to support HBase Client 2.x

Update the agent HBase plugin to support HBase Client 2.x

[ ] Add a test case for the new plugin, refer to the doc

[x] Add a component id in the component-libraries.yml

[x] Add a logo in the UI repo

[x] Relate the issue #6572

[x] Update the CHANGES log.

In HBase Client 2.x, the HTable constructor function has a little difference after 2.2.0.
So the agent plugin should have 3 ConstructorInterceptPoint, I have tested it all in test/plugin/scenarios/hbase-scenario, just change the dependency version.
The e2e test cases haven't committed, I will commit it later. Please take a look first, thanks.

Please don't close and reopen, You are making all CI process rerun.

@wu-sheng
After reading these docs as follows:

Contributing Guides
Plugin Development Guide
Apache SkyWalking Agent Test Tool Suite

You said that update the e2e test, does it meaning I just need to update the test/plugin/scenarios/hbase-scenario?
And consider the version, I could remove this module, and add 3 new scenario:

hbase-1.x-scenario (just rename hbase-scenario)
hbase-2.0.x-scenario
hbase-2.2.x-scenario

You said that update the e2e test, does it meaning I just need to update the test/plugin/scenarios/hbase-scenario?
And consider the version, I could remove this module, and add 3 new scenario:

It depends on which way you prefer? Does the client API change much? Which makes you have to separate the test cases?

Does the client API change much?

NO, but the HTable constructor function change in different versions.

Which makes you have to separate the test cases?

According to the difference constructor, we need different versions of maven dependency, or we will miss some scenarios.

NO, but the HTable constructor function change in different versions.

So, you can't use the same test codebase for different versions, right? If so, feel free to separate the cases.

you can't use the same test codebase for different versions, right?

The test code will be the same or similar, just the maven dependency version is different. We need to cover the HBase agent plugin for the different logic. In my opinion, it still should separate the cases.

you can't use the same test codebase for different versions, right?

The test code will be the same or similar, just the maven dependency version is different. We need to cover the HBase agent plugin for the different logic. In my opinion, it still should separate the cases.

Then you could add versions into the here, https://github.com/apache/skywalking/blob/master/test/plugin/scenarios/hbase-scenario/support-version.list. The test framework will cover the versions.

We need to be clear about the naming rules.

Supported-list.md should be updated
Plugin e2e tests should be updated.

I had added the comment about naming rules and update the support-version.list.
I had already tested the plugin in my local environment, and it works fine.
@wu-sheng PTAL.

Supported-list.md should be updated

This missed.