[DRAFT] Remove the custom padding in T4Rec's MerlinDataLoader

Goals :soccer:
The Merlin data loader has a Padding operator that we can pass via the transforms parameter. To avoid code redundancy, I created a PR in T4Rec that removes custom padding from the T4Rec's MerlinDataLoader class.
In this PR, the default behavior of the Trainer is to set the data loader without padding. If the user wants to perform padding outside of the model, they can explicitly provide a data loader with a Padding transform.
Implementation Details :construction:

Removes the sparse_max logic from MerlinDataLoader
Extend MerlinDataLoader to accept the transforms parameter where the user can pass the Padding op.

Testing Details :mag:

Add a unit test to check that we can provide a data loader with Padding op to the Trainer class

Closing this as a solution in favor of updating the T4Rec padding code is implemented in #690


Clickhouse build configuration

Hi there.
After updating my local repository with the current master state, I cannot build Clickhouse due to various errors similar to "sort is not a member of llvm". The errors vary depending on what version of compiler I use. I suppose my tools versions (gcc, llvm) do not match to the current codebase. Can anyone post here or update documentation regarding the versions of the compilers that are supported?
Also building with a docker does not work for me: run-with-docker.sh ubuntu:bionic jobs/quick-build/run.sh produces fatal error openssl/err.h file not found.
Thanks,
Maxim

Can you paste all commands with all flags (cmake .....) and all console output with errors?

The try to build on a local machine:
cd ClickHouse
mkdir build
cd build
cmake ..
ninja
produces this:
[248/1326] Building CXX object dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/driver.cpp.o
FAILED: /usr/bin/g++-7   -DUNALIGNED_OK -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -isystem /usr/lib/llvm-6.0/include -Icontrib/zlib-ng -I../contrib/zlib-ng -fdiagnostics-color=always -std=c++1z  -msse4.1 -msse4.2 -mpopcnt  -fno-omit-frame-pointer  -Wall  -Wnon-virtual-dtor -no-pie -Wextra -Werror -O2 -g -DNDEBUG -O3    -fno-tree-loop-distribute-patterns -Wno-error -Wno-unused-parameter -Wno-non-virtual-dtor -U_LIBCPP_DEBUG -fno-rtti -fno-exceptions -g0 -pthread -MMD -MT dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/driver.cpp.o -MF dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/driver.cpp.o.d -o dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/driver.cpp.o -c ../dbms/programs/clang/Compiler-6.0.0/driver.cpp
In file included from /usr/local/include/clang/Frontend/CompilerInvocation.h:22:0,
from ../dbms/programs/clang/Compiler-6.0.0/driver.cpp:22:
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h: In constructor ‘clang::AnalyzerOptions::AnalyzerOptions()’:
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:301:11: error: ‘sort’ is not a member of ‘llvm’
llvm::sort(AnalyzerConfigCmdFlags);
^~~~
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:301:11: note: suggested alternative:
In file included from /usr/include/c++/7/algorithm:62:0,
from /usr/include/llvm-6.0/llvm/Support/MathExtras.h:19,
from /usr/include/llvm-6.0/llvm/ADT/SmallVector.h:20,
from /usr/local/include/clang/Driver/Types.h:13,
from /usr/local/include/clang/Driver/Action.h:13,
from /usr/local/include/clang/Driver/Compilation.h:13,
from ../dbms/programs/clang/Compiler-6.0.0/driver.cpp:16:
/usr/include/c++/7/bits/stl_algo.h:4856:5: note:   ‘std::sort’
sort(_RandomAccessIterator __first, _RandomAccessIterator __last,
^~~~
[248/1326] Building CXX object dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/cc1_main.cpp.o
FAILED: /usr/bin/g++-7   -DUNALIGNED_OK -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -isystem /usr/lib/llvm-6.0/include -Icontrib/zlib-ng -I../contrib/zlib-ng -fdiagnostics-color=always -std=c++1z  -msse4.1 -msse4.2 -mpopcnt  -fno-omit-frame-pointer  -Wall  -Wnon-virtual-dtor -no-pie -Wextra -Werror -O2 -g -DNDEBUG -O3    -fno-tree-loop-distribute-patterns -Wno-error -Wno-unused-parameter -Wno-non-virtual-dtor -U_LIBCPP_DEBUG -fno-rtti -fno-exceptions -g0 -pthread -MMD -MT dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/cc1_main.cpp.o -MF dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/cc1_main.cpp.o.d -o dbms/programs/clang/Compiler-6.0.0/CMakeFiles/clickhouse-compiler-lib.dir/cc1_main.cpp.o -c ../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp
In file included from /usr/local/include/clang/Frontend/CompilerInvocation.h:22:0,
from /usr/local/include/clang/Frontend/CompilerInstance.h:15,
from ../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp:21:
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h: In constructor ‘clang::AnalyzerOptions::AnalyzerOptions()’:
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:301:11: error: ‘sort’ is not a member of ‘llvm’
llvm::sort(AnalyzerConfigCmdFlags);
^~~~
/usr/local/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h:301:11: note: suggested alternative:
In file included from /usr/include/c++/7/algorithm:62:0,
from /usr/include/llvm-6.0/llvm/Support/MathExtras.h:19,
from /usr/include/llvm-6.0/llvm/ADT/SmallVector.h:20,
from /usr/include/llvm-6.0/llvm/Option/Arg.h:18,
from ../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp:16:
/usr/include/c++/7/bits/stl_algo.h:4856:5: note:   ‘std::sort’
sort(_RandomAccessIterator __first, _RandomAccessIterator __last,
^~~~
../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp: In function ‘int cc1_main(llvm::ArrayRef<const char*>, const char*, void*)’:
../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp:237:5: error: ‘BuryPointer’ was not declared in this scope
BuryPointer(std::move(Clang));
^~~~~~~~~~~
../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp:237:5: note: suggested alternative:
In file included from /usr/local/include/clang/Frontend/CompilerInstance.h:24:0,
from ../dbms/programs/clang/Compiler-6.0.0/cc1_main.cpp:21:
/usr/local/include/llvm/Support/BuryPointer.h:23:28: note:   ‘llvm::BuryPointer’
template  void BuryPointer(std::unique_ptr Ptr) {
^~~~~~~~~~~
ninja: build stopped: subcommand failed.
Just in case: 'echo $CC, $CXX' produces gcc-7, g++-7
The try to use docker produces this:
cd ClickHouse/ci
./run-with-docker.sh ubuntu:bionic jobs/quick-build/run.sh
[...SKIP...]
[1794/2522] Building CXX object dbms/CMakeFiles/clickhouse_common_io.dir/src/Common/OpenSSLHelpers.cpp.o
FAILED: dbms/CMakeFiles/clickhouse_common_io.dir/src/Common/OpenSSLHelpers.cpp.o
/usr/bin/clang++-6.0  -DBOOST_SYSTEM_NO_DEPRECATED -DPOCO_STATIC -DPOCO_UNBUNDLED_ZLIB -DUNALIGNED_OK -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -I/ClickHouse/ci/workspace/sources/libs/libcommon/include -Ilibs/libcommon/include -isystem /ClickHouse/ci/workspace/sources/contrib/double-conversion -isystem /ClickHouse/ci/workspace/sources/contrib/brotli/c/include -isystem /ClickHouse/ci/workspace/sources/contrib/libhdfs3/include -isystem /ClickHouse/ci/workspace/sources/contrib/re2 -isystem /ClickHouse/ci/workspace/sources/contrib/poco/Data/include -I/ClickHouse/ci/workspace/sources/dbms/src -Idbms/src -isystem /ClickHouse/ci/workspace/sources/contrib/libpcg-random/include -isystem /ClickHouse/ci/workspace/sources/contrib/poco/Util/include -isystem /ClickHouse/ci/workspace/sources/contrib/poco/Foundation/include -I/ClickHouse/ci/workspace/sources/contrib/zlib-ng -Icontrib/zlib-ng -isystem /ClickHouse/ci/workspace/sources/contrib/poco/XML/include -isystem /ClickHouse/ci/workspace/sources/contrib/poco/JSON/include -I/ClickHouse/ci/workspace/sources/contrib/cityhash102/include -isystem /ClickHouse/ci/workspace/sources/contrib/boost -I/ClickHouse/ci/workspace/sources/libs/libwidechar_width -isystem /ClickHouse/ci/workspace/sources/contrib/poco/Net/include -isystem contrib/re2_st -I/ClickHouse/ci/workspace/sources/contrib/croaring -isystem /ClickHouse/ci/workspace/sources/contrib/libcpuid/include -I/ClickHouse/ci/workspace/sources/contrib/libgsasl -I/ClickHouse/ci/workspace/sources/contrib/libgsasl/gl -I/ClickHouse/ci/workspace/sources/contrib/libgsasl/src -I/ClickHouse/ci/workspace/sources/contrib/libgsasl/digest-md5 -I/ClickHouse/ci/workspace/sources/contrib/libgsasl/linux_x86_64/include -I/ClickHouse/ci/workspace/sources/contrib/libxml2-cmake/linux_x86_64/include -I/ClickHouse/ci/workspace/sources/contrib/libxml2/include -I/ClickHouse/ci/workspace/sources/contrib/protobuf/src -fdiagnostics-color=always  -msse4.1 -msse4.2 -mpopcnt  -fno-omit-frame-pointer  -Wall -Wno-unused-command-line-argument  -Wnon-virtual-dtor -stdlib=libc++ -Wextra -Werror -Wpedantic -Wno-vla-extension -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types -Wshadow -Wshadow-uncaptured-local -Wextra-semi -Wcomma -Winconsistent-missing-destructor-override -Wunused-exception-parameter -Wcovered-switch-default -Wold-style-cast -Wrange-loop-analysis -Wunused-member-function -Wunreachable-code -Wunreachable-code-return -Wnewline-eof -Wembedded-directive -Wgnu-case-range -Wunused-macros -Wconditional-uninitialized -Wdeprecated -Wundef -Wreserved-id-macro -Wredundant-parens -Wzero-as-null-pointer-constant -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-missing-noreturn -Wno-padded -Wno-switch-enum -Wno-shadow-field-in-constructor -Wno-deprecated-dynamic-exception-spec -Wno-float-equal -Wno-weak-vtables -Wno-shift-sign-overflow -Wno-sign-conversion -Wno-conversion -Wno-exit-time-destructors -Wno-undefined-func-template -Wno-documentation-unknown-command -Wno-missing-variable-declarations -Wno-unused-template -Wno-global-constructors -Wno-c99-extensions -Wno-missing-prototypes -Wno-weak-template-vtables -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types -Wno-double-promotion -Wno-disabled-macro-expansion -Wno-used-but-marked-unused -Wno-vla-extension -Wno-vla -Wno-packed -g -O0 -g3 -ggdb3 -fno-inline -g0 -D_LIBCPP_DEBUG=0   -pthread -std=c++1z -MD -MT dbms/CMakeFiles/clickhouse_common_io.dir/src/Common/OpenSSLHelpers.cpp.o -MF dbms/CMakeFiles/clickhouse_common_io.dir/src/Common/OpenSSLHelpers.cpp.o.d -o dbms/CMakeFiles/clickhouse_common_io.dir/src/Common/OpenSSLHelpers.cpp.o -c /ClickHouse/ci/workspace/sources/dbms/src/Common/OpenSSLHelpers.cpp
/ClickHouse/ci/workspace/sources/dbms/src/Common/OpenSSLHelpers.cpp:3:10: fatal error: 'openssl/err.h' file not found
#include <openssl/err.h>
^~~~~~~~~~~~~~~
1 error generated.

This is what cmake .. says:
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler using: Ninja
-- Check for working CXX compiler using: Ninja -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- IPO/LTO not enabled.
-- CMAKE_BUILD_TYPE is not set, set to default = RELWITHDEBINFO
-- CMAKE_BUILD_TYPE: RELWITHDEBINFO
-- Using linker: gold (selected from: LLD_PATH=/usr/local/bin/lld; GOLD_PATH=/usr/bin/gold; COMPILER_POSTFIX=-7)
-- Disabling compiler -pipe option (have only 6414 mb of memory)
-- Performing Test HAVE_SSE41
-- Performing Test HAVE_SSE41 - Success
-- Performing Test HAVE_SSE42
-- Performing Test HAVE_SSE42 - Success
-- Performing Test HAVE_SSSE3
-- Performing Test HAVE_SSSE3 - Success
-- Performing Test HAVE_AVX
-- Performing Test HAVE_AVX - Success
-- Performing Test HAVE_AVX2
-- Performing Test HAVE_AVX2 - Success
-- Performing Test HAVE_POPCNT
-- Performing Test HAVE_POPCNT - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Performing Test HAVE_NO_PIE
-- Performing Test HAVE_NO_PIE - Success
-- Tests are enabled
-- Building for: Linux-4.13.0-37-generic x86_64 x86_64-linux-gnu ; USE_STATIC_LIBRARIES=ON MAKE_STATIC_LIBRARIES=ON SPLIT_SHARED= UNBUNDLED=OFF CCACHE=CCACHE_FOUND-NOTFOUND
-- Using ssl=1: /home/bigdata/Projekte/ClickHouse/contrib/ssl/include : ssl;crypto
-- Found the following ICU libraries:
--   i18n (required)
--   uc (required)
--   data (required)
-- Found ICU: /usr/include/x86_64-linux-gnu (found version "55.1")
-- Using icu=1: /usr/include/x86_64-linux-gnu : /usr/lib/x86_64-linux-gnu/libicui18n.a;/usr/lib/x86_64-linux-gnu/libicuuc.a;/usr/lib/x86_64-linux-gnu/libicudata.a
-- Using Boost: /home/bigdata/Projekte/ClickHouse/contrib/boost : boost_program_options_internal,boost_system_internal,boost_filesystem_internal;boost_system_internal,boost_regex_internal
-- Using zlib-ng: /home/bigdata/Projekte/ClickHouse/contrib/zlib-ng;/home/bigdata/Projekte/ClickHouse/b2/contrib/zlib-ng : zlibstatic
-- Using zstd: /home/bigdata/Projekte/ClickHouse/contrib/zstd/lib : zstd
-- Using termcap: /usr/lib/x86_64-linux-gnu/libtermcap.a
-- Using odbc=1: /home/bigdata/Projekte/ClickHouse/contrib/unixodbc/include : unixodbc
-- Using Poco: /home/bigdata/Projekte/ClickHouse/contrib/poco/Foundation/include/;/home/bigdata/Projekte/ClickHouse/contrib/poco/Util/include/;/home/bigdata/Projekte/ClickHouse/contrib/zlib-ng/;/home/bigdata/Projekte/ClickHouse/b2/contrib/zlib-ng/ : PocoFoundation,PocoUtil,PocoNet,PocoNetSSL;ssl;crypto,PocoCrypto;ssl;crypto,PocoXML,PocoData,PocoDataODBC;unixodbc,,,PocoMongoDB; MongoDB=1, DataODBC=1, NetSSL=1
-- Using lz4: /home/bigdata/Projekte/ClickHouse/contrib/lz4/lib : lz4
-- Using xxhash=1: /home/bigdata/Projekte/ClickHouse/contrib/lz4/lib : lz4
-- Using sparsehash: /home/bigdata/Projekte/ClickHouse/contrib/libsparsehash
-- Using rt:
-- Using line editing libraries (readline): /usr/include : /usr/lib/x86_64-linux-gnu/libreadline.a;/usr/lib/x86_64-linux-gnu/libtermcap.a
-- Performing Test HAVE_READLINE_HISTORY
-- Performing Test HAVE_READLINE_HISTORY - Success
-- Using re2: /home/bigdata/Projekte/ClickHouse/contrib/re2 : re2;  : re2_st
-- Using libgsasl=1: /home/bigdata/Projekte/ClickHouse/contrib/libgsasl/src;/home/bigdata/Projekte/ClickHouse/contrib/libgsasl/linux_x86_64/include : libgsasl
-- Using librdkafka=1: /home/bigdata/Projekte/ClickHouse/contrib/librdkafka/src : rdkafka cppkafka
-- Build without capnp (support for Cap'n Proto format will be disabled)
CMake Warning at cmake/find_llvm.cmake:21 (find_package):
Could not find a configuration file for package "LLVM" that is compatible
with requested version "7".
The following configuration files were considered but not accepted:
/usr/local/lib/cmake/llvm/LLVMConfig.cmake, version: 9.0.0svn
/usr/lib/llvm-6.0/cmake/LLVMConfig.cmake, version: 6.0.0

Call Stack (most recent call first):
CMakeLists.txt:325 (include)
-- LLVM version: 6.0.0
-- LLVM include Directory: /usr/lib/llvm-6.0/include
-- LLVM library Directory: /usr/lib/llvm-6.0/lib
-- LLVM C++ compiler flags:
-- Using cpuid=1: /home/bigdata/Projekte/ClickHouse/contrib/libcpuid/include : cpuid
-- Using libxml2: /home/bigdata/Projekte/ClickHouse/contrib/libxml2/include;/home/bigdata/Projekte/ClickHouse/contrib/libxml2-cmake/linux_x86_64/include : libxml2
-- Using brotli=1: /home/bigdata/Projekte/ClickHouse/contrib/brotli/c/include : brotli
-- Using protobuf=1: /home/bigdata/Projekte/ClickHouse/contrib/protobuf/src : libprotobuf
-- Using pdqsort: /home/bigdata/Projekte/ClickHouse/contrib/pdqsort
-- Using hdfs3=1: /home/bigdata/Projekte/ClickHouse/contrib/libhdfs3/include : hdfs3
-- Using consistent-hashing:  : consistent-hashing
-- Using hyperscan=1: /home/bigdata/Projekte/ClickHouse/contrib/hyperscan/src : hs
-- Using simdjson=ON: simdjson
-- Using rapidjson=1: /home/bigdata/Projekte/ClickHouse/contrib/rapidjson/include
-- Using cityhash:  : cityhash
-- Using farmhash:  : farmhash
-- Using metrohash: /home/bigdata/Projekte/ClickHouse/contrib/libmetrohash/src : metrohash
-- Using btrie:  : btrie
-- Using double-conversion: /home/bigdata/Projekte/ClickHouse/contrib/double-conversion : double-conversion
-- Using snappy=1: /home/bigdata/Projekte/ClickHouse/contrib/snappy : snappy
-- Using Parquet: arrow_static: ; parquet_static: ; thrift_static
-- Using gtest=1: /home/bigdata/Projekte/ClickHouse/contrib/googletest/googletest : gtest_main;gtest :
-- Using jemalloc=1: /home/bigdata/Projekte/ClickHouse/contrib/jemalloc-cmake/include;/home/bigdata/Projekte/ClickHouse/contrib/jemalloc-cmake/include_linux_x86_64 : jemalloc
-- Using cctz: /home/bigdata/Projekte/ClickHouse/contrib/cctz/include : cctz
-- Using mysqlclient=1:  : mysqlclient; staticlib=
-- Using unwind=1: /home/bigdata/Projekte/ClickHouse/contrib/libunwind/include : unwind
-- compiler C   = /usr/bin/gcc-7  -fdiagnostics-color=always  -msse4.1 -msse4.2 -mpopcnt -fno-omit-frame-pointer  -Wall  -no-pie -O2 -g -DNDEBUG -O3
-- compiler CXX = /usr/bin/g++-7  -fdiagnostics-color=always -std=c++1z  -msse4.1 -msse4.2 -mpopcnt  -fno-omit-frame-pointer  -Wall  -Wnon-virtual-dtor -no-pie -O2 -g -DNDEBUG -O3
-- LINKER_FLAGS =  -fuse-ld=gold
-- ZSTD VERSION 1.3.17
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/gcc-7
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Architecture: x86_64
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for stdarg.h
-- Looking for stdarg.h - found
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
-- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
-- Performing Test HAVE_BUILTIN_CTZL
-- Performing Test HAVE_BUILTIN_CTZL - Success
-- Performing Test HAVE_SSE2_INTRIN
-- Performing Test HAVE_SSE2_INTRIN - Success
-- Performing Test HAVE_SSE42_INTRIN
-- Performing Test HAVE_SSE42_INTRIN - Success
-- Performing Test HAVE_SSE42CRC_INTRIN
-- Performing Test HAVE_SSE42CRC_INTRIN - Success
-- Performing Test HAVE_PCLMULQDQ_INTRIN
-- Performing Test HAVE_PCLMULQDQ_INTRIN - Success
-- Performing Test MFPU_NEON_AVAILABLE
-- Performing Test MFPU_NEON_AVAILABLE - Failed
-- Architecture-specific source files: arch/x86/x86.c;arch/x86/insert_string_sse.c;arch/x86/deflate_quick.c;arch/x86/fill_window_sse.c;arch/x86/crc_folding.c;arch/x86/crc_pclmulqdq.c
-- The following features have been enabled:

CMAKE_BUILD_TYPE , Build type: RELWITHDEBINFO (selected)
ZLIB_COMPAT , Provide a zlib-compatible API
WITH_GZFILEOP , Compile with support for gzFile-related functions
WITH_OPTIM , Build with optimisation
WITH_NEW_STRATEGIES , Use new strategies
SSE2 , Use the SSE2 instruction set, using "-msse2"
SSE4_CRC , Support CRC hash generation using the SSE4.2 instruction set, using "-msse4"
SSE4DEFLATE , Support SSE4.2-accelerated quick decompression
PCLMUL_CRC , Support CRC hash generation using PCLMULQDQ, using "-mpclmul"

-- The following OPTIONAL packages have been found:

Threads
ICU
LLVM (required version >= 6)

-- Looking for asprintf
-- Looking for asprintf - found
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Looking for strcasecmp
-- Looking for strcasecmp - found
-- Looking for strlcat
-- Looking for strlcat - not found
-- Looking for strlcpy
-- Looking for strlcpy - not found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for strsep
-- Looking for strsep - found
-- Looking for timegm
-- Looking for timegm - found
-- Looking for arc4random_buf
-- Looking for arc4random_buf - not found
-- Looking for arc4random_uniform
-- Looking for arc4random_uniform - not found
-- Looking for explicit_bzero
-- Looking for explicit_bzero - not found
-- Looking for getpagesize
-- Looking for getpagesize - found
-- Looking for timingsafe_bcmp
-- Looking for timingsafe_bcmp - not found
-- Looking for timingsafe_memcmp
-- Looking for timingsafe_memcmp - not found
-- Looking for memmem
-- Looking for memmem - found
-- Looking for include file err.h
-- Looking for include file err.h - found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Check size of time_t
-- Check size of time_t - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for byteswap.h
-- Looking for byteswap.h - found
-- Looking for sys/endian.h
-- Looking for sys/endian.h - not found
-- Looking for sys/mman.h
-- Looking for sys/mman.h - found
-- Looking for sys/resource.h
-- Looking for sys/resource.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/uio.h
-- Looking for sys/uio.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for windows.h
-- Looking for windows.h - not found
-- Looking for zlibVersion in z
-- Looking for zlibVersion in z - found
-- Looking for lzo1x_1_15_compress in lzo2
-- Looking for lzo1x_1_15_compress in lzo2 - not found
-- Performing Test HAVE_VISUAL_STUDIO_ARCH_AVX
-- Performing Test HAVE_VISUAL_STUDIO_ARCH_AVX - Failed
-- Performing Test HAVE_VISUAL_STUDIO_ARCH_AVX2
-- Performing Test HAVE_VISUAL_STUDIO_ARCH_AVX2 - Failed
-- Performing Test HAVE_CLANG_MAVX
-- Performing Test HAVE_CLANG_MAVX - Success
-- Performing Test HAVE_CLANG_MBMI2
-- Performing Test HAVE_CLANG_MBMI2 - Success
-- Performing Test HAVE_BUILTIN_EXPECT
-- Performing Test HAVE_BUILTIN_EXPECT - Success
-- Performing Test HAVE_BUILTIN_CTZ
-- Performing Test HAVE_BUILTIN_CTZ - Success
-- Performing Test SNAPPY_HAVE_SSSE3
-- Performing Test SNAPPY_HAVE_SSSE3 - Success
-- Performing Test SNAPPY_HAVE_BMI2
-- Performing Test SNAPPY_HAVE_BMI2 - Failed
-- Looking for mmap
-- Looking for mmap - found
-- Looking for sysconf
-- Looking for sysconf - found
-- Looking for arpa/inet.h
-- Looking for arpa/inet.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for getopt.h
-- Looking for getopt.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for netdb.h
-- Looking for netdb.h - found
-- Looking for netinet/in.h
-- Looking for netinet/in.h - found
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for sys/ioctl.h
-- Looking for sys/ioctl.h - found
-- Looking for sys/param.h
-- Looking for sys/param.h - found
-- Looking for sys/socket.h
-- Looking for sys/socket.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/un.h
-- Looking for sys/un.h - found
-- Looking for poll.h
-- Looking for poll.h - found
-- Looking for sys/poll.h
-- Looking for sys/poll.h - found
-- Looking for sys/select.h
-- Looking for sys/select.h - found
-- Looking for sched.h
-- Looking for sched.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for gethostbyname_r
-- Looking for gethostbyname_r - found
-- Looking for strerror_r
-- Looking for strerror_r - found
-- Looking for sched_get_priority_max
-- Looking for sched_get_priority_max - found
-- Looking for sched_get_priority_min
-- Looking for sched_get_priority_min - found
-- Performing Test STRERROR_R_CHAR_P
-- Performing Test STRERROR_R_CHAR_P - Success
-- Using OpenSSL from /home/bigdata/Projekte/ClickHouse/contrib/ssl
-- Looking for pthread_setaffinity_np in pthread
-- Looking for pthread_setaffinity_np in pthread - found
-- Performing Test HAVE_THREE_PARAM_SCHED_SETAFFINITY
-- Performing Test HAVE_THREE_PARAM_SCHED_SETAFFINITY - Success
-- Found ODBC: unixodbc
-- Found PythonInterp: /usr/bin/python (found version "2.7.12")
-- checking compiler: GCC (7.7.)
-- Looking for dladdr
-- Looking for dladdr - not found
-- Looking for nanosleep
-- Looking for nanosleep - found
-- Checking whether strerror_r returns an int
-- Checking whether strerror_r returns an int -- no
-- Performing Test ARCH_64_BIT
-- Performing Test ARCH_64_BIT - Success
-- Performing Test ARCH_32_BIT
-- Performing Test ARCH_32_BIT - Failed
-- Build type RELWITHDEBINFO
-- Boost version: 1.70.0
-- Build date: 2019-06-27
-- gcc version 7.4.0
-- g++ version 7.4.0
-- Looking for include file intrin.h
-- Looking for include file intrin.h - not found
-- Looking for C++ include intrin.h
-- Looking for C++ include intrin.h - not found
-- Looking for include file x86intrin.h
-- Looking for include file x86intrin.h - found
-- Looking for C++ include x86intrin.h
-- Looking for C++ include x86intrin.h - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for _aligned_malloc
-- Looking for _aligned_malloc - not found
-- Performing Test HAS_C_HIDDEN
-- Performing Test HAS_C_HIDDEN - Success
-- Performing Test HAS_CXX_HIDDEN
-- Performing Test HAS_CXX_HIDDEN - Success
-- Looking for _LIBCPP_VERSION
-- Looking for _LIBCPP_VERSION - not found
-- generator is Ninja
-- Performing Test HAS_C_ATTR_IFUNC
-- Performing Test HAS_C_ATTR_IFUNC - Success
-- Performing Test HAVE_AVX512
-- Performing Test HAVE_AVX512 - Failed
-- Building without AVX512 support
-- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED
-- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success
-- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED
-- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success
-- Performing Test HAVE__BUILTIN_CONSTANT_P
-- Performing Test HAVE__BUILTIN_CONSTANT_P - Success
-- Performing Test C_FLAG_Wvla
-- Performing Test C_FLAG_Wvla - Success
-- Performing Test C_FLAG_Wpointer_arith
-- Performing Test C_FLAG_Wpointer_arith - Success
-- Performing Test C_FLAG_Wstrict_prototypes
-- Performing Test C_FLAG_Wstrict_prototypes - Success
-- Performing Test C_FLAG_Wmissing_prototypes
-- Performing Test C_FLAG_Wmissing_prototypes - Success
-- Performing Test CXX_FLAG_Wvla
-- Performing Test CXX_FLAG_Wvla - Success
-- Performing Test CXX_FLAG_Wpointer_arith
-- Performing Test CXX_FLAG_Wpointer_arith - Success
-- Performing Test CC_SELF_ASSIGN
-- Performing Test CC_SELF_ASSIGN - Failed
-- Performing Test CXX_SELF_ASSIGN
-- Performing Test CXX_SELF_ASSIGN - Failed
-- Performing Test CC_PAREN_EQUALITY
-- Performing Test CC_PAREN_EQUALITY - Failed
-- Performing Test CXX_UNUSED_CONST_VAR
-- Performing Test CXX_UNUSED_CONST_VAR - Success
-- Performing Test CXX_IGNORED_ATTR
-- Performing Test CXX_IGNORED_ATTR - Success
-- Performing Test CXX_WEAK_VTABLES
-- Performing Test CXX_WEAK_VTABLES - Failed
-- Performing Test CXX_MISSING_DECLARATIONS
-- Performing Test CXX_MISSING_DECLARATIONS - Success
-- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS
-- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success
-- Performing Test CXX_WUNUSED_VARIABLE
-- Performing Test CXX_WUNUSED_VARIABLE - Success
-- Building for current host CPU:
-- Looking for mmap
-- Looking for mmap - found
-- Doxygen not found, unable to generate API reference
-- Sphinx not found, unable to generate developer reference
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'libpcre>=8.41'
-- PCRE version 8.41 or above not found
-- PCRE 8.41 or above not found
-- Could not find libpcap - some examples will not be built
-- Link jemalloc: jemalloc
-- /home/bigdata/Projekte/ClickHouse/utils: Have 6399 megabytes of memory. Limiting concurrent linkers jobs to 1 and compiler jobs to OFF
-- /home/bigdata/Projekte/ClickHouse/dbms: Have 6396 megabytes of memory. Limiting concurrent linkers jobs to 1 and compiler jobs to 4
-- Will build ClickHouse <IP_ADDRESS> revision 54422
-- Using gperf=1: /usr/bin/gperf
-- Using internal=OFF compiler=1: headers=/usr/share/clickhouse/headers/<IP_ADDRESS> root=/usr/share/clickhouse/headers/<IP_ADDRESS>:  /usr/local/bin/clickhouse-clang  -fdiagnostics-color=always -std=c++1z  -msse4.1 -msse4.2 -mpopcnt  -fno-omit-frame-pointer  -Wall  -Wnon-virtual-dtor  -Wextra -Werror -O2 -g -DNDEBUG -O3   -x c++ -march=native -fPIC -fvisibility=hidden -fno-implement-inlines -Wno-unused-command-line-argument -Bprefix=/usr/share/clickhouse -nostdinc -nostdinc++ -isysroot=/usr/share/clickhouse/headers/<IP_ADDRESS>; clickhouse-lld
-- Looking for readpassphrase
-- Looking for readpassphrase - not found
-- Using LLVM 6.0.0: /usr/lib/llvm-6.0/include : LLVMDemangle;LLVMSupport;LLVMTableGen;LLVMCore;LLVMFuzzMutate;LLVMIRReader;LLVMCodeGen;LLVMSelectionDAG;LLVMAsmPrinter;LLVMMIRParser;LLVMGlobalISel;LLVMBinaryFormat;LLVMBitReader;LLVMBitWriter;LLVMTransformUtils;LLVMInstrumentation;LLVMInstCombine;LLVMScalarOpts;LLVMipo;LLVMVectorize;LLVMObjCARCOpts;LLVMCoroutines;LLVMLinker;LLVMAnalysis;LLVMLTO;LLVMMC;LLVMMCParser;LLVMMCDisassembler;LLVMObject;LLVMObjectYAML;LLVMOption;LLVMDebugInfoDWARF;LLVMDebugInfoMSF;LLVMDebugInfoCodeView;LLVMDebugInfoPDB;LLVMSymbolize;LLVMExecutionEngine;LLVMInterpreter;LLVMMCJIT;LLVMOrcJIT;LLVMRuntimeDyld;LLVMTarget;LLVMAArch64CodeGen;LLVMAArch64Info;LLVMAArch64AsmParser;LLVMAArch64Disassembler;LLVMAArch64AsmPrinter;LLVMAArch64Desc;LLVMAArch64Utils;LLVMAMDGPUCodeGen;LLVMAMDGPUAsmParser;LLVMAMDGPUAsmPrinter;LLVMAMDGPUDisassembler;LLVMAMDGPUInfo;LLVMAMDGPUDesc;LLVMAMDGPUUtils;LLVMARMCodeGen;LLVMARMInfo;LLVMARMAsmParser;LLVMARMDisassembler;LLVMARMAsmPrinter;LLVMARMDesc;LLVMARMUtils;LLVMBPFCodeGen;LLVMBPFAsmParser;LLVMBPFDisassembler;LLVMBPFAsmPrinter;LLVMBPFInfo;LLVMBPFDesc;LLVMHexagonCodeGen;LLVMHexagonAsmParser;LLVMHexagonInfo;LLVMHexagonDesc;LLVMHexagonDisassembler;LLVMLanaiCodeGen;LLVMLanaiAsmParser;LLVMLanaiInfo;LLVMLanaiDesc;LLVMLanaiAsmPrinter;LLVMLanaiDisassembler;LLVMMipsCodeGen;LLVMMipsAsmPrinter;LLVMMipsDisassembler;LLVMMipsInfo;LLVMMipsDesc;LLVMMipsAsmParser;LLVMMSP430CodeGen;LLVMMSP430AsmPrinter;LLVMMSP430Info;LLVMMSP430Desc;LLVMNVPTXCodeGen;LLVMNVPTXInfo;LLVMNVPTXAsmPrinter;LLVMNVPTXDesc;LLVMPowerPCCodeGen;LLVMPowerPCAsmParser;LLVMPowerPCDisassembler;LLVMPowerPCAsmPrinter;LLVMPowerPCInfo;LLVMPowerPCDesc;LLVMSparcCodeGen;LLVMSparcInfo;LLVMSparcDesc;LLVMSparcAsmPrinter;LLVMSparcAsmParser;LLVMSparcDisassembler;LLVMSystemZCodeGen;LLVMSystemZAsmParser;LLVMSystemZDisassembler;LLVMSystemZAsmPrinter;LLVMSystemZInfo;LLVMSystemZDesc;LLVMX86CodeGen;LLVMX86AsmParser;LLVMX86Disassembler;LLVMX86AsmPrinter;LLVMX86Desc;LLVMX86Info;LLVMX86Utils;LLVMXCoreCodeGen;LLVMXCoreDisassembler;LLVMXCoreAsmPrinter;LLVMXCoreInfo;LLVMXCoreDesc;LLVMAsmParser;LLVMLineEditor;LLVMProfileData;LLVMCoverage;LLVMPasses;LLVMDlltoolDriver;LLVMLibDriver;LLVMXRay;LLVMWindowsManifest;/usr/lib/x86_64-linux-gnu/libtermcap.a;dl;zlibstatic
-- Target check already exists
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bigdata/Projekte/ClickHouse/b2

If you don't use the compile and compile_expressions settings, you can disable LLVM by adding
-D ENABLE_EMBEDDED_COMPILER=0 to your cmake invocation.

I can build now, thanks!

Some post-mortem writeup (HTH):
I also had this compile error with llvm::sort  etc., when trying to build IWYU (branch clang_8.0 , fe5275aa7b6a1372), on Ubuntu 18.04.3 LTS x64, with distro packages.
In my case it turned out that I had an inconsistent llvm config (IIRC, the resulting CMakeCache.txt listed LLVM_DIR with a 6 version, rather than the 8 which I had everything intended to be and which all other CMakeCache.txt entries referenced).
This was due to
following
https://github.com/include-what-you-use/include-what-you-use
in step-by-step mode, and erroneously hitting
OPTION 1:
For IWYU 0.10/Clang 6 and earlier
rather than
OPTION 2:
For IWYU 0.11/Clang 7 and later
, thus
setting
"-DIWYU_LLVM_ROOT_PATH=/usr/lib/llvm-6.0"
rather than
"-DCMAKE_PREFIX_PATH=/usr/lib/llvm-7" (using 8 obviously, in my case)
Once I followed their docs correctly (thus choosing option 2), CMakeCache.txt content ended up consistent,  and - drumroll - things built properly.
HTH!


fixes: warning MSB4011 Microsoft.NET.Sdk.WindowsDesktop.props cannot …

…be imported again.
What does the pull request do?

What is the current behavior?

What is the updated/expected behavior with this PR?

How was the solution implemented (if it's not obvious)?

Checklist

[ ] Added unit tests (if possible)?
[ ] Added XML documentation to any related classes?
[ ] Consider submitting a PR to https://github.com/AvaloniaUI/Documentation with user documentation

Breaking changes

Obsoletions / Deprecations

Fixed issues

Can we get some explanation of what this is doing? Links to documentation, etc?

We probably don't even need MSBuild.Sdk.Extras with .NET 6 SDK

Can we get some explanation of what this is doing? Links to documentation, etc?

I have no documentation, I was able to solve via the binlog.

We probably don't even need MSBuild.Sdk.Extras with .NET 6 SDK

probably. When I created this PR the .NET SDK it was 5.0

This seemed to have been force pushed to an older version and was showing up changes from unrelated PRs. I've updated the branch from master which means it's not not showing changes from other PRs, but I'm not 100% sure that the branch is in the state which @maxkatz6 approved it for.
@maxkatz6 does this still look right? I've removed your approval until you can check

@grokys yes, it looks look now


Facility for converting/importing Terraform created Azure resources (HCL) into Bicep declarations ?

Is your feature request related to a problem? Please describe.
We currently manage our Azure Resources entirely using Terraform. Terraform, as you know, manages state using it's own state files. Leveraging the native state management from ARM via Bicep is very compelling.
This means, however, we need to convert a massive amount of code currently written in Terraform HCL over to Bicep.
Describe the solution you'd like
Is there some facility that can allow us to convert our existing Terraform HCL into Bicep and thereby allow us an easy migration path ?

Definitely something we are interested in, but not something we are able to prioritize at the moment. It could be a great project for the community to tackle.
FWIW, it's likely a difficult problem since the HCL needed for an Azure resource will not match 1:1 with the equivalent in bicep. Property names are not always the same, etc. Like the decompiler we have to get you from ARM Template -> Bicep, it would have to be a best effort process.

Understood @alex-frankel . Thank you so much for the quick feedback.


Update adding-contacts.md

Description of the change:
Reason for the change:
Link to original source:

Closes #

Hello @sendgrid-ryan,Thanks again for the PR! It's HACKTOBERFEST! We want to show our appreciation by sending                                              you some special Hacktoberfest swag. If you have not already, could you please fill out                                              this form so we can send it to you? Thanks!                                             Team SendGrid DX

Hello @sendgrid-ryan,Thanks again for the PR!                                         We noticed that you have earned 2 out of the 5 points needed to receive glorious SendGrid Hacktoberfest swag.                                         Please take a moment to checkout this link to find more issues to get you past the required threshold.                                         Also, please be sure that you have officially registered here.                                         Thank you and Happy Hacktobering!                                         Team SendGrid DX

Hello @sendgrid-ryan,Thanks again for the PR!                                         You have earned 2 out of the 5 points needed to receive glorious SendGrid Hacktoberfest swag.                                         Please take a moment to checkout this link to find more issues to get you past the required threshold or to simply continue the celebration.                                         Also, please be sure you have officially registered with us here by November 1, 2018 to qualify.                                         If you have any questions you can email us at<EMAIL_ADDRESS>                                        Thank you and Happy Hacktobering!                                         Team SendGrid


issue with running files

Hello,
I am trying to use an Intel MacBook Air to run the python scripts.
I was wondering how you are supposed to download the files and then run them to light up the pixmob bracelet. I currently tried downloading the config.py file and the demo_single_effect.py files to a folder on my desktop. Are there others that I need to download? How do you run the files? Any help is appreciated.
Thanks in advance!

Hi,
Yes there is other files You  should see them in Imports at the begining of each python script,

pixmob_conversion_func.py
effect_definitions.py

Also make sure that the modules are installed too, like PySerial using PIP INSTALL
Be sure your directory structure is correct matching the repo.
You would probably be best cloning or downloading all the files from the Repo.
Check your Serial Port and Baud Rate to defined correctly in config.py
Hope this helps
Sean

Thank you. It says that it has successfully sent the code to the arduino. However, I am testing with a standard visible LED, and I am not getting it to light up. however, the 'L' LED on the arduino UNO circuit board is illuminating briefly when I run the demo code. Is there something that I should change in the arduino code? I already have it assigned to what pin the LED circuit is connected to. Thanks.

Hi,
It appears to be a PixMob X3, according to pictures on the website. It is from the Taylor swift 2023 concerts.
Is there anything else other than the data pin that is being used that needs to be entered into the PixMob_Transmitter_Arduino.ino file?
Thanks.

Hi again,
I guess it's tbis model you mean that Talor Swift uses.

It might be worth checking the batteries,  These don't hols charge very long,
With this model they made them NON-Recyclable by heat-staking them together,  but you can snip the snip the back of the melted plastic stakes and seperate it to test or  replace the batteries.  If cut flush it clips back together again.

With the Arduino script it is just the GPIO Pin that needs to be specified.
May I ask what type of Arduino you are using ?

Hello, Thank you for all of your help. I was able to get it to light up by setting WAIT_BEFORE_SEND = True  in Config.py.


Cross account ECS discovery

Greetings,
I have a question regarding cross account discovery of ECS tasks. I have two AWS accounts (A and B). I have the prometheus-ecs-discoverer running on a docker container on an EC2 instance in account A, which has a role with the required policy for it to work. I have setup a role in account B, with the appropriate policy, so that the EC2 instance in account A can assume it. I added another policy to account A's EC2 instance role so that it can assume the role I created in account B.  When I run the service on the container in account A, I can see that it picks up the ECS cluster tasks that are in account A, but it doesn't seem to be picking up the ones on account B. Is there any additional configuration steps required for the service in account A to be able to discover the ECS cluster tasks in account B?
The targets in both accounts are setup correctly to be discovered by the service. I ran the service independently on both accounts and I can see it scrape the targets within the respective account.
Thanks.

Hey @evelez-va, interesting use-case, I've have never considered it. I'd have to do a bit of research. No experience with working across multiple accounts.
You could check if https://github.com/signal-ai/prometheus-ecs-sd mentions something regarding cross account discovery. PromED is basically a "more refined" version of this project.
Let's say PromED is able to extract targets from other accounts into a file with respective IPs and ports for Prometheus, will Prometheus even be able to scrape the targets? The IPs would have to be public I assume. Or can you map subnets across accounts?

Hi @trallnag, thanks for the quick response. I'll take a look at that project to see if I find anything.
We will be setting up VPC peering between account A and B so that Prometheus in A is able to scrape the targets in B (along with the ones in A). In the meantime, I wanted to see if PromED on account A was at least able to discover the the ECS clusters on account B, even if Prometheus can't actually scrape them at the moment (have to wait for procedure for the peering).
We also considered deploying Prometheus along with PromED on account B like the recommendation, but we wanted to see if it was possible to do it this way and have one Prometheus to monitor our resources on both accounts. Might end up doing it that way if the current approach is not viable.

Okay so I have skipped through a few AWS docs and I don't see why it shouldn't work if the policies and roles are correctly set up. You could test it manually by deploying a task with a single Python container and the role attached you would also use for PromED and then try to use the AWS API like this:
import boto3

session = boto3.Session()
ecs = session.client("ecs")
arns = []
for page in ecs.get_paginator(method).paginate("list_clusters", "clusterArns"):
    arns += page.get(key, [])

Docs I have read:

https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-cross-account-lambda-auth-and-integration.html
https://docs.aws.amazon.com/AmazonECS/latest/userguide/security-iam.html

Another alternative is to just have PromED in the other account and only connect the Prometheus to the EFS with the file containing the targets.

Hello, I'm following up with what I found during testing. It seems that in order to be able to discover the ECS targets on another AWS account, the role that grants these permissions has to be explicitly assumed first. You're only able to discover targets within the boundaries of the role that is currently assumed by the EC2 instance.
In other words, if account A's EC2 role has a policy to assume a the role in account B that grants ECS access, it has to assume the cross account role first before being able to discover targets in account B. It would need an option that allows you to specify which role to use. It doesn't seem like it's possible to do both at the same time without constantly swapping roles (at least, not that I could see).
For now, I'm using this alternative solution which allows for specifying a role, and also custom docker labels (which we needed). I'm running two instances of the service: one in account A to discover local ECS targets, and another specifying it to assume the cross-account role to discover targets in account B. The downside is that this linked repo seems like it hasn't been updated in quite a while.

I see, very interesting. And nice that you have figured a solution that works for you. BTW the project you linked has been updated in June the last time, so not that old I'd argue ;D


(PDB-4730) Use Puppet 7 HTTP Client

CLA signed by all contributors.

I'm working on the puppetserver portion of this, and I think the changes here will fix some of the errors I'm seeing in the acceptance suites (notably https://github.com/puppetlabs/puppetdb/pull/3329/files#diff-b43043c3acd01a66d0886dd6f3b01d71R138). @austb are you aware of anything more that needs to happen with this?


Exclude org.codehaus.jackson

Exclude org.codehaus.jackson
This libraries are old (2013) and have plenty of CVEs. They were
migrated to org.fasterxml.jackson.

FYI @mneethiraj
See https://github.com/advisories/GHSA-c27h-mcmw-48hv and https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl

@mneethiraj why shim modules requires so many dependencies? I see that in hive plugin as well as in hbase. The more dependencies we put the higher risk for CVEs. How can I check what is actually used? I removed some of them and I see that project (module) compiles just fine, is this good enough for testing?

why shim modules requires so many dependencies

@kokosing  - you are right, shim modules only have dependency on libraries that include classes referenced in authorization interface (like HiveAuthorizer, HiveAuthorizerFactory), and ranger-plugin-classloader library.
About this patch, given org.codehaus.jackson libraries are not included in Hive plugin packaging, is it necessary to exclude them from pom.xml file?

About this patch, given org.codehaus.jackson libraries are not included in Hive plugin packaging, is it necessary to exclude them from pom.xml file?

I used mvn dependency:tree and I excluded it until the moment I no longer saw this dependency.

shim modules only have dependency on libraries that include classes referenced in authorization interface (like HiveAuthorizer, HiveAuthorizerFactory), and ranger-plugin-classloader library.

Is it safe to remove these dependencies and assume that if project compiles we are good?

Please rebase the PR to run all checks, 1 check is missing.

Done

@kokosing  - the patch is merged in master and ranger-2.5 branches. Thank you!

Thank you!


Use method namespace icrc1_ instead of icrc2_

I think all standards extending the ICRC-1 standard shall use the namespace icrc1_* for their methods.
Compare the following code snippet when multiple standards are implemented:
icrc2_transfer_from icrc3_get_transactions icrc4_batch_fransfer icrc1_get_metadata
vs
icrc1_transfer_from icrc1_get_transactions icrc1_batch_fransfer icrc1_get_metadata
I would say that the former is very confusing as the developer using the token needs to remember which methods belong to which ICRC number. Where in the latter case as long as the following ICRCs don't have overlapping names (which we can control as part of the ICRC process) it's very easy to use.

What about simply icrc_ prefix?
Going forward there may be hundred of ICRC standards and having them all point to ICRC1 seems wrong.


add url property

Would favor canonical URLs, but fall all the way back to window.location.href.

Added in 0.2.3


don't close publish modal automatically

currently if one of the target relays fail you get a nice modal with a list of all successes and failures, but if all relays succeed your screen blinks with an unreadable modal and you don't know to where the note was published.
with this even if all relays responded with an OK the modal will still remain open so the user can look at the list of relays the note was published to.

There was supposed to be a delay actually but i guess i forgot to add it


Issue80-include-new-wrappers-in-order-form-in-frontend

Extends order-form to include ts-sim, ts-modelprop and neptunus

@arnevogt : what will be the names of the parameters that ts-wrapper accepts?

lat (Latitude)
lon (Longitude)
mag (Magnitude)

https://riesgos.52north.org/wps?request=DescribeProcess&service=WPS&version=2.0.0&identifier=get_tsunamap


Add css hacks (sass)

Add some css hacks on 'src/_general.sass'. Just like clearfix.

Can't we just import the already existing hacks into it?

We can if they are under MIT license.

@svichas  add class for vertical align


add lastfm service

Creating the pr to track. Not ready to merge.

I'm having trouble getting the SearchForm to work. Seems like the state is always submitting: false. Any help @kudos ?

@matthewdias I'll try to get a look at this tonight, it's been a hectic week :)

@kudos have you had a chance to check this out yet? No worries if you're busy. Also to be clear, I haven't made any changes yet, just trying to get it to work locally.

@kudos any particular reason you closed this? honestly forgot about it, but id still like to build this out if you dont have any problems with having last.fm support.

@matthewdias opening a PR early is a good idea, but since it's been a year and there's nothing to give feedback on I figured it was abandoned. Let's open another PR when there's something more substantial?

sounds good 👍


Ensure all directories are kept under git

Add .gitkeep to empty dirs in cookiecutter

Closed by #75


Add comment sample

Solves #270
@rowan-m PTAL

Adding some comments here to just summarise some discussion.

If real comments are accepted, there's a lot of overhead. Input has to be sanitized, content has to be moderated, storage has to be maintained, etc. To me, the ROI in terms of the sample produced isn't enough to justify that. The backend code is not what people are really coming to look at.

One option would be to just have a static set of dummy comments and a form that when submitted shows another dummy comment being added.

The new comment only appears as the XHR response. Reloading the page goes back to to the static comments. The point of the sample is to demonstrate the potential UX.

Alternatively, the potential input could be greatly restricted. e.g. radio buttons / dropdown allowing the user to select from a few canned responses. These could be stored in the backend with a date and a random gravatar img or similar. Stored responses would just be FIFO limited to 10 or similar.

@rowan-m I tried the first suggested approach, PTAL.

@sebastianbenz @jmadler This is the comment sample in draft. You could add the login on top of that.

@rowan-m @sebastianbenz I added a commit with a login flow before being able to add a comment. The login page is a non AMP page as it's not required it to be an AMP page and can be the same page a publisher already has.

Some high-level comments:

the logout button leaves the preview page -> it should stay there
I find it confusing that it's not possible to add multiple comments. I assume this is because we don't want to implement the necessary server logic. Could we instead then hide the input field and comment button?
Aren't comments usually appended at the end?
s/Previous Comments/Comments/
Can you fake a normal comment field with author name and profile picture?
Explain your approach and how you're using the different components in the doc section.

@sebastianbenz I am trying to use server side templating to get the path of the sample (if it's either comment_section or comment_section/preview).
It works well for setting up the logout href, but it doesn't work for the authorization endpoint, where the string is inside a json.
This is what happens:
"authorization": "/samples_templates/comment_section/preview/authorization?rid=R....
where, server side, I am setting Path= "comment_section/preview" and in the template
"authorization": "/samples_templates/[[.Path]]/authorization?rid=READER_.....
As you can see, a backslash is added and the endpoint is not correct.
Have you got any idea how to solve this? I have been playing with it a bit with no luck; let me know if you have any idea, the latest code is committed.

Is it possible that the backslash is added to escape the trailing slash?

You can try if that's the cause by marking the path as HTML, e.g.  template.HTML(path)

@sebastianbenz tried template.html and template.URL, both same problem, look at here: https://play.golang.org/p/akuGK1VyVJ

Go's security model doesn't trust template parameters, you can either:

pass the whole string as a parameter: https://play.golang.org/p/9iTS5Ss6tc

or

pass the whole json object: https://play.golang.org/p/q6udq8MiZc

Thanks, I was trying to keep as much code as possible in the HTML so that the sample would be clearer, but if that is not possible, then I will pick one of the 2 approaches.

@sebastianbenz Addressed previous comments, PTAL

@sebastianbenz , answers belows:
Show comments to logged out users
I moved the comments on top, so that they are outside the form component and can be shown to the logged out users.
margin-top: 16px to
done
I'd prefer to have a single amp-access server endpoint which we can use across all samples
at the moment amp-access contains already some logic for the authorization endpoint related to the Hotel sample. The logic there is different from this sample. Me and @rowan-m already agreed that the Hotel sample could use the same code I am adding, but this will require some changes in the hotel sample code: for this reason this could be part of another PR.

"at the moment amp-access contains already some logic for the authorization endpoint related to the Hotel sample. The logic there is different from this sample. Me and @rowan-m already agreed that the Hotel sample could use the same code I am adding, but this will require some changes in the hotel sample code: for this reason this could be part of another PR."
A start would be to use the hotel sample and add an additional flag for your sample: "loggedId": true. This has the other advantage that you can simplify your amp-access contraints to:

...

I don't understand the suggestion. Do you mean removing span amp-access="user = 'test-user'" and using <span amp-access="loggedId"?

Yes. Testing for a specific user doesn't really make sense - comments should be open to everyone that is loggedIn.

@sebastianbenz I made the suggested change and now using

@sebastianbenz PTAL

@sebastianbenz PTAL.
I noticed that the after login-insert comment-logout-login the previous comment appears, while reloading the page will get the right behaviour.
The login.html is loading something like:
<input name="returnurl" type="hidden" value="https://cdn.ampproject.org/v0/amp-login-done-0.1.html?url=http%3A%2F%2Flocalhost%3A8080%2Fsamples_templates%2Fcomment_section%2Fpreview%2F">
will I need a way to flag that this should not be cached?

Applied changes, I noticed that there was some extra padding when a comment was added introduced by the form, so I added a css rule.

LGTM - great sample!


Use buildah commit and bud in podman

Vendor in buildah and use as much of commit and bud as possible for podman
build and commit.
Resolves #586
Signed-off-by: baude<EMAIL_ADDRESS>

LGTM

Can you grab the tests from buildah.bud to make sure they work.
bud.bats and all of the subdirs of bud.

you want to add bats tests back into libpod?

@baude or convert them to the new format.

Travis tests are failing with a seccomp error that I don't recall seeing before....
# github.com/projectatomic/libpod/vendor/github.com/seccomp/libseccomp-golang
vendor/github.com/seccomp/libseccomp-golang/seccomp_internal.go:25:2: error: #error Minimum supported version of Libseccomp is v2.2.0
 #error Minimum supported version of Libseccomp is v2.2.0
  ^
make: *** [podman] Error 2

Some failing commit tests

LGTM once tests are green

Need to manually build + install libseccomp >= 2.2.0 for travis

@rh-atomic-bot r+


Feat/dynamic relay configuration

📝 Summary

⛱ Motivation and Context

📚 References

✅ I have run these commands

[ ] make lint
[ ] make test-race
[ ] go mod tidy

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@ead84e5). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main       #1   +/-   ##
=======================================
  Coverage        ?   69.15%
=======================================
  Files           ?       17
  Lines           ?     1365
  Branches        ?        0
=======================================
  Hits            ?      944
  Misses          ?      386
  Partials        ?       35

Flag
Coverage Δ

unittests
69.15% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add a Task wrapper for MicroSoft OpenPAI

Description
Microsoft OpenPAI Job wrapper for Luigi.
Add luigi/contrib/pai.py
Add test/contrib/pai_test.py
Motivation and Context
Microsoft OpenPAI Job wrapper for Luigi.
"OpenPAI is an open source platform that provides complete AI model training and resource management capabilities,
it is easy to extend and supports on-premise, cloud and hybrid environments in various scale."
For more information about OpenPAI : https://github.com/Microsoft/pai/
Have you tested this? If so, how?
Tested in our local open pai cluster.

@dlstadther Thanks for your comments, see my response below:

The PaiJob/TaskRole/Port represent the OpenPAI Job structure, use slots to make sure no attributes other than predefined ones.  __slot_to_dict() used to json.dump as default function serialize the PaiJob object.
Done.
The camelCase was used in he PaiJob/TaskRole/Port to make sure the attribute name are same as OpenPAI job. Other than that, I use snake_case to follow python convention.
Added some checkings.
It should always have the same attributes when the response status_code are 200.

@dlstadther any new comments?

Sorry @liudongqing I'll try to re-review this this week. Thanks for following up!

@dlstadther Thanks for your comments, sorry for delayed response due to a long vacation.

@dlstadther , could you please help to follow? Thanks!

The unittests failed, looks like requests lib got an issued . I checked master branch, it failed as well.

@dlstadther passed UT by adding the requests>=2.20.0 per comment in the issue.

@dlstadther Thanks! Any more actions I need to take to merge that or just waiting someone to do it at convenience time?

@liudongqing I'm so sorry this fell off my radar! I noted this was ready to merge and i meant to do so. Could you update the merge conflict so that i can merge?

Thanks so much @liudongqing !


autogen: web session

update autogen to include web_session

https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-docs-superbuild-build/49/ contains the redis instructions

see https://github.com/ome/omeroweb-install/pull/53#issuecomment-564046477 -  I suppose this PR was used for that.
All went fine

It was. Thanks @pwalczysko


Changes requested to README

I fixed everything with those commits that were accidentally lumped into my previous PR, and those changes you requested for the README that be kept are in the commit below. Let me know if you want any changes made before it is merged. Thank you so much!

It looks like all the checks have passed, @acomodi. Let me know if you want any changes before you merge. Thanks!

@TheDHCreator looks good, just maybe leave out the python version in the example - python3.7 -> python3 should be fine.

@tmichalak Okay, I will make that change and push it right away.

That change has been made and pushed. Let me know @tmichalak if you would like any more changes before merging. Thanks again!

Thanks @TheDHCreator, LGTM. Merging.


fix: xreadgroup command keys parsing

@BridgeAR thank you for your help. I rewrite it and add some tests. I also add xread command features (basicly the same as xreadgroup)

@borsucok thanks a lot!


feat: jupyter-pytorch-full rock update

Summary of changes:

Added sercurity requirements.
Updated packages.

[I 2023-05-17 17:15:33.803 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-05-17 17:15:33.804 ServerApp] jupyterlab_git | extension was successfully linked.
[I 2023-05-17 17:15:33.806 ServerApp] nbclassic | extension was successfully linked.
[I 2023-05-17 17:15:33.806 ServerApp] nbdime | extension was successfully linked.
[I 2023-05-17 17:15:33.807 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-05-17 17:15:34.298 ServerApp] notebook_shim | extension was successfully linked.
[W 2023-05-17 17:15:34.335 ServerApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2023-05-17 17:15:34.343 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-05-17 17:15:34.343 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2023-05-17 17:15:34.344 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2023-05-17 17:15:34.344 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-05-17 17:15:34.346 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-05-17 17:15:34.347 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2023-05-17 17:15:34.351 ServerApp] nbclassic | extension was successfully loaded.

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2023-05-17 17:15:34.442 ServerApp] nbdime | extension was successfully loaded.
[C 2023-05-17 17:15:34.442 ServerApp] Running as root is not recommended. Use --allow-root to bypass.
2023-05-17T17:15:34.577Z [pebble] GET /v1/changes/1/wait 2.072504722s 200


Disallow emulator

Disallow emulator as the emu link links to a non-existent resource

Thanks! Yes, I think right now the emulator doesn't handle just widgets


Fix SVG sizing

Currently the SVG size is not updated in Firefox.
Also, the text of the first commit hash is truncated, regardless of which browser is used.
This PR fixes both issues.
Before

After

Thanks for the fix :)


Setting SOC before simulation

Summary
is there a convenient way to adjust SOC before the simulation?
Motivation
Setting SOC

Not yet, since this is dependent on how you define "SOC".
At the moment, you have to change the parameters "Initial concentration in negative electrode [mol.m-3]" and "Initial concentration in positive electrode [mol.m-3]"

thax for the info


Enhance survey filters in Django admin

Summary | Résumé
Trello
https://trello.com/c/eDGRMZDD/868-refine-filters-for-survey-send-feature

New filters on registrant including new columns
Default hardcoded 14 day filter

 Refine filters for survey send feature


[full dmmf] experiment with dmmf refactor and investigate what is required

Mostly done in 2.27.0.


Impossible to start writing after quote in WYSIWYG

[quote]text[/quote]

Insert the above BBC into the editor and toggle to WYSIWYG, it is impossible to place the cursor after the quote.
For the moment at elkarte.net I tweaked the code from:
content = '
' + sAuthor + ' ' + sDate + '' + content + '
';

to:
content = '
' + sAuthor + ' ' + sDate + '' + content + '

';

but this cases the adding of one empty line after each toggle if there is text after the quote.
If anyone has an idea on how to fix it "properly" it can be added to 1.1.4, otherwise I can commit the workaround and wait for the new version of SCEditor to be used in 2.0 and see if the problems is present.

Instead of the adding the
 ... try adding a


do u hv benchmark against redis and pedis?

do u hv benchmark against redis and pedis?

Not yet, the project is at the moment on hold but we'll work on benchmark when possible


M3-5973: Add Support for Linode and Backups available Field

Description 📝

Adds available field to Linodes and Linode Backups to @linode/api-v4
Shows a banner on a Linode's backups tab when Linode's backups are unavailable according to the API

Preview 📷

How to test 🧪

Currently only testable in local development with an API PR

Waiting on UX for this


Rewrite decompression to use less allocations and regrows

The old code used a StringBuilder with the default capacity and made a new String for each 128 byte chunk.

I am not entirely sure if this system is that much faster, as there might be a lot of array copies to enlarge the buffer of the ByteArrayOutputStream depending on the size of the decompressor output

At least the greater initial size limits the regrows, as was seen in the old StringBuilder approach, and actually doing the string coding at the end is a lot better for if there are multi byte characters, and string coders that might do validation?
I've been running it for some time, but forgot to take screenshots of my sessions.
You can take a look at my branch decompresser-test if you wish.

Would you please update this to follow our code style in bracket placing.

Sorry my bad, I will 😅

With only 6 guilds and rebuilding the shard it's apparent all those calls to the string constructor contribute to 1) more cpu time than necessary and 2) more allocations.
I don't have any hard evidence to support whether multi-byte characters would cause issues or not, but I suppose it would be pretty bad luck if it poses a problem in the real world, if those happen to align with the 128 byte chunks.

CPU call tree
Allocation call tree

(the last repush was just some import cleanup)

@Shredder121 Thank you for your contribution

:+1:


Make the default of notify_release_stages explicit

Someone reading the previous version might be tempted to call their production release stage prod. This might lead to no production errors being logged when relying on the default of notify_release_stages.
I've tried to make the default more clear, and changed the example release_stage to use the same name as the default.

Thanks @davich! This is indeed potentially confusing, and the change looks good to me.


rule(macro falco_privileged_images): add 'docker.io/falcosecurity/falco'

What type of PR is this?
/kind bug
/kind rule-update
Any specific area of the project related to this PR?
/area rules
What this PR does / why we need it:
Add 'docker.io/falcosecurity/falco' image to  'falco_privileged_images' macro. This prevent messages like this when booting up falco :
Warning Pod started with privileged container (user=system:serviceaccount:kube-system:daemon-set-controller pod=falco-42brw ns=monitoring images=docker.io/falcosecurity/falco:0.24.0)

Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign mstemm
You can assign the PR to them by writing /assign @mstemm in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

rules/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @nvanheuverzwijn! It looks like this is your first PR to falcosecurity/falco 🎉

This is a good idea! Thanks @nvanheuverzwijn

LGTM label has been added.  Git tree hash: 36eeb607a6475a36a2fb8121f26a6ef396c61b7f

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: fntlnz
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [fntlnz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: fntlnz, leodido
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [fntlnz,leodido]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Thanks for sending it here as discussed @nvanheuverzwijn :)
And congrats for your first contribution to Falco! 🎉


【help】HED

Thank you very much for your great work.
I want to draw the network structure of HED, but it has not been successful for a long time.
Can anyone share the original code of drawing HED? The deadline is coming. T-T
I'll be grateful.

Same here.

You can find code for HED here.


Unknown app id provided

I get a constant error  "message": "{"error":"Unknown app id testapp provided."}",  when I set in .env   PUSHER_APP_ID=testapp. How it works? Does it check with Pusher for real aplication IDs created with Pusher?

Did you use a custom handler?

For sure,
Web-site: http//allawitte.nl

----- Reply to message -----
Subject: Re: [beyondcode/laravel-websockets] Unknown app id provided (#294)
Date: ср, 4 мар. 2020, 15:40
From:  Ferri Sutanto<EMAIL_ADDRESS>To:  beyondcode/laravel-websockets<EMAIL_ADDRESS>Did you use a custom handler?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

php artisan config:clear ?

Same here. While developing, it was working fine on localhost. However, I am getting the same exact error after cloning to web server.
What is causing this?

Same problem... Any news? I haven't been able to decide for a week, do I really have to abandon laravel-websockets in favor of other solutions?


feat(imagegallerymodal): new component created to select an image

Closes #1786
Summary
Created a modal for selecting an image.
Change List (commits, features, bugs, etc)

Based the new modal on the ComposableModal component. The width is by default 100% and height is 80%, but can be modified to any value using scss.
Searching and filtering is done by the component on the data in the content prop
Searching defaults to the src property but can be modified via the searchProperty to be any of the image data properties, e.g. title.
The image label will be the image file name extracted by the path in the src unless the title property has a value.
A new ImageTile component based on the Tile component was created to handle both grid and list displays and selection of the images. The tiles will wrap depending on the width of the modal.
The data of the selected image is passed with the onSubmit callback
Fixed RTL issue for icon switch icon switch
Added stories
Added unit tests

Acceptance Test (how to verify the PR)

Got to https://deploy-preview-1836--carbon-addons-iot-react.netlify.app?path=/story/watson-iot-imagegallerymodal--basic

Test the described functionality

Go to https://deploy-preview-1836--carbon-addons-iot-react.netlify.app?path=/story/watson-iot-imagegallerymodal--with-i-18-n

Test that i18n can be modified

minor point, but we should probably clear the current selection on searching

:tada: This PR is included in version 2.141.0-next.30 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.141.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Porting ASP.NET Apps Ebook - add vertical slices incremental approach

Summary
Adding incremental migration guidance by vertical slice rather than by horizontal layer.

@IEvangelist @scottaddie This should be super-quick to approve, thanks!

I still think it's weird that ebooks written by individual authors are subject to general product documentation guidelines, but ok.

:shipit: ?


Obtaining Actual Depth Values with the L515

I currently have a depth map of 320240, and I also have an RGB image of 640480. How can I obtain the actual depth values in the depth map captured with my L515?
I can directly see the depth values using an application, but now I want to obtain the depth values from the captured images. I think this is a simple problem, but I've been stuck on it for a while. Please help me resolve this confusion. I would be very grateful.

Hi @CCCCCCCT  The RealSense SDK's rs-hello-realsense C++ example program demonstrates printing the actual real-world depth (distance) value in meters as a plain-text value.
https://github.com/IntelRealSense/librealsense/tree/master/examples/hello-realsense
If you have installed the RealSense SDK (librealsense) and its tools and examples then you can find a pre-built executable version of rs-hello-realsense at the following locations on your computer.
Windows
C: > Program Files (x86) > Intel RealSense SDK 2.0 >tools
Linux
/usr/local/bin
If you are using the Python language then https://github.com/IntelRealSense/librealsense/issues/9939 provides an example of code for printing the distance value.

Hi @CCCCCCCT  Do you require further assistance with this case, please?  Thanks!

If you are using get_distance then you do not need to multiply the values by the depth scale, as get_distance() outputs the real-world distance in meters.

Hi @CCCCCCCT  Do you require further assistance with this case, please?  Thanks!

Hello, I can now import photos, but I still can't use python code to get the distance in the photo. Now I can use the L515 to directly turn on the camera to check the distance inside the camera, but this is not what I want.
Now I use opencv to check the depth of the pixel in the photo, but I think this is not right because my depth map is a png file.
Thank you so much for helping out.

I am thinking whether I should read the raw file, because my Depth file is png

Saving RealSense depth data to PNG causes a lot of the depth information to be lost, whilst a .raw file preserves the depth.
An example of Python code for exporting depth to .raw that was verified as correct by a RealSense team member is at IntelRealSense/realsense-ros#1312
For C++ users there is an example provided at #1485 (comment)

Thank you very much, I'll do it right away

Hi @CCCCCCCT  Do you require further assistance with this case, please?  Thanks!

Case closed due to no further comments received.


Remove unnecessary pdf files, add root readme file

This PR

Removes PDF files we generated from Markdown files since this is not really needed anymore.
Removes a PDF file we most likely don't need anymore.
Adds a root readme file.
Adds a logo to our feature documentation

CI seems to be stuck (maybe because GitHub was down). I will make a small change to trigger it again.


replace ujson with rapidjson

Description
ujson seems to be long gone project.
What if we use something little more "today" like rapidjson
Expected behaviour
connexion uses actively developed library rapidjson instead of old ujson
Actual behaviour
connexion uses old ujson

@dtkav what do you think of that?

ujson is no longer used. It was removed in #851 to allow using a custom JSONEncoder which ujson didn't support.
rapidjson does not support the cls kwarg on dumps. rapidjson does however have an alternative to JSONEncoder. So a PR to support this would need to move that into the Jsonifier and then use Jsonifier to hide the difference between using stdlib json and rapidjson.
Here are some resources about the rapidjson Encoder that replaces dumps(cls=JSONEncoder)
https://github.com/python-rapidjson/python-rapidjson/issues/111#issuecomment-424219748
https://python-rapidjson.readthedocs.io/en/latest/encoder.html
Here are the pieces that might need to be adjusted to work with rapidjson:

connexion/jsonifier.py implements JSONEncoder and Jsonifier
connexion/apis/aiohttp_api.py has Jsonifier(cls=JSONEncoder) defaulting to using stdlib json and our custom JSONEncoder.
connexion/apis/flask_api.py has Jsonifier(flask.json, indent=2) which uses json from itsdangerous (either simplejson or stdlib json) via flask. And Flask provides its own JSONEncoder.
connexion/apis/abstract.py has Jsonifier() defaulting to stdlib json without the custom JSONEncoder - but that isn't used anywhere, since it is overriden for both aiohttp and flask.

Both Flask's and our JSONEncoder support datetime.datetime, datetime.date, and UUID. Flask's JSONEncoder also supports dataclasses, but our JSONEncoder (used with aiohttp) does not. That might be interesting to support.
This is the bug for supporting rapidjson in flask:
https://github.com/pallets/flask/issues/1602
It is closed, but looks like there was something that might allow rapidjson there.

Closing this as ujson was removed and the user can provide their own JSONEncoder.


storage.bucket.Bucket.copy_blob missing option to preserve ACL

Unlike gsutil.

Note that by default, the gsutil cp command does not copy the object ACL to the new object, and instead will use the default bucket ACL. You can override this behavior with the -p option.

--- a/bucket.py
+++ b/bucket.py
@@ -431,7 +431,7 @@
                     raise

     def copy_blob(self, blob, destination_bucket, new_name=None,
-                  client=None):
+                  client=None, preserve_acl=False):
         """Copy the given blob to the given bucket, optionally with a new name.

         :type blob: :class:`gcloud.storage.blob.Blob`
@@ -459,6 +459,8 @@
         copy_result = client.connection.api_request(
             method='POST', path=api_path, _target_object=new_blob)
         new_blob._set_properties(copy_result)
+        if preserve_acl is True:
+            new_blob.acl.save(blob.acl)
         return new_blob

     def rename_blob(self, blob, new_name, client=None):

@pdknsk thanks for letting us know!
I have a PR for this at #2409.

I've noticed that acl.save issues a separate request. Below is a better patch that sets the ACL with the same request.
--- a/bucket.py
+++ b/bucket.py
@@ -431,7 +431,7 @@
                     raise

     def copy_blob(self, blob, destination_bucket, new_name=None,
-                  client=None):
+                  client=None, preserve_acl=False):
         """Copy the given blob to the given bucket, optionally with a new name.

         :type blob: :class:`gcloud.storage.blob.Blob`
@@ -456,8 +456,12 @@
             new_name = blob.name
         new_blob = Blob(bucket=destination_bucket, name=new_name)
         api_path = blob.path + '/copyTo' + new_blob.path
+        data = None
+        if preserve_acl:
+            data = blob._properties
+            data.update({'acl': list(blob.acl)})
         copy_result = client.connection.api_request(
-            method='POST', path=api_path, _target_object=new_blob)
+            method='POST', path=api_path, _target_object=new_blob, data=data)
         new_blob._set_properties(copy_result)
         return new_blob

@pdknsk The new patch saves an API call only if the source blob's _properties and its acl are already loaded:  if _properties aren't loaded, then it would zero them out in the copied blob, while list(blob.acl) will actually trigger an API request if the acl isn't loaded.

Ah right, I missed some of those finer details.
I don't understand how _properties cannot be loaded. I think only when you manually make a Blob, with the same name as the file to be copied, and then use it to copy. In that case, you get what you asked for (even though it may not necessarily be what you wanted.)
I think there is still in advantage in only issuing a single request, for the case of a connection failure between both requests.


Pb in import of mxnet

I have a Pb in import of mxnet in python
Environment info
Operating System: windows 10
Compiler:
visual 2015 64
Package used (Python/R/Scala/Julia): Python
MXNet version:
0.10 or 0.11
installed from source:
If you are using python package, please provide
Python version and distribution:
2.7.13
Error Message:
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import mxnet
Traceback (most recent call last):
File "", line 1, in
File "mxnet_init_.py", line 25, in
from .base import MXNetError
File "mxnet\base.py", line 86, in
_LIB = _load_lib()
File "mxnet\base.py", line 78, in load_lib
lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
File "C:\Python27\lib\ctypes_init.py", line 362, in init
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 126] Le module spÚcifiÚ est introuvable
Minimum reproducible example
if you are using your own code, please provide a short script that reproduces the error.
Steps to reproduce
or if you are running standard examples, please provide the commands you have run that lead to the error.

AFTER COMPILATION : Import mxnet

What have you tried to solve it?

different version

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!
Also, do please check out our forum (and Chinese version) for general "how-to" questions.


[DRAFT] ⚡ Refactor thumbnail generation and scanner IO operations

This PR aims to (hopefully) improve the performance for both the thumbnail generation and scanner jobs. I don't think the changes are quite ready, but am making the PR to gather feedback and give myself time for proper review and testing.
The main thing to highlight is that I am trying to move away from rayon for anything IO-related, as that is actually a misuse of the library as I am learning, and towards dedicated, blocking threads. I've also added two concurrency-related configuration options in the hopes to provide better options for machines of varying power capacities.

Note to self: the last commit broke thumbnail loading on the client

So I ran a little experiment against these changes:

SMB mount library (~1543 books total) hooked up to external HDD 🐢

The initial scan:

5 hours 22 min to scan 1543 books (default concurrency of 200 files processed at a time)

Full details

For the first 4 hours memory didn't go above ~28-29MB during my periodic observations, but then slowly rose to ~35MB by the 5th hour and fluctuated between 33-36MB. I believe this is just the nature of a long-running process which tracks a growing internal state, which on a non-bottlenecked setup would likely be negligible. However I can't say for sure if this is the case here. I will say, no errors were thrown so the array of log to be dumped into the DB never grew, and while the task queue does "grow" I wouldn't expect the underlying data to be even close to the delta.

Once the scan completed, memory usage did start dropping immediately but the thumbnail generation started and.. well, it shot back up lol
When inspecting the opened files throughout the system related to the process, it does show the ~/.stump/Stump.log file remaining open consistently. I have to assume this is just a tailing operation, and that tracing isn't buffering the entire log in memory. Worth noting though.

Palpably slow media generation, but to be somewhat expected from this combination IMO

Side note: I do think the progress tracking on the UI is better than before by a large margin. However, the initial load time of provisioning the blocking threads on a severely bottlenecked setup like this still makes it feel like it's hanging for a second here and there. I think the indeterminate state I added helps, but still worth noting.

The job setup was actually very quick, quick enough that I wasn't able to time it before I had the thought to do so. This further suggests that basic traversal of the files isn't really the issue here, rather one of (or a combination of) the processing steps during media generation (hashing, metadata extraction, page counting).
DB writes were almost instantaneous, not surprising but supports filesystem IO being the bottleneck for slow scans

Some thoughts re: potential areas for performance improvements:

File hashing is currently a trait fn which each processor independently implements, using specific crates relevant to the format (e.g. zip for CBZ/ZIP) to take X pages as the sample to then hash. It might be better to have a single implementation using a smaller sample of the file, likely resulting in smaller read sizes. However, this could make the duplicate book detection less accurate, so it's a trade-off.

File hash is a nullable field in the DB, so it could be worth considering making this an opt-in feature for users who don't care about duplicate detection. That is practically the only use case for this field.

Metadata extraction is currently required, not opt-in. Besides the hashing, this should be the only read operation during the scan process. It might be worth considering a way to skip this step for users who don't care about metadata, or prefer metadata extraction to be done on-demand as a follow-up task.

Note that I am making assumptions about the crates used per format, e.g. I assume zip doesn't read the entire file to extract metadata used for iteration (e.g. file names), but I could be wrong.

TLDR; the scan was slow, but I think it's mostly due to the bottlenecked setup. Memory usage was relatively stable, and there are some potential areas for performance improvements around adding opt-in features for metadata extraction and file hashing.
The thumbnail generation:

Est ~7ish hours to generate 1543 thumbnails (200x350 px, 0.75 quality, webp format, default concurrency of 50 files processed at a time) 🐢
Note: I got impatient and stopped the process after an 2 hours, so this is a rough estimate
Note also: cancel functionality "worked" but is both bugged and improved with these changes lol (it successfully cancelled faster but marked as completed)

Full details

Averaged 4.4 (I know, fractional doesn't make sense) thumbnails per minute, which means it would take ~5-6 hours to generate all thumbs (1543/4.4 = 350.68, 351/60 = 5.85)
Memory relatively spiked for this job, at least initially, jumping to ~76MB as soon as the main sub-task loop started. It varied though, and I actually didn't see it go above 90MB. The lowest I saw was 44.2MB. I mostly saw it hovering between 48-50MB, which I'll attribute to varying sizes of images pulled from the files being held in memory since it was relatively stable for batches of files (50 are allowed at once, but I didn't keep track that finely to say for certain)

Some thoughts re: memory consumption:

Until the person who reported #427 has a chance to retest using the image with these changes, I can't really say for sure if the memory usage is improved. I do think at the very least it is faster now than before, but I couldn't replicate the growing memory issue with this setup.

For argument sake, I sampled the first 20 CBZ files in the library and found the first pages range from 2-4MB in size, so taking the average of 3MB and multiplying by 50 (concurrency limit) gives us 150MB we could hypothetically hold in memory while we generate a thumb. We resize the image in memory, so plus a little bit for each resized buffer, we then convert it, and finally dump it to disk. This doesn't quite add up to GBs of memory as reported though, but is also more than what I observed the process actually using
An additional note: each stream loop iteration (when a future resolves and the result is available), we handle the result. By the end of an iteration, I'd expect the returned buffer to be dropped and the memory to be freed (I'm not even using it). Perhaps a manual drop is needed?

TDLR; the generation was also slow, but I do think it's improved from before. I could not replicate the memory issue reported in #427, in fact I saw relatively stable and low memory usage throughout the process
All this has definitely been interesting to investigate, though I wish I had a clearer understanding of the linked issue. I'll try and setup a NFS for a follow-up, which hypothetically should result in improved speeds

Follow-up experiment:

NFS mount library (~1543 books total) hooked up to external HDD (still 🐢)

The initial scan:

3 hours 20 min to scan 1543 books (default concurrency of 200 files processed at a time)

Notes

Memory "spiked" earlier in the scan, getting to 44MB by 2 hours. I didn't really see it trend upwards, it would fluctuate down to 38-41MB and mostly hover in that range. I think this could just be correlated to the files being processed, which are unordered and differ each scan.

I did not notice memory drop after the scan completed, perhaps the OS kept the allocated memory as a cache intead of freeing it? I can experiment with a different allocator if needed but we are also talking about a relatively small MB so 🤷

Still overall slow media generation, but I'd say significantly faster than SMB (took ~2 hours less)

Some thoughts re: potential areas for performance improvements:

See note about allocator above, maybe something like mimalloc? Might be premature, I'm observing a small number of MBs hanging around, not GBs and not a steady increase as reported in #427. Also should note that the thumbnail generation tasks start soon after the scan, so it might not have had time to free the memory

TLDR; the scan was still slow, memory usage about the same, any speed diff is likely NFS vs SMB. I feel this doesn't really shed any light on the issues reported in #427, as I couldn't replicate the memory issues with either SMB or NFS mounts.
I didn't try analyzing the thumbnail generation, canceling it after a few minutes. I assume it would be similar to the scan re: overall faster but similar memory usage.

This comment https://github.com/stumpapp/stump/issues/427#issuecomment-2332857700 is really important context. A TLDR; is some internals/configurations in the zip crate changed which caused a huge performance regression. There are still some lingering memory issues, but magnitudes less after downgrading. For reference, the NFS test took 8 minutes after downgrading (down from 3 hours 20 min lol)

I'm going to do a last pass review and aim to merge this in today. I want to call out the warning in the PR description:

⚠ There are potentially breaking changes in this PR from the restructuring of library options. The migration was hand written, and while I tested it on my personal experimental instance without issue I am still weary enough to put out this disclaimer.

I will promote the current experimental to develop and then merge this shortly after (once I give it a last review). I'm somewhat eager to get these improvements out there, so I'll likely not let these changes sit in experimental for more than a week or two. If anyone is able to help with testing I'd appreciate it!


Phone reboots after finishing

-im using 3.0.0 b19 but it happened on previous versions, whenever I try to jailbreak it starts like normal but then it moves to a loading screen ( like in respring ) then the device reboots. This happens with EL and VS exploit

@Cryptiiiic & @sbinger

@Complex14 bad tweak, post your diagnostics.


It can't be installed by cocoapods

Did anyone meet the same issues?
Pod works well with other projects.

It seem that the version 1.8 is no available now.

I believe you are using some outdated mirror of the pods repo. Please update it. I attached the screenshot of current CocoaPods spec repo.

OK, I'm gonna figure it out.
Btw, thanks to reply me. XD
See you in Chinese Swift Developer Conference.
:)


Create configuration files for new Typescript project

We need to add the basic configuration files for this project:

package.json
tsconfig.json

Make sure to include the following devDependencies:

@types/jest
@types/lodash
@types/node

And the following dependencies:

jest
lodash

Issue Status: in-progress
Recent Activity
None


Feature/rst 1157 remove feedback link

Stath i don't think the issue i found got to do with your changes so happy for you to merge!


Add build tags to generated packr files

Closes #153

Currently travis is having problems with Go builds, so once that's cleared up and the tests pass, I'll merge this in. Thanks.

@techknowlogick it looks like you can't use - in a build tag:
packrd/packed-packr.go:1: invalid non-alphanumeric build constraint: !skip-packr

Thanks!


Avoid infinite loop if str contains only blanks

Proposed solution to:
https://github.com/ecmwf-ifs/ecrad/issues/14

Apologies for the delay. Thanks for this, nice find and solution!


'voxlingua107' for Language Classifier

This project has audio-samples for 107 languages:
http://bark.phon.ioc.ee/voxlingua107/
Would be great to improve the Language Classifier

Many thanks, I saw it some time ago, we are working on it albeit with a low priority
Theoretically we will even be able to make a vad out of it as well

done in https://github.com/snakers4/silero-vad/commit/395885b06b408b9ca0b84dcf05a42d8e8be59153
more data was used
probably will exclude some artificial or unspoken languages and train a bigger model

wow! this is great news....
I have been working on a language classifier using the common voice dataset, but I found it pretty hard to get a satisfying validation accuracy even on four languages. What is your validation accuracy?
I have been using 5s samples, STFT and classified them the small ATTRNN used in here
Do you have a tip on how to solve this task?

hard to get a satisfying validation accuracy even on four language
What is your validation accuracy?

We had 99%+ provided they were quite different (en, ru, de, es)
Though we just did random split, without regarding the speakers
The datasets are large enough not to care
For 100+ languages there are still some unresolved issues, i.e. English having low accuracy and mutually intelligible languges having orders of magnitude differences in available data

Do you have a tip on how to solve this task?

Just use our models
If you need higher quality for some particular cases - please dm for commercial inquiries

thanks for the quick reply!
When i did not care for speakers my acc was about 95% but failed hard in a real life scenario.
After i fixed the speakers issue, acc drastically decreased (85%) but real life performance is almost OK now.
I used 30k samples per language. I noticed there are flawed samples from cutting with Acoustic Audio Detection (auditok), so I checked for a VAD and ended up here.
Great work! I ll try to use your VAD for cleaner cuts on the samples.

I used 30k samples per language.
After i fixed the speakers issue, acc drastically decreased (85%) but real life performance is almost OK now.

also in domain / out of domain may be an issue if your dataset is not diverse enough (not enough augs)
as for language classifier, most likely we will update it soon, there are some obvious improvements


Registry: Self Help Guide

Description:
Prepare comprehensive self-help guides for the Registry. These guides should cover various aspects to assist users and developers. The following guides need to be created and maintained:

User Guide
Installation Guide
FAQ
Troubleshooting Guide

Ensure that all the guides are updated on the Confluence page.
Goals:

[ ] Prepare a detailed User Guide.
[ ] Create a comprehensive Installation Guide.
[ ] Compile a list of Frequently Asked Questions (FAQ).
[ ] Develop a Troubleshooting Guide.
[ ] Update all the guides on the Confluence page.

Expected Outcome:

A complete set of self-help guides available for users and developers.
All guides are accessible on the Confluence page for easy reference.
Improved user experience and support for the Registry.

Acceptance Criteria:

[ ] User Guide is prepared and covers all necessary aspects.
[ ] Installation Guide is created with step-by-step instructions.
[ ] FAQ list is compiled and addresses common queries.
[ ] Troubleshooting Guide is developed with solutions to common issues.
[ ] All guides are updated on the Confluence page.

Mockups / Wireframes:
NA
Product Name:
Beckn Protocol ONIX
Domain:
Documentation
Tech Skills Needed:
Technical Writing, Knowledge of Beckn Protocol, Documentation Tools
Complexity:
Medium
Category:
Documentation
Sub Category:
Self-Help Guides

Viraj will update the comment stating that he connected with Venky on Monday this week. He will include the details of the conversation and the latest document link, PR link, and close it today.
@viraj89 @emmayank @faizmagic

Following details are added and committed into the main branch here. This ticket is ready to close.

Add explanation for Schema URL field
Clarify KYC Complete checkbox functionality
Describe Core Version field in Network Role Information
Explain Status field options in Network Role Information
Add details about Operation Region field and its usage


The string "export default defineComponent" is being compiled abnormally

Related plugins

[ ] plugin-vue

[X] plugin-vue-jsx

Describe the bug
import { defineComponent } from 'vue';

const monacoEditorTemplateCode = 'export default defineComponent';

export default defineComponent({
  setup() {
    return () => monacoEditorTemplateCode;
  },
});

monacoEditorTemplateCode will be compiled to const __default__ = defineComponent under the following conditions: has default export、JSX、HMR or SSR
Reproduction
https://stackblitz.com/edit/vitejs-vite-rgsvev?file=src%2FApp.tsx
Steps to reproduce
# Reproduces only in the development environment.
npm run dev

System Info
System:
    OS: macOS 12.6
    CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
    Memory: 5.07 GB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.17.1 - ~/.nvm/versions/node/v18.17.1/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v18.17.1/bin/yarn
    npm: 9.6.7 - ~/.nvm/versions/node/v18.17.1/bin/npm
    pnpm: 7.15.0 - ~/Library/pnpm/pnpm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Edge: 121.0.2277.83
    Safari: 16.0

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

a workaround
const monacoEditorTemplateCode = 'export '+'default defineComponent';