USE_TENSORRT=1 not userfriendly

Description
I want to use tensorrt with my mxnet, i compile the mxnet with the source code. when I add the flag cmake -D USE_TENSORRT=1, i report compile error like this:
CMake Error at /home/yfzx/.local/lib/python3.7/site-packages/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
Call Stack (most recent call first):
/home/yfzx/.local/lib/python3.7/site-packages/cmake/data/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
/home/yfzx/.local/lib/python3.7/site-packages/cmake/data/share/cmake-3.18/Modules/FindProtobuf.cmake:626 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:240 (find_package)
Error Message
###reported CMakeError.log content is here:
Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/make cmTC_e1cfd/fast && /usr/bin/make  -f CMakeFiles/cmTC_e1cfd.dir/build.make CMakeFiles/cmTC_e1cfd.dir/build
make[1]: 进入目录“/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp”
Building C object CMakeFiles/cmTC_e1cfd.dir/src.c.o
/usr/bin/cc   -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -DCMAKE_HAVE_LIBC_PTHREAD -fPIE -o CMakeFiles/cmTC_e1cfd.dir/src.c.o -c /home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_e1cfd
/home/yfzx/.local/lib/python3.7/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_e1cfd.dir/link.txt --verbose=1
/usr/bin/cc  -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -DCMAKE_HAVE_LIBC_PTHREAD CMakeFiles/cmTC_e1cfd.dir/src.c.o -o cmTC_e1cfd
CMakeFiles/cmTC_e1cfd.dir/src.c.o：在函数‘main’中：
src.c:(.text.startup+0x23)：对‘pthread_create’未定义的引用
src.c:(.text.startup+0x2c)：对‘pthread_detach’未定义的引用
src.c:(.text.startup+0x35)：对‘pthread_cancel’未定义的引用
src.c:(.text.startup+0x40)：对‘pthread_join’未定义的引用
src.c:(.text.startup+0x4b)：对‘pthread_atfork’未定义的引用
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_e1cfd.dir/build.make:105: recipe for target 'cmTC_e1cfd' failed
make[1]: *** [cmTC_e1cfd] Error 1
make[1]: 离开目录“/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp”
Makefile:140: recipe for target 'cmTC_e1cfd/fast' failed
make: *** [cmTC_e1cfd/fast] Error 2
Source file was:
#include <pthread.h>
void* test_func(void* data)
{
return data;
}
int main(void)
{
pthread_t thread;
pthread_create(&thread, NULL, test_func, NULL);
pthread_detach(thread);
pthread_cancel(thread);
pthread_join(thread, NULL);
pthread_atfork(NULL, NULL, NULL);
pthread_exit(NULL);
return 0;
}
Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/make cmTC_9390f/fast && /usr/bin/make  -f CMakeFiles/cmTC_9390f.dir/build.make CMakeFiles/cmTC_9390f.dir/build
make[1]: 进入目录“/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp”
Building C object CMakeFiles/cmTC_9390f.dir/CheckFunctionExists.c.o
/usr/bin/cc   -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -DCHECK_FUNCTION_EXISTS=pthread_create -fPIE -o CMakeFiles/cmTC_9390f.dir/CheckFunctionExists.c.o -c /home/yfzx/.local/lib/python3.7/site-packages/cmake/data/share/cmake-3.18/Modules/CheckFunctionExists.c
Linking C executable cmTC_9390f
/home/yfzx/.local/lib/python3.7/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_9390f.dir/link.txt --verbose=1
/usr/bin/cc  -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -DCHECK_FUNCTION_EXISTS=pthread_create CMakeFiles/cmTC_9390f.dir/CheckFunctionExists.c.o -o cmTC_9390f  -lpthreads
/usr/bin/ld: 找不到 -lpthreads
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_9390f.dir/build.make:105: recipe for target 'cmTC_9390f' failed
make[1]: *** [cmTC_9390f] Error 1
make[1]: 离开目录“/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeTmp”
Makefile:140: recipe for target 'cmTC_9390f/fast' failed
make: *** [cmTC_9390f/fast] Error 2
To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
I use cuda 10.2, cudnn8.0.4, mxnet-src1.7.0, i have install tensorrt 7.2.1 with the offical tutorial, have pass the helloworld test
Steps to reproduce
(Paste the commands you ran that produced the error.)
1.i copy the $mxnet-src-path$/config/linux_gpu.cmake to $mxnet-src-path$/config.cmake, make some change like add one line "set(USE_NCCL 1)"
2.cd $mxnet-src-path$, then mkdir build,  cd build, cmake -D USE_TENSORRT=1 ..
What have you tried to solve it?

When i remove the -D USE_TENSORRT=1 flag, it can compile correctly, whe i add this flag, it will report like this issue;

Environment
We recommend using our script for collecting the diagnostic information with the following command
curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python3

cuda 10.2, cudnn8.0.4, mxnet-src1.7.0, python 3.7.4(anaconda), nvidia driver version:440.44
Environment Information
# Paste the diagnose.py command output here

problem resolved!
from the Reported message : Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
i install the protobuf ( sudo apt-get install libprotobuf-dev protobuf-compiler), then solved the problem.
but when compile again , it reported another problem:
CMake Error at CMakeLists.txt:242 (find_library):
Could not find ONNX_LIBRARY using the following names: libonnx.so
-- Configuring incomplete, errors occurred!
See also "/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeOutput.log".
See also "/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeError.log".
i install the onnx follow the tensorrt official tutorial, untar the .tar file, then cd TensorRT-${version}/onnx_graphsurgeon
pip3 install  onnx_graphsurgeon-0.2.6-py2.py3-none-any.whl
i guess the onnx is not installed correctly, is that another way to install onnx?

problem resolved!
from the Reported message : Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)
i install the protobuf ( sudo apt-get install libprotobuf-dev protobuf-compiler), then solved the problem.
but when compile again , it reported another problem:
CMake Error at CMakeLists.txt:242 (find_library):
Could not find ONNX_LIBRARY using the following names: libonnx.so
-- Configuring incomplete, errors occurred!
See also "/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeOutput.log".
See also "/home/yfzx/mxnet-src/apache-mxnet-src-1.7.0-incubating/build/CMakeFiles/CMakeError.log".
i install the onnx follow the tensorrt official tutorial, untar the .tar file, then cd TensorRT-${version}/onnx_graphsurgeon
pip3 install  onnx_graphsurgeon-0.2.6-py2.py3-none-any.whl
i guess the onnx is not installed correctly, is that another way to install onnx?

The build with tensorrt is currently not very user friendly. Nvidia plans to contribute a PR to improve it. For now, follow
https://github.com/apache/incubator-mxnet/blob/ae8c9748743ca98979964bd34643aca343f93c7c/ci/docker/runtime_functions.sh#L541-L594

The build with tensorrt is currently not very user friendly. Nvidia plans to contribute a PR to improve it. For now, follow
https://github.com/apache/incubator-mxnet/blob/ae8c9748743ca98979964bd34643aca343f93c7c/ci/docker/runtime_functions.sh#L541-L594

The build with tensorrt is currently not very user friendly. Nvidia plans to contribute a PR to improve it. For now, follow
https://github.com/apache/incubator-mxnet/blob/ae8c9748743ca98979964bd34643aca343f93c7c/ci/docker/runtime_functions.sh#L541-L594

Thank U very much!!
I'm not very clear about how to use this .sh file in docker, I will try to install onnx manually tomorrow, thank you again!

The build with tensorrt is currently not very user friendly. Nvidia plans to contribute a PR to improve it. For now, follow
https://github.com/apache/incubator-mxnet/blob/ae8c9748743ca98979964bd34643aca343f93c7c/ci/docker/runtime_functions.sh#L541-L594

Thank U very much!!
I'm not very clear about how to use this .sh file in docker, I will try to install onnx manually tomorrow, thank you again!

You don't need to use the .sh file, but you can just refer to the steps listed in the section of the .sh file referenced

You don't need to use the .sh file, but you can just refer to the steps listed in the section of the .sh file referenced

cc @Kh4L

cc @Kh4L

You don't need to use the .sh file, but you can just manually follow the steps listed in the section of the .sh file referenced

thanks, I still don't know how to install onnx manually, I get many errors in the follow the tutorial of the onnx installation guide.

You don't need to use the .sh file, but you can just manually follow the steps listed in the section of the .sh file referenced

thanks, I still don't know how to install onnx manually, I get many errors in the follow the tutorial of the onnx installation guide.

You don't need to use the .sh file, but you can just manually follow the steps listed in the section of the .sh file referenced

I'm very confused about the error message:
CMake Error at CMakeLists.txt:242 (find_library):
Could not find ONNX_LIBRARY using the following names: libonnx.so
I search in both onnx and onnx-tensorrt folders , there is no file named libonnx.so**, I only found some files named libnvonnxparser.so/libnvonnxparser.so.7/libnvonnxparser.so.7.2.1.
What should I do next?
Thank you!

You don't need to use the .sh file, but you can just manually follow the steps listed in the section of the .sh file referenced

I'm very confused about the error message:
CMake Error at CMakeLists.txt:242 (find_library):
Could not find ONNX_LIBRARY using the following names: libonnx.so
I search in both onnx and onnx-tensorrt folders , there is no file named libonnx.so**, I only found some files named libnvonnxparser.so/libnvonnxparser.so.7/libnvonnxparser.so.7.2.1.
What should I do next?
Thank you!

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

I will tell what I have done:
First, I download onnx-tensorrt from "https://github.com/onnx/onnx-tensorrt" to my /home dir, but in this project, the onnx-tensorrt/third_party/onnx/  directory is empty, I download the onnx from "https://github.com/onnx/onnx/tree/553df22c67bee5f0fe6599cff60f1afc6748c635" to the onnx-tensorrt/third_party/onnx/ dir.
Then I follow the onnx-tensorrt official tutorial, do the compile:
cd onnx-tensorrt
mkdir build && cd build
cmake .. -DTENSORRT_ROOT=<path_to_trt> && make -j
// Ensure that you update your LD_LIBRARY_PATH to pick up the location of the newly built library:
export LD_LIBRARY_PATH=$PWD:$LD_LIBRARY_PATH
It can compile correctly , but I found nothing like named libonnx.so**. And when I compile the mxnet with -DTENSORRT=1, It reported error again.
You said I must define some arguments , how many arguments should I have to define?
Now I have the mxnet-src file in my home directory like : /home/$USER_NAME/mxnet-src
and onnx-tensortrt in my home dir: /home/$USER_NAME/onnx-tensorrt
and onnx in the onnx-tensorrt directory :/home/$USER_NAME/onnx-tensorrt/third_party/onnx/
How should I define the eviron arguments? You said I must follow the .sh file in the docker file,  but I'm not sure how many arguments I shoud to define. Some args like ONNX_NAMESPACE,  LIBRARY_PATH, LD_LIBRARY_PATH, CPLUS_INCLUDE_PATH, CXXFLAGS, should I have to define it manually? And the popd, pushd, should I have to do that like the .sh file did?
I'm not very good at do compiling and define args, so thanks for your answering!!

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

I will tell what I have done:
First, I download onnx-tensorrt from "https://github.com/onnx/onnx-tensorrt" to my /home dir, but in this project, the onnx-tensorrt/third_party/onnx/  directory is empty, I download the onnx from "https://github.com/onnx/onnx/tree/553df22c67bee5f0fe6599cff60f1afc6748c635" to the onnx-tensorrt/third_party/onnx/ dir.
Then I follow the onnx-tensorrt official tutorial, do the compile:
cd onnx-tensorrt
mkdir build && cd build
cmake .. -DTENSORRT_ROOT=<path_to_trt> && make -j
// Ensure that you update your LD_LIBRARY_PATH to pick up the location of the newly built library:
export LD_LIBRARY_PATH=$PWD:$LD_LIBRARY_PATH
It can compile correctly , but I found nothing like named libonnx.so**. And when I compile the mxnet with -DTENSORRT=1, It reported error again.
You said I must define some arguments , how many arguments should I have to define?
Now I have the mxnet-src file in my home directory like : /home/$USER_NAME/mxnet-src
and onnx-tensortrt in my home dir: /home/$USER_NAME/onnx-tensorrt
and onnx in the onnx-tensorrt directory :/home/$USER_NAME/onnx-tensorrt/third_party/onnx/
How should I define the eviron arguments? You said I must follow the .sh file in the docker file,  but I'm not sure how many arguments I shoud to define. Some args like ONNX_NAMESPACE,  LIBRARY_PATH, LD_LIBRARY_PATH, CPLUS_INCLUDE_PATH, CXXFLAGS, should I have to define it manually? And the popd, pushd, should I have to do that like the .sh file did?
I'm not very good at do compiling and define args, so thanks for your answering!!

but in this project, the onnx-tensorrt/third_party/onnx/ directory is empty

You need to run git submodule update --init recursive inside the mxnet folder. Do not manually copy around folders

but in this project, the onnx-tensorrt/third_party/onnx/ directory is empty

You need to run git submodule update --init recursive inside the mxnet folder. Do not manually copy around folders

but in this project, the onnx-tensorrt/third_party/onnx/ directory is empty, so I download the onnx from "https://github.com/onnx/onnx/tree/553df22c67bee5f0fe6599cff60f1afc6748c635" and put it to the onnx-tensorrt/third_party/onnx/ dir.

You need to run git submodule update --init --recursive inside the mxnet folder. Do not manually copy around folders

Then I follow the onnx-tensorrt official tutorial, do the compile:

You need to follow the steps mentioned by @Kh4L from the mxnet folder after running git submodule update --init --recursive. No need to download any other repositories manually.

And the popd, pushd, should I have to do that like the .sh file did?

It's better to follow the instructions exactly if you are not clear about their meaning. If you know what they do, you can also use cd instead

I'm not very good at do compiling and define args, so thanks for your answering!!

Just start with a new git clone https://github.com/apache/incubator-mxnet/, run the git submodule --init --recursive and follow the instruction precisely step by step.

when I have done what you have told me, i download the mxnet source code again and do the git submodule update --init --recursive, I followed the .sh file as listed before, when it run to the "cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE .." line , it reported error :
NAMESPACE=$ONNX_NAMESPACE ..

-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/TENSORRT_INCLUDE_DIR
used as include directory in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
used as include directory in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_INFER
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_INFER_PLUGIN
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_MYELIN
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
cannot find Tensorrt Library, where can i find it?

but in this project, the onnx-tensorrt/third_party/onnx/ directory is empty, so I download the onnx from "https://github.com/onnx/onnx/tree/553df22c67bee5f0fe6599cff60f1afc6748c635" and put it to the onnx-tensorrt/third_party/onnx/ dir.

You need to run git submodule update --init --recursive inside the mxnet folder. Do not manually copy around folders

Then I follow the onnx-tensorrt official tutorial, do the compile:

You need to follow the steps mentioned by @Kh4L from the mxnet folder after running git submodule update --init --recursive. No need to download any other repositories manually.

And the popd, pushd, should I have to do that like the .sh file did?

It's better to follow the instructions exactly if you are not clear about their meaning. If you know what they do, you can also use cd instead

I'm not very good at do compiling and define args, so thanks for your answering!!

Just start with a new git clone https://github.com/apache/incubator-mxnet/, run the git submodule --init --recursive and follow the instruction precisely step by step.

when I have done what you have told me, i download the mxnet source code again and do the git submodule update --init --recursive, I followed the .sh file as listed before, when it run to the "cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE .." line , it reported error :
NAMESPACE=$ONNX_NAMESPACE ..

-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/TENSORRT_INCLUDE_DIR
used as include directory in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
used as include directory in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_INFER
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_INFER_PLUGIN
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
TENSORRT_LIBRARY_MYELIN
linked by target "nvonnxparser" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
linked by target "nvonnxparser_static" in directory /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt
-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
cannot find Tensorrt Library, where can i find it?

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

Yes , I already have installed Tensorrt. It is in my /usr/local/TensorRT-<IP_ADDRESS>/ directory.

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

Yes , I already have installed Tensorrt. It is in my /usr/local/TensorRT-<IP_ADDRESS>/ directory.

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

Yes , I already have installed Tensorrt. It is in my /usr/local/TensorRT-<IP_ADDRESS>/ directory.

@leezu
and follow the https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleMNIST tutorial, I run the sample_mnist program to confirm my tensorrt is installed correctly.
the output of the sample_mnist is below:
&&&& RUNNING TensorRT.sample_mnist # ./sample_mnist
[01/10/2021-20:19:35] [I] Building and running a GPU inference engine for MNIST
[01/10/2021-20:19:42] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[01/10/2021-20:19:42] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[01/10/2021-20:19:42] [I] Input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@+  #@@@@@@
@@@@@@@@@@@@@@@@@*   #@@@@@@
@@@@@@@@@@@@@@@@@=   #@@@@@@
@@@@@@@@@@@@@@@@#  .%@@@@@@@
@@@@@@@@@@@@@@@#.  @@@@@@@@
@@@@@@@@@@@@@@@-  :@@@@@@@@@
@@@@@@@@@@@@@@@   #@@@@@@@@@
@@@@@@@@@@@@@%.  =@@@@@@@@@@
@@@@@@@@@@@@@+  :@@@@@@@@@@@
@@@@@@@@@@@@  :%@@@@@@@@@@@
@@@@@@@@@@@%   -@@@@@@@@@@@@
@@@@@@@@@@@=   %@@@@@@@@@@@@
@@@@@@@@@@#.  -@@@@@@@@@@@@@
@@@@@@@@@@   :@@@@@@@@@@@@@@
@@@@@@@@@@   *@@@@@@@@@@@@@@
@@@@@@@@@-   %@@@@@@@@@@@@@@
@@@@@@@@#.  =@@@@@@@@@@@@@@@
@@@@@@@@#  .@@@@@@@@@@@@@@@@
@@@@@@@@#  .@@@@@@@@@@@@@@@@
@@@@@@@@%: .@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[01/10/2021-20:19:42] [I] Output:
0:
1: **********
2:
3:
4:
5:
6:
7:
8:
9:
&&&& PASSED TensorRT.sample_mnist # ./sample_mnist

You will need to install tensorrt first. Please see https://developer.nvidia.com/Tensorrt

Yes , I already have installed Tensorrt. It is in my /usr/local/TensorRT-<IP_ADDRESS>/ directory.

@leezu
and follow the https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleMNIST tutorial, I run the sample_mnist program to confirm my tensorrt is installed correctly.
the output of the sample_mnist is below:
&&&& RUNNING TensorRT.sample_mnist # ./sample_mnist
[01/10/2021-20:19:35] [I] Building and running a GPU inference engine for MNIST
[01/10/2021-20:19:42] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[01/10/2021-20:19:42] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[01/10/2021-20:19:42] [I] Input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@+  #@@@@@@
@@@@@@@@@@@@@@@@@*   #@@@@@@
@@@@@@@@@@@@@@@@@=   #@@@@@@
@@@@@@@@@@@@@@@@#  .%@@@@@@@
@@@@@@@@@@@@@@@#.  @@@@@@@@
@@@@@@@@@@@@@@@-  :@@@@@@@@@
@@@@@@@@@@@@@@@   #@@@@@@@@@
@@@@@@@@@@@@@%.  =@@@@@@@@@@
@@@@@@@@@@@@@+  :@@@@@@@@@@@
@@@@@@@@@@@@  :%@@@@@@@@@@@
@@@@@@@@@@@%   -@@@@@@@@@@@@
@@@@@@@@@@@=   %@@@@@@@@@@@@
@@@@@@@@@@#.  -@@@@@@@@@@@@@
@@@@@@@@@@   :@@@@@@@@@@@@@@
@@@@@@@@@@   *@@@@@@@@@@@@@@
@@@@@@@@@-   %@@@@@@@@@@@@@@
@@@@@@@@#.  =@@@@@@@@@@@@@@@
@@@@@@@@#  .@@@@@@@@@@@@@@@@
@@@@@@@@#  .@@@@@@@@@@@@@@@@
@@@@@@@@%: .@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[01/10/2021-20:19:42] [I] Output:
0:
1: **********
2:
3:
4:
5:
6:
7:
8:
9:
&&&& PASSED TensorRT.sample_mnist # ./sample_mnist

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>
OK, I compiled it correctly.
but when I do make, it reported error again....
I only paste part of it, like this:
In file included from /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/schema.h:24:0,
from /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/nn/defs.cc:7:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/shape_inference.h: In function ‘bool onnx2trt_onnx::getRepeatedAttribute(onnx2trt_onnx::InferenceContext&, std::__cxx11::string, std::vector&) [with T = long int]’:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/shape_inference.h:83:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
third_party/onnx/CMakeFiles/onnx.dir/build.make:315: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/old.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/old.cc.o] Error 1
third_party/onnx/CMakeFiles/onnx.dir/build.make:302: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/defs.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/defs.cc.o] Error 1
third_party/onnx/CMakeFiles/onnx.dir/build.make:81: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/checker.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/checker.cc.o] Error 1
CMakeFiles/Makefile2:321: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/all' failed
make[1]: *** [third_party/onnx/CMakeFiles/onnx.dir/all] Error 2
make[1]: *** 正在等待未完成的任务....
[ 59%] Linking CXX shared library libnvonnxparser.so
[ 59%] Built target nvonnxparser
[ 60%] Linking CXX static library libnvonnxparser_static.a
[ 60%] Built target nvonnxparser_static
Makefile:170: recipe for target 'all' failed
make: *** [all] Error 2

@leezu

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>
OK, I compiled it correctly.
but when I do make, it reported error again....
I only paste part of it, like this:
In file included from /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/schema.h:24:0,
from /home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/nn/defs.cc:7:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/shape_inference.h: In function ‘bool onnx2trt_onnx::getRepeatedAttribute(onnx2trt_onnx::InferenceContext&, std::__cxx11::string, std::vector&) [with T = long int]’:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/third_party/onnx/onnx/defs/shape_inference.h:83:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
third_party/onnx/CMakeFiles/onnx.dir/build.make:315: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/old.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/old.cc.o] Error 1
third_party/onnx/CMakeFiles/onnx.dir/build.make:302: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/defs.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/defs/nn/defs.cc.o] Error 1
third_party/onnx/CMakeFiles/onnx.dir/build.make:81: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/onnx/checker.cc.o' failed
make[2]: *** [third_party/onnx/CMakeFiles/onnx.dir/onnx/checker.cc.o] Error 1
CMakeFiles/Makefile2:321: recipe for target 'third_party/onnx/CMakeFiles/onnx.dir/all' failed
make[1]: *** [third_party/onnx/CMakeFiles/onnx.dir/all] Error 2
make[1]: *** 正在等待未完成的任务....
[ 59%] Linking CXX shared library libnvonnxparser.so
[ 59%] Built target nvonnxparser
[ 60%] Linking CXX static library libnvonnxparser_static.a
[ 60%] Built target nvonnxparser_static
Makefile:170: recipe for target 'all' failed
make: *** [all] Error 2

@leezu

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>

OK, I compiled again, and now it work correctly.
But I don't know at the bottom of  .sh file, the directory of /work/mxnet/lib, what's the meaning of the /work/mxnet?
Is that the directory of the mxnet source code? I think I must change the directory, because when i do the copy, the terminal will exit.

usr/local/TensorRT-<IP_ADDRESS> is a non-standard tensorrt directory. So you need to specify -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>

OK, I compiled again, and now it work correctly.
But I don't know at the bottom of  .sh file, the directory of /work/mxnet/lib, what's the meaning of the /work/mxnet?
Is that the directory of the mxnet source code? I think I must change the directory, because when i do the copy, the terminal will exit.

/work/mxnet  refers to the path of mxnet source code, yes.

/work/mxnet  refers to the path of mxnet source code, yes.

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

the result of the "ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx" is :
backend                  onnxifi_dummy.c           onnx_pb.h
bin                      onnxifi_ext.h             onnx.proto
checker.cc               onnxifi.h                 onnx.proto3
checker.h                onnxifi_loader.c          optimizer
checker.py               onnxifi_loader.h          optimizer.py
common                   onnxifi_utils.cc          proto_utils.h
cpp2py_export.cc         onnxifi_utils.h           py_utils.h
defs                     onnxifi_wrapper.c         shape_inference
examples                 onnx.in.proto             shape_inference.py
external_data_helper.py  onnx-ml.proto             string_utils.h
frontend                 onnx-ml.proto3            test
gen_proto.py             onnx-operators.in.proto   tools
helper.py                onnx-operators-ml.proto   utils.py
init.py              onnx-operators-ml.proto3  version_converter
mapping.py               onnx-operators_pb.h       version_converter.py
numpy_helper.py          onnx-operators.proto
onnx_cpp2py_export       onnx-operators.proto3
what's the meaning of these three  line?
mkdir -p /work/mxnet/lib/
cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/
the  /work/mxnet/lib/ is also must be in the $LIBRARY_PATH?
@Kh4L @leezu

Did you run this successfully:
     # Build ONNX
     pushd .
     echo "Installing ONNX."
     cd 3rdparty/onnx-tensorrt/third_party/onnx
     rm -rf build
     mkdir -p build
     cd build
     cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:`pwd`/onnx/:$LIBRARY_PATH
     export CPLUS_INCLUDE_PATH=`pwd`:$CPLUS_INCLUDE_PATH
     export CXXFLAGS=-I`pwd`

     popd

     # Build ONNX-TensorRT
     export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
     export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
     pushd .
     cd 3rdparty/onnx-tensorrt/
     mkdir -p build
     cd build
     cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE ..
     make -j$(nproc)
     export LIBRARY_PATH=`pwd`:$LIBRARY_PATH
     popd

     mkdir -p /work/mxnet/lib/
     cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
     cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/

If yes, what is the outputs of :
ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx

and finally, can you verify if this path is part of $LIBRARY_PATH?

the result of the "ls 3rdparty/onnx-tensorrt/third_party/onnx/onnx" is :
backend                  onnxifi_dummy.c           onnx_pb.h
bin                      onnxifi_ext.h             onnx.proto
checker.cc               onnxifi.h                 onnx.proto3
checker.h                onnxifi_loader.c          optimizer
checker.py               onnxifi_loader.h          optimizer.py
common                   onnxifi_utils.cc          proto_utils.h
cpp2py_export.cc         onnxifi_utils.h           py_utils.h
defs                     onnxifi_wrapper.c         shape_inference
examples                 onnx.in.proto             shape_inference.py
external_data_helper.py  onnx-ml.proto             string_utils.h
frontend                 onnx-ml.proto3            test
gen_proto.py             onnx-operators.in.proto   tools
helper.py                onnx-operators-ml.proto   utils.py
init.py              onnx-operators-ml.proto3  version_converter
mapping.py               onnx-operators_pb.h       version_converter.py
numpy_helper.py          onnx-operators.proto
onnx_cpp2py_export       onnx-operators.proto3
what's the meaning of these three  line?
mkdir -p /work/mxnet/lib/
cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /work/mxnet/lib/
cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /work/mxnet/lib/
the  /work/mxnet/lib/ is also must be in the $LIBRARY_PATH?
@Kh4L @leezu

@leezu
Thank you, it's 10:30PM in my country, I must go home and fix it tomorrow.
Thanks!

@leezu
Thank you, it's 10:30PM in my country, I must go home and fix it tomorrow.
Thanks!

@leezu @Kh4L
the incubator-mxnet/ci/docker/runtime_functions.sh file,
I can run to the line before the
" cmake -DUSE_CUDA=1                            \
-DUSE_CUDNN=1                           \
-DUSE_OPENCV=1                          \
-DUSE_TENSORRT=1                        \
-DUSE_OPENMP=0                          \
-DUSE_MKLDNN=0                          \
-DUSE_NVML=OFF                          \
-DUSE_MKL_IF_AVAILABLE=OFF              \
-DMXNET_CUDA_ARCH="$CI_CMAKE_CUDA_ARCH" \
-G Ninja                                \
/work/mxnet " line
But when every time I do cmake , terminal will exit automatically, so I follow the instructions of the /mxnet-src/incubator-mxnet/config/linux_gpu.cmake but added THE USE_TENSORRT flag: $ mkdir build; cd build #  $ cmake .. -DUSE_TENSORRT=1#  $ cmake --build . --parallel 32
THE cmake can do correctly, but when do cmake --build . --parallel 32, It reported:

In file included from /home/yfzx/mxnet-src/incubator-mxnet/src/operator/subgraph/tensorrt/./onnx_to_tensorrt.h:31:0,
from /home/yfzx/mxnet-src/incubator-mxnet/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc:29:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/NvOnnxParser.h:26:10: fatal error: NvInfer.h: 没有那个文件或目录
#include "NvInfer.h"
^~~~~~~~~~~
compilation terminated.
CMakeFiles/mxnet.dir/build.make:4813: recipe for target 'CMakeFiles/mxnet.dir/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc.o' failed
make[2]: *** [CMakeFiles/mxnet.dir/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc.o] Error 1
CMakeFiles/Makefile2:546: recipe for target 'CMakeFiles/mxnet.dir/all' failed
make[1]: *** [CMakeFiles/mxnet.dir/all] Error 2
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2

I'm still not sure about what the path /work/mxnet means, does the /work means the root of the source code or the /work/mxnet means the root of the source code?
And how to fix my bug?

@leezu @Kh4L
the incubator-mxnet/ci/docker/runtime_functions.sh file,
I can run to the line before the
" cmake -DUSE_CUDA=1                            \
-DUSE_CUDNN=1                           \
-DUSE_OPENCV=1                          \
-DUSE_TENSORRT=1                        \
-DUSE_OPENMP=0                          \
-DUSE_MKLDNN=0                          \
-DUSE_NVML=OFF                          \
-DUSE_MKL_IF_AVAILABLE=OFF              \
-DMXNET_CUDA_ARCH="$CI_CMAKE_CUDA_ARCH" \
-G Ninja                                \
/work/mxnet " line
But when every time I do cmake , terminal will exit automatically, so I follow the instructions of the /mxnet-src/incubator-mxnet/config/linux_gpu.cmake but added THE USE_TENSORRT flag: $ mkdir build; cd build #  $ cmake .. -DUSE_TENSORRT=1#  $ cmake --build . --parallel 32
THE cmake can do correctly, but when do cmake --build . --parallel 32, It reported:

In file included from /home/yfzx/mxnet-src/incubator-mxnet/src/operator/subgraph/tensorrt/./onnx_to_tensorrt.h:31:0,
from /home/yfzx/mxnet-src/incubator-mxnet/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc:29:
/home/yfzx/mxnet-src/incubator-mxnet/3rdparty/onnx-tensorrt/NvOnnxParser.h:26:10: fatal error: NvInfer.h: 没有那个文件或目录
#include "NvInfer.h"
^~~~~~~~~~~
compilation terminated.
CMakeFiles/mxnet.dir/build.make:4813: recipe for target 'CMakeFiles/mxnet.dir/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc.o' failed
make[2]: *** [CMakeFiles/mxnet.dir/src/operator/subgraph/tensorrt/onnx_to_tensorrt.cc.o] Error 1
CMakeFiles/Makefile2:546: recipe for target 'CMakeFiles/mxnet.dir/all' failed
make[1]: *** [CMakeFiles/mxnet.dir/all] Error 2
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2

I'm still not sure about what the path /work/mxnet means, does the /work means the root of the source code or the /work/mxnet means the root of the source code?
And how to fix my bug?

@benz725
/work/mxnet/lib is for the LD_LIBRARY_PATH, as it is done here:
https://github.com/apache/incubator-mxnet/blob/209a789e6cbc06d686b23893b892fbd58af6ffe8/ci/docker/runtime_functions.sh#L1099
/work/mxnet correspond to the root of the source code.
In your case, you should replace all occurrences of /work/mxnet by /home/yfzx/mxnet-src/.

@benz725
/work/mxnet/lib is for the LD_LIBRARY_PATH, as it is done here:
https://github.com/apache/incubator-mxnet/blob/209a789e6cbc06d686b23893b892fbd58af6ffe8/ci/docker/runtime_functions.sh#L1099
/work/mxnet correspond to the root of the source code.
In your case, you should replace all occurrences of /work/mxnet by /home/yfzx/mxnet-src/.

@leezu @Kh4L
I git clone the project agin to my folder : run "git clone --recursive https://github.com/apache/incubator-mxnet mxnet" at my "/home/yfzx/mxnet-src/mxnet_root" folder, then run below lines one by one:

cd /home/yfzx/mxnet-src/mxnet_root/mxnet
set -ex

export CC=gcc-7
export CXX=g++-7
export ONNX_NAMESPACE=onnx
Build ONNX
pushd .
echo "Installing ONNX."
cd 3rdparty/onnx-tensorrt/third_party/onnx
rm -rf build
mkdir -p build
cd build
cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
make -j$(nproc)
export LIBRARY_PATH=pwd:pwd/onnx/:$LIBRARY_PATH
export CPLUS_INCLUDE_PATH=pwd:$CPLUS_INCLUDE_PATH
export CXXFLAGS=-Ipwd
popd
Build ONNX-TensorRT
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
pushd .
cd 3rdparty/onnx-tensorrt/
mkdir -p build
cd build
cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE .. -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>
make -j$(nproc)
export LIBRARY_PATH=pwd:$LIBRARY_PATH
popd
mkdir -p /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
mkdir -p /home/yfzx/mxnet-src/mxnet_root/mxnet/build
cd /home/yfzx/mxnet-src/mxnet_root/mxnet/build
export LD_LIBRARY_PATH=/home/yfzx/mxnet-src/mxnet_root/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/TensorRT-<IP_ADDRESS>/include/:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/TensorRT-<IP_ADDRESS>/include/:$LIBRARY_PATH
rm -rf *
cmake .. -DUSE_CUDA=1 -DUSE_TENSORRT=1 -DUSE_CUDNN=1 -DUSE_OPENCV=1
cmake --build . --parellel 32
cmake .. can run correctly, but when run cmake --build ,the terminal will automatically exit. Why does these things happen?

@leezu @Kh4L
I git clone the project agin to my folder : run "git clone --recursive https://github.com/apache/incubator-mxnet mxnet" at my "/home/yfzx/mxnet-src/mxnet_root" folder, then run below lines one by one:

cd /home/yfzx/mxnet-src/mxnet_root/mxnet
set -ex

export CC=gcc-7
export CXX=g++-7
export ONNX_NAMESPACE=onnx
Build ONNX
pushd .
echo "Installing ONNX."
cd 3rdparty/onnx-tensorrt/third_party/onnx
rm -rf build
mkdir -p build
cd build
cmake -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER} -DBUILD_SHARED_LIBS=ON ..
make -j$(nproc)
export LIBRARY_PATH=pwd:pwd/onnx/:$LIBRARY_PATH
export CPLUS_INCLUDE_PATH=pwd:$CPLUS_INCLUDE_PATH
export CXXFLAGS=-Ipwd
popd
Build ONNX-TensorRT
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/local/cuda-10.2/targets/x86_64-linux/include/
pushd .
cd 3rdparty/onnx-tensorrt/
mkdir -p build
cd build
cmake -DONNX_NAMESPACE=$ONNX_NAMESPACE .. -DTENSORRT_ROOT=/usr/local/TensorRT-<IP_ADDRESS>
make -j$(nproc)
export LIBRARY_PATH=pwd:$LIBRARY_PATH
popd
mkdir -p /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
cp 3rdparty/onnx-tensorrt/third_party/onnx/build/*.so /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
cp -L 3rdparty/onnx-tensorrt/build/libnvonnxparser.so /home/yfzx/mxnet-src/mxnet_root/mxnet/lib/
mkdir -p /home/yfzx/mxnet-src/mxnet_root/mxnet/build
cd /home/yfzx/mxnet-src/mxnet_root/mxnet/build
export LD_LIBRARY_PATH=/home/yfzx/mxnet-src/mxnet_root/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/TensorRT-<IP_ADDRESS>/include/:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/TensorRT-<IP_ADDRESS>/include/:$LIBRARY_PATH
rm -rf *
cmake .. -DUSE_CUDA=1 -DUSE_TENSORRT=1 -DUSE_CUDNN=1 -DUSE_OPENCV=1
cmake --build . --parellel 32
cmake .. can run correctly, but when run cmake --build ,the terminal will automatically exit. Why does these things happen?

Maybe your machine is running out of memory and killing the terminal. Try using less parallel processes.

Maybe your machine is running out of memory and killing the terminal. Try using less parallel processes.

the docker which you infered can run tensort directly？where can I download the docker image？i donnt want to compile again. so many bugs make me so headacke.
@leezu

the docker which you infered can run tensort directly？where can I download the docker image？i donnt want to compile again. so many bugs make me so headacke.
@leezu

@Kh4L would you have time to submit the PR to simplify TensorRT build? It's really unfortunate the current build is not easy for users

@Kh4L would you have time to submit the PR to simplify TensorRT build? It's really unfortunate the current build is not easy for users

i think if there is a docker image i can download directly , it will work and can save me a lot of time .

i think if there is a docker image i can download directly , it will work and can save me a lot of time .

@benz725 it's not allowed due to license of tensorrt

@benz725 it's not allowed due to license of tensorrt

@benz725 NVIDIA has MXNet container at https://ngc.nvidia.com/catalog/containers/nvidia:mxnet

@benz725 NVIDIA has MXNet container at https://ngc.nvidia.com/catalog/containers/nvidia:mxnet

when I followed the instructions below, I do the last ninja step , it informed me that: 3rdparty/onnx-tensorrt/NvOnnxParser.h:26:10: fatal error: NvInfer.h: No such file or directory
however, I had added the $TENSORRT_ROOT/include and $TENSORRT_ROOT/lib into my $LD_LIBRARY_PATH, how does this happen?
And when I do ninja, many times it informed me :ninja: build stopped: subcommand failed.
I search this in the internet, and get replies that the ulimit number is to small, then I run ulimit -n 10240 to set ulimit. But when I do ninja, It informed me :"ninja: build stopped: subcommand failed." again. why does this happen？

The build with tensorrt is currently not very user friendly. Nvidia plans to contribute a PR to improve it. For now, follow
https://github.com/apache/incubator-mxnet/blob/ae8c9748743ca98979964bd34643aca343f93c7c/ci/docker/runtime_functions.sh#L541-L594

@leezu @Kh4L

If you use the latest v1.x branch, the instruction have been simplified thanks to @Kh4L
https://github.com/apache/incubator-mxnet/blob/4bd6d484bda46382a86e22fedd875e38af7dea47/ci/docker/runtime_functions.sh#L706-L730

If you use the latest commit from v1.x branch, the instruction have been simplified thanks to @Kh4L
You just need to specify -DUSE_TENSORRT=1 and -DONNX_NAMESPACE=onnx , for example
     cmake -DONNX_NAMESPACE=onnx -DUSE_TENSORRT=1 -GNinja ..
     ninja

Thanks very much.
I followed your instructions, but I failed when the make to nearly the 100%.
The bug reported this:
[100%] Linking CXX executable mxnet_unit_tests
/usr/bin/ld: warning: libcublas.so.10, needed by /usr/local/src/TensorRT-<IP_ADDRESS>/lib/libmyelin.so.1, may conflict with libcublas.so.9.1
/usr/bin/ld: CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o: undefined reference to symbol 'dposv_'
//usr/lib/x86_64-linux-gnu/liblapack.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
tests/CMakeFiles/mxnet_unit_tests.dir/build.make:427: recipe for target 'tests/mxnet_unit_tests' failed
make[2]: *** [tests/mxnet_unit_tests] Error 1
CMakeFiles/Makefile2:2288: recipe for target 'tests/CMakeFiles/mxnet_unit_tests.dir/all' failed
make[1]: *** [tests/CMakeFiles/mxnet_unit_tests.dir/all] Error 2
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2
It looks like the cublas conflicts in the make project. However, I have installed the OpenBlas in my /opt/OpenBLAS DIR, shold I have to make some change to the cmake. How can I fix the error?

You can specify cmake -DONNX_NAMESPACE=onnx -DUSE_TENSORRT=1 -GNinja -DOpenBLAS_Home=/opt/OpenBLAS ..

You can specify cmake -DONNX_NAMESPACE=onnx -DUSE_TENSORRT=1 -GNinja -DOpenBLAS_Home=/opt/OpenBLAS ..

Yes, I have tried what you said. But it still reported:
[ 97%] Linking CXX executable mxnet_unit_tests
/usr/bin/ld: warning: libcublas.so.10, needed by /usr/local/src/TensorRT-<IP_ADDRESS>/lib/libmyelin.so.1, may conflict with libcublas.so.9.1
/usr/bin/ld: CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o: undefined reference to symbol 'dposv_'
//usr/lib/x86_64-linux-gnu/liblapack.so.3: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
tests/CMakeFiles/mxnet_unit_tests.dir/build.make:427: recipe for target 'tests/mxnet_unit_tests' failed
make[2]: *** [tests/mxnet_unit_tests] Error 1
CMakeFiles/Makefile2:2288: recipe for target 'tests/CMakeFiles/mxnet_unit_tests.dir/all' failed
make[1]: *** [tests/CMakeFiles/mxnet_unit_tests.dir/all] Error 2
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2
I have searched in the google, but still cannot fix this bug.
I have followed the instructions in https://mxnet.apache.org/get_started/build_from_source#building-mxnet
to install OpenBLAS and LAPACK, but it still reported the error like this.
Can you help me to fix it? @leezu

It's not a bug in MXNet but your environment is broken. You can't use Cublas 9.1 and 10.0 at the same time


Laravel 5 support

Add support for Laravel 5. This means making the following system works:

[ ] Localized system/urls

Our localization system works using the following:

A modified Route class which provides the groupLocales method, which will take care of generating the route for each local within the closure.
A modified UrlGenerator class which overrides route to provide localized routes urls.
A modified Route class which contains data related to the locale it may represent (when used within a localized context).


Add support for pre-shared keys

Client and server should have the option to use out-of-band pre-shared keys rather than certificates.
https://commondatastorage.googleapis.com/chromium-boringssl-docs/ssl.h.html#Pre-shared-keys

Yes! I will add support for it.

As it turns out this is only for TLS1.2 and so not what we need... currently investigating how to add support for TLS1.3 which is used for QUIC.

We may need to wait for BoringSSL to add support /cc @davidben

@davidben is this something you support yet ? It's not clear to me atm

We don't currently support it in TLS 1.3. The problem is the original TLS 1.3 PSK mechanism was, IMO, defined wrong. It does this hash-matching thing, but that ties together your PSK configuration (managed by the caller), and TLS parameter selection (in many cases, library behavior).
Imported PSKs resolves this. My motivation there was, in fact, "someone is going to want PSKs someday and I want this specified before I need to implement it". 😄 But all the folks who wanted PSKs have since fizzled, so we never ended up implementing it. It's just sitting there as the plan of record, should we ever want to add it.

@davidben & @normanmaurer Any Updates on PSKs being supported by Netty? We currently have requirements to handle PSK with TLS 1.3.

@davidben Any update on this?


Adding alibi to mpt-7b_dolly_sft.yaml ?

I'd like to try the mpt-7b_dolly_sft.yaml config with alibi. Is this supported with llm-foundry?
For this config edit to the model section:

  # Model
  model:
    name: hf_causal_lm
    device: cpu
    pretrained: true
    pretrained_model_name_or_path: mosaicml/mpt-7b
    config_overrides:
      max_seq_len: ${max_seq_len}
      attn_config:
        attn_impl: triton
      alibi: true

I get:
Traceback (most recent call last):
  File "/llm-foundry/scripts/train/train.py", line 254, in <module>
    main(cfg)
  File "/llm-foundry/scripts/train/train.py", line 144, in main
    model = build_composer_model(cfg.model, tokenizer)
  File "/llm-foundry/scripts/train/train.py", line 67, in build_composer_model
    return COMPOSER_MODEL_REGISTRY[model_cfg.name](model_cfg, tokenizer)
  File "/llm-foundry/llmfoundry/models/hf/hf_causal_lm.py", line 55, in __init__
    raise ValueError(
ValueError: config does not have attribute "alibi" to override (alibi: True).

  # Model
  model:
    name: hf_causal_lm
    device: cpu
    pretrained: true
    pretrained_model_name_or_path: mosaicml/mpt-7b
    config_overrides:
      max_seq_len: ${max_seq_len}
      attn_config:
        attn_impl: triton
        alibi: true

alibi: true flag is an attn arg


Strategy for spawning & managing spot instances?

Taking this part.

@marpierc @smarru
Hello professors, I was trying to set up an autoscale group in AWS, but looks like I do not have permissions to do so. I could configure the settings, but the AWS wizard fails when "Creating launch configuration           Failure"
I am also unable to see the AutoScaling group dashboard which leads me to believe it is a permission issue.

The Autoscale group is setup and running in Ohio.
One question which popped in my mind regarding spot instance is, considering the fact that these can get shutdown anytime and the Autoscale group will try to get another EC2 instance -
We need a way to initialize this newly spawned EC2 instance with our application.

Use the userdata script
On Feb 2, 2017 2:22 AM, "Ameya Advankar"<EMAIL_ADDRESS>wrote:

The Autoscale group is setup and running in Ohio.
One question which popped in my mind regarding spot instance is,
considering the fact that these can get shutdown anytime and the Autoscale
group will try to get another EC2 instance -
We need a way to initialize this newly spawned EC2 instance with our
application.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/airavata-courses/spring17-laravel-portal/issues/7#issuecomment-276886049,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOOY4QtRpvWtsFAF9e7vXknwJkgpoQQ4ks5rYYQogaJpZM4LwI-c
.

FYI. For reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html


Shared secret for auto-renewable subscriptions

Hi,
Is it possible to use shared secret for auto-renewable subscriptions?
JSONObjectKey has 'password' key, but it seems currently it's not used.

Hey,
I actually dont know to be honest. I have never used auto-renewable subscriptions in my app before or worked with the StoreKit API for those.
Maybe google this, I think this tutorial talks about them
http://www.brianjcoleman.com/tutorial-receipt-validation-in-swift/
My apologises for that.

Thanks for the link!
Now it works perfectly in your project:
let payload = [JSONObjectKey.receiptData.rawValue: receiptData]
changed to:
let payload = [JSONObjectKey.receiptData.rawValue: receiptData, JSONObjectKey.password.rawValue: "sharedSecret"]

Hey,
Thanks very much for posting this. I will update the project in the future to include this. Happy coding

Is that all you need to do for shared secret or did you need to change anything else?
I was planning on maybe changing the code to something like this where you can pass in an optional shared secret in the validation method
func validateReceipt(forProductID: "SOMEID", sharedSecret: "SomeSharedSecret")....

and than I would change the payload bit to this
var payload = [JSONObjectKey.receiptData.rawValue: receiptData]

if let sharedSecret = sharedSecret {
        payload.updateValue(sharedSecret, forKey: JSONObjectKey.password.rawValue)
}

Yes, and maybe instead of updateValue just:
payload[JSONObjectKey.password.rawValue] = sharedSecret

Hey, thanks for the reply. I will push an update soon.
Thanks for the syntax tip, I know you can actually do it that way. I usually like to use the helper methods when working with arrays etc as I think at first glance it makes the code more readable.
Thanks again.

It will probably be tomorrow or something I want to read bit more about auto subscriptions. I am not sure that I am doing all the necessary check for them in the validation bit.

I updated the helper. It is also now returning the response so you can do additional checks such as checking expiry date etc. Have a look at the instructions again when you update, I changed the API a bit to make this more readable.
Thanks for the feedback.


Compile errors with fmt 10.1.0

Hi,
both HEAD and tag v1.2.0 do not build with fmt version 10.1.0. I stumbled over this due to Arch bumping its package, but this also occurs if I bump the version of fmt in etc/repos.json. It builds with 10.0.0, so this is not a major issue right now, but you will probably need to address this before updating fmt the next time.
The relevant error message is added below.
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/main/main.cpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/base_maps/entity_name.hpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/base_maps/entity_name_data.hpp:25:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/expression_ptr.hpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/function_map.hpp:21:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/linked_map.hpp:26:
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1577:63: error: implicit instantiation of undefined template
      'fmt::detail::type_is_unformattable_for<const nlohmann::basic_json<>, char>'
    type_is_unformattable_for<T, typename Context::char_type> _;
                                                              ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1810:23: note: in instantiation of function template specialization
      'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, const nlohmann::basic_json<>, 0>' requested here
        data_{detail::make_arg<is_packed, Context>(args)...} {
                      ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1828:10: note: in instantiation of function template specialization
      'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>,
      nlohmann::basic_json<>>::format_arg_store<const nlohmann::basic_json<>, const nlohmann::basic_json<>>' requested here
  return {args...};
         ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:130:48: note: in instantiation of function template specialization
      'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, const nlohmann::basic_json<>, const nlohmann::basic_json<>>'
      requested here
            auto fmsg = fmt::vformat(msg, fmt::make_format_args(args...));
                                               ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:93:13: note: in instantiation of function template specialization
      'Logger::FormatAndForward<const nlohmann::basic_json<> &, const nlohmann::basic_json<> &>' requested here
            FormatAndForward(nullptr,
            ^
/tmp/tmp3ltxf4ic/src/src/buildtool/main/main.cpp:475:17: note: in instantiation of function template specialization
      'Logger::Log<const nlohmann::basic_json<> &, const nlohmann::basic_json<> &>' requested here
        Logger::Log(LogLevel::Error,
                ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1555:45: note: template is declared here
template <typename T, typename Char> struct type_is_unformattable_for;
                                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1580:3: error: static assertion failed due to requirement 'formattable': Cannot
      format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1577:63: error: implicit instantiation of undefined template
      'fmt::detail::type_is_unformattable_for<nlohmann::basic_json<>, char>'
    type_is_unformattable_for<T, typename Context::char_type> _;
                                                              ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1810:23: note: in instantiation of function template specialization
      'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, 0>' requested here
        data_{detail::make_arg<is_packed, Context>(args)...} {
                      ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1828:10: note: in instantiation of function template specialization
      'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, std::basic_string<char>,
      std::basic_string<char>>::format_arg_store<nlohmann::basic_json<>, const std::basic_string<char>, const std::basic_string<char>>'
      requested here
  return {args...};
         ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:130:48: note: in instantiation of function template specialization
      'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, const std::basic_string<char>, const
      std::basic_string<char>>' requested here
            auto fmsg = fmt::vformat(msg, fmt::make_format_args(args...));
                                               ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:93:13: note: in instantiation of function template specialization
      'Logger::FormatAndForward<nlohmann::basic_json<> &, const std::basic_string<char> &, const std::basic_string<char> &>' requested here
            FormatAndForward(nullptr,
            ^
/tmp/tmp3ltxf4ic/src/src/buildtool/main/main.cpp:654:29: note: in instantiation of function template specialization
      'Logger::Log<nlohmann::basic_json<> &, const std::basic_string<char> &, const std::basic_string<char> &>' requested here
                    Logger::Log(LogLevel::Error,
                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1555:45: note: template is declared here
template <typename T, typename Char> struct type_is_unformattable_for;
                                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1580:3: error: static assertion failed due to requirement 'formattable': Cannot
      format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
4 errors generated.

It seems that {fmt} 10.x requires an explicit template specialization for formatting of types that can be formatted via operator<<(std::ostream& os, ...). This seems to be the case for nlohmann::basic_json<>. Adding the following code to main.cpp fixes the compiler error:
template <> struct fmt::formatter<nlohmann::basic_json<>> : ostream_formatter {};

I can create a PR for this addition or only patch it in the Arch Linux AUR build recipe. However, I'm not sure where to correctly place this template specialization.

I can create a PR for this addition or only patch it in the Arch Linux AUR build recipe. However, I'm not sure where to correctly place this template specialization.

The most natural place would probably be src/utils/cpp/json.hpp as you already suggested in your update of justbuild to 1.2.0 on AUR.


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
18 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
12 seconds
inspect-image

✅ Succeeded
11 seconds
deprecated-base-image-check

✅ Succeeded
28 seconds
clair-scan

✅ Succeeded
1 minute
clamav-scan

✅ Succeeded
13 seconds
sbom-json-check

✅ Succeeded
15 seconds
label-check

✅ Succeeded
14 seconds
optional-label-check

✅ Succeeded
5 seconds
show-summary


Add banner via upload

Feature Name: ICC Discord bot banner
Changes made:
uploaded banner
Issue num (must) - #4
Self Check(Tick After Making pull Request)

[x] This issue was assigned to me.
[x] One Change in one Pull Request
[x] My file is in proper folder
[x] I am following clean code and Documentation and my code is well linted with flake8.
[ ] I have updated requirements.txt

If issue was not assigned to you Please don't make a PR, It will be marked as invalid.

Discuss in discord server:- http://pragmaticprogrammer.in/discord
acknowledgement

[x] I have tested my code and distributing it under MIT license.

Merged ✨
Thank you for the contribution 😸


Blog post of OTel "getting started" survey

Blog post of OTel "getting started" survey
Assigned to Ana (not yet part of the OTel org)

Survey is published! https://opentelemetry.io/blog/2024/otel-get-started-survey/


Integrate GET Questions in Frontend

Closes: #784
Brief description of what is fixed or changed
We have get all questions in the the backend and also the questions has the upvote and downvote so i integrated in the frontend and added the UI for the question cards.
Types of changes

[x] As soon as someone visits on Q&A page, there should be the GET Questions call.
[x] The upvote functionality is integrated in frontend and downvote Also.

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes does not break the current system and it passes all the current test cases.

Screenshots
https://github.com/HITK-TECH-Community/Community-Website/assets/113225478/d098078e-4861-4694-9c62-b36b608f647b

Other Information
Please clear the test questions from mongodb database.

Yeah I added the loader in the question section by doing conditional rendering but its not showing and the toast message will get on upvote or when the user gets on the q&a page please specify.

@TejasNasre
For pt #1 please see the attached screenshot, i am not seeing the loader when get questions api call is happening.
For pt #2 We should see toast for all the api calls, getQuestions, upvotes and downvotes as well, so that user remains informed of anything wrong happening during the api call
https://github.com/HITK-TECH-Community/Community-Website/assets/44888949/b2315155-b806-4426-9c69-54e9f4c24350

Fix : Integrate-GET-Questions-in-Frontend
Type Of Changes

[x] loader when get questions api call is happening
[x] toast for all the api calls, getQuestions, upvotes and downvotes as well
[x] upvote and downvote both working

Screen Video
https://github.com/HITK-TECH-Community/Community-Website/assets/113225478/197e0a32-efed-49f6-8cdf-113d1c1f1e50

@TejasNasre  let me know asap if you are willing to work on https://github.com/HITK-TECH-Community/Community-Website/issues/885 this issue?

who is answering the questions.

who is answering the questions.

It can be answered by anyone but will appear only after admin's approval. As far as i remember that was implemented, let me check this its been a while i have checked this codebase

seems like api does exist but need a little tweak to have a flag isAdminApproved so will keep this integration on hold till the time the api gets fixed :)

I want your help - i recently worked on others repo and i raised some issue and they got merged but today in my forked repo i discard the commits unconditionally and the merge code is there on main but when I checked the PR all commits are deleted and some PR did not get merged yet and there commits also get  deleted so I will get points on GSSOC and what if I delete the fork repo and then then rebase it.

the points are only one the basis of labels assigned to the merged prs so yes, you will get the points for those prs

What if I delete the fork repo for this project and then rebase code and I got merged one PR so I will get points?

all merged prs with correct label will give you required points

Thanks @Kajol-Kumari For Helping

Hey @TejasNasre  i am assigning the levels based on the complexity of the issues only, if the required changes comes to be more than my expectation, i would be happy to revise the levels :)

Ok Thank 👍🏻


No such file or directory: 'prompts/execution.txt' (WSL2 env on Win11)

Description
I'm using v1.1.6-alpha
When i click the button "START PURSUING TASK", nothing happens in the UI
in the shell it says
Exception in thread Thread-2 (run_task):
Traceback (most recent call last):
  File "/home/mihai/miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/mihai/miniconda3/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mihai/Agent-LLM/AgentLLM.py", line 316, in run_task
    result = self.run(task=task["task_name"], prompt="execution")
  File "/home/mihai/Agent-LLM/AgentLLM.py", line 97, in run
    prompt = CustomPrompt().get_prompt(prompt)
  File "/home/mihai/Agent-LLM/CustomPrompt.py", line 17, in get_prompt
    with open(os.path.join("prompts", f"{prompt_name}.txt"), "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'prompts/execution.txt'

Steps to Reproduce the Bug

add OPENAI_API_BASE_URL=http://localhost:5001 to .env file
start app.py
start frontenv with npp run dev
open http://[localhost:3000/agent](http://localhost:3000/agent)
define new agent
configure agent with Agent Provider = openai and model gpt-3.5-turbo
navigate to tab Control Agent [agent name]
give the task of write the text 'Hello Mihai!' into a txt file named 'testing.txt'

Expected Behavior
After clicking the button to appear a message that say the task is started or some other kind of information to now it started
Actual Behavior
nothing happens in the UI.
only in terminal window can see it has errors
Additional Context / Screenshots
I'm running text-generation-webui with the command
python server.py --model TheBloke_GPT4All-13B-snoozy-GPTQ --disk --model_type Llama --listen --no-stream --auto-devices --verbose --xformers --sdp-attention --extensions api openai --bf16 --threads 4 --n_batch 128 --pre_layer 32 --gpu-memory 4 4 --chat --settings settings.json --cpu-memory 26 --wbits 4 --groupsize 128
If i try the CHAT WITH AGENT tab and write the same text, the response given is:
  File "/home/mihai/.local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/mihai/Agent-LLM/app.py", line 210, in chat
    response = agent.run(prompt.prompt, max_context_tokens=500, commands_enabled=False)
  File "/home/mihai/Agent-LLM/AgentLLM.py", line 113, in run
    self.response = self.CFG.instruct(formatted_prompt)
  File "/home/mihai/Agent-LLM/provider/openai.py", line 39, in instruct
    max_tokens=int(self.MAX_TOKENS - len(prompt)),
TypeError: unsupported operand type(s) for -: 'str' and 'int'

After i modify line 26 and 39  int the /home/mihai/Agent-LLM/provider/openai.py
from this
max_tokens=int(self.MAX_TOKENS - len(prompt)),
to this
max_tokens=int(self.MAX_TOKENS) - len(prompt),
then restart app.py, it works as exected, and the result is

Operating System

[ ] Microsoft Windows
[ ] Apple MacOS
[X] Linux
[ ] Android
[ ] iOS
[ ] Other

Python Version

[ ] Python <= 3.9
[X] Python 3.10
[ ] Python 3.11

Environment Type - Connection

[X] Local
[ ] Remote

Environment Type - Container

[ ] Using Docker
[X] Not Using Docker

Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of Agent-LLM.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

Pull latest, the file name was wrong and has been fixed.


Binding ObservableCollection updates one time

Hi,
I have problem with binding ObservableCollection, it reads values ones, not firing on collection modified.
In xaml code bellow I have GridView binded to ObservableCollection and GridView items having another ObservableCollections bindings.
The first TextBox have generic binding, and it works fine, value updates after collection modified
 <TextBlock Text="{Binding prop1.collection.Count}"></TextBlock>
But MultiBindingBehavior updates values ones, and not firing update on collection modify.
<behaviors:MultiBindingItem Value="{Binding prop1.collection}"/>
<GridView ItemsSource="{Binding prop.collection }">
            <GridView.ItemTemplate>
                <DataTemplate>
                    <StackPanel>
                        <TextBlock Text="{Binding prop1.collection.Count}"></TextBlock>
                        <TextBlock>
                             <interactivity:Interaction.Behaviors>
                                <behaviors:MultiBindingBehavior PropertyName="Text" Converter="{StaticResource MultiBindingValueConverter}" >
                                    <behaviors:MultiBindingItem Value="{Binding prop1.collection}"/>
                                    <behaviors:MultiBindingItem Value="{Binding prop2.collection}"/>
                                </behaviors:MultiBindingBehavior>
                            </interactivity:Interaction.Behaviors>
                        </TextBlock>
                    </StackPanel>
                </DataTemplate>
            </GridView.ItemTemplate>
</GridView>

P.S> I found if I am binding MultiBindingItem to ObservableCollection Count property it works, solution in this case will look a bit hacky, with unwanted extra items.
<behaviors:MultiBindingItem Value="{Binding prop1.collection.Count}"/> - to fire update event
<behaviors:MultiBindingItem Value="{Binding prop1.collection}"/> - to get actual data converter
Please, let me know if there is standard solution.
Regards,
Maksim

Hi Maksim, your code is a bit incomplete for me to test on my side, can you please provide the viewmodels for the binded properties? Thanks!

Hi Pedro.
Take a look test app.
App1.zip
Add/Remove buttons for adding and removing ObservableCollection items.
Regards,
Maksim

Thanks for the sample Zaduvalo, I now have the generic gist!
You have to remember there are two interfaces for notifications, INotifyPropertyChanged for property value changes and INotifyCollectionChanged for collection item changes.
When you bind to data.Count, you are effectively monitoring the Count property for changes and those are correctly raised, however, when you bind to data, that's monitoring for the data property changes and those are not raised as the instance didn't change, only the items inside the instance!
This is expected behavior from bindings and not a bug, so not much I can on my end I'm afraid...


blue screen with xrdp

after installing, with all files in right locations, xrdp comes up with a solid blue screen with a window claiming it starts to connect, then it fails and just says, "connection problem, giving up, some problem"

Show us the log as well as #952.

Yes i see #952
My log is different, first of all there is no [ERROR] log in my case.

Okay, then, can you show me xrdp-sesman.log as well as #952?

Exactly the same blue screen for me, I am installing from source - branch: devel, commit ae1514c1679e87edfdeba5ca3eb2b3659ef44b1a. Follow the instruction from https://github.com/neutrinolabs/xrdp/wiki/Building-on-Debian-8.
Attached logs
xrdp.log
xrdp-sesman.log

Sorry for necrobumping this old thread, but I'm facing the very same issue on manjaro, tried even with the very latest version of xrdp and xorgxrdp as for today (0.9.13).
xrdp.log
[20200311-11:00:09] [INFO ] Cannot find keymap file /etc/xrdp/km-0001040a.ini
[20200311-11:00:09] [INFO ] Loading keymap file /etc/xrdp/km-0000040a.ini
[20200311-11:00:09] [WARN ] local keymap file for 0x0001040a found and doesn't match built in keymap, using local keymap file
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: connecting to sesman ip <IP_ADDRESS> port 3350
[20200311-11:00:13] [INFO ] xrdp_wm_log_msg: sesman connect ok
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
[20200311-11:00:13] [DEBUG] return value from xrdp_mm_connect 0
[20200311-11:00:13] [INFO ] xrdp_wm_log_msg: login successful for display 10
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: started connecting
[20200311-11:00:17] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:00:21] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:00:24] [DEBUG] Closed socket 19 (AF_UNIX)
...
[20200311-11:03:41] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:03:44] [DEBUG] xrdp_wm_log_msg: connection problem, giving up
[20200311-11:03:44] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:03:44] [DEBUG] xrdp_wm_log_msg: some problem
[20200311-11:03:44] [DEBUG] xrdp_mm_module_cleanup
[20200311-11:03:44] [DEBUG] Closed socket 18 (AF_INET <IP_ADDRESS>:50424)

xrdp-sesman.log
[20200311-11:00:24] [ERROR] X server for display 10 startup timeout
[20200311-11:00:24] [ERROR] another Xserver might already be active on display 10 - see log
[20200311-11:00:24] [DEBUG] aborting connection...
[20200311-11:00:24] [CORE ] window manager (pid 121210) did exit, cleaning up session
[20200311-11:00:24] [INFO ] calling auth_stop_session and auth_end from pid 121209
[20200311-11:00:24] [DEBUG] cleanup_sockets:
[20200311-11:00:24] [DEBUG] cleanup_sockets: deleting /tmp/.xrdp/xrdp_chansrv_socket_10
[20200311-11:00:24] [DEBUG] cleanup_sockets: deleting /tmp/.xrdp/xrdpapi_10
[20200311-11:00:24] [DEBUG] cleanup_sockets: failed to delete /tmp/.xrdp/xrdpapi_10
[20200311-11:00:24] [INFO ] ++ terminated session:  username angel, display :10.0, session_pid 121209, ip <IP_ADDRESS>:59748 - socket: 12

Sorry for necrobumping this old thread, but I'm facing the very same issue on manjaro, tried even with the very latest version of xrdp and xorgxrdp as for today (0.9.13). xrdp.log
[20200311-11:00:09] [INFO ] Cannot find keymap file /etc/xrdp/km-0001040a.ini
[20200311-11:00:09] [INFO ] Loading keymap file /etc/xrdp/km-0000040a.ini
[20200311-11:00:09] [WARN ] local keymap file for 0x0001040a found and doesn't match built in keymap, using local keymap file
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: connecting to sesman ip <IP_ADDRESS> port 3350
[20200311-11:00:13] [INFO ] xrdp_wm_log_msg: sesman connect ok
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
[20200311-11:00:13] [DEBUG] return value from xrdp_mm_connect 0
[20200311-11:00:13] [INFO ] xrdp_wm_log_msg: login successful for display 10
[20200311-11:00:13] [DEBUG] xrdp_wm_log_msg: started connecting
[20200311-11:00:17] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:00:21] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:00:24] [DEBUG] Closed socket 19 (AF_UNIX)
...
[20200311-11:03:41] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:03:44] [DEBUG] xrdp_wm_log_msg: connection problem, giving up
[20200311-11:03:44] [DEBUG] Closed socket 19 (AF_UNIX)
[20200311-11:03:44] [DEBUG] xrdp_wm_log_msg: some problem
[20200311-11:03:44] [DEBUG] xrdp_mm_module_cleanup
[20200311-11:03:44] [DEBUG] Closed socket 18 (AF_INET <IP_ADDRESS>:50424)

xrdp-sesman.log
[20200311-11:00:24] [ERROR] X server for display 10 startup timeout
[20200311-11:00:24] [ERROR] another Xserver might already be active on display 10 - see log
[20200311-11:00:24] [DEBUG] aborting connection...
[20200311-11:00:24] [CORE ] window manager (pid 121210) did exit, cleaning up session
[20200311-11:00:24] [INFO ] calling auth_stop_session and auth_end from pid 121209
[20200311-11:00:24] [DEBUG] cleanup_sockets:
[20200311-11:00:24] [DEBUG] cleanup_sockets: deleting /tmp/.xrdp/xrdp_chansrv_socket_10
[20200311-11:00:24] [DEBUG] cleanup_sockets: deleting /tmp/.xrdp/xrdpapi_10
[20200311-11:00:24] [DEBUG] cleanup_sockets: failed to delete /tmp/.xrdp/xrdpapi_10
[20200311-11:00:24] [INFO ] ++ terminated session:  username angel, display :10.0, session_pid 121209, ip <IP_ADDRESS>:59748 - socket: 12

did you find a solution for that ?


Is Colossus under active development?

I am evaluating mail list managers for a large client. Is Colossus being actively developed and supported? If we have installation difficulties, can we get help?
Thanks!

Thanks for your reply. Is there any support for SAML2 authentication in the app?

Not actively being development but actively being used
I still use Colossus on a daily basis using this code as it is
No SAML2 auth support is planned for the time being

This should be under active development it looks a lot better compared to other django email packages out there. Vitor, as a fan of your blog and youtube channel, you should make a tutorial / videos on how to actually use this app. I think it would help out a lot of people!

Is there any roadmap ? we can contribute also, I have some ideas actually.

We forked colossus, and are building in SAML support. Let me know if you are interested in pulling it.

Hi @gcallah Are you able to provide the SAML support on your fork?
We're planning to use that on scale and would appreciate your help if you've built that already?

Hi Zaanu,
We did this project for the city of New York, and they have taken over the forked codebase. I am happy to help / intermediate here: Joel, and Martin, is the forked colossus repo still under development in your department? Is it possible to do remerge with the main branch? Or is there another way to offer SAML to colossus users?
Thanks,
Gene

On Dec 28, 2020, at 5:03 AM, Zaanu<EMAIL_ADDRESS>wrote:
Hi @gcallah https://github.com/gcallah Are you able to provide the SAML support on your fork?
We're planning to use that on scale and would appreciate your help if you've built that already?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/vitorfs/colossus/issues/33#issuecomment-751658507, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLNZOGMDAS2GLKUKJ7KAI3SXBJYXANCNFSM4I7J6YPQ.

@zaanu @gcallah if you guys submit a pull request I will be glad to review and help integrate in the code base

Well, I really hope the NYC people respond here, since I turned the fork over to them some months ago.

On Dec 28, 2020, at 9:04 AM, Vitor Freitas<EMAIL_ADDRESS>wrote:
@zaanu https://github.com/zaanu @gcallah https://github.com/gcallah if you guys submit a pull request I will be glad to review and help integrate in the code base
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/vitorfs/colossus/issues/33#issuecomment-751722903, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLNZOAXIJIJ6YMNMXNY37DSXCF6NANCNFSM4I7J6YPQ.

Well, I really hope the NYC people respond here, since I turned the fork over to them some months ago.
…
On Dec 28, 2020, at 9:04 AM, Vitor Freitas @.***> wrote: @zaanu https://github.com/zaanu @gcallah https://github.com/gcallah if you guys submit a pull request I will be glad to review and help integrate in the code base — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLNZOAXIJIJ6YMNMXNY37DSXCF6NANCNFSM4I7J6YPQ.

Hi there, any update on this?
Thanks

Perhaps Martin can help you here.
It has been several years since I have been involved in this project.

On Dec 23, 2021, at 4:06 PM, Michele Saba @.***> wrote:
Well, I really hope the NYC people respond here, since I turned the fork over to them some months ago.
… x-msg://15/#
On Dec 28, 2020, at 9:04 AM, Vitor Freitas @.***> wrote: @zaanu https://github.com/zaanu https://github.com/zaanu https://github.com/zaanu @gcallah https://github.com/gcallah https://github.com/gcallah https://github.com/gcallah if you guys submit a pull request I will be glad to review and help integrate in the code base — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment) https://github.com/vitorfs/colossus/issues/33#issuecomment-751722903>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLNZOAXIJIJ6YMNMXNY37DSXCF6NANCNFSM4I7J6YPQ https://github.com/notifications/unsubscribe-auth/ACLNZOAXIJIJ6YMNMXNY37DSXCF6NANCNFSM4I7J6YPQ.
Hi there, any update on this?
Thanks
—
Reply to this email directly, view it on GitHub https://github.com/vitorfs/colossus/issues/33#issuecomment-1000523368, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLNZOAXAIYHCROZSVHHZDDUSOFLXANCNFSM4I7J6YPQ.
You are receiving this because you were mentioned.


Upgrade libs to prevent security issues - CVE

It was identified by OWASP dependency-checker tool that exists vulnerabilities in odftoolkit-odftoolkit-0.9.0-RC2.
To prevent this security issues, please, upgrade to the latest version the following libraries (see image below):

Interesting! How did you trigger the OWASAP dependency-checker? Is it available as a GitHub bot?
There seems to be within the Simple API release an example ZIP, which is not created automatically and therefore contains an old version of Simple API.
I will take care of it in the next and final release of the deprecated Simple API.
Thanks for the report.
Svante

I was able to install 6.1.0 of
https://jeremylong.github.io/DependencyCheck/dependency-check-cli/index.html
on Linux and create an HTML report
running on the root directory 'dependency-check.sh -s .'
Nice Tool! Thanks again!

Hello Svante!
 :)
Sorry for the delay.
Normally, when I use a framework/lib, I trigger the OWASP dependency-checker to
verify the existence of security issues or vulnerabilities and report to the project holder.
I'm anxious for the new release and I hope to use on my maven repository soon!
Keep the good work!
If you need help, you are free to ask.
Thanks!
João Filipe

assuming this is fixed now


Fix NPE where the country code isn't specified in the sellAsset of Sep38Service

PR Checklist
PR Structure

[ ] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[ ] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[ ] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[ ] This PR adds tests for the most critical parts of the new functionality or fixes.

What
fix npe
Why
fix npe
Known limitations
n/a

Reference Server Preview is available here:https://anchor-ref-pr929.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr929.previews.kube001.services.stellar-ops.com/

All other places check that it's not null

Reference Server Preview is available here:https://anchor-ref-pr929.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr929.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr929.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr929.previews.kube001.services.stellar-ops.com/


On the otbr side, the SRP client registration information was lost after the the otbr-agent restarted.

On the otbr side, the SRP client registration information was lost after the the otbr-agent restarted.
There may be two ways to solve this problem:
1 The otbr-agent save the SRP registration information persistently and recover after restart.
2 SRP client is responsible to re-register. SRP client need to know when to re-register. Maybe otbr-agent can use a new port which is diffrent from previous one to servers SRP server and SRP client should re-register if there is a new SRP server.
If there is a mechanism to ensure that the SRP server port must be different from the previous one? If so, I think the second option works.
ot-br-posix git commit id: 5be35949c6f1fd4cc3a8e63169cec6a2211135a4

Which address mode is the SRP server running?
/**
 * This enumeration represents the address mode used by the SRP server.
 *
 * Address mode specifies how the address and port number are determined by the SRP server and how this info is
 * published in the Thread Network Data.
 *
 */
typedef enum otSrpServerAddressMode
{
    OT_SRP_SERVER_ADDRESS_MODE_UNICAST = 0, ///< Unicast address mode.
    OT_SRP_SERVER_ADDRESS_MODE_ANYCAST = 1, ///< Anycast address mode.
} otSrpServerAddressMode;

https://github.com/openthread/openthread/blob/b2132a903162be76eb4ea8a107e82c4e0737525f/include/openthread/srp_server.h#L145-L156
We are already selecting a different port (option 2) under unicast mode:
void Server::SelectPort(void)
{
    mPort = kUdpPortMin;

#if OPENTHREAD_CONFIG_SRP_SERVER_PORT_SWITCH_ENABLE
    {
        Settings::SrpServerInfo info;

        if (Get<Settings>().Read(info) == kErrorNone)
        {
            mPort = info.GetPort() + 1;
            if (mPort < kUdpPortMin || mPort > kUdpPortMax)
            {
                mPort = kUdpPortMin;
            }
        }
    }
#endif

    LogInfo("Selected port %u", mPort);
}

https://github.com/openthread/openthread/blob/b2132a903162be76eb4ea8a107e82c4e0737525f/src/core/net/srp_server.cpp#L513-L533

We can't use a different port in anycast mode since all SRP server need to share the same port.

The address mode of the SRP server is OT_SRP_SERVER_ADDRESS_MODE_UNICAST .

In this mode, SRP server select server port between kUdpPortMin(53535) and kUdpPortMax(default 53554). When selecting a new port, next to the current port will be use. My question is , is the current port be saved? Where is it?
Sometimes the SRP server port changes after otbr-agent restart, sometimes id doesn't.

If no SRP service has been registered to the SRP server, the server won't select a new port.

#if OPENTHREAD_CONFIG_SRP_SERVER_PORT_SWITCH_ENABLE
        if (!mHasRegisteredAnyService && (mAddressMode == kAddressModeUnicast))
        {
            Settings::SrpServerInfo info;

            mHasRegisteredAnyService = true;
            info.SetPort(GetSocket().mSockName.mPort);
            IgnoreError(Get<Settings>().Save(info));
        }
#endif

https://github.com/openthread/openthread/blob/b2132a903162be76eb4ea8a107e82c4e0737525f/src/core/net/srp_server.cpp#L479-L488
2. The server port is saved in settings, keyed by OT_SETTINGS_KEY_SRP_SERVER_INFO
// @superwhd

OK. Thank you.


Improve component data flexibility

This PR builds upon the 'component data' functionality by allowing you to pass inline data in addition to a reference to a data file (json), instead of being able to only do one or the other:
{% component 'component-name', 'some-data.json', { override: 'something' } %}
Changes:

Add support for passing inline data in addition to a file reference
Update readme to reflect this

@wolfpilot: good call, example added to home template!


Toggling  Picture and Avatar Configuration switches have no effect

Description
Toggling the Avatar switches in admin/config URL does not change the configuration as requested: reloading the page moves the switches back to their original position.
I verified the request is sent to the server, the request is also logged by the server:

2023/01/04 00:04:41 [63b4b489] router: completed POST /admin/config for <IP_ADDRESS>:44014, 200 OK in 2.6ms @ admin/config.go:193(admin.ChangeConfig)

BUT there's no change in the database (PostgreSQL) and reloading the page shows it (switches are back to their original position)
Gitea Version
1.19.0+dev-264-g32e863439 P
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
from build tree. Cannot reproduce on try.gitea.io only because I'm not admin there, but if you grant me admin (strk) I can try
Database
PostgreSQL

Bug seems to be in func SetSetting in models/system/setting.go whereas the upsertSettingValue function is ONLY invoked when the key is not already cached in the settings cache. Sounds wrong to only conditionally upsert...

@lunny git blame tells it was you the last to touch that SetSetting function

I Will take a look at it today. Thank you for your report.

The bug is possibly rooted in PR #22295 there's evidently an hole in testsuite regarding caches


Add Trino specific implementation of deduplicate()

This is a:

[ ] bug fix PR with no breaking changes — please ensure the base branch is main
[x] new functionality — please ensure the base branch is the latest dev/ branch
[ ] a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

Checklist

[ ] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)

[ ] BigQuery
[ ] Postgres
[ ] Redshift
[ ] Snowflake

[ ] I followed guidelines to ensure that my changes will work on "non-core" adapters by:

[ ] dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
[ ] using the limit_zero() macro in place of the literal string: limit 0
[ ] using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP

[ ] I have updated the README.md (if applicable)
[ ] I have added tests & descriptions to my models (and macros if applicable)
[ ] I have added an entry to CHANGELOG.md

Very impressive that you were able to figure out how to do deduplication across many different databases 🏅
I want to hold off on merging this since we've currently only included adapter-specific macros for databases that we have in CI (Postgres, Redshift, BigQuery, and Snowflake).
In the meantime, the closest appropriate repo I can find is this: https://github.com/lalalilo/athena-utils
It's an important conversation to determine how best to handle cases like this, and hopefully this discussion will help hash out a fruitful path:
https://github.com/dbt-labs/dbt-utils/discussions/590

Unfortunately, I'm pretty sure Athena doesn't support the WITH TIES syntax as it's still working on an Presto 0.217. There doesn't seem to be a Trino specific utils package.

Ah, I see.
I am going to apply a wontfix label for now, but definitely willing to revisit this pending the discussion in https://github.com/dbt-labs/dbt-utils/discussions/590
Far from the perfect solution, but in the meantime folks that need this can copy the relevant definition into their project:
{#
-- This is the ANSI SQL Standard way of doing this operation, however, it is so poorly
-- supported accross other SQL engines that it cannot realistically be the default!
#}
{%- macro _ansi__deduplicate(relation, partition_by, order_by) -%}

    select *
    from {{ relation }}
    order by row_number() over (
        partition by {{ partition_by }}
        order by {{ order_by }}
    ) fetch first row with ties

{%- endmacro -%}

{# Trino correctly supports the ANSI SQL Standard so we can use the ANSI implementation #}
{%- macro trino__deduplicate(relation, partition_by, order_by) -%}

    {{ return(dbt_utils._ansi__deduplicate(relation, partition_by, order_by )) }}

{%- endmacro -%}


Players can load and play unused prompts directly via url, even when not logged in

see https://wikispeedruns.com/play/125

Good first issue as an intro to prompt types and access privileges

I think this issue is resolved now


Update CLI env variable names

Description
With https://github.com/Shopify/cli/pull/1789, the CLI will deprecate some env var names. While they will still be present so previous apps won't break, we should migrate new apps as soon as possible to the new format.
This PR adds support for the new variable names, but falls back to the current ones so it won't affect existing apps.

[x] I have added tests that prove my fix is effective or that my feature works.
[x] I have added a changelog line.

We've decided to close this for now since we might want to take a slightly different approach in the future!