🐛 [Bug] torch_tensorrt::torchscript::compile gets stuck; bug caused by elimination exception

Bug Description
after calling
auto trt_mod = torch_tensorrt::torchscript::compile(module, compile_settings);
the process gets stuck in an infinite(?) loop. I can also observe that the GPU load drops back to 0% after about 1s.
According to this link: https://github.com/pytorch/TensorRT/pull/1409 the issue should already have been fixed.
Error message
1   __memmove_avx_unaligned                                                                                0x7fff79289cc1
2   std::vectortorch::jit::Use::_M_erase(__gnu_cxx::__normal_iterator<torch::jit::Use *, std::vectortorch::jit::Use>)                                                                                0x7fffab48412f
3   torch::jit::Value::replaceFirstUseWith(torch::jit::Value *)                                                                                0x7fffab46ff5d
4   torch::jit::Value::replaceAllUsesWith(torch::jit::Value *)                                                                                0x7fffab46ffcb
5   torch::jit::EliminateExceptions(torch::jit::Block *)                                                                                0x7fffab63c3c9
6   torch::jit::EliminateExceptions(std::shared_ptrtorch::jit::Graph&)                                                                                0x7fffab63c999
7   torch_tensorrt::core::lowering::LowerGraph(std::shared_ptrtorch::jit::Graph&, std::vectorc10::IValue&, torch_tensorrt::core::lowering::LowerInfo)                                                                                0x7fffd7426b0d
8   torch_tensorrt::core::lowering::Lower(torch::jit::Module const&, std::string, torch_tensorrt::core::lowering::LowerInfo const&)                                                                                0x7fffd742a181
9   torch_tensorrt::core::CompileGraph(torch::jit::Module const&, torch_tensorrt::core::CompileSpec)                                                                                0x7fffd732b5a8
10  torch_tensorrt::torchscript::compile(torch::jit::Module const&, torch_tensorrt::torchscript::CompileSpec)                                                                                0x7fffd7313a04
11  ModelLoader::optimizeWithTensorRT                                                                                modelloader.cpp      266  0x5ad43c
12  InferenceDisplay::<lambda()>::<lambda()>::operator()                                                                                inferencedisplay.cpp 1330 0x58c996
13  std::_Function_handler<void(), InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>::<lambda()>>::_M_invoke(const std::_Any_data &)                                                   std_function.h       316  0x58c996
14  std::function<void ()>::operator()() const                                                                                std_function.h       706  0x5cbcca
15  errorwrapper::loading(std::function<void ()>)                                                                                errorwrapper.cpp     11   0x5cbcca
16  InferenceDisplay::<lambda()>::operator()                                                                                inferencedisplay.cpp 1333 0x58e127
17  QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>>::call                                                 qobjectdefs_impl.h   146  0x58e127
18  QtPrivate::Functor<InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>, 0>::call<QtPrivate::List<>, void>                                                                            qobjectdefs_impl.h   256  0x58e127
19  QtPrivate::QFunctorSlotObject<InferenceDisplay::InferenceDisplay(QWidget *, DataController&)::<lambda()>, 0, QtPrivate::List<>, void>::impl(int, QtPrivate::QSlotObjectBase *, QObject *, void * *, bool *) qobjectdefs_impl.h   439  0x58e127
20  QMetaObject::activate(QObject *, int, int, void * *)                                                                                0x7fff7a163f8f
...
Expected behavior
successful torch-tensorrt optimization of a torchscript model
Environment

Torch-TensorRT Version: v1.3.0
PyTorch Version : 1.13.0
OS: Linux
PyTorch : libtorch 1.13+cu117
CUDA version: 11.7
cudnn version: <IP_ADDRESS>
TensorRT version: <IP_ADDRESS>

Can you provide us a reproducer script with the model for us to investigate?

So far I have tested it only with the latest release torch-tensorrt==1.3.
With some older libraries the tensorrt optimization was running fine (CUDA11.3, cudnn <IP_ADDRESS>, libtorch-1.10.2+cu113, TensorRT-<IP_ADDRESS> and the torch-tensorrt for 1.10.2).
I have built a minimal example with a pretrained model provided by pytorch to reproduce the issue:
1.) jupyter notebook to get torchscript model:
`import torch
https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/
model = torch.hub.load('pytorch/vision:v0.10.0', 'deeplabv3_resnet50', pretrained=True)
model.eval()
model_scripted = torch.jit.script(model) # Export to TorchScript
model_scripted.save('model_scripted.pth') # Save`
2.) TensorRT-app.cpp:
`#include <torch/script.h>
#include <torch_tensorrt/torch_tensorrt.h>
#include
#include
int main() {
//Paths
std::string path = "/PATH/TO/SCRIPTED/MODEL";
std::string path_save = "/PATH/WHERE/OPTIMIZED/MODEL/IS/SAVED";

// 1) load the model
torch::jit::Module module;

try {
// Deserialize the ScriptModule from a file using torch::jit::load().
module = torch::jit::load(path);
}
catch (const c10::Error &e) {
std::cerr << "Error loading the model\n";
return -1;
}
std::cout << "Scripted Model successfully loaded!\n";

module.to(torch::kCUDA); //run on GPU
module.eval();

// 3) Compile using Torch-TensorRT
//Example input
auto example_input = torch_tensorrt::Input(std::vector<int64_t>{1, 3, 224, 224}, torch::kHalf);
auto compile_settings = torch_tensorrt::torchscript::CompileSpec({example_input});
// FP16 execution
compile_settings.enabled_precisions = {torch::kHalf};

// Compile module to tensorrt
std::cout << "Its compiling......\n";
auto trt_mod = torch_tensorrt::torchscript::compile(module, compile_settings);

//TensorRT conversion successful
std::cout << "Conversion from Scripted Model to TensorRT Model was successful!\n";
std::cout << "TensorRT Model is saved at: " << path_save << "\n";

// Save TensorRT model for later
trt_mod.save(path_save);

}`
3.) CMakeLists.txt:
`cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(TensorRT-app)
add_executable(${PROJECT_NAME}
TensorRT-app.cpp)
check for CUDA
find_package(CUDA QUIET)
if (CUDA_FOUND)
# assumptions:
# CUDA in /usr/local/cuda
# cuDNN in /usr/local/cudnn
# TensorRT in /usr/local/tensorrt
# LibTorch in /usr/local/libtorch
# Torch-TensorRT in /usr/local/torch-tensorrt
# 1) cudNN
if(DEFINED ENV{CUDNN_ROOT})
  set(CUDNN_ROOT $ENV{CUDNN_ROOT})
else()
  set(CUDNN_ROOT /usr/local/cudnn)
endif()

set(CUDNN_INCLUDE_PATH ${CUDNN_ROOT}/include)
set(CUDNN_LIBRARY_PATH ${CUDNN_ROOT}/lib)
link_libraries(${CUDNN_LIBRARY_PATH})
link_directories(${CUDNN_LIBRARY_PATH})

# 2) TensorRT
if(DEFINED ENV{TENSORRT_ROOT})
  set(TENSORRT_ROOT $ENV{TENSORRT_ROOT})
else()
  set(TENSORRT_ROOT /usr/local/tensorrt)
endif()

find_path(TENSORRT_INCLUDE_DIR NvInfer.h
  HINTS ${TENSORRT_ROOT} ${CUDA_TOOLKIT_ROOT_DIR}
  PATH_SUFFIXES include)
MESSAGE(STATUS "Found TensorRT headers at ${TENSORRT_INCLUDE_DIR}")
find_library(TENSORRT_LIBRARY_INFER nvinfer
  HINTS ${TENSORRT_ROOT} ${TENSORRT_BUILD} ${CUDA_TOOLKIT_ROOT_DIR}
  PATH_SUFFIXES lib lib64 lib/x64)
find_library(TENSORRT_LIBRARY_INFER_PLUGIN nvinfer_plugin
  HINTS  ${TENSORRT_ROOT} ${TENSORRT_BUILD} ${CUDA_TOOLKIT_ROOT_DIR}
  PATH_SUFFIXES lib lib64 lib/x64)
set(TENSORRT_LIBRARY ${TENSORRT_LIBRARY_INFER} ${TENSORRT_LIBRARY_INFER_PLUGIN})
MESSAGE(STATUS "Found TensorRT libs at ${TENSORRT_LIBRARY}")
find_package_handle_standard_args(
  TENSORRT DEFAULT_MSG TENSORRT_INCLUDE_DIR TENSORRT_LIBRARY)
if(NOT TENSORRT_FOUND)
  message(ERROR "Cannot find TensorRT library.")
endif()

endif()
3) Libtorch
add_compile_definitions(CMAKE_TORCH_ENABLED)
find_package(Torch REQUIRED PATHS /usr/local/libtorch/share/cmake/Torch)
#find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 17)
if (CUDA_FOUND)
if(DEFINED ENV{TORCH_TENSORRT_ROOT})
    set(TORCH_TENSORRT_ROOT $ENV{TORCH_TENSORRT_ROOT})
else()
    set(TORCH_TENSORRT_ROOT /usr/local/torch-tensorrt)
endif()

# 4) Torch-TensorRT
include_directories(${TORCH_TENSORRT_ROOT}/include)
link_directories(${TORCH_TENSORRT_ROOT}/lib)

# 5) Linking
target_link_libraries(${PROJECT_NAME}
    /usr/local/cudnn/lib/libcudnn.so.8
    )

set(CMAKE_CUDA_COMPILER /usr/local/cuda/bin/nvcc)
set(TORCH_TENSORRT_LIBRARY ${TORCH_TENSORRT_ROOT}/lib/libtorchtrt.so)
target_link_libraries(${PROJECT_NAME}
${TENSORRT_LIBRARY}
${TORCH_TENSORRT_LIBRARY}
)
add_compile_definitions(CMAKE_CUDA_AVAILABLE)
endif()
target_link_libraries(${PROJECT_NAME} ${TORCH_LIBRARIES})
`
4.) file to build app build.sh:
mkdir -p build cd build cmake ../src -G Ninja -DCMAKE_C_COMPILER=gcc-11 -DCMAKE_CXX_COMPILER=g++-11 -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DFORCE_COLORED_OUTPUT=ON -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Debug  -DCMAKE_CXX_FLAGS="$*" cmake --build . cd ..

So far I have tested it only with the latest release torch-tensorrt=1.3.0
With some older libraries the tensorrt optimization was running fine (CUDA11.3, cudnn <IP_ADDRESS>, libtorch-1.10.2+cu113, TensorRT-<IP_ADDRESS> and the torch-tensorrt for 1.10.2).
I have built a minimal example with a pretrained model provided by pytorch to reproduce the issue:
tensorrt_api.zip

Hi @peri044,
could you reproduce the issue with the provided files?

Torch-TensorRT works for me when I follow this tutorial (https://developer.nvidia.com/blog/accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt/) . But I got the same issues with same traceback for my models. Any ideas to fix this bug? @bjaeger1 @peri044
I use pip to install the related packages as follow:
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113 ~/downloads/torch_tensorrt-1.2.0-cp38-cp38-linux_x86_64.whl nvidia-pyindex nvidia-tensorrt==<IP_ADDRESS>
Environment

Python Version : 3.8.16
OS: Ubuntu 20.04.5 LTS
Torch-TensorRT Version: v1.2.0
PyTorch Version : 1.12.1+cu113
cudnn version: 8.3.2
TensorRT version: <IP_ADDRESS>

@bobby-chiu
I provided @peri044 the files to reproduce the issue, but I did not get any feedback. My current workaround is to just downgrad libtorch to 1.10.2+cu113 and use the working torch-tensorrt for that version.
other libs:
CUDA: 11.3
CUDNN: <IP_ADDRESS>
tensorrt <IP_ADDRESS> (from nvidia)

I suspect this might be related to #1823

Hi @bjaeger1 - could you test your model with PR #1859? We recently identified an issue in torch::jit::EliminateExceptions, and fixed it in that PR, which is pending review. This may resolve the issue with your model as well.

Hi @gs-olive - I just tried out the PR #1859.
However I was not able to successfully build torch-tensorrt.
After building the docker image with:
DOCKER_BUILDKIT=1 docker build --build-arg BASE=23.06 --build-arg CUDNN_VERSION=8.9 --build-arg TENSORRT_VERSION=8.6 --build-arg PYTHON_VERSION=3.10 -f docker/Dockerfile -t torch_tensorrt:latest .
and running the container:
nvidia-docker run --gpus all -it --shm-size=8gb --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --name=torch_tensorrt --ipc=host --net=host torch_tensorrt:latest
it fails when checking if torch-tensorrt was compiled successfully:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary --config use_precompiled_torchtrt --config pre_cxx11_abi
ERROR:
2023/07/06 13:45:54 Downloading https://releases.bazel.build/5.2.0/release/bazel-5.2.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
ERROR: /opt/torch_tensorrt/WORKSPACE:57:21: fetching new_local_repository rule //external:libtorch_pre_cxx11_abi: java.io.IOException: The repository's path is "/opt/python3/site-packages/torch/" (absolute: "/opt/python3/site-packages/torch") but this directory does not exist.
ERROR: /opt/torch_tensorrt/tests/core/conversion/converters/BUILD:10:15: //tests/core/conversion/converters:test_activation depends on @libtorch_pre_cxx11_abi//:libtorch in repository @libtorch_pre_cxx11_abi which failed to fetch. no such package '@libtorch_pre_cxx11_abi//': The repository's path is "/opt/python3/site-packages/torch/" (absolute: "/opt/python3/site-packages/torch") but this directory does not exist.
ERROR: Analysis of target '//tests/core/conversion/converters:test_activation' failed; build aborted: Analysis failed
INFO: Elapsed time: 6.027s
INFO: 0 processes.

Fetching @googletest; Cloning 703bd9caab50b139428cea1aaff9974ebee5742e of https://github.com/google/googletest
Output when building the docker image:
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 5.17kB
=> [internal] load .dockerignore
=> => transferring context: 1.00kB
=> [internal] load metadata for docker.io/nvidia/cuda:11.8.0-devel-ubuntu22.04
=> [internal] load build context
=> => transferring context: 58.40MB
=> [base  1/21] FROM docker.io/nvidia/cuda:11.8.0-devel-ubuntu22.04
=> [base  2/21] RUN test -n "8.6" || (echo "No tensorrt version specified, please use --build-arg TENSORRT_VERSION=x.y to specify a version." && exit 1)
=> [base  3/21] RUN test -n "8.9" || (echo "No cudnn version specified, please use --build-arg CUDNN_VERSION=x.y to specify a version." && exit 1)
=> [base  4/21] RUN apt-get update
=> [base  5/21] RUN apt install -y build-essential manpages-dev wget zlib1g software-properties-common git libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev
=> [base  6/21] RUN wget -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer &&    chmod 755 pyenv-installer &&    bash pyenv-installer &&    eval "$(pyenv init -)"
=> [base  7/21] RUN pyenv install -v 3.10
=> [base  8/21] RUN pyenv global 3.10
=> [base  9/21] RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
=> [base 10/21] RUN mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
=> [base 11/21] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/7fa2af80.pub
=> [base 12/21] RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 536F8F1DE80F6A35
=> [base 13/21] RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC
=> [base 14/21] RUN add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
=> [base 15/21] RUN apt-get update
=> [base 16/21] RUN apt-get install -y libcudnn8=8.9* libcudnn8-dev=8.9*                                                                                108.6s
=> [base 17/21] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
=> [base 18/21] RUN add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
=> [base 19/21] RUN apt-get update
=> [base 20/21] RUN apt-get install -y libnvinfer8=8.6.* libnvinfer-plugin8=8.6.* libnvinfer-dev=8.6.* libnvinfer-plugin-dev=8.6.* libnvonnxparsers8=8.6.* libnvonnxparsers-dev=8.6.* libnvparsers8=8.6.*  libnvparsers-dev=8.6.*
=> [base 21/21] RUN wget -q https://github.com/bazelbuild/bazelisk/releases/download/v1.16.0/bazelisk-linux-amd64 -O /usr/bin/bazel &&    chmod a+x /usr/bin/bazel
=> [torch-tensorrt-builder-base 1/3] RUN apt-get install -y python3-setuptools
=> [torch-tensorrt 1/7] COPY . /opt/torch_tensorrt                                                                                => [torch-tensorrt-builder-base 2/3] RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
=> [torch-tensorrt-builder-base 3/3] RUN apt-get update &&    apt-get install -y --no-install-recommends locales ninja-build &&    rm -rf /var/lib/apt/lists/* &&    locale-gen en_US.UTF-8
=> [torch-tensorrt-builder 1/6] COPY . /workspace/torch_tensorrt/src
=> [torch-tensorrt-builder 2/6] WORKDIR /workspace/torch_tensorrt/src                                                                                => [torch-tensorrt-builder 3/6] RUN cp ./docker/WORKSPACE.docker WORKSPACE
=> [torch-tensorrt-builder 4/6] RUN mkdir -p "/opt/python3/" &&    ln -s "pyenv which python | xargs dirname | xargs dirname/lib/python3.10/site-packages" "/opt/python3/"
=> [torch-tensorrt-builder 5/6] RUN CUDA_BASE_IMG_VERSION_INTERMEDIATE=echo ${BASE_IMG#*:} &&    CUDA_BASE_IMG_VERSION=echo ${CUDA_BASE_IMG_VERSION_INTERMEDIATE%%-*} &&    CUDA_MAJOR_MINOR_VERSION=echo ${CUDA_BASE_IMG_VERSION%.*} &&    rm -fr /usr/local/cuda &&    ln  0.5s
=> [torch-tensorrt-builder 6/6] RUN bash ./docker/dist-build.sh
=> [torch-tensorrt 2/7] COPY --from=torch-tensorrt-builder  /workspace/torch_tensorrt/src/py/dist/ .
=> [torch-tensorrt 3/7] RUN cp /opt/torch_tensorrt/docker/WORKSPACE.docker /opt/torch_tensorrt/WORKSPACE
=> [torch-tensorrt 4/7] RUN pip install -r /opt/torch_tensorrt/py/requirements.txt
=> [torch-tensorrt 5/7] RUN pip install tensorrt==8.6.*
=> [torch-tensorrt 6/7] RUN pip install .whl && rm -fr /workspace/torch_tensorrt/py/dist/ *.whl
=> [torch-tensorrt 7/7] WORKDIR /opt/torch_tensorrt
=> exporting to image
=> => exporting layers
=> => writing image sha256:19df2980268d6c1d5692986c9d743acba55615d3e6cd59101537bf47d72b2d38
=> => naming to docker.io/library/torch_tensorrt:latest

Hi @bjaeger1 - I just rebased #1859 onto main, since a few dependencies were incorrect in the existing version. Additionally, regarding the docker build we recommend using the default (BASE_IMG=nvidia/cuda:12.1.1-devel-ubuntu22.04) as the base build; the following is working on my machine (when building from the #1859 PR branch directly):
DOCKER_BUILDKIT=1 docker build --build-arg CUDNN_VERSION=8.9 --build-arg TENSORRT_VERSION=8.6 --build-arg PYTHON_VERSION=3.10 -f docker/Dockerfile -t torch_tensorrt:latest .

Could you try the build again with a fresh pull of that branch?

Hi @gs-olive , thanks for the quick answer!
I just made a fresh pull of your eliminate_exceptions_removal-Branch and built a new image with the default base-image.
However, the command:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary --config use_precompiled_torchtrt --config pre_cxx11_abi
still fails with the same error message. It can't find /opt/python3/site-packages/torch/ (because there is no such directory).
But the docker-image build process stats:
=> CACHED [torch-tensorrt-builder 4/6] RUN mkdir -p "/opt/python3/" &&    ln -s "pyenv which python | xargs dirname | xargs dirname/lib/python3.10/site-packages" "/opt/python3/"

Hi @bjaeger1 - thanks for the follow-up. I was able to reproduce the issue and I addressed the problem in #2085, which adds the necessary /opt/python3 symlink in the final container. Could you add that code to the Dockerfile at the specified location, rebuild the container, and run:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary  --config pre_cxx11_abi

(I had to remove --config use_precompiled_torchtrt, since recompilation is needed here to use the C++ APIs).

Hi @gs-olive , after adding the mentioned lines for the symlink to the Dockerfile, I was finally able to successfully build torch_tensorrt. Thanks!
I currently have difficulties (linking torch_tensorrt against libtorch fails) when building my minimal example. I'll come back when I fixed that issue.

@gs-olive, I would assume that the linking error is because the torch_tensorrt I built is using CUDA-12.1 (BASE_IMG=nvidia/cuda:12.1.1-devel-ubuntu22.04) but the libtorch-version I downloaded from the official website is with CUDA-11.8.
Trying to build torch_tensorrt with BASE_IMG=nvidia/cuda:11.8.0-devel-ubuntu22.04 is somehow not possible and fails with:
36 225.8 No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
36 225.8 running bdist_wheel
36 225.8 2023/07/10 13:28:45 Downloading https://releases.bazel.build/5.2.0/release/bazel-5.2.0-linux-x86_64...
36 226.6 Extracting Bazel installation...
36 233.4 Starting local Bazel server and connecting to it...
36 234.6 Loading:
36 234.6 Loading: 0 packages loaded
36 235.7 Loading: 0 packages loaded
36 235.7     currently loading:
36 235.8 Analyzing: target //:libtorchtrt (1 packages loaded, 0 targets configured)
36 236.6 ERROR: /workspace/torch_tensorrt/src/WORKSPACE:35:21: fetching new_local_repository rule //external:cuda: java.io.IOException: The repository's path is "/usr/local/cuda/" (absolute: "/usr/local/cuda") but this directory does not exist.
36 236.7 ERROR: /root/.cache/bazel/_bazel_root/67e780960d5a9319501f4e5a06fc77dc/external/tensorrt/BUILD.bazel:108:11: @tensorrt//:nvinfer depends on @cuda//:cudart in repository @cuda which failed to fetch. no such package '@cuda//': The repository's path is "/usr/local/cuda/" (absolute: "/usr/local/cuda") but this directory does not exist.
36 236.7 ERROR: Analysis of target '//:libtorchtrt' failed; build aborted:
36 236.7 INFO: Elapsed time: 10.139s
36 236.7 INFO: 0 processes.
36 236.7 FAILED: Build did NOT complete successfully (67 packages loaded, 1423 targets configured)
36 236.7 FAILED: Build did NOT complete successfully (67 packages loaded, 1423 targets configured)
36 236.7 building libtorchtrt
------
executor failed running [/bin/sh -c bash ./docker/dist-build.sh]: exit code: 1

Hi @bjaeger1 - thanks for the follow-up. It seems likely that the mismatched libtorch/CUDA versions are contributing to this issue. We very recently updated the stack on main to CUDA 12.1 from 11.8. If you prefer to use CUDA 11.8, I would recommend applying commit https://github.com/pytorch/TensorRT/commit/f05290a4a7760759c3ad83497e25230f7908bad1, which rolls back to CUDA 11.8 everywhere in the code, except in the CI. For a branch containing these changes applied to main, see cuda_118_rollback. You can clone the branch and run the following to build the container, and run any necessary tests.
DOCKER_BUILDKIT=1 docker build --build-arg CUDNN_VERSION=8.8 --build-arg TENSORRT_VERSION=8.6 -f docker/Dockerfile -t torch_tensorrt:latest .

Hi @gs-olive , I prefer to use CUDA 11.8 and wait until the pytorch binaries are released with CUDA 12.1 instead of building it from source.
I pulled the cuda_118_rollback branch and built the image. The torch_tensorrt build fails with:
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //tests/core/conversion/converters:test_activation (75 packages loaded, 9001 targets configured).
INFO: Found 1 test target...
ERROR: /root/.cache/bazel/_bazel_root/272136ab8790bfb0c01be73c7cbef828/external/cuda/BUILD.bazel:12:11: SolibSymlink<EMAIL_ADDRESS>failed: missing input file 'external/cuda/lib64/libcudart.so', owner: '@cuda//:lib64/libcudart.so'
ERROR: /root/.cache/bazel/_bazel_root/272136ab8790bfb0c01be73c7cbef828/external/cuda/BUILD.bazel:12:11: SolibSymlink<EMAIL_ADDRESS>failed: 1 input file(s) do not exist
Target //tests/core/conversion/converters:test_activation failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /root/.cache/bazel/_bazel_root/272136ab8790bfb0c01be73c7cbef828/external/cuda/BUILD.bazel:12:11 SolibSymlink<EMAIL_ADDRESS>failed: 1 input file(s) do not exist
INFO: Elapsed time: 11.506s, Critical Path: 0.04s
INFO: 30 processes: 30 internal.
FAILED: Build did NOT complete successfully
//tests/core/conversion/converters:test_activation              FAILED TO BUILD

Again, thanks a lot for your effort!

Just to check - is this failure occurring during the docker build command, or once the container starts? I am able to build the container successfully on my machine, which is why I ask.

The docker build command works fine. In both cases described above, the problem arises inside the running docker container, either running:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary  --config pre_cxx11_abi
or:
/docker/dist-build.sh

Update: I just installed the libtorch nightly binary which comes with CUDA 12.1
I built a torch_tensorrt-docker-image with CUDA-12.1 and then the torch_tensorrt library.
When compiling my minimal example the compilation process fails with 3 undefined references to glibc:

~/Dokumente/Projekte/Torch-TensorRT-Minimal-Example/tensorrt_api> ./build.sh
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc-11 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++-11 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found TensorRT headers at /usr/local/TensorRT/include
-- Found TensorRT libs at /usr/local/TensorRT/lib/libnvinfer.so;/usr/local/TensorRT/lib/libnvinfer_plugin.so
-- Found TENSORRT: /usr/local/TensorRT/include
-- Found CUDA: /usr/local/cuda (found version "12.1")
-- The CUDA compiler identification is NVIDIA 12.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.1.105")
-- Caffe2: CUDA detected: 12.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 12.1
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is b51b459d
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- Autodetected CUDA architecture(s):  7.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75
-- Found Torch: /usr/local/libtorch/lib/libtorch.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/jrb/Dokumente/Projekte/Torch-TensorRT-Minimal-Example/tensorrt_api/build
[2/2] Linking CXX executable TensorRT-app
FAILED: TensorRT-app
: && /usr/bin/g++-11 -D_GLIBCXX_USE_CXX11_ABI=0 -g -rdynamic CMakeFiles/TensorRT-app.dir/TensorRT-app.cpp.o -o TensorRT-app -L/usr/local/torch_tensorrt/lib -Wl,-rpath,/usr/local/cudnn/lib:/usr/local/TensorRT/lib:/usr/local/torch_tensorrt/lib:/usr/local/libtorch/lib:/usr/local/cuda/lib64  /usr/local/cudnn/lib/libcudnn.so.8  /usr/local/TensorRT/lib/libnvinfer.so  /usr/local/TensorRT/lib/libnvinfer_plugin.so  -ltorchtrt  /usr/local/libtorch/lib/libtorch.so  /usr/local/libtorch/lib/libc10.so  /usr/local/libtorch/lib/libkineto.a  -lcuda  /usr/local/cuda/lib64/libnvrtc.so  /usr/local/cuda/lib64/libnvToolsExt.so  /usr/local/cuda/lib64/libcudart.so  /usr/local/libtorch/lib/libc10_cuda.so  -Wl,--no-as-needed,"/usr/local/libtorch/lib/libtorch_cpu.so" -Wl,--as-needed  -Wl,--no-as-needed,"/usr/local/libtorch/lib/libtorch_cuda.so" -Wl,--as-needed  /usr/local/libtorch/lib/libc10_cuda.so  /usr/local/libtorch/lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcublasLt.so  -Wl,--no-as-needed,"/usr/local/libtorch/lib/libtorch.so" -Wl,--as-needed && :

/usr/lib64/gcc/x86_64-suse-linux/11/../../../../x86_64-suse-linux/bin/ld: /usr/local/torch_tensorrt/lib/libtorchtrt.so: undefined reference to<EMAIL_ADDRESS>/usr/lib64/gcc/x86_64-suse-linux/11/../../../../x86_64-suse-linux/bin/ld: /usr/local/torch_tensorrt/lib/libtorchtrt.so: undefined reference to<EMAIL_ADDRESS>/usr/lib64/gcc/x86_64-suse-linux/11/../../../../x86_64-suse-linux/bin/ld: /usr/local/torch_tensorrt/lib/libtorchtrt.so: undefined reference to<EMAIL_ADDRESS>
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

That makes sense, thank you for the details. I was able to reproduce the error, and it seems to be caused by an issue in the WORKSPACE.docker file when linking cuda. I've addressed the issue in the cuda_118_rollback branch, and I am now able to build the container on that branch:
DOCKER_BUILDKIT=1 docker build --build-arg CUDNN_VERSION=8.8 --build-arg TENSORRT_VERSION=8.6 -f docker/Dockerfile -t torch_tensorrt:latest .

The following command succeeds from within the container, on my machine:
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary  --config pre_cxx11_abi

Please let me know if the latest updates work for you as well.

Hi, with the latest update I was able to build the image and run the container. You only changed
path = "/usr/local/cuda/" to path = "/usr/local/cuda-11.8/" in WORKSPACE.docker, right?
The command
bazel test //tests/core/conversion/converters:test_activation --compilation_mode=opt --test_output=summary  --config pre_cxx11_abi
works!
But when running:
/docker/dist-build.h
the following error occurs:

In file included from torch_tensorrt/csrc/tensorrt_classes.h:3,
from torch_tensorrt/csrc/register_tensorrt_classes.cpp:1:
/opt/torch_tensorrt/py/../core/compiler.h:3:10: fatal error: cuda_runtime.h: No such file or directory
3 | #include <cuda_runtime.h>
|          ^~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Yes, changing the path in the WORKSPACE.docker was the update that fixed the initial issue.
After discussing the issue with @peri044 - could you try the following from within the container prior to building your example. The issue may be with the use of pre_cxx11_abi naming in the container.
# Make directory for modeling files
mkdir modeling
cd modeling

# Make compilation/Torch-TRT build file
touch run.cpp
# See below for demo BUILD file
touch BUILD

# Build using Bazel
cd ..
bazel build modeling:my_custom_model--config pre_cxx11_abi

Demo BUILD file - adapted from cpp/bin/torchtrtc/BUILD
load("@rules_pkg//:pkg.bzl", "pkg_tar")

config_setting(
    name = "use_pre_cxx11_abi",
    values = {
        "define": "abi=pre_cxx11_abi",
    },
)

cc_binary(
    name = "my_custom_model",
    srcs = [
            "run.cpp"
    ],
    linkopts = [
        "-ldl",
    ],
    deps = [
        "//third_party/args",
        "//cpp:torch_tensorrt",
    ] + select({
        ":use_pre_cxx11_abi": [
            "@libtorch_pre_cxx11_abi//:libtorch",
            "@libtorch_pre_cxx11_abi//:caffe2",
        ],
        "//conditions:default": [
            "@libtorch//:libtorch",
            "@libtorch//:caffe2",
        ],
    }),
)

I tried your suggestion but the build failed:

root@pc:/opt/torch_tensorrt# bazel build modeling:my_custom_model --config pre_cxx11_abi
INFO: Analyzed target //modeling:my_custom_model (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /opt/torch_tensorrt/modeling/BUILD:10:10: Linking modeling/my_custom_model failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc @bazel-out/k8-fastbuild/bin/modeling/my_custom_model-2.params
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o:function _start: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
Target //modeling:my_custom_model failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.802s, Critical Path: 0.40s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

Thanks for testing that out. I was able to reproduce that message, but only when the run.cpp file is empty. If I populate the run.cpp file (only a main function is really required) as follows, for instance:
Demo run.cpp
#include "torch/csrc/autograd/grad_mode.h"
#include "torch/csrc/jit/runtime/graph_executor.h"
#include "torch/script.h"
#include "torch_tensorrt/logging.h"
#include "torch_tensorrt/torch_tensorrt.h"

int main(int argc, char** argv) {
  // Compile, infer, ...
  printf("Output ...\n");
  return 0;
}

Then I run the following:
cd /opt/torch_tensorrt
bazel build modeling:my_custom_model --config pre_cxx11_abi
./bazel-bin/modeling/my_custom_model

The above succeeds on my instance of the container.

hi @gs-olive , on my side your example now also works fine!

Thanks for the update, and good to hear it gets through more of the compilation! I just rebased the cuda_118_rollback branch onto main, which now includes the fix in #1859, which is intended to address this halting issue. Could you try pulling the branch and building the Docker container from scratch again, then running your example on the new container?

I have really good news! The torch-tensorrt compilation of my minimal example finally works inside the container! A big thank you!

Great to hear it works on the example now! After discussing with @narendasan, there are a few options to use the example outside the Docker container. One is to copy the torch_tensorrt directory to the local as you did, and then modify this line in the WORKSPACE to point to the path to the local folder:
https://github.com/pytorch/TensorRT/blob/e884820cece2c403c6e35060be2e4fcf71f4522e/WORKSPACE#L34-L38
Then, you can rebuild the bazel target similarly to what you did from the Docker container.
Alternatively, you could rebuild the library on your local and add a new target for the run.cpp file. See here for a sample/tutorial.

Short update: only applying our change is not enough. Bazel can somehow not fetch the libraries:
new_local_repository(
    name = "libtorch",
    path = "/usr/local/torch_tensorrt-1.5.0-dev2/opt/python3/site-packages/torch/",
    build_file = "third_party/libtorch/BUILD"
)

new_local_repository(
    name = "libtorch_pre_cxx11_abi",
    path = "/usr/local/torch_tensorrt-1.5.0-dev2/opt/python3/site-packages/torch/",
    build_file = "third_party/libtorch/BUILD"
)

I therefore added instead:
http_archive(
    name = "libtorch",
    build_file = "@//third_party/libtorch:BUILD",
    sha256 = "b0e5b7fd8935d317dd91ca39dbb1f35e0a9a261550eae77249d947a908c3493e",
    strip_prefix = "libtorch",
    urls = ["https://download.pytorch.org/libtorch/nightly/cu118/libtorch-cxx11-abi-shared-with-deps-2.1.0.dev20230703%2Bcu118.zip"],
)

http_archive(
    name = "libtorch_pre_cxx11_abi",
    build_file = "@//third_party/libtorch:BUILD",
    sha256 = "49ec0e4dbf332058bffdd011e83c597d35c1531e465c02cc2cda10959a6b5063",
    strip_prefix = "libtorch",
    urls = ["https://download.pytorch.org/libtorch/nightly/cu118/libtorch-shared-with-deps-2.1.0.dev20230703%2Bcu118.zip"],
)

also the local cudnn & torch_tensorrt are not found so I changed the paths from path="/usr" to  path="/usr/local/cudnn"  path="/usr/local/torch_tensorrt" (which makes sense).
The problem then is the following:
ERROR: /root/.cache/bazel/_bazel_root/58fc92e22fb39b158c690f9065b2b6f2/external/tensorrt/BUILD.bazel:29:11: @tensorrt//:nvinfer_headers: missing input file 'external/tensorrt/include/x86_64-linux-gnu/NvUtils.h', owner: '@tensorrt//:include/x86_64-linux-gnu/NvUtils.h'
Target //minimal_ex:my_custom_model failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /root/.cache/bazel/_bazel_root/58fc92e22fb39b158c690f9065b2b6f2/external/tensorrt/BUILD.bazel:97:10 1 input file(s) do not exist
INFO: Elapsed time: 0.387s, Critical Path: 0.01s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

which looks like it is related to this issue: https://github.com/pytorch/TensorRT/issues/45 although I am not sure which lines to modify in third_party/tensorrt/BUILD

Short update: only applying our change is not enough. Bazel can somehow not fetch the libraries:
new_local_repository(
   name = "libtorch",
   path = "/usr/local/torch_tensorrt-1.5.0-dev2/opt/python3/site-packages/torch/",
   build_file = "third_party/libtorch/BUILD"
)

new_local_repository(
   name = "libtorch_pre_cxx11_abi",
   path = "/usr/local/torch_tensorrt-1.5.0-dev2/opt/python3/site-packages/torch/",
   build_file = "third_party/libtorch/BUILD"
)

I therefore added instead:
http_archive(
   name = "libtorch",
   build_file = "@//third_party/libtorch:BUILD",
   sha256 = "b0e5b7fd8935d317dd91ca39dbb1f35e0a9a261550eae77249d947a908c3493e",
   strip_prefix = "libtorch",
   urls = ["https://download.pytorch.org/libtorch/nightly/cu118/libtorch-cxx11-abi-shared-with-deps-2.1.0.dev20230703%2Bcu118.zip"],
)

http_archive(
   name = "libtorch_pre_cxx11_abi",
   build_file = "@//third_party/libtorch:BUILD",
   sha256 = "49ec0e4dbf332058bffdd011e83c597d35c1531e465c02cc2cda10959a6b5063",
   strip_prefix = "libtorch",
   urls = ["https://download.pytorch.org/libtorch/nightly/cu118/libtorch-shared-with-deps-2.1.0.dev20230703%2Bcu118.zip"],
)

This is fine as long as 2.1.0.dev20230703+cu118 is the version you used to build torch-tensorrt in the container or you are building from source (make sure to set the cuda version to 11.8 for that dependency in the workspace)

also the local cudnn & torch_tensorrt are not found so I changed the paths >from path="/usr" to path="/usr/local/cudnn" path="/usr/local/torch_tensorrt" >(which makes sense).

The problem then is the following:
ERROR: /root/.cache/bazel/_bazel_root/58fc92e22fb39b158c690f9065b2b6f2/external/tensorrt/BUILD.bazel:29:11: @tensorrt//:nvinfer_headers: missing >input file 'external/tensorrt/include/x86_64-linux-gnu/NvUtils.h', owner: '@tensorrt//:include/x86_64-linux-gnu/NvUtils.h'
Target //minimal_ex:my_custom_model failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /root/.cache/bazel/_bazel_root/58fc92e22fb39b158c690f9065b2b6f2/external/tensorrt/BUILD.bazel:97:10 1 input file(s) do not exist
INFO: Elapsed time: 0.387s, Critical Path: 0.01s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

which looks like it is related to this issue: https://github.com/pytorch/TensorRT/issues/45 although I am not sure which lines to modify in third_party/tensorrt/BUILD

PS: in WORKSPACE I also had to remove the "-" in the workspace name ("Torch-TensorRT" --> "TorchTensorRT")

For building with bazel the easiest (i.e. least prone to error way) to include the cudnn and tensorrt dependencies is to use the tarballs without unpacking them as inputs to http archive:
On my systems after downloading the correct builds from developer.nvidia.com, my workspace will look like this (give or take a few changes related to version upgrades):
workspace(name = "Torch-TensorRT")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "863ba0fa944319f7e3d695711427d9ad80ba92c6edd0b7c7443b84e904689539",
    strip_prefix = "rules_python-0.22.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.22.0/rules_python-0.22.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

http_archive(
    name = "rules_pkg",
    sha256 = "8f9ee2dc10c1ae514ee599a8b42ed99fa262b757058f65ad3c384289ff70c4b8",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_pkg/releases/download/0.9.1/rules_pkg-0.9.1.tar.gz",
        "https://github.com/bazelbuild/rules_pkg/releases/download/0.9.1/rules_pkg-0.9.1.tar.gz",
    ],
)

load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")

rules_pkg_dependencies()

http_archive(
    name = "googletest",
    sha256 = "755f9a39bc7205f5a0c428e920ddad092c33c8a1b46997def3f1d4a82aded6e1",
    strip_prefix = "googletest-5ab508a01f9eb089207ee87fd547d290da39d015",
    urls = ["https://github.com/google/googletest/archive/5ab508a01f9eb089207ee87fd547d290da39d015.zip"],
)

# External dependency for torch_tensorrt if you already have precompiled binaries.
local_repository(
    name = "torch_tensorrt",
    path = "/opt/conda/lib/python3.8/site-packages/torch_tensorrt",
)

# CUDA should be installed on the system locally
new_local_repository(
    name = "cuda",
    build_file = "@//third_party/cuda:BUILD",
    path = "/usr/local/cuda-12.1/",
)

#############################################################################################################
# Tarballs and fetched dependencies (default - use in cases when building from precompiled bin and tarballs)
#############################################################################################################

http_archive(
    name = "libtorch",
    build_file = "@//third_party/libtorch:BUILD",
    sha256 = "1ae8366aaf7af7f68f142ba644fe26c837c6fa8347ec6bd9ce605ac60e7f7e5e",
    strip_prefix = "libtorch",
    urls = ["https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-2.1.0.dev20230703%2Bcu121.zip"],
)

http_archive(
    name = "libtorch_pre_cxx11_abi",
    build_file = "@//third_party/libtorch:BUILD",
    sha256 = "9add4832f4da9223866d85810820b816ab3319d5a227066101eeb6cbb76adb4b",
    strip_prefix = "libtorch",
    urls = ["https://download.pytorch.org/libtorch/nightly/cu121/libtorch-shared-with-deps-2.1.0.dev20230703%2Bcu121.zip"],
)

http_archive(
    name = "cudnn",
    build_file = "@//third_party/cudnn/archive:BUILD",
    sha256 = "79d77a769c7e7175abc7b5c2ed5c494148c0618a864138722c887f95c623777c",
    strip_prefix = "cudnn-linux-x86_64-<IP_ADDRESS>_cuda12-archive",
    urls = [
        "file:///<ABSOLUTE PATH TO DOWNLOAD ON SYSTEM>/cudnn-linux-x86_64-<IP_ADDRESS>_cuda12-archive.tar.xz",
        "https://developer.nvidia.com/downloads/compute/cudnn/secure/8.8.1/local_installers/12.0/cudnn-linux-x86_64-<IP_ADDRESS>_cuda12-archive.tar.xz",
    ],
)

http_archive(
    name = "tensorrt",
    build_file = "@//third_party/tensorrt/archive:BUILD",
    sha256 = "0f8157a5fc5329943b338b893591373350afa90ca81239cdadd7580cd1eba254",
    strip_prefix = "TensorRT-<IP_ADDRESS>",
    urls = [
        "file:///<ABSOLUTE PATH TO DOWNLOAD ON SYSTEM>/TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-12.0.tar.gz",
        "https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-12.0.tar.gz",
    ],
)

#########################################################################
# Development Dependencies (optional - comment out on aarch64)
#########################################################################

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "devtools_deps",
    requirements = "//:requirements-dev.txt",
)

load("@devtools_deps//:requirements.bzl", "install_deps")

install_deps()

The new_local_repository sources can be used but they are written by default to prefer the layout of files that a conventional install of the TensorRT and cuDNN debian packages will create. We have configurations for CentOS as well but that is enabled via a build flag. If you just unpacked the tarballs somewhere the archive:BUILD files will be correct compared to the local:BUILD ones


Crash on buggy version check

Describe the bug
Tried to do a check for updates for some plugins on my bungeecord server, and it just crashed
To Reproduce

Have the plugins I have installed
Run check all

Expected behavior
The app should either manage to check, or at least if there is a file that does not have a plugin.yml, should skip it rather than crash the application
Screenshots

Desktop (please complete the following information):

OS: Ubuntu 20.04 (5.4.0-77-generic)

Additional context
I am aware that this app is not designed to work on BungeCoord but this is more of a matter of validation rather than compatibility. App should not crash like this when something is wrong with one of the jar files.

Hey @nathanielcutajar,
i just released an update which fixes this issue.
Download the new version and you are good to go: v1.6.5
Thank you for flying with pluGET 🚀


Training

Hi, When I execute python train.py it works and the result is as follows:
Model restored from file: utils/checkpoint/gcn.ckpt
Model saved in file: utils/checkpoint/gcn.ckpt
Model saved in file: utils/checkpoint/gcn.ckpt
...
...
...
Training Finished!
According to the model save directory, I didn't find everything about the model under the checkpoint file. What's the problem?

tensorflow.python.framework.errors_impl.NotFoundError: .\p2m.external\tf_nndistance_so.so not found
Do you know how to solve it?

Same question please

Hi,
Do you know how to download the dataset and unzip it from the link below?
https://drive.google.com/open?id=131dH36qXCabym1JjSmEpSQZg4dmZVQid


Make log level changeable with normal logging (0) as default settings.

This is usable for testing/debugging purposes

LGTM


Updated Express code to serve assets

Assets where not served so the style was not applied, fixed this by adding the line app.use(express.static('dist/client/'))

What kind of changes does this PR include?

New or updated content

Description

What does this PR change? Give us a brief description.
Adds app.use(express.static('dist/client/')) to express's SSR code to serve assets.

Hi @EBits00! Thanks for this contribution to the docs! You did happen to find one of our pages that pulls in content from elsewhere, though, so the change has to be made at the original source which is in a different repository.
If you open this file again, and look at the very top, you'll see the link to the file in GitHub where this page's source exists. Would you mind making your PR there, to that file? (It's in the withastro/astro repository, and not actually the docs repository.) Then, the change will be brought into docs when next we sync. Let me know if you'd be willing to, or if you have any questions, or if you'd rather I take care of this, I can! But, I'd love for you to have the opportunity if you'd like to contribute directly!

Thanks for this @EBits00, just saw your PR got merged in the main repo withastro/astro#4402 so I will close this one. 💜


Support .properties files

.properties file are a common configuration file format.
Support for this format could be added.

I am interested to participate, May I go for this issue?


[CSM-332] Rewrite tx history - version for 0.5 release

See https://github.com/input-output-hk/cardano-sl/pull/1083

I have tested it in prod mode and it seems to be working fast at least. I can't be 100% about its correctness - but it seems to work


panic Tm::asctime on linux x86_64 systems

Panic when using Tm::asctime at Linux x86_64:
(simplified to reproduce the problem)
extern crate time;
use time::{Timespec, at};

fn main() {
    println!("{}", at(Timespec::new(-0x800000000000000, 0)).asctime());
}

Results in a unreachable panic:
thread '<main>' panicked at 'internal error: entered unreachable code', /home/willem/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.1.22/src/display.rs:134

At Linux 32-bit system, no panic happens.
Problem is caused by a invalid Tm struct created by 'at':
Tm { tm_sec: 24, tm_min: 21, tm_hour: 17, tm_mday: 32693, tm_mon: 37, tm_year: -1087444786, tm_wday: 0, tm_yday: 32693, tm_isdst: 0, tm_utcoff: 1172, tm_nsec: 0 }

This actually not caused by time crate itself, but is caused by GNU libc implementation of 'localtime_r'.

Oh dear, looks like we just weren't checking errors!


Blog Art : Admin Dashboard with Node-RED Dashboard 2.0

Related Issue
No response
Description
Considerations:

Where will the image be displayed/shared, product? blog post? social media? presentation? website?

Blog Post
Title: "Building Admin Dashboard with Node-RED Dashboard 2.0"
Reference content:

https://github.com/FlowFuse/website/pull/1883

Hi @sumitshinde-84 I've just pushed it to the PR


Compile_tb.py generate options not recogised by xrun

Observed Behavior

compile_tb_stdstreams.log
Expected Behavior

Steps to reproduce the issue

My Environment

EDA tool and version:
 Xcelium 23.09-s001
Operating system:
 Centos 7.9
Version of the Ibex source code:
 668233699df9ec2a40413e69e0de0a5b10185980

To save other people having to download the log file (which is only 3 lines long!), it contains:
xrun: *SE,BDOPT: Unknown option -rpath,/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++

I think this form of the -rpath argument is coming from _get_iss_pkgconfig_flags in compile_tb.py. Pinging @hcallahan-lowrisc, who might be able to help (given that he wrote the code there)

Hi @cern143
There has been a few issues with CentOS7 support in the past, so this could possibly be related to that. I can't reproduce this locally right now. (I am using NixOS, but I can also confirm it works fine on Ubuntu). But could I just check a few things with you first before we go down that route...
The xrun invocation I get locally looks as follows:
$ make --keep-going IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0

# compile_tb_stdstreams.log
<...>
xrun -64bit -q -f /home/harry/projects/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/harry/projects/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/harry/projects/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include/softfloat -I/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/include -Wld,-Xlinker,-rpath,/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib -lstdc++
<...>

Comparing it to your logfile, this suggests to me that your build has not been able to correctly use pkg-config to resolve the build-time flags pointing at your local spike build. Everywhere my build contains the path /nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0 (the path to my spike build), yours has no path there at all.
Also, strangely the -rpath argument looks to have been formed differently between our systems. Mine resolved to -Wld,-Xlinker,-rpath,<spike_path>/lib, whereas yours resolved to -Wld,-Xlinker -rpath,/lib, not only missing the resolved spike_path but also there is now a space instead of a comma between -Xlinker and -rpath.
To get a bit more information, I'll prepare a branch which prints out a bit more information from the build. In the meantime, could you just confirm a few things.

What your version of pkg-config is?

[harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ pkg-config --version
0.29.2

From the build instructions page, you have built spike and setup the PKG_CONFIG_PATH correctly, pointing to a path that exists. e.g.

[harry@neptune:~/projects/ibex/dv/uvm/core_ibex]$ echo $PKG_CONFIG_PATH | tr : '\n' | grep spike
/nix/store/88xrnh2bwrhxymlaq3xakh6957ak19ix-spike-1.1.0/lib/pkgconfig

Thanks

Hi @hcallahan-lowrisc
Thank for your reply. When i check pkg-config version, i got:
 pkg-config --version
0.27.1

and for PKG_CONFIG_PATH:
 echo $PKG_CONFIG_PATH
/usr/lib/pkgconfig:/usr/lib/pkgconfig

I can notice our environment are different. Is that ok?

Okay I see the issue. You need to follow the build instructions page, which describes how to build spike and setup your environment variables appropriately. A part of this is setting PKG_CONFIG_PATH to point towards the spike model you have just built. Without doing this, it is not possible to run the simulations, as we need to link against the spike model to perform cosimulation.
Thanks!

Hi @hcallahan-lowrisc, I rebuilt it in a different directory:
echo $PKG_CONFIG_PATH
/opt/lib/pkgconfig

And still get the same result:
xrun: *SE,BDOPT: Unknown option -rpath,/opt/lib.
Running compile_tb command :
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/opt/lib -I/opt/include -I/opt/include/softfloat -Wld,-Xlinker -rpath,/opt/lib -lstdc++

Im pretty sure i didn't specify the prefix at the first time i built it (which is not a smart move).
For more information, i also check the version of riscv-isa-sim
 ❯ git rev-parse HEAD
39612f93837122a980395487f55b2d97d29d70c1
❯ git rev-parse --abbrev-ref HEAD
ibex_cosim

Everything seem fine.
I also tried to edit compile.py to use -Wld option of xcelium and it seem to work, howerver the default g++ of xcelium can't understand -rpath option:
ake[2]: Entering directory `/home/usr14/ibex/dv/uvm/core_ibex'
building library run.so
g++: error: unrecognized command line option '-rpath'
make[2]: *** [/home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/xcelium.d/run.lnx8664.23.09.d/librun.so] Error 1
make[2]: Leaving directory `/home/usr14/ibex/dv/uvm/core_ibex'

Can you tell me what to do next? Best regard.

Could you try running the command directly, except with a comma between -Xlinker and -rpath instead of a space? See the message above where I pasted the command generated on my system for an example. I believe the xrun tool is parsing the CLI invocation differently due to this change, and hence g++ is being fed different arguments.
This suggests to me that pkg-config on our systems are somehow returning different outputs. I'll try to put together a branch today which prints some logging output to confirm this.

It is still the same problem:
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker -rpath,/lib -lstdc++
xrun: *SE,BDOPT: Unknown option -rpath,/lib.

I couldn't understand why that doesn't work

Hey @cern143
Could you try just running this command directly, and let me know what happens?
xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++

(I just copied the command from your message, but changed the space for a comma before the -rpath argument)
Thanks

I tried your command @hcallahan-lowrisc, and xrun seem ok with those parameters:
❯ xrun -64bit -q -f /home/usr14/ibex/dv/uvm/core_ibex/ibex_dv.f -sv -licqueue -uvm -uvmhome CDNS-1.2 -define UVM_REGEX_NO_DPI -elaborate -l /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb/compile_tb.log -xmlibdirpath /home/usr14/ibex/dv/uvm/core_ibex/out/build/tb -access rw -defparam core_ibex_tb_top.RV32E=0 -define IBEX_CFG_RV32M=ibex_pkg::RV32MSingleCycle -define IBEX_CFG_RV32B=ibex_pkg::RV32BOTEarlGrey -define IBEX_CFG_RegFile=ibex_pkg::RegFileFF -defparam core_ibex_tb_top.BranchTargetALU=1 -defparam core_ibex_tb_top.WritebackStage=1 -defparam core_ibex_tb_top.ICache=1 -defparam core_ibex_tb_top.ICacheECC=1 -defparam core_ibex_tb_top.ICacheScramble=1 -defparam core_ibex_tb_top.BranchPredictor=0 -defparam core_ibex_tb_top.DbgTriggerEn=1 -defparam core_ibex_tb_top.SecureIbex=1 -defparam core_ibex_tb_top.PMPEnable=1 -defparam core_ibex_tb_top.PMPGranularity=0 -defparam core_ibex_tb_top.PMPNumRegions=16 -defparam core_ibex_tb_top.MHPMCounterNum=10 -defparam core_ibex_tb_top.MHPMCounterWidth=32 -f ibex_dv_cosim_dpi.f -I/home/usr14/ibex/dv/cosim -lriscv -lsoftfloat -ldl -ldisasm -lfdt -lfesvr -L/lib -I/include -I/include/softfloat -Wld,-Xlinker,-rpath,/lib -lstdc++
xrun: *E,FILEMIS: Cannot find the provided file /dv/uvm/core_ibex/common/prim/prim_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_assert.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_util_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count_pkg.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_count.sv.
xrun: *E,FILEMIS: Cannot find the provided file /ip/prim/rtl/prim_secded_pkg.sv.
...

There are more log line but i these are enough to point out the problem, xrun can't find file listed in ibex_dv_cosim_dpi.f

Hey @hcallahan-lowrisc, seem like we fixed it. Just need to edit a line in compile.py (remove space between xlinker and formated string):
_flags = "'-Xlinker{}'".format(_flags.replace('-Wl,', ','))

and it will be ok.

Thanks for letting me know @cern143, glad to hear its working now. I've just opened #2209, which should fix this a bit more generally. Would you mind testing it out and confirming that it works for you?

Of course @hcallahan-lowrisc, i had ran your new script and it worked completely fine:
make --keep-going IBEX_CONFIG=opentitan SIMULATOR=xlm ISS=spike ITERATIONS=1 SEED=1 TEST=riscv_arithmetic_basic_test WAVES=0 COV=0
Build metadata already exists, not recreating from scratch.
Building RTL testbench
Generating core configuration file
Building randomized test generator
Running randomized test generator to create assembly file /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.S
Compiling riscvdv test assembly to create binary at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/test.bin
Running RTL simulation at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1
Collecting simulation results and checking logs of testcase at /home/usr19/ibex/dv/uvm/core_ibex/out/run/tests/riscv_arithmetic_basic_test.1/trr.yaml
Collecting up results of tests into report regr.log
100.00% PASS 1 PASSED, 0 FAILED


💄: prettier DMGs

Then:

Now:

I'm merging this. We can't change the font color (it isn't perfectly readable), but it looks much better. We can change anything later.


Automate Keycloak Updates

Context
To update to the newest version of Keycloak, we have to follow ~10 manual steps, including find out when a new version was released (https://quay.io/repository/keycloak/keycloak?tab=tags) and changing it on with commodore.
Related to SYN-352 Automated Component Dependency Updates
Current update procedure documented in internal wiki
Appeared during VT-1556
Alternatives
At the moment an update is done with helm_values, e.g.
  example_keycloak_prod:
    namespace: example-keycloak-prod
    fqdn: id.example.net
...
    helm_values:
      image:
        tag: 13.0.0

This is somehow bypassing component-keycloak and changing directly a value from the used helm chart.
As Operator of Keycloak on K8s...

I want to get informed if a new release is available
I want that a merge/pull request is created for my configured Keycloak statefulsets to update to that newest version
I want that after merging the MR/PR that the new version is rolled out automatically.

This is a generic problem that we have for all Commodore components. Our answer to that is a Renovate plugin. Renovate is bot that automatically searches for new updates and creates MRs/PRs. See https://syn.tools/syn/SDDs/0017-maintenance-with-renovate.html
However, it's currently not ready.

We now have Renovate support for component versions in Project Syn hierarchies as well as Helm chart and Docker image dependencies of components. All those are implemented in https://github.com/projectsyn/commodore-renovate/.
If you want automated updates for pinned versions of component dependencies (e.g. Helm chart or Docker image) in a Project Syn hierarchy repo (global or tenant config), you can follow the guide at https://syn.tools/syn/how-tos/tenant-version-pins.html.


add group_by_all support

This PR adds support for group_by_all parameter.
If it is true, all alert's labels will be used for grouping.
If each alert has unique set of labels, this setting will effectively disable aggregation of different alerts.

@stuartnelson3

Well, use cases are different, and sometimes such functionality is needed

also it will help in situations like #1587

Actually @brian-brazil this is a useful feature.
At the beginning I was also thinking that group_by: [] would do this. But it does the opposite.

 "Just talk to Prometheus directly" is throwing away the usefulness of the glue that is Alertmanager. Sure, this may not fit certain ideas of best practices, but not all environments are the same and not everyone has the same requirements with alert grouping. This seems like a simple enough feature to add, why not do so if it solves some peoples' problems?

From my POV, one big reason for not adding it is that it makes the AlertManager configuration even more difficult to get right and understand. Also I'm not convinced that it helps for #1587.
As for not grouping some alerts, I would rather add a special label to my alerting rules that would be based of all the labels necessary to uniquely identify an alert instance.
  - alert: MyAlert
    expr: my_metric == 0
    for: 1m
    labels:
      unique_key: "{{ $labels.foo }}/${{ $labels.bar }}/and/many/more"

And then use group_by [alertname,job,instance,unique_key] would be enough to have all my alerts ungrouped without having AlertManager to know about all possible label names.

It helps for #1587 because it guarantees that all alerts page individually, by never grouping anything. This is a plausible config for smaller shops, where even if everything is down the page storm is manageable.
Honestly, I don't see how mucking up all alert rules with an extra custom label is better than this one-shot config... at that point you might as well add all possible unique labels to group_by, it's less work than messing with each alert rule individually.

This is absolutely required where you want to feed through individual alerts to PD / OpsGenie etc... - its sort of useless at the moment when grouped together from a reporting perspective and actually knowing what to respond too.
But I still would want to retain grouping for informal slack notifications etc....

As @beorn7 already commented, features shouldn't be added to alleviate the lack of PD or OpsGenie. I can find at least one past example where a feature had been asked for similar reasons and eventually OpsGenie fixed the limitation on their end. You should try to get those people involved here.

That example is a different situation and its not clear what the lack of support actually from an opsgenie / pager duty perspective when its alert manager firing the alerts?

Okay, this is not a limitation of anything, it is more a limitation of HA in alertmanager.
Here is my usecase:
We have a webhook receiver that can only manage one alert at the time. That's it.
We tried to implement this in the webhook receiver (threating each alert individually)... but then.. it is too slow and the second alertmanager sends the alerts too!! Also, when alert 5 fails, we need to redo all the 4 first alerts, we can not fail atomically.
This can be avoided by this pull request. And it is more a way to deal with alertmanager HA.

We use queues for alerts too but in another use case which is out of scope here.
The idea here is:
I expect group_by: [] to +not group alerts+. Instead, it creates a massive group.
We have a receiver that, for exemple, posts to twitter. We are limited in lenght so we must send the alerts 1 by 1.
If we do that in the receiver, it is possible that out of 10 alerts, 1 fails, but we need to reply 500 to the alertmanager, so that the complete group would be re-sent. Or it will take a lot of time and the 2nd alert manager will also fire the alerts.
For those two reasons this feature makes sense.

The Alertmanager is fundamentally an at-least-once delivery system. I don't think we should be adding (contentious) features to support webhooks that aren't prepared to deal with that.

I expect group_by: [] to +not group alerts+. Instead, it creates a massive group.

From my POV, it does the right thing: grouping on an empty set of label names means that you want to put all alerts in the same bucket.
If you really want it, you could generate a label that identifies every alert uniquely:
  - alert: SomeAlert
    expr: my_metric == 0
    for: 1m
    labels:
      key: "{{ range $k, $v := $labels }}{{ $k }}={{ $v }}{{end}}"

I still fail to see how a simple toggle to disable existing behavior is so controversial, and why everyone keeps proposing horrible workarounds that replace a single line in the Alertmanager config with changes to every single alerting rule in Prometheus.
I get how this may not be something fitting certain ideas of how things should work or how external services should behave, but we've covered several use cases already where this simple feature would be useful to people. Clearly there is value here. Why the aversion to 1:1 alert routing? Alertmanager isn't just a grouper, it does more things, like silencing and converting alerting protocols. Why tie those things to mandatory grouping? This isn't some horrible subversion of what Alertmanager does or how it should behave in a stack. It's basically just trivial syntactic sugar for a behavior that people can already achieve in an ugly, non-ergonomic way.

that replace a single line in the Alertmanager config with changes to every single alerting rule in Prometheus.

We generally do not add features that can already be handled via configuration management.

Alertmanager isn't just a grouper, it does more things, like silencing and converting alerting protocols.  Why tie those things to mandatory grouping? This isn't some horrible subversion of what Alertmanager does or how it should behave in a stack.

The purpose of the Alertmanager is to increase the signal to noise ratio of notifications, grouping and throttling are innate parts of that. Allowing this would be a subversion of the goals of the alertmanager, as it is one of several features that users request for the purposes of spamming themselves due to being used to less powerful systems (without the notion of labels), rather than following https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview.

I don't see how asking all of our customers to add a certain label to every single alert config which looks horrible or alternatively having to pre-process their configs is a better idea than a simple configuration.

The purpose of the Alertmanager is to increase the signal to noise ratio of notifications, grouping and throttling are innate parts of that.
But not only these. There are also http_config, pagerduty_config, webhooks_config which means that they are also responsibilities of Alertmanager.

The purpose of the Alertmanager is to increase the signal to noise ratio of notifications, grouping and throttling are innate parts of that. Allowing this would be a subversion of the goals of the alertmanager, as it is one of several features that users request for the purposes of spamming themselves due to being used to less powerful systems (without the notion of labels), rather than following

The goal of software should not be to enforce a particular philosophy on its users. The goal of software should be to empower users to choose the configuration that suits them best. It's fine to nudge users towards not hurting themselves with documentation, but it's not productive to actively alienate those users with legitimate use cases for such features. While there is obviously no requirement that software implement every possible feature that every possible user might use, refusing to implement a simple "turn off this behavior I don't need or want" toggle on ideological grounds is, in my opinion, just poor form. You're basically saying "I know better than everyone else, so you can either do things my way or write your own code".
You know full well I was also an SRE at Google, and I agree with most of Rob's essay. But not everyone is Google. I've seen SRE teams at Google in monitoring hell with 10 pages per day. I've seen teams at Google that never got paged. There are many different scenarios that are very different from Google scale. In the extreme, small single-homed systems are often assumed to be able to stay up for years on end without paging. There is no point in optimizing alerting/monitoring rules for cases like that, because you don't even have the data to know what you need to monitor! E.g. symptom-based alerting is fine and dandy when you have a large distributed system with many frontends where you can afford to lose some with zero impact, but for a little single-homed service, damn right I care about finding out that it's running out of RAM before it actually goes down. There is no one-size-fits-all monitoring philosophy.

The pushgateway is an official exporter. This is a recognition that Prometheus someone's needs to do exceptions in its principles. In would see this as an exception needed for some well defined use cases.

FWIW, fully agree with @marcan here.

Has anybody ever run into a situation where the (current) group_by: [] (with an empty list, i.e. put every alert into one group) would have been useful? In my own AM experience, I have never used an empty group_by, and I had to think what it would do when I saw it discussed here.
Even if there is controversy if "no grouping" is useful, perhaps we can all agree that "no grouping" is still saner than "all alerts in one group"? In that case, we could change the behavior of group_by: []. The followers of pure Promethean philosophy would not notice as they would never do that. Those that want to switch off grouping could do it now easily (i.e. without creating an artificial label with a unique label), but at least there is not an explicit option for it so we wouldn't create the impression it would be a recommended practice. And those who argue doing no grouping is shooting into your own foot could be satisfied by the fact that grouping all alerts into one group is even worse foot-shooting.
Finally, in case somebody really wants to put all alerts into the same group, you could still say group_by: [some_random_label_name_that_will_never_be_used] to accomplish that.

Yes, a daily email with all currently firing alerts.

Fair point.
There goes my idea to reconcile all demands…

My initial reaction after reading some of the feedback is against including this. For a small alerting setup/individual route where it is reasonable to send a notification for each alert, it doesn't seem like it would be too hard to come up with a unique group_by labelset, and for a large setup, this feature would end up hurting users with spam.
However, it seems that this is a feature that some users very much want. If @mxinden is also fine with it, I think we can review and trial the feature.

@stuartnelson3 The problem with coming up with a unique group_by labelset is that it's fragile. It't what I'm doing now, sure, but what if tomorrow I add some alerts that make that labelset no longer unique? It's a ticking time bomb; having the Alertmanager config contain a labelset that has to stay in sync with the potential behavior of all Prometheus alerts (which ultimately depends on what targets you're scraping and what metrics they expose, which means it could even change behind your back with a software update) is definitely not what I'd consider hygienic or good engineering.
Or: just because it's at a scale small enough that I want to get notified for every alert doesn't mean I want my config to be fragile and to have to keep track of all potential labelsets.
Also, notifying for every alert is entirely reasonable for large systems if alerts are being grouped by a downstream receiver, as some support features like this themselves (and thus overlap with Alertmanager in functionality, so users should be able to choose whichever side they prefer to do the grouping).

@stuartnelson3 comment is very much inline with my own thoughts on this.
Given the high community involvement (which is very much appreciated) I am also voting for the "review and trial" of this feature. This is not an approval of the particular implementation of this pull request, but rather of the idea of a one-notification-per-alert feature.

Implementation-wise, I'd prefer to not have a separate config option for this. Perhaps a magic label name (that is an illegal label name otherwise) like * or ... could be used to mean "group by all".

Implementation-wise, I'd prefer to not have a separate config option for this. Perhaps a magic label name (that is an illegal label name otherwise) like * or ... could be used to mean "group by all".

This sounds good to me. I lean towards group_by: [*], I read that as an expansion to everything.
@beorn7 is this how you imagined it? @mxinden thoughts?

@stuartnelson3 Yes, that's the best I could come up with so far. (Is * something you can leave unquoted in Yaml?) I don't like "magic" strings (we already have too many of them in the Prometheus universe), but this one looks intuitive: group_by: [] vs group_by [*] vs group_by [alertname, zone].

I lean towards group_by: [*]

+1 if we need to pick something

Is * something you can leave unquoted in Yaml?

Yes

I'll reimplement with '*'

Unfortunately,
according to
http://yaml.org/spec/current.html#id2535812
and
http://yaml.org/spec/current.html#alias/syntax,
alias is starting with *, so it have to be quoted.
Do you agree with
group_by: ['*'] ?

Too bad…
However, group_by: [...] would be valid YAML. Should we go with the latter?

group_by: [...] would also avoid any wrong assumption that we support globbing. (Seeing group_by: ['*'], people might try group_by: ['some_prefix_*'] to group by all labels starting with some_prefix_.)

@beorn7 raises a good point. [...] sounds good, too.

Good point, +1 on [...]

Suggestion for the text:

To aggregate by all possible labels use '...' as the sole label name. This effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping.


build fail in cross compile env, [-Werror=vla-parameter]

command:
bbr-acs/sbbr/scripts$  ./build-scripts/build_sbbr.sh
output log:
....
gcc  -c -I ./brotli/c/include -I .. -I ../Include/Common -I ../Include/ -I ../Include/IndustryStandard -I-I . -I ../Include/X64/ -MD -fshort-wchar -fno-strict-aliasing -fwrapv -fno-delete-null-pointer-checks -Weprecated-declarations -Wno-stringop-truncation -Wno-restrict -Wno-unused-result -nostdlib -g -O2  brotlic -o brotli/c/dec/bit_reader.o
gcc  -c -I ./brotli/c/include -I .. -I ../Include/Common -I ../Include/ -I ../Include/IndustryStandard -I-I . -I ../Include/X64/ -MD -fshort-wchar -fno-strict-aliasing -fwrapv -fno-delete-null-pointer-checks -Weprecated-declarations -Wno-stringop-truncation -Wno-restrict -Wno-unused-result -nostdlib -g -O2  brotli brotli/c/dec/decode.o
brotli/c/dec/decode.c:2033:41: error: argument 2 of type ‘const uint8_t ’ {aka ‘const unsigned char ’} ter [-Werror=vla-parameter]
2033 |     size_t encoded_size, const uint8_t encoded_buffer, size_t decoded_size,
|                          ~^~
In file included from brotli/c/dec/decode.c:7:
./brotli/c/include/brotli/decode.h:204:19: note: previously declared as a variable length array ‘const uie]’ {aka ‘const unsigned char[*decoded_size]’}
204 |     const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
brotli/c/dec/decode.c:2034:14: error: argument 4 of type ‘uint8_t *’ {aka ‘unsigned char ’} declared as =vla-parameter]
2034 |     uint8_t decoded_buffer) {
|     ~^~~~~~
In file included from brotli/c/dec/decode.c:7:
./brotli/c/include/brotli/decode.h:206:13: note: previously declared as a variable length array ‘uint8_t[a ‘unsigned char[encoded_size]’}
206 |     uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[2]: *** [../Makefiles/footer.makefile:21: brotli/c/dec/decode.o] Error 1
make[2]: Leaving directory '/home/yin/c/acs/bbr-acs/sbbr/scripts/edk2/BaseTools/Source/C/BrotliCompress'
make[1]: *** [GNUmakefile:76: BrotliCompress] Error 2
make[1]: Leaving directory '/home/yin/c/acs/bbr-acs/sbbr/scripts/edk2/BaseTools/Source/C'
make: *** [GNUmakefile:19: Source/C] Error 2
make: Leaving directory '/home/yin/c/acs/bbr-acs/sbbr/scripts/edk2/BaseTools'
Build failed: error while running do_build at line 134 in ./build-scripts/build-sct.sh for [][].

Hello @WangYin3,
Can you share the toolchain version used for compilation?
Thanks,
Chetan

Hi Chetan,
Attach the toolchain info:
aarch64-linux-gnu-gcc -v
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)

Hi @WangYin3 ,
The issue is resolved as part of this commit : https://github.com/ARM-software/bbr-acs/commit/151faf8c7e002175255ca80c891e0163b98b1c76
Could you please verify and confirm.
ACS Team


readme updation

📚 Documentation
i want to update all these readme files in total there are 10 readme files i am updating so kindly can you label this issue with 20 points
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/cp/README.md
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/cryptography/README.md
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/dp/README.md
https://github.com/TesseractCoding/NeoAlgo/tree/master/C-Plus-Plus/ds
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/graphs/README.md
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/math/README.md
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/other/README.md
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/search/README.md
https://github.com/TesseractCoding/NeoAlgo/tree/master/C-Plus-Plus/sort
https://github.com/TesseractCoding/NeoAlgo/blob/master/C-Plus-Plus/blockchain/README.md
Have you read the Contributing Guidelines on Pull Requests?
yes

i want to update all these readme
in total there are 10 files

@NEEHITGOYAL No need of opening a new issue for readme update, you can directly put a pull request and mention me there so that I can approve it.


Fix TimeSeries error fallback glitch in small containers

Description of changes
This PR hides TimeSeries error fallback's text for small containers.
Checklist
Before merging to main:

[ ] Manually tested in React apps
[x] Release notes
[ ] Approved

Release notes
# Component changes

## Time Series

- Fixed Time Series error fallback on small containers, the fallback text used to overflow the container, now it will be hidden for containers/canvas smaller than `150px`.

# App changes

## React sample apps

- Added Time Series fix test case for React sample apps.

I was QA testing and it looks fine. But found some typing issues when I run build. I'm fixing here.

@felipecadavid additionally, can you take a stab at what the release notes for this would look like? Ideally, we communicate to the customer:

What the behavior use to be,
What it will be now, and
Any other important details to controlling the behavior.

@felipecadavid

The idea is just uncommenting/commenting it when it's necessary for testing manually

Should we have a dedicated page for testing that? Or should we build some kind of button/toggle into the React app for testing that? Ideally, the React apps enable testing without any kind of code-level intervention.

I agree, I was thinking about changing a little bit how the example apps work. Something I thought is creating a tab for each component and i.e adding a Card for this test case in the Time Series tab. This way the examples are organized and we wouldn't require intervention in the code for manual testing.

Closing as those changes are old and don't seem to be a problem with our current setup


"Cadence processor cores only" license restriction

There are 29 files in the repo which look like they have an MIT license header, but upon closer inspection it actually contains a restricted license. For example, from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/testing/test_xtensa_hifi_binary.sh:

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to use this Software with Cadence processor cores only and
not with any other processors and platforms, subject to
the following conditions:

The files with this license notice can be seen at https://github.com/tensorflow/tensorflow/search?q=cadence+processor+cores+only
The project may want to consider removing these files, or perhaps moving them to a different repository, as content with a use restriction like this might not be seen by the community as compatible with Apache-2.0. (I gather this code will be received by anyone who checks out the source code repo, whether or not they're using this xtensa section of it.)

@petewarden or @advaitjain can you comment?

@swinslow Could you please let us know if this is still an issue?
The URLs provided in the issue description seem to not be working.
In order to reproduce the issue reported here, could you please provide more details on this issue. Thank you!

Thanks @sushreebarsa! Looking at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro, it looks like the relevant code was moved to a separate repo.
And over in that repo, from a quick search I'm not seeing reference to the old license that was problematic. (I haven't done a deeper dive, so anyone who is concerned might want to do so, but this is not relevant for me any longer.)
So I'll go ahead and close this issue. Thanks again!


[CGP-0064P] - Celo India DAO (Regional DAO)

This PR doesn't include the another INDIA DAO PROPOSAL.

@willkraft @calicokittencat should we create a separate PR for the another DAO proposal? Kindly let me know

Hey, @willkraft @calicokittencat we have planned to reduce the budget further, and also edit the course of time (already mentioned 'Quarterly' in the proposal), after considering the feedback provided in the governance call.
cc: @cryptochitty @shaunjoseph


Erlang vs Haskell

Thanks for a great collection of docs. I have just discovered it and enjoyed it very much.
I would just like to point out that the Learn you Erlang book is actually inspired by the Learn you Haskell book. I think that book also deserves to be on the list. Reference.
Thanks

Absolutely. Just added the acknowledgment. Thanks for pointing it out.


fix model_name exception for turbomind

Motivation
fix model_name exception for turbomind
Modification
_build_turbomind
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

ping @lvhan028 @irexyc

ref https://github.com/InternLM/lmdeploy/pull/1085

I am gonna go with PR #1126

We want to unify the initialization process of pytorch turbomind engine and remove model name(chat template) from engine initializaton in the future. Therefore, we shoud handle it in pipeline api.


Convert to Equation Containing Brackets

From a fresh canvas, clicking '(' from the Math tab using the popup keyboard makes clicking ')'  unresponsive. Also you are not able to highlight anything afterwards unless you click/move the '('.

Sometimes the recognition is wrong resulting in bad latex format.

@hughdani
i think this issue only applies if you use the SVG type of brackets.
a center issue, closing bracket gets recognized in the SUPER region of i.
should be fixed later.
But if you type brackets with keyboard now, everything should work.

Correct, typing brackets on a physical keyboard works. SVG symbols are sketchy as it is supposed to programmatically wrap the expression perfectly but doesn't.
As a test, try and reproduce this expression using the built in keyboard while using SVG symbols when possible. If you can do this with minimal rearrangement, then the detection algorithm should be robust.

This is still happening:

From a fresh canvas, clicking '(' from the Math tab using the popup keyboard makes clicking ')' unresponsive. Also you are not able to highlight anything afterwards unless you click/move the '('.

Btw, the first two images recognize the incomplete expression correctly and I am not sure if we can do anything about it if its how LaTeX works.

Makes no sense to fix this before we have the SVG refactored.


Is main CI failing?

PR #7040 is failing because this Specs / ci (python, python, 2, 0, pip) (pull_request)  which is unrelated to anything in the PR.
Failed examples:
rspec ./spec/dependabot/python/file_updater/pipfile_file_updater_spec.rb:300 # Dependabot::Python::FileUpdater::PipfileFileUpdater#updated_dependency_files the updated Pipfile.lock with a subdependency from an extra doesn't remove the subdependency

Yes, I was just opening a PR with a dummy change to make sure this was not fault on any PR, but just CI failing on our main branch.
I fixed the issue now!


enable s3 support on chewieConfig

add configuration for s3 to enable s3 usage inside docker image

skeleton was not fully ready for the s3 pipeline and I had to pimp it up
implementation is done in the way that there is one image that can be still used on local for local development or remotely in the pipeline
you just pass env PIPELINE=true to say you want to use it on remote

fixed with https://github.com/YaaS/docpad-skeleton-apidocs/pull/56


uaa command for service with passcode

via @soccermax
there should be a uaa command to get not grant=client_credential token, but a grant=password token for a passcode

Tentative new list of uaa commands:
   === user authentication (uaa) ===
~  uaad   --uaa-decode TOKEN                                     decode JSON web token
~  uaac   --uaa-client [TENANT]                                  obtain uaa token for generic client
~  uaap   --uaa-passcode PASSCODE [TENANT]                       obtain uaa token for one-time passcode
~  uaau   --uaa-user USERNAME PASSWORD [TENANT]                  obtain uaa token for username password
~  uaasc  --uaa-service-client SERVICE [TENANT]                  obtain service token for generic client
~  uaasp  --uaa-service-passcode SERVICE PASSCODE [TENANT]       obtain service token for one-time passcode
~  uaasu  --uaa-service-user SERVICE USERNAME PASSWORD [TENANT]  obtain service token for username password
          ...    [TENANT]                                        obtain token for tenant, fallback to paas tenant
          ...    --decode                                        decode result token
          ...    --userinfo                                      add detailed user info for passcode

uaas and the dedicated userinfo command would change.


Unpackaged application doesn't start on release 1.0

Describe the bug
Blank application doesn't start in unpackaged mode. Application ends immediately with System.Runtime.InteropServices.COMException in Microsoft.WindowsAppRuntime.Bootstrap.Net.dll
Exit code<PHONE_NUMBER> (0x80131623).
Steps to reproduce the bug

Follow steps on https://docs.microsoft.com/pl-pl/windows/apps/winui/winui3/create-your-first-winui3-app?pivots=winui3-unpackaged-csharp

Expected behavior
A window with "Click Me" message appears
Screenshots
No response
NuGet package version
1.0.0
Packaging type
Unpackaged
Windows version
Windows 11 (22000)
IDE
Visual Studio 2022-preview, Visual Studio 2022
Additional context
Tried to upgrade Microsoft.Windows.SDK.BuildTools to latest versions 10.0.22000.196 and 10.0.22499-preview - no change

Closing as it works after installing updated WindowsAppRuntime, I didn't upgrade it from preview3.
Some meaningful message about missing runtime would be nice


Various improvements to pilot tests to make them more hermetic

Fixes a few tests in pilot that rely on global state, test ordering, or time.Sleep, and enabled a test that was skipped for a resolved issue.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: howardjohn
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [howardjohn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-master
8026f4245bba4acc7d982bb5327ea49bc37b666c
link
/test integ-new-install-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/assign @nmittler

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: howardjohn, nmittler
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [howardjohn,nmittler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


feat(offloader): support kerberos config set for hdfs offloader

Is your feature request related to a problem? Please describe.
Usually hdfs is kerberos is enabled in production environment.
Describe the solution you'd like
Support kerberos config set for hdfs offloader.

@zymap PTAL