CUDA runtime error: CUBLAS_STATUS_INVALID_VALUE when running any PyTorch code sample

Description
I could build and run C++ encoder samples successfully as described in the README. Output sample:
 $ ./bin/encoder_sample 32 12 32 12 64 0 0 0 0
Device Tesla V100-SXM2-16GB
Device Tesla V100-SXM2-16GB
before allocate free 15.44 GB total 15.78 GB
After allocate free 15.40 GB used 0.38 GB total 15.78 GB
[INFO] batch_size 32 seq_len 32 layer 12 FT-CPP-time 15.40 ms ( 50 iterations)

However, running any PyTorch sample fails with a RuntimeError: [FT][ERROR] CUDA runtime error: CUBLAS_STATUS_INVALID_VALUE /workspace/FasterTransformer/fastertransformer/cuda/open_attention.h:939
PyTorch encoder example

Commands:

./bin/encoder_gemm 32 32 12 64 0 0
python pytorch/encoder_sample.py 32 12 32 12 64 --time

Output from pytorch/encoder_sample.py:

2021-09-07 15:58:23.469196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled.
Traceback of TorchScript (most recent call last):
  File "/workspace/FasterTransformer/build/pytorch/utils/encoder.py", line 200, in forward
            trt_seq_len = torch.cat([trt_seq_len, torch.tensor([batch * max_seq_len], device='cuda').to(trt_seq_len.dtype)], dim=0).to(torch.int32)
        for i in range(self.layer_num):
            hidden_states = self.encoders[i].forward(hidden_states, attention_mask, trt_seq_len, sequence_id_offset)
                            ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        if self.remove_padding:
            hidden_states = self.rebuild_padding(hidden_states, sequence_id_offset, attention_mask, 0)
RuntimeError: [FT][ERROR] CUDA runtime error: CUBLAS_STATUS_INVALID_VALUE /workspace/FasterTransformer/fastertransformer/cuda/open_attention.h:939

Expected behavior
PyTorch samples should run without errors.
My goal is to run python pytorch/run_translation.py --batch_size 128 --beam_size 4 --model_type decoding_ext --data_type fp32 which has exactly the same issue
Environment info

docker image: nvcr.io/nvidia/tensorflow:20.12-tf1-py3
software versions:

amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ nvidia-smi
Tue Sep  7 16:34:08 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00001D0E:00:00.0 Off |                    0 |
| N/A   32C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00003D8F:00:00.0 Off |                    0 |
| N/A   33C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00006B96:00:00.0 Off |                    0 |
| N/A   32C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 000084FE:00:00.0 Off |                    0 |
| N/A   33C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 0000C3F5:00:00.0 Off |                    0 |
| N/A   34C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 0000DCCF:00:00.0 Off |                    0 |
| N/A   33C    P0    44W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 0000E931:00:00.0 Off |                    0 |
| N/A   32C    P0    44W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 0000F01A:00:00.0 Off |                    0 |
| N/A   33C    P0    42W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ g++ --version
g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
Resolving deltas: 100% (851/851), done.
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer (main) (DISPLAY=)
 $ mkdir -p build && cd build
mkdir: created directory 'build'
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ cmake -DSM=70 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON .. && make -j 32 # V100
-- The CXX compiler identification is GNU 9.3.0
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "11.1", minimum required is "10.1")
-- Add DBUILD_GPT, requires MPI and NCCL
-- Found MPI_CXX: /usr/local/mpi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found NCCL: /usr/include
-- Determining NCCL version from /usr/include/nccl.h...
-- Looking for NCCL_VERSION_CODE
-- Looking for NCCL_VERSION_CODE - not found
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.8.3)
-- Add DCUDA11_MODE
-- Assign GPU architecture (sm=70)
-- Use WMMA
-- Found CUDA: /usr/local/cuda (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found cuDNN: v8.0.5  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 3a20f2b6
CMake Warning at /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/utils.cmake:365 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:511 (torch_cuda_get_nvcc_gencode_flag)
  /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:211 (find_package)

-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
CMake Warning at /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /home/amralaa/.local/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:211 (find_package)

-- Found Torch: /home/amralaa/.local/lib/python3.8/site-packages/torch/lib/libtorch.so
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/FasterTransformer/build
Scanning dependencies of target nccl_utils
Scanning dependencies of target cuda_kernels
Scanning dependencies of target attention_kernels
Scanning dependencies of target online_softmax_beamsearch
Scanning dependencies of target topk
Scanning dependencies of target transformer_kernels
Scanning dependencies of target encoder_igemm_func
Scanning dependencies of target nvtx_utils
Scanning dependencies of target copy
Scanning dependencies of target encoder_gemm_func
Scanning dependencies of target trt_fused_multi_head_attention
Scanning dependencies of target gpt_gemm
Scanning dependencies of target decoding_gemm
[  0%] Building CUDA object fastertransformer/cuda/CMakeFiles/cuda_kernels.dir/cuda_kernels.cu.o
[  0%] Building CUDA object fastertransformer/cuda/CMakeFiles/attention_kernels.dir/attention_kernels.cu.o
[  1%] Building CXX object fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cpp.o
[  2%] Building CUDA object fastertransformer/cuda/CMakeFiles/transformer_kernels.dir/transformer_kernels.cu.o
[  3%] Building CXX object fastertransformer/utils/CMakeFiles/nccl_utils.dir/nccl_utils.cpp.o
[  4%] Building CXX object fastertransformer/gemm_test/CMakeFiles/encoder_igemm_func.dir/encoder_igemm_func.cc.o
[  4%] Built target copy
[  5%] Building CXX object fastertransformer/gemm_test/CMakeFiles/encoder_gemm_func.dir/encoder_gemm_func.cc.o
[  6%] Building CUDA object fastertransformer/cuda/CMakeFiles/online_softmax_beamsearch.dir/online_softmax_beamsearch_kernels.cu.o
[  7%] Building CXX object tools/gemm_test/CMakeFiles/gpt_gemm.dir/gpt_gemm.cc.o
[  7%] Building CXX object tools/gemm_test/CMakeFiles/decoding_gemm.dir/decoding_gemm.cc.o
[  8%] Building CUDA object fastertransformer/cuda/CMakeFiles/topk.dir/topk_kernels.cu.o
[  8%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm75.cpp.o
[  9%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm75.cpp.o
[  9%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm75.cpp.o
[ 10%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm80.cpp.o
[ 11%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm75.cpp.o
[ 12%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm80.cpp.o
[ 13%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/cudaDriverWrapper.cpp.o
[ 14%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm80.cpp.o
[ 15%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm80.cpp.o
[ 16%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm75.cpp.o
[ 17%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm80.cpp.o
[ 18%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm75.cpp.o
[ 19%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm80.cpp.o
[ 20%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp.o
[ 20%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm70.cpp.o
[ 21%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp.o
[ 22%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp.o
[ 24%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm70.cpp.o
[ 24%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp.o
[ 25%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp.o
[ 25%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm70.cpp.o
[ 26%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp.o
[ 27%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp.o
[ 28%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm70.cpp.o
[ 29%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp.o
[ 30%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm70.cpp.o
[ 30%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp.o
[ 31%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp.o
[ 32%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp.o
[ 33%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp.o
[ 34%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp.o
[ 35%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp.o
[ 35%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp.o
[ 36%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp.o
[ 37%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp.o
[ 38%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp.o
[ 39%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp.o
[ 40%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp.o
[ 40%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp.o
[ 41%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp.o
[ 42%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm75.cpp.o
[ 43%] Linking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o
[ 43%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm80.cpp.o
[ 44%] Building CUDA object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/qkvToContext.cu.o
[ 45%] Linking CXX static library ../../lib/libnvtx_utils.a
[ 45%] Built target nvtx_utils
[ 46%] Linking CUDA device code CMakeFiles/nccl_utils.dir/cmake_device_link.o
[ 47%] Linking CXX static library ../../lib/libnccl_utils.a
[ 47%] Built target nccl_utils
[ 47%] Linking CXX static library ../../lib/libencoder_igemm_func.a
[ 48%] Linking CXX executable ../../bin/decoding_gemm
[ 48%] Built target encoder_igemm_func
[ 48%] Linking CXX executable ../../bin/gpt_gemm
[ 48%] Built target decoding_gemm
[ 49%] Linking CUDA device code CMakeFiles/encoder_gemm_func.dir/cmake_device_link.o
[ 49%] Built target gpt_gemm
[ 50%] Linking CXX static library ../../lib/libencoder_gemm_func.a
[ 50%] Built target encoder_gemm_func
Scanning dependencies of target encoder_gemm
[ 50%] Building CXX object tools/gemm_test/CMakeFiles/encoder_gemm.dir/encoder_gemm.cc.o
[ 51%] Linking CXX executable ../../bin/encoder_gemm
[ 51%] Built target encoder_gemm
[ 52%] Linking CUDA device code CMakeFiles/attention_kernels.dir/cmake_device_link.o
[ 53%] Linking CUDA static library ../../lib/libattention_kernels.a
[ 53%] Built target attention_kernels
[ 54%] Linking CUDA device code CMakeFiles/transformer_kernels.dir/cmake_device_link.o
[ 55%] Linking CUDA static library ../../lib/libtransformer_kernels.a
[ 55%] Built target transformer_kernels
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In function 'void fastertransformer::set_alpha(uint32_t&, float, fastertransformer::Data_type)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:34:46: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
   34 |         alpha = reinterpret_cast<const uint32_t&>(h2);
      |                                              ^~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:38:46: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
   38 |         alpha = reinterpret_cast<const uint32_t&>(norm);
      |                                              ^~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In member function 'void fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:329:62: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  329 |         params.scale_bmm1 = reinterpret_cast<const uint32_t&>(scaleBmm1);
      |                                                              ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:330:62: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  330 |         params.scale_bmm2 = reinterpret_cast<const uint32_t&>(scaleBmm2);
      |                                                              ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:331:65: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  331 |         params.scale_softmax = reinterpret_cast<const uint32_t&>(scaleSoftmax);
      |                                                                 ^~~~~~~~~~~~
[ 56%] Linking CUDA device code CMakeFiles/cuda_kernels.dir/cmake_device_link.o
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In member function 'virtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:280:8: warning: 'warps_m' may be used uninitialized in this function [-Wmaybe-uninitialized]
  280 |         size_t warps_m, warps_n, warps_k = 1;
      |        ^~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:280:17: warning: 'warps_n' may be used uninitialized in this function [-Wmaybe-uninitialized]
  280 |         size_t warps_m, warps_n, warps_k = 1;
      |                 ^~~~~~~
[ 57%] Linking CUDA static library ../../lib/libcuda_kernels.a
[ 57%] Built target cuda_kernels
Scanning dependencies of target cuda_int8_kernels
Scanning dependencies of target decoder
[ 58%] Building CUDA object fastertransformer/cuda/CMakeFiles/cuda_int8_kernels.dir/cuda_int8_kernels.cu.o
[ 59%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoder.dir/open_decoder.cu.o
[ 60%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoder.dir/masked_multihead_attention.cu.o
[ 61%] Linking CUDA device code CMakeFiles/trt_fused_multi_head_attention.dir/cmake_device_link.o
[ 62%] Linking CXX static library ../../lib/libtrt_fused_multi_head_attention.a
[ 62%] Built target trt_fused_multi_head_attention
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu(303): warning: specified alignment (2) is different from alignment (4) specified on a previous declaration
          detected during:
            instantiation of "void fastertransformer::masked_attention_kernel(T *, T *, T *, const T *, T *, const T *, T *, const T *, T *, const __nv_bool *, int, int, int, int, T) [with T=half]"
(445): here
            instantiation of "void fastertransformer::masked_attention_dispatch(T *, T *, T *, const T *, T *, const T *, T *, const T *, T *, const __nv_bool *, int, int, int, int, int, int, cudaStream_t) [with T=half]"
(503): here

/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu(1420): warning: specified alignment (2) is different from alignment (4) specified on a previous declaration
          detected during:
            instantiation of "void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]"
(1555): here
            instantiation of "void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, cudaStream_t) [with T=half]"
(1576): here

[ 63%] Linking CUDA device code CMakeFiles/cuda_int8_kernels.dir/cmake_device_link.o
[ 64%] Linking CUDA static library ../../lib/libcuda_int8_kernels.a
[ 64%] Built target cuda_int8_kernels
Scanning dependencies of target encoder
[ 65%] Building CUDA object fastertransformer/cuda/CMakeFiles/encoder.dir/open_attention.cu.o
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::masked_attention_dispatch(T*, T*, T*, const T*, T*, const T*, T*, const T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:484:398:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:454:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  454 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::masked_attention_dispatch(T*, T*, T*, const T*, T*, const T*, T*, const T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:503:389:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:454:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  454 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:781:318:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:750:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  750 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:796:313:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:750:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  750 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch_v2(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, int, const int*, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:883:373:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:851:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  851 |   memset(&params, 0, sizeof(params));
      |   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch_v2(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, int, const int*, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:900:368:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:851:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  851 |   memset(&params, 0, sizeof(params));
      |   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 66%] Linking CUDA device code CMakeFiles/decoder.dir/cmake_device_link.o
[ 66%] Linking CXX static library ../../lib/libdecoder.a
[ 66%] Built target decoder
[ 67%] Linking CUDA device code CMakeFiles/encoder.dir/cmake_device_link.o
[ 68%] Linking CXX static library ../../lib/libencoder.a
[ 68%] Built target encoder
Scanning dependencies of target encoder_sample
[ 69%] Building CXX object sample/cpp/CMakeFiles/encoder_sample.dir/encoder_sample.cc.o
[ 70%] Linking CXX executable ../../bin/encoder_sample
[ 70%] Built target encoder_sample
[ 71%] Linking CUDA device code CMakeFiles/online_softmax_beamsearch.dir/cmake_device_link.o
[ 72%] Linking CUDA static library ../../lib/libonline_softmax_beamsearch.a
[ 72%] Built target online_softmax_beamsearch
[ 73%] Linking CUDA device code CMakeFiles/topk.dir/cmake_device_link.o
[ 74%] Linking CUDA static library ../../lib/libtopk.a
[ 74%] Built target topk
Scanning dependencies of target decoding
[ 75%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoding.dir/decoding_kernels.cu.o
[ 76%] Linking CUDA device code CMakeFiles/decoding.dir/cmake_device_link.o
[ 77%] Linking CUDA static library ../../lib/libdecoding.a
[ 77%] Built target decoding
Scanning dependencies of target gpt_triton_backend
Scanning dependencies of target decoding_sampling_sample
Scanning dependencies of target transformer-static
Scanning dependencies of target pyt_fastertransformer
Scanning dependencies of target gpt_sample
Scanning dependencies of target decoding_beamsearch_sample
[ 78%] Linking CUDA device code CMakeFiles/transformer-static.dir/cmake_device_link.o
[ 79%] Building CXX object sample/cpp/CMakeFiles/decoding_sampling_sample.dir/decoding_sampling_sample.cc.o
[ 80%] Building CXX object fastertransformer/triton_backend/CMakeFiles/gpt_triton_backend.dir/gpt_triton_backend.cc.o
[ 81%] Building CXX object sample/cpp/CMakeFiles/decoding_beamsearch_sample.dir/decoding_beamsearch_sample.cc.o
[ 82%] Building CXX object sample/cpp/CMakeFiles/gpt_sample.dir/gpt_sample.cc.o
[ 83%] Linking CXX static library lib/libtransformer-static.a
[ 84%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/ft_op.cc.o
[ 84%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/encoder.cc.o
[ 85%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/decoder.cc.o
[ 86%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/decoding.cc.o
[ 87%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/gpt.cc.o
[ 88%] Building CUDA object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/utils.cu.o
[ 89%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/weight_quantize_op.cc.o
[ 89%] Built target transformer-static
Scanning dependencies of target gpt
[ 90%] Building CXX object CMakeFiles/gpt.dir/sample/cpp/gpt_sample.cc.o
[ 91%] Linking CXX executable ../../bin/decoding_sampling_sample
[ 92%] Linking CXX executable ../../bin/decoding_beamsearch_sample
[ 92%] Built target decoding_sampling_sample
[ 92%] Built target decoding_beamsearch_sample
[ 93%] Linking CXX executable ../../bin/gpt_sample
[ 93%] Built target gpt_sample
[ 94%] Linking CXX executable bin/gpt
[ 94%] Built target gpt
[ 94%] Linking CXX shared library ../../lib/libgpt_triton_backend.so
[ 94%] Built target gpt_triton_backend
Scanning dependencies of target transformer-shared
Scanning dependencies of target gpt_thread_sample
Scanning dependencies of target gpt_triton_sample
[ 95%] Linking CUDA device code CMakeFiles/transformer-shared.dir/cmake_device_link.o
[ 97%] Building CXX object sample/cpp/CMakeFiles/gpt_triton_sample.dir/gpt_triton_sample.cc.o
[ 97%] Building CXX object sample/cpp/CMakeFiles/gpt_thread_sample.dir/gpt_thread_sample.cc.o
[ 97%] Linking CXX shared library lib/libtransformer-shared.so
[ 97%] Built target transformer-shared
[ 98%] Linking CXX executable ../../bin/gpt_triton_sample
[ 98%] Built target gpt_triton_sample
[ 99%] Linking CXX executable ../../bin/gpt_thread_sample
[ 99%] Built target gpt_thread_sample
[ 99%] Linking CUDA device code CMakeFiles/pyt_fastertransformer.dir/cmake_device_link.o
[100%] Linking CXX shared library ../../lib/libpyt_fastertransformer.so
[100%] Built target pyt_fastertransformer
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 python -c 'import torch; torch.ops.load_library("./lib/libpyt_fastertransformer.so")'                                                              130 ↵
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 ././bin/encoder_gemm 32 32 12 64 0 0                                                                                130 ↵
./bin/encoder_sample 32 12 32 12 64 0 0 0 0

Device Tesla V100-SXM2-16GB
***Encoder Gemm Testing Begin***
***Cublas Gemm Testing Begin***

-----------------------------
GEMM test 0: [M: 1024, K: 768, N: 768] from_tensor * weightQ/K/V, attr * output_kernel
algo_-1 costs 0.123ms
algo_2 costs 0.129ms
algo_3 costs 0.165ms
algo_4 costs 0.124ms
algo_5 costs 0.160ms
algo_6 costs 0.159ms
algo_7 costs 0.115ms
algo_8 costs 0.132ms
algo_9 costs 0.111ms
algo_10 costs 0.128ms
algo_11 costs 0.154ms
algo_18 costs 0.119ms
algo_19 costs 0.101ms
algo_20 costs 0.117ms
algo_21 costs 0.130ms
algo_22 costs 0.112ms
algo_23 costs 0.115ms
fast_algo 19 costs 0.101 ms

-----------------------------
GEMM test 1: [M: 1024, K: 768, N: 3072] attr_output * inter_kernel
algo_-1 costs 0.384ms
algo_2 costs 0.407ms
algo_3 costs 0.410ms
algo_4 costs 0.365ms
algo_5 costs 0.343ms
algo_6 costs 0.406ms
algo_7 costs 0.366ms
algo_8 costs 0.367ms
algo_9 costs 0.348ms
algo_10 costs 0.351ms
algo_11 costs 0.349ms
algo_18 costs 0.439ms
algo_19 costs 0.383ms
algo_20 costs 0.369ms
algo_21 costs 0.444ms
algo_22 costs 0.391ms
algo_23 costs 0.431ms
fast_algo 5 costs 0.343 ms

-----------------------------
GEMM test 2: [M: 1024, K: 3072, N: 768] inter_matmul * output_kernel
algo_-1 costs 0.366ms
algo_1 costs 0.631ms
algo_2 costs 0.414ms
algo_3 costs 0.543ms
algo_4 costs 0.402ms
algo_5 costs 0.525ms
algo_6 costs 0.525ms
algo_7 costs 0.361ms
algo_8 costs 0.377ms
algo_9 costs 0.347ms
algo_10 costs 0.370ms
algo_11 costs 0.375ms
algo_18 costs 0.444ms
algo_19 costs 0.360ms
algo_20 costs 0.404ms
algo_21 costs 0.441ms
algo_22 costs 0.383ms
algo_23 costs 0.407ms
fast_algo 9 costs 0.347 ms

-----------------------------
GEMM test 3: [M: 32, K: 64, N: 32] attention batched Gemm1
algo_-1 costs 0.029ms
algo_0 costs 0.022ms
algo_1 costs 0.026ms
algo_2 costs 0.022ms
algo_3 costs 0.036ms
algo_4 costs 0.066ms
algo_5 costs 0.022ms
algo_6 costs 0.026ms
algo_7 costs 0.022ms
algo_8 costs 0.036ms
algo_9 costs 0.066ms
algo_18 costs 0.019ms
algo_19 costs 0.024ms
algo_20 costs 0.039ms
algo_21 costs 0.019ms
algo_22 costs 0.024ms
algo_23 costs 0.039ms
fast_algo 21 costs 0.019 ms

-----------------------------
GEMM test 4: [M: 32, K: 32, N: 64] attention batched Gemm2
algo_-1 costs 0.017ms
algo_0 costs 0.024ms
algo_1 costs 0.017ms
algo_2 costs 0.016ms
algo_3 costs 0.023ms
algo_4 costs 0.040ms
algo_5 costs 0.024ms
algo_6 costs 0.017ms
algo_7 costs 0.016ms
algo_8 costs 0.023ms
algo_9 costs 0.040ms
algo_18 costs 0.022ms
algo_19 costs 0.020ms
algo_20 costs 0.031ms
algo_21 costs 0.022ms
algo_22 costs 0.020ms
algo_23 costs 0.031ms
fast_algo 7 costs 0.016 ms

-----------------------------
GEMM test 5: [M: 1024, K: 768, N: 768] from_tensor * weight_QKV in BatchGemm
algo_-1 costs 0.273ms
algo_0 costs 0.286ms
algo_1 costs 0.272ms
algo_2 costs 0.275ms
algo_3 costs 0.267ms
algo_4 costs 0.270ms
algo_5 costs 0.286ms
algo_6 costs 0.273ms
algo_7 costs 0.274ms
algo_8 costs 0.267ms
algo_9 costs 0.270ms
algo_18 costs 0.309ms
algo_19 costs 0.277ms
algo_20 costs 0.294ms
algo_21 costs 0.309ms
algo_22 costs 0.277ms
algo_23 costs 0.294ms
fast_algo 8 costs 0.267 ms
***cublas Gemm Testing End***

***Encoder Gemm Testing End***
Device Tesla V100-SXM2-16GB
Device Tesla V100-SXM2-16GB
before allocate free 15.44 GB total 15.78 GB
After allocate free 15.40 GB used 0.38 GB total 15.78 GB
[INFO] batch_size 32 seq_len 32 layer 12 FT-CPP-time 15.40 ms ( 50 iterations)
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ ./bin/encoder_gemm 32 32 12 64 0 0
python pytorch/encoder_sample.py 32 12 32 12 64 --time

Device Tesla V100-SXM2-16GB
***Encoder Gemm Testing Begin***
***Cublas Gemm Testing Begin***

-----------------------------
GEMM test 0: [M: 1024, K: 768, N: 768] from_tensor * weightQ/K/V, attr * output_kernel
algo_-1 costs 0.109ms
algo_2 costs 0.114ms
algo_3 costs 0.145ms
algo_4 costs 0.109ms
algo_5 costs 0.141ms
algo_6 costs 0.141ms
algo_7 costs 0.103ms
algo_8 costs 0.116ms
algo_9 costs 0.097ms
algo_10 costs 0.111ms
algo_11 costs 0.143ms
algo_18 costs 0.113ms
algo_19 costs 0.096ms
algo_20 costs 0.111ms
algo_21 costs 0.124ms
algo_22 costs 0.107ms
algo_23 costs 0.108ms
fast_algo 19 costs 0.096 ms

-----------------------------
GEMM test 1: [M: 1024, K: 768, N: 3072] attr_output * inter_kernel
algo_-1 costs 0.361ms
algo_2 costs 0.384ms
algo_3 costs 0.404ms
algo_4 costs 0.366ms
algo_5 costs 0.344ms
algo_6 costs 0.407ms
algo_7 costs 0.366ms
algo_8 costs 0.367ms
algo_9 costs 0.349ms
algo_10 costs 0.351ms
algo_11 costs 0.347ms
algo_18 costs 0.439ms
algo_19 costs 0.382ms
algo_20 costs 0.367ms
algo_21 costs 0.442ms
algo_22 costs 0.391ms
algo_23 costs 0.431ms
fast_algo 5 costs 0.344 ms

-----------------------------
GEMM test 2: [M: 1024, K: 3072, N: 768] inter_matmul * output_kernel
algo_-1 costs 0.365ms
algo_1 costs 0.629ms
algo_2 costs 0.414ms
algo_3 costs 0.543ms
algo_4 costs 0.402ms
algo_5 costs 0.526ms
algo_6 costs 0.530ms
algo_7 costs 0.363ms
algo_8 costs 0.377ms
algo_9 costs 0.348ms
algo_10 costs 0.370ms
algo_11 costs 0.375ms
algo_18 costs 0.445ms
algo_19 costs 0.357ms
algo_20 costs 0.404ms
algo_21 costs 0.442ms
algo_22 costs 0.384ms
algo_23 costs 0.406ms
fast_algo 9 costs 0.348 ms

-----------------------------
GEMM test 3: [M: 32, K: 64, N: 32] attention batched Gemm1
algo_-1 costs 0.029ms
algo_0 costs 0.022ms
algo_1 costs 0.026ms
algo_2 costs 0.022ms
algo_3 costs 0.036ms
algo_4 costs 0.066ms
algo_5 costs 0.022ms
algo_6 costs 0.026ms
algo_7 costs 0.022ms
algo_8 costs 0.036ms
algo_9 costs 0.066ms
algo_18 costs 0.019ms
algo_19 costs 0.024ms
algo_20 costs 0.039ms
algo_21 costs 0.019ms
algo_22 costs 0.024ms
algo_23 costs 0.039ms
fast_algo 21 costs 0.019 ms

-----------------------------
GEMM test 4: [M: 32, K: 32, N: 64] attention batched Gemm2
algo_-1 costs 0.017ms
algo_0 costs 0.024ms
algo_1 costs 0.017ms
algo_2 costs 0.016ms
algo_3 costs 0.023ms
algo_4 costs 0.040ms
algo_5 costs 0.024ms
algo_6 costs 0.017ms
algo_7 costs 0.016ms
algo_8 costs 0.023ms
algo_9 costs 0.040ms
algo_18 costs 0.022ms
algo_19 costs 0.020ms
algo_20 costs 0.031ms
algo_21 costs 0.022ms
algo_22 costs 0.020ms
algo_23 costs 0.031ms
fast_algo 7 costs 0.016 ms

-----------------------------
GEMM test 5: [M: 1024, K: 768, N: 768] from_tensor * weight_QKV in BatchGemm
algo_-1 costs 0.273ms
algo_0 costs 0.286ms
algo_1 costs 0.273ms
algo_2 costs 0.274ms
algo_3 costs 0.267ms
algo_4 costs 0.270ms
algo_5 costs 0.286ms
algo_6 costs 0.273ms
algo_7 costs 0.276ms
algo_8 costs 0.267ms
algo_9 costs 0.270ms
algo_18 costs 0.309ms
algo_19 costs 0.277ms
algo_20 costs 0.294ms
algo_21 costs 0.309ms
algo_22 costs 0.277ms
algo_23 costs 0.294ms
fast_algo 8 costs 0.267 ms
***cublas Gemm Testing End***

***Encoder Gemm Testing End***
2021-09-07 15:58:23.469196: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled.
Traceback of TorchScript (most recent call last):
  File "/workspace/FasterTransformer/build/pytorch/utils/encoder.py", line 200, in forward
            trt_seq_len = torch.cat([trt_seq_len, torch.tensor([batch * max_seq_len], device='cuda').to(trt_seq_len.dtype)], dim=0).to(torch.int32)
        for i in range(self.layer_num):
            hidden_states = self.encoders[i].forward(hidden_states, attention_mask, trt_seq_len, sequence_id_offset)
                            ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        if self.remove_padding:
            hidden_states = self.rebuild_padding(hidden_states, sequence_id_offset, attention_mask, 0)
RuntimeError: [FT][ERROR] CUDA runtime error: CUBLAS_STATUS_INVALID_VALUE /workspace/FasterTransformer/fastertransformer/cuda/open_attention.h:939

amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ nvidia-smi
Tue Sep  7 16:34:08 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00001D0E:00:00.0 Off |                    0 |
| N/A   32C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00003D8F:00:00.0 Off |                    0 |
| N/A   33C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00006B96:00:00.0 Off |                    0 |
| N/A   32C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 000084FE:00:00.0 Off |                    0 |
| N/A   33C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 0000C3F5:00:00.0 Off |                    0 |
| N/A   34C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 0000DCCF:00:00.0 Off |                    0 |
| N/A   33C    P0    44W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 0000E931:00:00.0 Off |                    0 |
| N/A   32C    P0    44W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 0000F01A:00:00.0 Off |                    0 |
| N/A   33C    P0    42W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

amralaa@19d361576f0b4b8ad39e81c5537c320a-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ g++ --version
g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
Compilation didn't go smoothly with PyTorch image nvcr.io/nvidia/pytorch:21.08-py3, CMake configure fails because it cannot find MPI (-- Add DBUILD_GPT, requires MPI and NCCL -- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS)).
When I try to disable BUILD_GPT I get many compilation errors like /workspace/FasterTransformer/fastertransformer/gpt.h:370:55: error: 'struct LayerParallelParam' has no member named 'nccl_comm'   370 |                                     l_parallel_param_.nccl_comm, decoding_params.stream);
Full log

amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace (DISPLAY=)
 $ cd /workspace/FasterTransformer
mkdir -p build && cd build
mkdir: created directory 'build'
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer/build (DISPLAY=)
 $ cmake -DSM=70 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON .. && make -j 32 # V100
-- The CXX compiler identification is GNU 9.3.0
-- The CUDA compiler identification is NVIDIA 11.4.100
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "11.4", minimum required is "10.1")
-- Add DBUILD_GPT, requires MPI and NCCL
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS)
CMake Error at /opt/conda/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_CXX_FOUND)
Call Stack (most recent call first):
  /opt/conda/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /opt/conda/share/cmake-3.21/Modules/FindMPI.cmake:1747 (find_package_handle_standard_args)
  CMakeLists.txt:27 (find_package)

-- Configuring incomplete, errors occurred!
See also "/workspace/FasterTransformer/build/CMakeFiles/CMakeOutput.log".
See also "/workspace/FasterTransformer/build/CMakeFiles/CMakeError.log".
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $                                                                                130 ↵
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ cd ..
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer (main) (DISPLAY=)
 $ rm -rf build
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer (main) (DISPLAY=)
 $ mkcd build
mkdir: created directory 'build'
amralaa@38499b6e03946c10280aea7b6393e4dc-master-0:/workspace/FasterTransformer/build (main) (DISPLAY=)
 $ cmake -DSM=70 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_GPT=OFF .. && make -j 32 # V100
-- The CXX compiler identification is GNU 9.3.0
-- The CUDA compiler identification is NVIDIA 11.4.100
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "11.4", minimum required is "10.1")
-- Add DCUDA11_MODE
-- Assign GPU architecture (sm=70)
-- Use WMMA
-- Found CUDA: /usr/local/cuda (found version "11.4")
-- Caffe2: CUDA detected: 11.4
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.4
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found cuDNN: v8.2.2  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 5efb87b3
CMake Warning at /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/utils.cmake:365 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:513 (torch_cuda_get_nvcc_gencode_flag)
  /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:211 (find_package)

-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
CMake Warning at /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /opt/conda/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:211 (find_package)

-- Found Torch: /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch.so
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/FasterTransformer/build
[  1%] Building CUDA object fastertransformer/cuda/CMakeFiles/topk.dir/topk_kernels.cu.o
[  2%] Building CXX object fastertransformer/utils/CMakeFiles/nccl_utils.dir/nccl_utils.cpp.o
[  3%] Building CUDA object fastertransformer/cuda/CMakeFiles/transformer_kernels.dir/transformer_kernels.cu.o
[  4%] Building CXX object fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cpp.o
[  4%] Building CUDA object fastertransformer/cuda/CMakeFiles/attention_kernels.dir/attention_kernels.cu.o
[  4%] Building CUDA object fastertransformer/cuda/CMakeFiles/online_softmax_beamsearch.dir/online_softmax_beamsearch_kernels.cu.o
[  5%] Building CXX object fastertransformer/gemm_test/CMakeFiles/encoder_igemm_func.dir/encoder_igemm_func.cc.o
[  6%] Building CXX object fastertransformer/gemm_test/CMakeFiles/encoder_gemm_func.dir/encoder_gemm_func.cc.o
[  7%] Building CXX object tools/gemm_test/CMakeFiles/decoding_gemm.dir/decoding_gemm.cc.o
[  8%] Building CXX object tools/gemm_test/CMakeFiles/gpt_gemm.dir/gpt_gemm.cc.o
[  9%] Building CUDA object fastertransformer/cuda/CMakeFiles/cuda_kernels.dir/cuda_kernels.cu.o
[ 10%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm75.cpp.o
[ 10%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm75.cpp.o
[ 11%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm80.cpp.o
[ 12%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm75.cpp.o
[ 13%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm80.cpp.o
[ 14%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm75.cpp.o
[ 15%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/cudaDriverWrapper.cpp.o
[ 16%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm80.cpp.o
[ 17%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm70.cpp.o
[ 18%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm80.cpp.o
[ 19%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm80.cpp.o
[ 20%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm75.cpp.o
[ 22%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm75.cpp.o
[ 22%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm80.cpp.o
[ 24%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp.o
[ 24%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp.o
[ 25%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp.o
[ 26%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp.o
[ 27%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm70.cpp.o
[ 28%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm70.cpp.o
[ 28%] Built target copy
[ 29%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp.o
[ 30%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp.o
[ 31%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm70.cpp.o
[ 32%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp.o
[ 32%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp.o
[ 33%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm70.cpp.o
[ 34%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp.o
[ 35%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp.o
[ 36%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp.o
[ 37%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp.o
[ 38%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp.o
[ 39%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp.o
[ 40%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp.o
[ 41%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp.o
[ 42%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp.o
[ 43%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp.o
[ 44%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp.o
[ 45%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp.o
[ 46%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp.o
[ 47%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp.o
[ 48%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm75.cpp.o
[ 49%] Building CXX object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm80.cpp.o
[ 50%] Linking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o
[ 51%] Building CUDA object fastertransformer/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/qkvToContext.cu.o
[ 52%] Linking CXX static library ../../lib/libnvtx_utils.a
[ 53%] Linking CUDA device code CMakeFiles/nccl_utils.dir/cmake_device_link.o
[ 53%] Built target nvtx_utils
[ 54%] Linking CXX static library ../../lib/libnccl_utils.a
[ 54%] Built target nccl_utils
[ 55%] Linking CXX executable ../../bin/decoding_gemm
[ 56%] Linking CXX static library ../../lib/libencoder_igemm_func.a
[ 56%] Built target encoder_igemm_func
[ 57%] Linking CXX executable ../../bin/gpt_gemm
[ 57%] Built target decoding_gemm
[ 58%] Linking CUDA device code CMakeFiles/encoder_gemm_func.dir/cmake_device_link.o
[ 58%] Built target gpt_gemm
[ 59%] Linking CXX static library ../../lib/libencoder_gemm_func.a
[ 59%] Built target encoder_gemm_func
[ 60%] Building CXX object tools/gemm_test/CMakeFiles/encoder_gemm.dir/encoder_gemm.cc.o
[ 61%] Linking CXX executable ../../bin/encoder_gemm
[ 61%] Built target encoder_gemm
[ 62%] Linking CUDA device code CMakeFiles/attention_kernels.dir/cmake_device_link.o
[ 63%] Linking CUDA static library ../../lib/libattention_kernels.a
[ 63%] Built target attention_kernels
[ 64%] Linking CUDA device code CMakeFiles/transformer_kernels.dir/cmake_device_link.o
[ 65%] Linking CUDA static library ../../lib/libtransformer_kernels.a
[ 65%] Built target transformer_kernels
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In function 'void fastertransformer::set_alpha(uint32_t&, float, fastertransformer::Data_type)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:34:46: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
   34 |         alpha = reinterpret_cast<const uint32_t&>(h2);
      |                                              ^~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:38:46: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
   38 |         alpha = reinterpret_cast<const uint32_t&>(norm);
      |                                              ^~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In member function 'void fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:329:62: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  329 |         params.scale_bmm1 = reinterpret_cast<const uint32_t&>(scaleBmm1);
      |                                                              ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:330:62: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  330 |         params.scale_bmm2 = reinterpret_cast<const uint32_t&>(scaleBmm2);
      |                                                              ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:331:65: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  331 |         params.scale_softmax = reinterpret_cast<const uint32_t&>(scaleSoftmax);
      |                                                                 ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu: In member function 'virtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int)':
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:280:8: warning: 'warps_m' may be used uninitialized in this function [-Wmaybe-uninitialized]
  280 |         size_t warps_m, warps_n, warps_k = 1;
      |        ^~~~~~~
/workspace/FasterTransformer/fastertransformer/trt_fused_multihead_attention/qkvToContext.cu:280:17: warning: 'warps_n' may be used uninitialized in this function [-Wmaybe-uninitialized]
  280 |         size_t warps_m, warps_n, warps_k = 1;
      |                 ^~~~~~~
[ 66%] Linking CUDA device code CMakeFiles/cuda_kernels.dir/cmake_device_link.o
[ 67%] Linking CUDA static library ../../lib/libcuda_kernels.a
[ 67%] Built target cuda_kernels
[ 69%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoder.dir/masked_multihead_attention.cu.o
[ 69%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoder.dir/open_decoder.cu.o
[ 70%] Building CUDA object fastertransformer/cuda/CMakeFiles/cuda_int8_kernels.dir/cuda_int8_kernels.cu.o
[ 71%] Linking CUDA device code CMakeFiles/trt_fused_multi_head_attention.dir/cmake_device_link.o
[ 72%] Linking CXX static library ../../lib/libtrt_fused_multi_head_attention.a
[ 72%] Built target trt_fused_multi_head_attention
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu(303): warning: specified alignment (2) is different from alignment (4) specified on a previous declaration
          detected during:
            instantiation of "void fastertransformer::masked_attention_kernel(T *, T *, T *, const T *, T *, const T *, T *, const T *, T *, const __nv_bool *, int, int, int, int, T) [with T=half]"
(445): here
            instantiation of "void fastertransformer::masked_attention_dispatch(T *, T *, T *, const T *, T *, const T *, T *, const T *, T *, const __nv_bool *, int, int, int, int, int, int, cudaStream_t) [with T=half]"
(503): here

/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu(1420): warning: specified alignment (2) is different from alignment (4) specified on a previous declaration
          detected during:
            instantiation of "void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]"
(1555): here
            instantiation of "void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, cudaStream_t) [with T=half]"
(1576): here

[ 73%] Linking CUDA device code CMakeFiles/cuda_int8_kernels.dir/cmake_device_link.o
[ 74%] Linking CUDA static library ../../lib/libcuda_int8_kernels.a
[ 74%] Built target cuda_int8_kernels
[ 75%] Building CUDA object fastertransformer/cuda/CMakeFiles/encoder.dir/open_attention.cu.o
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::masked_attention_dispatch(T*, T*, T*, const T*, T*, const T*, T*, const T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:484:398:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:454:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  454 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::masked_attention_dispatch(T*, T*, T*, const T*, T*, const T*, T*, const T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:503:389:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:454:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  454 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:781:318:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:750:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  750 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:796:313:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:750:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  750 |     memset(&params, 0, sizeof(params));
      |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch_v2(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, int, const int*, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:883:373:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:851:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<float>'; use assignment or value-initialization instead [-Wclass-memaccess]
  851 |   memset(&params, 0, sizeof(params));
      |   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<float>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu: In instantiation of 'void fastertransformer::fusedQKV_masked_attention_dispatch_v2(const T*, const T*, T*, T*, T*, const bool*, int, int, int, int, int, int, int, const int*, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]':
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:900:368:   required from here
/workspace/FasterTransformer/fastertransformer/cuda/open_decoder.cu:851:7: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct Masked_multihead_attention_params<short unsigned int>'; use assignment or value-initialization instead [-Wclass-memaccess]
  851 |   memset(&params, 0, sizeof(params));
      |   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/cuda/masked_multihead_attention.h:48:8: note: 'struct Masked_multihead_attention_params<short unsigned int>' declared here
   48 | struct Masked_multihead_attention_params {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 76%] Linking CUDA device code CMakeFiles/decoder.dir/cmake_device_link.o
[ 77%] Linking CXX static library ../../lib/libdecoder.a
[ 77%] Built target decoder
[ 78%] Linking CUDA device code CMakeFiles/encoder.dir/cmake_device_link.o
[ 79%] Linking CXX static library ../../lib/libencoder.a
[ 79%] Built target encoder
[ 80%] Building CXX object sample/cpp/CMakeFiles/encoder_sample.dir/encoder_sample.cc.o
[ 81%] Linking CXX executable ../../bin/encoder_sample
[ 81%] Built target encoder_sample
[ 82%] Linking CUDA device code CMakeFiles/online_softmax_beamsearch.dir/cmake_device_link.o
[ 83%] Linking CUDA static library ../../lib/libonline_softmax_beamsearch.a
[ 83%] Built target online_softmax_beamsearch
[ 84%] Linking CUDA device code CMakeFiles/topk.dir/cmake_device_link.o
[ 85%] Linking CUDA static library ../../lib/libtopk.a
[ 85%] Built target topk
[ 86%] Building CUDA object fastertransformer/cuda/CMakeFiles/decoding.dir/decoding_kernels.cu.o
[ 87%] Linking CUDA device code CMakeFiles/decoding.dir/cmake_device_link.o
[ 88%] Linking CUDA static library ../../lib/libdecoding.a
[ 88%] Built target decoding
[ 89%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/ft_op.cc.o
[ 90%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/encoder.cc.o
[ 91%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/decoder.cc.o
[ 92%] Building CUDA object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/utils.cu.o
[ 93%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/decoding.cc.o
[ 96%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/gpt.cc.o
[ 96%] Building CXX object sample/cpp/CMakeFiles/decoding_sampling_sample.dir/decoding_sampling_sample.cc.o
[ 96%] Building CXX object fastertransformer/th_op/CMakeFiles/pyt_fastertransformer.dir/weight_quantize_op.cc.o
[ 97%] Building CXX object sample/cpp/CMakeFiles/decoding_beamsearch_sample.dir/decoding_beamsearch_sample.cc.o
[ 97%] Linking CXX executable ../../bin/decoding_sampling_sample
[ 98%] Linking CXX executable ../../bin/decoding_beamsearch_sample
[ 98%] Built target decoding_sampling_sample
[ 98%] Built target decoding_beamsearch_sample
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/gpt.h: In member function 'void fastertransformer::DecodingGpt<OpType_>::forward_context(const fastertransformer::DecoderInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>*, fastertransformer::DecodingInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>)':
/workspace/FasterTransformer/fastertransformer/gpt.h:370:55: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  370 |                                     l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:414:55: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  414 |                                     l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h: In member function 'void fastertransformer::DecodingGpt<OpType_>::forward(const fastertransformer::DecoderInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>*, fastertransformer::DecodingInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>)':
/workspace/FasterTransformer/fastertransformer/gpt.h:494:87: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  494 |                                   l_parallel_param_.world_size - 1, l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                                                ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:561:57: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  561 |                                       l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                         ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:616:57: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  616 |                                       l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                         ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:840:124: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  840 |                     nccl_send(decoding_params.output_ids + step * m + ite * local_batch, local_batch, 0, l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                                                ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:869:45: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  869 |                           l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                             ^~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: At global scope:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:172:23: error: 'ncclUniqueId' has not been declared
  172 |   void init_nccl_comm(ncclUniqueId &tensor_para_nccl_uid, ncclUniqueId &layer_para_nccl_uid,
      |                       ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:172:59: error: 'ncclUniqueId' has not been declared
  172 |   void init_nccl_comm(ncclUniqueId &tensor_para_nccl_uid, ncclUniqueId &layer_para_nccl_uid,
      |                                                           ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:301:3: error: 'ncclComm_t' does not name a type
  301 |   ncclComm_t tensor_para_nccl_comm, layer_para_nccl_comm;
      |   ^~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/gpt.h:25,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: In constructor 'torch_ext::GPT<T>::GPT(int, int, int, int, int, int, int, float, float, int, int, int, int, bool, int, float, std::vector<std::vector<at::Tensor> >, std::vector<at::Tensor>)':
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:29: error: 'MPI_COMM_WORLD' was not declared in this scope
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |                             ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:15: error: there are no arguments to 'MPI_Comm_rank' that depend on a template parameter, so a declaration of 'MPI_Comm_rank' must be available [-fpermissive]
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |               ^~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:15: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |               ^~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:87:5: error: 'ncclUniqueId' was not declared in this scope
   87 |     ncclUniqueId tensor_para_nccl_uid;
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:88:17: error: expected ';' before 'layer_para_nccl_uid'
   88 |     ncclUniqueId layer_para_nccl_uid;
      |                 ^~~~~~~~~~~~~~~~~~~~
      |                 ;
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:89:20: error: 'tensor_para_nccl_uid' was not declared in this scope; did you mean 'tensor_para_size_'?
   89 |     init_nccl_comm(tensor_para_nccl_uid, layer_para_nccl_uid, rank, tensor_para_size, layer_para_size, tensor_para_rank, layer_para_rank);
      |                    ^~~~~~~~~~~~~~~~~~~~
      |                    tensor_para_size_
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:89:42: error: 'layer_para_nccl_uid' was not declared in this scope; did you mean 'layer_para_size_'?
   89 |     init_nccl_comm(tensor_para_nccl_uid, layer_para_nccl_uid, rank, tensor_para_size, layer_para_size, tensor_para_rank, layer_para_rank);
      |                                          ^~~~~~~~~~~~~~~~~~~
      |                                          layer_para_size_
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:99:27: error: 'struct TensorParallelParam' has no member named 'nccl_comm'
   99 |     tensor_parallel_param.nccl_comm = tensor_para_nccl_comm;
      |                           ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:99:39: error: 'tensor_para_nccl_comm' was not declared in this scope
   99 |     tensor_parallel_param.nccl_comm = tensor_para_nccl_comm;
      |                                       ^~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:105:26: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  105 |     layer_parallel_param.nccl_comm = layer_para_nccl_comm;
      |                          ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:105:38: error: 'layer_para_nccl_comm' was not declared in this scope
  105 |     layer_parallel_param.nccl_comm = layer_para_nccl_comm;
      |                                      ^~~~~~~~~~~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/gpt.h:25,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: In member function 'void torch_ext::GPT<T>::init_nccl_comm(int&, int&, int, int, int, int, int)':
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:185:7: note: in expansion of macro 'NCCLCHECK'
  185 |       NCCLCHECK( ncclGetUniqueId(&tensor_para_nccl_uid));
      |       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: error: 'ncclSuccess' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: note: in definition of macro 'NCCLCHECK'
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: error: there are no arguments to 'ncclGetErrorString' that depend on a template parameter, so a declaration of 'ncclGetErrorString' must be available [-fpermissive]
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: note: in definition of macro 'NCCLCHECK'
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:81: error: 'MPI_BYTE' was not declared in this scope
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                                                                                ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:104: error: 'MPI_COMM_WORLD' was not declared in this scope
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:19: error: there are no arguments to 'MPI_Send' that depend on a template parameter, so a declaration of 'MPI_Send' must be available [-fpermissive]
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                   ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:194:7: error: 'MPI_Status' was not declared in this scope; did you mean 'PyStatus'?
  194 |       MPI_Status status;
      |       ^~~~~~~~~~
      |       PyStatus
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/gpt.h:25,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:79: error: 'MPI_BYTE' was not declared in this scope
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                               ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:117: error: 'MPI_COMM_WORLD' was not declared in this scope
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:134: error: 'status' was not declared in this scope; did you mean 'statx'?
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:17: error: there are no arguments to 'MPI_Recv' that depend on a template parameter, so a declaration of 'MPI_Recv' must be available [-fpermissive]
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                 ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:204:7: note: in expansion of macro 'NCCLCHECK'
  204 |       NCCLCHECK( ncclGetUniqueId(&layer_para_nccl_uid));
      |       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: error: 'ncclSuccess' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: note: in definition of macro 'NCCLCHECK'
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: error: there are no arguments to 'ncclGetErrorString' that depend on a template parameter, so a declaration of 'ncclGetErrorString' must be available [-fpermissive]
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: note: in definition of macro 'NCCLCHECK'
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:208:79: error: 'MPI_BYTE' was not declared in this scope
  208 |         MPICHECK( MPI_Send(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank + i * tensor_para_size, 0, MPI_COMM_WORLD));
      |                                                                               ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:208:121: error: 'MPI_COMM_WORLD' was not declared in this scope
  208 |         MPICHECK( MPI_Send(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank + i * tensor_para_size, 0, MPI_COMM_WORLD));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:208:19: error: there are no arguments to 'MPI_Send' that depend on a template parameter, so a declaration of 'MPI_Send' must be available [-fpermissive]
  208 |         MPICHECK( MPI_Send(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank + i * tensor_para_size, 0, MPI_COMM_WORLD));
      |                   ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:213:7: error: 'MPI_Status' was not declared in this scope; did you mean 'PyStatus'?
  213 |       MPI_Status status;
      |       ^~~~~~~~~~
      |       PyStatus
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/gpt.h:25,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:215:77: error: 'MPI_BYTE' was not declared in this scope
  215 |       MPICHECK( MPI_Recv(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank % tensor_para_size, 0, MPI_COMM_WORLD, &status));
      |                                                                             ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:215:115: error: 'MPI_COMM_WORLD' was not declared in this scope
  215 |       MPICHECK( MPI_Recv(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank % tensor_para_size, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:215:132: error: 'status' was not declared in this scope; did you mean 'statx'?
  215 |       MPICHECK( MPI_Recv(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank % tensor_para_size, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:215:17: error: there are no arguments to 'MPI_Recv' that depend on a template parameter, so a declaration of 'MPI_Recv' must be available [-fpermissive]
  215 |       MPICHECK( MPI_Recv(&layer_para_nccl_uid, sizeof(layer_para_nccl_uid), MPI_BYTE, rank % tensor_para_size, 0, MPI_COMM_WORLD, &status));
      |                 ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:218:5: note: in expansion of macro 'NCCLCHECK'
  218 |     NCCLCHECK( ncclCommInitRank(&tensor_para_nccl_comm, tensor_para_size, tensor_para_nccl_uid, tensor_para_rank));
      |     ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: error: 'ncclSuccess' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: note: in definition of macro 'NCCLCHECK'
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: error: there are no arguments to 'ncclGetErrorString' that depend on a template parameter, so a declaration of 'ncclGetErrorString' must be available [-fpermissive]
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: note: in definition of macro 'NCCLCHECK'
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:219:5: note: in expansion of macro 'NCCLCHECK'
  219 |     NCCLCHECK( ncclCommInitRank(&layer_para_nccl_comm, layer_para_size, layer_para_nccl_uid, layer_para_rank));
      |     ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: error: 'ncclSuccess' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: note: in definition of macro 'NCCLCHECK'
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: error: there are no arguments to 'ncclGetErrorString' that depend on a template parameter, so a declaration of 'ncclGetErrorString' must be available [-fpermissive]
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: note: in definition of macro 'NCCLCHECK'
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.cc:17:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: In destructor 'torch_ext::GPT<T>::~GPT()':
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:223:21: error: 'tensor_para_nccl_comm' was not declared in this scope
  223 |     ncclCommDestroy(tensor_para_nccl_comm);
      |                     ^~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:223:5: error: there are no arguments to 'ncclCommDestroy' that depend on a template parameter, so a declaration of 'ncclCommDestroy' must be available [-fpermissive]
  223 |     ncclCommDestroy(tensor_para_nccl_comm);
      |     ^~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:224:21: error: 'layer_para_nccl_comm' was not declared in this scope
  224 |     ncclCommDestroy(layer_para_nccl_comm);
      |                     ^~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:224:5: error: there are no arguments to 'ncclCommDestroy' that depend on a template parameter, so a declaration of 'ncclCommDestroy' must be available [-fpermissive]
  224 |     ncclCommDestroy(layer_para_nccl_comm);
      |     ^~~~~~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/gpt.h:30,
                 from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:23:
/workspace/FasterTransformer/fastertransformer/gpt.h: In member function 'void fastertransformer::DecodingGpt<OpType_>::forward_context(const fastertransformer::DecoderInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>*, fastertransformer::DecodingInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>)':
/workspace/FasterTransformer/fastertransformer/gpt.h:370:55: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  370 |                                     l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:414:55: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  414 |                                     l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h: In member function 'void fastertransformer::DecodingGpt<OpType_>::forward(const fastertransformer::DecoderInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>*, fastertransformer::DecodingInitParam<typename fastertransformer::DecoderTransformerTraits<OpType_>::DataType>)':
/workspace/FasterTransformer/fastertransformer/gpt.h:494:87: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  494 |                                   l_parallel_param_.world_size - 1, l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                                                ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:561:57: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  561 |                                       l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                         ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:616:57: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  616 |                                       l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                         ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:840:124: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  840 |                     nccl_send(decoding_params.output_ids + step * m + ite * local_batch, local_batch, 0, l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                                                                ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/gpt.h:869:45: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  869 |                           l_parallel_param_.nccl_comm, decoding_params.stream);
      |                                             ^~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:23:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: At global scope:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:172:23: error: 'ncclUniqueId' has not been declared
  172 |   void init_nccl_comm(ncclUniqueId &tensor_para_nccl_uid, ncclUniqueId &layer_para_nccl_uid,
      |                       ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:172:59: error: 'ncclUniqueId' has not been declared
  172 |   void init_nccl_comm(ncclUniqueId &tensor_para_nccl_uid, ncclUniqueId &layer_para_nccl_uid,
      |                                                           ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:301:3: error: 'ncclComm_t' does not name a type
  301 |   ncclComm_t tensor_para_nccl_comm, layer_para_nccl_comm;
      |   ^~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/cuda/cuda_kernels.h:20,
                 from /workspace/FasterTransformer/fastertransformer/bert_encoder_transformer.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/encoder.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:20:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: In constructor 'torch_ext::GPT<T>::GPT(int, int, int, int, int, int, int, float, float, int, int, int, int, bool, int, float, std::vector<std::vector<at::Tensor> >, std::vector<at::Tensor>)':
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:29: error: 'MPI_COMM_WORLD' was not declared in this scope
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |                             ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:15: error: there are no arguments to 'MPI_Comm_rank' that depend on a template parameter, so a declaration of 'MPI_Comm_rank' must be available [-fpermissive]
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |               ^~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:83:15: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
   83 |     MPICHECK( MPI_Comm_rank(MPI_COMM_WORLD, &rank));
      |               ^~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:23:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:87:5: error: 'ncclUniqueId' was not declared in this scope
   87 |     ncclUniqueId tensor_para_nccl_uid;
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:88:17: error: expected ';' before 'layer_para_nccl_uid'
   88 |     ncclUniqueId layer_para_nccl_uid;
      |                 ^~~~~~~~~~~~~~~~~~~~
      |                 ;
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:89:20: error: 'tensor_para_nccl_uid' was not declared in this scope; did you mean 'tensor_para_size_'?
   89 |     init_nccl_comm(tensor_para_nccl_uid, layer_para_nccl_uid, rank, tensor_para_size, layer_para_size, tensor_para_rank, layer_para_rank);
      |                    ^~~~~~~~~~~~~~~~~~~~
      |                    tensor_para_size_
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:89:42: error: 'layer_para_nccl_uid' was not declared in this scope; did you mean 'layer_para_size_'?
   89 |     init_nccl_comm(tensor_para_nccl_uid, layer_para_nccl_uid, rank, tensor_para_size, layer_para_size, tensor_para_rank, layer_para_rank);
      |                                          ^~~~~~~~~~~~~~~~~~~
      |                                          layer_para_size_
In file included from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:23:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:99:27: error: 'struct TensorParallelParam' has no member named 'nccl_comm'
   99 |     tensor_parallel_param.nccl_comm = tensor_para_nccl_comm;
      |                           ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:99:39: error: 'tensor_para_nccl_comm' was not declared in this scope
   99 |     tensor_parallel_param.nccl_comm = tensor_para_nccl_comm;
      |                                       ^~~~~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:105:26: error: 'struct LayerParallelParam' has no member named 'nccl_comm'
  105 |     layer_parallel_param.nccl_comm = layer_para_nccl_comm;
      |                          ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:105:38: error: 'layer_para_nccl_comm' was not declared in this scope
  105 |     layer_parallel_param.nccl_comm = layer_para_nccl_comm;
      |                                      ^~~~~~~~~~~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/cuda/cuda_kernels.h:20,
                 from /workspace/FasterTransformer/fastertransformer/bert_encoder_transformer.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/encoder.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:20:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h: In member function 'void torch_ext::GPT<T>::init_nccl_comm(int&, int&, int, int, int, int, int)':
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:185:7: note: in expansion of macro 'NCCLCHECK'
  185 |       NCCLCHECK( ncclGetUniqueId(&tensor_para_nccl_uid));
      |       ^~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: error: 'ncclSuccess' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:13: note: in definition of macro 'NCCLCHECK'
   81 |     if (r!= ncclSuccess) {                                  \
      |             ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: error: there are no arguments to 'ncclGetErrorString' that depend on a template parameter, so a declaration of 'ncclGetErrorString' must be available [-fpermissive]
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:83:35: note: in definition of macro 'NCCLCHECK'
   83 |                 __FILE__,__LINE__,ncclGetErrorString(r));   \
      |                                   ^~~~~~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:81: error: 'MPI_BYTE' was not declared in this scope
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                                                                                ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:104: error: 'MPI_COMM_WORLD' was not declared in this scope
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:189:19: error: there are no arguments to 'MPI_Send' that depend on a template parameter, so a declaration of 'MPI_Send' must be available [-fpermissive]
  189 |         MPICHECK( MPI_Send(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank + i, 0, MPI_COMM_WORLD));
      |                   ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
In file included from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:23:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:194:7: error: 'MPI_Status' was not declared in this scope; did you mean 'PyStatus'?
  194 |       MPI_Status status;
      |       ^~~~~~~~~~
      |       PyStatus
In file included from /workspace/FasterTransformer/fastertransformer/utils/arguments.h:24,
                 from /workspace/FasterTransformer/fastertransformer/cuda/cuda_kernels.h:20,
                 from /workspace/FasterTransformer/fastertransformer/bert_encoder_transformer.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/encoder.h:26,
                 from /workspace/FasterTransformer/fastertransformer/th_op/ft_op.cc:20:
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:79: error: 'MPI_BYTE' was not declared in this scope
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                               ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:117: error: 'MPI_COMM_WORLD' was not declared in this scope
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:134: error: 'status' was not declared in this scope; did you mean 'statx'?
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                                                                                ^~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:196:17: error: there are no arguments to 'MPI_Recv' that depend on a template parameter, so a declaration of 'MPI_Recv' must be available [-fpermissive]
  196 |       MPICHECK( MPI_Recv(&tensor_para_nccl_uid, sizeof(tensor_para_nccl_uid), MPI_BYTE, rank - tensor_para_rank, 0, MPI_COMM_WORLD, &status));
      |                 ^~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:62:13: note: in definition of macro 'MPICHECK'
   62 |     int e = cmd;                                            \
      |             ^~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: error: 'MPI_SUCCESS' was not declared in this scope; did you mean 'EXIT_SUCCESS'?
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:63:14: note: in definition of macro 'MPICHECK'
   63 |     if( e != MPI_SUCCESS ) {                                \
      |              ^~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: error: 'ncclResult_t' was not declared in this scope
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:80:5: note: in definition of macro 'NCCLCHECK'
   80 |     ncclResult_t r = cmd;                                   \
      |     ^~~~~~~~~~~~
/workspace/FasterTransformer/fastertransformer/utils/nccl_utils.h:81:9: error: 'r' was not declared in this scope
   81 |     if (r!= ncclSuccess) {                                  \
      |         ^
/workspace/FasterTransformer/fastertransformer/th_op/gpt.h:204:7: note: in expansion of macro 'NCCLCHECK'