May i  use  triton-inference-server to run T5 model

I can see the T5 triton backed related file in https://github.com/NVIDIA/FasterTransformer/tree/dev/v5.0_beta_2021.11_tag/src/fastertransformer/triton_backend/t5
But i also find the https://github.com/triton-inference-server/fastertransformer_backend just support gpt
I wonder whether T5 is supported by triton inference server. Is there a doc or example to tell us how to use it?

FT triton backend already supports T5, please refer https://github.com/triton-inference-server/fastertransformer_backend/tree/dev/v1.1_beta#prepare-triton-t5-model-store-in-the-docker.

[I do as README， but i met some errors as follows:
Firstly, i start the server(the model path cannot be setted like --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/ or --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1 , because the program will failed with -- error: creating server: Internal - failed to load all models) as follows:
mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver --model-repository=/workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1/2-gpu/

Secondly, i start the client
sh /workspace/fastertransformer_backend/tools/run_client.sh
then i get this in logs:

](https://github.com/triton-inference-server/fastertransformer_backend/issues/15#issuecomment-1081617473)

Please set the model-repository=/workspace/build/fastertransformer_backend/all_models/t5

After i mv the model_checkpoint_path to 2-gpu

then i run the command:mpirun -n 1 --allow-run-as-root /opt/tritonserver/bin/tritonserver --model-repository
=/workspace/build/fastertransformer_backend/all_models/t5
i got this error , it sames the file name is wrong:

the dir struture is:

https://github.com/NVIDIA/FasterTransformer/blob/dev/v5.0_beta/src/fastertransformer/triton_backend/t5/T5TritonModel.cc#L116
Please read the guide https://github.com/triton-inference-server/fastertransformer_backend/tree/dev/v1.1_beta#prepare-triton-t5-model-store-in-the-docker
I guess the problem is because you should set model_checkpoint_path to /workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1/2-gpu/, but not 2_gpu/

I se t the model_checkpoint_path  to 2-gpu

then i also get the error.

you should set model_checkpoint_path to /workspace/build/fastertransformer_backend/all_models/t5/fastertransformer/1/2-gpu/, but not 2_gpu/

Thanx, it worked! but another question the cuda version must be 11.4 if i want to run t5 in triton ?

You can change the based triton server docker version in Docker file. The default version is 21.08, which is installed cuda 11.4.

i set the CONTAINER_VERSION=20.03, but i met not found error.
then, i set the CONTAINER_VERSION=20.10 , i also met this error.

but the branch   20.10  is available.

it sames the github access  errors in cmake file

i change the https to http  in fastertransformer_backend/CMakeLists.txt, then i also get this error:

I can build the Dockerfile successfully with default 21.08 CONTAINER_VERSION, then i met the cuda mismatch error.
So i  change the the CONTAINER_VERSION to 20.10  as you said and build the docker image with docker file , i met the 'unable to access github' error.

Have you tried other version? 21.07? 21.06?
You can also try to update your driver.
Since the error of 20.10 is on github access, I have no idea about it.

I just try 21.07, it's the same error.

And i can find the 20.10 in the https://github.com/triton-inference-server/server/branches/all?page=5
Maybe it's the network error.  Should i do some config about github in the Dockerfile?

Sorry, I have no idea about the error because it does not happen on FT side.

i add  the follows in the dockerfile before  #backend build
RUN sed -i "s/archive.ubuntu./mirrors.aliyun./g" /etc/apt/sources.list
RUN sed -i "s/deb.debian.org/mirrors.aliyun.com/g" /etc/apt/sources.list
RUN sed -i "s/security.debian.org/mirrors.aliyun.com/debian-security/g" /etc/apt/sources.list
RUN apt-get update
Then in the build process,i met this

-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "11.1", minimum required is "10.1")
-- Found MPI_C: /usr/local/mpi/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /usr/local/mpi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found NCCL: /usr/include
-- Determining NCCL version from /usr/include/nccl.h...
-- Looking for NCCL_VERSION_CODE
-- Looking for NCCL_VERSION_CODE - not found
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.7.8)
-- Found MPI (include: , library: /usr/local/mpi/lib/libmpi.so)
-- Add DCUDA11_MODE
-- RapidJSON found. Headers: /usr/include
-- RapidJSON found. Headers: /usr/include
-- Found CUDAToolkit: /usr/local/cuda/include (found version "11.1.74")
-- The CUDA compiler identification is NVIDIA 11.1.74
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Add DBUILD_GPT, requires MPI and NCCL
CMake Warning (dev) at build/_deps/repo-ft-src/CMakeLists.txt:32 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
command to set the policy and suppress this warning.
CMake variable NCCL_ROOT is set to:
/usr/local/cuda

For compatibility, CMake is ignoring the variable.
This warning is for project developers.  Use -Wno-dev to suppress it.
-- Assign GPU architecture (sm=70,75,80,86)
-- Determining NCCL version from /usr/include/nccl.h...
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.7.8)
-- Add DCUDA11_MODE
-- Found CUDAToolkit: /usr/local/cuda/include
-- Add DCUDA11_MODE
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/build/fastertransformer_backend/build
Scanning dependencies of target nvtx_utils
Scanning dependencies of target xlnet_preprocess_kernels
Scanning dependencies of target gpt_example_utils
Scanning dependencies of target activation_kernels
Scanning dependencies of target bert_preprocess_kernels
Scanning dependencies of target nccl_utils
Scanning dependencies of target cnpy
Scanning dependencies of target add_bias_transpose_kernels
Scanning dependencies of target triton-backend-utils
Scanning dependencies of target decoder_masked_multihead_attention
Scanning dependencies of target transpose_int8_kernels
Scanning dependencies of target gpt_kernels
Scanning dependencies of target layout_transformer_int8_kernels
Scanning dependencies of target quantize_weight
[  1%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cc.o
Scanning dependencies of target encoder_igemm_func
Scanning dependencies of target triton-common-table-printer
Scanning dependencies of target add_residual_kernels
Scanning dependencies of target xlnet_attention_kernels
Scanning dependencies of target decoding_kernels
Scanning dependencies of target activation_int8_kernels
[  1%] Building CXX object _deps/repo-ft-build/examples/cpp/multi_gpu_gpt/CMakeFiles/gpt_example_utils.dir/gpt_example_utils.cc.o
Scanning dependencies of target memory_utils
[  1%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/bert_preprocess_kernels.dir/bert_preprocess_kernels.cu.o
[  1%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/xlnet_preprocess_kernels.dir/xlnet_preprocess_kernels.cu.o
Scanning dependencies of target cublasAlgoMap
[  2%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_model_instance.cc.o
[  2%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_model.cc.o
[  2%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/activation_kernels.dir/activation_kernels.cu.o
[  2%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/decoder_masked_multihead_attention.dir/decoder_masked_multihead_attention.cu.o
Scanning dependencies of target gemm_func
Scanning dependencies of target quantization_int8_kernels
Scanning dependencies of target calibrate_quantize_weight_kernels
Scanning dependencies of target softmax_int8_kernels
[  2%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/transpose_int8_kernels.dir/transpose_int8_kernels.cu.o
Scanning dependencies of target layernorm_kernels
Scanning dependencies of target matrix_transpose_kernels
[  2%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_memory.cc.o
Scanning dependencies of target beam_search_topk_kernels
Scanning dependencies of target matrix_vector_multiplication
Scanning dependencies of target unfused_attention_int8_kernels
Scanning dependencies of target unfused_attention_kernels
Scanning dependencies of target longformer_kernels
[  2%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_output_responder.cc.o
Scanning dependencies of target sampling_penalty_kernels
Scanning dependencies of target layernorm_int8_kernels
Scanning dependencies of target beam_search_penalty_kernels
[  3%] Building CXX object _deps/repo-common-build/CMakeFiles/triton-common-table-printer.dir/src/table_printer.cc.o
Scanning dependencies of target sampling_topk_kernels
[  3%] Building CXX object _deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/cnpy.cpp.o
[  3%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_input_collector.cc.o
Scanning dependencies of target online_softmax_beamsearch_kernels
Scanning dependencies of target sampling_topp_kernels
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/memory_utils.dir/memory_utils.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/xlnet_attention_kernels.dir/xlnet_attention_kernels.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/quantize_weight.dir/quantize_weight.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/add_bias_transpose_kernels.dir/add_bias_transpose_kernels.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layout_transformer_int8_kernels.dir/layout_transformer_int8_kernels.cu.o
[  3%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_common.cc.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/gpt_kernels.dir/gpt_kernels.cu.o
[  3%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/encoder_igemm_func.dir/encoder_igemm_func.cc.o
[  3%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/gemm_func.dir/gemm_func.cc.o
[  3%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/nccl_utils.dir/nccl_utils.cc.o
[  3%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/cublasAlgoMap.dir/cublasAlgoMap.cc.o
[  4%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_penalty_kernels.dir/sampling_penalty_kernels.cu.o
[  5%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/matrix_vector_multiplication.dir/matrix_vector_multiplication.cu.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/decoding_kernels.dir/decoding_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/calibrate_quantize_weight_kernels.dir/calibrate_quantize_weight_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_topk_kernels.dir/sampling_topk_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/add_residual_kernels.dir/add_residual_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/softmax_int8_kernels.dir/softmax_int8_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/longformer_kernels.dir/longformer_kernels.cu.o
[  8%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/beam_search_penalty_kernels.dir/beam_search_penalty_kernels.cu.o
[  8%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layernorm_int8_kernels.dir/layernorm_int8_kernels.cu.o
[  9%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/activation_int8_kernels.dir/activation_int8_kernels.cu.o
[  9%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/online_softmax_beamsearch_kernels.dir/online_softmax_beamsearch_kernels.cu.o
[ 10%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/beam_search_topk_kernels.dir/beam_search_topk_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layernorm_kernels.dir/layernorm_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/quantization_int8_kernels.dir/quantization_int8_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/unfused_attention_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_topp_kernels.dir/sampling_topp_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/matrix_transpose_kernels.dir/matrix_transpose_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/unfused_attention_int8_kernels.dir/unfused_attention_int8_kernels.cu.o
Scanning dependencies of target trt_fused_multi_head_attention
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/cudaDriverWrapper.cpp.o
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm75.cpp.o
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm80.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm86.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm75.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm70.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm80.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm75.cpp.o
[ 14%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp.o
[ 14%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp.o
[ 14%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm86.cpp.o
[ 14%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm70.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm80.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm80.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm86.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp.o
[ 16%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm80.cpp.o
[ 18%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm86.cpp.o
[ 18%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm70.cpp.o
[ 19%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm80.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm86.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm86.cpp.o
[ 21%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm86.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm70.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm86.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm80.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp.o
[ 24%] Building CUDA object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/qkvToContext.cu.o
[ 24%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp.o
[ 24%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm86.cpp.o
[ 24%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp.o
[ 24%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm70.cpp.o
[ 25%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp.o
[ 25%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp.o
In file included from /workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/examples/cpp/xlnet/cnpy.cpp:6:0:
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/examples/cpp/xlnet/cnpy.h:19:10: fatal error: zlib.h: No such file or directory
#include <zlib.h>
^~~~~~~~
compilation terminated.
_deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/build.make:81: recipe for target '_deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/cnpy.cpp.o' failed
make[2]: *** [_deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/cnpy.cpp.o] Error 1
CMakeFiles/Makefile2:4928: recipe for target '_deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/all' failed
make[1]: *** [_deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 25%] Linking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o
[ 25%] Linking CUDA device code CMakeFiles/nccl_utils.dir/cmake_device_link.o
[ 25%] Linking CXX static library ../../../../../lib/libnvtx_utils.a
[ 26%] Linking CXX static library ../../../../../lib/libgpt_example_utils.a
[ 26%] Built target nvtx_utils
[ 26%] Built target gpt_example_utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
[ 26%] Linking CXX static library ../../../../../lib/libnccl_utils.a
[ 26%] Built target nccl_utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 26%] Linking CUDA device code CMakeFiles/gemm_func.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 27%] Linking CUDA device code CMakeFiles/encoder_igemm_func.dir/cmake_device_link.o
[ 28%] Linking CXX static library ../../../../../../lib/libgemm_func.a
[ 28%] Linking CUDA device code CMakeFiles/cublasAlgoMap.dir/cmake_device_link.o
[ 28%] Linking CXX static library libtritontableprinter.a
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
[ 28%] Linking CXX static library ../../../../../../lib/libencoder_igemm_func.a
[ 28%] Built target gemm_func
[ 28%] Built target encoder_igemm_func
[ 28%] Built target triton-common-table-printer
[ 28%] Linking CXX static library ../../../../../lib/libcublasAlgoMap.a
[ 28%] Built target cublasAlgoMap
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 28%] Linking CXX static library libtritonbackendutils.a
[ 28%] Built target triton-backend-utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
[ 28%] Linking CUDA device code CMakeFiles/quantize_weight.dir/cmake_device_link.o
[ 28%] Linking CUDA device code CMakeFiles/transpose_int8_kernels.dir/cmake_device_link.o
[ 28%] Linking CUDA device code CMakeFiles/quantization_int8_kernels.dir/cmake_device_link.o
[ 29%] Linking CUDA device code CMakeFiles/activation_kernels.dir/cmake_device_link.o
[ 29%] Linking CUDA device code CMakeFiles/gpt_kernels.dir/cmake_device_link.o
[ 29%] Linking CUDA device code CMakeFiles/add_residual_kernels.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 29%] Linking CUDA device code CMakeFiles/matrix_transpose_kernels.dir/cmake_device_link.o
[ 29%] Linking CUDA static library ../../../../../lib/libquantize_weight.a
[ 29%] Linking CUDA device code CMakeFiles/activation_int8_kernels.dir/cmake_device_link.o
[ 29%] Linking CUDA device code CMakeFiles/add_bias_transpose_kernels.dir/cmake_device_link.o
[ 30%] Linking CUDA static library ../../../../../lib/libquantization_int8_kernels.a
[ 31%] Linking CUDA static library ../../../../../lib/libtranspose_int8_kernels.a
[ 31%] Built target quantize_weight
[ 32%] Linking CUDA static library ../../../../../lib/libgpt_kernels.a
[ 32%] Linking CUDA static library ../../../../../lib/libactivation_kernels.a
[ 32%] Built target quantization_int8_kernels
[ 33%] Linking CUDA device code CMakeFiles/layout_transformer_int8_kernels.dir/cmake_device_link.o
[ 33%] Linking CUDA static library ../../../../../lib/libadd_residual_kernels.a
[ 33%] Built target transpose_int8_kernels
[ 33%] Linking CUDA static library ../../../../../lib/libmatrix_transpose_kernels.a
[ 33%] Built target gpt_kernels
[ 33%] Built target activation_kernels
[ 33%] Built target add_residual_kernels
[ 33%] Built target matrix_transpose_kernels
[ 33%] Linking CUDA static library ../../../../../lib/libactivation_int8_kernels.a
[ 34%] Linking CUDA static library ../../../../../lib/libadd_bias_transpose_kernels.a
[ 34%] Built target activation_int8_kernels
[ 34%] Built target add_bias_transpose_kernels
[ 34%] Linking CUDA static library ../../../../../lib/liblayout_transformer_int8_kernels.a
[ 34%] Built target layout_transformer_int8_kernels
[ 34%] Linking CUDA device code CMakeFiles/decoding_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA static library ../../../../../lib/libdecoding_kernels.a
[ 34%] Built target decoding_kernels
[ 34%] Linking CUDA device code CMakeFiles/sampling_penalty_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA device code CMakeFiles/xlnet_attention_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA static library ../../../../../lib/libsampling_penalty_kernels.a
[ 34%] Linking CUDA static library ../../../../../lib/libxlnet_attention_kernels.a
[ 34%] Built target sampling_penalty_kernels
[ 34%] Built target xlnet_attention_kernels
[ 34%] Linking CUDA device code CMakeFiles/calibrate_quantize_weight_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA device code CMakeFiles/xlnet_preprocess_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA device code CMakeFiles/bert_preprocess_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA static library ../../../../../lib/libcalibrate_quantize_weight_kernels.a
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of 'void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = float; int8_t = signed char; cudaStream_t = CUstream_st*]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:230:205:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: conversion lacks type at end of format [-Wformat=]
printf("[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 != 0).\n",
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: unknown conversion type character ' ' in format [-Wformat=]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:204:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
while (block.x * 4 > k)
~~~~~~~~~~~~~~~~~~^~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of 'void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = __half; int8_t = signed char; cudaStream_t = CUstream_st*]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:239:203:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: conversion lacks type at end of format [-Wformat=]
printf("[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 != 0).\n",
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: unknown conversion type character ' ' in format [-Wformat=]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:204:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
while (block.x * 4 > k)
~~~~~~~~~~~~~~~~~~^~
[ 35%] Linking CUDA static library ../../../../../lib/libxlnet_preprocess_kernels.a
[ 35%] Built target calibrate_quantize_weight_kernels
[ 35%] Built target xlnet_preprocess_kernels
[ 35%] Linking CUDA static library ../../../../../lib/libbert_preprocess_kernels.a
[ 35%] Linking CUDA device code CMakeFiles/matrix_vector_multiplication.dir/cmake_device_link.o
[ 35%] Built target bert_preprocess_kernels
[ 35%] Linking CUDA static library ../../../../../lib/libmatrix_vector_multiplication.a
[ 35%] Built target matrix_vector_multiplication
[ 35%] Linking CUDA device code CMakeFiles/layernorm_int8_kernels.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 35%] Linking CUDA static library ../../../../../lib/liblayernorm_int8_kernels.a
[ 35%] Built target layernorm_int8_kernels
[ 35%] Linking CUDA device code CMakeFiles/beam_search_penalty_kernels.dir/cmake_device_link.o
[ 35%] Linking CUDA static library ../../../../../lib/libbeam_search_penalty_kernels.a
[ 35%] Built target beam_search_penalty_kernels
[ 35%] Linking CUDA device code CMakeFiles/softmax_int8_kernels.dir/cmake_device_link.o
[ 35%] Linking CUDA device code CMakeFiles/layernorm_kernels.dir/cmake_device_link.o
[ 35%] Linking CUDA static library ../../../../../lib/liblayernorm_kernels.a
[ 35%] Linking CUDA static library ../../../../../lib/libsoftmax_int8_kernels.a
[ 35%] Built target layernorm_kernels
[ 35%] Built target softmax_int8_kernels
[ 36%] Linking CUDA device code CMakeFiles/unfused_attention_int8_kernels.dir/cmake_device_link.o
[ 36%] Linking CUDA static library ../../../../../lib/libunfused_attention_int8_kernels.a
[ 36%] Built target unfused_attention_int8_kernels
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In function 'void fastertransformer::set_alpha(uint32_t&, float, fastertransformer::Data_type)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:32:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
alpha = reinterpret_cast<const uint32_t&>(h2);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:36:50: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
alpha = reinterpret_cast<const uint32_t&>(norm);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In member function 'void fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:327:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_bmm1 = reinterpret_cast<const uint32_t&>(scaleBmm1);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:328:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_bmm2 = reinterpret_cast<const uint32_t&>(scaleBmm2);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:329:77: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_softmax = reinterpret_cast<const uint32_t&>(scaleSoftmax);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In member function 'virtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:278:17: warning: 'warps_n' may be used uninitialized in this function [-Wmaybe-uninitialized]
size_t warps_m, warps_n, warps_k = 1;
^~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:278:8: warning: 'warps_m' may be used uninitialized in this function [-Wmaybe-uninitialized]
size_t warps_m, warps_n, warps_k = 1;
^~~~~~~
[ 36%] Linking CUDA device code CMakeFiles/trt_fused_multi_head_attention.dir/cmake_device_link.o
[ 36%] Linking CXX static library ../../../../lib/libtrt_fused_multi_head_attention.a
[ 36%] Built target trt_fused_multi_head_attention
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 37%] Linking CUDA device code CMakeFiles/sampling_topk_kernels.dir/cmake_device_link.o
[ 37%] Linking CUDA static library ../../../../../lib/libsampling_topk_kernels.a
[ 37%] Built target sampling_topk_kernels
[ 37%] Linking CUDA device code CMakeFiles/longformer_kernels.dir/cmake_device_link.o
[ 38%] Linking CUDA static library ../../../../../lib/liblongformer_kernels.a
[ 38%] Built target longformer_kernels
[ 38%] Linking CUDA device code CMakeFiles/unfused_attention_kernels.dir/cmake_device_link.o
[ 39%] Linking CUDA static library ../../../../../lib/libunfused_attention_kernels.a
[ 39%] Built target unfused_attention_kernels
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/logger.h: In instantiation of 'void fastertransformer::Logger::log(fastertransformer::Logger::Level, std::__cxx11::string, const Args& ...) [with Args = {}; std::__cxx11::string = std::__cxx11::basic_string]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu:189:43:   required from 'int fastertransformer::loadWeightFromBin(T*, std::vector, std::__cxx11::string) [with T = float; std::__cxx11::string = std::__cxx11::basic_string]'
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu:213:100:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/logger.h:30:8: warning: format not a string literal and no format arguments [-Wformat-security]
fprintf(out, fmt.c_str(), args...);
^    ~~~~~~~~~~~~~~~~~~~~~~
[ 40%] Linking CUDA device code CMakeFiles/memory_utils.dir/cmake_device_link.o
[ 40%] Linking CUDA static library ../../../../../lib/libmemory_utils.a
[ 40%] Built target memory_utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
[ 40%] Linking CUDA device code CMakeFiles/beam_search_topk_kernels.dir/cmake_device_link.o
[ 40%] Linking CUDA static library ../../../../../lib/libbeam_search_topk_kernels.a
[ 40%] Built target beam_search_topk_kernels
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of 'int fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<float, int, 256, 2>]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:517:104:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:497:56: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if ( smem_size + BLOCK_THREADS * sizeof(float) > context.sm_shared_size || // dynamic + static memory
~~~~~~~~~~~~~~~~~~~~~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of 'int fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<float, int, 256, 1>]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:519:104:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:497:56: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of 'int fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<__half, int, 256, 4>]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:523:105:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:497:56: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of 'int fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<__half, int, 256, 1>]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:525:105:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu:497:56: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
[ 40%] Linking CUDA device code CMakeFiles/sampling_topp_kernels.dir/cmake_device_link.o
[ 41%] Linking CUDA static library ../../../../../lib/libsampling_topp_kernels.a
[ 41%] Built target sampling_topp_kernels
[ 42%] Linking CUDA device code CMakeFiles/online_softmax_beamsearch_kernels.dir/cmake_device_link.o
[ 42%] Linking CUDA static library ../../../../../lib/libonline_softmax_beamsearch_kernels.a
[ 42%] Built target online_softmax_beamsearch_kernels
[ 42%] Linking CUDA device code CMakeFiles/decoder_masked_multihead_attention.dir/cmake_device_link.o
[ 43%] Linking CUDA static library ../../../../../lib/libdecoder_masked_multihead_attention.a
[ 43%] Built target decoder_masked_multihead_attention
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2
The command '/bin/sh -c mkdir build -p &&     cd build &&     cmake       -D CMAKE_EXPORT_COMPILE_COMMANDS=1       -D CMAKE_BUILD_TYPE=Release       -D CMAKE_INSTALL_PREFIX=/opt/tritonserver       -D TRITON_COMMON_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"       -D TRITON_CORE_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"       -D TRITON_BACKEND_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"       .. &&     make -j"$(grep -c ^processor /proc/cpuinfo)" install' returned a non-zero code: 2

Sending build context to Docker daemon  289.8kB
Step 1/29 : ARG TRITON_VERSION=21.08
Step 2/29 : ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:${TRITON_VERSION}-py3
Step 3/29 : FROM ${BASE_IMAGE} as server-builder
---> 550b36cc0549
Step 4/29 : RUN apt-get update &&     apt-get install -y --no-install-recommends     zip unzip wget build-essential autoconf autogen gdb     python3.8 python3-pip python3-dev   rapidjson-dev &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*
---> Using cache
---> a251e61c245c
Step 5/29 : WORKDIR /workspace/build/
---> Using cache
---> 516033620fcb
Step 6/29 : RUN CMAKE_VERSION=3.18 &&     CMAKE_BUILD=3.18.4 &&     wget -nv https://cmake.org/files/v${CMAKE_VERSION}/cmake-${CMAKE_BUILD}.tar.gz &&     tar -xf cmake-${CMAKE  _BUILD}.tar.gz &&     cd cmake-${CMAKE_BUILD} &&     ./bootstrap --parallel=$(grep -c ^processor /proc/cpuinfo) -- -DCMAKE_USE_OPENSSL=OFF &&     make -j"$(grep -c ^processor   /proc/cpuinfo)" install &&     cd /workspace/build/ &&     rm -rf /workspace/build/cmake-${CMAKE_BUILD}
---> Using cache
---> ea939c613685
Step 7/29 : WORKDIR /workspace/build/fastertransformer_backend
---> Using cache
---> 387d12385d6c
Step 8/29 : ADD cmake cmake
---> Using cache
---> 83a8b8ec6682
Step 9/29 : ADD src src
---> Using cache
---> 85f0bf967f25
Step 10/29 : ADD CMakeLists.txt CMakeLists.txt
---> Using cache
---> 45ee59cde4b1
Step 11/29 : ADD README.md README.md
---> Using cache
---> c96543324780
Step 12/29 : ADD LICENSE LICENSE
---> Using cache
---> b743c11c3bd6
Step 13/29 : ARG FORCE_BACKEND_REBUILD=0
---> Using cache
---> c7922ed1874c
Step 14/29 : RUN sed -i "s/archive.ubuntu./mirrors.aliyun./g" /etc/apt/sources.list &&     sed -i "s/deb.debian.org/mirrors.aliyun.com/g" /etc/apt/sources.list &&     sed -i "  s/security.debian.org/mirrors.aliyun.com/debian-security/g" /etc/apt/sources.list &&     apt-get update &&     mkdir build -p &&     cd build &&     cmake       -D CMAKE_EXPO  RT_COMPILE_COMMANDS=1       -D CMAKE_BUILD_TYPE=Release       -D CMAKE_INSTALL_PREFIX=/opt/tritonserver       -D TRITON_COMMON_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"         -D TRITON_CORE_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"       -D TRITON_BACKEND_REPO_TAG="r${NVIDIA_TRITON_SERVER_VERSION}"       .. &&     make -j"$(grep -c ^processor /  proc/cpuinfo)" install
---> Running in 0bc5284d558c
Get:1 http://mirrors.aliyun.com/ubuntu bionic InRelease [242 kB]
Get:2 http://mirrors.aliyun.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://mirrors.aliyun.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://mirrors.aliyun.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:7 http://mirrors.aliyun.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:8 http://mirrors.aliyun.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:9 http://mirrors.aliyun.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:10 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 Packages [3133 kB]
Get:11 http://mirrors.aliyun.com/ubuntu bionic-updates/restricted amd64 Packages [918 kB]
Get:12 http://mirrors.aliyun.com/ubuntu bionic-updates/universe amd64 Packages [2268 kB]
Get:13 http://mirrors.aliyun.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Get:14 http://mirrors.aliyun.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [21.1 kB]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1490 kB]
Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2694 kB]
Get:18 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [884 kB]
Fetched 24.8 MB in 21s (1201 kB/s)
Reading package lists...
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/include/python3.6m (found version "3.6.9") found components: Development Development.Module Development.Embed
CMake Warning at CMakeLists.txt:54 (find_package):
By not providing "FindFasterTransformer.cmake" in CMAKE_MODULE_PATH this
project has asked CMake to find a package configuration file provided by
"FasterTransformer", but CMake did not find one.
Could not find a package configuration file provided by "FasterTransformer"
with any of the following names:
FasterTransformerConfig.cmake
fastertransformer-config.cmake

Add the installation prefix of "FasterTransformer" to CMAKE_PREFIX_PATH or
set "FasterTransformer_DIR" to a directory containing one of the above
files.  If "FasterTransformer" provides a separate development package or
SDK, be sure it has been installed.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "11.1", minimum required is "10.1")
-- Found MPI_C: /usr/local/mpi/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /usr/local/mpi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found NCCL: /usr/include
-- Determining NCCL version from /usr/include/nccl.h...
-- Looking for NCCL_VERSION_CODE
-- Looking for NCCL_VERSION_CODE - not found
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.7.8)
-- Found MPI (include: , library: /usr/local/mpi/lib/libmpi.so)
-- Add DCUDA11_MODE
-- RapidJSON found. Headers: /usr/include
-- RapidJSON found. Headers: /usr/include
-- Found CUDAToolkit: /usr/local/cuda/include (found version "11.1.74")
-- The CUDA compiler identification is NVIDIA 11.1.74
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Add DBUILD_GPT, requires MPI and NCCL
CMake Warning (dev) at build/_deps/repo-ft-src/CMakeLists.txt:32 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
command to set the policy and suppress this warning.
CMake variable NCCL_ROOT is set to:
/usr/local/cuda

For compatibility, CMake is ignoring the variable.
This warning is for project developers.  Use -Wno-dev to suppress it.
-- Determining NCCL version from /usr/include/nccl.h...
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.7.8)
-- Add DCUDA11_MODE
-- Assign GPU architecture (sm=70,75,80,86)
-- Found CUDAToolkit: /usr/local/cuda/include
-- Add DCUDA11_MODE
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/build/fastertransformer_backend/build
Scanning dependencies of target triton-common-table-printer
Scanning dependencies of target nvtx_utils
Scanning dependencies of target add_bias_transpose_kernels
Scanning dependencies of target nccl_utils
Scanning dependencies of target activation_kernels
Scanning dependencies of target quantization_int8_kernels
Scanning dependencies of target cnpy
Scanning dependencies of target decoder_masked_multihead_attention
Scanning dependencies of target transpose_int8_kernels
Scanning dependencies of target bert_preprocess_kernels
Scanning dependencies of target add_residual_kernels
Scanning dependencies of target triton-backend-utils
Scanning dependencies of target layout_transformer_int8_kernels
Scanning dependencies of target activation_int8_kernels
Scanning dependencies of target decoding_kernels
Scanning dependencies of target gpt_kernels
Scanning dependencies of target memory_utils
Scanning dependencies of target sampling_penalty_kernels
Scanning dependencies of target gpt_example_utils
Scanning dependencies of target cublasAlgoMap
Scanning dependencies of target quantize_weight
Scanning dependencies of target gemm_func
Scanning dependencies of target xlnet_attention_kernels
Scanning dependencies of target encoder_igemm_func
Scanning dependencies of target xlnet_preprocess_kernels
[  1%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cc.o
Scanning dependencies of target matrix_transpose_kernels
Scanning dependencies of target layernorm_int8_kernels
Scanning dependencies of target longformer_kernels
Scanning dependencies of target softmax_int8_kernels
Scanning dependencies of target calibrate_quantize_weight_kernels
[  1%] Building CXX object _deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/cnpy.dir/cnpy.cpp.o
[  1%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/add_bias_transpose_kernels.dir/add_bias_transpose_kernels.cu.o
Scanning dependencies of target unfused_attention_int8_kernels
[  2%] Building CXX object _deps/repo-common-build/CMakeFiles/triton-common-table-printer.dir/src/table_printer.cc.o
[  2%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/activation_kernels.dir/activation_kernels.cu.o
Scanning dependencies of target layernorm_kernels
Scanning dependencies of target matrix_vector_multiplication
Scanning dependencies of target beam_search_penalty_kernels
Scanning dependencies of target beam_search_topk_kernels
Scanning dependencies of target unfused_attention_kernels
[  2%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_output_responder.cc.o
Scanning dependencies of target online_softmax_beamsearch_kernels
[  2%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/nccl_utils.dir/nccl_utils.cc.o
[  2%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/cublasAlgoMap.dir/cublasAlgoMap.cc.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/activation_int8_kernels.dir/activation_int8_kernels.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/quantization_int8_kernels.dir/quantization_int8_kernels.cu.o
Scanning dependencies of target sampling_topk_kernels
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/transpose_int8_kernels.dir/transpose_int8_kernels.cu.o
[  3%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/add_residual_kernels.dir/add_residual_kernels.cu.o
[  3%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_model.cc.o
[  5%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/decoding_kernels.dir/decoding_kernels.cu.o
[  5%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_penalty_kernels.dir/sampling_penalty_kernels.cu.o
[  5%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_input_collector.cc.o
[  5%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/quantize_weight.dir/quantize_weight.cu.o
[  5%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_common.cc.o
[  5%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/bert_preprocess_kernels.dir/bert_preprocess_kernels.cu.o
[  6%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_model_instance.cc.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/decoder_masked_multihead_attention.dir/decoder_masked_multihead_attention.cu.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/memory_utils.dir/memory_utils.cu.o
[  6%] Building CXX object _deps/repo-ft-build/examples/cpp/multi_gpu_gpt/CMakeFiles/gpt_example_utils.dir/gpt_example_utils.cc.o
[  6%] Building CXX object _deps/repo-backend-build/CMakeFiles/triton-backend-utils.dir/src/backend_memory.cc.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layout_transformer_int8_kernels.dir/layout_transformer_int8_kernels.cu.o
Scanning dependencies of target sampling_topp_kernels
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/gpt_kernels.dir/gpt_kernels.cu.o
[  6%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/gemm_func.dir/gemm_func.cc.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layernorm_int8_kernels.dir/layernorm_int8_kernels.cu.o
[  6%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/softmax_int8_kernels.dir/softmax_int8_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/calibrate_quantize_weight_kernels.dir/calibrate_quantize_weight_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/longformer_kernels.dir/longformer_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/xlnet_preprocess_kernels.dir/xlnet_preprocess_kernels.cu.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/matrix_transpose_kernels.dir/matrix_transpose_kernels.cu.o
Scanning dependencies of target trt_fused_multi_head_attention
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/xlnet_attention_kernels.dir/xlnet_attention_kernels.cu.o
[  7%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/encoder_igemm_func.dir/encoder_igemm_func.cc.o
[  7%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/unfused_attention_int8_kernels.dir/unfused_attention_int8_kernels.cu.o
[  8%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/beam_search_topk_kernels.dir/beam_search_topk_kernels.cu.o
[  9%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/layernorm_kernels.dir/layernorm_kernels.cu.o
[  9%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_topk_kernels.dir/sampling_topk_kernels.cu.o
[ 10%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/matrix_vector_multiplication.dir/matrix_vector_multiplication.cu.o
[ 10%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/unfused_attention_kernels.cu.o
[ 10%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/online_softmax_beamsearch_kernels.dir/online_softmax_beamsearch_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/beam_search_penalty_kernels.dir/beam_search_penalty_kernels.cu.o
[ 11%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/kernels/CMakeFiles/sampling_topp_kernels.dir/sampling_topp_kernels.cu.o
[ 11%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm80.cpp.o
[ 11%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm75.cpp.o
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm75.cpp.o
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm86.cpp.o
[ 12%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp.o
[ 13%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm80.cpp.o
[ 14%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm86.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm86.cpp.o
[ 15%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm75.cpp.o
[ 16%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp.o
[ 16%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm80.cpp.o
[ 16%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm80.cpp.o
[ 16%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm80.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm80.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm70.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm86.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm75.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm70.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm75.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm70.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm70.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm86.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp.o
[ 17%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp.o
[ 18%] Building CUDA object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/qkvToContext.cu.o
[ 19%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp.o
[ 19%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp.o
[ 19%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp.o
[ 19%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/cudaDriverWrapper.cpp.o
[ 20%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm75.cpp.o
[ 21%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm80.cpp.o
[ 21%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm86.cpp.o
[ 21%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp.o
[ 21%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm75.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm86.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm70.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp.o
[ 22%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp.o
[ 23%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm86.cpp.o
[ 24%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp.o
[ 25%] Building CXX object _deps/repo-ft-build/3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm86.cpp.o
[ 25%] Linking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o
[ 25%] Linking CUDA device code CMakeFiles/nccl_utils.dir/cmake_device_link.o
[ 25%] Linking CXX static library ../../../../../lib/libnvtx_utils.a
[ 25%] Built target nvtx_utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 25%] Linking CXX static library ../../../../../lib/libnccl_utils.a
[ 25%] Built target nccl_utils
[ 26%] Linking CXX static library ../../../../../lib/libgpt_example_utils.a
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
[ 26%] Built target gpt_example_utils
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 27%] Linking CUDA device code CMakeFiles/encoder_igemm_func.dir/cmake_device_link.o
[ 27%] Linking CUDA device code CMakeFiles/gemm_func.dir/cmake_device_link.o
[ 27%] Linking CUDA device code CMakeFiles/cublasAlgoMap.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
[ 27%] Linking CXX static library libtritontableprinter.a
[ 27%] Built target triton-common-table-printer
[ 27%] Linking CXX static library ../../../../../../lib/libencoder_igemm_func.a
[ 28%] Linking CXX static library ../../../../../../lib/libgemm_func.a
[ 28%] Linking CXX static library ../../../../../lib/libcublasAlgoMap.a
[ 28%] Built target encoder_igemm_func
[ 28%] Built target gemm_func
Scanning dependencies of target xlnet_gemm_func
[ 28%] Built target cublasAlgoMap
Scanning dependencies of target gpt_gemm_func
Scanning dependencies of target decoding_gemm_func
Scanning dependencies of target encoder_gemm_func
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/xlnet_gemm_func.dir/xlnet_gemm_func.cc.o
Scanning dependencies of target t5_gemm_func
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/gpt_gemm_func.dir/gpt_gemm_func.cc.o
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/encoder_gemm_func.dir/encoder_gemm_func.cc.o
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/decoding_gemm_func.dir/decoding_gemm_func.cc.o
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/gemm_test/CMakeFiles/t5_gemm_func.dir/t5_gemm_func.cc.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
Scanning dependencies of target cublasMMWrapper
[ 28%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/cublasMMWrapper.dir/cublasMMWrapper.cc.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 28%] Linking CUDA device code CMakeFiles/xlnet_gemm_func.dir/cmake_device_link.o
[ 29%] Linking CUDA device code CMakeFiles/cublasMMWrapper.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
[ 30%] Linking CUDA device code CMakeFiles/gpt_gemm_func.dir/cmake_device_link.o
[ 30%] Linking CUDA device code CMakeFiles/decoding_gemm_func.dir/cmake_device_link.o
[ 31%] Linking CUDA device code CMakeFiles/encoder_gemm_func.dir/cmake_device_link.o
[ 32%] Linking CXX static library ../../../../../../lib/libxlnet_gemm_func.a
[ 32%] Linking CXX static library ../../../../../lib/libcublasMMWrapper.a
[ 32%] Linking CXX static library ../../../../../../lib/libgpt_gemm_func.a
[ 32%] Linking CXX static library ../../../../../../lib/libdecoding_gemm_func.a
[ 32%] Built target xlnet_gemm_func
[ 32%] Built target cublasMMWrapper
[ 32%] Linking CXX static library ../../../../../../lib/libencoder_gemm_func.a
[ 32%] Built target gpt_gemm_func
[ 32%] Built target decoding_gemm_func
[ 32%] Linking CUDA device code CMakeFiles/t5_gemm_func.dir/cmake_device_link.o
Scanning dependencies of target cublasINT8MMWrapper
[ 33%] Building CXX object _deps/repo-ft-build/src/fastertransformer/utils/CMakeFiles/cublasINT8MMWrapper.dir/cublasINT8MMWrapper.cc.o
[ 33%] Built target encoder_gemm_func
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 34%] Linking CXX static library ../../../../../../lib/libt5_gemm_func.a
[ 34%] Built target t5_gemm_func
[ 34%] Linking CXX static library libtritonbackendutils.a
[ 34%] Built target triton-backend-utils
[ 34%] Linking CUDA device code CMakeFiles/cublasINT8MMWrapper.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 34%] Linking CXX static library ../../../../../lib/libcublasINT8MMWrapper.a
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/bert_preprocess_kernels.cu(66): warning: variable "row_id" was declared but never referenced
detected during instantiation of "void fastertransformer::invokeBuildEncoderAttentionMask(T *, const int *, int, int, cudaStream_t) [with T=float]"
(86): here
[ 34%] Built target cublasINT8MMWrapper
[ 34%] Linking CUDA device code CMakeFiles/matrix_transpose_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA device code CMakeFiles/quantization_int8_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA static library ../../../../../lib/libmatrix_transpose_kernels.a
[ 34%] Linking CUDA device code CMakeFiles/quantize_weight.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(190): warning: invalid format string conversion
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu(134): warning: variable "input_val" was declared but never referenced
detected during instantiation of "void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]"
(230): here
[ 34%] Built target matrix_transpose_kernels
[ 34%] Linking CUDA device code CMakeFiles/transpose_int8_kernels.dir/cmake_device_link.o
[ 34%] Linking CUDA device code CMakeFiles/add_residual_kernels.dir/cmake_device_link.o
[ 35%] Linking CUDA static library ../../../../../lib/libquantization_int8_kernels.a
[ 35%] Linking CUDA static library ../../../../../lib/libquantize_weight.a
[ 36%] Linking CUDA device code CMakeFiles/layout_transformer_int8_kernels.dir/cmake_device_link.o
[ 36%] Built target quantization_int8_kernels
[ 36%] Built target quantize_weight
[ 37%] Linking CUDA device code CMakeFiles/activation_kernels.dir/cmake_device_link.o
[ 37%] Linking CUDA device code CMakeFiles/activation_int8_kernels.dir/cmake_device_link.o
[ 38%] Linking CUDA static library ../../../../../lib/libtranspose_int8_kernels.a
[ 38%] Linking CUDA static library ../../../../../lib/libadd_residual_kernels.a
[ 38%] Built target transpose_int8_kernels
[ 38%] Built target add_residual_kernels
[ 38%] Linking CUDA static library ../../../../../lib/liblayout_transformer_int8_kernels.a
[ 38%] Linking CUDA device code CMakeFiles/gpt_kernels.dir/cmake_device_link.o
[ 38%] Linking CUDA static library ../../../../../lib/libactivation_kernels.a
[ 38%] Built target layout_transformer_int8_kernels
[ 38%] Linking CUDA static library ../../../../../lib/libactivation_int8_kernels.a
[ 38%] Built target activation_kernels
[ 38%] Built target activation_int8_kernels
[ 39%] Linking CUDA static library ../../../../../lib/libgpt_kernels.a
[ 39%] Linking CUDA device code CMakeFiles/add_bias_transpose_kernels.dir/cmake_device_link.o
[ 39%] Built target gpt_kernels
[ 39%] Linking CUDA device code CMakeFiles/decoding_kernels.dir/cmake_device_link.o
[ 40%] Linking CUDA static library ../../../../../lib/libadd_bias_transpose_kernels.a
[ 40%] Built target add_bias_transpose_kernels
[ 41%] Linking CXX static library ../../../../../lib/libcnpy.a
[ 41%] Built target cnpy
[ 41%] Linking CUDA static library ../../../../../lib/libdecoding_kernels.a
[ 41%] Built target decoding_kernels
[ 41%] Linking CUDA device code CMakeFiles/sampling_penalty_kernels.dir/cmake_device_link.o
[ 41%] Linking CUDA static library ../../../../../lib/libsampling_penalty_kernels.a
[ 41%] Built target sampling_penalty_kernels
Scanning dependencies of target BaseSamplingLayer
[ 42%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/sampling_layers/CMakeFiles/BaseSamplingLayer.dir/BaseSamplingLayer.cc.o
[ 42%] Linking CUDA device code CMakeFiles/xlnet_preprocess_kernels.dir/cmake_device_link.o
[ 42%] Linking CUDA device code CMakeFiles/xlnet_attention_kernels.dir/cmake_device_link.o
[ 42%] Linking CUDA static library ../../../../../lib/libxlnet_attention_kernels.a
[ 43%] Linking CUDA static library ../../../../../lib/libxlnet_preprocess_kernels.a
[ 43%] Linking CUDA device code CMakeFiles/bert_preprocess_kernels.dir/cmake_device_link.o
[ 43%] Built target xlnet_preprocess_kernels
[ 43%] Built target xlnet_attention_kernels
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of 'void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = float; int8_t = signed char; cudaStream_t = CUstream_st*]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:230:205:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: conversion lacks type at end of format [-Wformat=]
printf("[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 != 0).\n",
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: unknown conversion type character ' ' in format [-Wformat=]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:204:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
while (block.x * 4 > k)
~~~~~~~~~~~~~~~~~~^~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of 'void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = __half; int8_t = signed char; cudaStream_t = CUstream_st*]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:239:203:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: conversion lacks type at end of format [-Wformat=]
printf("[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 != 0).\n",
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:188:7: warning: unknown conversion type character ' ' in format [-Wformat=]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/matrix_vector_multiplication.cu:204:26: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
while (block.x * 4 > k)
~~~~~~~~~~~~~~~~~~^~
[ 43%] Linking CUDA device code CMakeFiles/calibrate_quantize_weight_kernels.dir/cmake_device_link.o
[ 43%] Linking CUDA static library ../../../../../lib/libbert_preprocess_kernels.a
[ 43%] Built target bert_preprocess_kernels
[ 43%] Linking CUDA device code CMakeFiles/matrix_vector_multiplication.dir/cmake_device_link.o
[ 43%] Linking CUDA static library ../../../../../lib/libcalibrate_quantize_weight_kernels.a
[ 43%] Built target calibrate_quantize_weight_kernels
[ 43%] Linking CUDA static library ../../../../../lib/libmatrix_vector_multiplication.a
[ 43%] Built target matrix_vector_multiplication
[ 43%] Linking CUDA device code CMakeFiles/BaseSamplingLayer.dir/cmake_device_link.o
[ 43%] Linking CXX static library ../../../../../../lib/libBaseSamplingLayer.a
[ 43%] Built target BaseSamplingLayer
[ 43%] Linking CUDA device code CMakeFiles/layernorm_int8_kernels.dir/cmake_device_link.o
[ 43%] Linking CUDA device code CMakeFiles/beam_search_penalty_kernels.dir/cmake_device_link.o
[ 43%] Linking CUDA static library ../../../../../lib/liblayernorm_int8_kernels.a
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 43%] Built target layernorm_int8_kernels
[ 43%] Linking CUDA static library ../../../../../lib/libbeam_search_penalty_kernels.a
[ 43%] Built target beam_search_penalty_kernels
Scanning dependencies of target BaseBeamSearchLayer
[ 43%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/layers/beam_search_layers/CMakeFiles/BaseBeamSearchLayer.dir/BaseBeamSearchLayer.cu.o
[ 43%] Linking CUDA device code CMakeFiles/layernorm_kernels.dir/cmake_device_link.o
[ 43%] Linking CUDA device code CMakeFiles/softmax_int8_kernels.dir/cmake_device_link.o
[ 43%] Linking CUDA static library ../../../../../lib/liblayernorm_kernels.a
[ 43%] Built target layernorm_kernels
[ 43%] Linking CUDA static library ../../../../../lib/libsoftmax_int8_kernels.a
[ 43%] Built target softmax_int8_kernels
[ 44%] Linking CUDA device code CMakeFiles/unfused_attention_int8_kernels.dir/cmake_device_link.o
[ 44%] Linking CUDA static library ../../../../../lib/libunfused_attention_int8_kernels.a
[ 44%] Built target unfused_attention_int8_kernels
Scanning dependencies of target UnfusedAttentionLayerINT8
[ 44%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers_int8/CMakeFiles/UnfusedAttentionLayerINT8.dir/UnfusedAttentionLayerINT8.cc.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In function 'void fastertransformer::set_alpha(uint32_t&, float, fastertransformer::Data_type)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:32:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
alpha = reinterpret_cast<const uint32_t&>(h2);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:36:50: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
alpha = reinterpret_cast<const uint32_t&>(norm);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In member function 'void fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:327:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_bmm1 = reinterpret_cast<const uint32_t&>(scaleBmm1);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:328:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_bmm2 = reinterpret_cast<const uint32_t&>(scaleBmm2);
^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:329:77: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
params.scale_softmax = reinterpret_cast<const uint32_t&>(scaleSoftmax);
^
[ 45%] Linking CUDA device code CMakeFiles/UnfusedAttentionLayerINT8.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu: In member function 'virtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int)':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:278:17: warning: 'warps_n' may be used uninitialized in this function [-Wmaybe-uninitialized]
size_t warps_m, warps_n, warps_k = 1;
^~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:278:8: warning: 'warps_m' may be used uninitialized in this function [-Wmaybe-uninitialized]
size_t warps_m, warps_n, warps_k = 1;
^~~~~~~
[ 45%] Linking CXX static library ../../../../../../lib/libUnfusedAttentionLayerINT8.a
[ 45%] Built target UnfusedAttentionLayerINT8
[ 45%] Linking CUDA device code CMakeFiles/trt_fused_multi_head_attention.dir/cmake_device_link.o
[ 45%] Linking CXX static library ../../../../lib/libtrt_fused_multi_head_attention.a
[ 45%] Built target trt_fused_multi_head_attention
Scanning dependencies of target FusedAttentionLayerINT8
[ 46%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers_int8/CMakeFiles/FusedAttentionLayerINT8.dir/FusedAttentionLayerINT8.cu.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu(118): warning: floating-point value does not fit in required integral type
detected during:
instantiation of "void fastertransformer::cuda_random_uniform_kernel(T *, int) [with T=int8_t]"
(154): here
instantiation of "void fastertransformer::cudaRandomUniform(T *, int) [with T=int8_t]"
(28): here
instantiation of "void fastertransformer::deviceMalloc(T **, int, __nv_bool) [with T=int8_t]"
(37): here
[ 47%] Linking CUDA device code CMakeFiles/sampling_topk_kernels.dir/cmake_device_link.o
[ 47%] Linking CUDA static library ../../../../../lib/libsampling_topk_kernels.a
[ 47%] Built target sampling_topk_kernels
Scanning dependencies of target TopKSamplingLayer
[ 47%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/layers/sampling_layers/CMakeFiles/TopKSamplingLayer.dir/TopKSamplingLayer.cu.o
[ 47%] Linking CUDA device code CMakeFiles/longformer_kernels.dir/cmake_device_link.o
[ 48%] Linking CUDA static library ../../../../../lib/liblongformer_kernels.a
[ 48%] Built target longformer_kernels
[ 48%] Linking CUDA device code CMakeFiles/BaseBeamSearchLayer.dir/cmake_device_link.o
Scanning dependencies of target LongformerAttentionLayer
[ 49%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/LongformerAttentionLayer.dir/LongformerAttentionLayer.cc.o
[ 49%] Linking CUDA static library ../../../../../../lib/libBaseBeamSearchLayer.a
[ 49%] Built target BaseBeamSearchLayer
[ 49%] Linking CUDA device code CMakeFiles/unfused_attention_kernels.dir/cmake_device_link.o
[ 50%] Linking CUDA static library ../../../../../lib/libunfused_attention_kernels.a
[ 50%] Built target unfused_attention_kernels
[ 50%] Linking CUDA device code CMakeFiles/LongformerAttentionLayer.dir/cmake_device_link.o
[ 50%] Linking CXX static library ../../../../../../lib/libLongformerAttentionLayer.a
[ 50%] Built target LongformerAttentionLayer
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/kernels/sampling_topp_kernels.cu(491): warning: variable "KEYS_PER_LDG" was declared but never referenced
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/logger.h: In instantiation of 'void fastertransformer::Logger::log(fastertransformer::Logger::Level, std::__cxx11::string, const Args& ...) [with Args = {}; std::__cxx11::string = std::__cxx11::basic_string]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu:189:43:   required from 'int fastertransformer::loadWeightFromBin(T*, std::vector, std::__cxx11::string) [with T = float; std::__cxx11::string = std::__cxx11::basic_string]'
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/memory_utils.cu:213:100:   required from here
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/utils/logger.h:30:8: warning: format not a string literal and no format arguments [-Wformat-security]
fprintf(out, fmt.c_str(), args...);
^    ~~~~~~~~~~~~~~~~~~~~~~
[ 51%] Linking CUDA device code CMakeFiles/memory_utils.dir/cmake_device_link.o
[ 51%] Linking CUDA static library ../../../../../lib/libmemory_utils.a
[ 51%] Built target memory_utils
Scanning dependencies of target UnfusedAttentionLayer
Scanning dependencies of target gpt_gemm
Scanning dependencies of target ParallelGptDecoderLayerWeight
Scanning dependencies of target bert_gemm
Scanning dependencies of target decoding_gemm
Scanning dependencies of target XlnetAttentionLayer
Scanning dependencies of target FfnLayer
Scanning dependencies of target FfnLayerINT8
Scanning dependencies of target xlnet_gemm
Scanning dependencies of target GptContextAttentionLayer
Scanning dependencies of target GptDecoderLayerWeight
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/UnfusedAttentionLayer.dir/UnfusedAttentionLayer.cc.o
Scanning dependencies of target GptJDecoderLayerWeight
Scanning dependencies of target t5_gemm
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptDecoderLayerWeight.dir/ParallelGptDecoderLayerWeight.cc.o
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/gpt_gemm.dir/gpt_gemm.cc.o
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/decoding/CMakeFiles/decoding_gemm.dir/decoding_gemm.cc.o
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/bert/CMakeFiles/bert_gemm.dir/bert_gemm.cc.o
Scanning dependencies of target FusedAttentionLayer
[ 52%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/CMakeFiles/FfnLayerINT8.dir/FfnLayerINT8.cc.o
[ 53%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/xlnet_attention_layers/CMakeFiles/XlnetAttentionLayer.dir/XlnetAttentionLayer.cc.o
[ 53%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/xlnet/CMakeFiles/xlnet_gemm.dir/xlnet_gemm.cc.o
[ 53%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/GptContextAttentionLayer.dir/GptContextAttentionLayer.cc.o
[ 53%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/CMakeFiles/FfnLayer.dir/FfnLayer.cc.o
[ 54%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gpt/CMakeFiles/GptDecoderLayerWeight.dir/GptDecoderLayerWeight.cc.o
[ 55%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gptj/CMakeFiles/GptJDecoderLayerWeight.dir/GptJDecoderLayerWeight.cc.o
[ 56%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/t5/CMakeFiles/t5_gemm.dir/t5_gemm.cc.o
[ 56%] Building CUDA object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/FusedAttentionLayer.dir/FusedAttentionLayer.cu.o
[ 57%] Linking CUDA device code CMakeFiles/TopKSamplingLayer.dir/cmake_device_link.o
[ 57%] Linking CXX static library ../../../../../../lib/libTopKSamplingLayer.a
[ 57%] Built target TopKSamplingLayer
[ 57%] Linking CXX executable ../../../../../../bin/bert_gemm
[ 57%] Linking CXX executable ../../../../../../bin/xlnet_gemm
[ 57%] Linking CXX executable ../../../../../../bin/gpt_gemm
[ 57%] Linking CXX executable ../../../../../../bin/t5_gemm
[ 58%] Linking CXX executable ../../../../../../bin/decoding_gemm
[ 58%] Built target bert_gemm
[ 58%] Linking CUDA device code CMakeFiles/FfnLayerINT8.dir/cmake_device_link.o
[ 58%] Built target xlnet_gemm
[ 58%] Built target decoding_gemm
[ 58%] Built target t5_gemm
[ 58%] Built target gpt_gemm
[ 58%] Linking CUDA device code CMakeFiles/GptJDecoderLayerWeight.dir/cmake_device_link.o
[ 59%] Linking CUDA device code CMakeFiles/FfnLayer.dir/cmake_device_link.o
[ 60%] Linking CXX static library ../../../../../lib/libFfnLayerINT8.a
[ 60%] Built target FfnLayerINT8
[ 60%] Linking CUDA device code CMakeFiles/GptDecoderLayerWeight.dir/cmake_device_link.o
[ 60%] Linking CUDA device code CMakeFiles/UnfusedAttentionLayer.dir/cmake_device_link.o
[ 60%] Linking CXX static library ../../../../../../lib/libGptJDecoderLayerWeight.a
[ 60%] Linking CXX static library ../../../../../lib/libFfnLayer.a
[ 61%] Linking CUDA device code CMakeFiles/GptContextAttentionLayer.dir/cmake_device_link.o
[ 61%] Built target GptJDecoderLayerWeight
[ 61%] Built target FfnLayer
Scanning dependencies of target GptJWeight
Scanning dependencies of target LongformerEncoder
Scanning dependencies of target TensorParallelGeluFfnLayer
Scanning dependencies of target TensorParallelReluFfnLayer
[ 61%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/longformer/CMakeFiles/LongformerEncoder.dir/LongformerEncoder.cc.o
[ 61%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/CMakeFiles/TensorParallelReluFfnLayer.dir/TensorParallelReluFfnLayer.cc.o
[ 61%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gptj/CMakeFiles/GptJWeight.dir/GptJWeight.cc.o
[ 61%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/CMakeFiles/TensorParallelGeluFfnLayer.dir/TensorParallelGeluFfnLayer.cc.o
[ 61%] Linking CXX static library ../../../../../../lib/libUnfusedAttentionLayer.a
[ 61%] Linking CXX static library ../../../../../../lib/libGptDecoderLayerWeight.a
[ 61%] Built target GptDecoderLayerWeight
[ 61%] Built target UnfusedAttentionLayer
[ 61%] Linking CXX static library ../../../../../../lib/libGptContextAttentionLayer.a
Scanning dependencies of target GptWeight
Scanning dependencies of target TensorParallelUnfusedAttentionLayer
[ 61%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gpt/CMakeFiles/GptWeight.dir/GptWeight.cc.o
[ 62%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelUnfusedAttentionLayer.dir/TensorParallelUnfusedAttentionLayer.cc.o
[ 62%] Built target GptContextAttentionLayer
[ 63%] Linking CUDA device code CMakeFiles/ParallelGptDecoderLayerWeight.dir/cmake_device_link.o
Scanning dependencies of target TensorParallelGptContextAttentionLayer
Scanning dependencies of target GptContextDecoder
[ 63%] Building CXX object _deps/repo-ft-build/src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelGptContextAttentionLayer.dir/TensorParallelGptContextAttentionLayer.cc.o
[ 63%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gpt/CMakeFiles/GptContextDecoder.dir/GptContextDecoder.cc.o
[ 63%] Linking CUDA device code CMakeFiles/XlnetAttentionLayer.dir/cmake_device_link.o
[ 63%] Linking CXX static library ../../../../../../lib/libParallelGptDecoderLayerWeight.a
[ 63%] Linking CXX static library ../../../../../../lib/libXlnetAttentionLayer.a
[ 63%] Built target ParallelGptDecoderLayerWeight
Scanning dependencies of target ParallelGptWeight
[ 63%] Built target XlnetAttentionLayer
[ 63%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptWeight.dir/ParallelGptWeight.cc.o
Scanning dependencies of target Xlnet
[ 63%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/xlnet/CMakeFiles/Xlnet.dir/Xlnet.cc.o
[ 63%] Linking CUDA device code CMakeFiles/FusedAttentionLayerINT8.dir/cmake_device_link.o
[ 63%] Linking CUDA device code CMakeFiles/TensorParallelReluFfnLayer.dir/cmake_device_link.o
[ 63%] Linking CUDA device code CMakeFiles/TensorParallelGeluFfnLayer.dir/cmake_device_link.o
[ 63%] Linking CUDA device code CMakeFiles/TensorParallelUnfusedAttentionLayer.dir/cmake_device_link.o
[ 63%] Linking CXX static library ../../../../../../lib/libFusedAttentionLayerINT8.a
[ 63%] Linking CXX static library ../../../../../lib/libTensorParallelReluFfnLayer.a
[ 64%] Linking CXX static library ../../../../../lib/libTensorParallelGeluFfnLayer.a
[ 64%] Linking CUDA device code CMakeFiles/TensorParallelGptContextAttentionLayer.dir/cmake_device_link.o
[ 64%] Built target FusedAttentionLayerINT8
[ 64%] Built target TensorParallelReluFfnLayer
Scanning dependencies of target BertLayerINT8
[ 64%] Linking CXX static library ../../../../../../lib/libTensorParallelUnfusedAttentionLayer.a
[ 64%] Built target TensorParallelGeluFfnLayer
[ 65%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/bert_int8/CMakeFiles/BertLayerINT8.dir/BertLayerINT8.cc.o
[ 65%] Linking CUDA device code CMakeFiles/GptJWeight.dir/cmake_device_link.o
[ 65%] Built target TensorParallelUnfusedAttentionLayer
[ 66%] Linking CXX static library ../../../../../../lib/libTensorParallelGptContextAttentionLayer.a
[ 66%] Built target TensorParallelGptContextAttentionLayer
Scanning dependencies of target ParallelGptContextDecoder
Scanning dependencies of target GptJContextDecoder
[ 67%] Linking CXX static library ../../../../../../lib/libGptJWeight.a
[ 67%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptContextDecoder.dir/ParallelGptContextDecoder.cc.o
[ 67%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/gptj/CMakeFiles/GptJContextDecoder.dir/GptJContextDecoder.cc.o
[ 67%] Linking CUDA device code CMakeFiles/GptWeight.dir/cmake_device_link.o
[ 67%] Built target GptJWeight
[ 68%] Linking CUDA device code CMakeFiles/LongformerEncoder.dir/cmake_device_link.o
[ 68%] Linking CXX static library ../../../../../../lib/libGptWeight.a
[ 68%] Built target GptWeight
[ 68%] Linking CXX static library ../../../../../../lib/libLongformerEncoder.a
[ 68%] Built target LongformerEncoder
[ 68%] Linking CUDA device code CMakeFiles/ParallelGptWeight.dir/cmake_device_link.o
[ 68%] Linking CUDA device code CMakeFiles/GptContextDecoder.dir/cmake_device_link.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc: In instantiation of 'void fastertransformer::GptJContextDecoder::forward(std::vectorfastertransformer::Tensor, const std::vectorfastertransformer::Tensor, const std::vector<fastertransformer::GptJDecoderLayerWeight >) [with T = float]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:388:16:   required from here
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:302:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:302:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)local_batch_size' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
std::vector self_attention_input_tensors{
^~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'local_batch_size' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)seq_len' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'seq_len' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)seq_len' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'seq_len' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:322:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~~~~~~~~~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:322:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:339:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~~~~~~~~~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:339:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:344:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~~~~~~~~~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:344:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc: In instantiation of 'void fastertransformer::GptJContextDecoder::forward(std::vectorfastertransformer::Tensor, const std::vectorfastertransformer::Tensor, const std::vector<fastertransformer::GptJDecoderLayerWeight >) [with T = __half]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:389:16:   required from here
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:302:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:302:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)local_batch_size' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
std::vector self_attention_input_tensors{
^~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'local_batch_size' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)seq_len' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'seq_len' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of '(int)seq_len' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:300:33: warning: narrowing conversion of 'seq_len' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:322:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~^
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:322:42: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:339:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:339:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:344:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
{local_batch_size * seq_len, hidden_units},
~^
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/gptj/GptJContextDecoder.cc:344:46: warning: narrowing conversion of '(((int)local_batch_size) * ((int)seq_len))' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
[ 68%] Linking CXX static library ../../../../../../lib/libParallelGptWeight.a
[ 69%] Linking CXX static library ../../../../../../lib/libGptContextDecoder.a
[ 69%] Built target ParallelGptWeight
[ 69%] Built target GptContextDecoder
[ 69%] Linking CUDA device code CMakeFiles/Xlnet.dir/cmake_device_link.o
[ 70%] Linking CXX static library ../../../../../../lib/libXlnet.a
[ 70%] Built target Xlnet
Scanning dependencies of target xlnet_example
Scanning dependencies of target xlnet_correctness_example
[ 71%] Building CXX object _deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/xlnet_correctness_example.dir/xlnet_correctness_example.cc.o
[ 71%] Building CXX object _deps/repo-ft-build/examples/cpp/xlnet/CMakeFiles/xlnet_example.dir/xlnet_example.cc.o
[ 71%] Linking CUDA device code CMakeFiles/GptJContextDecoder.dir/cmake_device_link.o
[ 71%] Linking CUDA device code CMakeFiles/ParallelGptContextDecoder.dir/cmake_device_link.o
[ 71%] Linking CUDA device code CMakeFiles/BertLayerINT8.dir/cmake_device_link.o
[ 72%] Linking CXX static library ../../../../../../lib/libGptJContextDecoder.a
[ 72%] Built target GptJContextDecoder
[ 72%] Linking CXX static library ../../../../../../lib/libParallelGptContextDecoder.a
[ 72%] Linking CXX static library ../../../../../../lib/libBertLayerINT8.a
[ 72%] Built target ParallelGptContextDecoder
[ 72%] Built target BertLayerINT8
Scanning dependencies of target BertINT8
[ 72%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/bert_int8/CMakeFiles/BertINT8.dir/BertINT8.cc.o
[ 72%] Linking CXX executable ../../../../../bin/xlnet_example
[ 72%] Built target xlnet_example
[ 72%] Linking CXX executable ../../../../../bin/xlnet_correctness_example
[ 72%] Built target xlnet_correctness_example
[ 72%] Linking CUDA device code CMakeFiles/BertINT8.dir/cmake_device_link.o
[ 72%] Linking CXX static library ../../../../../../lib/libBertINT8.a
[ 72%] Built target BertINT8
Scanning dependencies of target bert_int8_example
[ 72%] Building CXX object _deps/repo-ft-build/examples/cpp/bert_int8/CMakeFiles/bert_int8_example.dir/bert_int8_example.cc.o
[ 73%] Linking CXX executable ../../../../../bin/bert_int8_example
[ 73%] Built target bert_int8_example
[ 73%] Linking CUDA device code CMakeFiles/FusedAttentionLayer.dir/cmake_device_link.o
[ 73%] Linking CXX static library ../../../../../../lib/libFusedAttentionLayer.a
[ 73%] Built target FusedAttentionLayer
Scanning dependencies of target Bert
Scanning dependencies of target T5Encoder
[ 73%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/bert/CMakeFiles/Bert.dir/Bert.cc.o
[ 74%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5Encoder.cc.o
[ 74%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5EncoderWeight.cc.o
[ 74%] Building CXX object _deps/repo-ft-build/src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5EncoderLayerWeight.cc.o
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderWeight.cc:17:9: warning: #pragma once in main file
#pragma once
^~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:17:9: warning: #pragma once in main file
#pragma once
^~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc: In instantiation of 'void fastertransformer::T5EncoderLayerWeight::loadModel(std::__cxx11::string) [with T = float; std::__cxx11::string = std::__cxx11::basic_string]':
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:239:17:   required from here
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:216:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight*)this)->fastertransformer::T5EncoderLayerWeight::weights_size[0]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[0],
~~~~~~^~
{weights_size[0]},
~~~~~~~~~~~~~~~~~~
dir_path + "layer.0.SelfAttention.q.weight." + std::to_string(tensor_para_rank) + ".bin");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:216:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[0]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:219:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[1]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[1],
~~~~~~^~
{weights_size[1]},
~~~~~~~~~~~~~~~~~~
dir_path + "layer.0.SelfAttention.k.weight." + std::to_string(tensor_para_rank) + ".bin");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:219:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[1]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:222:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[2]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[2],{weights_size[2]},
^~
dir_path + "layer.0.SelfAttention.v.weight." + std::to_string(tensor_para_rank) + ".bin");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:222:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[2]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:224:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[3]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[3],
~~~~~~^~
{weights_size[3]},
~~~~~~~~~~~~~~~~~~
dir_path + "layer.0.SelfAttention.o.weight." + std::to_string(tensor_para_rank) + ".bin");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:224:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[3]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:227:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[4]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[4], {weights_size[4]}, dir_path + "layer.0.layer_norm.weight.bin");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:227:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[4]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
/workspace/build/fastertransformer_backend/build/_deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:228:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[5]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]
loadWeightFromBin(weights_ptr[5],
~~~~~~^~
{weights_size[5]},
~~~~~~~~~~~~~~~~~~
dir_path + "layer.1.DenseReluDense.wi.weight." + std::to_string(tensor_para_rank) + ".bin");
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/workspace/build/fastertransformer_backend/build/deps/repo-ft-src/src/fastertransformer/models/t5/T5EncoderLayerWeight.cc:228:25: warning: narrowing conversion of '((fastertransformer::T5EncoderLayerWeight)this)->fastertransformer::T5EncoderLayerWeight::weights_size[5]' from 'size_t {aka long unsigned int}' to 'int' inside { } [-Wnarrowing]