[archlinux] libaio issues

@sgugger is trying to get offload/nvme to work on archlinux and has issues building the async_io extension.
First, I discovered Archlinux (I don't know this system) doesn't do dev packages, but packs everything into one package.
So there it is pacman libaio which he installed. I did check that that package https://archlinux.org/packages/core/x86_64/libaio/ contains libaio.h and  libaio.so.1
What else is missing? he is still getting:
 [WARNING]  async_io requires the libraries: ['libaio-dev'] but are missing. Can be fixed by: `apt install libaio-dev`.

On ubuntu:
$ dpkg-query -L libaio-dev
/usr/include/libaio.h
/usr/lib/x86_64-linux-gnu/libaio.a
/usr/lib/x86_64-linux-gnu/libaio.so

(I trimmed the output)
On archlinux: (package I manually downloaded from https://archlinux.org/packages/core/x86_64/libaio/)
$ tar xvf libaio-0.3.112-2-x86_64.pkg.tar
usr/include/libaio.h
usr/lib/libaio.so.1.0.1

(again trimmed to the essentials)
So the latter is missing libaio.a - does  async_io want the static object and won't use the shared object? Is that the problem?
You can see how archlinux package was built here:
https://github.com/archlinux/svntogit-packages/blob/packages/libaio/trunk/PKGBUILD
In any case I think it'd be useful to expose the build command and the error and not just have:
self = <deepspeed.ops.op_builder.async_io.AsyncIOBuilder object at 0x7fc52033a790>, verbose = True
    def jit_load(self, verbose=True):
        if not self.is_compatible():
            raise RuntimeError(
>               f"Unable to JIT load the {self.name} op due to it not being compatible due to hardware/software issue."
            )
E           RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue.

Since as you can see on non-Ubuntu system, this would be much more difficult to debug and need the actual compiler error.
@tjruwase

Also for the sake of the test suite, @tjruwase, could you think of a skip rule that could query whether the async_io extension can be built and so the test using nvme could be skipped if it can't be built?

@stas00, with regards to skip rule, do you mean deepspeed unit test and a rule like this?

Ah, great!
python -c "import deepspeed; from deepspeed.ops.aio import AsyncIOBuilder; print(deepspeed.ops.__compatible_ops__[AsyncIOBuilder.NAME])"

I get True, Sylvain gets False.
Can we reliably use this API in transformers tests?

Yes, you can reliably use that to detect availability of aio library.
Also, I believe building from source as follows will attempt to compile the aio sources and expose the compiler commands.
DS_BUILD_AIO=1 bash install.sh -s
Can you please try that?

Also, regarding building with static or shared libaio, can you confirm you want to investigate further?

As long as @sgugger is willing, yes, of course! I have no access to Archlinux - he does.
He will post the log shortly.

The command above produces the following:
(base:base) [sgugger@brahms DeepSpeed]$ DS_BUILD_AIO=1 bash install.sh -s
Attempting to remove deepspeed/git_version_info_installed.py
Attempting to remove dist
Attempting to remove build
Attempting to remove deepspeed.egg-info
No hostfile exists at /job/hostfile, installing locally
Building deepspeed wheel
DS_BUILD_OPS=0
/bin/sh: line 1: type: llvm-config-9: not found
/bin/sh: line 1: type: cmake: not found
 [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
 [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
 [WARNING]  async_io requires the libraries: ['libaio-dev'] but are missing. Can be fixed by: `apt install libaio-dev`.
/bin/sh: line 1: type: llvm-config-9: not found
/bin/sh: line 1: type: cmake: not found
 [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
 [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
 [WARNING]  async_io requires the libraries: ['libaio-dev'] but are missing. Can be fixed by: `apt install libaio-dev`.
Install Ops={'cpu_adam': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': False, 'transformer_inference': False, 'utils': False, 'quantizer': False}
version=0.4.0+11e94e6, git_hash=11e94e6, git_branch=master
install_requires=['torch>=1.2', 'torchvision>=0.4.0', 'tqdm', 'tensorboardX==1.8', 'ninja', 'numpy', 'psutil', 'packaging']
compatible_ops={'cpu_adam': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': False, 'transformer': True, 'stochastic_transformer': True, 'async_io': False, 'transformer_inference': True, 'utils': True, 'quantizer': True}
ext_modules=[]
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/deepspeed
copying deepspeed/constants.py -> build/lib/deepspeed
copying deepspeed/git_version_info.py -> build/lib/deepspeed
copying deepspeed/__init__.py -> build/lib/deepspeed
copying deepspeed/env_report.py -> build/lib/deepspeed
copying deepspeed/git_version_info_installed.py -> build/lib/deepspeed
creating build/lib/op_builder
copying op_builder/fused_adam.py -> build/lib/op_builder
copying op_builder/transformer_inference.py -> build/lib/op_builder
copying op_builder/builder.py -> build/lib/op_builder
copying op_builder/utils.py -> build/lib/op_builder
copying op_builder/fused_lamb.py -> build/lib/op_builder
copying op_builder/transformer.py -> build/lib/op_builder
copying op_builder/quantizer.py -> build/lib/op_builder
copying op_builder/__init__.py -> build/lib/op_builder
copying op_builder/async_io.py -> build/lib/op_builder
copying op_builder/cpu_adam.py -> build/lib/op_builder
copying op_builder/sparse_attn.py -> build/lib/op_builder
copying op_builder/stochastic_transformer.py -> build/lib/op_builder
creating build/lib/deepspeed/pipe
copying deepspeed/pipe/__init__.py -> build/lib/deepspeed/pipe
creating build/lib/deepspeed/elasticity
copying deepspeed/elasticity/config.py -> build/lib/deepspeed/elasticity
copying deepspeed/elasticity/constants.py -> build/lib/deepspeed/elasticity
copying deepspeed/elasticity/__init__.py -> build/lib/deepspeed/elasticity
copying deepspeed/elasticity/elasticity.py -> build/lib/deepspeed/elasticity
creating build/lib/deepspeed/module_inject
copying deepspeed/module_inject/replace_policy.py -> build/lib/deepspeed/module_inject
copying deepspeed/module_inject/inject.py -> build/lib/deepspeed/module_inject
copying deepspeed/module_inject/replace_module.py -> build/lib/deepspeed/module_inject
copying deepspeed/module_inject/__init__.py -> build/lib/deepspeed/module_inject
copying deepspeed/module_inject/module_quantize.py -> build/lib/deepspeed/module_inject
creating build/lib/deepspeed/utils
copying deepspeed/utils/zero_to_fp32.py -> build/lib/deepspeed/utils
copying deepspeed/utils/timer.py -> build/lib/deepspeed/utils
copying deepspeed/utils/__init__.py -> build/lib/deepspeed/utils
copying deepspeed/utils/logging.py -> build/lib/deepspeed/utils
copying deepspeed/utils/distributed.py -> build/lib/deepspeed/utils
creating build/lib/deepspeed/launcher
copying deepspeed/launcher/constants.py -> build/lib/deepspeed/launcher
copying deepspeed/launcher/launch.py -> build/lib/deepspeed/launcher
copying deepspeed/launcher/runner.py -> build/lib/deepspeed/launcher
copying deepspeed/launcher/__init__.py -> build/lib/deepspeed/launcher
copying deepspeed/launcher/multinode_runner.py -> build/lib/deepspeed/launcher
creating build/lib/deepspeed/ops
copying deepspeed/ops/__init__.py -> build/lib/deepspeed/ops
creating build/lib/deepspeed/runtime
copying deepspeed/runtime/state_dict_factory.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/weight_quantizer.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/utils.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/config.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/constants.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/progressive_layer_drop.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/config_utils.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/__init__.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/engine.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/lr_schedules.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/eigenvalue.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/dataloader.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/csr_tensor.py -> build/lib/deepspeed/runtime
copying deepspeed/runtime/quantize.py -> build/lib/deepspeed/runtime
creating build/lib/deepspeed/inference
copying deepspeed/inference/__init__.py -> build/lib/deepspeed/inference
copying deepspeed/inference/engine.py -> build/lib/deepspeed/inference
creating build/lib/deepspeed/profiling
copying deepspeed/profiling/config.py -> build/lib/deepspeed/profiling
copying deepspeed/profiling/constants.py -> build/lib/deepspeed/profiling
copying deepspeed/profiling/__init__.py -> build/lib/deepspeed/profiling
creating build/lib/deepspeed/ops/quantizer
copying deepspeed/ops/quantizer/quantizer.py -> build/lib/deepspeed/ops/quantizer
copying deepspeed/ops/quantizer/__init__.py -> build/lib/deepspeed/ops/quantizer
creating build/lib/deepspeed/ops/aio
copying deepspeed/ops/aio/__init__.py -> build/lib/deepspeed/ops/aio
creating build/lib/deepspeed/ops/transformer
copying deepspeed/ops/transformer/transformer.py -> build/lib/deepspeed/ops/transformer
copying deepspeed/ops/transformer/__init__.py -> build/lib/deepspeed/ops/transformer
creating build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/matmul.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/softmax.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/sparsity_config.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/bert_sparse_self_attention.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/__init__.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/sparse_attention_utils.py -> build/lib/deepspeed/ops/sparse_attention
copying deepspeed/ops/sparse_attention/sparse_self_attention.py -> build/lib/deepspeed/ops/sparse_attention
creating build/lib/deepspeed/ops/adam
copying deepspeed/ops/adam/fused_adam.py -> build/lib/deepspeed/ops/adam
copying deepspeed/ops/adam/__init__.py -> build/lib/deepspeed/ops/adam
copying deepspeed/ops/adam/cpu_adam.py -> build/lib/deepspeed/ops/adam
copying deepspeed/ops/adam/multi_tensor_apply.py -> build/lib/deepspeed/ops/adam
creating build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/fused_adam.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/transformer_inference.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/builder.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/utils.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/fused_lamb.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/transformer.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/quantizer.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/__init__.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/async_io.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/cpu_adam.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/sparse_attn.py -> build/lib/deepspeed/ops/op_builder
copying deepspeed/ops/op_builder/stochastic_transformer.py -> build/lib/deepspeed/ops/op_builder
creating build/lib/deepspeed/ops/lamb
copying deepspeed/ops/lamb/fused_lamb.py -> build/lib/deepspeed/ops/lamb
copying deepspeed/ops/lamb/__init__.py -> build/lib/deepspeed/ops/lamb
creating build/lib/deepspeed/ops/transformer/inference
copying deepspeed/ops/transformer/inference/transformer_inference.py -> build/lib/deepspeed/ops/transformer/inference
copying deepspeed/ops/transformer/inference/__init__.py -> build/lib/deepspeed/ops/transformer/inference
creating build/lib/deepspeed/ops/sparse_attention/trsrc
copying deepspeed/ops/sparse_attention/trsrc/__init__.py -> build/lib/deepspeed/ops/sparse_attention/trsrc
creating build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/schedule.py -> build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/__init__.py -> build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/engine.py -> build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/module.py -> build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/topology.py -> build/lib/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/p2p.py -> build/lib/deepspeed/runtime/pipe
creating build/lib/deepspeed/runtime/comm
copying deepspeed/runtime/comm/mpi.py -> build/lib/deepspeed/runtime/comm
copying deepspeed/runtime/comm/__init__.py -> build/lib/deepspeed/runtime/comm
copying deepspeed/runtime/comm/nccl.py -> build/lib/deepspeed/runtime/comm
creating build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/async_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/utils.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/constants.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/partitioned_param_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/aio_config.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/__init__.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py -> build/lib/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/optimizer_utils.py -> build/lib/deepspeed/runtime/swap_tensor
creating build/lib/deepspeed/runtime/fp16
copying deepspeed/runtime/fp16/loss_scaler.py -> build/lib/deepspeed/runtime/fp16
copying deepspeed/runtime/fp16/fused_optimizer.py -> build/lib/deepspeed/runtime/fp16
copying deepspeed/runtime/fp16/unfused_optimizer.py -> build/lib/deepspeed/runtime/fp16
copying deepspeed/runtime/fp16/__init__.py -> build/lib/deepspeed/runtime/fp16
creating build/lib/deepspeed/runtime/activation_checkpointing
copying deepspeed/runtime/activation_checkpointing/config.py -> build/lib/deepspeed/runtime/activation_checkpointing
copying deepspeed/runtime/activation_checkpointing/checkpointing.py -> build/lib/deepspeed/runtime/activation_checkpointing
copying deepspeed/runtime/activation_checkpointing/__init__.py -> build/lib/deepspeed/runtime/activation_checkpointing
creating build/lib/deepspeed/runtime/compression
copying deepspeed/runtime/compression/cupy.py -> build/lib/deepspeed/runtime/compression
copying deepspeed/runtime/compression/__init__.py -> build/lib/deepspeed/runtime/compression
creating build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/partition_parameters.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/utils.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/config.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/constants.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/stage3.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/linear.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/offload_constants.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/__init__.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/test.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/tiling.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/contiguous_memory_allocator.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/stage1.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/stage2.py -> build/lib/deepspeed/runtime/zero
copying deepspeed/runtime/zero/offload_config.py -> build/lib/deepspeed/runtime/zero
creating build/lib/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/adam.py -> build/lib/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/lamb.py -> build/lib/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/__init__.py -> build/lib/deepspeed/runtime/fp16/onebit
creating build/lib/deepspeed/profiling/flops_profiler
copying deepspeed/profiling/flops_profiler/profiler.py -> build/lib/deepspeed/profiling/flops_profiler
copying deepspeed/profiling/flops_profiler/__init__.py -> build/lib/deepspeed/profiling/flops_profiler
running egg_info
creating deepspeed.egg-info
writing deepspeed.egg-info/PKG-INFO
writing dependency_links to deepspeed.egg-info/dependency_links.txt
writing entry points to deepspeed.egg-info/entry_points.txt
writing requirements to deepspeed.egg-info/requires.txt
writing top-level names to deepspeed.egg-info/top_level.txt
writing manifest file 'deepspeed.egg-info/SOURCES.txt'
reading manifest file 'deepspeed.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.cc' under directory 'deepspeed'
warning: no files found matching '*.tr' under directory 'csrc'
warning: no files found matching '*.cc' under directory 'csrc'
writing manifest file 'deepspeed.egg-info/SOURCES.txt'
creating build/lib/deepspeed/ops/csrc
creating build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/compat.h -> build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/cpu_adam.cpp -> build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -> build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -> build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/multi_tensor_adam.cu -> build/lib/deepspeed/ops/csrc/adam
copying deepspeed/ops/csrc/adam/multi_tensor_apply.cuh -> build/lib/deepspeed/ops/csrc/adam
creating build/lib/deepspeed/ops/csrc/aio
creating build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.cpp -> build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.h -> build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.cpp -> build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.h -> build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.cpp -> build/lib/deepspeed/ops/csrc/aio/common
copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.h -> build/lib/deepspeed/ops/csrc/aio/common
creating build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.cpp -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.h -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.cpp -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.h -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.cpp -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.h -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.cpp -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.h -> build/lib/deepspeed/ops/csrc/aio/py_lib
copying deepspeed/ops/csrc/aio/py_lib/py_ds_aio.cpp -> build/lib/deepspeed/ops/csrc/aio/py_lib
creating build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/StopWatch.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/Timer.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/context.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/cpu_adam.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/cublas_wrappers.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/custom_cuda_layers.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/dropout.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/ds_transformer_cuda.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/feed_forward.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/gelu.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/gemm_test.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/general_kernels.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/normalize_layer.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/quantizer.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/softmax.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/strided_batch_gemm.h -> build/lib/deepspeed/ops/csrc/includes
copying deepspeed/ops/csrc/includes/type_shim.h -> build/lib/deepspeed/ops/csrc/includes
creating build/lib/deepspeed/ops/csrc/lamb
copying deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp -> build/lib/deepspeed/ops/csrc/lamb
copying deepspeed/ops/csrc/lamb/fused_lamb_cuda_kernel.cu -> build/lib/deepspeed/ops/csrc/lamb
creating build/lib/deepspeed/ops/csrc/quantization
copying deepspeed/ops/csrc/quantization/pt_binding.cpp -> build/lib/deepspeed/ops/csrc/quantization
copying deepspeed/ops/csrc/quantization/quantizer.cu -> build/lib/deepspeed/ops/csrc/quantization
creating build/lib/deepspeed/ops/csrc/sparse_attention
copying deepspeed/ops/csrc/sparse_attention/utils.cpp -> build/lib/deepspeed/ops/csrc/sparse_attention
creating build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/cublas_wrappers.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/dropout_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/ds_transformer_cuda.cpp -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/gelu_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/general_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/normalize_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/softmax_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
copying deepspeed/ops/csrc/transformer/transform_kernels.cu -> build/lib/deepspeed/ops/csrc/transformer
creating build/lib/deepspeed/ops/csrc/transformer/inference
creating build/lib/deepspeed/ops/csrc/transformer/inference/csrc
copying deepspeed/ops/csrc/transformer/inference/csrc/dequantize.cu -> build/lib/deepspeed/ops/csrc/transformer/inference/csrc
copying deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -> build/lib/deepspeed/ops/csrc/transformer/inference/csrc
copying deepspeed/ops/csrc/transformer/inference/csrc/normalize.cu -> build/lib/deepspeed/ops/csrc/transformer/inference/csrc
copying deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -> build/lib/deepspeed/ops/csrc/transformer/inference/csrc
copying deepspeed/ops/csrc/transformer/inference/csrc/softmax.cu -> build/lib/deepspeed/ops/csrc/transformer/inference/csrc
creating build/lib/deepspeed/ops/csrc/transformer/inference/includes
copying deepspeed/ops/csrc/transformer/inference/includes/context.h -> build/lib/deepspeed/ops/csrc/transformer/inference/includes
copying deepspeed/ops/csrc/transformer/inference/includes/cublas_wrappers.h -> build/lib/deepspeed/ops/csrc/transformer/inference/includes
copying deepspeed/ops/csrc/transformer/inference/includes/custom_cuda_layers.h -> build/lib/deepspeed/ops/csrc/transformer/inference/includes
creating build/lib/deepspeed/ops/csrc/utils
copying deepspeed/ops/csrc/utils/flatten_unflatten.cpp -> build/lib/deepspeed/ops/csrc/utils
copying deepspeed/ops/sparse_attention/trsrc/matmul.tr -> build/lib/deepspeed/ops/sparse_attention/trsrc
copying deepspeed/ops/sparse_attention/trsrc/softmax_bwd.tr -> build/lib/deepspeed/ops/sparse_attention/trsrc
copying deepspeed/ops/sparse_attention/trsrc/softmax_fwd.tr -> build/lib/deepspeed/ops/sparse_attention/trsrc
running build_scripts
creating build/scripts-3.7
copying and adjusting bin/deepspeed -> build/scripts-3.7
copying and adjusting bin/deepspeed.pt -> build/scripts-3.7
copying and adjusting bin/ds -> build/scripts-3.7
copying bin/ds_ssh -> build/scripts-3.7
copying and adjusting bin/ds_report -> build/scripts-3.7
copying and adjusting bin/ds_elastic -> build/scripts-3.7
changing mode of build/scripts-3.7/deepspeed from 644 to 755
changing mode of build/scripts-3.7/deepspeed.pt from 644 to 755
changing mode of build/scripts-3.7/ds from 644 to 755
changing mode of build/scripts-3.7/ds_report from 644 to 755
changing mode of build/scripts-3.7/ds_elastic from 644 to 755
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/deepspeed
creating build/bdist.linux-x86_64/wheel/deepspeed/pipe
copying build/lib/deepspeed/pipe/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/pipe
creating build/bdist.linux-x86_64/wheel/deepspeed/elasticity
copying build/lib/deepspeed/elasticity/config.py -> build/bdist.linux-x86_64/wheel/deepspeed/elasticity
copying build/lib/deepspeed/elasticity/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/elasticity
copying build/lib/deepspeed/elasticity/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/elasticity
copying build/lib/deepspeed/elasticity/elasticity.py -> build/bdist.linux-x86_64/wheel/deepspeed/elasticity
creating build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/module_inject/replace_policy.py -> build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/module_inject/inject.py -> build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/module_inject/replace_module.py -> build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/module_inject/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/module_inject/module_quantize.py -> build/bdist.linux-x86_64/wheel/deepspeed/module_inject
copying build/lib/deepspeed/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed
copying build/lib/deepspeed/git_version_info.py -> build/bdist.linux-x86_64/wheel/deepspeed
creating build/bdist.linux-x86_64/wheel/deepspeed/utils
copying build/lib/deepspeed/utils/zero_to_fp32.py -> build/bdist.linux-x86_64/wheel/deepspeed/utils
copying build/lib/deepspeed/utils/timer.py -> build/bdist.linux-x86_64/wheel/deepspeed/utils
copying build/lib/deepspeed/utils/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/utils
copying build/lib/deepspeed/utils/logging.py -> build/bdist.linux-x86_64/wheel/deepspeed/utils
copying build/lib/deepspeed/utils/distributed.py -> build/bdist.linux-x86_64/wheel/deepspeed/utils
creating build/bdist.linux-x86_64/wheel/deepspeed/launcher
copying build/lib/deepspeed/launcher/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/launcher
copying build/lib/deepspeed/launcher/launch.py -> build/bdist.linux-x86_64/wheel/deepspeed/launcher
copying build/lib/deepspeed/launcher/runner.py -> build/bdist.linux-x86_64/wheel/deepspeed/launcher
copying build/lib/deepspeed/launcher/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/launcher
copying build/lib/deepspeed/launcher/multinode_runner.py -> build/bdist.linux-x86_64/wheel/deepspeed/launcher
creating build/bdist.linux-x86_64/wheel/deepspeed/ops
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/quantizer
copying build/lib/deepspeed/ops/quantizer/quantizer.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/quantizer
copying build/lib/deepspeed/ops/quantizer/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/quantizer
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/aio
copying build/lib/deepspeed/ops/aio/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/aio
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer
copying build/lib/deepspeed/ops/transformer/transformer.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer
copying build/lib/deepspeed/ops/transformer/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer/inference
copying build/lib/deepspeed/ops/transformer/inference/transformer_inference.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer/inference
copying build/lib/deepspeed/ops/transformer/inference/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/transformer/inference
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_types.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_common.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_types.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_common.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
copying build/lib/deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/common
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/py_ds_aio.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
copying build/lib/deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/aio/py_lib
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/gelu_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/dropout_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/general_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/ds_transformer_cuda.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/softmax_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/cublas_wrappers.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
copying build/lib/deepspeed/ops/csrc/transformer/inference/csrc/dequantize.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
copying build/lib/deepspeed/ops/csrc/transformer/inference/csrc/softmax.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
copying build/lib/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
copying build/lib/deepspeed/ops/csrc/transformer/inference/csrc/normalize.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
copying build/lib/deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/csrc
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/includes
copying build/lib/deepspeed/ops/csrc/transformer/inference/includes/context.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/includes
copying build/lib/deepspeed/ops/csrc/transformer/inference/includes/cublas_wrappers.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/includes
copying build/lib/deepspeed/ops/csrc/transformer/inference/includes/custom_cuda_layers.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer/inference/includes
copying build/lib/deepspeed/ops/csrc/transformer/normalize_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
copying build/lib/deepspeed/ops/csrc/transformer/transform_kernels.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/transformer
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/quantization
copying build/lib/deepspeed/ops/csrc/quantization/pt_binding.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/quantization
copying build/lib/deepspeed/ops/csrc/quantization/quantizer.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/quantization
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/sparse_attention
copying build/lib/deepspeed/ops/csrc/sparse_attention/utils.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/sparse_attention
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/utils
copying build/lib/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/utils
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/compat.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/multi_tensor_apply.cuh -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
copying build/lib/deepspeed/ops/csrc/adam/cpu_adam.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/adam
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/context.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/feed_forward.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/type_shim.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/quantizer.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/general_kernels.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/gemm_test.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/gelu.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/ds_transformer_cuda.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/dropout.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/Timer.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/StopWatch.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/strided_batch_gemm.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/cublas_wrappers.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/cpu_adam.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/custom_cuda_layers.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/normalize_layer.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
copying build/lib/deepspeed/ops/csrc/includes/softmax.h -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/includes
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/lamb
copying build/lib/deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/lamb
copying build/lib/deepspeed/ops/csrc/lamb/fused_lamb_cuda_kernel.cu -> build/bdist.linux-x86_64/wheel/deepspeed/ops/csrc/lamb
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/matmul.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/softmax.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/sparsity_config.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/bert_sparse_self_attention.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
copying build/lib/deepspeed/ops/sparse_attention/sparse_attention_utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention/trsrc
copying build/lib/deepspeed/ops/sparse_attention/trsrc/softmax_fwd.tr -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention/trsrc
copying build/lib/deepspeed/ops/sparse_attention/trsrc/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention/trsrc
copying build/lib/deepspeed/ops/sparse_attention/trsrc/matmul.tr -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention/trsrc
copying build/lib/deepspeed/ops/sparse_attention/trsrc/softmax_bwd.tr -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention/trsrc
copying build/lib/deepspeed/ops/sparse_attention/sparse_self_attention.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/sparse_attention
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/adam
copying build/lib/deepspeed/ops/adam/fused_adam.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/adam
copying build/lib/deepspeed/ops/adam/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/adam
copying build/lib/deepspeed/ops/adam/cpu_adam.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/adam
copying build/lib/deepspeed/ops/adam/multi_tensor_apply.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/adam
copying build/lib/deepspeed/ops/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/fused_adam.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/transformer_inference.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/builder.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/fused_lamb.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/transformer.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/quantizer.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/async_io.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/cpu_adam.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/sparse_attn.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
copying build/lib/deepspeed/ops/op_builder/stochastic_transformer.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/op_builder
creating build/bdist.linux-x86_64/wheel/deepspeed/ops/lamb
copying build/lib/deepspeed/ops/lamb/fused_lamb.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/lamb
copying build/lib/deepspeed/ops/lamb/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/ops/lamb
copying build/lib/deepspeed/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/schedule.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/engine.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/module.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/topology.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
copying build/lib/deepspeed/runtime/pipe/p2p.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/pipe
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/comm
copying build/lib/deepspeed/runtime/comm/mpi.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/comm
copying build/lib/deepspeed/runtime/comm/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/comm
copying build/lib/deepspeed/runtime/comm/nccl.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/comm
copying build/lib/deepspeed/runtime/state_dict_factory.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/weight_quantizer.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/async_swapper.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/aio_config.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/swap_tensor/optimizer_utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/swap_tensor
copying build/lib/deepspeed/runtime/config.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16
copying build/lib/deepspeed/runtime/fp16/loss_scaler.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16
copying build/lib/deepspeed/runtime/fp16/fused_optimizer.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16
copying build/lib/deepspeed/runtime/fp16/unfused_optimizer.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16
copying build/lib/deepspeed/runtime/fp16/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16/onebit
copying build/lib/deepspeed/runtime/fp16/onebit/adam.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16/onebit
copying build/lib/deepspeed/runtime/fp16/onebit/lamb.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16/onebit
copying build/lib/deepspeed/runtime/fp16/onebit/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/fp16/onebit
copying build/lib/deepspeed/runtime/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/activation_checkpointing
copying build/lib/deepspeed/runtime/activation_checkpointing/config.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/activation_checkpointing
copying build/lib/deepspeed/runtime/activation_checkpointing/checkpointing.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/activation_checkpointing
copying build/lib/deepspeed/runtime/activation_checkpointing/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/activation_checkpointing
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/compression
copying build/lib/deepspeed/runtime/compression/cupy.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/compression
copying build/lib/deepspeed/runtime/compression/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/compression
copying build/lib/deepspeed/runtime/progressive_layer_drop.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/config_utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
creating build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/partition_parameters.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/utils.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/config.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/stage3.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/linear.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/offload_constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/test.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/tiling.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/contiguous_memory_allocator.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/stage1.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/stage2.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/zero/offload_config.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime/zero
copying build/lib/deepspeed/runtime/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/engine.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/lr_schedules.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/eigenvalue.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/dataloader.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/csr_tensor.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/runtime/quantize.py -> build/bdist.linux-x86_64/wheel/deepspeed/runtime
copying build/lib/deepspeed/env_report.py -> build/bdist.linux-x86_64/wheel/deepspeed
creating build/bdist.linux-x86_64/wheel/deepspeed/inference
copying build/lib/deepspeed/inference/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/inference
copying build/lib/deepspeed/inference/engine.py -> build/bdist.linux-x86_64/wheel/deepspeed/inference
copying build/lib/deepspeed/git_version_info_installed.py -> build/bdist.linux-x86_64/wheel/deepspeed
creating build/bdist.linux-x86_64/wheel/deepspeed/profiling
copying build/lib/deepspeed/profiling/config.py -> build/bdist.linux-x86_64/wheel/deepspeed/profiling
copying build/lib/deepspeed/profiling/constants.py -> build/bdist.linux-x86_64/wheel/deepspeed/profiling
copying build/lib/deepspeed/profiling/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/profiling
creating build/bdist.linux-x86_64/wheel/deepspeed/profiling/flops_profiler
copying build/lib/deepspeed/profiling/flops_profiler/profiler.py -> build/bdist.linux-x86_64/wheel/deepspeed/profiling/flops_profiler
copying build/lib/deepspeed/profiling/flops_profiler/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeed/profiling/flops_profiler
creating build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/fused_adam.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/transformer_inference.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/builder.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/utils.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/fused_lamb.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/transformer.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/quantizer.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/__init__.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/async_io.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/cpu_adam.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/sparse_attn.py -> build/bdist.linux-x86_64/wheel/op_builder
copying build/lib/op_builder/stochastic_transformer.py -> build/bdist.linux-x86_64/wheel/op_builder
running install_egg_info
Copying deepspeed.egg-info to build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data
creating build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/deepspeed -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/ds_report -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/ds_ssh -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/deepspeed.pt -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/ds_elastic -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
copying build/scripts-3.7/ds -> build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/deepspeed to 755
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/ds_report to 755
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/ds_ssh to 755
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/deepspeed.pt to 755
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/ds_elastic to 755
changing mode of build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.data/scripts/ds to 755
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.linux-x86_64/wheel/deepspeed-0.4.0+11e94e6.dist-info/WHEEL
creating 'dist/deepspeed-0.4.0+11e94e6-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'deepspeed/__init__.py'
adding 'deepspeed/constants.py'
adding 'deepspeed/env_report.py'
adding 'deepspeed/git_version_info.py'
adding 'deepspeed/git_version_info_installed.py'
adding 'deepspeed/elasticity/__init__.py'
adding 'deepspeed/elasticity/config.py'
adding 'deepspeed/elasticity/constants.py'
adding 'deepspeed/elasticity/elasticity.py'
adding 'deepspeed/inference/__init__.py'
adding 'deepspeed/inference/engine.py'
adding 'deepspeed/launcher/__init__.py'
adding 'deepspeed/launcher/constants.py'
adding 'deepspeed/launcher/launch.py'
adding 'deepspeed/launcher/multinode_runner.py'
adding 'deepspeed/launcher/runner.py'
adding 'deepspeed/module_inject/__init__.py'
adding 'deepspeed/module_inject/inject.py'
adding 'deepspeed/module_inject/module_quantize.py'
adding 'deepspeed/module_inject/replace_module.py'
adding 'deepspeed/module_inject/replace_policy.py'
adding 'deepspeed/ops/__init__.py'
adding 'deepspeed/ops/adam/__init__.py'
adding 'deepspeed/ops/adam/cpu_adam.py'
adding 'deepspeed/ops/adam/fused_adam.py'
adding 'deepspeed/ops/adam/multi_tensor_apply.py'
adding 'deepspeed/ops/aio/__init__.py'
adding 'deepspeed/ops/csrc/adam/compat.h'
adding 'deepspeed/ops/csrc/adam/cpu_adam.cpp'
adding 'deepspeed/ops/csrc/adam/custom_cuda_kernel.cu'
adding 'deepspeed/ops/csrc/adam/fused_adam_frontend.cpp'
adding 'deepspeed/ops/csrc/adam/multi_tensor_adam.cu'
adding 'deepspeed/ops/csrc/adam/multi_tensor_apply.cuh'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_common.cpp'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_common.h'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_types.cpp'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_types.h'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.cpp'
adding 'deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.h'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.cpp'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.h'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.cpp'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.h'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.cpp'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.h'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.cpp'
adding 'deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.h'
adding 'deepspeed/ops/csrc/aio/py_lib/py_ds_aio.cpp'
adding 'deepspeed/ops/csrc/includes/StopWatch.h'
adding 'deepspeed/ops/csrc/includes/Timer.h'
adding 'deepspeed/ops/csrc/includes/context.h'
adding 'deepspeed/ops/csrc/includes/cpu_adam.h'
adding 'deepspeed/ops/csrc/includes/cublas_wrappers.h'
adding 'deepspeed/ops/csrc/includes/custom_cuda_layers.h'
adding 'deepspeed/ops/csrc/includes/dropout.h'
adding 'deepspeed/ops/csrc/includes/ds_transformer_cuda.h'
adding 'deepspeed/ops/csrc/includes/feed_forward.h'
adding 'deepspeed/ops/csrc/includes/gelu.h'
adding 'deepspeed/ops/csrc/includes/gemm_test.h'
adding 'deepspeed/ops/csrc/includes/general_kernels.h'
adding 'deepspeed/ops/csrc/includes/normalize_layer.h'
adding 'deepspeed/ops/csrc/includes/quantizer.h'
adding 'deepspeed/ops/csrc/includes/softmax.h'
adding 'deepspeed/ops/csrc/includes/strided_batch_gemm.h'
adding 'deepspeed/ops/csrc/includes/type_shim.h'
adding 'deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp'
adding 'deepspeed/ops/csrc/lamb/fused_lamb_cuda_kernel.cu'
adding 'deepspeed/ops/csrc/quantization/pt_binding.cpp'
adding 'deepspeed/ops/csrc/quantization/quantizer.cu'
adding 'deepspeed/ops/csrc/sparse_attention/utils.cpp'
adding 'deepspeed/ops/csrc/transformer/cublas_wrappers.cu'
adding 'deepspeed/ops/csrc/transformer/dropout_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/ds_transformer_cuda.cpp'
adding 'deepspeed/ops/csrc/transformer/gelu_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/general_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/normalize_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/softmax_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/transform_kernels.cu'
adding 'deepspeed/ops/csrc/transformer/inference/csrc/dequantize.cu'
adding 'deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu'
adding 'deepspeed/ops/csrc/transformer/inference/csrc/normalize.cu'
adding 'deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp'
adding 'deepspeed/ops/csrc/transformer/inference/csrc/softmax.cu'
adding 'deepspeed/ops/csrc/transformer/inference/includes/context.h'
adding 'deepspeed/ops/csrc/transformer/inference/includes/cublas_wrappers.h'
adding 'deepspeed/ops/csrc/transformer/inference/includes/custom_cuda_layers.h'
adding 'deepspeed/ops/csrc/utils/flatten_unflatten.cpp'
adding 'deepspeed/ops/lamb/__init__.py'
adding 'deepspeed/ops/lamb/fused_lamb.py'
adding 'deepspeed/ops/op_builder/__init__.py'
adding 'deepspeed/ops/op_builder/async_io.py'
adding 'deepspeed/ops/op_builder/builder.py'
adding 'deepspeed/ops/op_builder/cpu_adam.py'
adding 'deepspeed/ops/op_builder/fused_adam.py'
adding 'deepspeed/ops/op_builder/fused_lamb.py'
adding 'deepspeed/ops/op_builder/quantizer.py'
adding 'deepspeed/ops/op_builder/sparse_attn.py'
adding 'deepspeed/ops/op_builder/stochastic_transformer.py'
adding 'deepspeed/ops/op_builder/transformer.py'
adding 'deepspeed/ops/op_builder/transformer_inference.py'
adding 'deepspeed/ops/op_builder/utils.py'
adding 'deepspeed/ops/quantizer/__init__.py'
adding 'deepspeed/ops/quantizer/quantizer.py'
adding 'deepspeed/ops/sparse_attention/__init__.py'
adding 'deepspeed/ops/sparse_attention/bert_sparse_self_attention.py'
adding 'deepspeed/ops/sparse_attention/matmul.py'
adding 'deepspeed/ops/sparse_attention/softmax.py'
adding 'deepspeed/ops/sparse_attention/sparse_attention_utils.py'
adding 'deepspeed/ops/sparse_attention/sparse_self_attention.py'
adding 'deepspeed/ops/sparse_attention/sparsity_config.py'
adding 'deepspeed/ops/sparse_attention/trsrc/__init__.py'
adding 'deepspeed/ops/sparse_attention/trsrc/matmul.tr'
adding 'deepspeed/ops/sparse_attention/trsrc/softmax_bwd.tr'
adding 'deepspeed/ops/sparse_attention/trsrc/softmax_fwd.tr'
adding 'deepspeed/ops/transformer/__init__.py'
adding 'deepspeed/ops/transformer/transformer.py'
adding 'deepspeed/ops/transformer/inference/__init__.py'
adding 'deepspeed/ops/transformer/inference/transformer_inference.py'
adding 'deepspeed/pipe/__init__.py'
adding 'deepspeed/profiling/__init__.py'
adding 'deepspeed/profiling/config.py'
adding 'deepspeed/profiling/constants.py'
adding 'deepspeed/profiling/flops_profiler/__init__.py'
adding 'deepspeed/profiling/flops_profiler/profiler.py'
adding 'deepspeed/runtime/__init__.py'
adding 'deepspeed/runtime/config.py'
adding 'deepspeed/runtime/config_utils.py'
adding 'deepspeed/runtime/constants.py'
adding 'deepspeed/runtime/csr_tensor.py'
adding 'deepspeed/runtime/dataloader.py'
adding 'deepspeed/runtime/eigenvalue.py'
adding 'deepspeed/runtime/engine.py'
adding 'deepspeed/runtime/lr_schedules.py'
adding 'deepspeed/runtime/progressive_layer_drop.py'
adding 'deepspeed/runtime/quantize.py'
adding 'deepspeed/runtime/state_dict_factory.py'
adding 'deepspeed/runtime/utils.py'
adding 'deepspeed/runtime/weight_quantizer.py'
adding 'deepspeed/runtime/activation_checkpointing/__init__.py'
adding 'deepspeed/runtime/activation_checkpointing/checkpointing.py'
adding 'deepspeed/runtime/activation_checkpointing/config.py'
adding 'deepspeed/runtime/comm/__init__.py'
adding 'deepspeed/runtime/comm/mpi.py'
adding 'deepspeed/runtime/comm/nccl.py'
adding 'deepspeed/runtime/compression/__init__.py'
adding 'deepspeed/runtime/compression/cupy.py'
adding 'deepspeed/runtime/fp16/__init__.py'
adding 'deepspeed/runtime/fp16/fused_optimizer.py'
adding 'deepspeed/runtime/fp16/loss_scaler.py'
adding 'deepspeed/runtime/fp16/unfused_optimizer.py'
adding 'deepspeed/runtime/fp16/onebit/__init__.py'
adding 'deepspeed/runtime/fp16/onebit/adam.py'
adding 'deepspeed/runtime/fp16/onebit/lamb.py'
adding 'deepspeed/runtime/pipe/__init__.py'
adding 'deepspeed/runtime/pipe/engine.py'
adding 'deepspeed/runtime/pipe/module.py'
adding 'deepspeed/runtime/pipe/p2p.py'
adding 'deepspeed/runtime/pipe/schedule.py'
adding 'deepspeed/runtime/pipe/topology.py'
adding 'deepspeed/runtime/swap_tensor/__init__.py'
adding 'deepspeed/runtime/swap_tensor/aio_config.py'
adding 'deepspeed/runtime/swap_tensor/async_swapper.py'
adding 'deepspeed/runtime/swap_tensor/constants.py'
adding 'deepspeed/runtime/swap_tensor/optimizer_utils.py'
adding 'deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py'
adding 'deepspeed/runtime/swap_tensor/partitioned_param_swapper.py'
adding 'deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py'
adding 'deepspeed/runtime/swap_tensor/utils.py'
adding 'deepspeed/runtime/zero/__init__.py'
adding 'deepspeed/runtime/zero/config.py'
adding 'deepspeed/runtime/zero/constants.py'
adding 'deepspeed/runtime/zero/contiguous_memory_allocator.py'
adding 'deepspeed/runtime/zero/linear.py'
adding 'deepspeed/runtime/zero/offload_config.py'
adding 'deepspeed/runtime/zero/offload_constants.py'
adding 'deepspeed/runtime/zero/partition_parameters.py'
adding 'deepspeed/runtime/zero/stage1.py'
adding 'deepspeed/runtime/zero/stage2.py'
adding 'deepspeed/runtime/zero/stage3.py'
adding 'deepspeed/runtime/zero/test.py'
adding 'deepspeed/runtime/zero/tiling.py'
adding 'deepspeed/runtime/zero/utils.py'
adding 'deepspeed/utils/__init__.py'
adding 'deepspeed/utils/distributed.py'
adding 'deepspeed/utils/logging.py'
adding 'deepspeed/utils/timer.py'
adding 'deepspeed/utils/zero_to_fp32.py'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/deepspeed'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/deepspeed.pt'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/ds'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/ds_elastic'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/ds_report'
adding 'deepspeed-0.4.0+11e94e6.data/scripts/ds_ssh'
adding 'op_builder/__init__.py'
adding 'op_builder/async_io.py'
adding 'op_builder/builder.py'
adding 'op_builder/cpu_adam.py'
adding 'op_builder/fused_adam.py'
adding 'op_builder/fused_lamb.py'
adding 'op_builder/quantizer.py'
adding 'op_builder/sparse_attn.py'
adding 'op_builder/stochastic_transformer.py'
adding 'op_builder/transformer.py'
adding 'op_builder/transformer_inference.py'
adding 'op_builder/utils.py'
adding 'deepspeed-0.4.0+11e94e6.dist-info/LICENSE'
adding 'deepspeed-0.4.0+11e94e6.dist-info/METADATA'
adding 'deepspeed-0.4.0+11e94e6.dist-info/WHEEL'
adding 'deepspeed-0.4.0+11e94e6.dist-info/entry_points.txt'
adding 'deepspeed-0.4.0+11e94e6.dist-info/top_level.txt'
adding 'deepspeed-0.4.0+11e94e6.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
deepspeed build time = 0.27840399742126465 secs
Installing deepspeed
[sudo] password for sgugger:
WARNING: Skipping deepspeed as it is not installed.
Processing ./dist/deepspeed-0.4.0+11e94e6-py3-none-any.whl
Collecting torchvision>=0.4.0
  Downloading torchvision-0.9.1-cp37-cp37m-manylinux1_x86_64.whl (17.4 MB)
     |████████████████████████████████| 17.4 MB 5.4 MB/s
Collecting psutil
  Downloading psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)
     |████████████████████████████████| 296 kB 95.4 MB/s
Collecting packaging
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 11.2 MB/s
Collecting ninja
  Downloading ninja-1.10.0.post2-py3-none-manylinux1_x86_64.whl (107 kB)
     |████████████████████████████████| 107 kB 138.5 MB/s
Collecting numpy
  Downloading numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.3 MB)
     |████████████████████████████████| 15.3 MB 124.8 MB/s
Collecting torch>=1.2
  Downloading torch-1.8.1-cp37-cp37m-manylinux1_x86_64.whl (804.1 MB)
     |████████████████████████████████| 804.1 MB 3.7 kB/s
Collecting tqdm
  Downloading tqdm-4.61.0-py2.py3-none-any.whl (75 kB)
     |████████████████████████████████| 75 kB 7.6 MB/s
Collecting tensorboardX==1.8
  Downloading tensorboardX-1.8-py2.py3-none-any.whl (216 kB)
     |████████████████████████████████| 216 kB 86.3 MB/s
Collecting pillow>=4.1.1
  Downloading Pillow-8.2.0-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 150.3 MB/s
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 12.2 MB/s
Collecting typing-extensions
  Downloading typing_extensions-<IP_ADDRESS>-py3-none-any.whl (26 kB)
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting protobuf>=3.2.0
  Downloading protobuf-3.17.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 80.5 MB/s
Installing collected packages: pillow, numpy, typing-extensions, torch, torchvision, psutil, pyparsing, packaging, ninja, tqdm, six, protobuf, tensorboardX, deepspeed
Successfully installed deepspeed-0.4.0+11e94e6 ninja-1.10.0.post2 numpy-1.20.3 packaging-20.9 pillow-8.2.0 protobuf-3.17.1 psutil-5.8.0 pyparsing-2.4.7 six-1.16.0 tensorboardX-1.8 torch-1.8.1 torchvision-0.9.1 tqdm-4.61.0 typing-extensions-<IP_ADDRESS>
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
/bin/sh: line 1: type: llvm-config-9: not found
/bin/sh: line 1: type: cmake: not found
 [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
 [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the libraries: ['libaio-dev'] but are missing. Can be fixed by: `apt install libaio-dev`.
async_io ............... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch']
torch version .................... 1.8.1+cu111
torch cuda version ............... 11.1
nvcc version ..................... 11.1
deepspeed install path ........... ['/home/sgugger/git/DeepSpeed/deepspeed']
deepspeed info ................... 0.4.0+11e94e6, 11e94e6, master
deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1

But then I get into the same problem, specifically:
E           RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue.

so looking at libraries_installed - it assumes the world to be a .deb oyster. And that's why it fails.
perhaps trying to compile a simple c program and linking it to -laio could be a more reliable platform independent test?

@stas00, great observation. This is not surprising as we have only developed on deb (Ubuntu) systems.
Perhaps an issue could be opened for archlinux support in our installation process. Unfortunately, we don't have access to such systems so that would be a problem.
In the meantime, I think your suggestion of linking a simple c program to -laio should give a good estimate of how much work is required.

@stas00, great observations. This is not surprising as we have only developed on deb (Ubuntu) systems.
Perhaps an issue could be opened for archlinux support in our installation process. Unfortunately, we don't have access to such systems so that would be a problem.

I'm not following your suggestion, isn't this ticket not it already?
I don't think we need to have access to a non-debian system. We just need to remove dependency on using dpkg - or use it first as it'd account for probably 90% (total guess) of users, and fall-back to a secondary linker test if it fails.
Also the error message wouldn't fit on non-debian systems, as it no longer will be libaio-dev - e.g. Redhat it's libaio-devel I believe. So it might be a painful path of:

detect the OS
switch(OS): do_this

yet another alternative path is to say that deepspeed supports only debian systems by default and for other systems here are the instructions on how to build libai

wget ...libaio.tar.gz
untar and cd
make install

build deepspeed.

Specifically to @sgugger we perhaps need SKIP_LIBAIO_INSTALLED_CHECK and after pacman libaio it whould just work.
@sgugger, if you do:
diff --git a/op_builder/async_io.py b/op_builder/async_io.py
index 31ab9d3..88fcb61 100644
--- a/op_builder/async_io.py
+++ b/op_builder/async_io.py
@@ -48,6 +48,7 @@ class AsyncIOBuilder(OpBuilder):
         return ['-laio']

     def is_compatible(self):
+        return
         aio_libraries = ['libaio-dev']
         aio_compatible = self.libraries_installed(aio_libraries)
         if not aio_compatible:

it should just work now.

I'm not following your suggestion, isn't this ticket not it already?

Yes, you are correct. This is good enough.
@sgugger, can you please confirm that you are unblocked.

To clarify my low-hanging poor man solution is:

to have an option to skip the check
to have the user figure out they have libaio.so somewhere in their ldconf/LD_LIBRARY_PATH
done.

@tjruwase, now it gives a weird gcc error - with a blank file name:
`    building 'deepspeed.ops.aio.async_io_op' extension
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -g -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/TH -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/THC -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/include -I/home/sgugger/.pyenv/versions/3.7.9/include/python3.7m -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.7/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
    gcc: error: : No such file or directory
    error: command 'gcc' failed with exit status 1

but -c csrc/aio/py_lib/deepspeed_py_copy.cpp is in the command line, so where did it get the blank file from? (the missing filename should be gcc: error: missing_file.c : No such file or directory

btw, @tjruwase - I see it has both -O3 and -O0 as you can see. Not sure which one takes precedence.

@stas00, I was about to respond when I saw your update. It is very odd. I am not sure -O3 and -O0 should cause a missing file error?

no, the problem was old build dir. removing it fixed the problem.
I was flagging  -O3 and -O0 not as a cause of the problem, but as something that's perhaps important in general? (hence the separate comment).

ok, success with my hack.
stochastic_transformer_op won't link, so we had to remove DS_BUILD_STOCHASTIC_TRANSFORMER=1

I was flagging -O3 and -O0 not as a cause of the problem, but as something that's perhaps important in general? (hence the separate comment).

Got it. I seem to recall that gcc would pick the last option, but that was ages ago :).

oh, wait, not done yet... will post update

OK, so gcc: error: : No such file or directory is still there. Apparently removing build dir didn't help :(
Full log:
/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/commands/install.py:230: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Using pip 21.0.1 from /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip (python 3.7)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-jq7dvwjn
Created temporary directory: /tmp/pip-req-tracker-ui59j_xi
Initialized build tracking at /tmp/pip-req-tracker-ui59j_xi
Created build tracker: /tmp/pip-req-tracker-ui59j_xi
Entered build tracker: /tmp/pip-req-tracker-ui59j_xi
Created temporary directory: /tmp/pip-install-17_4wjam
Obtaining file:///home/sgugger/git/DeepSpeed
  Added file:///home/sgugger/git/DeepSpeed to build tracker '/tmp/pip-req-tracker-ui59j_xi'
    Running setup.py (path:/home/sgugger/git/DeepSpeed/setup.py) egg_info for package from file:///home/sgugger/git/DeepSpeed
    Created temporary directory: /tmp/pip-pip-egg-info-tczkqgc8
    Running command python setup.py egg_info
    /bin/sh: line 1: type: llvm-config-9: not found
    /bin/sh: line 1: type: cmake: not found
    /bin/sh: line 1: type: llvm-config-9: not found
    /bin/sh: line 1: type: cmake: not found
    DS_BUILD_OPS=0
     [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
     [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
     [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
     [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
    Install Ops={'cpu_adam': 1, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': 1, 'transformer_inference': False, 'utils': 1, 'quantizer': False}
    version=0.4.0+11e94e6, git_hash=11e94e6, git_branch=master
    install_requires=['torch>=1.2', 'torchvision>=0.4.0', 'tqdm', 'tensorboardX==1.8', 'ninja', 'numpy', 'psutil', 'packaging']
    compatible_ops={'cpu_adam': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': False, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'transformer_inference': True, 'utils': True, 'quantizer': True}
    ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x7fef878daed0>, <setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7fef878dae90>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x7fef878e2350>]
    running egg_info
    creating /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info
    writing /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/dependency_links.txt
    writing entry points to /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/entry_points.txt
    writing requirements to /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/requires.txt
    writing top-level names to /tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.cc' under directory 'deepspeed'
    warning: no files found matching '*.tr' under directory 'csrc'
    warning: no files found matching '*.cc' under directory 'csrc'
    writing manifest file '/tmp/pip-pip-egg-info-tczkqgc8/deepspeed.egg-info/SOURCES.txt'
    deepspeed build time = 0.12819957733154297 secs
  Source in /home/sgugger/git/DeepSpeed has version 0.4.0+11e94e6, which satisfies requirement deepspeed==0.4.0+11e94e6 from file:///home/sgugger/git/DeepSpeed
  Removed deepspeed==0.4.0+11e94e6 from file:///home/sgugger/git/DeepSpeed from build tracker '/tmp/pip-req-tracker-ui59j_xi'
Requirement already satisfied: torch>=1.2 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (1.8.1+cu111)
Requirement already satisfied: torchvision>=0.4.0 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (0.9.1+cu111)
Requirement already satisfied: tqdm in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (4.49.0)
Requirement already satisfied: tensorboardX==1.8 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (1.8)
Requirement already satisfied: ninja in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (1.10.0.post2)
Requirement already satisfied: numpy in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (1.19.5)
Requirement already satisfied: psutil in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (5.7.2)
Requirement already satisfied: packaging in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from deepspeed==0.4.0+11e94e6) (20.4)
Requirement already satisfied: six in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from tensorboardX==1.8->deepspeed==0.4.0+11e94e6) (1.15.0)
Requirement already satisfied: protobuf>=3.2.0 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from tensorboardX==1.8->deepspeed==0.4.0+11e94e6) (3.13.0)
Requirement already satisfied: setuptools in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from protobuf>=3.2.0->tensorboardX==1.8->deepspeed==0.4.0+11e94e6) (47.1.0)
Requirement already satisfied: typing-extensions in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from torch>=1.2->deepspeed==0.4.0+11e94e6) (<IP_ADDRESS>)
Requirement already satisfied: pillow>=4.1.1 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from torchvision>=0.4.0->deepspeed==0.4.0+11e94e6) (7.2.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages (from packaging->deepspeed==0.4.0+11e94e6) (2.4.7)
Created temporary directory: /tmp/pip-unpack-xkyalnfm

Installing collected packages: deepspeed
  Running setup.py develop for deepspeed
    Running command /home/sgugger/.pyenv/versions/3.7.9/envs/base/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"'; __file__='"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext develop --no-deps
    /bin/sh: line 1: type: llvm-config-9: not found
    /bin/sh: line 1: type: cmake: not found
    /bin/sh: line 1: type: llvm-config-9: not found
    /bin/sh: line 1: type: cmake: not found
    DS_BUILD_OPS=0
     [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
     [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
     [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
     [WARNING]  sparse_attn requires CUDA version 10.1+, does not currently support >=11 or <10.1
    Install Ops={'cpu_adam': 1, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': 1, 'transformer_inference': False, 'utils': 1, 'quantizer': False}
    version=0.4.0+11e94e6, git_hash=11e94e6, git_branch=master
    install_requires=['torch>=1.2', 'torchvision>=0.4.0', 'tqdm', 'tensorboardX==1.8', 'ninja', 'numpy', 'psutil', 'packaging']
    compatible_ops={'cpu_adam': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': False, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'transformer_inference': True, 'utils': True, 'quantizer': True}
    ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x7f74f15810d0>, <setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7f74f17b8510>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x7f74f084aed0>]
    running build_ext
    building 'deepspeed.ops.adam.cpu_adam_op' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/csrc
    creating build/temp.linux-x86_64-3.7/csrc/adam
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Icsrc/includes -I/usr/local/cuda/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/TH -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/include -I/home/sgugger/.pyenv/versions/3.7.9/include/python3.7m -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-3.7/csrc/adam/cpu_adam.o -O3 -std=c++14 -L/usr/local/cuda/lib64 -lcudart -lcublas -g -Wno-reorder -march=native -fopenmp -D__SCALAR__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
    /usr/local/cuda/bin/nvcc -Icsrc/includes -I/usr/local/cuda/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/TH -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/include -I/home/sgugger/.pyenv/versions/3.7.9/include/python3.7m -c csrc/adam/custom_cuda_kernel.cu -o build/temp.linux-x86_64-3.7/csrc/adam/custom_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/deepspeed
    creating build/lib.linux-x86_64-3.7/deepspeed/ops
    creating build/lib.linux-x86_64-3.7/deepspeed/ops/adam
    g++ -pthread -shared -L/home/sgugger/.pyenv/versions/3.7.9/lib -L/home/sgugger/.pyenv/versions/3.7.9/lib build/temp.linux-x86_64-3.7/csrc/adam/cpu_adam.o build/temp.linux-x86_64-3.7/csrc/adam/custom_cuda_kernel.o -L/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/lib -L/usr/local/cuda/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.7/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so
    building 'deepspeed.ops.aio.async_io_op' extension
    creating build/temp.linux-x86_64-3.7/csrc/aio
    creating build/temp.linux-x86_64-3.7/csrc/aio/py_lib
    creating build/temp.linux-x86_64-3.7/csrc/aio/common
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/TH -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/torch/include/THC -I/home/sgugger/.pyenv/versions/3.7.9/envs/base/include -I/home/sgugger/.pyenv/versions/3.7.9/include/python3.7m -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.7/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
    gcc: error: : No such file or directory
    error: command 'gcc' failed with exit status 1
ERROR: Command errored out with exit status 1: /home/sgugger/.pyenv/versions/3.7.9/envs/base/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"'; __file__='"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext develop --no-deps Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 400, in run
    pycompile=options.compile,
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 88, in install_given_reqs
    pycompile=pycompile,
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 774, in install
    unpacked_source_directory=self.unpacked_source_directory,
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/operations/install/editable_legacy.py", line 51, in install_editable
    cwd=unpacked_source_directory,
  File "/home/sgugger/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 258, in call_subprocess
    raise InstallationSubprocessError(proc.returncode, command_desc)
pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: /home/sgugger/.pyenv/versions/3.7.9/envs/base/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"'; __file__='"'"'/home/sgugger/git/DeepSpeed/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext develop --no-deps Check the logs for full command output.
Removed build tracker: '/tmp/pip-req-tracker-ui59j_xi'

More info. I'm getting the same error on a totally different system:
    gcc -pthread -B /home/stas/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/stas/anaconda3/lib/python3.8/site-packages/torch/include -I/home/stas/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/stas/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/stas/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/home/stas/anaconda3/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
    gcc: error: : No such file or directory

This time someone else's Ubuntu!
Oddly enough it builds just fine on mine. Investigating.

Hmm, this is a super interesting mystery. Can you add these gcc options to get more verbose compilation -v --save-temps?

with your addition:
 gcc -pthread -B /home/stas/anaconda3/envs/py38-pt18/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC -I/home/stas/anaconda3/envs/py38-pt18/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio -v --save-temps  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    gcc: error: : No such file or directory

Note what happens if I run the command alone:
$ gcc -pthread -B /home/stas/anaconda3/envs/py38-pt18/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC -I/home/stas/anaconda3/envs/py38-pt18/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio -v --save-temps  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0

Using built-in specs.
 search starts here:
#include <...> search starts here:
 csrc/aio/py_lib
 csrc/aio/common
 /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include
 /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include
 /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH
 /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC
 /home/stas/anaconda3/envs/py38-pt18/include/python3.8
 /usr/include/c++/9
 /usr/include/x86_64-linux-gnu/c++/9
 /usr/include/c++/9/backward
 /usr/lib/gcc/x86_64-linux-gnu/9/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
COLLECT_GCC_OPTIONS='-pthread' '-B' '/home/stas/anaconda3/envs/py38-pt18/compiler_compat' '-Wsign-compare' '-D' 'NDEBUG' '-g' '-fwrapv' '-O3' '-Wstrict-prototypes' '-I' 'csrc/aio/py_lib' '-I' 'csrc/aio/common' '-I' '/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include' '-I' '/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include' '-I' '/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH' '-I' '/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC' '-I' '/home/stas/anaconda3/envs/py38-pt18/include/python3.8' '-c' '-o' 'build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o' '-g' '-Wall' '-O0' '-std=c++14' '-shared' '-fPIC' '-Wno-reorder' '-march=native' '-fopenmp' '-v' '-save-temps' '-D' 'TORCH_API_INCLUDE_EXTENSION_H' '-D' 'PYBIND11_COMPILER_TYPE=_gcc' '-D' 'PYBIND11_STDLIB=_libstdcpp' '-D' 'PYBIND11_BUILD_ABI=_cxxabi1011' '-D' 'TORCH_EXTENSION_NAME=async_io_op' '-D' '_GLIBCXX_USE_CXX11_ABI=0' '-pthread'
 /usr/lib/gcc/x86_64-linux-gnu/9/cc1plus -fpreprocessed deepspeed_py_copy.ii -march=znver2 -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -msse4a -mcx16 -msahf -mmovbe -maes -msha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mno-sgx -mbmi2 -mno-pconfig -mwbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mclwb -mno-mwaitx -mclzero -mno-pku -mrdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg -mno-avx512vpopcntdq -mno-movdiri -mno-movdir64b -mno-waitpkg -mno-cldemote -mno-ptwrite --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=znver2 -quiet -dumpbase deepspeed_py_copy.cpp -auxbase-strip build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -g -O3 -O0 -Wsign-compare -Wstrict-prototypes -Wall -Wno-reorder -std=c++14 -version -fwrapv -fPIC -fopenmp -fasynchronous-unwind-tables -fstack-protector-strong -Wformat-security -fstack-clash-protection -fcf-protection -o deepspeed_py_copy.s
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
GNU C++14 (Ubuntu 9.3.0-17ubuntu1~20.04) version 9.3.0 (x86_64-linux-gnu)
        compiled by GNU C version 9.3.0, GMP version 6.2.0, MPFR version 4.0.2, MPC version 1.1.0, isl version isl-0.22.1-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
GNU C++14 (Ubuntu 9.3.0-17ubuntu1~20.04) version 9.3.0 (x86_64-linux-gnu)
        compiled by GNU C version 9.3.0, GMP version 6.2.0, MPFR version 4.0.2, MPC version 1.1.0, isl version isl-0.22.1-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 466f818abe2f30ba03783f22bd12d815
In file included from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/cast.h:16,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/attr.h:13,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h:45,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/extension.h:6,
                 from csrc/aio/py_lib/deepspeed_py_copy.h:14,
                 from csrc/aio/py_lib/deepspeed_py_copy.cpp:8:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/detail/internals.h:269:55: error: expected ‘)’ before ‘_gcc’
  269 |     PYBIND11_STR_TYPE id(PYBIND11_INTERNALS_ID);
      |                       ~                               ^
      |                                                       )
In file included from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/attr.h:13,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h:45,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/extension.h:6,
                 from csrc/aio/py_lib/deepspeed_py_copy.h:14,
                 from csrc/aio/py_lib/deepspeed_py_copy.cpp:8:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/cast.h: In member function ‘bool pybind11::detail::type_caster_generic::try_load_foreign_module_local(pybind11::handle)’:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/cast.h:638:72: error: expected ‘,’ or ‘;’ before ‘_gcc’
  638 |         constexpr auto *local_key = PYBIND11_MODULE_LOCAL_ID;
      |                                                                        ^
In file included from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
                 from /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/extension.h:6,
                 from csrc/aio/py_lib/deepspeed_py_copy.h:14,
                 from csrc/aio/py_lib/deepspeed_py_copy.cpp:8:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h: In member function ‘void pybind11::detail::generic_type::initialize(const pybind11::detail::type_record&)’:
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h:1130:62: error: expected ‘)’ before ‘_gcc’
 1130 |             setattr(m_ptr, PYBIND11_MODULE_LOCAL_ID, capsule(tinfo));
      |                    ~                                         ^~~~~
      |

OK, i got rid of pip, doing now bare setup tools:
DS_BUILD_AIO=1 python setup.py build_ext develop
DS_BUILD_OPS=0
Install Ops={'cpu_adam': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': 1, 'transformer_inference': False, 'utils': False, 'quantizer': False}
version=0.4.0+26e3841, git_hash=26e3841, git_branch=master
install_requires=['torch>=1.2', 'torchvision>=0.4.0', 'tqdm', 'tensorboardX==1.8', 'ninja', 'numpy', 'psutil', 'packaging', '#triton==1.0']
compatible_ops={'cpu_adam': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'transformer_inference': True, 'utils': True, 'quantizer': True}
ext_modules=[<setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7f1e0f9b8ca0>]
running build_ext
building 'deepspeed.ops.aio.async_io_op' extension
gcc -pthread -B /home/stas/anaconda3/envs/py38-pt18/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC -I/home/stas/anaconda3/envs/py38-pt18/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio -v --save-temps  -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
gcc: error: : No such file or directory
Using built-in specs.
COLLECT_GCC=gcc
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
error: command 'gcc' failed with exit status 1

On my home Ubuntu I get almost identical command and it builds just fine. The difference is:
mine (works)
 -D__AVX256_ -DPYBIND11_BUILD_ABI="_cxxabi1013" -D_GLIBCXX_USE_CXX11_ABI=1

another Ubuntu (breaks):
 -DPYBIND11_BUILD_ABI="_cxxabi1011"  -D_GLIBCXX_USE_CXX11_ABI=0

everything else is identical.

I don't think -D__AVX256_ should matter. The *ABI flags on the other hand look suspicious. I am not familiar with them. I also don't know where they are coming from? Is it possible to try the failing pattern on your home Ubuntu?

Good call. if I replace -DPYBIND11_BUILD_ABI="_cxxabi1013" -D_GLIBCXX_USE_CXX11_ABI=1 with -DPYBIND11_BUILD_ABI="_cxxabi1011"  -D_GLIBCXX_USE_CXX11_ABI=0 on my box it now breaks too.
Actually, just using -DPYBIND11_BUILD_ABI="_cxxabi1011"  is enough to break it.

OK, it looks I can't just run that gcc command alone - it fails everywhere on its own. There must be something else happening around it. That's why I'm asking if there is a Makefile to follow

@stas00, I was not expecting the gcc command to work alone, I was hoping the extra flags would provide more debug information.
Unfortunately, I killed the Makefile path when we switched to jit compilation.  I will search my old branches to see if I can find a Makefile.

Is there a way to tell it to save the build files, so that one could investigate what exactly it does? I suspect it doesn't full disclose what it is doing.
Incidentally the 2 Ubuntu boxes  I'm comparing are almost identical - even the same conda setup, and the same path - just different pytorch and cuda builds. I'm going to try other builds.

Tried cuda-10 (as compared to 11.x), tried different pytorch versions including 1.7.1 - all have the same problem. Must be something in the environment.

@stas00, can you share which combinations fail/work?
-DPYBIND11_BUILD_ABI="_cxxabi1011"  -D_GLIBCXX_USE_CXX11_ABI=0
-DPYBIND11_BUILD_ABI="_cxxabi1011"  -D_GLIBCXX_USE_CXX11_ABI=1
-DPYBIND11_BUILD_ABI="_cxxabi1013"  -D_GLIBCXX_USE_CXX11_ABI=0
-DPYBIND11_BUILD_ABI="_cxxabi1013"  -D_GLIBCXX_USE_CXX11_ABI=1

If I try to copy-n-paste just the gcc command it doesn't work with any of these combinations on any of the setups. If I remove all of the -Ds that get appended at the end it works on all setups. i.e. this completes w/o any errors:
gcc -pthread -B /home/stas/anaconda3/envs/py38-pt18/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/TH -I/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/THC -I/home/stas/anaconda3/envs/py38-pt18/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -laio

Trying to figure out where those defines come from

That was a hard one. But this one got solved too: https://github.com/microsoft/DeepSpeed/pull/1129

So to bring this to a closure, I was able to pre-build deespeed on this Archlinux box, but can't run tests as it crashes the box due to AMD Ryzen 9 deepspeed issue https://github.com/microsoft/DeepSpeed/issues/925#issuecomment-853984722
So this compatility checking can be approached in various ways.

create a map of distro-to-packaging system - hopefully someone has already done the ground work as it'd be a pain to develop from scratch.

ubuntu: dpkg
arch: pacman
etc

so the first step is to detect the distro, e.g. via https://github.com/nir0s/distro
python -c "import distro; print(distro.linux_distribution(full_distribution_name=False))"
('ubuntu', '20.04', 'focal')
python -c "import distro; print(distro.linux_distribution(full_distribution_name=False))"
('manjaro', '21.0.5', 'Ornara')

as you can see we already have a problem here, since manjaro is built upon Archlinux, so it's not even reporting arch
There are some 20 distros it can detect: https://distro.readthedocs.io/en/latest/#distro.id

probably a more pragmatic approach is to simply try which of the dpkg/pacman/rpm utils is available and derive the action based on that. The problem here might be to recommend the right package if it's not installed. But perhaps this simple map would be sufficient (expanding it as needed):

dpkg: libaio-dev
pacman: libaio
rpm: libaio-devel

To find if the executable exists in PATH:
python -c "import distutils.spawn; print(distutils.spawn.find_executable('pacman'))"
/usr/bin/pacman

So something like:
import distutils.spawn
import subprocess

libs = dict(
    dpkg=["-l", "libaio-dev"],
    pacman=["-Q", "libaio"],
    rpm=["-q", "libaio-devel"],
)

found = False
for pkgmgr, data in libs.items():
    flag, lib = data
    path = distutils.spawn.find_executable(pkgmgr)
    if path is not None:
        cmd = f"{pkgmgr} {flag} {lib}"
        print("Checking", cmd)
        # do the actual check
        result = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
        if result.wait() == 0:
            print("Success")
            found = True
            break

OK, so we figured out the system crashing issue - was low RAM.
I verified that with my hack patch from  https://github.com/microsoft/DeepSpeed/issues/1126#issuecomment-853271755 the nvme support works on archlinux with pacman -Q libaio
So @tjruwase we just need to complete this Issue by probably merging a variation of my proposed code at https://github.com/microsoft/DeepSpeed/issues/1126#issuecomment-854101593