MPI Simulation fails when the --multisend flag is enabled

Describe the issue
I am trying to enable the multisend option of spike exchanging.
However, when I tried to run the simulation as :
mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/10000Sim/RoundRobin-core-8 --mpi --gpu --multisend

My program failed, while the command without "--multisend" could run smoothly.
I am not sure if there is something wrong with my environment or some option that I failed to enable, or the code had some bug.
To Reproduce, and the corresponding Logs
Steps to reproduce the behavior:

If I don't enable the "--multisend" option, the program runs smoothly, and the "normal" log is as below:

$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/10000Sim/RoundRobin-core-8 --mpi --gpu
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[36744,1],7]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
[dgxone:76177] [[36744,0],0] ORTE_ERROR_LOG: Data unpack had inadequate space in file ../../orte/util/show_help.c at line 513
 num_mpi=8
 num_omp_thread=1

 Info : 8 GPUs shared by 8 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.2031, Min 215.6484, Avg 215.9121
 Memory (MBs) :            After MPI_Init : Max 216.2031, Min 215.6484, Avg 215.9395
 Memory (MBs) :          Before nrn_setup : Max 217.0234, Min 216.5938, Avg 216.8501
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
[dgxone:76177] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:76177] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:76177] 6 more processes have sent help message help-btl-vader.txt / cma-permission-denied
 Setup Done   : 8.70 seconds
 Memory (MBs) :          After nrn_setup  : Max 1550.6797, Min 1519.4180, Avg 1538.5410
GENERAL PARAMETERS
--mpi=true
--gpu=true
--dt=0.025
--tstop=1000

GPU
--nwarp=0
--cell-permute=0

INPUT PARAMETERS
--voltage=-65
--seed=-1
--datpath=./networks/10000Sim/RoundRobin-core-8
--filesdat=files.dat
--pattern=
--report-conf=
--restore=

PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=false

SPIKE EXCHANGE
--ms_phases=2
--ms_subintervals=2
--multisend=false
--spk_compress=0
--binqueue=false

CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=35
--mindelay=1.00875
--report-buffer-size=4

OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 1550.6797, Min 1519.4180, Avg 1538.5410
 Memory (MBs) :     After nrn_finitialize : Max 1551.0352, Min 1519.8086, Avg 1538.8823

 psolve |========================================================| t: 1000.00 ETA: 0h02m44s

Solver Time : 164.163

 Simulation Statistics
 Number of cells: 10000
 Number of compartments: 2494952
 Number of presyns: 4689782
 Number of input presyns: 69968
 Number of synapses: 11968052
 Number of point processes: 16710070
 Number of transfer (gap) targets: 0
 Number of spikes: 220340
 Number of spikes with non negative gid-s: 220340

However, if the "--multisend" flag is appended, the program fails quickly with the log below:
$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/10000Sim/RoundRobin-core-8 --mpi --gpu --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[35160,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
 num_mpi=8
 num_omp_thread=1

 Info : 8 GPUs shared by 8 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.1602, Min 215.7266, Avg 215.9482
 Memory (MBs) :            After MPI_Init : Max 216.1602, Min 215.7266, Avg 215.9771
 Memory (MBs) :          Before nrn_setup : Max 217.0703, Min 216.7070, Avg 216.8882
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
all2allv_int gidin to intermediate space=17523 total=371.438 time=0.000624772
all2allv_int gidout space=2532 total=371.438 time=0.000354822
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[35160,1],2]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:12096] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:12096] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:12096] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

Expected behavior
As the non-multisend command runs well, I think there is nothing wrong with my environment.
However, when the multisend flag is appended, the program fails, which is hard to understand or debug.
System (please complete the following information)

OS: Ubuntu 20.04
Compiler: PGI 20.7
Version: master branch
Backend: CPU, GPU

Does the simulation work with --multisend but without --gpu?

Does the simulation work with --multisend but without --gpu?

@nrnhines No, the simulation fails with --multisend but without --gpu. The log is quiet similar:
$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/10000Sim/RoundRobin-core-8 --mpi --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[38544,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.0508, Min 215.5312, Avg 215.7192
 Memory (MBs) :            After MPI_Init : Max 216.0625, Min 215.5312, Avg 215.8867
 Memory (MBs) :          Before nrn_setup : Max 217.3984, Min 216.7070, Avg 217.1641
all2allv_int gidin to intermediate space=17523 total=371.051 time=0.000610346
all2allv_int gidout space=2532 total=371.402 time=0.00050868
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[38544,1],6]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:12424] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:12424] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:12424] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

@nrnhines No, the simulation fails with --multisend but without --gpu. The log is quiet similar:
$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/10000Sim/RoundRobin-core-8 --mpi --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[38544,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.0508, Min 215.5312, Avg 215.7192
 Memory (MBs) :            After MPI_Init : Max 216.0625, Min 215.5312, Avg 215.8867
 Memory (MBs) :          Before nrn_setup : Max 217.3984, Min 216.7070, Avg 217.1641
all2allv_int gidin to intermediate space=17523 total=371.051 time=0.000610346
all2allv_int gidout space=2532 total=371.402 time=0.00050868
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[38544,1],6]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:12424] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:12424] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:12424] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

@HolyLow : is your model or smaller test example available somewhere that we can use to reproduce the issue ?

@HolyLow : is your model or smaller test example available somewhere that we can use to reproduce the issue ?

@pramodk All the networks I use are generated by Snudda and exported with the Neuron's nrnbbcore_write API.
To meet your advice, I tested with a smaller network generated by Snudda called tinySim which consisted of 100 neurons.
When I tested the same network with -np=8, the program failed:
$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/tinySim/RoundRobin-core-8 --mpi --gpu --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[33977,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Info : 8 GPUs shared by 8 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.1992, Min 215.5781, Avg 215.8984
 Memory (MBs) :            After MPI_Init : Max 216.2734, Min 215.5781, Avg 215.9272
 Memory (MBs) :          Before nrn_setup : Max 217.0977, Min 216.5352, Avg 216.8506
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
all2allv_int gidin to intermediate space=192 total=218.223 time=0.000168855
all2allv_int gidout space=58 total=218.223 time=7.7529e-05
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[33977,1],0]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:08865] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:08865] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:08865] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

However, when I tested the small network with -np=2, the program succeeded:
$ mpiexec -np 2 ./x86_64/special-core --tstop 1000 --datpath ./networks/tinySim/Rou
ndRobin-core-2 --mpi --gpu --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[53236,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=2
 num_omp_thread=1

 Info : 8 GPUs shared by 2 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 215.1992, Min 215.0039, Avg 215.1016
 Memory (MBs) :            After MPI_Init : Max 215.1992, Min 215.0039, Avg 215.1016
 Memory (MBs) :          Before nrn_setup : Max 216.1953, Min 216.0273, Avg 216.1113
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
all2allv_int gidin to intermediate space=108 total=220.668 time=1.94621e-05
all2allv_int gidout space=108 total=220.668 time=1.5568e-05
all2allv_int lists space=408 total=220.668 time=1.94809e-05
 Setup Done   : 0.31 seconds
 Memory (MBs) :          After nrn_setup  : Max 266.2656, Min 262.9492, Avg 264.6074
GENERAL PARAMETERS
--mpi=true
--gpu=true
--dt=0.025
--tstop=1000

GPU
--nwarp=0
--cell-permute=0

INPUT PARAMETERS
--voltage=-65
--seed=-1
--datpath=./networks/tinySim/RoundRobin-core-2
--filesdat=files.dat
--pattern=
--report-conf=
--restore=

PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=false

SPIKE EXCHANGE
--ms_phases=2
--ms_subintervals=2
--multisend=true
--spk_compress=0
--binqueue=false

CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=35
--mindelay=1.01375
--report-buffer-size=4

OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 266.2656, Min 262.9492, Avg 264.6074
 Memory (MBs) :     After nrn_finitialize : Max 266.5938, Min 263.4727, Avg 265.0332

[dgxone:27116] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics34s
[dgxone:27116] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:27116] 1 more process has sent help message help-btl-vader.txt / cma-permission-denied
 psolve |========================================================| t: 1000.00 ETA: 0h09m55s

Solver Time : 595.167

 Simulation Statistics
 Number of cells: 100
 Number of compartments: 25006
 Number of presyns: 47647
 Number of input presyns: 100
 Number of synapses: 55873
 Number of point processes: 103510
 Number of transfer (gap) targets: 0
 Number of spikes: 4162
 Number of spikes with non negative gid-s: 4162

I did more experiments, and it turned out that regardless of the network size, np=2 or np=4 would work fine, but np=8 would fail.
With the logs above, I found that the problem might lay in the all2allv_int initialization "all2allv_int  lists space=408 total=220.668 time=1.94809e-05“.
But I am not able to find out anything more, as I am not familiar with the code.

@pramodk All the networks I use are generated by Snudda and exported with the Neuron's nrnbbcore_write API.
To meet your advice, I tested with a smaller network generated by Snudda called tinySim which consisted of 100 neurons.
When I tested the same network with -np=8, the program failed:
$ mpiexec -np 8 ./x86_64/special-core --tstop 1000 --datpath ./networks/tinySim/RoundRobin-core-8 --mpi --gpu --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[33977,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Info : 8 GPUs shared by 8 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 216.1992, Min 215.5781, Avg 215.8984
 Memory (MBs) :            After MPI_Init : Max 216.2734, Min 215.5781, Avg 215.9272
 Memory (MBs) :          Before nrn_setup : Max 217.0977, Min 216.5352, Avg 216.8506
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
all2allv_int gidin to intermediate space=192 total=218.223 time=0.000168855
all2allv_int gidout space=58 total=218.223 time=7.7529e-05
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[33977,1],0]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:08865] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:08865] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:08865] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

However, when I tested the small network with -np=2, the program succeeded:
$ mpiexec -np 2 ./x86_64/special-core --tstop 1000 --datpath ./networks/tinySim/Rou
ndRobin-core-2 --mpi --gpu --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[53236,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=2
 num_omp_thread=1

 Info : 8 GPUs shared by 2 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 215.1992, Min 215.0039, Avg 215.1016
 Memory (MBs) :            After MPI_Init : Max 215.1992, Min 215.0039, Avg 215.1016
 Memory (MBs) :          Before nrn_setup : Max 216.1953, Min 216.0273, Avg 216.1113
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
all2allv_int gidin to intermediate space=108 total=220.668 time=1.94621e-05
all2allv_int gidout space=108 total=220.668 time=1.5568e-05
all2allv_int lists space=408 total=220.668 time=1.94809e-05
 Setup Done   : 0.31 seconds
 Memory (MBs) :          After nrn_setup  : Max 266.2656, Min 262.9492, Avg 264.6074
GENERAL PARAMETERS
--mpi=true
--gpu=true
--dt=0.025
--tstop=1000

GPU
--nwarp=0
--cell-permute=0

INPUT PARAMETERS
--voltage=-65
--seed=-1
--datpath=./networks/tinySim/RoundRobin-core-2
--filesdat=files.dat
--pattern=
--report-conf=
--restore=

PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=false

SPIKE EXCHANGE
--ms_phases=2
--ms_subintervals=2
--multisend=true
--spk_compress=0
--binqueue=false

CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=35
--mindelay=1.01375
--report-buffer-size=4

OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 266.2656, Min 262.9492, Avg 264.6074
 Memory (MBs) :     After nrn_finitialize : Max 266.5938, Min 263.4727, Avg 265.0332

[dgxone:27116] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics34s
[dgxone:27116] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:27116] 1 more process has sent help message help-btl-vader.txt / cma-permission-denied
 psolve |========================================================| t: 1000.00 ETA: 0h09m55s

Solver Time : 595.167

 Simulation Statistics
 Number of cells: 100
 Number of compartments: 25006
 Number of presyns: 47647
 Number of input presyns: 100
 Number of synapses: 55873
 Number of point processes: 103510
 Number of transfer (gap) targets: 0
 Number of spikes: 4162
 Number of spikes with non negative gid-s: 4162

I did more experiments, and it turned out that regardless of the network size, np=2 or np=4 would work fine, but np=8 would fail.
With the logs above, I found that the problem might lay in the all2allv_int initialization "all2allv_int  lists space=408 total=220.668 time=1.94809e-05“.
But I am not able to find out anything more, as I am not familiar with the code.

Ok thanks!
i am not familiar with Snudda, will take a look. But if you have commands or scripts already to export such model, you can provide here. (Just would be bit easy to reproduce)

Ok thanks!
i am not familiar with Snudda, will take a look. But if you have commands or scripts already to export such model, you can provide here. (Just would be bit easy to reproduce)

@pramodk The original Snudda doesn't have the functionality of exporting network to CoreNeuron, and my modified version is too dirty to share.
I upload my generated tinySim data so you could download it.

@pramodk The original Snudda doesn't have the functionality of exporting network to CoreNeuron, and my modified version is too dirty to share.
I upload my generated tinySim data so you could download it.

if the "--multisend" flag is appended, the program fails quickly

Just a datapoint as my desktop takes 20s to simulate 20ms on 8 cores when configures with -DCMAKE_BUILD_TYPE=Debug.
Without a --gpu, I experience
hines@hines-T7500:~/Downloads/tinySim$ mpiexec -np 8 ./x86_64/special-core --tstop 20 --datpath ./original-RoundRobin-core-8 --mpi --multisend`

runs successfully, generating 4 spikes
hines@hines-T7500:~/Downloads/tinySim$ cat out.dat
2.8	52
5.6	27
7.425	66
13.675	47

And I see the output
...
all2allv_int gidin to intermediate space=192 total=15.4336 time=0.000116663
all2allv_int gidout space=58 total=15.4336 time=0.000155916
all2allv_int lists space=394 total=15.4336 time=4.424e-05
 Setup Done   : 0.21 seconds
...

If you are experiencing an error under these conditions, then it seems that we will need to reproduce more accurately your hardware/software environment.

if the "--multisend" flag is appended, the program fails quickly

Just a datapoint as my desktop takes 20s to simulate 20ms on 8 cores when configures with -DCMAKE_BUILD_TYPE=Debug.
Without a --gpu, I experience
hines@hines-T7500:~/Downloads/tinySim$ mpiexec -np 8 ./x86_64/special-core --tstop 20 --datpath ./original-RoundRobin-core-8 --mpi --multisend`

runs successfully, generating 4 spikes
hines@hines-T7500:~/Downloads/tinySim$ cat out.dat
2.8	52
5.6	27
7.425	66
13.675	47

And I see the output
...
all2allv_int gidin to intermediate space=192 total=15.4336 time=0.000116663
all2allv_int gidout space=58 total=15.4336 time=0.000155916
all2allv_int lists space=394 total=15.4336 time=4.424e-05
 Setup Done   : 0.21 seconds
...

If you are experiencing an error under these conditions, then it seems that we will need to reproduce more accurately your hardware/software environment.

@nrnhines I tried your datapoint, and without --multisend it worked fine on my environment as well. The outputs are also identical to what you've shown.
But if I add the --multisend flag, the program fails. Logs are as below:
$ mpiexec -np 8 ./x86_64/special-core --tstop 20 --datpath ./networks/tinySim/RoundRobin-core-8 --mpi --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[16699,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 215.6602, Min 215.3125, Avg 215.5264
 Memory (MBs) :            After MPI_Init : Max 215.9609, Min 215.3125, Avg 215.6260
 Memory (MBs) :          Before nrn_setup : Max 216.6797, Min 216.3984, Avg 216.5669
all2allv_int gidin to intermediate space=192 total=217.977 time=0.000178425
all2allv_int gidout space=58 total=217.977 time=7.27859e-05
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[16699,1],5]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:59171] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:59171] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:59171] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

@nrnhines I tried your datapoint, and without --multisend it worked fine on my environment as well. The outputs are also identical to what you've shown.
But if I add the --multisend flag, the program fails. Logs are as below:
$ mpiexec -np 8 ./x86_64/special-core --tstop 20 --datpath ./networks/tinySim/RoundRobin-core-8 --mpi --multisend
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
--------------------------------------------------------------------------
[[16699,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: dgxone

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: dgxone
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 c40b39f (2020-12-30 09:29:10 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 215.6602, Min 215.3125, Avg 215.5264
 Memory (MBs) :            After MPI_Init : Max 215.9609, Min 215.3125, Avg 215.6260
 Memory (MBs) :          Before nrn_setup : Max 216.6797, Min 216.3984, Avg 216.5669
all2allv_int gidin to intermediate space=192 total=217.977 time=0.000178425
all2allv_int gidout space=58 total=217.977 time=7.27859e-05
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[16699,1],5]
  Exit code:    1
--------------------------------------------------------------------------
[dgxone:59171] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[dgxone:59171] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[dgxone:59171] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied

@nrnhines @pramodk Hello, I tried to dig into the source code, and the error seemed to come from the use_phase2_ procedure in nrnmultisend_setup.cpp.
But I could hardly understand what the use_phase2_ was doing...
When I disabled the use_phase2_ with "--ms-phases 1" option in the cmd, things worked fine again.
I was wondering if the use_phase2_ related code was buggy.

@nrnhines @pramodk Hello, I tried to dig into the source code, and the error seemed to come from the use_phase2_ procedure in nrnmultisend_setup.cpp.
But I could hardly understand what the use_phase2_ was doing...
When I disabled the use_phase2_ with "--ms-phases 1" option in the cmd, things worked fine again.
I was wondering if the use_phase2_ related code was buggy.

@HolyLow : thanks for an update.
Just FYI : I haven't used multi-send option often. This helps for spike exchange in specific scenario. Are you testing this option for any specific use case?
I didn't get time to look into details yet. Will try to debug with your provided dataset during this week.

@HolyLow : thanks for an update.
Just FYI : I haven't used multi-send option often. This helps for spike exchange in specific scenario. Are you testing this option for any specific use case?
I didn't get time to look into details yet. Will try to debug with your provided dataset during this week.

@pramodk Thanks for your attention.
I was just trying to compare the behavior and performance when --multisend option was (or not) enabled.
And I ran into the crash accidentally.
When digging into the code, I found another interesting performance issue, maybe not related to this issue.
When I dig into deliver_net_events related node, I notice that the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) seems to be executed on CPU side, which causes update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) to transfer data from CPU to GPU.
Is there any chance to put the procedure on GPU side to eliminate the redundant data transfer?
It seems that mod2c generates openacc code for _net_buffer_receive function, but _net_receive function is never parallelized by openacc.
I want to know how to put the _net_receive procedure on GPU side, if possible.

@pramodk Thanks for your attention.
I was just trying to compare the behavior and performance when --multisend option was (or not) enabled.
And I ran into the crash accidentally.
When digging into the code, I found another interesting performance issue, maybe not related to this issue.
When I dig into deliver_net_events related node, I notice that the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) seems to be executed on CPU side, which causes update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) to transfer data from CPU to GPU.
Is there any chance to put the procedure on GPU side to eliminate the redundant data transfer?
It seems that mod2c generates openacc code for _net_buffer_receive function, but _net_receive function is never parallelized by openacc.
I want to know how to put the _net_receive procedure on GPU side, if possible.

Perhaps this could be improved. I don't know.  The multisend method is an interprocessor spike exchange method, which, under some circumstances, has better performance than MPI_Allgather (followed by, if there are more spikes than can fit into the allgather source buffer, an MPI_Allgatherv). These spike exchange methods are not directly involved in delivery. When spikes arrive at the cpu, those spikes are placed in an event queue to await delivery. Interprocessor spike exchange and enqueuing/dequeuing are done on the cpu. At the time of dequeuing, the spike is place in a buffer specific to a mod file and at each time step that buffer is copied to the gpu. The gpu then calls the proper NET_RECEIVE block instance for the spike.
I didn't know if it was possible to efficiently manage a priority queue on the GPU.

Perhaps this could be improved. I don't know.  The multisend method is an interprocessor spike exchange method, which, under some circumstances, has better performance than MPI_Allgather (followed by, if there are more spikes than can fit into the allgather source buffer, an MPI_Allgatherv). These spike exchange methods are not directly involved in delivery. When spikes arrive at the cpu, those spikes are placed in an event queue to await delivery. Interprocessor spike exchange and enqueuing/dequeuing are done on the cpu. At the time of dequeuing, the spike is place in a buffer specific to a mod file and at each time step that buffer is copied to the gpu. The gpu then calls the proper NET_RECEIVE block instance for the spike.
I didn't know if it was possible to efficiently manage a priority queue on the GPU.

@nrnhines Thanks for your explanation!

At the time of dequeuing, the spike is place in a buffer specific to a mod file and at each time step that buffer is copied to the gpu.
Do you mean that the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) only places the spike in a buffer specific to a mod file, and if we directly change the buffer on the gpu, the update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) which transfers data from CPU to GPU can be eliminated?

I didn't know if it was possible to efficiently manage a priority queue on the GPU.
I think if the deliver procedure is to be moved to the GPU side, then maybe we won't choose to use a priority queue. Possibly we can record all the spikes according to their arrival times, and check if the fired spikes can trigger some netcon at each dt.

@nrnhines Thanks for your explanation!

At the time of dequeuing, the spike is place in a buffer specific to a mod file and at each time step that buffer is copied to the gpu.
Do you mean that the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) only places the spike in a buffer specific to a mod file, and if we directly change the buffer on the gpu, the update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) which transfers data from CPU to GPU can be eliminated?

I didn't know if it was possible to efficiently manage a priority queue on the GPU.
I think if the deliver procedure is to be moved to the GPU side, then maybe we won't choose to use a priority queue. Possibly we can record all the spikes according to their arrival times, and check if the fired spikes can trigger some netcon at each dt.

Do you mean

Yes. I would only disambiguate, or stress a bit more, that on dequeuing on the cpu from the destination cpu priority queue that the cpu copies the spike to a modfile specific buffer on the cpu. That buffer is copied every time step to the gpu. The gpu delivers the contents of the buffer to  the NET_RECEIVE block instances every time step.  The presumption was that cpu->gpu transfer is better performance as an array than as one at a time.  If that presumption is vitiated by the performance improvement of eliminating the

update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) which transfers data from CPU to GPU can be eliminated

Then it makes sense to do so.

Possibly we can record all the spikes according to their arrival times

I'm not sure I know exactly what you mean. There is a spike generation time ts, an arrival time on the destination node, and a delivery time, ts + NetCon.delay. The latter is what goes onto the priority queue and when that time occurs the spike is sent to the (buffered) destination for immediate (that time step) delivery.

check if the fired spikes can trigger some netcon at each dt.

That seems to imply some sort of queue or  insertable, removable list on the gpu which is either global, per mod type, or per mod instance.  generation order, is not correlated with arrival on node order, which is not correlated with delivery order. (latter due to different NetCon.delay) (although in practice delivery is in the same order as generation due to globally constant NetCon.delay).

Do you mean

Yes. I would only disambiguate, or stress a bit more, that on dequeuing on the cpu from the destination cpu priority queue that the cpu copies the spike to a modfile specific buffer on the cpu. That buffer is copied every time step to the gpu. The gpu delivers the contents of the buffer to  the NET_RECEIVE block instances every time step.  The presumption was that cpu->gpu transfer is better performance as an array than as one at a time.  If that presumption is vitiated by the performance improvement of eliminating the

update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) which transfers data from CPU to GPU can be eliminated

Then it makes sense to do so.

Possibly we can record all the spikes according to their arrival times

I'm not sure I know exactly what you mean. There is a spike generation time ts, an arrival time on the destination node, and a delivery time, ts + NetCon.delay. The latter is what goes onto the priority queue and when that time occurs the spike is sent to the (buffered) destination for immediate (that time step) delivery.

check if the fired spikes can trigger some netcon at each dt.

That seems to imply some sort of queue or  insertable, removable list on the gpu which is either global, per mod type, or per mod instance.  generation order, is not correlated with arrival on node order, which is not correlated with delivery order. (latter due to different NetCon.delay) (although in practice delivery is in the same order as generation due to globally constant NetCon.delay).

I really appreciate your patience!! I have some more questions:

the cpu copies the spike to a modfile specific buffer on the cpu.

Do you refer to the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) ? I failed to find out the data structure of the "modfile specific buffer".

That buffer is copied every time step to the gpu.

Is the copy carried out in update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) ? As I am not able to find the modfile specific buffer on CPU, I failed to understand this procedure as well...

The gpu delivers the contents of the buffer to the NET_RECEIVE block instances every time step.

Where does this procedure happen? In update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) ?

I'm not sure I know exactly what you mean. There is a spike generation time ts, an arrival time on the destination node, and a delivery time, ts + NetCon.delay.

I mean the spike generation time ts, which is transferred in MPI_Spike structure. I think maybe we could record all the spikes in a table according to their generation time ts, and then we check the table each time we want to "pop" netcons. We could directly check all the spike's netcons, calculate spike.ts + netcon.delay to see if the netcon should be activated. As the netcons have a maximum delay, the spikes would be outdated after the maximum delay and can be safely discarded, which keeps the number of tracked spikes small enough. A round-robin queue might be just ok.
Actually, I have another question for the existing priority queue. It seems that all kinds of DiscreteEvents are inserted into the queue, such as NetCons, SelfEvents, ConditionEvents, NetParEvents, etc. Currently I only know that in deliver_net_events related node, the netcons would be inserted into the queue. But what about other kinds of events, including SelfEvents, ConditionEvents and NetParEvents? Where are they produced and inserted into the priority queue?

I really appreciate your patience!! I have some more questions:

the cpu copies the spike to a modfile specific buffer on the cpu.

Do you refer to the function (*corenrn.get_pnt_receive()[typ])(target_, u.weight_index_, 0) (called in NetCon::deliver) ? I failed to find out the data structure of the "modfile specific buffer".

That buffer is copied every time step to the gpu.

Is the copy carried out in update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) ? As I am not able to find the modfile specific buffer on CPU, I failed to understand this procedure as well...

The gpu delivers the contents of the buffer to the NET_RECEIVE block instances every time step.

Where does this procedure happen? In update_net_receive_buffer(nt) (called in NetCvode::deliver_net_events) ?

I'm not sure I know exactly what you mean. There is a spike generation time ts, an arrival time on the destination node, and a delivery time, ts + NetCon.delay.

I mean the spike generation time ts, which is transferred in MPI_Spike structure. I think maybe we could record all the spikes in a table according to their generation time ts, and then we check the table each time we want to "pop" netcons. We could directly check all the spike's netcons, calculate spike.ts + netcon.delay to see if the netcon should be activated. As the netcons have a maximum delay, the spikes would be outdated after the maximum delay and can be safely discarded, which keeps the number of tracked spikes small enough. A round-robin queue might be just ok.
Actually, I have another question for the existing priority queue. It seems that all kinds of DiscreteEvents are inserted into the queue, such as NetCons, SelfEvents, ConditionEvents, NetParEvents, etc. Currently I only know that in deliver_net_events related node, the netcons would be inserted into the queue. But what about other kinds of events, including SelfEvents, ConditionEvents and NetParEvents? Where are they produced and inserted into the priority queue?

the cpu copies the spike to a modfile specific buffer on the cpu.
That buffer is copied every time step to the gpu.
The gpu delivers the contents of the buffer to the NET_RECEIVE block instances every time step.

void NetCvode::deliver_net_events(NrnThread* nt) {  // for default method
    ...
    deliver_events(tm, nt);   Though need to deal with interthread events, the principle call here is for each event to call
                                           deliver_event which calls (for our purposes) NetCon.deliver.
                                           That calls (*corenrn.get_pnt_receive()[typ]) which is in the the nmodl translated cxx file with the function
                                            name (e.g. in build/x86_64/corenrn/mod2c/expsyn.cpp) _net_buf_receive which appends the
                                            spike to the membrane_list specific _net_receive_buffer (on the cpu)
    ...
    /*before executing on gpu, we have to update the NetReceiveBuffer_t on GPU */
    update_net_receive_buffer(nt);

    for (auto& net_buf_receive : corenrn.get_net_buf_receive()) {
        (*net_buf_receive.first)(nt);  This calls the  _net_buf_receive function in the proper cxx translated mod file where,
                                                     on the GPU, the events in the _ml->_net_receive_buffer are looped over and
                                                     net_receive_kernel is called for each of then.

I think maybe we could record all the spikes in a table according to their generation time ts, and then we check the table each time we want to "pop" netcons.

That is very similar to the binqueue for NetCon events except td is in the binqueue so that
spike.ts+netcon.delay is calculated only once.

what about other kinds of events

They are never interprocessor or interthread events. They come from a mod file instance and get sent back to the same instance. See netcvode.cpp::  void net_send(...

the cpu copies the spike to a modfile specific buffer on the cpu.
That buffer is copied every time step to the gpu.
The gpu delivers the contents of the buffer to the NET_RECEIVE block instances every time step.

void NetCvode::deliver_net_events(NrnThread* nt) {  // for default method
    ...
    deliver_events(tm, nt);   Though need to deal with interthread events, the principle call here is for each event to call
                                           deliver_event which calls (for our purposes) NetCon.deliver.
                                           That calls (*corenrn.get_pnt_receive()[typ]) which is in the the nmodl translated cxx file with the function
                                            name (e.g. in build/x86_64/corenrn/mod2c/expsyn.cpp) _net_buf_receive which appends the
                                            spike to the membrane_list specific _net_receive_buffer (on the cpu)
    ...
    /*before executing on gpu, we have to update the NetReceiveBuffer_t on GPU */
    update_net_receive_buffer(nt);

    for (auto& net_buf_receive : corenrn.get_net_buf_receive()) {
        (*net_buf_receive.first)(nt);  This calls the  _net_buf_receive function in the proper cxx translated mod file where,
                                                     on the GPU, the events in the _ml->_net_receive_buffer are looped over and
                                                     net_receive_kernel is called for each of then.

I think maybe we could record all the spikes in a table according to their generation time ts, and then we check the table each time we want to "pop" netcons.

That is very similar to the binqueue for NetCon events except td is in the binqueue so that
spike.ts+netcon.delay is calculated only once.

what about other kinds of events

They are never interprocessor or interthread events. They come from a mod file instance and get sent back to the same instance. See netcvode.cpp::  void net_send(...

@nrnhines

That is very similar to the binqueue for NetCon events except td is in the binqueue so that spike.ts+netcon.delay is calculated only once.

Wow, I haven't read the code about binqueue, and I think this query method would be good for GPU parallelization rather than CPU single thread. So if the query method is moved to GPU, the performance should increase greatly.

They are never interprocessor or interthread events. They come from a mod file instance and get sent back to the same instance. See netcvode.cpp:: void net_send(...

Do you mean that in the priority queue there are only netcons, and other events such as SelfEvents, ConditionEvents, and NetParEvents won't appear in the queue?
Besides, if I want to let the translated cxx file's _net_buf_receive directly modifies the gpu version _net_receive_buffer to avoid the  update_net_receive_buffer(nt) (which is strangely slow in my profiling...), what should I do? Currently, the _net_receive_buffer is pinned between CPU and GPU because of acc_copyin, and I guess if the size of the buffer components changes, during the acc_update_device the gpu buffer might be freed, reallocated and transferred again, which greatly hurts performance.

@nrnhines

That is very similar to the binqueue for NetCon events except td is in the binqueue so that spike.ts+netcon.delay is calculated only once.

Wow, I haven't read the code about binqueue, and I think this query method would be good for GPU parallelization rather than CPU single thread. So if the query method is moved to GPU, the performance should increase greatly.

They are never interprocessor or interthread events. They come from a mod file instance and get sent back to the same instance. See netcvode.cpp:: void net_send(...

Do you mean that in the priority queue there are only netcons, and other events such as SelfEvents, ConditionEvents, and NetParEvents won't appear in the queue?
Besides, if I want to let the translated cxx file's _net_buf_receive directly modifies the gpu version _net_receive_buffer to avoid the  update_net_receive_buffer(nt) (which is strangely slow in my profiling...), what should I do? Currently, the _net_receive_buffer is pinned between CPU and GPU because of acc_copyin, and I guess if the size of the buffer components changes, during the acc_update_device the gpu buffer might be freed, reallocated and transferred again, which greatly hurts performance.

...binqueue... if the query method is moved to GPU, the performance should increase greatly.

I agree it could be on the GPU. Whether you can get better performance (at least for interprocessor spikes) is an experimental question. But I wouldn't be surprised if a single cpu thread model on the GPU could greatly benefit with a spike staying on the GPU from send to receive. That was beyond the scope of my GPU understanding since spikes are generated and delivered randomly without regard to ordering related to the otherwise "structure of array" memory organization.

SelfEvents, ConditionEvents, and NetParEvents won't appear in the queue?

The don't appear in the binqueue. They go into the heap  or splay tree queue for exact delivery time. These really are candidates for staying on the GPU (at least SelfEvents and ConditionEvents that get sent and delivered to the same object. NetParEvents are just for synchronization at "minimum NetCon (interprocessor/interthread) delay integration intervals" to ensure every event has arrived at its thread destination in time for delivery. The NEURON version has a flag for "SelfEvent not on queue" that takes advantage of this but it did not make it into CoreNEURON.

(which is strangely slow in my profiling...), what should I do? ... guess if the size of the buffer components changes

You would have to establish whether buffer size changes are a significant performance issue. Presently the ml->_net_receive_buffer is doubled in size every reallocation and starts out with size equal to the number of instances of the type.  The usual pattern is that the number of NetCon events destined for an instance within a time step is usually 0 and never > 1.   We should consult with @pramodk  about your performance results.

...binqueue... if the query method is moved to GPU, the performance should increase greatly.

I agree it could be on the GPU. Whether you can get better performance (at least for interprocessor spikes) is an experimental question. But I wouldn't be surprised if a single cpu thread model on the GPU could greatly benefit with a spike staying on the GPU from send to receive. That was beyond the scope of my GPU understanding since spikes are generated and delivered randomly without regard to ordering related to the otherwise "structure of array" memory organization.

SelfEvents, ConditionEvents, and NetParEvents won't appear in the queue?

The don't appear in the binqueue. They go into the heap  or splay tree queue for exact delivery time. These really are candidates for staying on the GPU (at least SelfEvents and ConditionEvents that get sent and delivered to the same object. NetParEvents are just for synchronization at "minimum NetCon (interprocessor/interthread) delay integration intervals" to ensure every event has arrived at its thread destination in time for delivery. The NEURON version has a flag for "SelfEvent not on queue" that takes advantage of this but it did not make it into CoreNEURON.

(which is strangely slow in my profiling...), what should I do? ... guess if the size of the buffer components changes

You would have to establish whether buffer size changes are a significant performance issue. Presently the ml->_net_receive_buffer is doubled in size every reallocation and starts out with size equal to the number of instances of the type.  The usual pattern is that the number of NetCon events destined for an instance within a time step is usually 0 and never > 1.   We should consult with @pramodk  about your performance results.

@nrnhines I've done some more detailed profiling, and it suggests that you are right about the GPU buffer allocations. The GPU buffers are not reallocated, so the performance is not influenced by it.
The more detailed profiling suggests that the (*corenrn.get_pnt_receive()[typ]) called by the delivered netcons and the update_net_receive_buffer(nt) are consuming a great partial of time when the simulation size increases.
I think If we could collect all the netcons that is going to fire in a timestep, and let them fire (aka calling the *corenrn.get_pnt_receive()[typ]) in parallel on GPU, and eliminate the data transfer of update_net_receive_buffer(nt), then the performance should be improved. But I have no idea how to make it, as the mod files are auto-generated. Do I have to modify the generated mod c codes on my self? Could you give me some advices on how to realize it?
I have another question on the deliver event functions. It appears that there are two functions, deliver_net_events and nrn_deliver_events, that are related to the event delivery, but when I dig into the code, only the deliver_net_events function would call (input)presyn->send and insert the netcons into the priority queue. So what does nrn_deliver_events do? Why it is seperated from the deliver_net_events?

@nrnhines I've done some more detailed profiling, and it suggests that you are right about the GPU buffer allocations. The GPU buffers are not reallocated, so the performance is not influenced by it.
The more detailed profiling suggests that the (*corenrn.get_pnt_receive()[typ]) called by the delivered netcons and the update_net_receive_buffer(nt) are consuming a great partial of time when the simulation size increases.
I think If we could collect all the netcons that is going to fire in a timestep, and let them fire (aka calling the *corenrn.get_pnt_receive()[typ]) in parallel on GPU, and eliminate the data transfer of update_net_receive_buffer(nt), then the performance should be improved. But I have no idea how to make it, as the mod files are auto-generated. Do I have to modify the generated mod c codes on my self? Could you give me some advices on how to realize it?
I have another question on the deliver event functions. It appears that there are two functions, deliver_net_events and nrn_deliver_events, that are related to the event delivery, but when I dig into the code, only the deliver_net_events function would call (input)presyn->send and insert the netcons into the priority queue. So what does nrn_deliver_events do? Why it is seperated from the deliver_net_events?

@nrnhines : could you copy / attach one of your profile output and also coreneuron's stdout? I am curious because in other models I haven't seen event related parts being bottleneck.

@nrnhines : could you copy / attach one of your profile output and also coreneuron's stdout? I am curious because in other models I haven't seen event related parts being bottleneck.

@pramodk I ran a huge network with 100000 neurons, on 8 top-tired A100 GPUs. Here is my output log:
mpiexec -np 8 ./profile_gpu_install/bin/special-core --tstop 1000 --datpath ./networks/100000Sim/RoundRobin-core-8 --mpi --gpu --multisend --ms-phases 1
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs6
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs8
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs5
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs7
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs9
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs4
--------------------------------------------------------------------------
[[20140,1],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: onebrain-dgx-a100-01

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: onebrain-dgx-a100-01
--------------------------------------------------------------------------
 num_mpi=8
 num_omp_thread=1

 Info : 8 GPUs shared by 8 ranks per node

 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2020
 Version : 0.21.0 43fe5fa (2021-01-21 20:12:20 +0800)

 Additional mechanisms from files
 bk_fs.mod bk_ms.mod cadyn_fs.mod cadyn_ms.mod cal12_ms.mod cal13_ms.mod caldyn_ms.mod can_fs.mod can_ms.mod caq_fs.mod caq_ms.mod car_fs.mod car_ms.mod cat32_ms.mod cat33_ms.mod exp2syn.mod expsyn.mod h_lts.mod hh.mod im_lts.mod it_lts.mod kaf_fs.mod kaf_ms.mod kas_fs.mod kas_ms.mod kdr_fs.mod kdr_ms.mod kdrbca1_lts.mod kir_fs.mod kir_ms.mod na3n_lts.mod naf_fs.mod naf_lts.mod naf_ms.mod netstim.mod par_ggap.mod passive.mod pattern.mod sk_fs.mod sk_ms.mod stim.mod svclmp.mod tmampa.mod tmgabaa.mod tmglut.mod tmnmda.mod vecevent.mod

 Memory (MBs) :             After mk_mech : Max 202.8984, Min 202.4102, Avg 202.5913
 Memory (MBs) :            After MPI_Init : Max 202.8984, Min 202.4102, Avg 202.5913
 Memory (MBs) :          Before nrn_setup : Max 204.4688, Min 204.0898, Avg 204.2632
 WARNING : GPU execution requires --cell-permute type 1 or 2. Setting it to 1.
[onebrain-dgx-a100-01:2475650] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[onebrain-dgx-a100-01:2475650] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[onebrain-dgx-a100-01:2475650] 7 more processes have sent help message help-btl-vader.txt / cma-permission-denied
all2allv_int gidin to intermediate space=175014 total=1833.57 time=0.00667806
all2allv_int gidout space=25034 total=1836.1 time=0.000639373
all2allv_int lists space=225026 total=1836.9 time=0.00040563
 Setup Done   : 52.94 seconds
 Memory (MBs) :          After nrn_setup  : Max 13454.4922, Min 13310.5234, Avg 13362.8823
GENERAL PARAMETERS
--mpi=true
--gpu=true
--dt=0.025
--tstop=1000

GPU
--nwarp=0
--cell-permute=0

INPUT PARAMETERS
--voltage=-65
--seed=-1
--datpath=./networks/100000Sim/RoundRobin-core-8
--filesdat=files.dat
--pattern=
--report-conf=
--restore=

PARALLEL COMPUTATION PARAMETERS
--threading=false
--skip_mpi_finalize=false

SPIKE EXCHANGE
--ms_phases=1
--ms_subintervals=2
--multisend=true
--spk_compress=0
--binqueue=false

CONFIGURATION
--spikebuf=100000
--prcellgid=-1
--forwardskip=0
--celsius=35
--mindelay=1.00875
--report-buffer-size=4

OUTPUT PARAMETERS
--dt_io=0.1
--outpath=.
--checkpoint=

 Start time (t) = 0

 Memory (MBs) :  After mk_spikevec_buffer : Max 13454.4922, Min 13310.5234, Avg 13362.8823
 Memory (MBs) :     After nrn_finitialize : Max 13454.8281, Min 13310.8945, Avg 13363.2485

 psolve |========================================================| t: 1000.00 ETA: 0h14m08s

Solver Time : 848.615

 Simulation Statistics
 Number of cells: 100001
 Number of compartments: 24948985
 Number of presyns: 46885564
 Number of input presyns: 699903
 Number of synapses: 142227989
 Number of point processes: 189819298
 Number of transfer (gap) targets: 0
 Number of spikes: 1705882
 Number of spikes with non negative gid-s: 1705882
Path                                            Min time/rank Max time/rank Avg time/rank Time %
main                                               910.918379    910.971648    910.943178 99.999924
  checkpoint                                         0.000014      0.000017      0.000016  0.000002
  output-spike                                       0.181329      0.181383      0.181364  0.019909
  simulation                                       848.614906    848.614913    848.614910 93.157761
    nrn_multisend_receive                           48.211981     77.066952     65.366869  7.175730
      nrnmpi_multisend_advance                       9.009972      9.801592      9.344053  1.025755
    timestep                                       771.164518    799.889700    782.842262 85.937486
      state-update                                  21.478632     23.407606     22.235478  2.440927
        state-tmGlut                                 0.600395      0.657703      0.621035  0.068175
        state-tmGabaA                                0.603162      0.670587      0.633518  0.069545
        state-sk_ms                                  0.630907      0.675192      0.649857  0.071339
        state-gGapPar                                0.198618      0.209459      0.204231  0.022420
        state-naf_ms                                 0.624028      0.681327      0.646556  0.070977
        state-naf_fs                                 0.621885      0.693116      0.653311  0.071718
        state-kir_ms                                 0.615571      0.683735      0.639699  0.070224
        state-kir_fs                                 0.621746      0.702599      0.645035  0.070810
        state-kdr_ms                                 0.605290      0.691195      0.639563  0.070209
        state-kdr_fs                                 0.609506      0.685924      0.640477  0.070309
        state-kas_ms                                 0.612295      0.683557      0.642188  0.070497
        state-kas_fs                                 0.621509      0.713599      0.654592  0.071859
        state-kaf_ms                                 0.617341      0.693093      0.644476  0.070748
        state-kaf_fs                                 0.624762      0.703548      0.653654  0.071756
        state-cat33_ms                               0.622820      0.670690      0.647910  0.071125
        state-cat32_ms                               0.623263      0.677978      0.644940  0.070799
        state-car_ms                                 0.620437      0.680921      0.649741  0.071326
        state-caq_fs                                 0.624777      0.685835      0.651047  0.071470
        state-can_ms                                 0.641906      0.691721      0.663833  0.072873
        state-cal13_ms                               0.630679      0.674601      0.650541  0.071414
        state-cal12_ms                               0.642529      0.690433      0.662547  0.072732
        state-bk_ms                                  0.634718      0.687286      0.658555  0.072294
        state-bk_fs                                  0.641710      0.709077      0.662374  0.072713
        state-cadyn_fs                               0.624075      0.685532      0.655934  0.072006
        state-cadyn_ms                               0.635891      0.710540      0.666689  0.073187
        state-caldyn_ms                              0.670242      0.715727      0.693189  0.076096
        state-pas                                    0.230345      0.246662      0.236413  0.025953
      update                                         0.761722      0.831632      0.800757  0.087904
      second_order_cur                               0.206082      0.237223      0.217908  0.023921
      matrix-solver                                260.285747    263.816372    262.734427 28.842000
      setup_tree_matrix                             27.783332     31.105136     29.111301  3.195729
        cur-tmGlut                                   0.698753      0.778214      0.734702  0.080653
        cur-tmGabaA                                  0.701016      0.771259      0.731793  0.080334
        cur-sk_ms                                    0.640352      0.731186      0.676640  0.074279
        cur-gGapPar                                  0.712011      0.801755      0.743627  0.081633
        cur-naf_ms                                   0.642256      0.736338      0.681916  0.074858
        cur-naf_fs                                   0.651287      0.712772      0.683288  0.075009
        cur-kir_ms                                   0.647686      0.754648      0.679597  0.074604
        cur-kir_fs                                   0.665318      0.731789      0.693071  0.076083
        cur-kdr_ms                                   0.636183      0.731089      0.673716  0.073958
        cur-kdr_fs                                   0.642821      0.718732      0.678598  0.074494
        cur-kas_ms                                   0.669113      0.766549      0.703603  0.077239
        cur-kas_fs                                   0.654087      0.745909      0.696810  0.076493
        cur-kaf_ms                                   0.653428      0.722170      0.686797  0.075394
        cur-kaf_fs                                   0.670647      0.759065      0.707676  0.077686
        cur-cat33_ms                                 0.660693      0.763976      0.701601  0.077019
        cur-cat32_ms                                 0.668792      0.768218      0.705791  0.077479
        cur-car_ms                                   0.665881      0.754181      0.700573  0.076906
        cur-caq_fs                                   0.670234      0.766199      0.715233  0.078516
        cur-can_ms                                   0.672516      0.767212      0.710777  0.078026
        cur-cal13_ms                                 0.665043      0.752573      0.706560  0.077564
        cur-cal12_ms                                 0.671335      0.782261      0.713954  0.078375
        cur-bk_ms                                    0.653320      0.735494      0.688944  0.075630
        cur-bk_fs                                    0.658014      0.748753      0.699011  0.076735
        cur-cadyn_fs                                 0.670311      0.765523      0.717275  0.078740
        cur-cadyn_ms                                 0.653098      0.745702      0.694606  0.076251
        cur-caldyn_ms                                0.658266      0.750415      0.696119  0.076417
        cur-cal_ion                                  0.556998      0.647780      0.587886  0.064536
        cur-ca_ion                                   0.563171      0.632140      0.595445  0.065366
        cur-k_ion                                    0.574322      0.640588      0.602244  0.066112
        cur-na_ion                                   0.648642      0.746462      0.688948  0.075630
        cur-pas                                      0.678330      0.787698      0.719481  0.078982
      deliver_events                               450.160572    473.903018    460.721194 50.576244
        nrn_deliver_events                         132.393860    135.454351    134.115292 14.722674
          netbuf_receive_device                      2.765391      2.927384      2.830566  0.310729
          transfer_netbuf_host2device               50.994765     58.777755     55.119894  6.050855
            update_net_receive_buffer               50.215810     58.042484     54.365246  5.968013
              acc_update_device                     24.804827     33.915664     29.601895  3.249585
              net_receive_buffer_order_refactor     22.872733     24.352687     23.540262  2.584162
          cvode_instance_deliver_events             72.225182     77.753242     74.663350  8.196262
        deliver_net_events                         313.929017    337.217062    325.070261 35.684993
          netbuf_receive_device                      2.750386      2.912784      2.820341  0.309607
          transfer_netbuf_host2device               34.304229     36.817370     35.225450  3.866918
            update_net_receive_buffer               33.573392     35.991656     34.462581  3.783173
              acc_update_device                      9.373999      9.934178      9.533084  1.046506
              net_receive_buffer_order_refactor     23.000350     24.758430     23.701044  2.601812
          deque_deliver_host                        72.178038     77.727124     74.609715  8.190375
          nrn_multisend_advance_host               167.421934    179.408870    173.218302 19.015255
            nrnmpi_multisend_advance                 0.931094      1.120856      1.029664  0.113033
          get_watch_host                             0.198831      0.225281      0.209353  0.022982
          send_presyn_host                          31.480995     35.580274     33.327281  3.658544
            nrnmpi_multisend                         3.652442      4.098563      3.879833  0.425914
          transfer_spike_deveice2host                0.948560      1.023150      0.972604  0.106769
          collect_spike_device                       1.901445      2.134007      2.011170  0.220779
  finitialize                                        6.514536      6.514597      6.514570  0.715145
    nrn_multisend_receive                            0.000184      0.437879      0.237697  0.026093
      nrnmpi_multisend_advance                       0.000047      0.000101      0.000067  0.000007
    cur-tmGlut                                       0.000021      0.000028      0.000023  0.000003
    cur-tmGabaA                                      0.000020      0.000035      0.000024  0.000003
    cur-sk_ms                                        0.000018      0.000025      0.000021  0.000002
    cur-gGapPar                                      0.000019      0.000035      0.000023  0.000002
    cur-naf_ms                                       0.000019      0.000024      0.000021  0.000002
    cur-naf_fs                                       0.000019      0.000025      0.000021  0.000002
    cur-kir_ms                                       0.000019      0.000025      0.000021  0.000002
    cur-kir_fs                                       0.000019      0.000026      0.000022  0.000002
    cur-kdr_ms                                       0.000017      0.000025      0.000021  0.000002
    cur-kdr_fs                                       0.000018      0.000025      0.000021  0.000002
    cur-kas_ms                                       0.000019      0.000027      0.000022  0.000002
    cur-kas_fs                                       0.000019      0.000026      0.000022  0.000002
    cur-kaf_ms                                       0.000019      0.000024      0.000021  0.000002
    cur-kaf_fs                                       0.000020      0.000026      0.000022  0.000002
    cur-cat33_ms                                     0.000019      0.000024      0.000021  0.000002
    cur-cat32_ms                                     0.000020      0.000025      0.000022  0.000002
    cur-car_ms                                       0.000019      0.000025      0.000022  0.000002
    cur-caq_fs                                       0.000020      0.000027      0.000023  0.000002
    cur-can_ms                                       0.000019      0.000026      0.000022  0.000002
    cur-cal13_ms                                     0.000019      0.000024      0.000021  0.000002
    cur-cal12_ms                                     0.000020      0.000026      0.000022  0.000002
    cur-bk_ms                                        0.000018      0.000024      0.000020  0.000002
    cur-bk_fs                                        0.000020      0.000025      0.000021  0.000002
    cur-cadyn_fs                                     0.000018      0.000029      0.000021  0.000002
    cur-cadyn_ms                                     0.000017      0.000027      0.000020  0.000002
    cur-caldyn_ms                                    0.000018      0.000030      0.000021  0.000002
    cur-cal_ion                                      0.000013      0.000022      0.000015  0.000002
    cur-ca_ion                                       0.000013      0.000019      0.000015  0.000002
    cur-k_ion                                        0.000014      0.000018      0.000015  0.000002
    cur-na_ion                                       0.000019      0.000024      0.000021  0.000002
    cur-pas                                          0.000025      0.000036      0.000031  0.000003
    nrn_deliver_events                               0.006263      0.014525      0.013337  0.001464
      netbuf_receive_device                          0.005979      0.014208      0.013023  0.001430
      transfer_netbuf_host2device                    0.000091      0.000114      0.000099  0.000011
        update_net_receive_buffer                    0.000033      0.000044      0.000038  0.000004
      cvode_instance_deliver_events                  0.000036      0.000047      0.000041  0.000005
  load-model                                        55.412495     55.606147     55.501959  6.092797

The profiler is Caliper, and note that I've added many detailed profiling to the deliver_events.