CUDA_TOO_MANY_PEERS exception when launching on p2.16xlarge on aws

This issue is identical to the issue discussed here.
When launching tensorflow to train on more than 8 gpu instances, tensorflow will cause a CUDA_TOO_MANY_PEERS error in the driver. The Tesla K80 shows up as 2 gpu instances internally so even though the p2.16xlarge technically only has 8 gpus it looks to the driver like it has 16. It seems that the cuda p2p system limits any gpu to only connect to a maximum 8 other gpus via p2p, but when the graph is launched, tensorflow (quite understandably) attempts to create a p2p connection between every gpu in the graph.
Is there some way to disable p2p or is there a way via intelligent graph construction to limit the number of connections any given gpu requires, or even use a resource pool for gpu p2p connections?
ubuntu@host:~/workspace/nn$ nvidia-smi
Wed Nov  2 20:53:20 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 361.93.02              Driver Version: 361.93.02                 |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:00:0F.0     Off |                    0 |
| N/A   45C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:00:10.0     Off |                    0 |
| N/A   39C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:00:11.0     Off |                    0 |
| N/A   50C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:00:12.0     Off |                    0 |
| N/A   43C    P8    31W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 0000:00:13.0     Off |                    0 |
| N/A   50C    P8    57W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 0000:00:14.0     Off |                    0 |
| N/A   41C    P8    70W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 0000:00:15.0     Off |                    0 |
| N/A   51C    P0    56W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 0000:00:16.0     Off |                    0 |
| N/A   43C    P0    71W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   8  Tesla K80           Off  | 0000:00:17.0     Off |                    0 |
| N/A   46C    P0    56W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   9  Tesla K80           Off  | 0000:00:18.0     Off |                    0 |
| N/A   39C    P0    71W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  10  Tesla K80           Off  | 0000:00:19.0     Off |                    0 |
| N/A   49C    P0    58W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  11  Tesla K80           Off  | 0000:00:1A.0     Off |                    0 |
| N/A   40C    P0    73W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  12  Tesla K80           Off  | 0000:00:1B.0     Off |                    0 |
| N/A   49C    P0    58W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  13  Tesla K80           Off  | 0000:00:1C.0     Off |                    0 |
| N/A   40C    P0    70W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  14  Tesla K80           Off  | 0000:00:1D.0     Off |                    0 |
| N/A   49C    P0    60W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|  15  Tesla K80           Off  | 0000:00:1E.0     Off |                    0 |
| N/A   41C    P0    69W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
ubuntu@host:~/workspace/nn$ python3 deploy_local.py
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:0f.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3bb3f120
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:10.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3bf86790
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:11.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x3c3cf0c0
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:12.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4a7fe070
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:13.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4ac4dd20
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:14.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4b0a16a0
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:15.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4b4f9220
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:16.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4b9545d0
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 8 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:17.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4bdb3920
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 9 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:18.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4c216790
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 10 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:19.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4c67d450
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 11 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1a.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4cae8580
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 12 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1b.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x28999af0
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 13 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1c.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x28e0bf00
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 14 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1d.0
Total memory: 11.17GiB
Free memory: 11.11GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x29282290
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 15 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1e.0
Total memory: 11.17GiB
Free memory: 11.11GiB
E tensorflow/core/common_runtime/direct_session.cc:132] Internal: Internal: failed to enable peer access from 0x1c961680 to 0x18f1ac60: CUDA_ERROR_TOO_MANY_PEERS
Traceback (most recent call last):
  File "deploy_local.py", line 39, in <module>
    n_test_batches=4,
  File "/home/ubuntu/workspace/nn/nn/model/network.py", line 271, in train
    test_steps=test_steps, save_steps=save_steps, load_all=load_all, debug=debug, **kwargs)
  File "/home/ubuntu/workspace/nn/nn/train/train.py", line 86, in train_model
    with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, gpu_options=gpu_options)) as sesh:
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 1138, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 502, in __init__
    self._session = tf_session.TF_NewSession(opts, status)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors.py", line 463, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InternalError: Failed to create session.

@zhengxq, it seems like the recommended way to go about this is to use distributed and give each process on the same host half of the available GPUs using CUDA_VISIBLE_DEVICES as discussed on the link article. Could you confirm?

I can confirm that deploying as a cluster to half of the devices per worker does not reduce the number of p2p connections attempted when starting a session.
creating ec2 cluster
creating security group
launching instances
waiting for ip addresses to be assigned
cluster config: {
  "ps_hosts": [
    "<IP_ADDRESS>"
  ],
  "worker_hosts": [
    "<IP_ADDRESS>",
    "<IP_ADDRESS>"
  ]
}
setting rules for security group.
waiting for cluster instances to boot up.
starting ps host at: <IP_ADDRESS>
starting worker host at: <IP_ADDRESS>
starting master worker host at: <IP_ADDRESS>
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
<IP_ADDRESS>: INFO: starting parameter server 0
<IP_ADDRESS>: INFO: starting master worker.
<IP_ADDRESS>: INFO: starting training worker 1
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd5093eb0
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7838bed0
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6e672bb0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd54e5d30
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6eaa38b0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b78778410
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd59365b0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6eed7d10
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b78b66470
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd5d8add0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6f310590
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b78f55830
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd61e3400
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b65446460
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b79346310
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd663fad0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b65837ee0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b79738150
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:15.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd6aa04e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:15.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b65c2b780
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:15.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b79b2b630
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:16.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd6f04a60
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:16.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6601f5b0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:16.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b79f1f670
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 8 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:17.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd736cfc0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 8 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:17.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b66414bf0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 8 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:17.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7a314800
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 9 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:18.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd77d9480
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 9 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:18.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b6680b590
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 9 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:18.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7a70acf0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 10 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:19.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7bd7c49a40
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 10 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:19.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b66c03120
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 10 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:19.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7ab02880
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 11 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1a.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f79f80be5b0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 11 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1a.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b66ffced0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 11 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1a.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7aefc630
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 12 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1b.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f79f85369e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 12 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1b.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b673f71f0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 12 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1b.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7b2f6950
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 13 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1c.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.04GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f79f89b2d40
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 13 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1c.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b677f23c0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 13 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1c.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7b6f1b20
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 14 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1d.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f79f8e32fe0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 14 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1d.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.98GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b67bee8f0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 14 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1d.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.98GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f7b7baee050
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 15 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1e.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 15 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1e.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.98GiB
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 15 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1e.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.98GiB
Traceback (most recent call last):
  File "/Users/alexatknit/Workspace/nn/deploy_local.py", line 42, in <module>
    n_test_batches=4,
  File "/Users/alexatknit/Workspace/nn/nn/model/network.py", line 293, in train_cluster
    n_test_batches=n_test_batches, test_steps=test_steps, save_steps=save_steps, load_all=load_all, **kwargs)
  File "/Users/alexatknit/Workspace/nn/nn/train/deploy.py", line 81, in deploy_to_cluster
    **kwargs).result()
  File "/usr/local/lib/python3.5/site-packages/KnitPyUtils-0.0.39-py3.5.egg/knitutils/util/executor.py", line 25, in result
    raise self._result
  File "/usr/local/lib/python3.5/site-packages/KnitPyUtils-0.0.39-py3.5.egg/knitutils/util/executor.py", line 119, in _worker
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/KnitPyUtils-0.0.39-py3.5.egg/knitutils/parallel/cluster.py", line 83, in _submit
    return self.proxy.submit(fn, *args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/KnitPyUtils-0.0.39-py3.5.egg/knitutils/util/server.py", line 202, in method
security group removed.
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.39-py3.4.egg/knitutils/util/server.py", line 122, in handle
    result = ('#RETURN', func(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.39-py3.4.egg/knitutils/util/functions.py", line 48, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.39-py3.4.egg/knitutils/parallel/cluster.py", line 39, in submit
    return self.future_pool.submit(fn, *args, **kwargs).result()
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.39-py3.4.egg/knitutils/parallel/future.py", line 75, in result
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.39-py3.4.egg/knitutils/parallel/future.py", line 18, in _deploy_worker
    result = ("#RETURN", fn(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/KnitNN-0.5.29-py3.4.egg/nn/train/train.py", line 398, in train_master
    server = tf.train.Server(cluster, job_name='worker', task_index=0)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/server_lib.py", line 152, in __init__
    self._server_def.SerializeToString(), status)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors.py", line 463, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InternalError: Internal: failed to enable peer access from 0x7f7b8c286310 to 0x7f7b8e226740: CUDA_ERROR_TOO_MANY_PEERS
scheduling instances for termination: ['<IP_ADDRESS>']
waiting for termination to complete.

@alexatknit, could you confirm that you have set CUDA_VISIBLE_DEVICES? TensorFlow should not even see the invisible devices in that case.
Even if it is not set, this is a bit weird. Before trying to enabling peer-access to each other GPU, TF calls cuDeviceCanAccessPeer, only if it returns True, TF proceeds to call the next cuCtxEnablePeerAccess. It seems to cuDeviceCanAccessPeer returns True even if the next cuCtxEnablePeerAccess will fail with CUDA_ERROR_TOO_MANY_PEERS.
+@benbarsdell, our friend from NVIDIA. Could you confirm whether cuCtxEnablePeerAccess should fail in this case?

I have not altered CUDA_VISIBLE_DEVICES, I'll try it out.

cuDeviceCanAccessPeer returns True even if the next cuCtxEnablePeerAccess will fail with CUDA_ERROR_TOO_MANY_PEERS

This is correct.

@benbarsdell, would it be too big a feature request to have future Cuda drivers return False in that case?

I've set CUDA_VISIBLE_DEVICES and still have the issue

@alexatknit, could you share your command line, your CUDA_VISIBLE_DEVICES setting, and the resulting logs? If the log is too large, feel free to upload somewhere such as pastebin.com and share the URL here.

My cluster is launched remotely, I set the environment variable in python

Depending on where you set it, it might be too late. It needs to be set the first time Cuda driver is loaded.
The safest thing to try is to set it in the script that launches your worker and ps.

There are far too many variables associated with launching a model, I would need to rework my entire system to try that. My listener process seems to be importing tensorflow during the depickle phase which loads the driver, and this state is passed to the child processes.

@zheng-xq

would it be too big a feature request to have future Cuda drivers return False in that case?

cuDeviceCanAccessPeer only considers the topology of the system, independent of any other state; I believe that changing this behavior could potentially break existing codes. There are also other reasons why calls to cuCtxEnablePeerAccess may still fail.

Alright, I updated the architecture to support setting environment variables before deserialization and it appears that it still fails:
creating ec2 cluster
launching instances
waiting for ip addresses to be assigned
cluster config: {
  "worker_hosts": [
    "<IP_ADDRESS>",
    "<IP_ADDRESS>"
  ],
  "ps_hosts": [
    "<IP_ADDRESS>"
  ]
}
setting rules for security group.
waiting for cluster instances to boot up.
starting ps host at: <IP_ADDRESS>
starting worker host at: <IP_ADDRESS>
starting master worker host at: <IP_ADDRESS>
<IP_ADDRESS>: env = {}
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
<IP_ADDRESS>: env = {'CUDA_VISIBLE_DEVICES': '0,1,2,3,4,5,6,7'}
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
<IP_ADDRESS>: env = {'CUDA_VISIBLE_DEVICES': '8,9,10,11,12,13,14,15'}
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
<IP_ADDRESS>: I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
<IP_ADDRESS>: INFO: starting parameter server 0
<IP_ADDRESS>: INFO: starting master worker.
<IP_ADDRESS>: INFO: starting training worker 1
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:17.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e64e3f6b0
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e60e3c540
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebd0934f0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:18.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e6522b7e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebd4e4f80
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e612288e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:0f.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e65619840
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebd9357e0
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 2 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e61616d00
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:10.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e65a08fc0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebdd89f40
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 3 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e61a060c0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:11.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e65df9aa0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebe1e2600
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 4 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e61df67e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:12.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e661eb520
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebe63ec40
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 5 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e621e8260
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:13.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e665dedc0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:15.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebea9f5a0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 6 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:15.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4e625dbb00
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:14.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 10.99GiB
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:16.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebef03bb0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 7 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:16.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.05GiB
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 1 2 3 4 5 6 7
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 1:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 2:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 3:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 4:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 5:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 6:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 7:   Y Y Y Y Y Y Y Y
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:17.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K80, pci bus id: 0000:00:18.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:2) -> (device: 2, name: Tesla K80, pci bus id: 0000:00:0f.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:3) -> (device: 3, name: Tesla K80, pci bus id: 0000:00:10.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:4) -> (device: 4, name: Tesla K80, pci bus id: 0000:00:11.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:5) -> (device: 5, name: Tesla K80, pci bus id: 0000:00:12.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:6) -> (device: 6, name: Tesla K80, pci bus id: 0000:00:13.0)
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:7) -> (device: 7, name: Tesla K80, pci bus id: 0000:00:14.0)
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 8 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:17.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 565.56MiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebf36c090
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 9 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:18.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 504.81MiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebf7d8550
<IP_ADDRESS>: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:197] Initialize GrpcChannelCache for job ps -> {0 -> <IP_ADDRESS>:40842}
<IP_ADDRESS>: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:197] Initialize GrpcChannelCache for job worker -> {0 -> <IP_ADDRESS>:50425, 1 -> localhost:52197}
<IP_ADDRESS>: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:206] Started server with target: grpc://localhost:52197
<IP_ADDRESS>: INFO: training model res_net_v10_20161103_2_fcn.0
<IP_ADDRESS>: INFO: setting up device: '/job:worker/task:1'
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 10 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:19.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ebfc48b00
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 11 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1a.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ce40bd6e0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 12 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1b.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ce4535a80
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 13 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1c.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ce49b1e00
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 14 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1d.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
<IP_ADDRESS>: W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x7f4ce4e320b0
<IP_ADDRESS>: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
<IP_ADDRESS>: I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 15 with properties:
<IP_ADDRESS>: name: Tesla K80
<IP_ADDRESS>: major: 3 minor: 7 memoryClockRate (GHz) 0.8235
<IP_ADDRESS>: pciBusID 0000:00:1e.0
<IP_ADDRESS>: Total memory: 11.17GiB
<IP_ADDRESS>: Free memory: 11.11GiB
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.40-py3.4.egg/knitutils/parallel/future.py", line 18, in _deploy_worker
    result = ("#RETURN", fn(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/KnitPyUtils-0.0.40-py3.4.egg/knitutils/parallel/cluster.py", line 23, in _run_worker
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/KnitNN-0.5.30-py3.4.egg/nn/train/train.py", line 406, in train_master
    server = tf.train.Server(cluster, job_name='worker', task_index=0)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/server_lib.py", line 152, in __init__
    self._server_def.SerializeToString(), status)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors.py", line 463, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InternalError: Internal: failed to enable peer access from 0x7f4e79688120 to 0x7f4e7aa4ebb0: CUDA_ERROR_TOO_MANY_PEERS
scheduling instances for termination: ['<IP_ADDRESS>']
waiting for termination to complete.

If we change it from an error to a warning, would that make people happy? :)

(I'm going to send out a change that does this, and then we can refine it if it turns out to be bad).  I think enough people are bitten by this that don't really care, that we should just fix it for the common case.

@vrv does the change make sure to prioritize the p2p connections that are in the graph? I wouldn't want to have an issue where gpus 8-15 can't talk to each other even though they're the only ones the graph is deployed to in that particular tf session.

Yeah, I would try to use CUDA_VISIBLE_DEVICES if you really don't plan on using the GPUs in your graph.   All the change will do is not fail when it currently fails, so things might get a bit slower, but at least it will tell you via the warning logs, instead of just failing the job.


[SPARK-9908] [SQL] When spark.sql.tungsten.enabled is false, broadcast join does not work

https://issues.apache.org/jira/browse/SPARK-9908

Merged build triggered.

Merged build started.

Test build #1509 has started for   PR 8149 at commit 16c3438.

Merged build finished. Test FAILed.

Test build #1509 has finished for   PR 8149 at commit 16c3438.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1524 has started for   PR 8149 at commit 16c3438.

I'm merging this since the Python test is flaky.

Test build #1524 has finished for   PR 8149 at commit 16c3438.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.


🚧 Dev ➾ On compile, emit additional files only if a main function is present

🧰 Work Description
When I compile an arbitrary file using taq compile xyz.mligo the tool emits xyz.{parameter,storage}List.mligo` (what I'll refer to as, "additional files").
It assumes that I'm writing a contract, for which yes I might like those additional files.
However, many contract projects use multiple files per "actual" contract, so all these extra files are slightly inconvenient while well-intentioned.
✅ To Do
Have taq scan for a main function  within the compilation target, and if found only then produce the additional files.
The ligo tool provides a way to list the functions in a given source-file:
ligo info list-declarations <file-name.mligo>
If main is listed therein, at that point taq should generate the additional files.
⚖ Acceptance Criteria
Compiling arbitrary files does not produce the additional files.
Compiling any file with a main function produces the additional files.
🔭 Scope
🙅♂ Out of Scope:

...

Code of Conduct

[X] I agree to follow this project's Code of Conduct

We can't use the function from ligo to check if there is "main" function or the substitute.
Right now, it works with 'main' though.


UX Researcher: Creating User Interview Script - Batch 1

Creating the interview script for each interview based off our interview questions (https://ibm.box.com/s/i7jpsm9wayrh5t6pzfok7hlzwd1nculq) that should cover and provide some data for the following research topics:

User roles + team mgt (#294)
(Approval) workflows (#296)
Content creation (#301)
Editing content (#295)

Content types (understanding/navigating to content types) (#297)
Alerts + notifications (#300)
First time experience on the CMS (#341)
Searching for content (#299)

Feature: Create interview script for user interviews
  As a UX researcher
  I will take the interview questions I've created and create an interview script to use during interviews
  So that I make sure I only ask the interview questions that are pertinent to each user's role

  Scenario: Craft interview script for interviewees
      Given that I will have created a large list of interview questions to use
      And that I will be interviewing people from many differing roles
     Then I will create interview scripts for each interviewee comprised of interview questions that make sense for that user's role

Research Interview Script

[ ] Create interview script for Allen
[ ] Create interview script for Jeremy
[ ] Create interview script for Lecia (may separate this out for next sprint)

Closing this - decided it was too related to my other story - I don't need this.


ENH: support toctree captions as first navigation level?

Ah damn, run into an issue. In the top level documentation we use multiple toctree`s with captions to structure the sections. This is fine in RTD, but pydata-sphinx-theme doesn't take these captions into account when constructing the top bar:

Source: https://github.com/aiidateam/aiida-core/blob/develop/docs/source/index.rst
RTD: https://130-77234579-gh.circle-artifacts.com/0/html/index.html
pydata: https://132-77234579-gh.circle-artifacts.com/0/html/index.html

@jorisvandenbossche any quick fix you can think of, or shall I raise a separate issue?
I reckon pydata-sphinx should be able to handle this in some way, since I don't think it's an uncommon practice.
The only other minor difference, is that RTD shows the documentation version at the top-right. I'm not sure if that is strictly necessary. But it might be nice to have the option to show this in some fashion
Originally posted by @chrisjsewell in https://github.com/pandas-dev/pydata-sphinx-theme/issues/188#issuecomment-635871957

In the top level documentation we use multiple toctree`s with captions to structure the sections. This is fine in RTD, but pydata-sphinx-theme doesn't take these captions into account when constructing the top bar

Yeah, that's a known issue, and @choldgraf looked at this before, and I was recently trying to pick it up again, see https://github.com/pandas-dev/pydata-sphinx-theme/pull/135
It's indeed something we should handle, as it is commonly used.
Now, in your case, I think you want to use those captions as the "first level of navigation", right? So to use those items for the top navbar?
(the PR I linked too is more about allowing those captions in the sidebar)

So this theme uses the toctree as inferred by sphinx from your project (but ignoring captions at the moment ..) to structure the layout.
That also means that you might need to structure your sphinx doc directory layout (or the toctrees) in such a way that it fits what you want to see in the html docs.
Meaning that you need nested toctrees (not necessarily a nested directory layout, as sphinx just looks at the toctree directives). While in your example of AIIDA, you have single flat toctree (just split in separate toctree directives with a caption, but for sphinx that's basically a single flat level)

Yeh, it is a bit of a weird one, because there is technically nothing to actually link to for the captions.
I had the same issue with the panel links, where I am currently just linking to the first document in each section.

Meaning that you need nested toctrees (not necessarily a nested directory layout, as sphinx just looks at the toctree directives)

yeh I might have a play around with this quickly

I have to say, I do prefer the RTD toctree bar though, where you can quickly link to anywhere in the documentation,
e.g. in https://pydata-sphinx-theme.readthedocs.io/en/latest/index.html, it feels a bit redundant that there is a blank toc bar, and you are having to instead have the toctree show on the page

I commented on https://github.com/aiidateam/aiida-core/pull/4129, I think it would actually not be a big change to get it working with the current release.
One thing is that you indeed get those "ugly" landing pages with only the toctree, while it would be nice to directly go to for example the first subpage (eg in pandas this page is quite useless: https://pandas.pydata.org/docs/user_guide/index.html, for the getting started index.rst, we added some more content which makes this approach OK: https://pandas.pydata.org/docs/getting_started/index.html. But you might not necessarily have such content for each index.rst ..)
Related issue: https://github.com/pandas-dev/pydata-sphinx-theme/issues/60

I have to say, I do prefer the RTD toctree bar though, where you can quickly link to anywhere in the documentation (to depth 2),

Then you might want to use the "single level" version of this theme instead? (without the top navbar)
See https://github.com/pandas-dev/pydata-sphinx-theme/issues/90 (which basically is the sphinx-book-theme, or a slight variation on the layout of this theme like in eg https://contextily.readthedocs.io/en/latest/ (only that we want caption support there, which sphinx-book-theme already does).
I would personally still like to see this "no-top-bar-layout" included in this repo as well, but that's a discussion for https://github.com/pandas-dev/pydata-sphinx-theme/issues/90

e.g. in https://pydata-sphinx-theme.readthedocs.io/en/latest/index.html, it feels a bit redundant that there is a blank toc bar, and you are having to instead have the toctree show on the page

Yes, for home pages, that's certainly a problem that should be solved. One way to add more content to the home page (and hide the in-page toctree), but you might also want a different layout for the home page (eg no left sidebar), see https://github.com/pandas-dev/pydata-sphinx-theme/issues/146

I guess essentially what I'd like is the bulk of the pydata theme, but with the "fixed" left toctree from the RTD theme.
At least for the amount/depth of content in the AiiDA documentation, I think it's a lot more user-friendly that you can scan down the left ToC to get a good/quick outline of the documentation and quickly select e.g. Topics/Workflows, rather than having this extra step of having to first navigate to Topics then looking what's there.

What do you understand exactly under "the bulk" ?
Also the top navbar? I would assume not, since that would then duplicate the captions of the left sidebar toctrees ?
And if you want everything except the top navbar, isn't that basically the "single level" version what I mentioned before (so basically something like https://jupyterbook.org/intro.html or like https://jupyterbook.org/intro.html (but with captions in the sidebar))

At least for the amount/depth of content in the AiiDA documentation

Yes, it certainly depends on the amount of links that are in the left sidebar whether it becomes beneficial to splitting this up into "first level nav in topbar and second+ level navigation in sidebar", or whether you prefer keeping all levels of navigation in the sidebar.
It's mainly when the number of links / pages in the sidebar get that big that you need to do a lot of scrolling in it that I personally prefer moving part of the navigation to the top navbar, but of course that is still subjective ;)

Sorry this is mainly just me thinking out loud...
I think the top banner works better as part of a larger website, where your "documentation" is a section of the overall site, like:

pandas and AiiDA are both a bit similar in this respect, in that they actually have separate top-level and documentation sites:

https://pandas.pydata.org/ and https://pandas.pydata.org/docs/
http://www.aiida.net/ and https://aiida.readthedocs.io

Its kind of odd for both that clicking on the documentation section of the banner on the main site leads you to a completely different site, with its own separate layout.

What do you understand exactly under "the bulk" ?

I guess just the general CSS, plus the right "page level" ToC

BTW I was recently pointed toward this "search as you type" extension https://readthedocs-sphinx-search.readthedocs.io which is pretty nice. I haven't check yet if it works already with the pydata theme, but it might be a consideration to include/embed it or something similar by default.

since we're talking about RTD-only extensions, there's also this nifty extension to provide hover tooltips: https://pypi.org/project/sphinx-hoverxref/

nice!

@chrisjsewell yeah, I am hoping we can convince RTD to special-case jupyter books so that people can use these extensions in their books if they use RTD...

convince RTD to special-case jupyter books

What do you mean by special case?

right RTD only supports "regular" sphinx builds, mkdocs, and maybe one or two other documentation engines. Since Jupyter Book builds books with jupyter-book build mybook/ rather than make html w/ Sphinx, then we need RTD to support this build command. (unless you know of a way to use custom build commands with RTD?)

unless you know of a way to use custom build commands with RTD?

Yeh not that I know, so sounds like a good idea

This issue has no activity since 2020 is it still a wanted feature ?

Since there has not been a follow-up and this issue seems stale anyway, I will go ahead and close it. If there is need for this at a later time we can re-open this issue.


Harrison/activeloop ai tql deprecation

Is there anything more needed for this PR? Or is it ready to merge?


Unnecessary overwrite confirmations

Whenever I try to save patient's data I get dialog similar to:

This feels counterproductive - I would expect such dialogs to only appear if for example try to create a patient record with the same account number as another in the database. Saving should IMHO just happen. Spamming the user with too many confirmations will just lead to them ignoring those and mindlesssly clicking "Yes" - including the cases when they do something actually dangerous (for example opening a new patient without saving the currently opened one).
The dialog also appears even if I made no changes to the patient.
Further the message dialog

is IMHO redundant and distracts from work. I would suggest you instead have some area for messages that don't necessarily require attention (like this one) and show them there without requiring user input and let them fade (either over time or when the user makes additional changes).
Note also that most modern UIs actually save everything immediately as the user writes it in. You might want to consider doing the same - this is however just a suggestion and I don't think it is required.

You are incorrect in your remarks. Every time a patient's record is overwritten, the user is asked for permission because a person could accidentally overwrite a medical record. I believe that dialog box is important to forewarn the user.
I don't have the time to make the changes you ask for and I don't think they affect the functionality of the software.

Overwrites are important to tell the user not accidentally overwrite the data.


rake-db: custom collation bug

From discord: collation name is single-quoted by rake-db, but seems like it should be double-quoted instead.

Fixed this and added methods for creating and dropping a collation: docs


Route title function isn't reactive

Here's a example of route definition:
FlowRouter.route('/:storyId', {
  name: 'story',
  title({storyId}) {
    // This is only fired once, cursors are empty
    // Subscriptions are done in templates
    // When data is available, this doesn't run again
    const story = Stories.find(storyId).fetch()[0]
    return story && story.title
  },
  action: function(params, queryParams) {
    BlazeLayout.render(...)
  }
})

I also tried const story = Stories.find(storyId).fetch() (without [0]),  or even const story = Stories.find(storyId), no luck. The title() function should run again when the cursor isn't empty anymore. Also tried with some Session.get() tests, nothing happens neither.
Thanks.
NB. This follows https://github.com/VeliovGroup/Meteor-flow-router-title/issues/6

@nicooprat should be fixed now, please update to the latest release and let me know if it works on your end.

Many thanks for your reactivity and your work!
I can confirm it's now reactive, but... too much :) I set a debugger in my title() function, and it runs 5 times on any route change: 2 times on the exiting route, 3 times on the entering route. Any idea why it happens?

Show me your code
If possible debug it with console.trace()
Do you use is in a group, or nested group?

My code is like the one on the first comment.
Yes the route is in a group, which has a default title() function by the way (which is not run in this case, as expected).

What browser that log from? Can't differ call from call.

Yes the route is in a group, which has a default title() function by the way (which is not run in this case, as expected).

Please explain

It's from Chrome 61. How could I get a better trace?
I knew I wasn't clear :) Here's a sample of my code:
const allRoutes = FlowRouter.group({
  name: 'allRoutes',
  title() {
    // This is not run, as expected
    return 'Default title'
  }
}

allRoutes.route('/:storyId', {
  name: 'story',
  title({storyId}) {
    // Runs a lot of times
    const story = Stories.find(storyId).fetch()[0]
    return story && story.title
  }
})

Do you meant to use titlePrefix instead of title in a group?
Because title in a group will be used as default title for routes where is no title.

No it's meant to be like this, I have some routes where I don't set the title, so the default one is the group is set. I just meant to say that I'm actually using groups, but in my understanding it's not the cause of the issue, as it behaves as expected.
By the way, a titleSuffix option would be great too, as I always need to append the name of the app to titles. I think it's a pretty common use case.

By the way, a titleSuffix option would be great too, as I always need to append the name of the app to titles. I think it's a pretty common use case.

Then why don't you use titlePrefix 🤣 ?

How could I get a better trace?

Mb using warn expanding trace and taking screenshot.

No it's meant to be like this, I have some routes where I don't set the title, so the default one is the group is set. I just meant to say that I'm actually using groups, but in my understanding it's not the cause of the issue, as it behaves as expected.

If it's used in groups it will re-computed for each title in each group of nested groups and its routes

I was actually speaking of appending text, not prepending. Like « prefix > page - app title ». Unless I misunderstood its use?
I’ll get that trace as soon as possible, afk for now, too bad we are not in the same timezone :D
I’ll try without this group default title then.

Well:
route's title: 'Title'
group's titlePrefix: 'Prefix > '

// will result in
Prefix > Title

too bad we are not in the same timezone :D

1:30 am at my zone :D

I was actually speaking of appending text, not prepending. Like « prefix > page - app title ». Unless I misunderstood its use?

Got you now... yeah, not possible now. Not sure if will be ever possible, I can detect first definition to add prepending text, but won't ever now if that one is the last one, thanks to the ability to nest groups

Understood. Not sure if it’s really importan thought, that could even be something static, as I can’t really think of another use than for the global app title. But anyway, that’s not really important.

But anyway, that’s not really important.

And it will be difficult to keep title in the recommended range of 70 chars