TF-GPU v1.14 CUDA runs out of memory

Hello fellow machine learning people :)
I am hoping you can please pull down my custom TF-GPU, CUDA X, and Anaconda container solution with Jupyter. I am hoping you can assist because whatever the problem is, the fix for the container is same as Ubuntu.
Here is the error when running Tensorboard and benchmarks.py (see instructions in README.md as to how to run it after spinning up container).
This is the error that I get, scroll right to read entire line:
root@e71bda560638:/apps/apps/gpu_benchmarks# python tensorboard.py
WARNING: Logging before flag parsing goes to stderr.
W0721 23:23:02.790582<PHONE_NUMBER>45184 __init__.py:308] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0721 23:23:02.810480<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:191: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W0721 23:23:02.810840<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:161: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W0721 23:23:02.810928<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:163: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0721 23:23:02.811073<PHONE_NUMBER>45184 deprecation.py:323] From tensorboard.py:20: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0721 23:23:02.811128<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
W0721 23:23:02.811218<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
W0721 23:23:04.074006<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
W0721 23:23:04.547974<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
W0721 23:23:04.550884<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
W0721 23:23:05.421007<PHONE_NUMBER>45184 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0721 23:23:05.558143<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:22: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.

2019-07-21 23:23:05.558423: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-21 23:23:05.562806: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-21 23:23:05.563094: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e55a519380 executing computations on platform Host. Devices:
2019-07-21 23:23:05.563108: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-21 23:23:05.563894: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-21 23:23:05.580061: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.580637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
2019-07-21 23:23:05.580748: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:23:05.581760: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:23:05.582704: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-07-21 23:23:05.582852: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-07-21 23:23:05.583850: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-07-21 23:23:05.584484: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-07-21 23:23:05.586678: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-21 23:23:05.586776: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.587222: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.587602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-21 23:23:05.587629: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:23:05.646174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-21 23:23:05.646191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-07-21 23:23:05.646197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-07-21 23:23:05.646385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.646756: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.647144: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:05.647446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 170 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-21 23:23:05.649957: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e55ba1ac60 executing computations on platform CUDA. Devices:
2019-07-21 23:23:05.649969: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
W0721 23:23:05.650787<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:27: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0721 23:23:05.652657<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:32: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0721 23:23:05.653614<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:37: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0721 23:23:05.658960<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:49: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0721 23:23:05.666438<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:55: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0721 23:23:05.681653<PHONE_NUMBER>45184 deprecation.py:506] From tensorboard.py:84: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0721 23:23:05.714577<PHONE_NUMBER>45184 deprecation.py:323] From tensorboard.py:100: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

W0721 23:23:05.727325<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:106: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0721 23:23:05.796838<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:118: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W0721 23:23:05.797843<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:119: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0721 23:23:05.817948<PHONE_NUMBER>45184 deprecation_wrapper.py:119] From tensorboard.py:121: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

2019-07-21 23:23:06.136829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:23:06.224853: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-21 23:23:06.226553: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-21 23:23:06.228049: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.228406: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.228869: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.229269: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.229750: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.230151: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.230537: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.230914: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.231278: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.231646: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.232019: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.232382: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.232740: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.233104: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.233457: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.233823: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.234184: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.234533: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.234906: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.235263: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.235914: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.236498: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.236888: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:06.237265: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:16.229311: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:16.230144: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:16.230171: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.91MiB (rounded to 31360000).  Current allocation summary follows.
2019-07-21 23:23:16.230190: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): 	Total Chunks: 15, Chunks in use: 15. 3.8KiB allocated for chunks. 3.8KiB in use in bin. 240B client-requested in use in bin.
2019-07-21 23:23:16.230203: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230215: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-07-21 23:23:16.230226: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): 	Total Chunks: 5, Chunks in use: 5. 10.0KiB allocated for chunks. 10.0KiB in use in bin. 9.8KiB client-requested in use in bin.
2019-07-21 23:23:16.230237: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230250: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230275: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): 	Total Chunks: 5, Chunks in use: 4. 97.8KiB allocated for chunks. 79.0KiB in use in bin. 78.1KiB client-requested in use in bin.
2019-07-21 23:23:16.230286: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230299: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): 	Total Chunks: 1, Chunks in use: 1. 78.2KiB allocated for chunks. 78.2KiB in use in bin. 78.1KiB client-requested in use in bin.
2019-07-21 23:23:16.230311: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230325: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): 	Total Chunks: 2, Chunks in use: 1. 833.0KiB allocated for chunks. 390.8KiB in use in bin. 390.6KiB client-requested in use in bin.
2019-07-21 23:23:16.230336: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230348: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): 	Total Chunks: 3, Chunks in use: 3. 4.49MiB allocated for chunks. 4.49MiB in use in bin. 4.49MiB client-requested in use in bin.
2019-07-21 23:23:16.230361: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): 	Total Chunks: 2, Chunks in use: 1. 4.50MiB allocated for chunks. 2.00MiB in use in bin. 1.50MiB client-requested in use in bin.
2019-07-21 23:23:16.230373: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): 	Total Chunks: 1, Chunks in use: 0. 5.01MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230386: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230398: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230409: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230420: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230434: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230444: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:16.230454: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 29.91MiB was 16.00MiB, Chunk State:
2019-07-21 23:23:16.230463: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-07-21 23:23:16.230473: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3b73000000 next 15 of size 1568000
2019-07-21 23:23:16.230484: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f3b7317ed00 next<PHONE_NUMBER>3709551615 of size 2626304
2019-07-21 23:23:16.230493: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-07-21 23:23:16.230502: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3b73400000 next 19 of size 1568000
2019-07-21 23:23:16.230511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3b7357ed00 next 27 of size 1568000
2019-07-21 23:23:16.230521: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f3b736fda00 next<PHONE_NUMBER>3709551615 of size 5252608
2019-07-21 23:23:16.230529: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576
2019-07-21 23:23:16.230540: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000000 next 1 of size 256
2019-07-21 23:23:16.230555: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000100 next 2 of size 256
2019-07-21 23:23:16.230570: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000200 next 3 of size 2048
2019-07-21 23:23:16.230583: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000a00 next 4 of size 256
2019-07-21 23:23:16.230597: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000b00 next 5 of size 256
2019-07-21 23:23:16.230611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000c00 next 6 of size 256
2019-07-21 23:23:16.230625: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf000d00 next 7 of size 2048
2019-07-21 23:23:16.230654: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf001500 next 8 of size 256
2019-07-21 23:23:16.230669: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf001600 next 10 of size 20224
2019-07-21 23:23:16.230684: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf006500 next 11 of size 1280
2019-07-21 23:23:16.230697: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf006a00 next 12 of size 256
2019-07-21 23:23:16.230709: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf006b00 next 13 of size 256
2019-07-21 23:23:16.230723: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf006c00 next 16 of size 2048
2019-07-21 23:23:16.230735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf007400 next 29 of size 256
2019-07-21 23:23:16.230747: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf007500 next 30 of size 256
2019-07-21 23:23:16.230760: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf007600 next 32 of size 256
2019-07-21 23:23:16.230773: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf007700 next 33 of size 256
2019-07-21 23:23:16.230785: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f3bdf007800 next 17 of size 19200
2019-07-21 23:23:16.230796: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf00c300 next 20 of size 256
2019-07-21 23:23:16.230809: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf00c400 next 21 of size 20224
2019-07-21 23:23:16.230823: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf011300 next 22 of size 20224
2019-07-21 23:23:16.230836: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf016200 next 23 of size 2048
2019-07-21 23:23:16.230848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf016a00 next 24 of size 2048
2019-07-21 23:23:16.230861: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf017200 next 25 of size 256
2019-07-21 23:23:16.230874: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf017300 next 26 of size 256
2019-07-21 23:23:16.230886: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf017400 next 28 of size 20224
2019-07-21 23:23:16.230899: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf01c300 next 31 of size 400128
2019-07-21 23:23:16.230911: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf07de00 next 35 of size 80128
2019-07-21 23:23:16.230924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f3bdf091700 next<PHONE_NUMBER>3709551615 of size 452864
2019-07-21 23:23:16.230938: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2097152
2019-07-21 23:23:16.230953: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f3bdf400000 next<PHONE_NUMBER>3709551615 of size 2097152
2019-07-21 23:23:16.230967: I tensorflow/core/common_runtime/bfc_allocator.cc:809]      Summary of in-use Chunks by size:
2019-07-21 23:23:16.230983: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 15 Chunks of size 256 totalling 3.8KiB
2019-07-21 23:23:16.230995: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-07-21 23:23:16.231007: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 2048 totalling 10.0KiB
2019-07-21 23:23:16.231023: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 20224 totalling 79.0KiB
2019-07-21 23:23:16.231039: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 80128 totalling 78.2KiB
2019-07-21 23:23:16.231053: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 400128 totalling 390.8KiB
2019-07-21 23:23:16.231068: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 1568000 totalling 4.49MiB
2019-07-21 23:23:16.231085: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 2097152 totalling 2.00MiB
2019-07-21 23:23:16.231102: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 7.04MiB
2019-07-21 23:23:16.231111: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 15728640 memory_limit_: 178782208 available bytes: 163053568 curr_region_allocation_bytes_: 33554432
2019-07-21 23:23:16.231125: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit:                   178782208
InUse:                     7377664
MaxInUse:                 10028288
NumAllocs:                      61
MaxAllocSize:              2626304

2019-07-21 23:23:16.231150: W tensorflow/core/common_runtime/bfc_allocator.cc:319] **********________________*********************_________________________________****__***********xxx
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
	 [[layer1/Wx_plus_b/add/_25]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tensorboard.py", line 191, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "tensorboard.py", line 164, in main
    train()
  File "tensorboard.py", line 139, in train
    summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
	 [[layer1/Wx_plus_b/add/_25]]
0 successful operations.
0 derived errors ignored.
root@e71bda560638:/apps/apps/gpu_benchmarks# python tensorboard.py
WARNING: Logging before flag parsing goes to stderr.
W0721 23:23:48.399826<PHONE_NUMBER>13568 __init__.py:308] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0721 23:23:48.418498<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:191: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W0721 23:23:48.418813<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:161: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W0721 23:23:48.418894<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:162: The name tf.gfile.DeleteRecursively is deprecated. Please use tf.io.gfile.rmtree instead.

W0721 23:23:48.419052<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:163: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0721 23:23:48.419144<PHONE_NUMBER>13568 deprecation.py:323] From tensorboard.py:20: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0721 23:23:48.419198<PHONE_NUMBER>13568 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
W0721 23:23:48.419269<PHONE_NUMBER>13568 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
W0721 23:23:48.555204<PHONE_NUMBER>13568 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
W0721 23:23:48.555730<PHONE_NUMBER>13568 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
W0721 23:23:48.580618<PHONE_NUMBER>13568 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0721 23:23:48.672224<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:22: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.

2019-07-21 23:23:48.672475: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-21 23:23:48.676436: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-21 23:23:48.676919: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e235e7cac0 executing computations on platform Host. Devices:
2019-07-21 23:23:48.676941: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-21 23:23:48.677678: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-21 23:23:48.699374: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.699807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
2019-07-21 23:23:48.699921: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:23:48.701017: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:23:48.702035: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-07-21 23:23:48.702201: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-07-21 23:23:48.703269: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-07-21 23:23:48.704016: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-07-21 23:23:48.706595: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-21 23:23:48.706691: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.707180: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.707594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-21 23:23:48.707623: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:23:48.767926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-21 23:23:48.767946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-07-21 23:23:48.767953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-07-21 23:23:48.768148: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.768525: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.768871: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:23:48.769198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 161 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-21 23:23:48.770262: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e237153f80 executing computations on platform CUDA. Devices:
2019-07-21 23:23:48.770273: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
W0721 23:23:48.770924<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:27: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0721 23:23:48.772640<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:32: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0721 23:23:48.773568<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:37: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0721 23:23:48.778774<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:49: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0721 23:23:48.786193<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:55: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0721 23:23:48.801232<PHONE_NUMBER>13568 deprecation.py:506] From tensorboard.py:84: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0721 23:23:48.833851<PHONE_NUMBER>13568 deprecation.py:323] From tensorboard.py:100: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

W0721 23:23:48.846037<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:106: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0721 23:23:48.914657<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:118: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W0721 23:23:48.915626<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:119: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0721 23:23:48.934774<PHONE_NUMBER>13568 deprecation_wrapper.py:119] From tensorboard.py:121: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

2019-07-21 23:23:49.254414: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:23:49.324353: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-21 23:23:49.330331: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-21 23:23:49.331837: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.332226: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.332706: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.333106: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.333517: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.333936: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.334331: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.334722: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.335239: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.335604: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.336030: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.336411: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.336789: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.337165: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.337532: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.337899: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.338252: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:49.338602: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:59.333998: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:59.336335: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:23:59.336404: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.91MiB (rounded to 31360000).  Current allocation summary follows.
2019-07-21 23:23:59.336453: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): 	Total Chunks: 15, Chunks in use: 15. 3.8KiB allocated for chunks. 3.8KiB in use in bin. 240B client-requested in use in bin.
2019-07-21 23:23:59.336491: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): 	Total Chunks: 2, Chunks in use: 0. 1.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336534: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-07-21 23:23:59.336588: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): 	Total Chunks: 5, Chunks in use: 5. 10.0KiB allocated for chunks. 10.0KiB in use in bin. 9.8KiB client-requested in use in bin.
2019-07-21 23:23:59.336636: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336683: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336711: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): 	Total Chunks: 5, Chunks in use: 4. 96.5KiB allocated for chunks. 79.0KiB in use in bin. 78.1KiB client-requested in use in bin.
2019-07-21 23:23:59.336719: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336725: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): 	Total Chunks: 1, Chunks in use: 1. 78.2KiB allocated for chunks. 78.2KiB in use in bin. 78.1KiB client-requested in use in bin.
2019-07-21 23:23:59.336732: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336740: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): 	Total Chunks: 2, Chunks in use: 1. 833.0KiB allocated for chunks. 390.8KiB in use in bin. 390.6KiB client-requested in use in bin.
2019-07-21 23:23:59.336748: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336758: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): 	Total Chunks: 3, Chunks in use: 2. 4.49MiB allocated for chunks. 2.99MiB in use in bin. 2.99MiB client-requested in use in bin.
2019-07-21 23:23:59.336765: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): 	Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 2.99MiB client-requested in use in bin.
2019-07-21 23:23:59.336774: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): 	Total Chunks: 1, Chunks in use: 0. 5.01MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336782: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336789: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336796: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336803: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336811: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336818: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:23:59.336827: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 29.91MiB was 16.00MiB, Chunk State:
2019-07-21 23:23:59.336832: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-07-21 23:23:59.336840: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1bdf000000 next 16 of size 1568000
2019-07-21 23:23:59.336846: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1bdf17ed00 next<PHONE_NUMBER>3709551615 of size 2626304
2019-07-21 23:23:59.336850: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-07-21 23:23:59.336856: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1bdf400000 next 20 of size 1568000
2019-07-21 23:23:59.336862: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1bdf57ed00 next 28 of size 1568000
2019-07-21 23:23:59.336869: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1bdf6fda00 next<PHONE_NUMBER>3709551615 of size 5252608
2019-07-21 23:23:59.336875: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576
2019-07-21 23:23:59.336880: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000000 next 1 of size 256
2019-07-21 23:23:59.336886: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000100 next 2 of size 256
2019-07-21 23:23:59.336892: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000200 next 3 of size 2048
2019-07-21 23:23:59.336898: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000a00 next 4 of size 256
2019-07-21 23:23:59.336903: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000b00 next 5 of size 256
2019-07-21 23:23:59.336909: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000c00 next 6 of size 256
2019-07-21 23:23:59.336915: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f000d00 next 7 of size 2048
2019-07-21 23:23:59.336920: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f001500 next 8 of size 256
2019-07-21 23:23:59.336926: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f001600 next 10 of size 20224
2019-07-21 23:23:59.336933: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f006500 next 11 of size 1280
2019-07-21 23:23:59.336939: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f006a00 next 29 of size 256
2019-07-21 23:23:59.336945: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f006b00 next 30 of size 256
2019-07-21 23:23:59.336951: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1c4f006c00 next 34 of size 768
2019-07-21 23:23:59.336957: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f006f00 next 35 of size 256
2019-07-21 23:23:59.336963: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1c4f007000 next 38 of size 512
2019-07-21 23:23:59.336968: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f007200 next 39 of size 256
2019-07-21 23:23:59.336974: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1c4f007300 next 12 of size 17920
2019-07-21 23:23:59.336980: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f00b900 next 13 of size 256
2019-07-21 23:23:59.336986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f00ba00 next 14 of size 256
2019-07-21 23:23:59.336992: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f00bb00 next 17 of size 20224
2019-07-21 23:23:59.336998: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f010a00 next 18 of size 20224
2019-07-21 23:23:59.337004: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f015900 next 21 of size 256
2019-07-21 23:23:59.337010: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f015a00 next 22 of size 2048
2019-07-21 23:23:59.337016: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f016200 next 23 of size 2048
2019-07-21 23:23:59.337022: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f016a00 next 24 of size 256
2019-07-21 23:23:59.337028: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f016b00 next 25 of size 2048
2019-07-21 23:23:59.337034: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f017300 next 26 of size 256
2019-07-21 23:23:59.337040: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f017400 next 27 of size 20224
2019-07-21 23:23:59.337046: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f01c300 next 32 of size 400128
2019-07-21 23:23:59.337052: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f07de00 next 36 of size 80128
2019-07-21 23:23:59.337058: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7f1c4f091700 next<PHONE_NUMBER>3709551615 of size 452864
2019-07-21 23:23:59.337064: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2097152
2019-07-21 23:23:59.337070: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f1c4f400000 next<PHONE_NUMBER>3709551615 of size 2097152
2019-07-21 23:23:59.337076: I tensorflow/core/common_runtime/bfc_allocator.cc:809]      Summary of in-use Chunks by size:
2019-07-21 23:23:59.337084: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 15 Chunks of size 256 totalling 3.8KiB
2019-07-21 23:23:59.337091: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-07-21 23:23:59.337097: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 2048 totalling 10.0KiB
2019-07-21 23:23:59.337104: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 20224 totalling 79.0KiB
2019-07-21 23:23:59.337111: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 80128 totalling 78.2KiB
2019-07-21 23:23:59.337118: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 400128 totalling 390.8KiB
2019-07-21 23:23:59.337124: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 1568000 totalling 2.99MiB
2019-07-21 23:23:59.337130: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 2097152 totalling 2.00MiB
2019-07-21 23:23:59.337137: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 2626304 totalling 2.50MiB
2019-07-21 23:23:59.337143: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 8.04MiB
2019-07-21 23:23:59.337149: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 15728640 memory_limit_: 168951808 available bytes: 153223168 curr_region_allocation_bytes_: 33554432
2019-07-21 23:23:59.337158: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit:                   168951808
InUse:                     8435968
MaxInUse:                 10029568
NumAllocs:                      61
MaxAllocSize:              2626304

2019-07-21 23:23:59.337168: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ********************xxxxxxx_________***********_________________________________****__***********xxx
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
	 [[layer1/Wx_plus_b/add/_25]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tensorboard.py", line 191, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "tensorboard.py", line 164, in main
    train()
  File "tensorboard.py", line 139, in train
    summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_input/x-input_0_1}}]]
	 [[layer1/Wx_plus_b/add/_25]]
0 successful operations.
0 derived errors ignored.
root@e71bda560638:/apps/apps/gpu_benchmarks# ls
__pycache__  benchmark.py  tensorboard.py
root@e71bda560638:/apps/apps/gpu_benchmarks# python benchmark.py gpu 10000
WARNING: Logging before flag parsing goes to stderr.
W0721 23:26:34.769160<PHONE_NUMBER>67552 __init__.py:308] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0721 23:26:34.788027<PHONE_NUMBER>67552 deprecation_wrapper.py:119] From benchmark.py:18: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0721 23:26:34.792794<PHONE_NUMBER>67552 deprecation_wrapper.py:119] From benchmark.py:24: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0721 23:26:34.792872<PHONE_NUMBER>67552 deprecation_wrapper.py:119] From benchmark.py:24: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2019-07-21 23:26:34.792978: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-21 23:26:34.818743: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-21 23:26:34.819354: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560751921f80 executing computations on platform Host. Devices:
2019-07-21 23:26:34.819370: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-21 23:26:34.819978: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-21 23:26:34.834675: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.835206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
2019-07-21 23:26:34.835319: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:26:34.836303: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:26:34.837230: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-07-21 23:26:34.837382: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-07-21 23:26:34.838334: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-07-21 23:26:34.838907: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-07-21 23:26:34.840924: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-21 23:26:34.841007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.841449: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.841823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-21 23:26:34.841845: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-21 23:26:34.894491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-21 23:26:34.894509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-07-21 23:26:34.894513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-07-21 23:26:34.894703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.895050: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.895367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-21 23:26:34.895665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 159 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-21 23:26:34.896687: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560752b2a3a0 executing computations on platform CUDA. Devices:
2019-07-21 23:26:34.896697: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
2019-07-21 23:26:34.897194: I tensorflow/core/common_runtime/direct_session.cc:296] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device

random_uniform/RandomUniform: (RandomUniform): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897731: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/RandomUniform: (RandomUniform)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform/sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897741: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/sub: (Sub)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform/mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897746: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/mul: (Mul)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform: (Add): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897751: I tensorflow/core/common_runtime/placer.cc:54] random_uniform: (Add)/job:localhost/replica:0/task:0/device:GPU:0
transpose: (Transpose): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897757: I tensorflow/core/common_runtime/placer.cc:54] transpose: (Transpose)/job:localhost/replica:0/task:0/device:GPU:0
MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897761: I tensorflow/core/common_runtime/placer.cc:54] MatMul: (MatMul)/job:localhost/replica:0/task:0/device:GPU:0
Sum: (Sum): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897771: I tensorflow/core/common_runtime/placer.cc:54] Sum: (Sum)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform/shape: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897781: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/shape: (Const)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform/min: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897790: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/min: (Const)/job:localhost/replica:0/task:0/device:GPU:0
random_uniform/max: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897802: I tensorflow/core/common_runtime/placer.cc:54] random_uniform/max: (Const)/job:localhost/replica:0/task:0/device:GPU:0
transpose/perm: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897810: I tensorflow/core/common_runtime/placer.cc:54] transpose/perm: (Const)/job:localhost/replica:0/task:0/device:GPU:0
Const: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.897817: I tensorflow/core/common_runtime/placer.cc:54] Const: (Const)/job:localhost/replica:0/task:0/device:GPU:0
2019-07-21 23:26:34.901219: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-21 23:26:35.082216: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 159.69M (167444480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.082576: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 143.72M (150700032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.082925: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 129.35M (135630080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.083263: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 116.41M (122067200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.083600: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 104.77M (109860608 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.083937: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 94.29M (98874624 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.084275: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 84.86M (88987392 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:35.084614: E tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 76.38M (80088832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-07-21 23:26:45.085467: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 381.47MiB (rounded to 400000000).  Current allocation summary follows.
2019-07-21 23:26:45.085552: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085613: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085661: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-07-21 23:26:45.085701: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085739: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085779: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085816: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085857: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085897: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085936: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.085975: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086012: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086043: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086078: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086122: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086154: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086187: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086217: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086247: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): 	Total Chunks: 1, Chunks in use: 0. 68.74MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086281: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086317: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-21 23:26:45.086361: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 381.47MiB was 256.00MiB, Chunk State:
2019-07-21 23:26:45.086394: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 72080128
2019-07-21 23:26:45.086432: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fa540000000 next 1 of size 1280
2019-07-21 23:26:45.086468: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free  at 0x7fa540000500 next<PHONE_NUMBER>3709551615 of size 72078848
2019-07-21 23:26:45.086493: I tensorflow/core/common_runtime/bfc_allocator.cc:809]      Summary of in-use Chunks by size:
2019-07-21 23:26:45.086521: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-07-21 23:26:45.086546: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 1.2KiB
2019-07-21 23:26:45.086575: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 72080128 memory_limit_: 167444480 available bytes: 95364352 curr_region_allocation_bytes_: 334888960
2019-07-21 23:26:45.086610: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit:                   167444480
InUse:                        1280
MaxInUse:                     1280
NumAllocs:                       1
MaxAllocSize:                 1280

2019-07-21 23:26:45.086709: W tensorflow/core/common_runtime/bfc_allocator.cc:319] *___________________________________________________________________________________________________
2019-07-21 23:26:45.086813: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at random_op.cc:76 : Resource exhausted: OOM when allocating tensor with shape[10000,10000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[10000,10000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node random_uniform/RandomUniform}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[10000,10000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node random_uniform/RandomUniform}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Sum/_1]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "benchmark.py", line 25, in <module>
    result = session.run(sum_operation)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[10000,10000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node random_uniform/RandomUniform (defined at benchmark.py:18) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[10000,10000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node random_uniform/RandomUniform (defined at benchmark.py:18) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Sum/_1]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Original stack trace for 'random_uniform/RandomUniform':
  File "benchmark.py", line 18, in <module>
    random_matrix = tf.random_uniform(shape=shape, minval=0, maxval=1)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform
    rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform
    name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Addt'l errors:

root@85fe2dc4f4c7:/# cd /apps/apps/gpu_benchmarks
root@85fe2dc4f4c7:/apps/apps/gpu_benchmarks# python tensorboard.py

WARNING: Logging before flag parsing goes to stderr.
W0722 01:03:54.204519<PHONE_NUMBER>30112 __init__.py:308] Limited tf.compat.v2.summary API due to missing TensorBoard installation.
W0722 01:03:54.223043<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:191: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W0722 01:03:54.223351<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:161: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.

W0722 01:03:54.223432<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:163: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0722 01:03:54.223577<PHONE_NUMBER>30112 deprecation.py:323] From tensorboard.py:20: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0722 01:03:54.223633<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
W0722 01:03:54.223726<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
W0722 01:03:55.224481<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
W0722 01:03:55.593722<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
W0722 01:03:55.596628<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
W0722 01:03:56.321027<PHONE_NUMBER>30112 deprecation.py:323] From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
W0722 01:03:56.457564<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:22: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.

2019-07-22 01:03:56.457849: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-07-22 01:03:56.462771: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-22 01:03:56.463093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d6dbc67d40 executing computations on platform Host. Devices:
2019-07-22 01:03:56.463107: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-07-22 01:03:56.463852: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-07-22 01:03:56.491958: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.492486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
2019-07-22 01:03:56.492643: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-22 01:03:56.493766: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-07-22 01:03:56.494731: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-07-22 01:03:56.494898: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-07-22 01:03:56.495933: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-07-22 01:03:56.496564: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-07-22 01:03:56.498758: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-07-22 01:03:56.498845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.499567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.500184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-07-22 01:03:56.500229: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-07-22 01:03:56.583505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 01:03:56.583522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-07-22 01:03:56.583529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-07-22 01:03:56.583817: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.584268: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.584688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 01:03:56.585086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9514 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-07-22 01:03:56.585977: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d6dcf3e350 executing computations on platform CUDA. Devices:
2019-07-22 01:03:56.585989: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
W0722 01:03:56.586563<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:27: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0722 01:03:56.588290<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:32: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0722 01:03:56.589203<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:37: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

W0722 01:03:56.594342<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:49: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0722 01:03:56.601807<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:55: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0722 01:03:56.617331<PHONE_NUMBER>30112 deprecation.py:506] From tensorboard.py:84: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0722 01:03:56.648849<PHONE_NUMBER>30112 deprecation.py:323] From tensorboard.py:100: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

W0722 01:03:56.660882<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:106: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0722 01:03:56.728627<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:118: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W0722 01:03:56.729586<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:119: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0722 01:03:56.748252<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:121: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

2019-07-22 01:03:57.061654: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
Accuracy at step 0: 0.0829
Accuracy at step 10: 0.6891
Accuracy at step 20: 0.7829
Accuracy at step 30: 0.8521
Accuracy at step 40: 0.8806
Accuracy at step 50: 0.8902
Accuracy at step 60: 0.9007
Accuracy at step 70: 0.9038
Accuracy at step 80: 0.9052
Accuracy at step 90: 0.9078
W0722 01:03:59.865827<PHONE_NUMBER>30112 deprecation_wrapper.py:119] From tensorboard.py:144: The name tf.RunOptions is deprecated. Please use tf.compat.v1.RunOptions instead.

2019-07-22 01:03:59.866129: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:03:59.866604: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcupti.so.10.1
2019-07-22 01:03:59.967085: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2019-07-22 01:04:00.018909: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:00.019111: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 99
Accuracy at step 100: 0.9156
Accuracy at step 110: 0.9158
Accuracy at step 120: 0.9211
Accuracy at step 130: 0.9229
Accuracy at step 140: 0.9238
Accuracy at step 150: 0.9239
Accuracy at step 160: 0.9256
Accuracy at step 170: 0.924
Accuracy at step 180: 0.928
Accuracy at step 190: 0.9292
2019-07-22 01:04:02.574073: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:02.574520: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2019-07-22 01:04:02.586044: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:02.586146: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 199
Accuracy at step 200: 0.9299
Accuracy at step 210: 0.9304
Accuracy at step 220: 0.9352
Accuracy at step 230: 0.936
Accuracy at step 240: 0.9397
Accuracy at step 250: 0.9371
Accuracy at step 260: 0.939
Accuracy at step 270: 0.9396
Accuracy at step 280: 0.9408
Accuracy at step 290: 0.9407
2019-07-22 01:04:05.113320: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:05.113360: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:05.124815: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:05.124918: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 299
Accuracy at step 300: 0.9424
Accuracy at step 310: 0.9438
Accuracy at step 320: 0.9465
Accuracy at step 330: 0.9465
Accuracy at step 340: 0.9477
Accuracy at step 350: 0.9469
Accuracy at step 360: 0.9475
Accuracy at step 370: 0.9485
Accuracy at step 380: 0.944
Accuracy at step 390: 0.9491
2019-07-22 01:04:07.640317: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:07.640363: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:07.651981: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:07.652109: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 399
Accuracy at step 400: 0.9495
Accuracy at step 410: 0.9509
Accuracy at step 420: 0.9501
Accuracy at step 430: 0.9522
Accuracy at step 440: 0.9484
Accuracy at step 450: 0.9542
Accuracy at step 460: 0.9548
Accuracy at step 470: 0.9518
Accuracy at step 480: 0.9528
Accuracy at step 490: 0.9535
2019-07-22 01:04:10.202758: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:10.202795: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:10.214207: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:10.214319: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 499
Accuracy at step 500: 0.9563
Accuracy at step 510: 0.9533
Accuracy at step 520: 0.9577
Accuracy at step 530: 0.9565
Accuracy at step 540: 0.9559
Accuracy at step 550: 0.9563
Accuracy at step 560: 0.9567
Accuracy at step 570: 0.9596
Accuracy at step 580: 0.9609
Accuracy at step 590: 0.9603
2019-07-22 01:04:12.781947: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:12.781992: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:12.793775: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:12.793883: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 599
Accuracy at step 600: 0.9623
Accuracy at step 610: 0.9625
Accuracy at step 620: 0.9619
Accuracy at step 630: 0.9628
Accuracy at step 640: 0.9616
Accuracy at step 650: 0.9607
Accuracy at step 660: 0.9625
Accuracy at step 670: 0.9608
Accuracy at step 680: 0.9616
Accuracy at step 690: 0.963
2019-07-22 01:04:15.406032: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:15.406075: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:15.417561: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:15.417671: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 699
Accuracy at step 700: 0.9616
Accuracy at step 710: 0.964
Accuracy at step 720: 0.9638
Accuracy at step 730: 0.9652
Accuracy at step 740: 0.9642
Accuracy at step 750: 0.964
Accuracy at step 760: 0.961
Accuracy at step 770: 0.9629
Accuracy at step 780: 0.9634
Accuracy at step 790: 0.9651
2019-07-22 01:04:17.950988: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:17.951028: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:17.962709: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:17.962825: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 799
Accuracy at step 800: 0.9646
Accuracy at step 810: 0.9643
Accuracy at step 820: 0.9654
Accuracy at step 830: 0.9651
Accuracy at step 840: 0.9654
Accuracy at step 850: 0.9671
Accuracy at step 860: 0.9632
Accuracy at step 870: 0.9674
Accuracy at step 880: 0.9665
Accuracy at step 890: 0.9685
2019-07-22 01:04:20.496482: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:20.496523: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:20.508072: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:20.508190: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 899
Accuracy at step 900: 0.9676
Accuracy at step 910: 0.9664
Accuracy at step 920: 0.9693
Accuracy at step 930: 0.9689
Accuracy at step 940: 0.9692
Accuracy at step 950: 0.9674
Accuracy at step 960: 0.9683
Accuracy at step 970: 0.9684
Accuracy at step 980: 0.9684
Accuracy at step 990: 0.9704
2019-07-22 01:04:23.068611: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-07-22 01:04:23.068659: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI_ERROR_NOT_INITIALIZED
2019-07-22 01:04:23.080319: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-07-22 01:04:23.080431: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI_ERROR_INVALID_PARAMETER
Adding run metadata for 999

The odd thing is, tensorboard.py actually runs w/ all those errors, but it executes very slowly:

Potential fix in the works:
Going to test sudo chown -R user anaconda3 during docker build stage to correct and re-$PATH Tensorboard to Conda env.

Any update on this? I am experiencing a similar issue with TF 2.1.0.

I have been meaning to get to this, but with upcoming holidays I dont have
time. I am sorry.
On Fri, Dec 6, 2019 at 4:33 PM Lizzie<EMAIL_ADDRESS>wrote:

Any update on this? I am experiencing a similar issue with TF 2.1.0.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/joehoeller/Anaconda-CUDA-Accelerated-TensorFlowGPU-Development-Environment/issues/1?email_source=notifications&email_token=ABHVQHD5WGBGRWT5KG42PU3QXLHNFA5CNFSM4IFTXV6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGFRNFY#issuecomment-562763415,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABHVQHFNGXNK6SEQ7X3B4MLQXLHNFANCNFSM4IFTXV6A
.