TF does not seem to map all available VRAM in multi-gpu config

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.7.0-3-g024aecf414 1.7.0
Python version: 3.5.2
Bazel version (if compiling from source): /
GCC/Compiler version (if compiling from source): /
CUDA/cuDNN version: 9.0/7.0.4
GPU model and memory: Tesla K80 x2
Exact command to reproduce: See script in my comment below to reproduce

Describe the problem
I am facing an OOM error (Out of memory) error while running a large TF app on a multi-gpu config (2 Tesla K80).
The memory Limit displayed by the error log accounts for the VRAM of a single GPU (12GB), while I was expecting to account both (24GB):
Limit:<PHONE_NUMBER>9
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>8
NumAllocs:                   41821
MaxAllocSize:<PHONE_NUMBER>

I am 100% certain both GPUs are used, since they both show up in the log:
2018-04-10 12:08:37.937431: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use:
 AVX2 FMA
2018-04-10 12:08:38.052749: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must
 be at least one NUMA node, so returning NUMA node zero
2018-04-10 12:08:38.053215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2018-04-10 12:08:38.129345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must
 be at least one NUMA node, so returning NUMA node zero
2018-04-10 12:08:38.129745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 1 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:05.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-04-10 12:08:38.130022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0, 1
2018-04-10 12:08:38.658026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-10 12:08:38.658156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 1
2018-04-10 12:08:38.658180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N Y
2018-04-10 12:08:38.658201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 1:   Y N
2018-04-10 12:08:38.658793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10
749 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2018-04-10 12:08:38.853578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10
765 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:00:05.0, compute capability: 3.7)

This behavior is inconsistent with TF documentation: https://www.tensorflow.org/programmers_guide/using_gpu#allowing_gpu_memory_growth

By default, TensorFlow maps nearly all of the GPU memory of all GPUs

logs
2018-04-10 12:09:40.993943: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.42GiB.  Current allocation summary follows.
2018-04-10 12:09:40.994041: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256):   Total Chunks: 76, Chunks in use: 76. 19.0KiB allocated for chunks. 19.0KiB in use in bin. 1.8KiB client-requested in use in bin.
2018-04-10 12:09:40.994060: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512):   Total Chunks: 6, Chunks in use: 6. 3.0KiB allocated for chunks. 3.0KiB in use in bin. 3.0KiB client-requested in use in bin.
2018-04-10 12:09:40.994075: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024):  Total Chunks: 10, Chunks in use: 10. 11.0KiB allocated for chunks. 11.0KiB in use in bin. 10.0KiB client-requested in use in bin.
2018-04-10 12:09:40.994086: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048):  Total Chunks: 14, Chunks in use: 14. 28.0KiB allocated for chunks. 28.0KiB in use in bin. 28.0KiB client-requested in use in bin.
2018-04-10 12:09:40.994097: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096):  Total Chunks: 3, Chunks in use: 2. 20.0KiB allocated for chunks. 13.5KiB in use in bin. 13.5KiB client-requested in use in bin.
2018-04-10 12:09:40.994107: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 12:09:40.994118: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384):         Total Chunks: 5, Chunks in use: 5. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin.
2018-04-10 12:09:40.994126: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 12:09:40.994137: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536):         Total Chunks: 9, Chunks in use: 9. 704.2KiB allocated for chunks. 704.2KiB in use in bin. 600.6KiB client-requested in use in bin.
2018-04-10 12:09:40.994148: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072):        Total Chunks: 6, Chunks in use: 5. 1.03MiB allocated for chunks. 919.5KiB in use in bin. 919.3KiB client-requested in use in bin.
2018-04-10 12:09:40.994158: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144):        Total Chunks: 6, Chunks in use: 6. 1.67MiB allocated for chunks. 1.67MiB in use in bin. 1.66MiB client-requested in use in bin.
2018-04-10 12:09:40.994169: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288):        Total Chunks: 8, Chunks in use: 7. 6.09MiB allocated for chunks. 5.27MiB in use in bin. 5.26MiB client-requested in use in bin.
2018-04-10 12:09:40.994179: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576):       Total Chunks: 4, Chunks in use: 4. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.18MiB client-requested in use in bin.
2018-04-10 12:09:40.994190: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152):       Total Chunks: 4, Chunks in use: 4. 9.00MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin.
2018-04-10 12:09:40.994200: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304):       Total Chunks: 2, Chunks in use: 2. 9.00MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin.
2018-04-10 12:09:40.994213: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608):       Total Chunks: 42, Chunks in use: 39. 467.91MiB allocated for chunks. 433.54MiB in use in bin. 426.99MiB client-requested in use in bin.
2018-04-10 12:09:40.994224: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216):      Total Chunks: 10, Chunks in use: 10. 170.47MiB allocated for chunks. 170.47MiB in use in bin. 157.25MiB client-requested in use in bin.
2018-04-10 12:09:40.994233: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432):      Total Chunks: 11, Chunks in use: 10. 404.75MiB allocated for chunks. 369.16MiB in use in bin. 350.70MiB client-requested in use in bin.
2018-04-10 12:09:40.994269: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864):      Total Chunks: 13, Chunks in use: 13. 919.70MiB allocated for chunks. 919.70MiB in use in bin. 918.65MiB client-requested in use in bin.
2018-04-10 12:09:40.994281: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728):     Total Chunks: 3, Chunks in use: 3. 423.99MiB allocated for chunks. 423.99MiB in use in bin. 353.33MiB client-requested in use in bin.
2018-04-10 12:09:40.994291: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456):     Total Chunks: 4, Chunks in use: 3. 8.13GiB allocated for chunks. 4.97GiB in use in bin. 4.97GiB client-requested in use in bin.
2018-04-10 12:09:40.994300: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 4.42GiB was 256.00MiB, Chunk State:
2018-04-10 12:09:40.994314: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 3.17GiB | Requested Size: 2.13MiB | in_use: 0, prev:   Size: 70.67MiB | Requested Size: 70.67MiB | in_use: 1
2018-04-10 12:09:40.994325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280000 of size 1280
2018-04-10 12:09:40.994331: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280500 of size 1280
2018-04-10 12:09:40.994339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280a00 of size 256
2018-04-10 12:09:40.994363: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280b00 of size 256
2018-04-10 12:09:40.994371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280c00 of size 256
2018-04-10 12:09:40.994379: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280d00 of size 256
2018-04-10 12:09:40.994383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280e00 of size 256
2018-04-10 12:09:40.994390: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a280f00 of size 256
2018-04-10 12:09:40.994398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281000 of size 256
2018-04-10 12:09:40.994406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281100 of size 512
2018-04-10 12:09:40.994414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281300 of size 256
2018-04-10 12:09:40.994422: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281400 of size 256
2018-04-10 12:09:40.994429: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281500 of size 256
2018-04-10 12:09:40.994437: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281600 of size 256
2018-04-10 12:09:40.994444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281700 of size 1024
2018-04-10 12:09:40.994452: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281b00 of size 256
2018-04-10 12:09:40.994460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281c00 of size 256
2018-04-10 12:09:40.994478: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281d00 of size 256
2018-04-10 12:09:40.994485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281e00 of size 256
2018-04-10 12:09:40.994494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a281f00 of size 2048
2018-04-10 12:09:40.994518: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282700 of size 256
2018-04-10 12:09:40.994523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282800 of size 256
2018-04-10 12:09:40.994528: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282900 of size 256
2018-04-10 12:09:40.994532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282a00 of size 256
2018-04-10 12:09:40.994537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282b00 of size 256
2018-04-10 12:09:40.994545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282c00 of size 256
2018-04-10 12:09:40.994554: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282d00 of size 256
2018-04-10 12:09:40.994559: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282e00 of size 256
2018-04-10 12:09:40.994567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a282f00 of size 256
2018-04-10 12:09:40.994574: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283000 of size 256
2018-04-10 12:09:40.994580: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283100 of size 256
2018-04-10 12:09:40.994588: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283200 of size 256
2018-04-10 12:09:40.994593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283300 of size 256
2018-04-10 12:09:40.994600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283400 of size 256
2018-04-10 12:09:40.994605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283500 of size 256
2018-04-10 12:09:40.994609: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283600 of size 256
2018-04-10 12:09:40.994615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283700 of size 256
2018-04-10 12:09:40.994628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283800 of size 512
2018-04-10 12:09:40.994635: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283a00 of size 256
2018-04-10 12:09:40.994642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283b00 of size 256
2018-04-10 12:09:40.994648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283c00 of size 256
2018-04-10 12:09:40.994655: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283d00 of size 256
2018-04-10 12:09:40.994661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a283e00 of size 1536
2018-04-10 12:09:40.994667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284400 of size 256
2018-04-10 12:09:40.994672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284500 of size 256
2018-04-10 12:09:40.994680: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284600 of size 256
2018-04-10 12:09:40.994685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284700 of size 256
2018-04-10 12:09:40.994693: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284800 of size 256
2018-04-10 12:09:40.994699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a284900 of size 2048
2018-04-10 12:09:40.994706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285100 of size 256
2018-04-10 12:09:40.994712: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285200 of size 256
2018-04-10 12:09:40.994719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285300 of size 256
2018-04-10 12:09:40.994725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285400 of size 256
2018-04-10 12:09:40.994733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285500 of size 256
2018-04-10 12:09:40.994738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285600 of size 256
2018-04-10 12:09:40.994746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285700 of size 256
2018-04-10 12:09:40.994751: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285800 of size 256
2018-04-10 12:09:40.994758: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285900 of size 256
2018-04-10 12:09:40.994764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285a00 of size 256
2018-04-10 12:09:40.994772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285b00 of size 256
2018-04-10 12:09:40.994778: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285c00 of size 256
2018-04-10 12:09:40.994785: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285d00 of size 256
2018-04-10 12:09:40.994791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285e00 of size 256
2018-04-10 12:09:40.994798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a285f00 of size 256
2018-04-10 12:09:40.994803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286000 of size 256
2018-04-10 12:09:40.994810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286100 of size 256
2018-04-10 12:09:40.994817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286200 of size 256
2018-04-10 12:09:40.994824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286300 of size 256
2018-04-10 12:09:40.994829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286400 of size 256
2018-04-10 12:09:40.994837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286500 of size 256
2018-04-10 12:09:40.994843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286600 of size 256
2018-04-10 12:09:40.994850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286700 of size 256
2018-04-10 12:09:40.994856: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286800 of size 256
2018-04-10 12:09:40.994864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286900 of size 256
2018-04-10 12:09:40.994869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286a00 of size 256
2018-04-10 12:09:40.994876: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286b00 of size 256
2018-04-10 12:09:40.994882: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286c00 of size 256
2018-04-10 12:09:40.994890: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286d00 of size 256
2018-04-10 12:09:40.994896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286e00 of size 256
2018-04-10 12:09:40.994903: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a286f00 of size 256
2018-04-10 12:09:40.994908: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a287000 of size 512
2018-04-10 12:09:40.994917: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a287200 of size 512
2018-04-10 12:09:40.994922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a287400 of size 1024
2018-04-10 12:09:40.994929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a287800 of size 1024
2018-04-10 12:09:40.994935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a287c00 of size 1024
2018-04-10 12:09:40.994942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a288000 of size 2048
2018-04-10 12:09:40.994950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a288800 of size 2048
2018-04-10 12:09:40.994959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a289000 of size 2048
2018-04-10 12:09:40.994964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a289800 of size 2048
2018-04-10 12:09:40.994968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a28a000 of size 2048
2018-04-10 12:09:40.994973: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a28a800 of size 2048
2018-04-10 12:09:40.994978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a28b000 of size 6912
2018-04-10 12:09:40.994990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a28cb00 of size 113152
2018-04-10 12:09:40.994998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2a8500 of size 256
2018-04-10 12:09:40.995008: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2a8600 of size 65536
2018-04-10 12:09:40.995018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2b8600 of size 81920
2018-04-10 12:09:40.995025: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2cc600 of size 147456
2018-04-10 12:09:40.995040: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2f0600 of size 512
2018-04-10 12:09:40.995052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a2f0800 of size 294912
2018-04-10 12:09:40.995062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a338800 of size 294912
2018-04-10 12:09:40.995074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a380800 of size 512
2018-04-10 12:09:40.995080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a380a00 of size 289536
2018-04-10 12:09:40.995086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3c7500 of size 71936
2018-04-10 12:09:40.995092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3d8e00 of size 16384
2018-04-10 12:09:40.995098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3dce00 of size 16384
2018-04-10 12:09:40.995106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3e0e00 of size 16384
2018-04-10 12:09:40.995111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3e4e00 of size 71936
2018-04-10 12:09:40.995118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a3f6700 of size 107264
2018-04-10 12:09:40.995127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a410a00 of size 589824
2018-04-10 12:09:40.995134: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a4a0a00 of size 1024
2018-04-10 12:09:40.995154: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a4a0e00 of size 2359296
2018-04-10 12:09:40.995159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a6e0e00 of size 16384
2018-04-10 12:09:40.995164: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a6e4e00 of size 65536
2018-04-10 12:09:40.995171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a6f4e00 of size 16384
2018-04-10 12:09:40.995179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a6f8e00 of size 289536
2018-04-10 12:09:40.995184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a73f900 of size 287488
2018-04-10 12:09:40.995191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a785c00 of size 71936
2018-04-10 12:09:40.995199: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a797500 of size 215552
2018-04-10 12:09:40.995207: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a7cbf00 of size 256
2018-04-10 12:09:40.995212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a7cc000 of size 256
2018-04-10 12:09:40.995219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a7cc100 of size 256
2018-04-10 12:09:40.995227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x50a7cc200 of size 6656
2018-04-10 12:09:40.995235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a7cdc00 of size 71936
2018-04-10 12:09:40.995240: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x50a7df500 of size 137472
2018-04-10 12:09:40.995247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a800e00 of size 1179648
2018-04-10 12:09:40.995253: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a920e00 of size 1024
2018-04-10 12:09:40.995260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a921200 of size 1024
2018-04-10 12:09:40.995266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50a921600 of size 2359296
2018-04-10 12:09:40.995273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50ab61600 of size 1157888
2018-04-10 12:09:40.995281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50ac7c100 of size 1201408
2018-04-10 12:09:40.995289: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50ada1600 of size 2048
2018-04-10 12:09:40.995294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50ada1e00 of size 9437184
2018-04-10 12:09:40.995301: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50b6a1e00 of size 9437184
2018-04-10 12:09:40.995306: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50bfa1e00 of size 9437184
2018-04-10 12:09:40.995314: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50c8a1e00 of size 9437184
2018-04-10 12:09:40.995319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50d1a1e00 of size 9437184
2018-04-10 12:09:40.995326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa1e00 of size 2048
2018-04-10 12:09:40.995334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa2600 of size 2048
2018-04-10 12:09:40.995342: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa2e00 of size 2048
2018-04-10 12:09:40.995347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa3600 of size 2048
2018-04-10 12:09:40.995354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa3e00 of size 2048
2018-04-10 12:09:40.995360: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa4600 of size 6912
2018-04-10 12:09:40.995367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daa6100 of size 147456
2018-04-10 12:09:40.995372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50daca100 of size 294912
2018-04-10 12:09:40.995380: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50db12100 of size 589824
2018-04-10 12:09:40.995387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50dba2100 of size 1179648
2018-04-10 12:09:40.995396: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50dcc2100 of size 2359296
2018-04-10 12:09:40.995400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50df02100 of size 4718592
2018-04-10 12:09:40.995407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50e382100 of size 9437184
2018-04-10 12:09:40.995413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50ec82100 of size 16777216
2018-04-10 12:09:40.995421: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x50fc82100 of size 16777216
2018-04-10 12:09:40.995426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x510c82100 of size 8388608
2018-04-10 12:09:40.995433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x511482100 of size 16777216
2018-04-10 12:09:40.995439: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x512482100 of size 16777216
2018-04-10 12:09:40.995446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x513482100 of size 8388608
2018-04-10 12:09:40.995454: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x513c82100 of size 8388608
2018-04-10 12:09:40.995462: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x514482100 of size 14155776
2018-04-10 12:09:40.995468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x515202100 of size 9437184
2018-04-10 12:09:40.995475: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x515b02100 of size 9437184
2018-04-10 12:09:40.995485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x516402100 of size 9437184
2018-04-10 12:09:40.995492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x516d02100 of size 9437184
2018-04-10 12:09:40.995500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x517602100 of size 2359296
2018-04-10 12:09:40.995505: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x517842100 of size 4718592
2018-04-10 12:09:40.995512: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x517cc2100 of size 9437184
2018-04-10 12:09:40.995518: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5185c2100 of size 74097664
2018-04-10 12:09:40.995525: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x51cc6c500 of size 74097664
2018-04-10 12:09:40.995530: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x521316900 of size 36773888
2018-04-10 12:09:40.995538: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x523628900 of size 36773888
2018-04-10 12:09:40.995545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x52593a900 of size 16777216
2018-04-10 12:09:40.995553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x52693a900 of size 74097664
2018-04-10 12:09:40.995560: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x52afe4d00 of size 16777216
2018-04-10 12:09:40.995567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x52bfe4d00 of size 16777216
2018-04-10 12:09:40.995573: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x52cfe4d00 of size 74097664
2018-04-10 12:09:40.995594: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53168f100 of size 74097664
2018-04-10 12:09:40.995603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x535d39500 of size 13893376
2018-04-10 12:09:40.995611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x536a79400 of size 36773888
2018-04-10 12:09:40.995618: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x538d8b400 of size 74097664
2018-04-10 12:09:40.995626: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53d435800 of size 13893376
2018-04-10 12:09:40.995634: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53e175700 of size 13893376
2018-04-10 12:09:40.995642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53eeb5600 of size 8388608
2018-04-10 12:09:40.995649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53f6b5600 of size 8388608
2018-04-10 12:09:40.995657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x53feb5600 of size 36773888
2018-04-10 12:09:40.995665: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5421c7600 of size 36773888
2018-04-10 12:09:40.995672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5444d9600 of size 13893376
2018-04-10 12:09:40.995680: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x545219500 of size 16777216
2018-04-10 12:09:40.995688: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x546219500 of size 13893376
2018-04-10 12:09:40.995695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x546f59400 of size 8388608
2018-04-10 12:09:40.995703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x547759400 of size 13893376
2018-04-10 12:09:40.995711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x548499300 of size 36773888
2018-04-10 12:09:40.995718: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54a7ab300 of size 13893376
2018-04-10 12:09:40.995723: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54b4eb200 of size 13893376
2018-04-10 12:09:40.995746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54c22b100 of size 16777216
2018-04-10 12:09:40.995754: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54d22b100 of size 13893376
2018-04-10 12:09:40.995763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54df6b000 of size 13893376
2018-04-10 12:09:40.995770: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54ecaaf00 of size 13893376
2018-04-10 12:09:40.995778: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x54f9eae00 of size 8388608
2018-04-10 12:09:40.995785: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5501eae00 of size 13893376
2018-04-10 12:09:40.995793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x550f2ad00 of size 13893376
2018-04-10 12:09:40.995800: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x551c6ac00 of size 13893376
2018-04-10 12:09:40.995808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5529aab00 of size 868352
2018-04-10 12:09:40.995813: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x552a7eb00 of size 868352
2018-04-10 12:09:40.995820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x552b52b00 of size 868352
2018-04-10 12:09:40.995828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x552c26b00 of size 11288320
2018-04-10 12:09:40.995836: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5536eaa00 of size 868352
2018-04-10 12:09:40.995843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5537bea00 of size 215552
2018-04-10 12:09:40.995851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5537f3400 of size 215552
2018-04-10 12:09:40.995856: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x553827e00 of size 868352
2018-04-10 12:09:40.995863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5538fbe00 of size 868352
2018-04-10 12:09:40.995871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x5539cfe00 of size 10857216
2018-04-10 12:09:40.995879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55442a900 of size 55573504
2018-04-10 12:09:40.995886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55792a500 of size 13893376
2018-04-10 12:09:40.995893: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55866a400 of size 13893376
2018-04-10 12:09:40.995901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5593aa300 of size 13893376
2018-04-10 12:09:40.995909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55a0ea200 of size 13893376
2018-04-10 12:09:40.995916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55ae2a100 of size 13945088
2018-04-10 12:09:40.995924: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55bb76a00 of size 13893376
2018-04-10 12:09:40.995946: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x55c8b6900 of size 74097664
2018-04-10 12:09:40.995954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x560f60d00 of size 75203584
2018-04-10 12:09:40.995961: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x565719100 of size 147089408
2018-04-10 12:09:40.995965: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x56e35f900 of size 13893376
2018-04-10 12:09:40.995969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x56f09f800 of size 296390656
2018-04-10 12:09:40.995974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x580b48800 of size 27760896
2018-04-10 12:09:40.995978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5825c2100 of size 74097664
2018-04-10 12:09:40.996009: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x586c6c500 of size 36773888
2018-04-10 12:09:40.996017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x588f7e500 of size 37323776
2018-04-10 12:09:40.996047: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x58b316900 of size 149298176
2018-04-10 12:09:40.996055: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x594178500 of size 296390656
2018-04-10 12:09:40.996063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5a5c21500 of size 74097664
2018-04-10 12:09:40.996075: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5aa2cb900 of size 36773888
2018-04-10 12:09:40.996080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5ac5dd900 of size 37323776
2018-04-10 12:09:40.996086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5ae975d00 of size 148195328
2018-04-10 12:09:40.996091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5b76ca500 of size 74097664
2018-04-10 12:09:40.996098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5bbd74900 of size 74097664
2018-04-10 12:09:40.996106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x5c041ed00 of size<PHONE_NUMBER>
2018-04-10 12:09:40.996115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x6daeaed00 of size 74097664
2018-04-10 12:09:40.996123: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x6df559100 of size<PHONE_NUMBER>
2018-04-10 12:09:40.996130: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size:
2018-04-10 12:09:40.996141: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 76 Chunks of size 256 totalling 19.0KiB
2018-04-10 12:09:40.996150: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 512 totalling 3.0KiB
2018-04-10 12:09:40.996156: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 1024 totalling 7.0KiB
2018-04-10 12:09:40.996164: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB
2018-04-10 12:09:40.996172: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1536 totalling 1.5KiB
2018-04-10 12:09:40.996184: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 14 Chunks of size 2048 totalling 28.0KiB
2018-04-10 12:09:40.996193: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 6912 totalling 13.5KiB
2018-04-10 12:09:40.996201: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 16384 totalling 80.0KiB
2018-04-10 12:09:40.996210: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 65536 totalling 128.0KiB
2018-04-10 12:09:40.996218: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 71936 totalling 281.0KiB
2018-04-10 12:09:40.996226: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 81920 totalling 80.0KiB
2018-04-10 12:09:40.996234: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 107264 totalling 104.8KiB
2018-04-10 12:09:40.996243: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 113152 totalling 110.5KiB
2018-04-10 12:09:40.996251: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 147456 totalling 288.0KiB
2018-04-10 12:09:40.996259: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 215552 totalling 631.5KiB
2018-04-10 12:09:40.996267: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 287488 totalling 280.8KiB
2018-04-10 12:09:40.996276: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 289536 totalling 565.5KiB
2018-04-10 12:09:40.996284: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 294912 totalling 864.0KiB
2018-04-10 12:09:40.996295: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 589824 totalling 1.12MiB
2018-04-10 12:09:40.996304: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 868352 totalling 4.14MiB
2018-04-10 12:09:40.996312: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1157888 totalling 1.10MiB
2018-04-10 12:09:40.996324: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1179648 totalling 2.25MiB
2018-04-10 12:09:40.996330: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1201408 totalling 1.15MiB
2018-04-10 12:09:40.996338: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 2359296 totalling 9.00MiB
2018-04-10 12:09:40.996350: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4718592 totalling 9.00MiB
2018-04-10 12:09:40.996359: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 8388608 totalling 56.00MiB
2018-04-10 12:09:40.996371: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 9437184 totalling 99.00MiB
2018-04-10 12:09:40.996378: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 19 Chunks of size 13893376 totalling 251.75MiB
2018-04-10 12:09:40.996386: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 13945088 totalling 13.30MiB
2018-04-10 12:09:40.996395: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 14155776 totalling 13.50MiB
2018-04-10 12:09:40.996403: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 16777216 totalling 144.00MiB
2018-04-10 12:09:40.996411: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 27760896 totalling 26.47MiB
2018-04-10 12:09:40.996420: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 36773888 totalling 280.56MiB
2018-04-10 12:09:40.996428: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 37323776 totalling 35.59MiB
2018-04-10 12:09:40.996436: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 55573504 totalling 53.00MiB
2018-04-10 12:09:40.996445: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 74097664 totalling 847.98MiB
2018-04-10 12:09:40.996453: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 75203584 totalling 71.72MiB
2018-04-10 12:09:40.996461: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 147089408 totalling 140.28MiB
2018-04-10 12:09:40.996470: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 148195328 totalling 141.33MiB
2018-04-10 12:09:40.996478: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 149298176 totalling 142.38MiB
2018-04-10 12:09:40.996486: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 296390656 totalling 565.32MiB
2018-04-10 12:09:40.996494: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size<PHONE_NUMBER> totalling 4.42GiB
2018-04-10 12:09:40.996503: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 7.26GiB
2018-04-10 12:09:40.996512: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit:<PHONE_NUMBER>9
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>8
NumAllocs:                   41821
MaxAllocSize:<PHONE_NUMBER>

Here a simple program that allocates a large tensor filled with zeros:
import numpy as np
import tensorflow as tf
import sys

amount = int(sys.argv[1])
print('Generating tensor with {} zeros' % amount)
_data = np.zeros(amount * (1 << 20), dtype=np.float32)

data_init = tf.placeholder(tf.float32, shape=_data.shape)
data = tf.Variable(data_init, trainable=False, collections=[])
a = tf.contrib.memory_stats.MaxBytesInUse()
b = tf.constant('Completed successfully')

with tf.Session() as sess:
    print(sess.run([data.initializer, a, b], feed_dict={data_init: _data}))

Generating 1200 or 1300 zeros works ok
$ python alloc.py 1200
/home/remi_beges/.local/share/virtualenvs/analog-OR2zGvr-/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Generating amount  1200
WARNING:tensorflow:From /home/remi_beges/.local/share/virtualenvs/analog-OR2zGvr-/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
2018-04-10 13:59:20.700598: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-04-10 13:59:20.816981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-04-10 13:59:20.817404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2018-04-10 13:59:20.892493: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-04-10 13:59:20.892925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 1 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:05.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-04-10 13:59:20.892991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0, 1
2018-04-10 13:59:21.439399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-10 13:59:21.439464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 1
2018-04-10 13:59:21.439474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N N
2018-04-10 13:59:21.439479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 1:   N N
2018-04-10 13:59:21.439977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10750 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2018-04-10 13:59:21.627795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10765 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:00:05.0, compute capability: 3.7)
[None, 1280, b'I ran OK']

But as soon as I go over 1300 the allocation fails:
2018-04-10 13:58:38.000952: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 5.27GiB.  Current allocation summary follows.
2018-04-10 13:58:38.001045: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001086: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001098: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-04-10 13:58:38.001111: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001117: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001125: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001135: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001145: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001151: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001158: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001179: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001191: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001201: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001209: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001219: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001226: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001232: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001239: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001246: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001254: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-04-10 13:58:38.001265: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456):     Total Chunks: 2, Chunks in use: 1. 10.50GiB allocated for chunks. 5.27GiB in use in bin. 5.27GiB client-requested in use in bin.
2018-04-10 13:58:38.001278: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 5.27GiB was 256.00MiB, Chunk State:
2018-04-10 13:58:38.001293: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 5.22GiB | Requested Size: 0B | in_use: 0, prev:   Size: 5.27GiB | Requested Size: 5.27GiB | in_use: 1
2018-04-10 13:58:38.001307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x70a280000 of size 1280
2018-04-10 13:58:38.001318: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x70a280500 of size<PHONE_NUMBER>
2018-04-10 13:58:38.001326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x85ba80500 of size<PHONE_NUMBER>
2018-04-10 13:58:38.001336: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size:
2018-04-10 13:58:38.001346: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2018-04-10 13:58:38.001354: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size<PHONE_NUMBER> totalling 5.27GiB
2018-04-10 13:58:38.001364: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.27GiB
2018-04-10 13:58:38.001376: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit:<PHONE_NUMBER>2
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>
NumAllocs:                       2
MaxAllocSize:<PHONE_NUMBER>

In fact, it seems that the allocator is not even able to allocate half of available memory for a single GPU. Is there a maximum limit for single memory allocations ?

@jlebar, can you take a look?

Hi, Reed.
I work on XLA.  Is this somehow related to that?

No. I only assigned to your because you authored 9fe7c29605a5ee1519cceba8e67a6d8413444fac. But I can take a look instead.

Thank you for looking into this. Today, I tried the same thing on a Tesla P100 that has 16GB of VRAM, same outcome. It fails to allocate ~4GB:
2018-04-11 06:58:33.346240: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.42GiB.  Current allocation summary follows.

2018-04-11 06:58:33.346580: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f0e30800000 of size<PHONE_NUMBER>
2018-04-11 06:58:33.346588: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f0f4b290000 of size 74097664
2018-04-11 06:58:33.346596: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x7f0f4f93a400 of size<PHONE_NUMBER>
2018-04-11 06:58:33.346603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1084000000 of size 16777216
2018-04-11 06:58:33.346611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1085000000 of size 74097664
2018-04-11 06:58:33.346619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f10896aa400 of size 16777216
2018-04-11 06:58:33.346630: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f108a6aa400 of size 16777216
2018-04-11 06:58:33.346635: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f108b6aa400 of size 74097664
2018-04-11 06:58:33.346643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f108fd54800 of size 74097664
2018-04-11 06:58:33.346649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x7f10943fec00 of size<PHONE_NUMBER>
2018-04-11 06:58:33.346657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1184000000 of size 36773888
2018-04-11 06:58:33.346664: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1186312000 of size 36773888
2018-04-11 06:58:33.346671: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1188624000 of size 74097664
2018-04-11 06:58:33.346684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f118ccce400 of size 120790016
2018-04-11 06:58:33.346690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1194000000 of size 8388608
2018-04-11 06:58:33.346695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1194800000 of size 8388608
2018-04-11 06:58:33.346703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1195000000 of size 16777216
2018-04-11 06:58:33.346711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1196000000 of size 16777216
2018-04-11 06:58:33.346719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1197000000 of size 2359296
2018-04-11 06:58:33.346729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1197240000 of size 16515072
2018-04-11 06:58:33.346736: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1198200000 of size 9437184
2018-04-11 06:58:33.346744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1198b00000 of size 9437184
2018-04-11 06:58:33.346750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1199400000 of size 9437184
2018-04-11 06:58:33.346756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x7f1199d00000 of size 9437184
2018-04-11 06:58:33.346763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119a600000 of size 294912
2018-04-11 06:58:33.346769: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119a648000 of size 589824
2018-04-11 06:58:33.346775: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119a6d8000 of size 1179648
2018-04-11 06:58:33.346781: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119a7f8000 of size 2359296
2018-04-11 06:58:33.346787: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119aa38000 of size 4718592
2018-04-11 06:58:33.346793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f119aeb8000 of size 18120704
2018-04-11 06:58:33.346799: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c8400000 of size 9437184
2018-04-11 06:58:33.346806: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c8d00000 of size 589824
2018-04-11 06:58:33.346812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c8d90000 of size 1179648
2018-04-11 06:58:33.346819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c8eb0000 of size 2359296
2018-04-11 06:58:33.346829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c90f0000 of size 4718592
2018-04-11 06:58:33.346835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c9570000 of size 9437184
2018-04-11 06:58:33.346841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11c9e70000 of size 16777216
2018-04-11 06:58:33.346847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11cae70000 of size 22609920
2018-04-11 06:58:33.346854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11cc400000 of size 9437184
2018-04-11 06:58:33.346863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11ccd00000 of size 9437184
2018-04-11 06:58:33.346869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11cd600000 of size 14680064
2018-04-11 06:58:33.346875: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11ce400000 of size 16777216
2018-04-11 06:58:33.346885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11cf400000 of size 8388608
2018-04-11 06:58:33.346891: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11cfc00000 of size 4194304
2018-04-11 06:58:33.346896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11d0000000 of size 289536
2018-04-11 06:58:33.346903: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f11d0046b00 of size 289536
2018-04-11 06:58:33.346909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x7f11d008d600 of size 1518080
2018-04-11 06:58:33.346916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400000 of size 1280
2018-04-11 06:58:33.346921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400500 of size 1280
2018-04-11 06:58:33.346928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400a00 of size 256
2018-04-11 06:58:33.346934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400b00 of size 256
2018-04-11 06:58:33.346940: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400c00 of size 256
2018-04-11 06:58:33.346945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400d00 of size 256
2018-04-11 06:58:33.346952: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400e00 of size 256
2018-04-11 06:58:33.346957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210400f00 of size 256
2018-04-11 06:58:33.346966: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401000 of size 256
2018-04-11 06:58:33.346972: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401100 of size 512
2018-04-11 06:58:33.346981: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401300 of size 256
2018-04-11 06:58:33.346988: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401400 of size 256
2018-04-11 06:58:33.346994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401500 of size 256
2018-04-11 06:58:33.347000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401600 of size 256
2018-04-11 06:58:33.347006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401700 of size 1024
2018-04-11 06:58:33.347012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401b00 of size 256
2018-04-11 06:58:33.347022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401c00 of size 256
2018-04-11 06:58:33.347028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401d00 of size 256
2018-04-11 06:58:33.347035: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401e00 of size 256
2018-04-11 06:58:33.347041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210401f00 of size 2048
2018-04-11 06:58:33.347049: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402700 of size 256
2018-04-11 06:58:33.347055: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402800 of size 256
2018-04-11 06:58:33.347061: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402900 of size 256
2018-04-11 06:58:33.347068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402a00 of size 256
2018-04-11 06:58:33.347074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402b00 of size 256
2018-04-11 06:58:33.347079: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402c00 of size 256
2018-04-11 06:58:33.347086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402d00 of size 256
2018-04-11 06:58:33.347092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402e00 of size 256
2018-04-11 06:58:33.347100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210402f00 of size 256
2018-04-11 06:58:33.347106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403000 of size 256
2018-04-11 06:58:33.347112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403100 of size 256
2018-04-11 06:58:33.347119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403200 of size 256
2018-04-11 06:58:33.347126: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403300 of size 256
2018-04-11 06:58:33.347131: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403400 of size 256
2018-04-11 06:58:33.347137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403500 of size 256
2018-04-11 06:58:33.347144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403600 of size 256
2018-04-11 06:58:33.347151: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403700 of size 256
2018-04-11 06:58:33.347158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403800 of size 512
2018-04-11 06:58:33.347165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403a00 of size 256
2018-04-11 06:58:33.347171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403b00 of size 256
2018-04-11 06:58:33.347177: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403c00 of size 256
2018-04-11 06:58:33.347183: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403d00 of size 256
2018-04-11 06:58:33.347190: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210403e00 of size 1536
2018-04-11 06:58:33.347197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404400 of size 256
2018-04-11 06:58:33.347203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404500 of size 256
2018-04-11 06:58:33.347210: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404600 of size 256
2018-04-11 06:58:33.347216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404700 of size 256
2018-04-11 06:58:33.347222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404800 of size 256
2018-04-11 06:58:33.347229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210404900 of size 2048
2018-04-11 06:58:33.347236: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405100 of size 256
2018-04-11 06:58:33.347242: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405200 of size 256
2018-04-11 06:58:33.347248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405300 of size 256
2018-04-11 06:58:33.347254: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405400 of size 256
2018-04-11 06:58:33.347261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405500 of size 256
2018-04-11 06:58:33.347266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405600 of size 256
2018-04-11 06:58:33.347273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405700 of size 256
2018-04-11 06:58:33.347278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210405800 of size 256
2018-04-11 06:58:33.347284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x7f1210405900 of size 5376
2018-04-11 06:58:33.347290: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210406e00 of size 256
2018-04-11 06:58:33.347297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210406f00 of size 256
2018-04-11 06:58:33.347305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210407000 of size 512
2018-04-11 06:58:33.347312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210407200 of size 512
2018-04-11 06:58:33.347317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210407400 of size 1024
2018-04-11 06:58:33.347326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210407800 of size 1024
2018-04-11 06:58:33.347334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210407c00 of size 1024
2018-04-11 06:58:33.347341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210408000 of size 2048
2018-04-11 06:58:33.347346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210408800 of size 2048
2018-04-11 06:58:33.347354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210409000 of size 2048
2018-04-11 06:58:33.347360: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210409800 of size 2048
2018-04-11 06:58:33.347366: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121040a000 of size 2048
2018-04-11 06:58:33.347374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121040a800 of size 2048
2018-04-11 06:58:33.347380: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121040b000 of size 6912
2018-04-11 06:58:33.347385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121040cb00 of size 113152
2018-04-11 06:58:33.347392: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210428500 of size 256
2018-04-11 06:58:33.347398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210428600 of size 65536
2018-04-11 06:58:33.347406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210438600 of size 81920
2018-04-11 06:58:33.347411: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121044c600 of size 147456
2018-04-11 06:58:33.347418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210470600 of size 512
2018-04-11 06:58:33.347427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210470800 of size 512
2018-04-11 06:58:33.347435: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210470a00 of size 1024
2018-04-11 06:58:33.347443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210470e00 of size 1024
2018-04-11 06:58:33.347448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210471200 of size 1024
2018-04-11 06:58:33.347456: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210471600 of size 2048
2018-04-11 06:58:33.347461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210471e00 of size 2048
2018-04-11 06:58:33.347468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210472600 of size 2048
2018-04-11 06:58:33.347473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210472e00 of size 2048
2018-04-11 06:58:33.347481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210473600 of size 2048
2018-04-11 06:58:33.347486: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210473e00 of size 2048
2018-04-11 06:58:33.347493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210474600 of size 6912
2018-04-11 06:58:33.347499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f1210476100 of size 147456
2018-04-11 06:58:33.347505: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f121049a100 of size 417536
2018-04-11 06:58:33.347511: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size:
2018-04-11 06:58:33.347519: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 52 Chunks of size 256 totalling 13.0KiB
2018-04-11 06:58:33.347528: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 512 totalling 3.0KiB
2018-04-11 06:58:33.347536: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 1024 totalling 7.0KiB
2018-04-11 06:58:33.347543: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB
2018-04-11 06:58:33.347551: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1536 totalling 1.5KiB
2018-04-11 06:58:33.347557: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 14 Chunks of size 2048 totalling 28.0KiB
2018-04-11 06:58:33.347564: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 6912 totalling 13.5KiB
2018-04-11 06:58:33.347571: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 65536 totalling 64.0KiB
2018-04-11 06:58:33.347577: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 81920 totalling 80.0KiB
2018-04-11 06:58:33.347585: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 113152 totalling 110.5KiB
2018-04-11 06:58:33.347591: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 147456 totalling 288.0KiB
2018-04-11 06:58:33.347598: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 289536 totalling 565.5KiB
2018-04-11 06:58:33.347605: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 294912 totalling 288.0KiB
2018-04-11 06:58:33.347611: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 417536 totalling 407.8KiB
2018-04-11 06:58:33.347618: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 589824 totalling 1.12MiB
2018-04-11 06:58:33.347624: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1179648 totalling 2.25MiB
2018-04-11 06:58:33.347630: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 2359296 totalling 6.75MiB
2018-04-11 06:58:33.347637: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4194304 totalling 4.00MiB
2018-04-11 06:58:33.347642: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4718592 totalling 9.00MiB
2018-04-11 06:58:33.347651: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 8388608 totalling 24.00MiB
2018-04-11 06:58:33.347657: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 9437184 totalling 63.00MiB
2018-04-11 06:58:33.347663: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 14680064 totalling 14.00MiB
2018-04-11 06:58:33.347670: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16515072 totalling 15.75MiB
2018-04-11 06:58:33.347686: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 16777216 totalling 112.00MiB
2018-04-11 06:58:33.347693: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 18120704 totalling 17.28MiB
2018-04-11 06:58:33.347705: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 22609920 totalling 21.56MiB
2018-04-11 06:58:33.347713: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 36773888 totalling 70.14MiB
2018-04-11 06:58:33.347720: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 74097664 totalling 353.33MiB
2018-04-11 06:58:33.347734: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 120790016 totalling 115.19MiB
2018-04-11 06:58:33.347742: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size<PHONE_NUMBER> totalling 4.42GiB
2018-04-11 06:58:33.347752: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.23GiB
2018-04-11 06:58:33.347761: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit:<PHONE_NUMBER>9
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>
NumAllocs:                   27368
MaxAllocSize:<PHONE_NUMBER>

Let me know if I can do anything to help debug this.

For the short example you posted, my guess is that when you initialize a Variable with a tf.placeholder fed with a feed dict, you need to allocate memory both for the Variable data and the tf.placeholder, requiring double the memory as what just the Variable requires. (@mrry can you confirm this fact?)

The memory Limit displayed by the error log accounts for the VRAM of a single GPU (12GB), while I was expecting to account both (24GB):

Note that the limit is supposed to show only a single GPU. Each GPU has it's own memory, and ops assigned to that GPU will use that GPU's memory. If there is not enough memory on a GPU to run all the ops assigned to that GPU, you will get an OOM even if there is memory left on the other GPU.
In your case, I would guess either you are running all/most your ops on GPU:0, or both GPUs would run out of memory and GPU:0 happens to run out of memory first.

@reedwm I think that's correct, yes. Here's where the copy happens:
https://github.com/tensorflow/tensorflow/blob/0c1ea2db7fb137dd4cf6f334dda9eb9623935d9d/tensorflow/core/kernels/assign_op.h#L106-L121
You might be able to avoid this using tf.get_variable(..., use_resource=True), because a ResourceVariable will reuse the fed-in value (as long as it has no other users in the same step):
https://github.com/tensorflow/tensorflow/blob/0c1ea2db7fb137dd4cf6f334dda9eb9623935d9d/tensorflow/core/kernels/resource_variable_ops.cc#L251-L261

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 29 days with no activity and the awaiting response label was assigned. Is this still an issue?

I don't have access to the GPU machine for now, when I do I'll try the use_resource flag. I'm closing this for now because I don't think there is anything to fix in TF, rather I need to redesign a few things to eliminate this op that requires so much vram in my final app.
But it's good to keep in mind that although TF maps memory of multiple GPUs, each op must fit into a single GPU, otherwise a memory allocation will be raised.


Were update and upgrade purposely removed from the DSL?

Ah looks like maybe it was an internal brew feature that got canned?

There was an old DSL that wasn't well designed. This one is better. upgrade is done implicitly but update needs done manually now.

Interesting.  How is this better than say a shell script with calls to brew?

Well, the previous DSL wasn't any better. This one understands Homebrew versions and the like in a slightly smarter way and allows you to embed some Ruby logic inside your Brewfile. It's basically more like a Gemfile.


Cannot customize style of md-tabs

Bug, feature request, or proposal:
Adding custom style to the tabs header
What is the expected behavior?
affect css style of header (font size, color, display....)
What is the current behavior?
Style is beeing totally ignored due to attribute selectors generated by the own element ignoring the styleUrls ones
What are the steps to reproduce?
Create a simple element (html):
<md-tab-group>
    <md-tab>
        <template md-tab-label>Personal info</template>
        <template md-tab-content>

Name, address

                Some data here

        </template>
    </md-tab>

    <md-tab>
        <template md-tab-label>Payment</template>
        <template md-tab-content>

                Payment setup

        </template>
    </md-tab>
</md-tab-group>

Add this style in the .scss:
md-tab-group {
    .md-tab-header {
        font-size: 12px !important;
        display: flex;
        flex-flow: row wrap;
        align-items: center;
        .md-tab-label {
            flex: 1;
            min-width: 1px;
        }
    }
}

Plunk demo http://plnkr.co/edit/K1iVNV
What is the use-case or motivation for changing an existing behavior?
Need to customize md tabs style/colors/responsive
Which versions of Angular, Material, OS, browsers are affected?
Angular 2.0, Material alpha 8-2
Is there anything else we should know?

I see two problems with your Plunk

Your styles are not valid CSS:

/*CSS styles should not be nested*/
md-tab-group {
   .md-tab-header{
   	...
   }
}

Styles in components are isolated, so the Material component (md-tab-group) cannot see your App component styles.  Put the styles you want to override in a main CSS file that you link to from your index.html.

On this Plunk, I have implemented some of the styles you were going for.

I just copy pasted it, I am using scss in my code. But I am able to customize style of md-card for example. I  edit the plunk example http://plnkr.co/edit/K1iVNV?p=preview.
I notice that my component is generating this css for my style:
md-tab-group[_ngcontent-yed-6]   .md-tab-header[_ngcontent-yed-6] {
  font-size: 12px !important;
  display: flex;
  flex-flow: row wrap;
  align-items: center; }
  md-tab-group[_ngcontent-yed-6]   .md-tab-header[_ngcontent-yed-6]   .md-tab-label[_ngcontent-yed-6] {
    flex: 1;
    min-width: 1px; }

But this is being override by this css, where the attribute selector _nghost-yed-7 get priority since it is present in the tag of the tab header:
[_nghost-yed-7] { display: flex; flex-direction: column; font-family: Roboto, "Helvetica Neue", sans-serif; }  .md-tab-header[_ngcontent-yed-7] { overflow: hidden; position: relative; display: flex; flex-direction: row; border-bottom: 1px solid #e0e0e0; flex-shrink: 0; }  .md-tab-label[_ngcontent-yed-7] { line-height: 48px; height: 48px; padding: 0 12px; font-size: 14px; font-family: Roboto, "Helvetica Neue", sans-serif; font-weight: 500; cursor: pointer; box-sizing: border-box; color: currentColor; opacity: 0.6; min-width: 160px; text-align: center; } .md-tab-label[_ngcontent-yed-7]:focus { outline: none; opacity: 1; background-color: rgba(178, 223, 219, 0.3); } .md-tab-disabled[_ngcontent-yed-7] { cursor: default; pointer-events: none; }  .md-tab-body-wrapper[_ngcontent-yed-7] { position: relative; overflow: hidden; flex-grow: 1; display: flex; }  .md-tab-body[_ngcontent-yed-7] { display: none; overflow: auto; box-sizing: border-box; flex-grow: 1; flex-shrink: 1; } .md-tab-body.md-tab-active[_ngcontent-yed-7] { display: block; }  md-ink-bar[_ngcontent-yed-7] { position: absolute; bottom: 0; height: 2px; background-color: #009688; transition: 350ms ease-out; }/*# sourceMappingURL=tab-group.css.map */

I really cannot understand this behavior, it seems that md-tabs are a sort of subcomponent...?!

You need to use /deep/:
/deep/ md-tab-group {
...

Even using /deep/ the styles are not applied. You must use !important to ensure the style will be applied.
So, the question is, what is the correct way of doing style modification in md components? I my case, I could not change md-checkbox margins when placing it inside a table . So I ended up with this code to be possible to change the checkbox style
/deep/ .md-checkbox-inner-container{
	margin: 4px 0px 0px 0px !important;
}

The problem is that I do not want to use !important, so, how can I do this without !important?

Sometimes I need to put a more specific selector, for example:
body <selector> {...

instead of
<selector> {...

If you use SCSS you can use the hierarchy feature to get a better specification of the selectors, and they will win this "competition".

What do you mean by "the hierarchy feature"?
Let's say I have a component that is a table and then I place an md-checkbox at the first column of the tbody rows, so, how can I use the hierarchy feature to change the md-checkbox positioning without using !important and /deep/?

Let's say we have:

    <md-checkbox></md-checkbox>

So you would style in SCSS in this way:
.articles{
  color: blue; /*style for entire articles section */
  article{
    color: red; /* style for each article*/
    /deep/ md-checkbox .md-checkbox-inner-container{
      margin: 4px 0px 0px 0px;
    }
  }
}

It will be translated to:
.articles{
  color: blue;
}
.articles article{
  color: red;
}
.articles article /deep/ md-checkbox .md-checkbox-inner-container{
  margin: 4px 0px 0px 0px;
}

So .articles article /deep/ md-checkbox .md-checkbox-inner-container may be specific enough to style the proper component without !important. As md-checkbox is a component, I think we need to edit the component style itself to change it without /deep/.

Thanks @anderflash. Using /deep/ md-checkbox .md-checkbox-inner-container worked for me.
So we can assume that it is the right behavior because Angular 2 styles (except those that are loaded by index.html) are scoped to the component so one component should not be able to change nested components unless you use /deep/?

That's also my view about this issue. I'm also learning it and the tools are evolving rapidly. It still seems like a hack to put /deep/ wherever we want back default css behavior (changing nested components without context). So I think it will change in the future.

There are three ways to resolve this right now.
(1) Set view encapsulation to NONE on the component that hosts your tabs and use your component style sheet
(2) Load a separate style sheet apart from Angular (e.g. through index.html) that affects the tabs
(3) Use /deep/, which has so far been deprecated from the spec but still supported by Angular for the time being. Not recommended.

@andrewseguin None of that 3 methods help to customize some styles right now the style of a md-tab header. The only way for now, is to use the !important keyword on the style (and this is not recommended really).
I can set the encapsulation to NONE of my component, but this will affect only my component, not Tabs component.
You're closing an issue that is an issue, style should be customized.

Perhaps I'm misunderstanding your issue. I am able to change the styling on this tab header removing view encapsulation and applying styles that have a higher specificity than those defined in the default md CSS.
http://plnkr.co/edit/J6Wnn0?p=preview

No you've not misunderstood it. In your case, as you can see, the encapsulation was not enought, you've added also .my-tabs class

Yes, it is necessary to use higher specificity in CSS selectors in order to override the default style of the tab header. Does this solution not fit your needs?

Let says yes but in general I would like to preserve encapsulation of  my component and be able to customize the style of my app at the same time.

I would like to be able to do that as well, but unfortunately this is beyond the bounds of what can be done in material (or any UI toolkit) due to how angular 2 is implemented. Feel free to file an issue with angular to suggest this.

encapsulation to NONE is the better way to customize tabs component?

@andrewseguin why you didn't set "md-tab-group" component encapsulated to none ?

I resolve my problem
/deep/ .mat-tab-label{
line-height: 28px !important;
height: 28px !important;
}
/deep/ .mat-tab-header {
background-color: #0288D1;
color: white;
border-radius: 5px 5px 0px 0px;
}
/deep/ .mat-ink-bar {
background-color: #d14b02  !important;
min-width: 120px;
}
result

Maybe the same way you can style a toolbar like this:
<md-toolbar color="primary">Title</md-toolbar>
<md-tab-group color="primary">   <md-tab label="Title">Content</md-tab> </md-tab-group>

Hello, it works indeed as @ivanntis said. But even if I can change all of the tabs, how can I change only one tab style? I want to add a + at the end in order to add a tab.
Thank you.

@andrewseguin I am unable to change styles via ViewEncapsulation and the plnkr you have attached is not working. Thank you.

Already mentioned, but I see a few adopting the /deep/ solution. /deep/ is being deprecated, sooner or later your apps will break.

So what is the best solution?

@mist3r3 the choices you have are as reported by @andrewseguin earlier in this thread. Since /deep/ is being deprecated the solutions you are left with are (copied from @andrewseguin above):
(1) Set view encapsulation to NONE on the component that hosts your tabs and use your component style sheet
(2) Load a separate style sheet apart from Angular (e.g. through index.html) that affects the tabs
Personally I use option 1.

The issue with (1) setting the view encapsulation to NONE is those styles bleed throughout the entire app.  If my component's stylesheet I had h1 { color: blue } then when changing the component from Emulated to None it makes ALL h1 elements blue that don't have a color defined by a more specific rule.
I would like to say "make this component and all it's children" h1's blue.  Setting encapsulation to none I would have to add an application-wide unique selector to my component and use that in the component's stylesheet.  But that's what I'm trying to get away from doing and why I fell in love with component view encapsulation.

@anderflash i did the same thing!
i wrapped it into a css class e.g --> class ="test"
--> test{
/deep/ md-checkbox .md-checkbox-inner-container{
margin:..;
}
}
and its works!


Remove models and architectures in preparation for pypi publication.

Models have been moved to a separate repo [https://github.com/wearepal/legacy_models] in light of the fact that most (all?) are broken or unused and pre-defining models is generally not very useful as the callbacks need to be tailored to the task in question.

You might want to move conduit/models/utils.py somewhere else because it is now the only file in conduit/models.

I've given great thought to this but decided to leave it as it is for now so as not to break the repos which import from conduit.models.utils.


Is selectable tree node implemented?

In the demo from Dear ImGui, there is an advanced tree node (TreeNodeEx) which can be selected by single-click, and expanded by double-click. Can I achieve the same behavior by using "tree_node"?

Answer my own question. Code below achieves what I want:
ui.tree_node(&im_str!("Child {}", i))
    .flags(ImGuiTreeNodeFlags::OpenOnDoubleClick | ImGuiTreeNodeFlags::OpenOnArrow)
    .build(...)


[Help requested] Clarification on Generating Customized Positive Queries.

Hi, guys! Great work!
@TempleX98, I have been studying your paper on "DETRs with Collaborative Hybrid Assignments Training" and I am particularly interested in the Customized Positive Queries Generation process.
In DAB-DETR, there are two types of queries used by the decoder: content queries and positional queries. Please confirm if my understanding is correct on how to generate these queries when integrating your method into DAB-DETR. And even more, do I need to use DAB DETR with content and positional queries?
Here is my understanding:

Extract positive coordinates from the auxiliary heads to identify likely object locations.
Take those coordinates as positional queries of shape [x, y, w, h].
Generate Content Queries by extracting corresponding features from the encoder’s output at these positive coordinates and transforming them appropriately.
Combine these content and positional queries to form the input queries for the decoder.

I would appreciate your confirmation or any additional insights on this process.
Thank you for your assistance.
Best regards,
Aleksandr Gordeev

Similar to Co-Deformable-DETR, the content queries are obtained via an MLP layer.


Error while loading custom finetuned QLoRA model in 4 bit : size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).

Hi Team,
I have successfully finetuned a QLoRA adapter on a custom dataset. When I try to load it in full precision, it gets loaded and works well
But this takes too much time and GPU memory to run inference. So I wanted to load the model in 4-bit precision. So I pass in the load_4bit parameter:
model_path = '/home/apoorv/development/videollama2/VideoLLaMA2/work_dirs/videollama2_vllava/finetune_videollama2_vllava_qlora'
model_name = get_model_name_from_path(model_path)
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_4bit=True)

While running this I get the following error:
RuntimeError: Error(s) in loading state_dict for Videollama2MistralForCausalLM:
	size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
	size mismatch for model.mm_projector.readout.2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).

I have debugged and understood the reason that when we pass in the  load_4bit=True, the VideoLLama2 model params are loaded as 4bit params. The LLM part weights gets initialized from the base model (in this case mistralai/Mistral-7B-Instruct-v0.2) in 4-bit but the mm_projector does not (I am guessing it is initialized from the random values but in Params4bit class).
Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L76
model = Videollama2MistralForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained, **kwargs)
Now to initialize the weights for the mm_projector, we try to load it from the previously saved non_lora_trainables.bin which was stored in full precision format.
Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L101
model.load_state_dict(non_lora_trainables, strict=False)
Debugging Outputs
Model params
**(Pdb) model.model.mm_projector.readout**
Sequential(
  (0): Linear4bit(in_features=4096, out_features=4096, bias=True)
  (1): GELU(approximate='none')
  (2): Linear4bit(in_features=4096, out_features=4096, bias=True)
)
**(Pdb) model.model.mm_projector.readout[0]**
Linear4bit(in_features=4096, out_features=4096, bias=True)
**(Pdb) model.model.mm_projector.readout[0].weight**
Parameter containing:
Parameter(Params4bit([[193],
            [108],
            [250],
            ...,
            [231],
            [107],
            [ 92]], device='cuda:1', dtype=torch.uint8))
**(Pdb) model.model.mm_projector.readout[0].weight.shape**
torch.Size([8388608, 1])

Previously saved non_lora_trainables param
(Pdb) non_lora_trainables[ 'model.model.mm_projector.readout.0.weight'].shape
torch.Size([4096, 4096])

Please advise on how to resolve this.

Hi Team,
Any help is highly appreciated.
Thanks :)

You can check this issue https://github.com/DAMO-NLP-SG/VideoLLaMA2/issues/78

Hi Team,
I have successfully finetuned a QLoRA adapter on a custom dataset. When I try to load it in full precision, it gets loaded and works well
But this takes too much time and GPU memory to run inference. So I wanted to load the model in 4-bit precision. So I pass in the load_4bit parameter:
model_path = '/home/apoorv/development/videollama2/VideoLLaMA2/work_dirs/videollama2_vllava/finetune_videollama2_vllava_qlora'
model_name = get_model_name_from_path(model_path)
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_4bit=True)

While running this I get the following error:
RuntimeError: Error(s) in loading state_dict for Videollama2MistralForCausalLM:
	size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
	size mismatch for model.mm_projector.readout.2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).

I have debugged and understood the reason that when we pass in the load_4bit=True, the VideoLLama2 model params are loaded as 4bit params. The LLM part weights gets initialized from the base model (in this case mistralai/Mistral-7B-Instruct-v0.2) in 4-bit but the mm_projector does not (I am guessing it is initialized from the random values but in Params4bit class).
Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L76 model = Videollama2MistralForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained, **kwargs)
Now to initialize the weights for the mm_projector, we try to load it from the previously saved non_lora_trainables.bin which was stored in full precision format.
Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L101 model.load_state_dict(non_lora_trainables, strict=False)
Debugging Outputs
Model params
**(Pdb) model.model.mm_projector.readout**
Sequential(
  (0): Linear4bit(in_features=4096, out_features=4096, bias=True)
  (1): GELU(approximate='none')
  (2): Linear4bit(in_features=4096, out_features=4096, bias=True)
)
**(Pdb) model.model.mm_projector.readout[0]**
Linear4bit(in_features=4096, out_features=4096, bias=True)
**(Pdb) model.model.mm_projector.readout[0].weight**
Parameter containing:
Parameter(Params4bit([[193],
            [108],
            [250],
            ...,
            [231],
            [107],
            [ 92]], device='cuda:1', dtype=torch.uint8))
**(Pdb) model.model.mm_projector.readout[0].weight.shape**
torch.Size([8388608, 1])

Previously saved non_lora_trainables param
(Pdb) non_lora_trainables[ 'model.model.mm_projector.readout.0.weight'].shape
torch.Size([4096, 4096])

Please advise on how to resolve this.


Get tests running in GitHub actions

GitHub action tests currently fail with the following error:
Run dotnet test src/graphiql.tests/ --configuration Development --nologo
##[warning]BaseTest.cs(40,30): warning CS0168: The variable 'e' is declared but never used [/home/runner/work/graphiql-dotnet/graphiql-dotnet/src/graphiql.tests/graphiql.tests.csproj]
Test run for /home/runner/work/graphiql-dotnet/graphiql-dotnet/src/graphiql.tests/bin/Development/netcoreapp3.1/graphiql.tests.dll(.NETCoreApp,Version=v3.1)
Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {06adbfda-45e1-4c4d-8b95-3d842d41852f} may be persisted to storage in unencrypted form.

Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Connection refused Connection refused
Connection refused Connection refused
[xUnit.net 00:00:07.45]     GraphiQl.tests.GraphQlTests.CanQueryGraphQl [FAIL]
  X GraphiQl.tests.GraphQlTests.CanQueryGraphQl [1ms]
  Error Message:
   OpenQA.Selenium.WebDriverException : unknown error: Chrome failed to start: exited abnormally.
  (chrome not reachable)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  Stack Trace:
     at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
   at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(ICommandExecutor commandExecutor, ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Chrome.ChromeDriver..ctor(ChromeDriverService service, ChromeOptions options, TimeSpan commandTimeout)
   at OpenQA.Selenium.Chrome.ChromeDriver..ctor(ChromeOptions options)
   at GraphiQl.tests.BaseTest..ctor(Boolean runHeadless) in /home/runner/work/graphiql-dotnet/graphiql-dotnet/src/graphiql.tests/BaseTest.cs:line 31
   at GraphiQl.tests.GraphQlTests..ctor(HostFixture fixture) in /home/runner/work/graphiql-dotnet/graphiql-dotnet/src/graphiql.tests/GraphQlTests.cs:line 14

Fixed by using the Windows image.

Hi Joseph,
Please can you provide more info on how you fixed this?


add dynamic shape support for cumsum/grid

Description
add dynamic shape support for cumsum/grid
Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

[ ] My code follows the style guidelines of this project (You can use the linters)
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas and hacks
[ ] I have made corresponding changes to the documentation
[ ] I have added tests to verify my fix or my feature
[ ] New and existing unit tests pass locally with my changes
[ ] I have added the relevant labels to my PR in so that relevant reviewers are notified

Need to rebase from main.