Ran out of GPU memory

I have a network similar to A3C network described here, with lots of syncing and copying tensor values between different duplicates of the model. The code crashes with a Ran out of memory exception (see below). This might be a little bit hard to reproduce somewhere other than my machine (since the code is not open source ... yet) but I would be more than happy to provide any logs or help to track down the issue.
What happens is that I receiving a lot of pool_allocator messages like below:

I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 40664481 get requests, put_count=40664484 evicted_count=39000 eviction_rate=0.000959068 and unsatisfied allocation rate=0.000959265

and then finally it crashes like this:

I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072):    Total Chunks: 1, Chunks in use: 0 204.5KiB allocated for chunks. 1.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 220.5KiB was 128.0KiB, Chunk State:
I tensorflow/core/common_runtime/bfc_allocator.cc:662]   Size: 204.5KiB | Requested Size: 1.0KiB | in_use: 0, prev:   Size: 16.0KiB | Requested Size: 16.0KiB | in_use: 1, next:   Size: 16.0KiB | Requested Size: 16.0KiB | in_use: 1
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a064a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a068a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a070a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a078a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a440a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a808a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a808e00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a809200 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a809e00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a80aa00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a80ea00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a812a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a813600 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a81b600 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a823600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0abeb600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb3600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb3a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb3e00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb4a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb5600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afb5a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afbda00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afc1a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afc1e00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afc9e00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afcde00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afcea00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afcee00 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afdd300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afddf00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afe1f00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afe5f00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afe6b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afe6f00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0afeef00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0aff2f00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0aff3b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0aff3f00 of size 39168
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0affd800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0affdc00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b005c00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b009c00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b00a800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b00ac00 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b019100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b019d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b01dd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b01e900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b01ed00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b022d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b023900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b023d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b02bd00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b02fd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b030900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b030d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b038d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b03cd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b03d900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b03dd00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b045d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b049d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b04a900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b04ad00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b052d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b056d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b057900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b057d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b05fd00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b063d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b064900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b064d00 of size 51968
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b071800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b079800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b07d800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b07e400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b07e800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b082800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b083400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b083800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b08b800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b08f800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b090400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b090800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b094800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b095400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b095800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b09d800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0a1800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0a2400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0a2800 of size 39168
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0ac100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0b4100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0b8100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0b8d00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0b9100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0c1100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0c5100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0c5500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0cd500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0d1500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0d2100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0d2500 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0e0a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0e1600 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0e5600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0e5a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0e9a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0ea600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0eaa00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0f2a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0f6a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0f7600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b0f7a00 of size 39168
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b101300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b109300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b10d300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b10df00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b10e300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b116300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b11a300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b11af00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b11b300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b123300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b127300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b127f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b128300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b130300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b134300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b134f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b135300 of size 41984
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b13f700 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b147700 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b148300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b14c300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b14cf00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b14d300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b155300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b159300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b159f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b15a300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b162300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b166300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b166f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b167300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b16f300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b173300 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b173700 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b17b700 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b17f700 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b180300 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b180700 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b18ec00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b18f800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b193800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b194400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b194800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b19c800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1a0800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1a1400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1a1800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1a9800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1ad800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1adc00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1b5c00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1b9c00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1ba800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1bac00 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1c9100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1c9d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1cdd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1ce900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1ced00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1d2d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1d6d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1d7900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1d7d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1dfd00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1e3d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1e4900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1e4d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1ecd00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1f0d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1f1900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1f1d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1f9d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1fdd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1fe900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b1fed00 of size 29184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b205f00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b20df00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b20e300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b216300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b21a300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b21af00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b21b300 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b229800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b22a400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b22a800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b232800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b236800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b237400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b237800 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b245d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b246900 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b247500 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b247900 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b24f900 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b253900 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b254500 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b254900 of size 55552
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b262200 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b26a200 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b26e200 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b26ee00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b276e00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b27ae00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b27ba00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b27be00 of size 59648
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b28a700 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b28ab00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b28eb00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b28f700 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b28fb00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b297b00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b29bb00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b29c700 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b29cb00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2a4b00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2a8b00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2a9700 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2a9b00 of size 45568
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2b4d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2b8d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2bcd00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2bd900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2bdd00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2c5d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2c9d00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2ca100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2d2100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2d6100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2d6d00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2d7100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2df100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2e3100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2e3500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2eb500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2ef500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2f0100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2f0500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2f8500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2fc500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2fd100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b2fd500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b305500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b309500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b30a100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b30a500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b312500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b316500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b317100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b317500 of size 21248
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b31c800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b31d400 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b31e000 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b326000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b326c00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b327000 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b32b000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b32bc00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b32c000 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b334000 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b338000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b338c00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b339000 of size 39168
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b342900 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b34a900 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b34e900 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b34f500 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b34f900 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b357900 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b35b900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b35bd00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b363d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b367d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b368900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b368d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b370d00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b374d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b375900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b375d00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b37dd00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b381d00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b382900 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b382d00 of size 26880
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b389600 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b38a200 of size 33792
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b392600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b392a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b396a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0b75ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0bb26a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0beeea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0c2b6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0c67ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0ca46a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0ce0ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0d1d6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0d59ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0d966a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0dd2ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0e0f6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0e4bea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0e886a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0ec4ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0f016a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0f3dea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0f7a6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0fb6ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0ff36a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb102fea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb106c6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb10a8ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb10e56a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1121ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb115e6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb119aea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb11d76a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1213ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb12506a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb128cea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb12c96a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1305ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb13426a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb137eea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb13bb6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb13f7ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb14346a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1470ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb14ad6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb14e9ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb15266a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1562ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb159f6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb15dbea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb16186a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1654ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb16916a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb16cdea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb170a6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1746ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb17836a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb17bfea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb17fc6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1838ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb18756a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb18b1ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb18ee6a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb192aea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb19676a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb19a3ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb19e06a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1a1cea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1a596a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1a95ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ad26a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1b0eea00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1b0ef600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1b4b7600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1b87f600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1bc47600 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1bc4f600 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c017600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c017a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c01fa00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c023a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c023e00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c02be00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c02fe00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c033e00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c034a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c034e00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c03ce00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c040e00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c041a00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c041e00 of size 51712
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c04e800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c04f400 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c417400 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c418000 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7e0000 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7e0400 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7e8400 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7ec400 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7ed000 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7ed400 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7fb900 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1c7fc500 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1cbc4500 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1cf8c500 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d354500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d358500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d359100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d359500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d361500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d365500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d366100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d366500 of size 56320
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d374100 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1d73c100 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1db04100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1db08100 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ded0100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ded0d00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ded1100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ded9100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1dedd100 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e2a5100 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e2a5d00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e2a6100 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e2ae100 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e2b2100 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e67a100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e67a500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e682500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e686500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e687100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e687500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e68f500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e693500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e694100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e694500 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e69c500 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e6a0500 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e6a1100 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e6a1500 of size 65024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e6b1300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1e6b1f00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ea79f00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1ee41f00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f209f00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f20df00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f5d5f00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f5d6b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f5d6f00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f5def00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f5e2f00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9aaf00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9abb00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9abf00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9aff00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9b0b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9b0f00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9b4f00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9b5b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9b5f00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9bdf00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9c1f00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9c2b00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9c2f00 of size 39168
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9cc800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9d4800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9d8800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9d9400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9d9800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9e1800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9e5800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9e6400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9e6800 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9ee800 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9f2800 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9f3400 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9f3800 of size 45568
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1f9fea00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1fa06a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb1fdcea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb20196a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2055ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb20926a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb20ceea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb210b6a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb210baa00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21482a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21483600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21483a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2148ba00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2148fa00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21857a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21858600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21858a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21860a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21864a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21c2ca00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21c2d600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21c2da00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21c35a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb21c39a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22001a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22002600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22002a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2200aa00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2200ea00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb223d6a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb223d7600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb223d7a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb223dfa00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb223e3a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb227aba00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb227ac600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb227aca00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb227b4a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb227b8a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b80a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b81600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b81a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b85a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b86600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b86a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b8ea00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b92a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b93600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b93a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b9ba00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22b9fa00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22ba0600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22ba0a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22ba8a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22baca00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22bad600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22bada00 of size 45568
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22bb8c00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22bc0c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb22f88c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23350c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23718c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23ae0c00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23ae4c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23eacc00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23ead800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23eadc00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23eb5c00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb23eb9c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24281c00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24282800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24282c00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2428ac00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2428ec00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24656c00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24657800 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24657c00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2465fc00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24663c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a2bc00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a2c000 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a34000 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a38000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a38c00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a39000 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a41000 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a45000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a45c00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a46000 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a4e000 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a52000 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a52c00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a53000 of size 65024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a62e00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24a63a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb24e2ba00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb251f3a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb255bba00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb255bfa00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25987a00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25988600 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25988a00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25990a00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25994a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d5ca00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d5ce00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d60e00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d61200 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d69200 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d6d200 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d6de00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d6e200 of size 58624
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d7c700 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d7d300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d81300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d81f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d82300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d8a300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d8e300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d8ef00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d8f300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d97300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d9b300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d9bf00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25d9c300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25da4300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25da4f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25da5300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dad300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25db1300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25db1f00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25db2300 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dba300 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dbe300 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dbef00 of size 1024
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dbf300 of size 47872
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dcae00 of size 3072
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dcba00 of size 32768
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb25dd3a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2619ba00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb26563a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb2692ba00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb26cf3a00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb270bba00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb270f2c00 of size 16384
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb270f6c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb274bec00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb27886c00 of size 3964928
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb27c4ec00 of size 4219136
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0xb270bfa00 of size 209408
I tensorflow/core/common_runtime/bfc_allocator.cc:689]      Summary of in-use Chunks by size:
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 10 Chunks of size 256 totalling 2.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 125 Chunks of size 1024 totalling 125.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 125 Chunks of size 3072 totalling 375.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 122 Chunks of size 16384 totalling 1.91MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 21248 totalling 20.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 26880 totalling 26.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 29184 totalling 28.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 98 Chunks of size 32768 totalling 3.06MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 33792 totalling 33.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 39168 totalling 191.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 41984 totalling 41.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 45568 totalling 133.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 47872 totalling 46.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 51712 totalling 50.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 51968 totalling 50.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 55552 totalling 54.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 56320 totalling 55.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 9 Chunks of size 58624 totalling 515.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 59648 totalling 58.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 2 Chunks of size 65024 totalling 127.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 124 Chunks of size 3964928 totalling 468.88MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 4219136 totalling 4.02MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 479.76MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats:
Limit:                   503270604
InUse:                   503061248
MaxInUse:                503061248
NumAllocs:                20717987
MaxAllocSize:              4219136
W tensorflow/core/common_runtime/bfc_allocator.cc:270] ****************************************************************************************************
W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 220.5KiB.  See logs for memory state.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072):    Total Chunks: 1, Chunks in use: 0 204.5KiB allocated for chunks. 1.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 330.8KiB was 256.0KiB, Chunk State:
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0xb0a060a00 of size 16384
Exception in thread Predictor3:
Traceback (most recent call last):
File "/home/mbz/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 715, in _do_call
return fn(*args)
File "/home/mbz/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 697, in _run_fn
status, run_metadata)
File "/home/mbz/anaconda3/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/mbz/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.InternalError: Dst tensor is not initialized.
[[Node: _recv_network1/X_0/_28 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_9__recv_network1/X_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:1"]]
[[Node: network1/Squeeze/_30 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:1", send_device_incarnation=1, tensor_name="edge_10_network1/Squeeze", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Environment info
Operating System: Ubuntu 14.04
Installed version of CUDA and cuDNN:
-rw-r--r-- 1 root root   322936 Aug 15  2015 /usr/local/cuda/lib64//libcudadevrt.a
lrwxrwxrwx 1 root root       16 Aug 15  2015 /usr/local/cuda/lib64//libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root       19 Aug 15  2015 /usr/local/cuda/lib64//libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root   383336 Aug 15  2015 /usr/local/cuda/lib64//libcudart.so.7.5.18
-rw-r--r-- 1 root root   720192 Aug 15  2015 /usr/local/cuda/lib64//libcudart_static.a
-rwxr-xr-x 1 root root 61453024 May 19 18:24 /usr/local/cuda/lib64//libcudnn.so
-rwxr-xr-x 1 root root 61453024 May 19 18:24 /usr/local/cuda/lib64//libcudnn.so.4
-rwxr-xr-x 1 root root 61453024 May 19 18:24 /usr/local/cuda/lib64//libcudnn.so.4.0.7
-rw-r--r-- 1 root root 62025862 May 19 18:24 /usr/local/cuda/lib64//libcudnn_static.a
If installed from binary pip package, provide:
Anaconda3 (python3)
TF v0.9

It might not be easy to find the problems without the code...
@zheng-xq Could you take a look at this please?

@mbz, are you using the latest TensorFlow? Both both GPU and pinned memory, the latest TensorFlow uses BFC allocator, which has a lot less fragmentation.

Maybe you need to close and restart your Terminal every time after you run your previous program.

Also please upload the entire log to website such as the following, and paste the link here.
http://pastebin.com/
As @jmchen-g mentioned, it is better to have a small repro case. Otherwise, it might be expected to fail if the model requests more memory than what the GPU has.

@mbz, uploading the full log to pastebin.com and paste the URL here is very helpful.
If you are using a 12G GPU, something is wrong with the memory limit BFC allocator sees:
I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats:
Limit: 503270604
It is only allowed 500M GPU memory. A few possibilities:

It is currently running on a smaller GPU.
The GPU config in the session only allows that amount of memory.

The full log is helpful in this case.

my bad, I should have been more clear. I'm ran the same experiment on a 12G gpu and it crashed after a few days. In order to reproduce it, I re ran the code with gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=0.05)) which resulted in the logs above.
I'm not sure if I understand what you mean by full log. TensorFlow also outputs See logs for memory state., but it doesn't say where can I find it.

Could you install the latest Cuda driver as well? From another issue, older Cuda drivers could leak memory.
https://github.com/tensorflow/tensorflow/issues/2728
You can redirect your TensorFlow output into another file, and then upload it.
 |& tee /tmp/tf_output.log

Closing due to lack of response, but please reopen if there is still a problem.

Sorry for late answer, wasn't available.
I put the full log here: http://pastebin.com/WkU1buwx
I think I have the latest drivers.

It seems that I cannot reopen the issue, it would be great if you do so.
Thanks

@mbz, it seems that you have two different GPUs active at the same time.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980 Ti, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K20c, pci bus id: 0000:02:00.0)
Could you try the following?

Set the environment variable CUDA_VISIBLE_DEVICES=1 before running TensorFlow. It should mask off the other GPU.
Your BFC log shows only about 250MB memory is allowed. Could you make sure you session config code didn't specify a smaller GPU memory limit?

Limit:                   251635302

maybe change the batch size would help?

When I change batch_size, the quality of training decreases. It should have had a better optimization.
Cuda version=10.0
Tensorflow-gpu==1.14.0
Python=3.6.4
...
Limit:<PHONE_NUMBER>
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>
NumAllocs:                  545033
MaxAllocSize:<PHONE_NUMBER>
W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at concat_op.cc:153 : Resource exhausted: OOM when allocating tensor with shape[64,2048,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Change the batch size

I'm getting this behaviour loading TF models with Triton Inference Server during the model warmup. I'm getting dumps like this:
...
2020-11-03 04:16:48.356608: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 2506240 totalling 7.17MiB
2020-11-03 04:16:48.356612: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2640384 totalling 2.52MiB
2020-11-03 04:16:48.356616: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 3051520 totalling 2.91MiB
2020-11-03 04:16:48.356619: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 3571968 totalling 3.41MiB
2020-11-03 04:16:48.356623: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 74 Chunks of size 3735552 totalling 263.62MiB
2020-11-03 04:16:48.356627: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 74 Chunks of size 4194304 totalling 296.00MiB
2020-11-03 04:16:48.356631: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 294 Chunks of size 6291456 totalling 1.72GiB
2020-11-03 04:16:48.356635: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 7641344 totalling 7.29MiB
2020-11-03 04:16:48.356639: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 22 Chunks of size 9980928 totalling 209.41MiB
2020-11-03 04:16:48.356643: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 10460160 totalling 9.98MiB
2020-11-03 04:16:48.356647: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 11162880 totalling 10.65MiB
2020-11-03 04:16:48.356651: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 12476160 totalling 11.90MiB
2020-11-03 04:16:48.356655: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 12478464 totalling 11.90MiB
2020-11-03 04:16:48.356659: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 12487168 totalling 35.73MiB
2020-11-03 04:16:48.356663: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 14589184 totalling 13.91MiB
2020-11-03 04:16:48.356667: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 14600192 totalling 13.92MiB
2020-11-03 04:16:48.356671: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 14982400 totalling 57.15MiB
2020-11-03 04:16:48.356675: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 15250944 totalling 29.09MiB
2020-11-03 04:16:48.356679: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 16777216 totalling 32.00MiB
2020-11-03 04:16:48.356683: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16951296 totalling 16.17MiB
2020-11-03 04:16:48.356687: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 17095424 totalling 32.61MiB
2020-11-03 04:16:48.356692: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 17847808 totalling 34.04MiB
2020-11-03 04:16:48.356696: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 19283456 totalling 36.78MiB
2020-11-03 04:16:48.356700: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 19288832 totalling 18.39MiB
2020-11-03 04:16:48.356705: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 19654656 totalling 18.74MiB
2020-11-03 04:16:48.356709: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 24789248 totalling 23.64MiB
2020-11-03 04:16:48.356719: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 7 Chunks of size 26758144 totalling 178.63MiB
2020-11-03 04:16:48.356724: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 27076352 totalling 25.82MiB
2020-11-03 04:16:48.356728: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 29264384 totalling 83.73MiB
2020-11-03 04:16:48.356732: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 29265920 totalling 27.91MiB
2020-11-03 04:16:48.356736: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 33860096 totalling 32.29MiB
2020-11-03 04:16:48.356740: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 34943232 totalling 33.32MiB
2020-11-03 04:16:48.356744: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 39245312 totalling 37.43MiB
2020-11-03 04:16:48.356748: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 41305344 totalling 39.39MiB
2020-11-03 04:16:48.356752: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 49077760 totalling 46.80MiB
2020-11-03 04:16:48.356756: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 9.00GiB
2020-11-03 04:16:48.356760: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_:<PHONE_NUMBER>0 memory_limit_:<PHONE_NUMBER>1 available bytes: 41 curr_region_allocation_bytes_:<PHONE_NUMBER>4
2020-11-03 04:16:48.356767: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit:<PHONE_NUMBER>1
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>6
NumAllocs:                   55572
MaxAllocSize:             49077760

2020-11-03 04:16:48.358419: W tensorflow/core/common_runtime/bfc_allocator.cc:424] ****************************************************************************************************

All the GPU memory is consumed and the warmup takes hours as we have 75 models on a 12GB 2080 :(


Mobile Landing

Hi guys.  Our beautiful design begins to show its weak spots on mobile.  We have an issue with overflow where a white area shows.  Also, I suggest on mobile, we make the .intro section a
min-height: calc(100vh - 6rem)

and that we get rid of some of the headers and P text.  Just the logo and the main header, plus button should be good.
I can do this if no one gets to it in the next couple days, but wanted to throw it out there.
Thanks!

About the "white space" I will look into that but on my chrome dev tool in any mobile view that does not show, not even on my phone so it's kinda weird. I will work on fixing any bug here and there, if anyone notice new ones let me know and I will work on it!


Replace MethodDoc.[Ch] with generation at CMake time

Is your feature request related to a problem?
MethodDoc.[Ch] are generated from .rst files. This allows us to have easily human readable and editable content documenting the CLI but still have that content appear in help(foo) from within the VisIt CLI python interpreter.
Having the generated files be part of our repo creates confusion especially for newbie or occasional developers. And, there is really no reason we need to keep the generated files in the repo.
Lets remove them from the repo and generate them at CMake time. This would eliminate any confusion over what to edit and eliminate the need for any CI (see proposed https://github.com/visit-dav/visit/pull/19879) too.

Can we get rid of .C and .h files associated with .xml and .code files???

Can we get rid of .C and .h files associated with .xml and .code files???

Seems a tad beyond the scope of this issue.

Can we get rid of .C and .h files associated with .xml and .code files???

Seems a tad beyond the scope of this issue.

Besides which, there are some that cannot be auto-regenerated due to specialized logic not easily codified in our xml generation scheme.

Besides which, there are some that cannot be auto-regenerated due to specialized logic not easily codified in our xml generation scheme.

Right but those cases are tagged as such and they represent a substantial minority of the cases, right? If so, I kinda sorta get why it might make sense to get rid of the majority of them. But, I think thats a different issue ticket if someone wants to file it...hint hint...for a special topics discussion...

We talked about this and decided it might be more trouble than its worth


NPM3D training question

Hi Thomas
Thank you for making your work publicly available.  I am trying to recreate your results from https://npm3d.fr/paris-lille-3d and do not get the same results (these are validation results).

mean
ground  buildings   poles    bollards trash_cans barriers pedestrians   cars    natural

63.1
97.3      89.1      64.1      38.4      27.1      59.7      14.3      89.8      87.9

I took the information from parameters.txt for the pretrained version but could you please confirm what parameters I should be using, what the validation split should be and if I am correct in using the NPM3D dataset as is i.e. Paris, Lille1 and 2 in the training folder and Dijon etc in the test folder.
Thanks
Justin

Hi @j2oj2o,
The standard validation cloud is Paris, as it is from a different city. If you used the parameters form the pretrained model, they should be pretty good. and you should obtain the same performances as the pretrained model.
Keep in mind that Paris is a very difficult cloud and the performances at validation are usually worse than on the test set.

Thanks, I did use Paris and what you say may explain what I am seeing.


az functionapp keys set returns invalid status 'Bad Request', properties object not present in request body

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az functionapp keys set
I also tried to use az rest to see if that would work:
az rest --method put --url "https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.Web/sites/REDACTED/host/default/functionKeys/testKey?api-version=2022-03-01" --body '{"testKey": ""}' --debug --verbose
(Also tried some non-empty text values for the key value)
Every time got this kind of error:
Bad Request({"Code":"BadRequest","Message":"Properties object is not present in the request body.","Target":null,"Details":[{"Message":"Properties object is not present in the request body."},{"Code":"BadRequest"},{"ErrorEntity":{"ExtendedCode":"51006","MessageTemplate":"{0} object is not present in the request body.","Parameters":["Properties"],"Code":"BadRequest","Message":"Properties object is not present in the request body."}}],"Innererror":null})
This is consistent in an Azure DevOps Ubuntu agent (Microsoft-hosted) as well as locally, Azure CLI 2.42 (reported below) and 2.43 - I updated after generating this ticket but same behavior.
Errors:
Operation returned an invalid status 'Bad Request'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az functionapp keys set --key-name {} --key-type {} --name {} -g {} --verbose --debug

Expected Behavior
The function app key gets created/updated correctly.
Environment Summary
Windows-10-10.0.19044-SP0
Python 3.10.8
Installer: MSI

azure-cli 2.42.0

Extensions:
account 0.2.3
datafactory 0.7.0
monitor-control-service 0.3.0

Dependencies:
msal 1.20.0
azure-mgmt-resource 21.1.0b1

Additional Context

route to CXP team

@plzm I believe this is not a cmdlet issue.  You are getting this error because your function app is not able to access the configured storage account either because of a network misconfiguration or because of an issue on the storage side.   This storage account gets referenced by the Function App Application Setting “AzureWebJobsStorage”.

Can you try creating the key from Azure Portal? or try creating a new function app and then using the cmdlet.

Thanks @SaurabhSharma-MSFT.
I was able to create keys once I created a Function without any private link/endpoint, and a storage account that accepted all public network traffic. I expect that Function with VNET_ROUTE_ALL and a storage account with private endpoint for all services is the issue.
Is there any way to parse the output of az functionapp keys set and determine that "Bad Request" actually means that a storage or network configuration issue is the problem? I ran with --debug and didn't see anything obvious.
I'll go ahead and close this issue but it would be great to have the error message be more specific - apologies if I missed this reason in the output somewhere.

One other comment - on the Function where this failed, I created another storage account that accepted all public network traffic and set its conn string on the Function AzureWebJobStorage, saved Config, also explicitly restarted Function app. That didn't help - no keys, and no blob containers created in the new storage account. I checked Function docs for storage acct requirements (no Premium, must be GP, same region etc. etc.) and confirmed meeting all of them.
Does reconfiguring a Function for a wide open storage account not fix the key issue?

One other comment - on the Function where this failed, I created another storage account that accepted all public network traffic and set its conn string on the Function AzureWebJobStorage, saved Config, also explicitly restarted Function app. That didn't help - no keys, and no blob containers created in the new storage account. I checked Function docs for storage acct requirements (no Premium, must be GP, same region etc. etc.) and confirmed meeting all of them.
Does reconfiguring a Function for a wide open storage account not fix the key issue?

@plzm I believe reconfiguration should fix this key issue.  In case you feel you don't any network issues connecting storage account, I suggest you to please open a Azure Support Ticket so that a Support engineer can investigate this connectivity this issue.


refacterd code for better flow/readablity

Q
A

Bug fix?
no - yes?

New feature?
no - yes?

New sample?
no - yes?

Related issues?
fixes #X, partially #Y, mentioned in #Z

What's in this Pull Request?
Removed function return statement (it is a :void function definition)
Execute code WHEN order exist
Removed HTML string building variable with Array.map method,
and removed it to the end of the code as right-hand assignment to .innerHTML
This makes code execution order and "flow" (human reading the code) the same
Less code is less options for errors
Note: this code still errors when the .data table was deleted from the DOM. Could be fixed with:
     const table: Element = this.domElement.querySelector(".data");
     if (this.orders && table) {

Guidance
You can delete this section when you are submitting the pull request.

Please update this PR information accordingly. We'll use this as part of our release notes in monthly communications.
Please target your PR to 'dev' branch.

Thanks for your contribution! Sharing is caring.

@Danny-Engelman,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

thx Danny, appreciated.


Feature Implementation: Community Showcase

Display user-created content, high scores, or interesting gameplay moments.
@YadavAkhileshh I would love to work on it, could you please assign this task to me?

@YadavAkhileshh hello!?


[simplify-cfg] Add a visitedBlocks set to hasSameUltimateSuccessor to…

… prevent infinite loops.
Previously, we were not handling properly blocks that we could visit multiple
times. In this commit, I added a SmallPtrSet to ensure that we handle all of the
same cases that we handled previously.
The key reason that we want to follow this approach rather than something else
is that the previous algorithm on purpose allowed for side-entrances from other
checks since often times when we have multiple checks, all of the .none branches
funnel together into a single ultimate block.
This can be seen by the need of this code to support the test two_chained_calls
in simplify_switch_enum_objc.sil.
rdar://55861081
(cherry picked from commit aa00865715aa4416c326fbd9cf3a490dcf4f3a8e)
Cherry-pick radar: rdar://55951429

— CCC Info —

Radar: rdar://55951429
Explanation: The optimizer was infinite looping in certain cases in Simplify CFG.
Scope of issue: Causes the compiler to never finish given the appropriate code.
Origination: This was introduced in swift-5.1 to reduce code-size.
Risk: Low.
Testing: Added a regression test for this case and verified that we do not infinite loop anymore.
Reviewers: Erik Eckstein

@swift-ci test

Build failed
Swift Test OS X Platform
Git Sha - db34e5f18eaeac660100b677a220bdb62b98f86d

Forgot to include the other cherry-pick commit that adds the option to enable testing with branch merging turned off.

@swift-ci test

@swift-ci test

Restarted the job.

Build failed
Swift Test OS X Platform
Git Sha - db34e5f18eaeac660100b677a220bdb62b98f86d

Build failed
Swift Test Linux Platform
Git Sha - db34e5f18eaeac660100b677a220bdb62b98f86d

@swift-ci nominate


Per-message deflate causes bad_alloc on message length of 0

The return code of ::deflate is not checked  - it returns Z_BUF_ERROR when:

It has already processed data before; and
It is passed a buffer with a length of zero

This happens when sending a message (other than the first message) over a websocket with a size of 0.
Then, resizing the output vector to size() - 4, which is a huge number as size() is zero, throws a bad_alloc exception.
I believe following RFC 7692 7.2.1 to the letter would fix this:
   2.  If the resulting data does not end with an empty DEFLATE block
       with no compression (the "BTYPE" bits are set to 00), append an
       empty DEFLATE block with no compression to the tail end.

@cake4289 - Thanks for the bug report! Is it possible for you to send a pull request?

Thanks @cake4289 for providing a fix that fast! :+1:

@cake4289 - Great, thanks!


abort trap error

to be fair, i think this is a node error, like many things:
> sharp('/Users/jong/Downloads/We0opiV.jpg').resize(50).toFile('test.jpg', function (){})
{ options:
   { width: 50,
     height: -1,
     canvas: 'c',
     angle: 0,
     withoutEnlargement: false,
     sharpen: false,
     progressive: false,
     sequentialRead: false,
     quality: 80,
     compressionLevel: 6,
     output: '__jpeg',
     fileIn: '/Users/jong/Downloads/We0opiV.jpg' } }
> Assertion failed: (enter->IsFunction()), function MakeDomainCallback, file ../src/node.cc, line 1027.
Abort trap: 6

This is when i execute it in the REPL.
OS X 10.9.3
Node 0.11.13

I can no longer reproduce this problem using the latest unstable Node.js v0.11.15. Feel free to re-open if the error shows up again.


updating links from rtd to jdox

Last commit! Just a quick update to the link on contam pages.

Hey @mfixstsci , I just merged this branch with the most recent master . I'll merge when checks have passed. Thanks for reviewing!


Add AuthController

Add AuthController with enpoints (see #43).
Add AccountRefreshToken entity for storing refresh tokens.
Made corrections in .editorconfig.

Closes #43

Looks like a very nice job done! 😎
I do, however, think that maybe @ZhakalenDk also should look through it if possible; it is approved from my side though. 😊

I'll check it out tomortow morning


403 CSRF Token Mismatch after some time

We use [nuxt-csurf](https://github.com/Morgbn/nuxt-csurf) implicitly via nuxt-security.
After visiting the application the CSRF Tokens are valid and all forms (with server side validation or submissins) work as expected.
If we keep the window open or use only client side navigation, the CSRF Token verification fails (403 CSRF Token Mismatch) after a while (approximately 10-15 minutes, not exactly measured) although the cookie __Host-csrf  and the header Csrf-Token are still send with the corresponding requests.
If the page is reloaded the requests work for another certain time again. The cookie remains the same, but the Csrf-Token gets updated.

Any update on the above? I'm also getting a similar issue.

Hello, does this happen in production ? or just in dev mode ?

This doesn't happen in the dev mode. It happens in the Production mode and that too intermitently. Like the first call gets passed but the 2nd fails. Sometimes the second call passes even though the first had failed.

Do you run your nuxt in multiple instances and without Session Affinity? if so I think we have a problem with this module!!!

@Morgbn We are facing the same kind of issue where we are running nuxt instances in different machine and we get into the same issue. Do you have any work around/implementation for this?

In the case of multiple instances behind a load balancer, sticky-session may need to be activated (you can find more information and how to enable sticky session here)


A question about pre-trained model

Hello, thanks for your great work. I wonder what dataset have you used to pre-trained. Did you use Synth90k and ICDAR2015? Thank you!

No, it's just used to show the net works.

No, it's just used to show the net works.

what dataset have you used？I don't understand the reply "No, it's just used to show the net works."

Hello, thanks for your great work. I wonder what dataset have you used to pre-trained. Did you use Synth90k and ICDAR2015? Thank you!

Could you please clarify this query?. I would also like to know the datasets that you used for the pretrained model.


STM32 Mailbox API communication issues

Information:
Board: STM32H747i_DISCOVERY
Zephyr version: 3.5.99
Zephyr SDK version: 0.16.5
Commit: https://github.com/zephyrproject-rtos/zephyr/commit/023d4838fbb8513f9d9ff6f26b065591131c347d

Hi everyone,
I've recently begun developing a CAN to OPC-UA modeler on the STM32H747i-Discovery Board. The idea is to utilize the M4 chip for CAN communication and the M7 for OPC-UA. The pre-defined device tree bindings support the mailbox feature for IPM (Inter-Processor Communication).
My code is equivalent to the example found at samples/boards/stm32/h7_dual_core.
The code compiles, and I can successfully send pings between both boards. However, when attempting to send data between the two devices, issues arise. The channel fails to send data with error code 122 (EMSGSIZE).
The code structure resembles the example found at samples/drivers/ipm/ipm_mcux/src, which demonstrates IPM communication working with a NXP board.
Is there a potential issue with the STM32 implementation of the mailbox? Would it be preferable to use IPC (Inter-Process Communication)? How can I encapsulate IPM within an IPC interface? Is this even possible?

Advice to check on https://github.com/zephyrproject-rtos/zephyr/pull/68758