GPU Usage Too Much

This model seems not very big, but it occupies fully GPU usages. And sometime memory not enough to allocate.
ERROR during Learning (not Training Loop):

WARNING:tensorflow:Calling GradientTape.gradient on a persistent tape inside its context is significantly less efficient than calling it outside the context (it causes the gradient ops to be recorded on the tape, leading to increased CPU and memory usage). Only call GradientTape.gradient inside the context if you actually want to trace the gradient in order to compute higher order derivatives.
W1003 11:44:05.744225<PHONE_NUMBER>94688 backprop.py:1039] Calling GradientTape.gradient on a persistent tape inside its context is significantly less efficient than calling it outside the context (it causes the gradient ops to be recorded on the tape, leading to increased CPU and memory usage). Only call GradientTape.gradient inside the context if you actually want to trace the gradient in order to compute higher order derivatives.
2020-10-03 11:44:07.931671: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:118] None of the MLIR optimization passes are enabled (registered 1)
2020-10-03 11:44:14.733424: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:118] None of the MLIR optimization passes are enabled (registered 1)
2020-10-03 11:44:23.213334: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 784.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-03 11:44:23.512142: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 784.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-03 11:44:24.609945: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.02GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-10-03 11:44:24.609968: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2020-10-03 11:44:37.214851: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.00GiB (rounded to<PHONE_NUMBER>)requested by op swap_in_gradient_tape/conv2_d/convolution_2/Conv2DBackpropFilter_0
Current allocation summary follows.
2020-10-03 11:44:37.214914: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc
2020-10-03 11:44:37.214936: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): 	Total Chunks: 108, Chunks in use: 107. 27.0KiB allocated for chunks. 26.8KiB in use in bin. 4.6KiB client-requested in use in bin.
2020-10-03 11:44:37.214955: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): 	Total Chunks: 37, Chunks in use: 36. 19.0KiB allocated for chunks. 18.2KiB in use in bin. 18.0KiB client-requested in use in bin.
2020-10-03 11:44:37.214967: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): 	Total Chunks: 7, Chunks in use: 7. 7.2KiB allocated for chunks. 7.2KiB in use in bin. 7.0KiB client-requested in use in bin.
2020-10-03 11:44:37.214978: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): 	Total Chunks: 3, Chunks in use: 2. 7.0KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2020-10-03 11:44:37.214989: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): 	Total Chunks: 1, Chunks in use: 0. 6.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215001: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): 	Total Chunks: 3, Chunks in use: 3. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.0KiB client-requested in use in bin.
2020-10-03 11:44:37.215015: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): 	Total Chunks: 5, Chunks in use: 5. 124.2KiB allocated for chunks. 124.2KiB in use in bin. 114.0KiB client-requested in use in bin.
2020-10-03 11:44:37.215028: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): 	Total Chunks: 2, Chunks in use: 1. 95.5KiB allocated for chunks. 51.0KiB in use in bin. 51.0KiB client-requested in use in bin.
2020-10-03 11:44:37.215041: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): 	Total Chunks: 18, Chunks in use: 16. 1.31MiB allocated for chunks. 1.13MiB in use in bin. 1011.0KiB client-requested in use in bin.
2020-10-03 11:44:37.215051: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215063: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): 	Total Chunks: 10, Chunks in use: 10. 2.50MiB allocated for chunks. 2.50MiB in use in bin. 2.50MiB client-requested in use in bin.
2020-10-03 11:44:37.215075: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): 	Total Chunks: 1, Chunks in use: 0. 896.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215085: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215094: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215104: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215114: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215123: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215139: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): 	Total Chunks: 1, Chunks in use: 1. 48.00MiB allocated for chunks. 48.00MiB in use in bin. 48.00MiB client-requested in use in bin.
2020-10-03 11:44:37.215150: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215161: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): 	Total Chunks: 1, Chunks in use: 0. 191.50MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:37.215173: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): 	Total Chunks: 14, Chunks in use: 11. 8.76GiB allocated for chunks. 7.00GiB in use in bin. 7.00GiB client-requested in use in bin.
2020-10-03 11:44:37.215183: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 1.00GiB was 256.00MiB, Chunk State:
2020-10-03 11:44:37.215205: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 268.98MiB | Requested Size: 1.0KiB | in_use: 0 | bin_num: 20, prev:   Size: 48.00MiB | Requested Size: 48.00MiB | in_use: 1 | bin_num: -1, next:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:37.215223: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 512.00MiB | Requested Size: 289.05MiB | in_use: 0 | bin_num: 20, prev:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1, next:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:37.215236: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 1017.22MiB | Requested Size: 512B | in_use: 0 | bin_num: 20, prev:   Size: 1.00GiB | Requested Size: 1.00GiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:37.215244: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size<PHONE_NUMBER>
2020-10-03 11:44:37.215256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000000 of size 256 next 1
2020-10-03 11:44:37.215265: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000100 of size 1280 next 2
2020-10-03 11:44:37.215272: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000600 of size 256 next 3
2020-10-03 11:44:37.215280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000700 of size 256 next 4
2020-10-03 11:44:37.215287: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000800 of size 256 next 5
2020-10-03 11:44:37.215295: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000900 of size 256 next 6
2020-10-03 11:44:37.215302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000a00 of size 256 next 7
2020-10-03 11:44:37.215309: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000b00 of size 256 next 13
2020-10-03 11:44:37.215316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000c00 of size 256 next 21
2020-10-03 11:44:37.215323: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000d00 of size 256 next 10
2020-10-03 11:44:37.215331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000e00 of size 256 next 11
2020-10-03 11:44:37.215338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000f00 of size 256 next 9
2020-10-03 11:44:37.215345: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001000 of size 256 next 8
2020-10-03 11:44:37.215352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001100 of size 256 next 25
2020-10-03 11:44:37.215360: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001200 of size 256 next 15
2020-10-03 11:44:37.215367: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001300 of size 256 next 16
2020-10-03 11:44:37.215392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001400 of size 256 next 17
2020-10-03 11:44:37.215406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001500 of size 256 next 18
2020-10-03 11:44:37.215419: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001600 of size 256 next 20
2020-10-03 11:44:37.215440: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001700 of size 512 next 29
2020-10-03 11:44:37.215455: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001900 of size 256 next 32
2020-10-03 11:44:37.215464: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001a00 of size 512 next 46
2020-10-03 11:44:37.215471: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001c00 of size 256 next 31
2020-10-03 11:44:37.215479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001d00 of size 512 next 33
2020-10-03 11:44:37.215486: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001f00 of size 512 next 34
2020-10-03 11:44:37.215493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002100 of size 256 next 37
2020-10-03 11:44:37.215500: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002200 of size 512 next 40
2020-10-03 11:44:37.215508: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002400 of size 130816 next 30
2020-10-03 11:44:37.215516: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200022300 of size 86784 next 19
2020-10-03 11:44:37.215527: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200037600 of size 12288 next 14
2020-10-03 11:44:37.215535: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a600 of size 512 next 35
2020-10-03 11:44:37.215542: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a800 of size 256 next 43
2020-10-03 11:44:37.215550: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a900 of size 256 next 39
2020-10-03 11:44:37.215557: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003aa00 of size 256 next 42
2020-10-03 11:44:37.215564: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ab00 of size 256 next 47
2020-10-03 11:44:37.215571: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ac00 of size 512 next 28
2020-10-03 11:44:37.215578: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ae00 of size 256 next 58
2020-10-03 11:44:37.215586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003af00 of size 256 next 51
2020-10-03 11:44:37.215593: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b000 of size 512 next 52
2020-10-03 11:44:37.215600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b200 of size 512 next 44
2020-10-03 11:44:37.215608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b400 of size 1024 next 45
2020-10-03 11:44:37.215615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b800 of size 256 next 89
2020-10-03 11:44:37.215622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b900 of size 256 next 53
2020-10-03 11:44:37.215630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ba00 of size 512 next 57
2020-10-03 11:44:37.215637: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003bc00 of size 256 next 95
2020-10-03 11:44:37.215644: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003bd00 of size 256 next 96
2020-10-03 11:44:37.215652: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003be00 of size 512 next 100
2020-10-03 11:44:37.215659: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c000 of size 512 next 178
2020-10-03 11:44:37.215668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c200 of size 256 next 226
2020-10-03 11:44:37.215675: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c300 of size 512 next 97
2020-10-03 11:44:37.215683: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c500 of size 256 next 101
2020-10-03 11:44:37.215690: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c600 of size 256 next 102
2020-10-03 11:44:37.215697: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c700 of size 256 next 103
2020-10-03 11:44:37.215704: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c800 of size 256 next 104
2020-10-03 11:44:37.215711: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c900 of size 256 next 105
2020-10-03 11:44:37.215719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ca00 of size 1024 next 106
2020-10-03 11:44:37.215726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ce00 of size 256 next 107
2020-10-03 11:44:37.215733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003cf00 of size 256 next 108
2020-10-03 11:44:37.215741: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d000 of size 256 next 109
2020-10-03 11:44:37.215748: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d100 of size 256 next 110
2020-10-03 11:44:37.215755: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d200 of size 256 next 111
2020-10-03 11:44:37.215762: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d300 of size 256 next 112
2020-10-03 11:44:37.215769: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d400 of size 256 next 113
2020-10-03 11:44:37.215777: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d500 of size 256 next 114
2020-10-03 11:44:37.215784: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d600 of size 256 next 115
2020-10-03 11:44:37.215791: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d700 of size 1024 next 116
2020-10-03 11:44:37.215798: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003db00 of size 256 next 118
2020-10-03 11:44:37.215806: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003dc00 of size 2048 next 119
2020-10-03 11:44:37.215813: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e400 of size 256 next 120
2020-10-03 11:44:37.215821: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e500 of size 256 next 121
2020-10-03 11:44:37.215828: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e600 of size 256 next 122
2020-10-03 11:44:37.215835: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e700 of size 256 next 123
2020-10-03 11:44:37.215842: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e800 of size 256 next 124
2020-10-03 11:44:37.215849: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e900 of size 256 next 125
2020-10-03 11:44:37.215857: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ea00 of size 256 next 64
2020-10-03 11:44:37.215864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003eb00 of size 512 next 62
2020-10-03 11:44:37.215871: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ed00 of size 256 next 79
2020-10-03 11:44:37.215879: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ee00 of size 12288 next 71
2020-10-03 11:44:37.215886: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200041e00 of size 512 next 48
2020-10-03 11:44:37.215893: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042000 of size 512 next 73
2020-10-03 11:44:37.215902: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042200 of size 256 next 68
2020-10-03 11:44:37.215910: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042300 of size 256 next 130
2020-10-03 11:44:37.215917: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042400 of size 1024 next 131
2020-10-03 11:44:37.215924: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042800 of size 256 next 132
2020-10-03 11:44:37.215931: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042900 of size 1024 next 135
2020-10-03 11:44:37.215938: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042d00 of size 256 next 136
2020-10-03 11:44:37.215946: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042e00 of size 256 next 137
2020-10-03 11:44:37.215953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042f00 of size 512 next 138
2020-10-03 11:44:37.215961: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200043100 of size 30208 next 36
2020-10-03 11:44:37.215969: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520004a700 of size 65536 next 38
2020-10-03 11:44:37.215977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520005a700 of size 26112 next 59
2020-10-03 11:44:37.215984: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f5200060d00 of size 256 next 87
2020-10-03 11:44:37.215991: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200060e00 of size 512 next 76
2020-10-03 11:44:37.215998: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061000 of size 512 next 63
2020-10-03 11:44:37.216005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061200 of size 512 next 67
2020-10-03 11:44:37.216013: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061400 of size 512 next 80
2020-10-03 11:44:37.216020: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061600 of size 512 next 70
2020-10-03 11:44:37.216027: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061800 of size 512 next 60
2020-10-03 11:44:37.216035: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061a00 of size 512 next 84
2020-10-03 11:44:37.216042: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061c00 of size 512 next 74
2020-10-03 11:44:37.216049: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061e00 of size 256 next 61
2020-10-03 11:44:37.216056: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061f00 of size 768 next 82
2020-10-03 11:44:37.216064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062200 of size 256 next 77
2020-10-03 11:44:37.216071: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062300 of size 256 next 85
2020-10-03 11:44:37.216078: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062400 of size 512 next 86
2020-10-03 11:44:37.216086: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062600 of size 13056 next 83
2020-10-03 11:44:37.216094: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065900 of size 256 next 90
2020-10-03 11:44:37.216101: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065a00 of size 256 next 81
2020-10-03 11:44:37.216108: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065b00 of size 256 next 91
2020-10-03 11:44:37.216115: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065c00 of size 256 next 92
2020-10-03 11:44:37.216122: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065d00 of size 256 next 93
2020-10-03 11:44:37.216133: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065e00 of size 18688 next 54
2020-10-03 11:44:37.216143: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520006a700 of size 65536 next 56
2020-10-03 11:44:37.216151: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520007a700 of size 65536 next 55
2020-10-03 11:44:37.216158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520008a700 of size 26112 next 99
2020-10-03 11:44:37.216165: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200090d00 of size 65536 next 66
2020-10-03 11:44:37.216173: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000a0d00 of size 104704 next 22
2020-10-03 11:44:37.216183: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000ba600 of size 262144 next 23
2020-10-03 11:44:37.216191: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000fa600 of size 262144 next 12
2020-10-03 11:44:37.216199: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520013a600 of size 262144 next 27
2020-10-03 11:44:37.216209: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520017a600 of size 536870912 next 94
2020-10-03 11:44:37.216220: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f522017a600 of size 541065216 next 24
2020-10-03 11:44:37.216228: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f524057a600 of size 536870912 next 26
2020-10-03 11:44:37.216235: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526057a600 of size 262144 next 88
2020-10-03 11:44:37.216243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605ba600 of size 52224 next 117
2020-10-03 11:44:37.216251: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52605c7200 of size 6144 next 222
2020-10-03 11:44:37.216258: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605c8a00 of size 512 next 215
2020-10-03 11:44:37.216265: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52605c8c00 of size 45568 next 127
2020-10-03 11:44:37.216273: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605d3e00 of size 65536 next 75
2020-10-03 11:44:37.216280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605e3e00 of size 65536 next 49
2020-10-03 11:44:37.216288: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605f3e00 of size 262144 next 72
2020-10-03 11:44:37.216295: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260633e00 of size 262144 next 65
2020-10-03 11:44:37.216302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260673e00 of size 262144 next 126
2020-10-03 11:44:37.216309: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52606b3e00 of size 65536 next 129
2020-10-03 11:44:37.216317: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52606c3e00 of size 262144 next 133
2020-10-03 11:44:37.216324: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260703e00 of size 65536 next 139
2020-10-03 11:44:37.216331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260713e00 of size 262144 next 140
2020-10-03 11:44:37.216338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260753e00 of size 256 next 141
2020-10-03 11:44:37.216346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260753f00 of size 26112 next 142
2020-10-03 11:44:37.216353: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526075a500 of size 256 next 143
2020-10-03 11:44:37.216360: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526075a600 of size 262144 next 144
2020-10-03 11:44:37.216368: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079a600 of size 512 next 145
2020-10-03 11:44:37.216375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079a800 of size 512 next 146
2020-10-03 11:44:37.216383: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079aa00 of size 512 next 147
2020-10-03 11:44:37.216391: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079ac00 of size 512 next 148
2020-10-03 11:44:37.216398: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079ae00 of size 512 next 149
2020-10-03 11:44:37.216406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079b000 of size 65536 next 150
2020-10-03 11:44:37.216413: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607ab000 of size 512 next 151
2020-10-03 11:44:37.216420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607ab200 of size 65536 next 152
2020-10-03 11:44:37.216428: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb200 of size 256 next 153
2020-10-03 11:44:37.216435: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb300 of size 256 next 154
2020-10-03 11:44:37.216442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb400 of size 256 next 155
2020-10-03 11:44:37.216450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb500 of size 256 next 156
2020-10-03 11:44:37.216457: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb600 of size 256 next 157
2020-10-03 11:44:37.216464: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb700 of size 256 next 158
2020-10-03 11:44:37.216472: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb800 of size 256 next 159
2020-10-03 11:44:37.216479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb900 of size 256 next 160
2020-10-03 11:44:37.216487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bba00 of size 256 next 161
2020-10-03 11:44:37.216494: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbb00 of size 256 next 162
2020-10-03 11:44:37.216501: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbc00 of size 256 next 163
2020-10-03 11:44:37.216508: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbd00 of size 256 next 164
2020-10-03 11:44:37.216515: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbe00 of size 256 next 165
2020-10-03 11:44:37.216523: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbf00 of size 256 next 166
2020-10-03 11:44:37.216530: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc000 of size 256 next 167
2020-10-03 11:44:37.216537: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc100 of size 256 next 168
2020-10-03 11:44:37.216545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc200 of size 256 next 169
2020-10-03 11:44:37.216552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc300 of size 256 next 170
2020-10-03 11:44:37.216559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc400 of size 256 next 171
2020-10-03 11:44:37.216566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc500 of size 256 next 172
2020-10-03 11:44:37.216574: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607bc600 of size 3072 next 184
2020-10-03 11:44:37.216581: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd200 of size 256 next 185
2020-10-03 11:44:37.216588: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd300 of size 256 next 186
2020-10-03 11:44:37.216596: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd400 of size 256 next 187
2020-10-03 11:44:37.216603: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd500 of size 256 next 188
2020-10-03 11:44:37.216610: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd600 of size 256 next 189
2020-10-03 11:44:37.216619: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd700 of size 256 next 190
2020-10-03 11:44:37.216626: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd800 of size 256 next 191
2020-10-03 11:44:37.216634: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd900 of size 256 next 192
2020-10-03 11:44:37.216641: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bda00 of size 256 next 193
2020-10-03 11:44:37.216648: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdb00 of size 256 next 194
2020-10-03 11:44:37.216656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdc00 of size 256 next 195
2020-10-03 11:44:37.216669: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdd00 of size 256 next 196
2020-10-03 11:44:37.216677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bde00 of size 256 next 197
2020-10-03 11:44:37.216684: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdf00 of size 256 next 198
2020-10-03 11:44:37.216691: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be000 of size 256 next 199
2020-10-03 11:44:37.216698: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be100 of size 256 next 200
2020-10-03 11:44:37.216706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be200 of size 256 next 201
2020-10-03 11:44:37.216713: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be300 of size 1024 next 183
2020-10-03 11:44:37.216720: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be700 of size 512 next 243
2020-10-03 11:44:37.216728: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be900 of size 512 next 204
2020-10-03 11:44:37.216735: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607beb00 of size 768 next 212
2020-10-03 11:44:37.216742: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bee00 of size 2048 next 213
2020-10-03 11:44:37.216750: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607bf600 of size 917504 next 245
2020-10-03 11:44:37.216757: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526089f600 of size 65536 next 246
2020-10-03 11:44:37.216764: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608af600 of size 65536 next 244
2020-10-03 11:44:37.216772: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52608bf600 of size 65536 next 221
2020-10-03 11:44:37.216779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608cf600 of size 117760 next 239
2020-10-03 11:44:37.216787: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52608ec200 of size 78848 next 236
2020-10-03 11:44:37.216794: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608ff600 of size 200807424 next 220
2020-10-03 11:44:37.216802: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526c880a00 of size 50331648 next 224
2020-10-03 11:44:37.216810: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f526f880a00 of size 282042368 next 50
2020-10-03 11:44:37.216818: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f528057aa00 of size 536871936 next 41
2020-10-03 11:44:37.216826: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52a057ae00 of size 536870912 next 78
2020-10-03 11:44:37.216833: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52c057ae00 of size 536870912 next 69
2020-10-03 11:44:37.216840: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52e057ae00 of size 536871936 next 128
2020-10-03 11:44:37.216848: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f530057b200 of size 536871936 next 134
2020-10-03 11:44:37.216856: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f532057b600 of size<PHONE_NUMBER> next 219
2020-10-03 11:44:37.216864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f536057b600 of size<PHONE_NUMBER> next 241
2020-10-03 11:44:37.216871: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f53a057b600 of size 536870912 next 242
2020-10-03 11:44:37.216879: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f53c057b600 of size<PHONE_NUMBER> next 225
2020-10-03 11:44:37.216886: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f540057b600 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2020-10-03 11:44:37.216893: I tensorflow/core/common_runtime/bfc_allocator.cc:1031]      Summary of in-use Chunks by size:
2020-10-03 11:44:37.216907: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 107 Chunks of size 256 totalling 26.8KiB
2020-10-03 11:44:37.216917: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 35 Chunks of size 512 totalling 17.5KiB
2020-10-03 11:44:37.216926: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 768 totalling 768B
2020-10-03 11:44:37.216934: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 6 Chunks of size 1024 totalling 6.0KiB
2020-10-03 11:44:37.216943: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-03 11:44:37.216952: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 2048 totalling 4.0KiB
2020-10-03 11:44:37.216961: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 12288 totalling 24.0KiB
2020-10-03 11:44:37.216970: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13056 totalling 12.8KiB
2020-10-03 11:44:37.216979: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18688 totalling 18.2KiB
2020-10-03 11:44:37.216988: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 26112 totalling 76.5KiB
2020-10-03 11:44:37.216998: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 30208 totalling 29.5KiB
2020-10-03 11:44:37.217007: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 52224 totalling 51.0KiB
2020-10-03 11:44:37.217018: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 65536 totalling 768.0KiB
2020-10-03 11:44:37.217027: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 78848 totalling 77.0KiB
2020-10-03 11:44:37.217037: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 86784 totalling 84.8KiB
2020-10-03 11:44:37.217046: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 104704 totalling 102.2KiB
2020-10-03 11:44:37.217055: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 130816 totalling 127.8KiB
2020-10-03 11:44:37.217064: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 10 Chunks of size 262144 totalling 2.50MiB
2020-10-03 11:44:37.217073: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 50331648 totalling 48.00MiB
2020-10-03 11:44:37.217082: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 536870912 totalling 2.00GiB
2020-10-03 11:44:37.217091: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 536871936 totalling 1.50GiB
2020-10-03 11:44:37.217100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 541065216 totalling 516.00MiB
2020-10-03 11:44:37.217109: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size<PHONE_NUMBER> totalling 3.00GiB
2020-10-03 11:44:37.217117: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 7.05GiB
2020-10-03 11:44:37.217126: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_:<PHONE_NUMBER> memory_limit_:<PHONE_NUMBER> available bytes: 128 curr_region_allocation_bytes_:<PHONE_NUMBER>4
2020-10-03 11:44:37.217142: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit:<PHONE_NUMBER>
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>
NumAllocs:                       30989
MaxAllocSize:<PHONE_NUMBER>
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2020-10-03` 11:44:37.217175: W tensorflow/core/common_runtime/bfc_allocator.cc:439] _______************************************************___________ 2020-10-03 11:44:37.217215: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at gpu_swapping_kernels.cc:72 : Resource exhausted: OOM when allocating tensor with shape[256,64,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2020-10-03 11:44:37.217248: W tensorflow/core/common_runtime/executor.cc:1086] [/job:localhost/replica:0/task:0/device:GPU:0] Executor start aborting: Resource exhausted: OOM when allocating tensor with shape[256,64,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node swap_in_gradient_tape/conv2_d/convolution_2/Conv2DBackpropFilter_0}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2020-10-03 11:44:47.217504: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.00GiB (rounded to<PHONE_NUMBER>)requested by op conv2_d/BiasAdd_3-0-0-TransposeNCHWToNHWC-LayoutOptimizer
Current allocation summary follows.
2020-10-03 11:44:47.217571: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc
2020-10-03 11:44:47.217596: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): 	Total Chunks: 108, Chunks in use: 107. 27.0KiB allocated for chunks. 26.8KiB in use in bin. 4.6KiB client-requested in use in bin.
2020-10-03 11:44:47.217611: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): 	Total Chunks: 37, Chunks in use: 36. 19.0KiB allocated for chunks. 18.2KiB in use in bin. 18.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217623: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): 	Total Chunks: 7, Chunks in use: 7. 7.2KiB allocated for chunks. 7.2KiB in use in bin. 7.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217633: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): 	Total Chunks: 3, Chunks in use: 2. 7.0KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217645: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): 	Total Chunks: 1, Chunks in use: 0. 6.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217657: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): 	Total Chunks: 3, Chunks in use: 3. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217698: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): 	Total Chunks: 5, Chunks in use: 5. 124.2KiB allocated for chunks. 124.2KiB in use in bin. 114.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217723: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): 	Total Chunks: 2, Chunks in use: 1. 95.5KiB allocated for chunks. 51.0KiB in use in bin. 51.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217746: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): 	Total Chunks: 18, Chunks in use: 16. 1.31MiB allocated for chunks. 1.13MiB in use in bin. 1011.0KiB client-requested in use in bin.
2020-10-03 11:44:47.217767: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217786: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): 	Total Chunks: 10, Chunks in use: 10. 2.50MiB allocated for chunks. 2.50MiB in use in bin. 2.50MiB client-requested in use in bin.
2020-10-03 11:44:47.217799: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): 	Total Chunks: 1, Chunks in use: 0. 896.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217809: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217819: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217829: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217839: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217848: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217865: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): 	Total Chunks: 1, Chunks in use: 1. 48.00MiB allocated for chunks. 48.00MiB in use in bin. 48.00MiB client-requested in use in bin.
2020-10-03 11:44:47.217876: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217888: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): 	Total Chunks: 1, Chunks in use: 0. 191.50MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-03 11:44:47.217900: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): 	Total Chunks: 14, Chunks in use: 11. 8.76GiB allocated for chunks. 7.00GiB in use in bin. 7.00GiB client-requested in use in bin.
2020-10-03 11:44:47.217914: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 1.00GiB was 256.00MiB, Chunk State:
2020-10-03 11:44:47.217949: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 268.98MiB | Requested Size: 1.0KiB | in_use: 0 | bin_num: 20, prev:   Size: 48.00MiB | Requested Size: 48.00MiB | in_use: 1 | bin_num: -1, next:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:47.217984: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 512.00MiB | Requested Size: 289.05MiB | in_use: 0 | bin_num: 20, prev:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1, next:   Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:47.218011: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 1017.22MiB | Requested Size: 512B | in_use: 0 | bin_num: 20, prev:   Size: 1.00GiB | Requested Size: 1.00GiB | in_use: 1 | bin_num: -1
2020-10-03 11:44:47.218023: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size<PHONE_NUMBER>
2020-10-03 11:44:47.218036: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000000 of size 256 next 1
2020-10-03 11:44:47.218045: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000100 of size 1280 next 2
2020-10-03 11:44:47.218052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000600 of size 256 next 3
2020-10-03 11:44:47.218059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000700 of size 256 next 4
2020-10-03 11:44:47.218066: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000800 of size 256 next 5
2020-10-03 11:44:47.218074: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000900 of size 256 next 6
2020-10-03 11:44:47.218081: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000a00 of size 256 next 7
2020-10-03 11:44:47.218088: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000b00 of size 256 next 13
2020-10-03 11:44:47.218095: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000c00 of size 256 next 21
2020-10-03 11:44:47.218102: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000d00 of size 256 next 10
2020-10-03 11:44:47.218109: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000e00 of size 256 next 11
2020-10-03 11:44:47.218117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200000f00 of size 256 next 9
2020-10-03 11:44:47.218124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001000 of size 256 next 8
2020-10-03 11:44:47.218131: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001100 of size 256 next 25
2020-10-03 11:44:47.218138: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001200 of size 256 next 15
2020-10-03 11:44:47.218145: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001300 of size 256 next 16
2020-10-03 11:44:47.218164: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001400 of size 256 next 17
2020-10-03 11:44:47.218173: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001500 of size 256 next 18
2020-10-03 11:44:47.218180: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001600 of size 256 next 20
2020-10-03 11:44:47.218189: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001700 of size 512 next 29
2020-10-03 11:44:47.218196: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001900 of size 256 next 32
2020-10-03 11:44:47.218204: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001a00 of size 512 next 46
2020-10-03 11:44:47.218211: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001c00 of size 256 next 31
2020-10-03 11:44:47.218218: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001d00 of size 512 next 33
2020-10-03 11:44:47.218225: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200001f00 of size 512 next 34
2020-10-03 11:44:47.218232: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002100 of size 256 next 37
2020-10-03 11:44:47.218239: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002200 of size 512 next 40
2020-10-03 11:44:47.218247: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200002400 of size 130816 next 30
2020-10-03 11:44:47.218255: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200022300 of size 86784 next 19
2020-10-03 11:44:47.218262: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200037600 of size 12288 next 14
2020-10-03 11:44:47.218269: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a600 of size 512 next 35
2020-10-03 11:44:47.218276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a800 of size 256 next 43
2020-10-03 11:44:47.218283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003a900 of size 256 next 39
2020-10-03 11:44:47.218291: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003aa00 of size 256 next 42
2020-10-03 11:44:47.218298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ab00 of size 256 next 47
2020-10-03 11:44:47.218305: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ac00 of size 512 next 28
2020-10-03 11:44:47.218312: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ae00 of size 256 next 58
2020-10-03 11:44:47.218319: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003af00 of size 256 next 51
2020-10-03 11:44:47.218326: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b000 of size 512 next 52
2020-10-03 11:44:47.218333: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b200 of size 512 next 44
2020-10-03 11:44:47.218340: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b400 of size 1024 next 45
2020-10-03 11:44:47.218348: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b800 of size 256 next 89
2020-10-03 11:44:47.218355: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003b900 of size 256 next 53
2020-10-03 11:44:47.218362: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ba00 of size 512 next 57
2020-10-03 11:44:47.218369: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003bc00 of size 256 next 95
2020-10-03 11:44:47.218377: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003bd00 of size 256 next 96
2020-10-03 11:44:47.218384: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003be00 of size 512 next 100
2020-10-03 11:44:47.218391: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c000 of size 512 next 178
2020-10-03 11:44:47.218400: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c200 of size 256 next 226
2020-10-03 11:44:47.218407: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c300 of size 512 next 97
2020-10-03 11:44:47.218415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c500 of size 256 next 101
2020-10-03 11:44:47.218422: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c600 of size 256 next 102
2020-10-03 11:44:47.218429: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c700 of size 256 next 103
2020-10-03 11:44:47.218437: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c800 of size 256 next 104
2020-10-03 11:44:47.218444: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003c900 of size 256 next 105
2020-10-03 11:44:47.218451: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ca00 of size 1024 next 106
2020-10-03 11:44:47.218458: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ce00 of size 256 next 107
2020-10-03 11:44:47.218465: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003cf00 of size 256 next 108
2020-10-03 11:44:47.218472: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d000 of size 256 next 109
2020-10-03 11:44:47.218479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d100 of size 256 next 110
2020-10-03 11:44:47.218487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d200 of size 256 next 111
2020-10-03 11:44:47.218494: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d300 of size 256 next 112
2020-10-03 11:44:47.218501: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d400 of size 256 next 113
2020-10-03 11:44:47.218508: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d500 of size 256 next 114
2020-10-03 11:44:47.218515: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d600 of size 256 next 115
2020-10-03 11:44:47.218522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003d700 of size 1024 next 116
2020-10-03 11:44:47.218529: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003db00 of size 256 next 118
2020-10-03 11:44:47.218538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003dc00 of size 2048 next 119
2020-10-03 11:44:47.218545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e400 of size 256 next 120
2020-10-03 11:44:47.218552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e500 of size 256 next 121
2020-10-03 11:44:47.218560: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e600 of size 256 next 122
2020-10-03 11:44:47.218566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e700 of size 256 next 123
2020-10-03 11:44:47.218573: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e800 of size 256 next 124
2020-10-03 11:44:47.218581: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003e900 of size 256 next 125
2020-10-03 11:44:47.218588: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ea00 of size 256 next 64
2020-10-03 11:44:47.218595: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003eb00 of size 512 next 62
2020-10-03 11:44:47.218602: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ed00 of size 256 next 79
2020-10-03 11:44:47.218610: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520003ee00 of size 12288 next 71
2020-10-03 11:44:47.218617: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200041e00 of size 512 next 48
2020-10-03 11:44:47.218624: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042000 of size 512 next 73
2020-10-03 11:44:47.218632: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042200 of size 256 next 68
2020-10-03 11:44:47.218640: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042300 of size 256 next 130
2020-10-03 11:44:47.218647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042400 of size 1024 next 131
2020-10-03 11:44:47.218655: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042800 of size 256 next 132
2020-10-03 11:44:47.218662: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042900 of size 1024 next 135
2020-10-03 11:44:47.218669: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042d00 of size 256 next 136
2020-10-03 11:44:47.218676: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042e00 of size 256 next 137
2020-10-03 11:44:47.218683: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200042f00 of size 512 next 138
2020-10-03 11:44:47.218691: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200043100 of size 30208 next 36
2020-10-03 11:44:47.218699: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520004a700 of size 65536 next 38
2020-10-03 11:44:47.218707: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520005a700 of size 26112 next 59
2020-10-03 11:44:47.218715: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f5200060d00 of size 256 next 87
2020-10-03 11:44:47.218722: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200060e00 of size 512 next 76
2020-10-03 11:44:47.218730: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061000 of size 512 next 63
2020-10-03 11:44:47.218737: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061200 of size 512 next 67
2020-10-03 11:44:47.218744: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061400 of size 512 next 80
2020-10-03 11:44:47.218751: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061600 of size 512 next 70
2020-10-03 11:44:47.218759: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061800 of size 512 next 60
2020-10-03 11:44:47.218766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061a00 of size 512 next 84
2020-10-03 11:44:47.218773: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061c00 of size 512 next 74
2020-10-03 11:44:47.218780: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061e00 of size 256 next 61
2020-10-03 11:44:47.218788: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200061f00 of size 768 next 82
2020-10-03 11:44:47.218795: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062200 of size 256 next 77
2020-10-03 11:44:47.218802: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062300 of size 256 next 85
2020-10-03 11:44:47.218809: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062400 of size 512 next 86
2020-10-03 11:44:47.218817: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200062600 of size 13056 next 83
2020-10-03 11:44:47.218825: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065900 of size 256 next 90
2020-10-03 11:44:47.218832: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065a00 of size 256 next 81
2020-10-03 11:44:47.218839: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065b00 of size 256 next 91
2020-10-03 11:44:47.218846: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065c00 of size 256 next 92
2020-10-03 11:44:47.218853: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065d00 of size 256 next 93
2020-10-03 11:44:47.218861: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200065e00 of size 18688 next 54
2020-10-03 11:44:47.218870: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520006a700 of size 65536 next 56
2020-10-03 11:44:47.218877: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520007a700 of size 65536 next 55
2020-10-03 11:44:47.218885: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520008a700 of size 26112 next 99
2020-10-03 11:44:47.218892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5200090d00 of size 65536 next 66
2020-10-03 11:44:47.218899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000a0d00 of size 104704 next 22
2020-10-03 11:44:47.218907: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000ba600 of size 262144 next 23
2020-10-03 11:44:47.218915: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52000fa600 of size 262144 next 12
2020-10-03 11:44:47.218922: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520013a600 of size 262144 next 27
2020-10-03 11:44:47.218930: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f520017a600 of size 536870912 next 94
2020-10-03 11:44:47.218938: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f522017a600 of size 541065216 next 24
2020-10-03 11:44:47.218946: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f524057a600 of size 536870912 next 26
2020-10-03 11:44:47.218953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526057a600 of size 262144 next 88
2020-10-03 11:44:47.218960: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605ba600 of size 52224 next 117
2020-10-03 11:44:47.218968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52605c7200 of size 6144 next 222
2020-10-03 11:44:47.218975: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605c8a00 of size 512 next 215
2020-10-03 11:44:47.218983: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52605c8c00 of size 45568 next 127
2020-10-03 11:44:47.218990: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605d3e00 of size 65536 next 75
2020-10-03 11:44:47.218997: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605e3e00 of size 65536 next 49
2020-10-03 11:44:47.219005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52605f3e00 of size 262144 next 72
2020-10-03 11:44:47.219012: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260633e00 of size 262144 next 65
2020-10-03 11:44:47.219019: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260673e00 of size 262144 next 126
2020-10-03 11:44:47.219026: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52606b3e00 of size 65536 next 129
2020-10-03 11:44:47.219033: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52606c3e00 of size 262144 next 133
2020-10-03 11:44:47.219040: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260703e00 of size 65536 next 139
2020-10-03 11:44:47.219047: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260713e00 of size 262144 next 140
2020-10-03 11:44:47.219055: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260753e00 of size 256 next 141
2020-10-03 11:44:47.219062: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f5260753f00 of size 26112 next 142
2020-10-03 11:44:47.219069: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526075a500 of size 256 next 143
2020-10-03 11:44:47.219076: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526075a600 of size 262144 next 144
2020-10-03 11:44:47.219083: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079a600 of size 512 next 145
2020-10-03 11:44:47.219090: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079a800 of size 512 next 146
2020-10-03 11:44:47.219099: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079aa00 of size 512 next 147
2020-10-03 11:44:47.219106: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079ac00 of size 512 next 148
2020-10-03 11:44:47.219114: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079ae00 of size 512 next 149
2020-10-03 11:44:47.219121: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526079b000 of size 65536 next 150
2020-10-03 11:44:47.219128: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607ab000 of size 512 next 151
2020-10-03 11:44:47.219135: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607ab200 of size 65536 next 152
2020-10-03 11:44:47.219142: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb200 of size 256 next 153
2020-10-03 11:44:47.219149: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb300 of size 256 next 154
2020-10-03 11:44:47.219156: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb400 of size 256 next 155
2020-10-03 11:44:47.219163: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb500 of size 256 next 156
2020-10-03 11:44:47.219170: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb600 of size 256 next 157
2020-10-03 11:44:47.219177: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb700 of size 256 next 158
2020-10-03 11:44:47.219184: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb800 of size 256 next 159
2020-10-03 11:44:47.219191: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bb900 of size 256 next 160
2020-10-03 11:44:47.219198: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bba00 of size 256 next 161
2020-10-03 11:44:47.219206: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbb00 of size 256 next 162
2020-10-03 11:44:47.219213: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbc00 of size 256 next 163
2020-10-03 11:44:47.219220: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbd00 of size 256 next 164
2020-10-03 11:44:47.219227: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbe00 of size 256 next 165
2020-10-03 11:44:47.219234: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bbf00 of size 256 next 166
2020-10-03 11:44:47.219241: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc000 of size 256 next 167
2020-10-03 11:44:47.219248: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc100 of size 256 next 168
2020-10-03 11:44:47.219255: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc200 of size 256 next 169
2020-10-03 11:44:47.219262: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc300 of size 256 next 170
2020-10-03 11:44:47.219269: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc400 of size 256 next 171
2020-10-03 11:44:47.219276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bc500 of size 256 next 172
2020-10-03 11:44:47.219283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607bc600 of size 3072 next 184
2020-10-03 11:44:47.219290: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd200 of size 256 next 185
2020-10-03 11:44:47.219297: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd300 of size 256 next 186
2020-10-03 11:44:47.219304: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd400 of size 256 next 187
2020-10-03 11:44:47.219311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd500 of size 256 next 188
2020-10-03 11:44:47.219318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd600 of size 256 next 189
2020-10-03 11:44:47.219327: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd700 of size 256 next 190
2020-10-03 11:44:47.219334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd800 of size 256 next 191
2020-10-03 11:44:47.219342: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bd900 of size 256 next 192
2020-10-03 11:44:47.219348: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bda00 of size 256 next 193
2020-10-03 11:44:47.219355: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdb00 of size 256 next 194
2020-10-03 11:44:47.219362: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdc00 of size 256 next 195
2020-10-03 11:44:47.219370: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdd00 of size 256 next 196
2020-10-03 11:44:47.219377: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bde00 of size 256 next 197
2020-10-03 11:44:47.219384: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bdf00 of size 256 next 198
2020-10-03 11:44:47.219391: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be000 of size 256 next 199
2020-10-03 11:44:47.219398: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be100 of size 256 next 200
2020-10-03 11:44:47.219405: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be200 of size 256 next 201
2020-10-03 11:44:47.219412: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be300 of size 1024 next 183
2020-10-03 11:44:47.219419: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be700 of size 512 next 243
2020-10-03 11:44:47.219426: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607be900 of size 512 next 204
2020-10-03 11:44:47.219434: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607beb00 of size 768 next 212
2020-10-03 11:44:47.219441: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52607bee00 of size 2048 next 213
2020-10-03 11:44:47.219448: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52607bf600 of size 917504 next 245
2020-10-03 11:44:47.219456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526089f600 of size 65536 next 246
2020-10-03 11:44:47.219463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608af600 of size 65536 next 244
2020-10-03 11:44:47.219470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52608bf600 of size 65536 next 221
2020-10-03 11:44:47.219477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608cf600 of size 117760 next 239
2020-10-03 11:44:47.219484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52608ec200 of size 78848 next 236
2020-10-03 11:44:47.219492: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52608ff600 of size 200807424 next 220
2020-10-03 11:44:47.219500: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f526c880a00 of size 50331648 next 224
2020-10-03 11:44:47.219507: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f526f880a00 of size 282042368 next 50
2020-10-03 11:44:47.219515: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f528057aa00 of size 536871936 next 41
2020-10-03 11:44:47.219522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52a057ae00 of size 536870912 next 78
2020-10-03 11:44:47.219530: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f52c057ae00 of size 536870912 next 69
2020-10-03 11:44:47.219537: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f52e057ae00 of size 536871936 next 128
2020-10-03 11:44:47.219544: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f530057b200 of size 536871936 next 134
2020-10-03 11:44:47.219554: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f532057b600 of size<PHONE_NUMBER> next 219
2020-10-03 11:44:47.219562: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f536057b600 of size<PHONE_NUMBER> next 241
2020-10-03 11:44:47.219569: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f53a057b600 of size 536870912 next 242
2020-10-03 11:44:47.219577: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f53c057b600 of size<PHONE_NUMBER> next 225
2020-10-03 11:44:47.219585: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f540057b600 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2020-10-03 11:44:47.219591: I tensorflow/core/common_runtime/bfc_allocator.cc:1031]      Summary of in-use Chunks by size:
2020-10-03 11:44:47.219604: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 107 Chunks of size 256 totalling 26.8KiB
2020-10-03 11:44:47.219614: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 35 Chunks of size 512 totalling 17.5KiB
2020-10-03 11:44:47.219623: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 768 totalling 768B
2020-10-03 11:44:47.219631: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 6 Chunks of size 1024 totalling 6.0KiB
2020-10-03 11:44:47.219640: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-03 11:44:47.219648: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 2048 totalling 4.0KiB
2020-10-03 11:44:47.219658: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 12288 totalling 24.0KiB
2020-10-03 11:44:47.219667: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13056 totalling 12.8KiB
2020-10-03 11:44:47.219675: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18688 totalling 18.2KiB
2020-10-03 11:44:47.219685: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 26112 totalling 76.5KiB
2020-10-03 11:44:47.219694: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 30208 totalling 29.5KiB
2020-10-03 11:44:47.219702: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 52224 totalling 51.0KiB
2020-10-03 11:44:47.219711: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 65536 totalling 768.0KiB
2020-10-03 11:44:47.219721: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 78848 totalling 77.0KiB
2020-10-03 11:44:47.219730: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 86784 totalling 84.8KiB
2020-10-03 11:44:47.219739: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 104704 totalling 102.2KiB
2020-10-03 11:44:47.219748: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 130816 totalling 127.8KiB
2020-10-03 11:44:47.219757: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 10 Chunks of size 262144 totalling 2.50MiB
2020-10-03 11:44:47.219766: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 50331648 totalling 48.00MiB
2020-10-03 11:44:47.219775: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 536870912 totalling 2.00GiB
2020-10-03 11:44:47.219783: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 536871936 totalling 1.50GiB
2020-10-03 11:44:47.219793: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 541065216 totalling 516.00MiB
2020-10-03 11:44:47.219802: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size<PHONE_NUMBER> totalling 3.00GiB
2020-10-03 11:44:47.219810: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 7.05GiB
2020-10-03 11:44:47.219818: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_:<PHONE_NUMBER> memory_limit_:<PHONE_NUMBER> available bytes: 128 curr_region_allocation_bytes_:<PHONE_NUMBER>4
2020-10-03 11:44:47.219835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit:<PHONE_NUMBER>
InUse:<PHONE_NUMBER>
MaxInUse:<PHONE_NUMBER>
NumAllocs:                       30989
MaxAllocSize:<PHONE_NUMBER>
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0
2020-10-03 11:44:47.219868: W tensorflow/core/common_runtime/bfc_allocator.cc:439] _______************************************************___________
2020-10-03 11:44:47.219907: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at transpose_op.cc:198 : Resource exhausted: OOM when allocating tensor with shape[256,128,128,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/yiwen/pjs/200902_rimrl_v3/ws/main.py", line 158, in
app.run(main)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/home/yiwen/pjs/200902_rimrl_v3/ws/main.py", line 149, in main
train_loop.run(num_episodes=FLAGS.num_episodes_per_eval)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/acme/environment_loop.py", line 99, in run
self._actor.update()
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/acme/agents/agent.py", line 87, in update
self._learner.step()
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/acme/agents/tf/d4pg/learning.py", line 251, in step
fetches = self._step()
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in call
result = self._call(*args, **kwds)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
return self._stateless_fn(*args, **kwds)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2844, in call
return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1847, in _filtered_call
cancellation_manager=cancellation_manager)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 550, in call
ctx=ctx)
File "/home/yiwen/anaconda3/envs/acme/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[256,64,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node swap_in_gradient_tape/conv2_d/convolution_2/Conv2DBackpropFilter_0}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[Op:__inference__step_10907]
Function call stack:
_step
[reverb/cc/platform/default/server.cc:64] Shutting down replay server
W1003 11:44:52.483008<PHONE_NUMBER>94688 client.py:112] Writer-object deleted without calling .close explicitly.
[reverb/cc/writer.cc:244] Received error when closing the stream: [14] Socket closed

Code

> `
> """Example running D4PG on the OpenAI Gym."""
> import os
> import sys
> path_env=os.path.join(os.getcwd(),"pybullet_env")
> sys.path.append(path_env)
>
> from typing import Mapping, Sequence
> from absl import app
> from absl import flags
> import acme
> from acme import specs
> from acme import types
> from acme import wrappers
> from acme.agents.tf import actors
> from acme.agents.tf import d4pg
> from acme.tf import networks
> from acme.tf import utils as tf2_utils
> import tensorflow as tf
> import dm_env
> import gym
> import numpy as np
> import sonnet as snt
> from pybullet_env.My_Env import yw_robotics_env
>
>
>
> FLAGS = flags.FLAGS
> flags.DEFINE_integer('num_episodes', 100000,
>                      'Number of training episodes to run for.')
> flags.DEFINE_integer('num_episodes_per_eval', 50,
>                      'Number of training episodes to run between evaluation '
>                      'episodes.')
>
>
> def make_environment(
>     task: str = 'MountainCarContinuous-v0') -> dm_env.Environment:
>   """Creates an OpenAI Gym environment."""
>
>   # Load the gym environment.
>   # environment = gym.make(task)
>   environment = yw_robotics_env("yw_insert_v1img3cm")
>
>   # Make sure the environment obeys the dm_env.Environment interface.
>   environment = wrappers.GymWrapper(environment)
>   environment = wrappers.SinglePrecisionWrapper(environment)
>
>   return environment
>
>
> # The default settings in this network factory will work well for the
> # MountainCarContinuous-v0 task but may need to be tuned for others. In
> # particular, the vmin/vmax and num_atoms hyperparameters should be set to
> # give the distributional critic a good dynamic range over possible discounted
> # returns. Note that this is very different than the scale of immediate rewards.
> def make_networks(
>     action_spec: specs.BoundedArray,
>     policy_layer_sizes: Sequence[int] = (128, 128, 128),
>     critic_layer_sizes: Sequence[int] = (128, 128, 128),
>     vmin: float = -150.,
>     vmax: float = 150.,
>     num_atoms: int = 51,
> ) -> Mapping[str, types.TensorTransformation]:
>   """Creates the networks used by the agent."""
>
>   # Get total number of action dimensions from action spec.
>   num_dimensions = np.prod(action_spec.shape, dtype=int)
>
>   # Create the shared observation network; here simply a state-less operation.
>   # observation_network = tf2_utils.batch_concat
>
>   observation_network = snt.Sequential(
>       [
>       snt.Conv2D(64,4),
>       snt.Conv2D(64,4),
>       snt.Conv2D(64,4),
>       snt.Conv2D(64,4),
>       snt.Flatten(),
>       ]
>   )
>
>
>   # Create the policy network.
>   policy_network = snt.Sequential([
>       networks.LayerNormMLP(policy_layer_sizes, activate_final=False),
>       networks.NearZeroInitializedLinear(num_dimensions),
>       networks.TanhToSpec(action_spec),
>   ])
>
>   # Create the critic network.
>   critic_network = snt.Sequential([
>       # The multiplexer concatenates the observations/actions.
>       networks.CriticMultiplexer(),
>       networks.LayerNormMLP(critic_layer_sizes, activate_final=True),
>       networks.DiscreteValuedHead(vmin, vmax, num_atoms),
>   ])
>
>   return {
>       'policy': policy_network,
>       'critic': critic_network,
>       'observation': observation_network,
>   }
>
>
> def main(_):
>   # Create an environment, grab the spec, and use it to create networks.
>   environment = make_environment()
>   environment_spec = specs.make_environment_spec(environment)
>   agent_networks = make_networks(environment_spec.actions)
>
>   # Construct the agent.
>   agent = d4pg.D4PG(
>       environment_spec=environment_spec,
>       policy_network=agent_networks['policy'],
>       critic_network=agent_networks['critic'],
>       observation_network=agent_networks['observation'],
>       sigma=1.0,  # pytype: disable=wrong-arg-types
>   )
>
>   # Create the environment loop used for training.
>   train_loop = acme.EnvironmentLoop(environment, agent, label='train_loop')
>
>   # Create the evaluation policy.
>   # eval_policy = snt.Sequential([
>   #     agent_networks['observation'],
>   #     agent_networks['policy'],
>   # ])
>
>   # # Create the evaluation actor and loop.
>   # eval_actor = actors.FeedForwardActor(policy_network=eval_policy)
>   # eval_env = make_environment()
>   # eval_loop = acme.EnvironmentLoop(eval_env, eval_actor, label='eval_loop')
>
>   for _ in range(FLAGS.num_episodes // FLAGS.num_episodes_per_eval):
>     train_loop.run(num_episodes=FLAGS.num_episodes_per_eval)
>     # eval_loop.run(num_episodes=1)
>
>   tf.saved_model.save(agent_networks['policy'], "saved_model/policy")
>   tf.saved_model.save(agent_networks['observation'], "saved_model/obs")
>   tf.saved_model.save(agent_networks['critic'], "saved_model/crit")
>
>
> if __name__ == '__main__':
>   app.run(main)
>
> `

Sorry for not replying earlier. This question is quite old and I assume it's probably not relevant any more. I'll close it as I don't think it's worth spending time to debug this.
If you still have the issue, I'd encourage you to try our JAX agents and let us know if you encounter further problems.


IOS 17 - Attachments - When you reopen the image, the preload splash screen rotates indefinitely

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open a New Expensify app
Log in with any account
Navigate to any conversation where there are 4-5 attached images or send them
Open the last image
Navigate to another conversation
Return to the conversation with the images
Open the last image opened in step 4

Expected Result:
When you reopen an image in a conversation, the preload splash screen should disappear when the image is fully loaded
Actual Result:
When reopening an image in the conversation, the prereload splash screen spins infinitely and won't go away
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[x] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.74-1
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/78819774/ffd038e9-1a7f-452f-b159-7c98a5ebc81b
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub

Can't repro


Minor: Refine doc comments for BuiltinScalarFunction::return_dimension

Which issue does this PR close?
Related to https://github.com/apache/arrow-datafusion/pull/7008
Rationale for this change
I only just now realized what "dimension" meant in the context of ListArray / Arrays so I tried to document it
What changes are included in this PR?
Doc comments

Are these changes tested?

Are there any user-facing changes?

@izveigor or @jayzhan211 I wonder if you have some time to review this PR?

Thanks @izveigor  -- I'll keep chipping away at keeping the codebase readable and documented :)

I thought that ListArray is 1D...
But, this definition looks good to me too.

I have different thoughts on this, List() is 1D, List(List()) is 2D, Int64 is 0D, dimension is only meaningful to List, not non-List type,

@jayzhan211  That is what I was trying to say.

But, If we want a new definition of dimensions, it is also fine.

My intention was not to change the definition but to capture what this function does. I am pretty sure that given the input int64 this function would return 1

@alamb In your opinion, which one do you prefer? I think both are fine but need to be consistent everywhere

@alamb In your opinion, which one do you prefer?

Int64 is 0D,
List() is 1D
List(List()) is 2D,

Makes the most sense to me given my understanding of multi-dimensional array terminology.
I don't know the code well enough to understand how large a code change updating to that definition would be

I don't think there are many to change so changing the code might not be a problem. We just need to have a consensus on the definition of list dimensions, also add them to code comment and documentation
These are some code I know that is related to List dimensions
https://github.com/apache/arrow-datafusion/blob/11b7b5c215012231e5768fc5be3445c0254d0169/datafusion/expr/src/built_in_function.rs#L439-L460
https://github.com/apache/arrow-datafusion/blob/11b7b5c215012231e5768fc5be3445c0254d0169/datafusion/physical-expr/src/array_expressions.rs#L128-L162
https://github.com/apache/arrow-datafusion/blob/11b7b5c215012231e5768fc5be3445c0254d0169/datafusion/expr/src/built_in_function.rs#L494-L517
@izveigor Do you know any code that considers the list dimensions?


Proposal to add flow-type definitions to flow-typed

I am using following flow-types in my project, can someone help me put them to flow-typed? I do not have experience with adding flow-typed definitions for modules.
/**
 * Once filled out, we encourage you to share your work with the
 * community by sending a pull request to:
 * https://github.com/flowtype/flow-typed
 */

declare module 'styled-system' {
  declare export type NumberOrString = number | string;

  declare export type Responsive = NumberOrString | Array<NumberOrString>;

  declare export type ResponsiveBoolean = boolean | Array<boolean>;

  declare export type Width = {
    /** width */
    width?: Responsive,
    /** width */
    w?: Responsive,
  };

  declare export type Space = {
    /** margin */
    m?: Responsive,
    /** margin-top */
    mt?: Responsive,
    /** margin-right */
    mr?: Responsive,
    /** margin-bottom */
    mb?: Responsive,
    /** margin-left */
    ml?: Responsive,
    /** margin-x */
    mx?: Responsive,
    /** margin-y */
    my?: Responsive,
    /** padding */
    p?: Responsive,
    /** padding-top */
    pt?: Responsive,
    /** padding-right */
    pr?: Responsive,
    /** padding-bottom */
    pb?: Responsive,
    /** padding-left */
    pl?: Responsive,
    /** padding-x */
    px?: Responsive,
    /** padding-y */
    py?: Responsive,
  };

  declare export type FontSize = {
    /** fontSize */
    fontSize?: Responsive,
    /** fontSize */
    f?: Responsive,
  };

  declare export type Color = {
    color?: Responsive,
    /** background */
    bg?: Responsive,
  };

  declare export type TextAlign = {
    align?: Responsive,
  };

  declare export type FontWeight = {
    fontWeight?: NumberOrString,
  };

  declare export type LineHeight = {
    lineHeight?: NumberOrString,
  };

  declare export type AlignItems = {
    align?: Responsive,
  };

  declare export type JustifyContent = {
    justify?: Responsive,
  };

  declare export type FlexWrap = {
    wrap?: ResponsiveBoolean,
  };

  declare export type FlexDirection = {
    flexDirection?: Responsive,
  };

  declare export type Flex = {
    flex?: Responsive,
  };

  declare export type AlignSelf = {
    alignSelf?: Responsive,
  };

  declare export type MaxWidth = {
    maxWidth?: NumberOrString,
  };

  declare export type BorderRadius = {
    borderRadius?: NumberOrString,
  };

  declare export type BorderWidth = {
    borderWidth?: NumberOrString,
    borderTop?: boolean,
    borderRight?: boolean,
    borderBottom?: boolean,
    borderLeft?: boolean,
  };

  declare export type BorderColor = {
    borderColor?: string,
  };

  declare export type BoxShadow = {
    boxShadow?: NumberOrString,
  };

  declare export type Hover = {
    hover?: Object,
  };

  declare export type Focus = {
    focus?: Object,
  };

  declare export type Active = {
    active?: Object,
  };

  declare export type Disabled = {
    disabledStyle?: Object,
  };

  // custom
  declare export type Order = {
    order?: Responsive,
  };

  declare export type Props =
    Width &
    Space &
    FontSize &
    Color &
    TextAlign &
    FontWeight &
    LineHeight &
    AlignItems &
    JustifyContent &
    FlexWrap &
    FlexDirection &
    Flex &
    AlignSelf &
    MaxWidth &
    BorderRadius &
    BorderWidth &
    BorderColor &
    BoxShadow &
    Hover &
    Focus &
    Active &
    Disabled;
}

declare module 'grid-styled' {
  declare export type Is = {is?: React.ElementType};

  declare export type BoxProps = Is &
    Width &
    Space &
    FontSize &
    Color &
    Flex &
    Order;

  declare export type FlexProps = BoxProps &
    FlexWrap &
    FlexDirection &
    AlignItems &
    JustifyContent;
}

Thanks for the proposal and sorry no one has chimed in here. I'm going to close this out since this won't likely be a change in this repo


Remove SQL fields

Hi all,
Here is a simple request :
Collection.select(:id).near("New York", 10)
The SQL request generated is :
SELECT "collection"."id", collection.*, 6371.0 * 2 * ASIN(SQRT(POWER(SIN((40.7127753 - collection.latitude) * PI() / 180 / 2), 2) + COS(40.7127753 * PI() / 180) * COS(collection.latitude * PI() / 180) * POWER(SIN((-74.0059728 - collection.longitude) * PI() / 180 / 2), 2))) AS distance, MOD(CAST((ATAN2( ((collection.longitude - -74.0059728) / 57.2957795), ((collection.latitude - 40.7127753) / 57.2957795)) * 57.2957795) + 360 AS decimal), 360) AS bearing FROM "collection" WHERE (collection.latitude BETWEEN 40.622843139408126 AND 40.80270746059187 AND collection.longitude BETWEEN -74.12461854589681 AND -73.88732705410318 AND (6371.0 * 2 * ASIN(SQRT(POWER(SIN((40.7127753 - collection.latitude) * PI() / 180 / 2), 2) + COS(40.7127753 * PI() / 180) * COS(collection.latitude * PI() / 180) * POWER(SIN((-74.0059728 - collection.longitude) * PI() / 180 / 2), 2)))) BETWEEN 0.0 AND 10) ORDER BY distance ASC
The problem is that i make a projection only on the :id field but we can see that collection.*, distance and bearing fields are selected too.
Is there a solution to remove distance and bearing field please ?

Collection.select(:id).near("New York", 10)

I found the problem. It was due to the sort. I did it to resolve the problem :
Collection.select(:id).near("New York", 10, order: false)


Update fs2-core to 2.5.8

Updates co.fs2:fs2-core from 2.5.7 to 2.5.8.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "co.fs2", artifactId = "fs2-core" } ]

labels: library-update, semver-patch

Superseded by #34.


Handle collections without optional item_assets property

Description
Veda-docs example stac_ipyleaflet notebook is throwing exception
'message': 'Error caught with collection', 'error': KeyError('item_assets')
Surfaced in veda-docs PR https://github.com/NASA-IMPACT/veda-docs/pull/160
I think
has_cog = True if data["item_assets"] else False
could be updated to something like has_cog = True if data.get("item_assets", None) else False
AC

[ ] https://github.com/NASA-IMPACT/veda-docs/blob/main/notebooks/tutorials/stac_ipyleaflet.ipynb is runnable

I think has_cog = True if data["item_assets"] else False could be updated to something like has_cog = True if data.get("item_assets", None) else False

Perhaps simply this? has_cog = "item_assets" in data


webpack 4 e.search error

I recently migrated to web pack 4 and found this error:
[Error] TypeError: value.search is not a function. (In 'value.search(pattern)', 'value.search' is undefined)
	hasHtmlTags (utils.js:31)
	getLocalizedElement (utils.js:26)
	render (index.jsx:151)
	finishClassComponent (react-dom.development.js:7873)
	performUnitOfWork (react-dom.development.js:10224)
	workLoop (react-dom.development.js:10288)
	callCallback (react-dom.development.js:542)
	dispatchEvent
	invokeGuardedCallbackDev (react-dom.development.js:581)
	invokeGuardedCallback (react-dom.development.js:438)
	renderRoot (react-dom.development.js:10366)
	performWorkOnRoot (react-dom.development.js:11014)
	performWork (react-dom.development.js:10967)
	requestWork (react-dom.development.js:10878)
	scheduleWorkImpl (react-dom.development.js:10732)
	enqueueSetState (react-dom.development.js:6212)
	setState (react.development.js:237)
	onStateChange (connectAdvanced.js:215)
	onStateChange
	dispatch (createStore.js:178)
	(función anónima) (App.js:128)
	promiseReactionJob

This is the only library that presents this error its something bad or could be a web pack bug?

Closing for now as no further information has been provided.


Drop Synapse schemas

Resolves #3600
What is being addressed
Init and Drop results and temp schemas in ohdsi synapse data source

/test