Getting error Enabling Large Model Support may avoid this failure although I am enabling LMS

I am getting the following error message, and have tried everything to the best of my knowledge to solve this issue. My code and full error file can be found on this link . I am getting the following error report. I would be grateful for any help you can provide. Thank you
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
/opt/gridengine/default/spool/compute-0-3/job_scripts/108309: line 8: $'\342\200\213': command not found
/opt/gridengine/default/spool/compute-0-3/job_scripts/108309: line 10: $'\342\200\213': command not found
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
2023-08-23 01:05:55.059242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-08-23 01:05:57.035024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
2023-08-23 01:05:57.048181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7
2023-08-23 01:06:00.498131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2023-08-23 01:06:00.616351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:00.620182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0
coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s
2023-08-23 01:06:00.620215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-08-23 01:06:00.620254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-08-23 01:06:00.659120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2023-08-23 01:06:00.749331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2023-08-23 01:06:00.804330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2023-08-23 01:06:00.848649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2023-08-23 01:06:00.848698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-08-23 01:06:00.848818: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:00.850190: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:00.851426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0
2023-08-23 01:06:00.863760: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-08-23 01:06:00.889080: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency:<PHONE_NUMBER> Hz
2023-08-23 01:06:00.893827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55bd9465b360 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-08-23 01:06:00.893864: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-08-23 01:06:01.085878: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:01.086821: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55bd946c1ca0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-08-23 01:06:01.086845: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, Compute Capability 8.0
2023-08-23 01:06:01.087126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:01.087938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0
coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s
2023-08-23 01:06:01.087966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-08-23 01:06:01.087979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-08-23 01:06:01.087997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2023-08-23 01:06:01.088005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2023-08-23 01:06:01.088013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2023-08-23 01:06:01.088021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2023-08-23 01:06:01.088028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-08-23 01:06:01.088084: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:01.088847: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:06:01.089579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0
2023-08-23 01:06:01.089608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-08-23 01:10:45.778914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-08-23 01:10:45.779150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105]      0
2023-08-23 01:10:45.779162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0:   N
2023-08-23 01:10:45.779879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:10:45.780875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-23 01:10:45.782233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37896 MB memory) -> physical GPU (device: 0, name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, pci bus id: 2c8f7:00:00.0, compute capability: 8.0)
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
2023-08-23 01:11:37.778808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-08-23 01:13:39.025763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-08-23 01:23:06.888341: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: ptxas exited with non-zero error code 65280, output: ptxas fatal   : Value 'sm_80' is not defined for option 'gpu-name'
Relying on driver to perform ptx compilation. This message will be only logged once.
2023-08-23 01:24:39.942612: W tensorflow/core/kernels/gpu_utils.cc:48] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2023-08-23 01:25:45.789332: W tensorflow/core/common_runtime/bfc_allocator.cc:801] Allocator (gpu_host_bfc) ran out of memory trying to allocate 6.09GiB (rounded to<PHONE_NUMBER>).  Current allocation summary follows.
2023-08-23 01:25:45.789898: I tensorflow/core/common_runtime/bfc_allocator.cc:1288] BFCAllocator dump for gpu_host_bfc
2023-08-23 01:25:45.789916: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (256): 	Total Chunks: 1762, Chunks in use: 1762. 440.5KiB allocated for chunks. 440.5KiB in use in bin. 81.4KiB client-requested in use in bin.
2023-08-23 01:25:45.789924: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (512): 	Total Chunks: 84, Chunks in use: 84. 54.8KiB allocated for chunks. 54.8KiB in use in bin. 41.5KiB client-requested in use in bin.
2023-08-23 01:25:45.789931: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1024): 	Total Chunks: 30, Chunks in use: 30. 34.5KiB allocated for chunks. 34.5KiB in use in bin. 30.4KiB client-requested in use in bin.
2023-08-23 01:25:45.789937: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2048): 	Total Chunks: 15, Chunks in use: 15. 42.8KiB allocated for chunks. 42.8KiB in use in bin. 40.2KiB client-requested in use in bin.
2023-08-23 01:25:45.789943: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4096): 	Total Chunks: 51, Chunks in use: 51. 241.5KiB allocated for chunks. 241.5KiB in use in bin. 240.5KiB client-requested in use in bin.
2023-08-23 01:25:45.789948: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8192): 	Total Chunks: 150, Chunks in use: 150. 2.14MiB allocated for chunks. 2.14MiB in use in bin. 2.12MiB client-requested in use in bin.
2023-08-23 01:25:45.789955: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16384): 	Total Chunks: 6, Chunks in use: 6. 137.2KiB allocated for chunks. 137.2KiB in use in bin. 136.8KiB client-requested in use in bin.
2023-08-23 01:25:45.789961: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (32768): 	Total Chunks: 21, Chunks in use: 21. 978.0KiB allocated for chunks. 978.0KiB in use in bin. 975.2KiB client-requested in use in bin.
2023-08-23 01:25:45.789966: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (65536): 	Total Chunks: 27, Chunks in use: 27. 2.58MiB allocated for chunks. 2.58MiB in use in bin. 2.57MiB client-requested in use in bin.
2023-08-23 01:25:45.789972: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (131072): 	Total Chunks: 49, Chunks in use: 49. 9.03MiB allocated for chunks. 9.03MiB in use in bin. 9.02MiB client-requested in use in bin.
2023-08-23 01:25:45.789978: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (262144): 	Total Chunks: 47, Chunks in use: 47. 14.46MiB allocated for chunks. 14.46MiB in use in bin. 14.23MiB client-requested in use in bin.
2023-08-23 01:25:45.789984: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (524288): 	Total Chunks: 39, Chunks in use: 39. 29.41MiB allocated for chunks. 29.41MiB in use in bin. 29.41MiB client-requested in use in bin.
2023-08-23 01:25:45.790002: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-08-23 01:25:45.790008: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2097152): 	Total Chunks: 6, Chunks in use: 6. 15.48MiB allocated for chunks. 15.48MiB in use in bin. 14.66MiB client-requested in use in bin.
2023-08-23 01:25:45.790014: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4194304): 	Total Chunks: 74, Chunks in use: 74. 438.40MiB allocated for chunks. 438.40MiB in use in bin. 432.52MiB client-requested in use in bin.
2023-08-23 01:25:45.790020: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8388608): 	Total Chunks: 7, Chunks in use: 7. 69.55MiB allocated for chunks. 69.55MiB in use in bin. 65.25MiB client-requested in use in bin.
2023-08-23 01:25:45.790026: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16777216): 	Total Chunks: 5, Chunks in use: 5. 114.75MiB allocated for chunks. 114.75MiB in use in bin. 114.75MiB client-requested in use in bin.
2023-08-23 01:25:45.790031: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (33554432): 	Total Chunks: 76, Chunks in use: 76. 3.53GiB allocated for chunks. 3.53GiB in use in bin. 3.53GiB client-requested in use in bin.
2023-08-23 01:25:45.790037: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (67108864): 	Total Chunks: 10, Chunks in use: 10. 835.00MiB allocated for chunks. 835.00MiB in use in bin. 771.00MiB client-requested in use in bin.
2023-08-23 01:25:45.790042: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (134217728): 	Total Chunks: 9, Chunks in use: 8. 1.54GiB allocated for chunks. 1.41GiB in use in bin. 1.41GiB client-requested in use in bin.
2023-08-23 01:25:45.790048: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (268435456): 	Total Chunks: 74, Chunks in use: 73. 57.42GiB allocated for chunks. 52.88GiB in use in bin. 52.70GiB client-requested in use in bin.
2023-08-23 01:25:45.790054: I tensorflow/core/common_runtime/bfc_allocator.cc:1311] Bin for 6.09GiB was 256.00MiB, Chunk State:
2023-08-23 01:25:45.790067: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 4.55GiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev:   Size: 1.50GiB | Requested Size: 1.50GiB | in_use: 1 | bin_num: -1
.........
2023-08-23 01:25:45.802092: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size<PHONE_NUMBER> totalling 9.06GiB
2023-08-23 01:25:45.802097: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 59.32GiB
2023-08-23 01:25:45.802101: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>6
2023-08-23 01:25:45.802137: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8
MaxInUse:<PHONE_NUMBER>8
NumAllocs:                          2884
MaxAllocSize:<PHONE_NUMBER>
BytesInactive:                         0
BytesActive:<PHONE_NUMBER>8
PeakBytesActive:<PHONE_NUMBER>8
BytesReclaimed:                        0
NumSingleReclaims:                     0
NumFullReclaims:                       0
NumDefrags:                            0
BytesDefragged:                        0
2023-08-23 01:25:45.802262: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************************************************************_______
2023-08-23 01:25:45.802268: W tensorflow/core/common_runtime/bfc_allocator.cc:809] Enabling Large Model Support may avoid this failure.
2023-08-23 01:25:45.802510: F tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:153] Check failed: result.ok()
/opt/gridengine/default/spool/compute-0-3/job_scripts/108309: line 13: 122674 Aborted                 python $PYTHON_SCRIPT

You are hitting the limit on the CPU (GPU host) side and need to increase the amount of memory allocated for the tensor swap space. This section of the readme discusses it: https://github.com/IBM/tensorflow-large-model-support#increase-the-system-memory-gpu-host-memory-allocation
I also see that you are using TensorFlow 2.1.0. The 2.2.0 level had enhancements and bug fixes and I would suggest using that level.
Also note, that TensorFlow Large Model Support development ended more than 3 years ago and it is not actively maintained or supported.

@junaidjawaid1 in https://github.com/junaidjawaid1/3d_U-Net-TFLMS/issues/1#issuecomment-1692219388 you mentioned that you are still getting this error.
Have you set the TF_GPU_HOST_MEM_LIMIT_IN_MB env var in the environment you are running this in before starting your Python job as documented here?
If you successfully set it higher, the Limit and related values in the Stats part of your error message will be higher values:
Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8

The Allocator (gpu_host_bfc) ran out of memory part of your error message is noting that it's not your GPU that's running out of memory, but rather the amount of memory you have allocated for tensor swap space is running out. This is from the gpu_host_bfc part. If a GPU runs out of memory, that value is different.

@junaidjawaid1 in junaidjawaid1/3d_U-Net-TFLMS#1 (comment) you mentioned that you are still getting this error.
Have you set the TF_GPU_HOST_MEM_LIMIT_IN_MB env var in the environment you are running this in before starting your Python job as documented here?
If you successfully set it higher, the Limit and related values in the Stats part of your error message will be higher values:
Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8

The Allocator (gpu_host_bfc) ran out of memory part of your error message is noting that it's not your GPU that's running out of memory, but rather the amount of memory you have allocated for tensor swap space is running out. This is from the gpu_host_bfc part. If a GPU runs out of memory, that value is different.

Thank you so much for the response. I did not run TF_GPU_HOST_MEM_LIMIT_IN_MB in the env before running the Python job, I was running it in the script. I'll try it out.

@junaidjawaid1 in junaidjawaid1/3d_U-Net-TFLMS#1 (comment) you mentioned that you are still getting this error.
Have you set the TF_GPU_HOST_MEM_LIMIT_IN_MB env var in the environment you are running this in before starting your Python job as documented here?
If you successfully set it higher, the Limit and related values in the Stats part of your error message will be higher values:
Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8

The Allocator (gpu_host_bfc) ran out of memory part of your error message is noting that it's not your GPU that's running out of memory, but rather the amount of memory you have allocated for tensor swap space is running out. This is from the gpu_host_bfc part. If a GPU runs out of memory, that value is different.

I set the TF_GPU_HOST_MEM_LIMIT_IN_MB=500000 in the environment before running the Python script, but I am still getting the same error
**Error: 2023-09-15 04:16:57.287529: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************2023-09-15 04:16:57.287535: W tensorflow/core/common_runtime/bfc_allocator.cc:809] Enabling Large Model Support may avoid this failure.
**********************************************_______
2023-09-15 04:16:57.287700: F tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:153] Check failed: result.ok()
/opt/gridengine/default/spool/compute-0-3/job_scripts/108504: line 13: 226121 Aborted                 python $PYTHON_SCRIPT
I also tried the 2.2.0 version and have been struggling with another error for quite a while, even though the Cuda libraries seem to be the same as those downloaded with the 2.1.0 version which did not have this problem.
Error : 023-09-15 03:50:50.643371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1250] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37681 MB memory) -> physical GPU (device: 0, name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, pci bus id: 2c8f7:00:00.0, compute capability: 8.0)
2023-09-15 03:50:51.042966: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device
Please note that I am using Nvidia A100 vitually divided in to two 40GB  GPUs and I am using one of them. Also note that in both cases (2.1.0 and 2.2.0) cudatoolkit and cudnn were downloaded from the same channels as the tensorflow-gpu and tensorflow-base.

@junaidjawaid1 There is not enough information in your last comment for me to give any guidance. I don't know if the GPU ran out of memory or if the host allocator (GPU swap space) ran out of memory. Two sections of your error output would help.

The error containing the memory allocator name which will be either the GPU or the host. The line looks like this in your above output:

Allocator (gpu_host_bfc) ran out of memory trying to allocate 6.09GiB (rounded to<PHONE_NUMBER>). Current allocation summary follows.

The stats section, like this:

Sum Total of in-use chunks: 59.32GiB
2023-08-23 01:25:45.802101: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>6
2023-08-23 01:25:45.802137: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8
MaxInUse:<PHONE_NUMBER>8
NumAllocs: 2884
MaxAllocSize:<PHONE_NUMBER>
BytesInactive: 0
BytesActive:<PHONE_NUMBER>8
PeakBytesActive:<PHONE_NUMBER>8
BytesReclaimed: 0
NumSingleReclaims: 0
NumFullReclaims: 0
NumDefrags: 0
BytesDefragged: 0

This is the error I got after setting  TF_GPU_HOST_MEM_LIMIT_IN_MB=500000 in the environment before running the python script
**2023-09-15 18:18:54.619307: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>6
2023-09-15 18:18:54.619343: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>8
MaxInUse:<PHONE_NUMBER>8
NumAllocs:                          2885
MaxAllocSize:<PHONE_NUMBER>
BytesInactive:                         0
BytesActive:<PHONE_NUMBER>8
PeakBytesActive:<PHONE_NUMBER>8
BytesReclaimed:                        0
NumSingleReclaims:                     0
NumFullReclaims:                       0
NumDefrags:                            0
BytesDefragged:                        0
2023-09-15 18:18:54.619480: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *******************************************************************************************_______
2023-09-15 18:18:54.619485: W tensorflow/core/common_runtime/bfc_allocator.cc:809] Enabling Large Model Support may avoid this failure.
2023-09-15 18:18:54.619692: F tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:153] Check failed: result.ok()
/opt/gridengine/default/spool/compute-0-3/job_scripts/108549: line 13: 171842 Aborted                 python $PYTHON_SCRIPT

You did not include the section from the error message which lists the allocator. However, given the Limit:<PHONE_NUMBER>6 stayed the same I assume it is likely the host allocator.
In your latest error message the Limit:<PHONE_NUMBER>6 has not changed and is the same as previous errors at 64GB, which is the default. It seems the setting of TF_GPU_HOST_MEM_LIMIT_IN_MB=500000 in your environment isn't taking effect.
Are you using export TF_GPU_HOST_MEM_LIMIT_IN_MB=500000 ?
Secondly, do you have 488.28 GB memory free on your system?

I have a new update. I also included os.environ['TF_GPU_HOST_MEM_LIMIT_IN_MB'] = '500000' in the python script and it seems to be taking an effect, I got new error which concerns memory fragmentation. Note that this error is with tensroflow 2.1, and I am unable to run GPU with tensorflow 2.2. I will try again to get everything working and will get back to you. Thank you so much for the support.
2023-09-18 13:01:56.036865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-09-18 13:04:19.088625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-18 13:15:48.266102: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: ptxas exited with non-zero error code 65280, output: ptxas fatal   : Value 'sm_80' is not defined for option 'gpu-name'
Relying on driver to perform ptx compilation. This message will be only logged once.
2023-09-18 13:17:08.612305: W tensorflow/core/kernels/gpu_utils.cc:48] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2023-09-18 13:20:13.335178: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2023-09-18 13:20:23.843480: W tensorflow/core/common_runtime/bfc_allocator.cc:801] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.84GiB (rounded to<PHONE_NUMBER>).  Current allocation summary follows.
2023-09-18 13:20:23.843915: I tensorflow/core/common_runtime/bfc_allocator.cc:1288] BFCAllocator dump for GPU_0_bfc
2023-09-18 13:20:23.843932: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (256): 	Total Chunks: 181, Chunks in use: 174. 45.2KiB allocated for chunks. 43.5KiB in use in bin. 3.7KiB client-requested in use in bin.
2023-09-18 13:20:23.843939: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (512): 	Total Chunks: 2, Chunks in use: 0. 1.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843945: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1024): 	Total Chunks: 6, Chunks in use: 4. 7.0KiB allocated for chunks. 4.2KiB in use in bin. 3.9KiB client-requested in use in bin.
2023-09-18 13:20:23.843950: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2048): 	Total Chunks: 1, Chunks in use: 0. 2.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843956: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4096): 	Total Chunks: 3, Chunks in use: 3. 13.5KiB allocated for chunks. 13.5KiB in use in bin. 13.5KiB client-requested in use in bin.
2023-09-18 13:20:23.843962: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8192): 	Total Chunks: 13, Chunks in use: 12. 184.0KiB allocated for chunks. 168.8KiB in use in bin. 167.5KiB client-requested in use in bin.
2023-09-18 13:20:23.843968: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843972: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843977: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843981: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.843987: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (262144): 	Total Chunks: 8, Chunks in use: 6. 2.18MiB allocated for chunks. 1.62MiB in use in bin. 1.62MiB client-requested in use in bin.
2023-09-18 13:20:23.843997: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (524288): 	Total Chunks: 1, Chunks in use: 0. 610.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844002: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844007: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844011: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844016: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8388608): 	Total Chunks: 1, Chunks in use: 0. 14.96MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844021: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16777216): 	Total Chunks: 1, Chunks in use: 0. 17.01MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844026: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844030: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844035: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-18 13:20:23.844041: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (268435456): 	Total Chunks: 7, Chunks in use: 4. 36.97GiB allocated for chunks. 27.38GiB in use in bin. 27.38GiB client-requested in use in bin.
2023-09-18 13:20:23.844047: I tensorflow/core/common_runtime/bfc_allocator.cc:1311] Bin for 6.84GiB was 256.00MiB, Chunk State:
2023-09-18 13:20:23.844061: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 1.16GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-18 13:20:23.844067: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 2.31GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-18 13:20:23.844073: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 6.13GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-18 13:20:23.844077: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 1048576
2023-09-18 13:20:23.844085: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6200000 of size 1280 next 1
2023-09-18 13:20:23.844090: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6200500 of size 283904 next 371
2023-09-18 13:20:23.844095: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d6245a00 of size 283904 next 363
2023-09-18 13:20:23.844099: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628af00 of size 10752 next 2142
2023-09-18 13:20:23.844104: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628d900 of size 256 next 2180
2023-09-18 13:20:23.844108: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628da00 of size 256 next 356
2023-09-18 13:20:23.844112: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628db00 of size 256 next 2167
2023-09-18 13:20:23.844119: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628dc00 of size 256 next 360
2023-09-18 13:20:23.844123: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d628dd00 of size 256 next 342
2023-09-18 13:20:23.844127: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628de00 of size 256 next 183
2023-09-18 13:20:23.844131: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628df00 of size 256 next 2309
2023-09-18 13:20:23.844135: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e000 of size 256 next 2183
2023-09-18 13:20:23.844138: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e100 of size 256 next 344
2023-09-18 13:20:23.844142: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e200 of size 256 next 2301
2023-09-18 13:20:23.844146: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d628e300 of size 256 next 359
2023-09-18 13:20:23.844150: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e400 of size 256 next 1801
2023-09-18 13:20:23.844154: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e500 of size 256 next 364
2023-09-18 13:20:23.844158: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e600 of size 256 next 372
2023-09-18 13:20:23.844162: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e700 of size 256 next 1800
2023-09-18 13:20:23.844166: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e800 of size 256 next 2296
2023-09-18 13:20:23.844170: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628e900 of size 256 next 2176
2023-09-18 13:20:23.844174: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628ea00 of size 256 next 144
2023-09-18 13:20:23.844177: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628eb00 of size 256 next 351
2023-09-18 13:20:23.844181: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628ec00 of size 256 next 2172
2023-09-18 13:20:23.844185: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628ed00 of size 256 next 2136
2023-09-18 13:20:23.844189: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628ee00 of size 256 next 2163
2023-09-18 13:20:23.844193: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628ef00 of size 256 next 353
2023-09-18 13:20:23.844197: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f000 of size 256 next 347
2023-09-18 13:20:23.844201: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f100 of size 256 next 354
2023-09-18 13:20:23.844205: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f200 of size 256 next 349
2023-09-18 13:20:23.844208: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f300 of size 256 next 2144
2023-09-18 13:20:23.844212: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f400 of size 256 next 2317
2023-09-18 13:20:23.844216: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f500 of size 256 next 2145
2023-09-18 13:20:23.844220: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f600 of size 256 next 350
2023-09-18 13:20:23.844224: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f700 of size 256 next 2177
2023-09-18 13:20:23.844228: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f800 of size 256 next 2308
2023-09-18 13:20:23.844232: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628f900 of size 256 next 2170
2023-09-18 13:20:23.844236: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628fa00 of size 256 next 2178
2023-09-18 13:20:23.844239: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d628fb00 of size 256 next 2141
2023-09-18 13:20:23.844246: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628fc00 of size 256 next 2139
2023-09-18 13:20:23.844250: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628fd00 of size 256 next 2315
2023-09-18 13:20:23.844254: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d628fe00 of size 15616 next 2182
2023-09-18 13:20:23.844258: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6293b00 of size 15616 next 2162
2023-09-18 13:20:23.844262: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297800 of size 256 next 343
2023-09-18 13:20:23.844266: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297900 of size 256 next 209
2023-09-18 13:20:23.844270: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297a00 of size 256 next 417
2023-09-18 13:20:23.844273: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297b00 of size 256 next 2265
2023-09-18 13:20:23.844277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297c00 of size 256 next 2259
2023-09-18 13:20:23.844281: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6297d00 of size 256 next 2263
2023-09-18 13:20:23.844285: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d6297e00 of size 768 next 2262
2023-09-18 13:20:23.844289: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6298100 of size 256 next 2171
2023-09-18 13:20:23.844293: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6298200 of size 256 next 2169
2023-09-18 13:20:23.844297: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6298300 of size 256 next 2261
2023-09-18 13:20:23.844301: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6298400 of size 256 next 2257
2023-09-18 13:20:23.844304: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d6298500 of size 256 next 2267
2023-09-18 13:20:23.844308: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6298600 of size 15616 next 246
2023-09-18 13:20:23.844312: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c300 of size 256 next 306
2023-09-18 13:20:23.844316: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c400 of size 256 next 341
2023-09-18 13:20:23.844320: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c500 of size 256 next 2166
2023-09-18 13:20:23.844324: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d629c600 of size 256 next 2285
2023-09-18 13:20:23.844328: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c700 of size 256 next 419
2023-09-18 13:20:23.844332: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c800 of size 256 next 2283
2023-09-18 13:20:23.844336: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629c900 of size 256 next 315
2023-09-18 13:20:23.844340: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629ca00 of size 4608 next 2165
2023-09-18 13:20:23.844344: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629dc00 of size 1024 next 2091
2023-09-18 13:20:23.844348: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629e000 of size 256 next 2310
2023-09-18 13:20:23.844352: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d629e100 of size 768 next 2200
2023-09-18 13:20:23.844356: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629e400 of size 256 next 362
2023-09-18 13:20:23.844360: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629e500 of size 256 next 361
2023-09-18 13:20:23.844364: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629e600 of size 256 next 2268
2023-09-18 13:20:23.844370: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629e700 of size 256 next 2174
2023-09-18 13:20:23.844374: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d629e800 of size 1024 next 2307
2023-09-18 13:20:23.844378: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629ec00 of size 256 next 2188
2023-09-18 13:20:23.844382: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629ed00 of size 256 next 2197
2023-09-18 13:20:23.844386: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629ee00 of size 1024 next 2181
2023-09-18 13:20:23.844390: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629f200 of size 1024 next 2190
2023-09-18 13:20:23.844394: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629f600 of size 256 next 373
2023-09-18 13:20:23.844398: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629f700 of size 256 next 348
2023-09-18 13:20:23.844402: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d629f800 of size 1792 next 357
2023-09-18 13:20:23.844405: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d629ff00 of size 256 next 2186
2023-09-18 13:20:23.844409: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0000 of size 256 next 2218
2023-09-18 13:20:23.844413: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0100 of size 256 next 367
2023-09-18 13:20:23.844417: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0200 of size 256 next 284
2023-09-18 13:20:23.844421: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0300 of size 256 next 374
2023-09-18 13:20:23.844425: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0400 of size 256 next 2194
2023-09-18 13:20:23.844429: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0500 of size 256 next 2175
2023-09-18 13:20:23.844433: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0600 of size 256 next 2220
2023-09-18 13:20:23.844437: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d62a0700 of size 256 next 2191
2023-09-18 13:20:23.844451: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a0800 of size 4608 next 2189
2023-09-18 13:20:23.844455: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a1a00 of size 4608 next 2198
2023-09-18 13:20:23.844459: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a2c00 of size 15616 next 2208
2023-09-18 13:20:23.844463: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62a6900 of size 15616 next 2299
2023-09-18 13:20:23.844467: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62aa600 of size 15616 next 334
2023-09-18 13:20:23.844471: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62ae300 of size 15616 next 365
2023-09-18 13:20:23.844475: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d62b2000 of size 15616 next 2291
2023-09-18 13:20:23.844479: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d62b5d00 of size 303872 next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.844483: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 2097152
2023-09-18 13:20:23.844487: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6400000 of size 283904 next 2199
2023-09-18 13:20:23.844491: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6445500 of size 15616 next 2179
2023-09-18 13:20:23.844495: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d6449200 of size 15616 next 2147
2023-09-18 13:20:23.844499: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d644cf00 of size 10752 next 2316
2023-09-18 13:20:23.844503: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d644f900 of size 10752 next 346
2023-09-18 13:20:23.844509: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6452300 of size 283904 next 2151
2023-09-18 13:20:23.844514: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6497800 of size 283904 next 2286
2023-09-18 13:20:23.844518: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d64dcd00 of size 283904 next 366
2023-09-18 13:20:23.844522: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b65d6522200 of size 283904 next 2184
2023-09-18 13:20:23.844526: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b65d6567700 of size 624896 next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.844529: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 33554432
2023-09-18 13:20:23.844534: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b673e000000 of size 17836544 next 1439
2023-09-18 13:20:23.844538: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f102a00 of size 256 next 1440
2023-09-18 13:20:23.844542: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b673f102b00 of size 2048 next 1448
2023-09-18 13:20:23.844546: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103300 of size 256 next 1449
2023-09-18 13:20:23.844550: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103400 of size 256 next 1450
2023-09-18 13:20:23.844554: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103500 of size 256 next 1451
2023-09-18 13:20:23.844557: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103600 of size 256 next 1452
2023-09-18 13:20:23.844561: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103700 of size 256 next 1453
2023-09-18 13:20:23.844565: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103800 of size 256 next 1454
2023-09-18 13:20:23.844569: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103900 of size 256 next 1455
2023-09-18 13:20:23.844573: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103a00 of size 256 next 1456
2023-09-18 13:20:23.844577: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103b00 of size 256 next 1457
2023-09-18 13:20:23.844580: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103c00 of size 256 next 1458
2023-09-18 13:20:23.844584: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103d00 of size 256 next 1459
2023-09-18 13:20:23.844588: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103e00 of size 256 next 1460
2023-09-18 13:20:23.844592: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f103f00 of size 256 next 1461
2023-09-18 13:20:23.844596: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104000 of size 256 next 1462
2023-09-18 13:20:23.844600: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104100 of size 256 next 1463
2023-09-18 13:20:23.844603: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104200 of size 256 next 1464
2023-09-18 13:20:23.844607: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104300 of size 256 next 1465
2023-09-18 13:20:23.844611: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104400 of size 256 next 1466
2023-09-18 13:20:23.844615: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104500 of size 256 next 1467
2023-09-18 13:20:23.844619: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104600 of size 256 next 1468
2023-09-18 13:20:23.844623: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104700 of size 256 next 1469
2023-09-18 13:20:23.844626: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104800 of size 256 next 1470
2023-09-18 13:20:23.844630: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104900 of size 256 next 1471
2023-09-18 13:20:23.844641: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104a00 of size 256 next 1472
2023-09-18 13:20:23.844645: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104b00 of size 256 next 1473
2023-09-18 13:20:23.844649: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104c00 of size 256 next 1474
2023-09-18 13:20:23.844653: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104d00 of size 256 next 1475
2023-09-18 13:20:23.844657: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104e00 of size 256 next 1476
2023-09-18 13:20:23.844661: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f104f00 of size 256 next 1477
2023-09-18 13:20:23.844665: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105000 of size 256 next 1478
2023-09-18 13:20:23.844668: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105100 of size 256 next 1479
2023-09-18 13:20:23.844672: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105200 of size 256 next 1480
2023-09-18 13:20:23.844676: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105300 of size 256 next 1481
2023-09-18 13:20:23.844680: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105400 of size 256 next 1482
2023-09-18 13:20:23.844684: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105500 of size 256 next 1483
2023-09-18 13:20:23.844688: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105600 of size 256 next 1484
2023-09-18 13:20:23.844692: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105700 of size 256 next 1485
2023-09-18 13:20:23.844695: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105800 of size 256 next 1486
2023-09-18 13:20:23.844699: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105900 of size 256 next 1487
2023-09-18 13:20:23.844703: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105a00 of size 256 next 1488
2023-09-18 13:20:23.844707: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105b00 of size 256 next 1489
2023-09-18 13:20:23.844711: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105c00 of size 256 next 1490
2023-09-18 13:20:23.844715: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105d00 of size 256 next 1491
2023-09-18 13:20:23.844719: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105e00 of size 256 next 1492
2023-09-18 13:20:23.844723: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f105f00 of size 256 next 1493
2023-09-18 13:20:23.844726: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106000 of size 256 next 1494
2023-09-18 13:20:23.844730: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106100 of size 256 next 1495
2023-09-18 13:20:23.844734: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106200 of size 256 next 1496
2023-09-18 13:20:23.844738: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106300 of size 256 next 1497
2023-09-18 13:20:23.844742: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106400 of size 256 next 1498
2023-09-18 13:20:23.844746: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106500 of size 256 next 1499
2023-09-18 13:20:23.844750: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106600 of size 256 next 1500
2023-09-18 13:20:23.844754: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106700 of size 256 next 1501
2023-09-18 13:20:23.844758: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106800 of size 256 next 1502
2023-09-18 13:20:23.844764: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106900 of size 256 next 1503
2023-09-18 13:20:23.844768: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106a00 of size 256 next 1504
2023-09-18 13:20:23.844772: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106b00 of size 256 next 1505
2023-09-18 13:20:23.844776: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106c00 of size 256 next 1506
2023-09-18 13:20:23.844780: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106d00 of size 256 next 1507
2023-09-18 13:20:23.844784: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106e00 of size 256 next 1508
2023-09-18 13:20:23.844787: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f106f00 of size 256 next 1509
2023-09-18 13:20:23.844791: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107000 of size 256 next 1510
2023-09-18 13:20:23.844795: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107100 of size 256 next 1511
2023-09-18 13:20:23.844799: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107200 of size 256 next 1512
2023-09-18 13:20:23.844803: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107300 of size 256 next 1513
2023-09-18 13:20:23.844807: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107400 of size 256 next 1514
2023-09-18 13:20:23.844810: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107500 of size 256 next 1515
2023-09-18 13:20:23.844814: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107600 of size 256 next 1516
2023-09-18 13:20:23.844818: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107700 of size 256 next 1517
2023-09-18 13:20:23.844822: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107800 of size 256 next 1518
2023-09-18 13:20:23.844826: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107900 of size 256 next 1519
2023-09-18 13:20:23.844830: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107a00 of size 256 next 1520
2023-09-18 13:20:23.844834: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107b00 of size 256 next 1521
2023-09-18 13:20:23.844837: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107c00 of size 256 next 1522
2023-09-18 13:20:23.844841: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107d00 of size 256 next 1523
2023-09-18 13:20:23.844845: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107e00 of size 256 next 1524
2023-09-18 13:20:23.844849: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f107f00 of size 256 next 1525
2023-09-18 13:20:23.844853: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108000 of size 256 next 1526
2023-09-18 13:20:23.844857: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108100 of size 256 next 1527
2023-09-18 13:20:23.844861: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108200 of size 256 next 1528
2023-09-18 13:20:23.844864: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108300 of size 256 next 1529
2023-09-18 13:20:23.844868: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108400 of size 256 next 1530
2023-09-18 13:20:23.844872: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108500 of size 256 next 1531
2023-09-18 13:20:23.844876: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108600 of size 256 next 1532
2023-09-18 13:20:23.844880: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108700 of size 256 next 1533
2023-09-18 13:20:23.844884: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108800 of size 256 next 1534
2023-09-18 13:20:23.844890: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108900 of size 256 next 1535
2023-09-18 13:20:23.844894: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108a00 of size 256 next 1536
2023-09-18 13:20:23.844898: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108b00 of size 256 next 1537
2023-09-18 13:20:23.844901: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108c00 of size 256 next 1538
2023-09-18 13:20:23.844905: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108d00 of size 256 next 1539
2023-09-18 13:20:23.844909: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108e00 of size 256 next 1540
2023-09-18 13:20:23.844913: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f108f00 of size 256 next 1541
2023-09-18 13:20:23.844917: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109000 of size 256 next 1542
2023-09-18 13:20:23.844921: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109100 of size 256 next 1543
2023-09-18 13:20:23.844924: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109200 of size 256 next 1544
2023-09-18 13:20:23.844928: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109300 of size 256 next 1545
2023-09-18 13:20:23.844932: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109400 of size 256 next 1546
2023-09-18 13:20:23.844936: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109500 of size 256 next 1547
2023-09-18 13:20:23.844940: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109600 of size 256 next 1548
2023-09-18 13:20:23.844944: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109700 of size 256 next 1549
2023-09-18 13:20:23.844948: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109800 of size 256 next 1550
2023-09-18 13:20:23.844951: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109900 of size 256 next 1551
2023-09-18 13:20:23.844955: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109a00 of size 256 next 1552
2023-09-18 13:20:23.844959: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109b00 of size 256 next 1553
2023-09-18 13:20:23.844963: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b673f109c00 of size 256 next 1554
2023-09-18 13:20:23.844967: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b673f109d00 of size 256 next 1555
2023-09-18 13:20:23.844971: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b673f109e00 of size 15688192 next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.844975: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>
2023-09-18 13:20:23.844979: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b753a000000 of size<PHONE_NUMBER> next 2161
2023-09-18 13:20:23.844983: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b76f0000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.844987: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>4
2023-09-18 13:20:23.844991: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b773a000000 of size<PHONE_NUMBER> next 2195
2023-09-18 13:20:23.844995: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b78f0000000 of size<PHONE_NUMBER> next 2202
2023-09-18 13:20:23.844999: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b7aa6000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.845003: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>0
2023-09-18 13:20:23.845007: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2b9da4000000 of size<PHONE_NUMBER> next 1598
2023-09-18 13:20:23.845014: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b9f5a000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-18 13:20:23.845018: I tensorflow/core/common_runtime/bfc_allocator.cc:1349]      Summary of in-use Chunks by size:
2023-09-18 13:20:23.845024: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 174 Chunks of size 256 totalling 43.5KiB
2023-09-18 13:20:23.845029: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 1024 totalling 3.0KiB
2023-09-18 13:20:23.845033: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 1280 totalling 1.2KiB
2023-09-18 13:20:23.845037: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 4608 totalling 13.5KiB
2023-09-18 13:20:23.845041: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 10752 totalling 31.5KiB
2023-09-18 13:20:23.845046: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 9 Chunks of size 15616 totalling 137.2KiB
2023-09-18 13:20:23.845050: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 6 Chunks of size 283904 totalling 1.62MiB
2023-09-18 13:20:23.845055: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 4 Chunks of size<PHONE_NUMBER> totalling 27.38GiB
2023-09-18 13:20:23.845059: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 27.38GiB
2023-09-18 13:20:23.845064: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>72
2023-09-18 13:20:23.845079: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>6
MaxInUse:<PHONE_NUMBER>6
NumAllocs:                         10380
MaxAllocSize:<PHONE_NUMBER>
BytesInactive:                         0
BytesActive:<PHONE_NUMBER>6
PeakBytesActive:<PHONE_NUMBER>6
BytesReclaimed:<PHONE_NUMBER>76
NumSingleReclaims:                     6
NumFullReclaims:                       4
NumDefrags:                            0
BytesDefragged:                        0
2023-09-18 13:20:23.845096: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************************___********************________________
2023-09-18 13:20:23.845102: W tensorflow/core/common_runtime/bfc_allocator.cc:816] Enough free memory to satisfy the allocation request exists but it is fragmented. Enabling Large Model Support defragmentation may avoid this failure.
2023-09-18 13:20:23.845178: F tensorflow/stream_executor/cuda/cuda_driver.cc:216] Check failed: is_host_ptr == points_to_host_memory (0 vs. 1)dst pointer is not actually on GPU: (nil)
/opt/gridengine/default/spool/compute-0-3/job_scripts/108619: line 13: 85507 Aborted                 python $PYTHON_SCRIPT

Large Model Support defragmentation in 2.1.0 was buggy and the feature was removed for 2.2.0. I would recommend that it be disabled and not used.

Large Model Support defragmentation in 2.1.0 was buggy and the feature was removed for 2.2.0. I would recommend that it be disabled and not used.

The program keeps stopping due to a memory fragmentation error, can you please suggest me something to by pass this. I cant find any option except for enabling defragmentation which I do not want to do

You are right in that it is failing with fragmentation. Most of the memory of the GPU is taken by 4 tensors of size 6.8GB each. The process failed in trying to allocate space for another 6.8GB. Unfortunately in some these cases the memory allocation and memory free pattern driven by a specific neural net leads to fragmentation and you can't do much of anything about it. In the case above large model support has already swapped out as many tensors as it can out of the GPU. All that would be left are things that are needed for input to your next network calculation or are needed for the outputs of that calculation.
In this case you could actually try enabling large model support defragmentation in 2.1.0. It may work, it did work for some of the models we tried. It just ran into some cases where memory pointers were not updated correctly and things would crash.

Thank you for the feedback. I tried running the program with defragmentation enabled and now I am getting a new error. I looked it up, but there's nothing useful regarding this. Please let me know if you have any idea about this.
2023-09-20 15:05:14.369254: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109700 of size 256 next 1549
2023-09-20 15:05:14.369259: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109800 of size 256 next 1550
2023-09-20 15:05:14.369264: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109900 of size 256 next 1551
2023-09-20 15:05:14.369269: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109a00 of size 256 next 1552
2023-09-20 15:05:14.369273: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b4c9b109b00 of size 256 next 1553
2023-09-20 15:05:14.369277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109c00 of size 256 next 1554
2023-09-20 15:05:14.369292: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109d00 of size 256 next 1555
2023-09-20 15:05:14.369299: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b4c9b109e00 of size 15688192 next<PHONE_NUMBER>3709551615
2023-09-20 15:05:14.369303: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>
2023-09-20 15:05:14.369308: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5a9a000000 of size<PHONE_NUMBER> next 371
2023-09-20 15:05:14.369312: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b5c50000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 15:05:14.369316: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>4
2023-09-20 15:05:14.369321: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5cac000000 of size<PHONE_NUMBER> next 2187
2023-09-20 15:05:14.369325: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5e62000000 of size<PHONE_NUMBER> next 2184
2023-09-20 15:05:14.369329: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b6018000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 15:05:14.369333: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>0
2023-09-20 15:05:14.369339: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b830c000000 of size<PHONE_NUMBER> next 1596
2023-09-20 15:05:14.369344: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2b84c2000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 15:05:14.369348: I tensorflow/core/common_runtime/bfc_allocator.cc:1349]      Summary of in-use Chunks by size:
2023-09-20 15:05:14.369354: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 174 Chunks of size 256 totalling 43.5KiB
2023-09-20 15:05:14.369360: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 1024 totalling 3.0KiB
2023-09-20 15:05:14.369364: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 1280 totalling 1.2KiB
2023-09-20 15:05:14.369368: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 4608 totalling 13.5KiB
2023-09-20 15:05:14.369373: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 10752 totalling 31.5KiB
2023-09-20 15:05:14.369378: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 9 Chunks of size 15616 totalling 137.2KiB
2023-09-20 15:05:14.369382: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 6 Chunks of size 283904 totalling 1.62MiB
2023-09-20 15:05:14.369386: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 4 Chunks of size<PHONE_NUMBER> totalling 27.38GiB
2023-09-20 15:05:14.369391: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 27.38GiB
2023-09-20 15:05:14.369396: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>72
2023-09-20 15:05:14.369413: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>6
MaxInUse:<PHONE_NUMBER>0
NumAllocs:                         10381
MaxAllocSize:<PHONE_NUMBER>
BytesInactive:                         0
BytesActive:<PHONE_NUMBER>6
PeakBytesActive:<PHONE_NUMBER>6
BytesReclaimed:<PHONE_NUMBER>60
NumSingleReclaims:                     6
NumFullReclaims:                       4
NumDefrags:                            0
BytesDefragged:                        0
2023-09-20 15:05:14.369432: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************************___********************________________
2023-09-20 15:05:14.369564: F tensorflow/stream_executor/cuda/cuda_driver.cc:216] Check failed: is_host_ptr == points_to_host_memory (0 vs. 1)dst pointer is not actually on GPU: (nil)
/opt/gridengine/default/spool/compute-0-3/job_scripts/108639: line 13: 99471 Aborted                 python $PYTHON_SCRIPT

Thank you for the feedback. I tried running the program with defragmentation enabled and now I am getting a new error. I looked it up, but there's nothing useful regarding this. Please let me know if you have any idea about this.
2023-09-20 15:05:14.369254: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109700 of size 256 next 1549 2023-09-20 15:05:14.369259: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109800 of size 256 next 1550 2023-09-20 15:05:14.369264: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109900 of size 256 next 1551 2023-09-20 15:05:14.369269: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109a00 of size 256 next 1552 2023-09-20 15:05:14.369273: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 2b4c9b109b00 of size 256 next 1553 2023-09-20 15:05:14.369277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109c00 of size 256 next 1554 2023-09-20 15:05:14.369292: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b4c9b109d00 of size 256 next 1555 2023-09-20 15:05:14.369299: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 2b4c9b109e00 of size 15688192 next<PHONE_NUMBER>3709551615 2023-09-20 15:05:14.369303: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER> 2023-09-20 15:05:14.369308: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5a9a000000 of size<PHONE_NUMBER> next 371 2023-09-20 15:05:14.369312: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 2b5c50000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615 2023-09-20 15:05:14.369316: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>4 2023-09-20 15:05:14.369321: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5cac000000 of size<PHONE_NUMBER> next 2187 2023-09-20 15:05:14.369325: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b5e62000000 of size<PHONE_NUMBER> next 2184 2023-09-20 15:05:14.369329: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 2b6018000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615 2023-09-20 15:05:14.369333: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>0 2023-09-20 15:05:14.369339: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2b830c000000 of size<PHONE_NUMBER> next 1596 2023-09-20 15:05:14.369344: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free at 2b84c2000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615 2023-09-20 15:05:14.369348: I tensorflow/core/common_runtime/bfc_allocator.cc:1349] Summary of in-use Chunks by size: 2023-09-20 15:05:14.369354: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 174 Chunks of size 256 totalling 43.5KiB 2023-09-20 15:05:14.369360: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 1024 totalling 3.0KiB 2023-09-20 15:05:14.369364: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 1280 totalling 1.2KiB 2023-09-20 15:05:14.369368: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 4608 totalling 13.5KiB 2023-09-20 15:05:14.369373: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 10752 totalling 31.5KiB 2023-09-20 15:05:14.369378: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 9 Chunks of size 15616 totalling 137.2KiB 2023-09-20 15:05:14.369382: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 6 Chunks of size 283904 totalling 1.62MiB 2023-09-20 15:05:14.369386: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 4 Chunks of size<PHONE_NUMBER> totalling 27.38GiB 2023-09-20 15:05:14.369391: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 27.38GiB 2023-09-20 15:05:14.369396: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>72 2023-09-20 15:05:14.369413: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats: Limit:<PHONE_NUMBER>6 InUse:<PHONE_NUMBER>6 MaxInUse:<PHONE_NUMBER>0 NumAllocs: 10381 MaxAllocSize:<PHONE_NUMBER> BytesInactive: 0 BytesActive:<PHONE_NUMBER>6 PeakBytesActive:<PHONE_NUMBER>6 BytesReclaimed:<PHONE_NUMBER>60 NumSingleReclaims: 6 NumFullReclaims: 4 NumDefrags: 0 BytesDefragged: 0
2023-09-20 15:05:14.369432: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************************************************_____________ 2023-09-20 15:05:14.369564: F tensorflow/stream_executor/cuda/cuda_driver.cc:216] Check failed: is_host_ptr == points_to_host_memory (0 vs. 1)dst pointer is not actually on GPU: (nil) /opt/gridengine/default/spool/compute-0-3/job_scripts/108639: line 13: 99471 Aborted python $PYTHON_SCRIPT

@smatzek is it possible that I am getting this error cause I am using Nvidia A100 which is a Cuda 11 card, is it possibly a driver issue? I have tried everything I could but cannot solve this issue.

I doubt it is because of the A100. Cuda 11 may play into it though. It has been nearly 3.5 years since this code has been maintained and I do not remember if we used Cuda 10 or 11 at that time.
When the process runs out of memory, the bottom of the error message can vary. There usually is little value in that part of the error message. In your output above I don't see any fragmentation being attempted:
NumDefrags: 0
BytesDefragged: 0

It may be that the in-use memory regions surrounding the free regions were unmoveable. The error message is in the comment is also missing the top lines that state how much was trying to be allocated.
The OOM (out of memory) is still just likely due to the memory usage pattern of the model, the size of the tensors it requires, and the order in which TensorFlow is executing the instructions.

Thanks you for your response. This is the whole error message I am getting.
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
/opt/gridengine/default/spool/compute-0-3/job_scripts/108641: line 8: $'\342\200\213': command not found
/opt/gridengine/default/spool/compute-0-3/job_scripts/108641: line 10: $'\342\200\213': command not found
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'nvidia'
dirname: missing operand
Try 'dirname --help' for more information.
2023-09-20 15:54:35.233773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-09-20 15:54:39.354047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
2023-09-20 15:54:39.377620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7
2023-09-20 15:54:46.245237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2023-09-20 15:54:46.363581: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.367404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0
coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s
2023-09-20 15:54:46.367459: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-09-20 15:54:46.367501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-09-20 15:54:46.430341: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2023-09-20 15:54:46.542567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2023-09-20 15:54:46.600798: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2023-09-20 15:54:46.660684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2023-09-20 15:54:46.660765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-20 15:54:46.660909: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.662197: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.663343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0
2023-09-20 15:54:46.675533: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-09-20 15:54:46.701330: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency:<PHONE_NUMBER> Hz
2023-09-20 15:54:46.705703: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637df87a220 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-09-20 15:54:46.705744: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-09-20 15:54:46.871866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.873023: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637df8e0b80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-09-20 15:54:46.873047: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, Compute Capability 8.0
2023-09-20 15:54:46.873356: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.874187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0
coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s
2023-09-20 15:54:46.874219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-09-20 15:54:46.874233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-09-20 15:54:46.874252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2023-09-20 15:54:46.874261: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2023-09-20 15:54:46.874270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2023-09-20 15:54:46.874278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2023-09-20 15:54:46.874285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-20 15:54:46.874343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.875107: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 15:54:46.875817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0
2023-09-20 15:54:46.875847: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2023-09-20 16:00:53.004545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-09-20 16:00:53.004814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105]      0
2023-09-20 16:00:53.004832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0:   N
2023-09-20 16:00:53.005351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 16:00:53.006312: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-20 16:00:53.007510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 37896 MB memory) -> physical GPU (device: 0, name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, pci bus id: 2c8f7:00:00.0, compute capability: 8.0)
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
2023-09-20 16:01:46.443811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2023-09-20 16:03:30.999882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-09-20 16:14:31.988353: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: ptxas exited with non-zero error code 65280, output: ptxas fatal   : Value 'sm_80' is not defined for option 'gpu-name'
Relying on driver to perform ptx compilation. This message will be only logged once.
2023-09-20 16:15:56.600984: W tensorflow/core/kernels/gpu_utils.cc:48] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2023-09-20 16:18:55.411831: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2023-09-20 16:19:05.926236: W tensorflow/core/common_runtime/bfc_allocator.cc:801] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.84GiB (rounded to<PHONE_NUMBER>).  Current allocation summary follows.
2023-09-20 16:19:05.926604: I tensorflow/core/common_runtime/bfc_allocator.cc:1288] BFCAllocator dump for GPU_0_bfc
2023-09-20 16:19:05.926622: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (256): 	Total Chunks: 180, Chunks in use: 174. 45.0KiB allocated for chunks. 43.5KiB in use in bin. 3.7KiB client-requested in use in bin.
2023-09-20 16:19:05.926631: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (512): 	Total Chunks: 1, Chunks in use: 0. 768B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926637: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1024): 	Total Chunks: 7, Chunks in use: 4. 8.0KiB allocated for chunks. 4.2KiB in use in bin. 3.9KiB client-requested in use in bin.
2023-09-20 16:19:05.926643: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2048): 	Total Chunks: 1, Chunks in use: 0. 2.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926649: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4096): 	Total Chunks: 3, Chunks in use: 3. 13.5KiB allocated for chunks. 13.5KiB in use in bin. 13.5KiB client-requested in use in bin.
2023-09-20 16:19:05.926656: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8192): 	Total Chunks: 13, Chunks in use: 12. 184.0KiB allocated for chunks. 168.8KiB in use in bin. 167.5KiB client-requested in use in bin.
2023-09-20 16:19:05.926661: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926665: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926670: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926675: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926680: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (262144): 	Total Chunks: 8, Chunks in use: 6. 2.18MiB allocated for chunks. 1.62MiB in use in bin. 1.62MiB client-requested in use in bin.
2023-09-20 16:19:05.926690: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (524288): 	Total Chunks: 1, Chunks in use: 0. 610.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926695: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926700: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926704: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926709: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (8388608): 	Total Chunks: 1, Chunks in use: 0. 14.96MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926714: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (16777216): 	Total Chunks: 1, Chunks in use: 0. 17.01MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926719: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926724: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926728: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-09-20 16:19:05.926734: I tensorflow/core/common_runtime/bfc_allocator.cc:1295] Bin (268435456): 	Total Chunks: 7, Chunks in use: 4. 36.97GiB allocated for chunks. 27.38GiB in use in bin. 27.38GiB client-requested in use in bin.
2023-09-20 16:19:05.926740: I tensorflow/core/common_runtime/bfc_allocator.cc:1311] Bin for 6.84GiB was 256.00MiB, Chunk State:
2023-09-20 16:19:05.926760: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 1.16GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-20 16:19:05.926767: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 2.31GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-20 16:19:05.926773: I tensorflow/core/common_runtime/bfc_allocator.cc:1317]   Size: 6.13GiB | Requested Size: 384.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 6.84GiB | Requested Size: 6.84GiB | in_use: 1 | bin_num: -1
2023-09-20 16:19:05.926777: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 1048576
2023-09-20 16:19:05.926787: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUse  at 2ac5aa200000 of size 1280 next 1
2023-09-20 16:19:05.926817: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa200500 of size 283904 next 372
2023-09-20 16:19:05.926834: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa245a00 of size 10752 next 183
2023-09-20 16:19:05.926843: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa248400 of size 283904 next 341
2023-09-20 16:19:05.926862: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28d900 of size 256 next 2033
2023-09-20 16:19:05.926871: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28da00 of size 256 next 1682
2023-09-20 16:19:05.926875: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa28db00 of size 256 next 2267
2023-09-20 16:19:05.926890: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28dc00 of size 256 next 2288
2023-09-20 16:19:05.926904: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28dd00 of size 256 next 2154
2023-09-20 16:19:05.926912: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28de00 of size 256 next 344
2023-09-20 16:19:05.926922: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28df00 of size 256 next 2174
2023-09-20 16:19:05.926932: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e000 of size 256 next 2301
2023-09-20 16:19:05.926938: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e100 of size 256 next 2172
2023-09-20 16:19:05.926943: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e200 of size 256 next 2309
2023-09-20 16:19:05.926947: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa28e300 of size 256 next 2265
2023-09-20 16:19:05.926955: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e400 of size 256 next 2285
2023-09-20 16:19:05.926960: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e500 of size 256 next 2162
2023-09-20 16:19:05.926972: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e600 of size 256 next 2161
2023-09-20 16:19:05.926981: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e700 of size 256 next 1789
2023-09-20 16:19:05.926987: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e800 of size 256 next 155
2023-09-20 16:19:05.926998: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28e900 of size 256 next 144
2023-09-20 16:19:05.927010: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28ea00 of size 256 next 347
2023-09-20 16:19:05.927015: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28eb00 of size 256 next 354
2023-09-20 16:19:05.927020: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28ec00 of size 256 next 2173
2023-09-20 16:19:05.927024: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28ed00 of size 256 next 2139
2023-09-20 16:19:05.927029: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28ee00 of size 256 next 2179
2023-09-20 16:19:05.927038: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28ef00 of size 256 next 351
2023-09-20 16:19:05.927043: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f000 of size 256 next 350
2023-09-20 16:19:05.927047: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f100 of size 256 next 353
2023-09-20 16:19:05.927051: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa28f200 of size 256 next 2170
2023-09-20 16:19:05.927060: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f300 of size 256 next 2137
2023-09-20 16:19:05.927071: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f400 of size 256 next 2315
2023-09-20 16:19:05.927078: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f500 of size 256 next 2181
2023-09-20 16:19:05.927084: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f600 of size 256 next 2182
2023-09-20 16:19:05.927098: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f700 of size 256 next 2270
2023-09-20 16:19:05.927105: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f800 of size 256 next 2165
2023-09-20 16:19:05.927111: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28f900 of size 256 next 2125
2023-09-20 16:19:05.927116: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28fa00 of size 256 next 2163
2023-09-20 16:19:05.927121: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28fb00 of size 256 next 349
2023-09-20 16:19:05.927128: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28fc00 of size 256 next 2143
2023-09-20 16:19:05.927132: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa28fd00 of size 256 next 2259
2023-09-20 16:19:05.927137: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa28fe00 of size 15616 next 2140
2023-09-20 16:19:05.927143: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa293b00 of size 15616 next 2316
2023-09-20 16:19:05.927154: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa297800 of size 256 next 2160
2023-09-20 16:19:05.927160: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa297900 of size 256 next 209
2023-09-20 16:19:05.927166: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa297a00 of size 256 next 2300
2023-09-20 16:19:05.927174: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa297b00 of size 256 next 2293
2023-09-20 16:19:05.927179: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa297c00 of size 256 next 417
2023-09-20 16:19:05.927189: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa297d00 of size 256 next 2262
2023-09-20 16:19:05.927193: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa297e00 of size 768 next 2287
2023-09-20 16:19:05.927197: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298100 of size 256 next 2283
2023-09-20 16:19:05.927201: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298200 of size 256 next 2260
2023-09-20 16:19:05.927204: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298300 of size 256 next 2284
2023-09-20 16:19:05.927209: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298400 of size 256 next 2269
2023-09-20 16:19:05.927218: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298500 of size 256 next 338
2023-09-20 16:19:05.927222: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa298600 of size 15616 next 2168
2023-09-20 16:19:05.927231: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c300 of size 256 next 2176
2023-09-20 16:19:05.927236: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c400 of size 256 next 2167
2023-09-20 16:19:05.927247: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c500 of size 256 next 318
2023-09-20 16:19:05.927255: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c600 of size 256 next 2171
2023-09-20 16:19:05.927260: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c700 of size 256 next 2175
2023-09-20 16:19:05.927273: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c800 of size 256 next 343
2023-09-20 16:19:05.927286: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29c900 of size 256 next 2264
2023-09-20 16:19:05.927294: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29ca00 of size 4608 next 246
2023-09-20 16:19:05.927315: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29dc00 of size 1024 next 306
2023-09-20 16:19:05.927324: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa29e000 of size 1024 next 356
2023-09-20 16:19:05.927335: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29e400 of size 256 next 2195
2023-09-20 16:19:05.927349: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29e500 of size 256 next 2190
2023-09-20 16:19:05.927362: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29e600 of size 256 next 346
2023-09-20 16:19:05.927374: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29e700 of size 256 next 2201
2023-09-20 16:19:05.927398: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa29e800 of size 1024 next 2186
2023-09-20 16:19:05.927409: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29ec00 of size 256 next 363
2023-09-20 16:19:05.927420: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29ed00 of size 256 next 2203
2023-09-20 16:19:05.927428: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29ee00 of size 1024 next 2142
2023-09-20 16:19:05.927456: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29f200 of size 1024 next 2310
2023-09-20 16:19:05.927468: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29f600 of size 256 next 2197
2023-09-20 16:19:05.927475: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29f700 of size 256 next 2178
2023-09-20 16:19:05.927479: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa29f800 of size 1792 next 2200
2023-09-20 16:19:05.927490: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa29ff00 of size 256 next 361
2023-09-20 16:19:05.927497: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0000 of size 256 next 2192
2023-09-20 16:19:05.927510: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0100 of size 256 next 2194
2023-09-20 16:19:05.927522: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0200 of size 256 next 2187
2023-09-20 16:19:05.927528: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0300 of size 256 next 2210
2023-09-20 16:19:05.927538: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0400 of size 256 next 2147
2023-09-20 16:19:05.927548: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0500 of size 256 next 2298
2023-09-20 16:19:05.927557: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0600 of size 256 next 365
2023-09-20 16:19:05.927566: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0700 of size 256 next 2307
2023-09-20 16:19:05.927577: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a0800 of size 4608 next 2191
2023-09-20 16:19:05.927593: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a1a00 of size 4608 next 2205
2023-09-20 16:19:05.927606: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a2c00 of size 15616 next 2196
2023-09-20 16:19:05.927617: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2a6900 of size 15616 next 315
2023-09-20 16:19:05.927622: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2aa600 of size 15616 next 2177
2023-09-20 16:19:05.927627: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2ae300 of size 15616 next 362
2023-09-20 16:19:05.927632: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa2b2000 of size 15616 next 2308
2023-09-20 16:19:05.927636: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa2b5d00 of size 303872 next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.927640: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 2097152
2023-09-20 16:19:05.927645: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa400000 of size 283904 next 364
2023-09-20 16:19:05.927649: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa445500 of size 15616 next 373
2023-09-20 16:19:05.927653: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa449200 of size 15616 next 357
2023-09-20 16:19:05.927664: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa44cf00 of size 10752 next 2199
2023-09-20 16:19:05.927675: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa44f900 of size 10752 next 419
2023-09-20 16:19:05.927680: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa452300 of size 283904 next 2169
2023-09-20 16:19:05.927693: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa497800 of size 283904 next 2145
2023-09-20 16:19:05.927702: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa4dcd00 of size 283904 next 2193
2023-09-20 16:19:05.927706: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac5aa522200 of size 283904 next 334
2023-09-20 16:19:05.927710: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac5aa567700 of size 624896 next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.927714: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size 33554432
2023-09-20 16:19:05.927718: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac704000000 of size 17836544 next 1439
2023-09-20 16:19:05.927722: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705102a00 of size 256 next 1440
2023-09-20 16:19:05.927726: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac705102b00 of size 2048 next 1448
2023-09-20 16:19:05.927733: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103300 of size 256 next 1449
2023-09-20 16:19:05.927739: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103400 of size 256 next 1450
2023-09-20 16:19:05.927751: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103500 of size 256 next 1451
2023-09-20 16:19:05.927765: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103600 of size 256 next 1452
2023-09-20 16:19:05.927771: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103700 of size 256 next 1453
2023-09-20 16:19:05.927776: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103800 of size 256 next 1454
2023-09-20 16:19:05.927782: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103900 of size 256 next 1455
2023-09-20 16:19:05.927793: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103a00 of size 256 next 1456
2023-09-20 16:19:05.927802: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103b00 of size 256 next 1457
2023-09-20 16:19:05.927813: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103c00 of size 256 next 1458
2023-09-20 16:19:05.927822: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103d00 of size 256 next 1459
2023-09-20 16:19:05.927826: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103e00 of size 256 next 1460
2023-09-20 16:19:05.927831: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705103f00 of size 256 next 1461
2023-09-20 16:19:05.927837: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104000 of size 256 next 1462
2023-09-20 16:19:05.927842: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104100 of size 256 next 1463
2023-09-20 16:19:05.927847: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104200 of size 256 next 1464
2023-09-20 16:19:05.927852: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104300 of size 256 next 1465
2023-09-20 16:19:05.927861: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104400 of size 256 next 1466
2023-09-20 16:19:05.927866: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104500 of size 256 next 1467
2023-09-20 16:19:05.927882: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104600 of size 256 next 1468
2023-09-20 16:19:05.927894: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104700 of size 256 next 1469
2023-09-20 16:19:05.927900: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104800 of size 256 next 1470
2023-09-20 16:19:05.927904: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104900 of size 256 next 1471
2023-09-20 16:19:05.927911: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104a00 of size 256 next 1472
2023-09-20 16:19:05.927917: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104b00 of size 256 next 1473
2023-09-20 16:19:05.927922: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104c00 of size 256 next 1474
2023-09-20 16:19:05.927928: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104d00 of size 256 next 1475
2023-09-20 16:19:05.927933: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104e00 of size 256 next 1476
2023-09-20 16:19:05.927939: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705104f00 of size 256 next 1477
2023-09-20 16:19:05.927944: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105000 of size 256 next 1478
2023-09-20 16:19:05.927949: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105100 of size 256 next 1479
2023-09-20 16:19:05.927954: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105200 of size 256 next 1480
2023-09-20 16:19:05.927960: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105300 of size 256 next 1481
2023-09-20 16:19:05.927965: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105400 of size 256 next 1482
2023-09-20 16:19:05.927969: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105500 of size 256 next 1483
2023-09-20 16:19:05.927974: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105600 of size 256 next 1484
2023-09-20 16:19:05.927979: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105700 of size 256 next 1485
2023-09-20 16:19:05.927984: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105800 of size 256 next 1486
2023-09-20 16:19:05.927990: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105900 of size 256 next 1487
2023-09-20 16:19:05.927995: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105a00 of size 256 next 1488
2023-09-20 16:19:05.928000: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105b00 of size 256 next 1489
2023-09-20 16:19:05.928005: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105c00 of size 256 next 1490
2023-09-20 16:19:05.928010: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105d00 of size 256 next 1491
2023-09-20 16:19:05.928015: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105e00 of size 256 next 1492
2023-09-20 16:19:05.928020: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705105f00 of size 256 next 1493
2023-09-20 16:19:05.928025: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106000 of size 256 next 1494
2023-09-20 16:19:05.928030: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106100 of size 256 next 1495
2023-09-20 16:19:05.928035: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106200 of size 256 next 1496
2023-09-20 16:19:05.928040: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106300 of size 256 next 1497
2023-09-20 16:19:05.928046: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106400 of size 256 next 1498
2023-09-20 16:19:05.928051: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106500 of size 256 next 1499
2023-09-20 16:19:05.928056: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106600 of size 256 next 1500
2023-09-20 16:19:05.928061: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106700 of size 256 next 1501
2023-09-20 16:19:05.928066: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106800 of size 256 next 1502
2023-09-20 16:19:05.928071: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106900 of size 256 next 1503
2023-09-20 16:19:05.928079: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106a00 of size 256 next 1504
2023-09-20 16:19:05.928084: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106b00 of size 256 next 1505
2023-09-20 16:19:05.928089: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106c00 of size 256 next 1506
2023-09-20 16:19:05.928094: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106d00 of size 256 next 1507
2023-09-20 16:19:05.928099: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106e00 of size 256 next 1508
2023-09-20 16:19:05.928104: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705106f00 of size 256 next 1509
2023-09-20 16:19:05.928110: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107000 of size 256 next 1510
2023-09-20 16:19:05.928113: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107100 of size 256 next 1511
2023-09-20 16:19:05.928119: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107200 of size 256 next 1512
2023-09-20 16:19:05.928124: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107300 of size 256 next 1513
2023-09-20 16:19:05.928129: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107400 of size 256 next 1514
2023-09-20 16:19:05.928143: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107500 of size 256 next 1515
2023-09-20 16:19:05.928151: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107600 of size 256 next 1516
2023-09-20 16:19:05.928156: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107700 of size 256 next 1517
2023-09-20 16:19:05.928161: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107800 of size 256 next 1518
2023-09-20 16:19:05.928167: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107900 of size 256 next 1519
2023-09-20 16:19:05.928172: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107a00 of size 256 next 1520
2023-09-20 16:19:05.928177: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107b00 of size 256 next 1521
2023-09-20 16:19:05.928183: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107c00 of size 256 next 1522
2023-09-20 16:19:05.928188: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107d00 of size 256 next 1523
2023-09-20 16:19:05.928198: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107e00 of size 256 next 1524
2023-09-20 16:19:05.928207: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705107f00 of size 256 next 1525
2023-09-20 16:19:05.928212: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108000 of size 256 next 1526
2023-09-20 16:19:05.928217: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108100 of size 256 next 1527
2023-09-20 16:19:05.928222: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108200 of size 256 next 1528
2023-09-20 16:19:05.928228: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108300 of size 256 next 1529
2023-09-20 16:19:05.928233: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108400 of size 256 next 1530
2023-09-20 16:19:05.928238: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108500 of size 256 next 1531
2023-09-20 16:19:05.928244: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108600 of size 256 next 1532
2023-09-20 16:19:05.928249: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108700 of size 256 next 1533
2023-09-20 16:19:05.928254: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108800 of size 256 next 1534
2023-09-20 16:19:05.928259: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108900 of size 256 next 1535
2023-09-20 16:19:05.928267: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108a00 of size 256 next 1536
2023-09-20 16:19:05.928272: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108b00 of size 256 next 1537
2023-09-20 16:19:05.928277: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108c00 of size 256 next 1538
2023-09-20 16:19:05.928283: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108d00 of size 256 next 1539
2023-09-20 16:19:05.928288: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108e00 of size 256 next 1540
2023-09-20 16:19:05.928293: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705108f00 of size 256 next 1541
2023-09-20 16:19:05.928298: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109000 of size 256 next 1542
2023-09-20 16:19:05.928303: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109100 of size 256 next 1543
2023-09-20 16:19:05.928309: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109200 of size 256 next 1544
2023-09-20 16:19:05.928314: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109300 of size 256 next 1545
2023-09-20 16:19:05.928318: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109400 of size 256 next 1546
2023-09-20 16:19:05.928322: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109500 of size 256 next 1547
2023-09-20 16:19:05.928327: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109600 of size 256 next 1548
2023-09-20 16:19:05.928333: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109700 of size 256 next 1549
2023-09-20 16:19:05.928338: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109800 of size 256 next 1550
2023-09-20 16:19:05.928343: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109900 of size 256 next 1551
2023-09-20 16:19:05.928349: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109a00 of size 256 next 1552
2023-09-20 16:19:05.928360: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109b00 of size 256 next 1553
2023-09-20 16:19:05.928367: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac705109c00 of size 256 next 1554
2023-09-20 16:19:05.928376: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ac705109d00 of size 256 next 1555
2023-09-20 16:19:05.928380: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ac705109e00 of size 15688192 next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.928384: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>
2023-09-20 16:19:05.928388: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ad4fe000000 of size<PHONE_NUMBER> next 342
2023-09-20 16:19:05.928393: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ad6b4000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.928397: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>4
2023-09-20 16:19:05.928402: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ad6fe000000 of size<PHONE_NUMBER> next 2202
2023-09-20 16:19:05.928406: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2ad8b4000000 of size<PHONE_NUMBER> next 355
2023-09-20 16:19:05.928410: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2ada6a000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.928414: I tensorflow/core/common_runtime/bfc_allocator.cc:1324] Next region of size<PHONE_NUMBER>0
2023-09-20 16:19:05.928418: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] InUseA at 2afd56000000 of size<PHONE_NUMBER> next 1599
2023-09-20 16:19:05.928422: I tensorflow/core/common_runtime/bfc_allocator.cc:1344] Free   at 2aff0c000000 of size<PHONE_NUMBER> next<PHONE_NUMBER>3709551615
2023-09-20 16:19:05.928428: I tensorflow/core/common_runtime/bfc_allocator.cc:1349]      Summary of in-use Chunks by size:
2023-09-20 16:19:05.928435: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 174 Chunks of size 256 totalling 43.5KiB
2023-09-20 16:19:05.928449: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 1024 totalling 3.0KiB
2023-09-20 16:19:05.928453: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 1 Chunks of size 1280 totalling 1.2KiB
2023-09-20 16:19:05.928457: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 4608 totalling 13.5KiB
2023-09-20 16:19:05.928462: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 3 Chunks of size 10752 totalling 31.5KiB
2023-09-20 16:19:05.928466: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 9 Chunks of size 15616 totalling 137.2KiB
2023-09-20 16:19:05.928470: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 6 Chunks of size 283904 totalling 1.62MiB
2023-09-20 16:19:05.928475: I tensorflow/core/common_runtime/bfc_allocator.cc:1352] 4 Chunks of size<PHONE_NUMBER> totalling 27.38GiB
2023-09-20 16:19:05.928479: I tensorflow/core/common_runtime/bfc_allocator.cc:1356] Sum Total of in-use chunks: 27.38GiB
2023-09-20 16:19:05.928484: I tensorflow/core/common_runtime/bfc_allocator.cc:1358] total_region_allocated_bytes_:<PHONE_NUMBER>6 memory_limit_:<PHONE_NUMBER>6 available bytes: 0 curr_region_allocation_bytes_:<PHONE_NUMBER>72
2023-09-20 16:19:05.928504: I tensorflow/core/common_runtime/bfc_allocator.cc:1364] Stats:
Limit:<PHONE_NUMBER>6
InUse:<PHONE_NUMBER>6
MaxInUse:<PHONE_NUMBER>0
NumAllocs:                         10381
MaxAllocSize:<PHONE_NUMBER>
BytesInactive:                         0
BytesActive:<PHONE_NUMBER>6
PeakBytesActive:<PHONE_NUMBER>6
BytesReclaimed:<PHONE_NUMBER>60
NumSingleReclaims:                     6
NumFullReclaims:                       4
NumDefrags:                            0
BytesDefragged:                        0
2023-09-20 16:19:05.928526: W tensorflow/core/common_runtime/bfc_allocator.cc:806] *********************************************************___********************________________
2023-09-20 16:19:05.928652: F tensorflow/stream_executor/cuda/cuda_driver.cc:216] Check failed: is_host_ptr == points_to_host_memory (0 vs. 1)dst pointer is not actually on GPU: (nil)
/opt/gridengine/default/spool/compute-0-3/job_scripts/108641: line 13: 125680 Aborted                 python $PYTHON_SCRIPT

Based on your error message, 2023-09-20 16:19:05.926236: W tensorflow/core/common_runtime/bfc_allocator.cc:801] Allocator (GPU_0_bfc) ran out of memory trying to allocate 6.84GiB (rounded to<PHONE_NUMBER>). Current allocation summary follows. and your current memory allocation, the free ~10GiB of space is too fragmented to allow another ~6.84GiB allocation. At this point TFLMS has already swapped out all the memory it can, the remaining memory allocations are still marked as "Active" for either base-line operation of TensorFlow or are required for the next operation that your model will execute on the GPU.  I do not know what else you can do with your model to try and change its execution order or memory behavior. You may need to reduce the size of your tensors to free up more memory.